Generating Random Datasets
Generating Random Datasets
Challenge
Challenge
Quickly create random datasets with a specific column names and random data.
Since this is a recurring need, I do not want to spend time reinventing this wheel every time.
Since this is a recurring need, I do not want to spend time reinventing this wheel every time.
Use Cases
Use Cases
Currently, I have the following specific use cases in mind, though I believe that many more will present themselves.
Generating Unit Test Data
Generating Unit Test Data
While I am a big believer in Test Driven Development, I often fail due to commitments made to produce something of value to clients. In order to more consistently, as well as more often, apply Test Driven Development principles, I need a way to quickly generate useable test data.
In order to avoid the false confidence that we can get from seeing the same set of unit tests succeed, when appropriate, I like to create additional unit tests using random data.
In order to avoid the false confidence that we can get from seeing the same set of unit tests succeed, when appropriate, I like to create additional unit tests using random data.
Generating Data For Proofs-of-Concepts and Examples
Generating Data For Proofs-of-Concepts and Examples
I often find myself needing a quick dataset for a proof-of-concept, create an example or to collaborate with others while keeping proprietary data private.
Generating Scaled Down Datasets for End-To-End Testing
Generating Scaled Down Datasets for End-To-End Testing
I try to take the Test Driven Development strategy beyond unit testing. While it would be great to always use real-world data for testing, it is often not feasible. For example, I often work on applications that can take hours or even days to run. If I used real-world data in those situations, the time to complete a test-cycle could severely limit my productivity.
Getting Started Before The Client Gives Me Data
Getting Started Before The Client Gives Me Data
I sometimes find myself needing to get started before the data I need is available.
Solution
Solution
The Function
The Function
In[]:=
dsGenerateRandomDataset[assoc_Association,nRows_Integer]:=Block[{eval},eval[key_String,{f_,args_}]:=keyf[Sequence[args]];eval[key_String,{f_}]:=keyf[];Table[Prepend[Association[KeyValueMap[eval,assoc]],<|"ID"id|>],{id,1,nRows}]//Dataset];
Example Usage
Example Usage
Sales Data
Sales Data
If I am working on a project that will analyze product sales by region, I can use this function to generate a dataset of any size.
In[]:=
dsGenerateRandomDataset[<|"Region"{RandomChoice,{"North","South","East","West"}},"Product"{RandomChoice,{"T-Shrits","Pants","Hats","Dresses"}},"Sales"{RandomInteger,{0,100}}|>,10]
Out[]=
Daily Observations
Daily Observations
@Rohit_Namjoshi provided another use case. What if we need a dataset of random observations based on specific distributions. We can do something like:
In[]:=
randomNormal[]:=RandomVariate[NormalDistribution[]];randomPoisson[]:=RandomVariate[PoissonDistribution[3]];dsGenerateRandomDataset[<|"Ob 1"{randomNormal},"Ob 2"{randomPoisson}|>,10][All,<|"Date"DatePlus[DateObject[{2020,1,1}],#["ID"]],#|>&]
Out[]=
Next Steps
Next Steps
Get Feedback from the Community
Get Feedback from the Community
I am posting this to the community hoping to get feedback on:
◼
How are others implementing Test Driven Development strategies
◼
My implementation? Are there better ways to do this?
◼
Other use cases where this solution might apply
Refactor
Refactor
Since I can imagine other uses for the private eval[] function, I should consider making it a readily available reusable function.
Expand this Concept Outside of Datasets
Expand this Concept Outside of Datasets
Consider using this approach for other types of test data generation:
◼
Networks
◼
Matrices
◼
Hierarchical Lists/Associations
◼
etc.