Introduction
Introduction
In this notebook we show how to ingest Airbnb reviews data.
(The data can be used for exemplifying relevant Machine Learning algorithms.)
Sources:
Sources:
Downloading the data
Downloading the data
In[]:=
WebImage["http://insideairbnb.com/get-the-data.html"]
Out[]=
Load data
Load data
In this section the data is imported. (You have to download the data first, and replace the corresponding names.)
In[]:=
lsFileNames=FileNames["~/Datasets/AirBnB/*reviews.csv"]
Out[]=
{~/Datasets/AirBnB/AirBnB-Austin-Texas-United-States-reviews.csv,~/Datasets/AirBnB/AirBnB-Broward-County-Florida-United-States-reviews.csv}
AbsoluteTimingdsReviews=Join@@MapResourceFunction,lsFileNames;
Out[]=
{18.0559,Null}
In[]:=
Dimensions[dsReviews]
Out[]=
{524759,6}
In[]:=
SeedRandom[342];RandomSample[dsReviews,12]
Out[]=
Summaries
Summaries
Summary of all records:
In[]:=
ResourceFunction["RecordsSummary"][dsReviews]
Out[]=
,
,
,
,
,
1 listing_id | ||||||||||||
|
2 id | ||||||||||||
|
3 date | ||||||||||||||
|
4 reviewer_id | ||||||||||||
|
5 reviewer_name | ||||||||||||||
|
6 comments | ||||||||||||||
|
Summary of counts per unique “listing_id” value:
In[]:=
ResourceFunction["RecordsSummary"][Query[GroupBy[#["listing_id"]&],Length]@dsReviews]
Out[]=
1 column 1 | ||||||||||||
|
Summary of counts per unique “reviewer_id” value:
In[]:=
ResourceFunction["RecordsSummary"][Query[GroupBy[#["reviewer_id"]&],Length]@dsReviews]
Out[]=
1 column 1 | ||||||||||||
|
Summary of counts per unique “id” value:
In[]:=
ResourceFunction["RecordsSummary"][Query[GroupBy[#["id"]&],Length]@dsReviews]
Out[]=
1 column 1 | ||||||||||||
|
Pareto Principle adherence
Pareto Principle adherence
Here we see a typical Pareto principle manifestation for the number of reviews distribution per listing ID:
ResourceFunction[Values[Normal[Query[GroupBy[#["listing_id"]&],Length]@dsReviews]]]
Out[]=
Here we see that many reviewers do one review only (within that dataset):
ResourceFunction[Values[Normal[Query[GroupBy[#["reviewer_id"]&],Length]@dsReviews]]]
Contingency matrix
Contingency matrix
Here we compute the contingency matrix for listings vs reviewers:
Latent Semantic Analysis
Latent Semantic Analysis
The Latent Semantic Analysis (LSA) computations are mostly done in order to compute different text derived statistics. (See [AA1, AA2].)
Text data
Text data
In this sub-section we extract the text data from the dataset into an association with text values and keys that correspond to the review ID’s.
Monad objects
Monad objects
Here we import the package [AA2]:
All reviews
All reviews
Tighter reviews
Tighter reviews
For faster LSA computations -- and better results -- we want to use a tighter set of reviews.
Here we select review with string length between 80 and 400 symbols:
Document × term statistics -- all reviews
Document × term statistics -- all reviews
Document × term statistics -- tight reviews
Document × term statistics -- tight reviews
Topics extraction (over tighter reviews)
Topics extraction (over tighter reviews)
Statistical thesaurus
Statistical thesaurus
References
References
[AA1] Anton Antonov, A monad for Latent Semantic Analysis workflows, (2019), MathematicaForPrediction at GitHub.