Introduction
Introduction
Discovering a website's last updated date is a difficult task, since websites rarely keep update data, and even when they do store it, it is difficult to decide whether the update was a major change to the website code or a simple adjustment. This project approaches this challenge by scraping data from the Internet Archive and using string comparison methods to date HTML and plot code modifications.
Capture the HTML Data
Capture the HTML Data
Internet Archive's API outputs data in multiple different formats which required complicated and slow preprocessing, so I needed to bypass the original API and use my own web-scraping program.
Take an input of a date in the format YYYYMMDD and a URL and return its HTML code:
In[]:=
htmlTime[date1_String,url1_String]:=Quiet[URLRead[HTTPRequest[StringJoin["https://web.archive.org/web/",date1,"/http://",url1]],"Body"]]
A section of Wolfram.com's code from 2005:
In[]:=
StringTake[htmlTime["20050101","wolfram.com"],{2025,3576}]
Out[]=
</script><link rel="stylesheet" type="text/css" href="/_static/css/banner-styles.css?v=_99BHQfY" /><link rel="stylesheet" type="text/css" href="/_static/css/iconochive.css?v=qtvMKcIJ" /><!-- End Wayback Rewrite JS Include --><title>Wolfram Research, Inc.</title><meta name="description" content="Wolfram Research, makers of Mathematica, the only fully integrated technical computing software"><meta name="keywords" content="wolfram research Inc,wolfram research,wolfram,technical software,mathematica,software,wri,math typesetting,mathematical typesetting,math notation,mathsource,mathlink,engineering software,mathematics software,mathematical software,finance software,financial software,education software,science software,scientific software,notebook,stephen wolfram,structural mechanics,control system professional,wavelet explorer,experimental data analyst,time series,finance essentials,fuzzy logic,optica,dynamic visualizer,mechanical systems,signals and systems,electrical engineering,scientific astronomer,windows software,macintosh software,unix software,graphics software,mathematics books,mathematics education,calculus education,technical publishing,math teaching,finance,math,engineering,typesetting,typeset, technical computing, math software, math programming, mathematical programming, Mathematica Link for Excel, Mathematica Link for Microsoft Word, Calculus Wiz, symbolic computation software, computer algebra software, Wolfram Research, Stephen Wolfram, CalculationCenter, InstantCalculator, SmartPlot, Calculation Center">
My First Attempts
My First Attempts
After I had developed my scraping code, I was initially planning on training an AI to check the code age. However, first, I needed to develop a training and test dataset. This proved to be quite a large challenge, as calculating how similar two very long strings were took prohibitively long.
The length of the current code of wolfram.com:
In[]:=
StringLength[htmlTime[StringDelete[DateString["ISODate"],"-"],"wolfram.com"]]
Out[]=
105899
Early on, I used EditDistance as my distance algorithm and used monthly website data, but the runtime of this program was almost 24 hours for a single website, making it infeasible to generate a dataset. After that, I found LongestCommonSequence runs enough faster and worked well enough to replace EditDistance, as long as I changed my distance threshold.
Change a date in the format that htmltime takes into a 2-digit number corresponding to a year:
In[]:=
extract[n_String]:=Interpreter["Number"][StringJoin[StringPart[n,{3,4}]]]
Create a list of dates every 6 months in the correct format for htmltime from 2-digit numbers corresponding to years:
In[]:=
dates[start_,end_]:=StringJoin[#,"01"]&/@Flatten[Table[StringJoin[#,If[i<10,StringJoin["0",ToString[i]],ToString[i]]],{i,1,12,6}]&/@Table[StringJoin["20",If[i<10,StringJoin["0",ToString[i]],ToString[i]]],{i,start,end}]]
Calculate the first date at which approximately 2/3 of the code is identical to the current version of the code:
In[]:=
getDate[website_String,{startdate_,enddate_}]:=dates[extract[startdate],extract[enddate]][[FirstPosition[StringLength[LongestCommonSequence[htmlTime[StringDelete[DateString["ISODate"],"-"],website],htmlTime[#,website]]]&/@dates[extract[startdate],extract[enddate]],_?(StringLength[htmlTime[StringDelete[DateString["ISODate"],"-"],website]]-#-10000<=Divide[StringLength[htmlTime[StringDelete[DateString["ISODate"],"-"],website]]-10000,3]&)]]][[1]]
What follows is a graph produced by a variant of getDate which I have not included here which plots the length of the longest common sequences, demonstrating how the code works:
getDate was quite difficult to develop and bugfix, but I eventually decided on using the following approach: 1. Call LongestCommonSequence on each date outputted by dates and find the length of the sequences. 2. Find the first position at which the current code length minus the length of a code at a specific date minus 10000 characters (the approximate amount of formatting characters which are constant) is less than or equal to the current length minus 10000 characters over three. 3. Get the n'th element of the dates, where n is the first index at which the code is mostly the same. This will return a date as the output.This code worked quite well and returned quite accurate results. For example, depending on the parameters of the cutoff equation, it detected both the major change to wolfram.com in mid-2014 when Mathematica 10 was released and the release of Mathematica 11 and the additional redesign of the homepage. However, it was still taking several hours to run and would still occasionally fail, so I realized that using this program to develop a training dataset was far too slow, so I needed a different approach.
A Change of Approach
A Change of Approach
After brainstorming for a while, I realized that instead of training an AI to predict website ages, I could instead just use my program which I intended to develop the dataset as my function, since it did what I was planning for the AI to do without requiring a complicated training/test dataset. This made the rest of the project much easier in some ways, as I didn't need to run my dataset creation code anymore (which only ran on four of the thirty websites which were planned for it) and didn't have to worry about selecting good training data. However, this decision also meant I needed to aggressively optimize my algorithm to make it fast enough for general use, since without the AI algorithm as a quick frontend, the string comparison algorithm would need to run in only a few minutes to function well enough. What follows this is a variant of my code which I used to optimize my algorithm.
Download the code from wolfram.com's current version on Archive:
In[]:=
wolframSite=htmlTime[StringDelete[DateString["ISODate"],"-"],"wolfram.com"];
Get a list of code from wolfram.com at different dates:
In[]:=
wolframList=Quiet[URLRead[HTTPRequest[StringJoin[StringJoin["https://web.archive.org/web/",#],"/http://www.wolfram.com/"]],"Body"]&/@dates[5,20]];
Create a feature extraction vector as our distance function, which simplifies the similarities and differences in the text into a single long vector:
In[]:=
wolframVector=FeatureExtraction[wolframList,"TFIDF"]
Out[]=
FeatureExtractorFunction
Plot the feature distance for wolfram.com:
In[]:=
DiscretePlot[Labeled[FeatureDistance[wolframList[[i]],wolframSite,wolframVector],dates[5,20][[i]]],{i,1,32},ExtentSizeFull,PlotTheme"Marketing"]
Out[]=
This code runs much faster than getDate, but is less accurate and harder to work with for the purposes of this code, as it considers any point at which there was very little code to be very similar to the current version of the code and barely detects the code change in mid-2017, unlike the previous algorithm.
The Finalized Version
The Finalized Version
Below is my code which is (relatively) complete, which takes an input of a website, an optional input of a list of two numbers for the start and end year (the function Extract listed above combined with DateString allows for DateObject inputs), and an optional step parameter, and, depending on the function, either returns a plot of the website's changes or the date at which the website was last majorly updated.
Upgrade dates to have a step parameter:
In[]:=
dates2[start_,end_,step_:6]:=StringJoin[#,"01"]&/@Flatten[Table[StringJoin[#,If[i<10,StringJoin["0",ToString[i]],(*Addzerostomonthdigitsiftheyaresingle-digit*)ToString[i]]],{i,1,12,step}]&/@Table[StringJoin["20",If[i<10,StringJoin["0",ToString[i]],(*Addzerostoyeardigitsiftheyaresingledigit*)ToString[i]]],{i,start,end}]]
Get the html data across all the dates in the list outputted by dates2:
In[]:=
htmlExtract[site_String,se_,step_:6]:=htmlTime[#,site]&/@dates2[se[[1]],se[[2]],step]
Define the feature extraction vector, which is a vector that finds characteristics which appear multiple times between code versions, but gives extra weight to those which are less common:
In[]:=
featureVector[site_String,se_:{5,20},step_:6]:=FeatureExtraction[htmlExtract[site,se,step],"TFIDF"]
get the feature distance of the site's version on the dates from dates2 from the current version of the site (m is an input where the feature vector is inputted):
Determine the threshold at which a site is considered different compared to the current version, where y is the output of getDistance:
Get the date at which the code was last updated, where y is the output from getDistance:
Get the date at which the website was updated last from the url, dates, and step:
Plot the changes in code over time:
Plot the changes of several websites over time:
Green is wolfram, orange is google, red is github, blue is espn, and yellow is facebook.
A graph of the speeds of some alternate distance functions:
The Future of the Project
The Future of the Project
Keywords
Keywords
◼
Data Analysis
◼
Wolfram Summer Camp 2020
◼
Wolfram Language
◼
Websites
◼
HTML
◼
Code