Get open books in nested WEB pages, for unstructured data analysis, WEB indexing, and data science research
As websites change very quickly, probably the code below does not work anymore for those links, but you can do just simple adjustments for WEB Scraping and WEB Crawling in any other website or other contexts with Wolfram Language, and feels like a hacker!! :-)
Many publishers kindly shared free books during the pandemics in 2020 and 2021.
If you are mining lots of data (Big Data), it is good to do it slowly to be kind with the WEB server and networks. And it is important too to remember that your local hard drive cant have all the Internet info, and if you fill all your company storage space (or cloud) somebody from IT support will kick you out! :-)
Don’t be evil!!
It is important to remember that Google, Bing, DuckDuckGo, Common Crawl, and others do this for a living.
Link book source: https://marcusnunes.me/posts/livros-gratuitos-da-springer/
Link book source: https://marcusnunes.me/posts/livros-gratuitos-da-springer/
Get the book page links:
In[]:=
Import["https://marcusnunes.me/posts/livros-gratuitos-da-springer/","Hyperlinks"]
Out[]//Short=
{https://marcusnunes.me,https://marcusnunes.me/posts/,https://marcusnunes.me/sobre/,https://marcusnunes.me/ensino/,https://marcusnunes.me/pesquisa/,https://marcusnunes.me/shiny/,https://marcusnunes.me/midia/,https://twitter.com/_marcus_nunes_,76,https://disqus.com/?ref_noscript,https://disqus.com,https://marcusnunes.me,https://creativecommons.org/licenses/by-nc/4.0/,https://gohugo.io/,https://github.com/Track3/hermit,https://marcusnunes.me/posts/index.xml}
Get one book page links to check:
Import["https://link.springer.com/book/10.1007%2F1-84628-168-7","Hyperlinks"]
Out[]//Short=
{http://activatejavascript.org,https://link.springer.com/book//,76,https://www.springernature.com/,https://www.springernature.com/}
Identify links (URLs) with PDF files:
StringContainsQ[%,"pdf"]
Out[]=
{False,False,False,False,False,False,False,True,True,False,True,False,True,False,True,False,True,False,True,False,True,False,True,False,True,False,True,False,True,False,True,False,True,False,True,False,True,False,True,False,True,False,True,False,True,False,True,False,True,False,True,False,True,False,True,False,True,False,True,False,True,False,True,False,True,True,False,False,True,False,False,False,False,False,False,False,False,False,False,False}
Clean up the list:
Pick[%%,%]
Out[]//Short=
{https://link.springer.com/content/pdf/10.1007%2F1-84628-168-7.pdf,https://link.springer.com/content/pdf/bfm%3A978-1-84628-168-6%2F1.pdf,28,https://link.springer.com/content/pdf/bbm%3A978-1-84628-168-6%2F1.pdf,https://link.springer.com/content/pdf/10.1007%2F1-84628-168-7.pdf}
Download test...
URLDownload[%,"~/Downloads/SpringerBooks",CreateIntermediateDirectories->True]
File,File,38,File,File
Get all PDF books list, and download:
In[]:=
bookList=Select[Flatten[Import[#,"Hyperlinks"]&/@Import["https://marcusnunes.me/posts/livros-gratuitos-da-springer/","Hyperlinks"]],StringContainsQ[#,".pdf"]&];
URLDownload[bookList,"~/Downloads/SpringerBooks",CreateIntermediateDirectories->True]
File,File,1200,File,File
Save the list of PDF books, just in case:
In[]:=
Export["SpringerBookList.txt",bookList]
Out[]=
SpringerBookList.txt
View the list:
In[]:=
bookList
Out[]//Short=
{https://marcusnunes.me/images/papers/marcus_nunes_thesis.pdf,https://marcusnunes.me/images/papers/marcus_nunes_dissertacao.pdf,https://github.com/mnunes/2nd-csds/raw/master/presentation/presentation.pdf,https://marcusnunes.me/images/minicursos/rbras-2016/slides.pdf,1196,https://link.springer.com/content/pdf/10.1007%2F978-1-4614-6227-9_14.pdf,https://link.springer.com/content/pdf/10.1007%2F978-1-4614-6227-9_15.pdf,https://link.springer.com/content/pdf/bbm%3A978-1-4614-6227-9%2F1.pdf,https://link.springer.com/content/pdf/10.1007%2F978-1-4614-6227-9.pdf}
Put it all together:
Parallelize[URLDownload[Select[Flatten[Import[#,"Hyperlinks"]&/@Import["https://marcusnunes.me/posts/livros-gratuitos-da-springer/","Hyperlinks"]],StringContainsQ[#,".pdf"]&],"~/Downloads/SpringerBooks",CreateIntermediateDirectories->True]]
Another JOB: Notes for Professional books
Parallelize[URLDownload[Select[Flatten[Import[#,"Hyperlinks"]&/@Import["https://books.goalkicker.com/","Hyperlinks"]],StringContainsQ[#,".pdf"]&],"~/Downloads/NotesForProfessionalsBooks",CreateIntermediateDirectories->True]]
File,File,44,File,File
Now you can process all the PDF books from a local directory for data analysis, NLP (Natural Language Processing) and research!
Behave!
Behave!