#web scraper

Step One:

Create a folder for your downloads. With Chrome’s settings you can then switch your download folder to this new folder.

Trust me, you don’t want your thousands of historical fic mixing in with all the other porn crap in your downloads folder. The scraping won’t be maintaining subfolder file structure so you’ll have all your group’s files in one heap of a mess. Possibly half of them helpfully named ‘Chapter_One.htm’ The less sorting you have to do later the better.

Step Two:

Install a browser extension scraper. In this case we’re using Chrome’s Web Scraper:

Once installed you can access it on any webpage by opening Chrome’s developer tools using F12 and selecting ‘Web Scraper’ from the tool’s menu bar. Note: You’ll want to dock the developer tools window to the bottom of the screen using the buttons I’ve highlighted below. vvv

If you want more details on how to use Web Scraper, click on the blue web symbol that will appear to the right of your address bar in Chrome. You’ll find tutorials and documentation. I didn’t read any of it. Why any of you are listening to me is a mystery. But if you’re happy with the blind leading the blind, jump to our next step.

Step Three:

Now we can import your Sitemap for scraping. Start by navigating in Chrome to the yahoo groups file page. Something like,  https://groups.yahoo.com/neo/groups/awesomefanfic/files

Open the Web Scraper in developer tools (F12) for this page, if you haven’t already, and click “create new sitemap”>“Import Sitemap” 

Then paste the following code into the "Sitemap JSON” field:

{"_id":"testscrape1","startUrl":["YOUR FILES"],"selectors":[{"id":"folderName","type":"SelectorLink","parentSelectors":["_root"],"selector":".yg-list-title a","multiple":true,"delay":0},{"id":"subfolder","type":"SelectorLink","parentSelectors":["folderName"],"selector":".yg-list-title a","multiple":true,"delay":0},{"id":"subfiles","type":"SelectorLink","parentSelectors":["subfiles"],"selector":".yg-list-title a","multiple":true,"delay":0},{"id":"Files","type":"SelectorText","parentSelectors":["folderName"],"selector":".yg-list-title a","multiple":true,"regex":"","delay":0},{"id":"subFiles","type":"SelectorLink","parentSelectors":["subfolder"],"selector":".yg-list-title a","multiple":true,"delay":0},{"id":"SubFileNames","type":"SelectorText","parentSelectors":["subfolder"],"selector":".yg-list-title a","multiple":true,"regex":"","delay":0},{"id":"dummy","type":"SelectorText","parentSelectors":["subFiles"],"selector":".yg-list-title a","multiple":true,"regex":"","delay":0}]}

You’ll going to need to change where it says “YOUR FILES” to your yahoo group’s files page. Copy it from the address bar. It will look something like: "startUrl": ["https://groups.yahoo.com/neo/groups/awesomefanfic/files"] 

You may want to rename your Sitemap in the second field. Otherwise it will be named “testscrape1″

Once you’ve made your changes, hit the import button and you’re ready to start scraping.

Note: This imported JSON code provides a Sitemap structure designed to download all the documents up to a level two subfolder and place them all into that download folder. It will also create a spreadsheet record of each file’s location.

Step Four:

Now we Scrape. 

Once it starts scraping it will open a new window and slowly (SOOO SLOWLY) start downloading your old files. Note: Do NOT close Web Scraper window while it is running a scrape or you will have to start again from beginning. Again, it will not be downloading folders so all the files will be dumped in one big heap. Luckily the spreadsheet it creates showing where every file came from, will help.

Step Five:

Once you’ve walked the dog, gone to work, eaten a feast for the elder gods, and danced with your coven at midnight the scraping should be about done (results dependent on number of grizzled files to download). The Web Scraper window should close on it’s own. You’re tomb-of-ancient-knowledge will be ready in the location you specified in step one. It’s possible the Scrape missed some hoary files because yahoo was refusing to open a folder or the file was in a deeper subfolder than level two (the bot checking for subfolders is why this scrape is so slow so I didn’t go deeper). For best results you will need to review the scraped file data and compare it to what you downloaded. Use “Export data as CSV” and then use your new spreadsheet to rebuild your file structure and check for any missing files. NOTHING listed in the “dummy” column will be downloaded. You will have download everything in that column (and deeper) manually.

With luck, only a handful of files will need to be downloaded manually after your review. Treasure your prize and if you’re a mod of the group, help save the files for the gen z fic readers by sharing your hoard on Hugo-award-winning AO3.

Bonus (Expert Mode):

Using the spreadsheet and batch file scripts you can recreate your file structure quickly and move the files into the new folders. A few changes will turn your list of urls into folder locations on your PC and bam! You’ve got a everything organized again! A few things to watch for: 1. You will have to remove spaces and special characters from folder names. 2. File names should be in double quotes in your batch file or have all spaces removed. 3. Files with the same name from different folders (Like chapter1.htm) should probably be done manually before running the batch to move the rest.

-------------------------------

Thank you for reading.