Isn’t the data you just posted the same as the data last week?
Thank you, eagle-eyed anon! Yes, yes it was but now it has been fixed, so the newest post for 02/05/2024 is accurate.
And while I was trying to figure out why the same data showed up twice, I found a bunch of other issues in my code, so keep reading if you want to hear about the 10 other things I fixed while I was fixing that issue (thanks ADHD!)
As it turns out, it was the exact same info as last week because there was an issue in my workflow that caused it to not run properly so the data in the spreadsheet that powers the dashboard was stale.
But that prompted me to ask myself "didn't I build in a notification system that's supposed to let me know when something fails?" and when I investigated I realized they had stopped working last June and I never noticed (it sends EITHER a success or failure email every week, so I really should've noticed).
When I investigated that, I realized June was when I split the backend data sources between two Google Sheets because they were getting so big - originally I had them in separate tabs of the same sheet. However the notifications were still set up for only the original sheet, so I fixed that.
Also while I was looking at the sheets, I realized there were still a bunch of "Uncategorized" fandoms in the data, even though I removed that category from the workflow after realizing that they don't have fic counts on that Category page. So instead of pulling fic count, I was accidentally pulling a bunch of years and passing them off as fic count. I fixed that a couple weeks ago, but I didn't clear old data so I did that now.
THEN when I finally remembered the original problem I had been solving, I started trying to figure out whether the data hadn't been recorded for 2/5 at all (I save it in a JSON first) or if the sheet just hadn't been properly updated. While doing some checks, I realized that most of the categories had data for 2/5 EXCEPT Video Games & Theater?? Which honestly I still haven't fully figured out why, but I was able to manually run the size scrape for those two pages again, so at least they have data now.
Finally, while I was re-running the scripts to try to figure out why it wasn't running for Video Games, I had it start printing out various pieces in the loops and noticed it was taking forever to get through a category and seemed to go slower and slower by letter. Turns out a small error in my code had it looping through all the fandoms for a category starting with a specific letter (A) but then instead of moving on and doing B next, it was doing (A, B), then (A,B,C) etc. This wasn't actually causing errors of any kind in the results, but it was slowing the whole thing down and unnecessarily writing over the size count 25+ times for the fandoms at the beginning of the alphabet, so I've now fixed that.
The best part is even though I just wrote all of this out, I'm still not quite sure where in there I fixed the issue that caused the stale data in the first place lol. But at some point I fixed some other things, re-ran the "write_to_sheets" script (which I of course tried first thing and it didn't do anything) and this time it worked and actually updated. So hooray! And clearly this whole thing is being held together by spit & glue (it was a project I set for myself when I was learning Python) so maybe I should go back and rewrite some things now that I'm actually paid to do coding for my job and have a bit more experience. If you're still reading, KUDOS TO YOU and hope you enjoy my little project. [[Maybe leave me a heart in the comments if you do, because I accidentally spent way too long on this tonight and it's super late and I still have actual work to do (YIKES).]]
