Hearing that public works on ao3 were scrapped for Huggingface is so infuriating. Scrapping in general is just both a morally reprehensible way to gather data, and stupid too. If you're really going to build datasets, having a narrower parameter then "literally everything" is the very start.
According to this article it used to be a chatbot but moved onto "primarily used for natural language processing tasks, such as text generation, sentiment analysis, and chatbot development." At a glance, I would not be scrapping a huge fandom archive for that sorta thing, considering not-insignificant number of fics that aren't written with the best grammar and also crackfics.
Going to the website itself and scrolling down I see this
I am not tech savvy enough to have extended commentary on the models or datasets outside of the fact that if this company wanted to be a hub for legitimate and ethically ai development, you think they would put more care into how they were getting their information. It's those spaces i'm more concerned about.
Now, it seems than they're uploaded by different people and not just Hugging face itself, but I digress. A skim has already shown different dialogue and character creation related bots made to generate.
Here is the post of the user on hugging face who made and uploaded at dataset of ao3 fics. Needless to say, do not make accounts just to make threats and drag this creator. Knowing its a user and not the action of the website itself is only comforting until you wonder why/when will the website screen the datasets being uploaded to make sure they're ethically sourced. (This also doesn't appear to be the only dataset made with fanfiction, but by far the largest.) It appears that Hugging Face is doing what they're legally obligated to do.
Access to it has been restricted thankfully. Following that link leads me to this
and the comments are as ugly as you can expect them to be. (No, fanfiction isn't theft. It is protected under parody and fair use laws. Regardless, i cannot fathom why you would want to use them anyways.)
If you're curious if it's been scrapped, this tiktok said any work with the work id under 63,200,000 appears to be scrapped. Basically anything made mid this march and before. All of my fics, including my one original work has been scrapped. Since ao3 lawyers are on the case, there's nothing anyone else can do now but wait. There is no point in submitting your own DMCA claim because Ao3's covers all of us.
That being said, it's irritable and disheartening. This is not in the spirit of fanfiction, this is not legally or morally ethical, and it's just irritating. None of these works had consent to be reuploaded or used in this way (no matter how those ai idiots think things work. Just because it's on the internet does not mean it's free use)
The conversation now is how to prevent this from happening again. Many users are setting their work to be privated, so that way only those with an account can read fics. I fear that is a very bandaid on the bullet wound type solution, because there's nothing stopping someone from making an account and then scrapping. Ao3 themselves have their own methods to try and counteract such things. And largely, we will be relaying on the team to handle such things. (This is why that donation drive is important, i can imagine the poor legal department is going to be clocking a lot of overtime soon.) I suppose another method is just to go old school. Printed zines for yourself and your circle of friends and not online I guess.
What now? Not sure. Obviously arguing with ai bros on their own turf isn't doing anything constructive, not that I really expected anything else from the general fandom nerds tbh. It's just. A super fucking frustrating reality of the internet now. I can only hope that when enough idiots do something like this and face repercussions, it'll be a widespread lesson on Ethics in Business and 101 for all.
It sucks, because back before "AI" was associated with chatGPT and datascrapping, it was understood to be broad range of tech and code that can be really good and helpful. Unfortunately, aibros of today do not know how to, nor will put in the hard work, to making something more constructive than ArtTheftArtGeneratornumber3000 and ChatBotThatSucks2000. With so much of the slop, it's taking away time, energy, and resources from people trying to development meaningful AIs to recognize cancer and process information humans can't in a timely manner.
Edit: Its worth knowing this is not the first, nor likely the last time, ao3 has been scrapped. This isn't new but it's getting annoyingly common. And this particular dataset is *huge*