aza

What did you miss?

Another Collaboration?

In case you haven't kept up with the news on other platforms, I'm now in talks with the Open Doors Committee of the Organization for Transformative Works (the people behind Archive of Our Own) for a possible collaboration!! We had our first meeting in which we discussed the different possibilities for us working together and what it would take, so let's hope for the best! This would be a great opportunity and a massive help to me too.

New Database!

Originally, I was just using MongoDB because it was the easiest to just set up a scraper and start ingesting data, however, for an actual website deployment MongoDB would be very unsuited. It's slow, it's clunky and aggregations are really slow.

So I decided to switch to a new Database: ClickHouse! ClickHouse allows me to do extremely complex aggregations with full table scans in milliseconds, something that would take the old MongoDB deployed multiple dozens of seconds if not over a minute! This will also make the database easier to share when it's published to SciOp

New Infrastructure!

To be completely honest, I planned to just deploy everything locally in Europe and hope for the best with CDN response times in the US. Now, I've realized that this would be pretty suboptimal so I've decided to geo replicate the CDN and DB to the US as well as Europe, that way the site, when it finally releases, will hopefully be fast on both sides of the oceans!

Additionally, since I'm in the process of getting my hands on a full Terabyte of previously unarchived media I decided to invest some more into new storage infrastructure allowing me to host and cache the most accessed files directly instead of relying on the sometimes slow response times of the Internet Archive.

New Costs :c

Sadly, all of that infra didn't just come for free (even though one person actually provisioned a VM in Canada for me, allowing me to improve geo replication). I'll have to see the exact infra bill next month since I've done some restructuring and got massive discounts on my current infra thanks to my server host. The last infrastructure bill was 50€/mo. which is... quite a lot for a project that hasn't even released yet haha! In case you want to support my endeavor of keeping this free for everyone, you can donate to my buy me a coffee page

New Content!

This is probably one of the better news I have to give. I extracted a list of archived pages from the Internet Archive, that means, that my archive will be a joint archive of everything still available from Amino in any way!! About 2-3 Million new posts will be added to the archive in the next few days!!

Sneak Peek...

Bonus Content!

After the DB migration, I wanted to try out the new aggregation capabilities by doing some rudimentary analysis, all of the queries took <0.1s to compute!!

Because I didn't just want to do these calculations without sharing them, here are the Amino records brought to you by the Amino Archive!

🥇Amino Awards🥇

🥇 World's Biggest Yapper🥇

With over 13 million posts globally on Amino, the biggest yapper award goes to: BTS εµρɦσɾเαᵇᵘᵗᵗᵉʳ | JK

(important note: not all of these posts were archived due to limitations on Amino's end)

🥇 World's Most Entitled 🥇

Everyone knows that the amount of titles you can receive on your profile in a given community is limited, one person broke those limits and achieved a whopping 66 million titles on a single community!

🥇 World's Most Reputable 🥇

As a member I honestly thought level 20 was unreachable it just always seemed like way too high of a bar, but one person was so dedicated to not just get the 500K rep required for Level 20 but get nearly 200K additional rep!

🥇 World's Most Followed🥇

Anime Amino is at it again for world record contending profiles with the most followed profile on Amino with nearly 1 million followers!!! Now imagine all of those people genuinely following you on the street.

🥇 World's Most Likeable🥇

This person has proven time and time again that their content is likeable and deserves your attention (I've never once seen a post by them before archiving the platform lmao)

The ArchiveTeam collab

So, one of the major parts of the project was collaboration with ArchiveTeam to get the media archived. That actually went pretty well, I gave them a list of 55M URLs and they archived 54M URLs before Amino shut down the media servers, that means that nearly every single image, video, gif, profile frame, whatever is archived fully & functionally! Private media was not archived and as such media for larger communities that were set to private to prevent bot attacks might be gone. There are separate 7M files that got archived which might fill some of the holes but at most 61M URLs are functionally archived.

Public Preview

Because I got >80 E-Mails asking me whether a community was archived or not I decided to make a quick public preview that allows you to easily search through the archived communities as well as find out what has been archived from it. You can find that hosted at https://archive.azalea.sh/ please make sure to read the about page as that contains some useful information!

Rant (this is where I go crashout)

You really don't need to read this unless you're interested in the technical side of things but I CAN'T KEEP IT TOGETHER ANYMORE

I wanted to implement text search in a way where it's handled by a different service than actual DB lookups. Mostly to reduce strain but also to make text search quick and snappy. For that purpose I wanted to use MeiliSearch because I've used it on a bunch of other smaller projects (like archiving & indexing every Strawpage, that's a story for another day) and it generally worked great!

It turns out that it really does not scale well and I kept having a bunch of stupid problems with MeiliSearch which eventually just lead to me programming around the pitfalls that MeiliSearch inherently brings with it, that is, I started to write my own ingestion pipeline to auto compact the db because it would just grow exponentially otherwise. I started writing systems to partition the data automagically to prevent building monolithic indexes that do nothing but waste space.

I started using Amino in 2018 and was a fairly active user. Due to my interest in coding, I started to reverse engineer Amino working alongside others on the project of "Amino.py" an unofficial API wrapper for Amino. This project has since been deleted from GitHub, but my contributions to the community can still be seen. Whilst working on Amino.py I had learned a lot about the inner workings of Amino. I started reporting bugs alongside potential fixes. I started reporting exploits alongside write-ups.

Note: this post was written in mid 2024, since I didn't want to change everything, I'm publishing it as it was then. Amino has since shut down.

How it came to be

Eventually, this lead to me contacting Sam Chen on Twitter/X due to them complaining about how the codebase was developed.

This conversation eventually lead to Sam contacting me and us getting to talk settling on an employment starting in October 2021.

How it started out

To be honest, working under Sam was a very nice learning experience. I was still 15 so there was a lot to teach me, but I learned to do official translation, Jira ticketing, team communication, communication with the community under the pseudonym "Dente" and also doing Q&A for amino and testing new features.

Since I was too young to make a PayPal account in Germany, I had to ask my brother to invoice Amino for me and hand me the money. I didn't think much of it but this was the first of a lot of problems that I was about to face.

Payment at MediaLab was/is handled through Deel. As said already, I was too young for any banking so I also couldn't sign up for Deel either. This was an issue however. Contracts were established through Deel and with Deel being explicitly mentioned as the payment provider. As a consequence, I never got a written contract.

Sadly, Sam had to leave MediaLab shortly after I started working there. Sam was one of the most pleasant people I ever had the chance to work with and to this day I genuinely believe that Sam only wanted good for me and the other workers.

Enter Julien Marlatt

After Sam was gone, there was a lot of confusion about who would lead the team now that the previous product owner had to leave. We didn't really get an introduction to him but we saw a new person in the Slack group - Julien Marlatt.

Julien, who is now the product manager of Imgur, introduced himself to the team in a meeting in which I wasn't present. Despite that, I had high hopes of continuing my work at MediaLab and was excited to work under Julien.

This is where things started to go south. Julien would spend a lot of time ignoring mine and a co-workers requests and inquires. Anytime we'd message him it'd take days to get a response back which would often not address our inquiry at all or not be helpful. I thought that this was just due to him being busy. There was a lot to do on Amino and the platform was put under huge amounts of stress with new exploits being discovered by the public (For context, I found most of the exploits that still exist on Amino to this day. I reported a lot of them before they got widely known and abused. They weren't fixed in a timely manner though which lead the community to discover the exploits as well due to leaks from the Amino.py staff).

Problems start to arise

A huge issue Amino suffered before the acquisition by MediaLab was a lack of transparency. There would rarely be updates on what was happening internally. Under Sam, this issue was addressed. Us workers were allowed to communicate in the community under our pseudonyms establishing the transparency which was previously lacking. When Julien started managing Amino, this transparency began to fade. Official Q&As stopped being published, the amount of events began to decrease and things went back to how they were before Sam.

This is where it goes from bad to worse. Our Jira tickets would suddenly start to be put into the backlog again or just moved to "Done" despite me still being able to replicate all of them. On inquiring about this, there were a lot of weird statements being made.

For example, there is an exploit that allows you to generate Amino Coins (the in-app currency of Amino) in a way that scales to infinity. At the peak of this exploit, I had over 16 Million amino coins. I opened a ticket for this on Jira because it would directly influence MediaLabs profit. The response I got, was a bit... weird.

To an exploit to generate coins, Julien responded that bots farming coins should be prioritised. I can somewhat understand this. The API should be secure in the first place after all, but it obviously wasn't and instead of fixing what could easily be done to buy time and to stop a big part of coin generation there was now no plan to fix this at all. This is just an example and there are a lot of tickets that would just get marked as resolved for reasons still unclear to me.

Not understanding tech is one thing. Sam also had trouble to understand my conversations with the engineers from time to time. I don't expect anyone to know everything.

As product manager something that you should know though is paying the people who work for you. Under Julien I didn't receive a single cent. Despite that I continued to work for MediaLab regardless of not receiving any compensation whatsoever. I thought that this would be resolved somehow. Keep in mind I was 16 when Julien took over at Amino and I didn't really know anything about how the world works. The lack of compensation and lack of any progress with the app lead to me eventually quitting my employment after approximately one year of work under Julien. So I sent a Slack message explaining why I was departing and that I'd quit.

The journey to receiving a reference

(this part will be the most detailed one due to occupying a huge chunk of my mind as of late and at the time of writing still being an ongoing event)

When I sent that message I also asked Julien for a reference for my CV. Despite me not knowing much about life I knew that an employment at MediaLab at the age of 15 would be quite impressive on a CV, with that in mind I asked Julien for a reference when I quit to which he responded that I should ask again when I need it.

In June of 2023 I started applying for universities and jobs for a dual course study, which is a special form of studying at a university where you also have vocational training and study at a university in tandem. This was the perfect time to contact Julien so I sent him an email hoping for a quick response. And then... Nothing happened.

One month later in July of 2023 Julien replied, CCing my email to an HR member asking whether Deel provides a function to export such a reference. After that, there was radio silence again. I managed to get accepted for the job and the university I applied to regardless. That and me being preoccupied with myself for the time being lead to me forgetting the email correspondence.

In December of 2023 I reached out to Julien again hoping to still get the reference/certificate of work. You might not know this but in Germany there are laws which make such a request permissible and which declare a duty for companies to provide a reference on leave when requested (e.g. §630 BGB, §109 GeWo).

At that time I realised that Julien didn't manage Amino anymore, so I decided to reach out to the HR Team with my request instead, mentioning the legal basis for my request and explaining why I'd like to receive the aforementioned reference. I got a reply by the same person Julien added to the email chain earlier explaining that due to the festivities in December, I would get a response on Tuesday 02/01/2024.

I accepted that wait. If I could wait for half a year, I can wait for another week or so. Then Tuesday came and there was no email. Wednesday came and there still was no email. But then Thursday (04/01/2024) arrived and I finally received another email from Julien.

Starting with this email, I began to get a bit annoyed. I never got paid so the payroll is completely irrelevant, so is the last question. I never got sent a contract, so the first question is also meaningless.

I mentioned this in my response to him

What you're reading there is true by the way. I didn't charge MediaLab for the full sum of my work which I now fully regret. I charged at least 10% less than what I could, and probably should, have.

Since I responded in about an hour I was hoping to get a response at latest on Friday (05/01/204). Sadly, that wasn't the case. On Monday (08/01/2024) I decided to send an email to Andrew Gordon, the general counsel for MediaLab in hopes of escalating this matter due to the legal basis of my request and the fact that nearly all of Germany's laws were disregarded when I was hired and subsequently quit. Mr. Gordon didn't respond either.

Nearly two weeks after the last response from Julien on 17/01/2024 I decided to send another email asking for an update on the progress. This email as well has gone unanswered. On Monday 22/01/2024 I tried to push this matter again by messaging the HR Team and articulating exactly why this is an urgent matter. Shortly thereafter on the same day, I got a response from Julien

This email was just adding salt to the wound. Not only was there still a complete disregard to my request but there also was a huge misunderstanding of what I did at Amino. I didn't just "bust bugs", I was pretty much the only person finding new exploits on Amino and had a lot of knowledge on the inner workings of the API. I also did way more than just that including Q&A, recommending and basically managing new people, maintain relations on the German side of Amino which was way underrepresented and also continued to show the engineering team where and how they could improve security.

Despite that, I was happy to finally see some transparency and progress being made due to legal and HR getting involved now. So, again I wrote a reply to Julien hoping to maintain the transparency and progression.

The aftermath

I'm now 18. It has been three years since my employment started and I still only ever got compensated under Sam. Amino has been slowly dying and the outbreak of exploits combined with lack of communication probably played a major role in this.

I now know that you should never trust in big companies, especially not when you're 15. I also know now that this was partly my fault. I shouldn't have kept working and I was too naïve to quit.

I also learned that your higher ups will certainly try to find a way to not do what you want them to. Even if your request is as simple as a piece of paper saying that you worked for their company.

I'm writing this to get the word out about what happened at MediaLab and to raise awareness about the necessity of worker's rights. Even if I am partially to blame, robbing a 15 year old or their well earned money and afterwards refusing to provide a reference is not justifiable.

What now?

I've given up receiving the compensation and I missed out on over 4,000USD.

I won't work remotely to the US again and I'll try to stay out of trouble like that in the future.

I know that I probably won't change anything with this, but I had to get it out of my head and write it down. This has been one of the most stressful and annoying experiences I've ever had so far and I want to do everything I can do prevent something like this happening again in the future.

I'm deeply saddened that all of my work and everything I had done seemingly just went to waste.

You might ask: how did we get here in the first place? What even is Amino? And what's my connection to it?

Basically: Amino is was a social media platform where teens were trusted to run and moderate subcommunities about anything you can think of, with the occasional grooming sprinkled in. Basically: reddit with the modern customizability and chatting functionality of discord and more!! Quizzes, Photo Albums, if you can think of a feature for a social media app, Amino probably had it – in a lot of cases, even before a lot of social media platforms have started using it themselves.

I started using Amino in 2018, started contributing to a python library for amino in 2019-2020, got employed for Amino in 2021 and then left in late 2022. I've written a huge draft that goes into my employment at MediaLab and how it wasn't really... good nor lawful, but that's neither here nor there for now.

After my tenure I've started to use the platform less and less but one thing was always certain for me: if Amino were to disappear, I would be extremely sad to go out empty handed. So in 2023 I started working on an archival tool for Amino, which had matured in 2024 where I had started renting servers to run it on, as well as using my own hardware for running the tools in parallel with a selfhosted database cluster which I had to maintain. In total, I spent over 800€ during the runtime of this project to scrape amino as much as I could. Leading me to a database with about 30 million records

Developing that archival tool, was a huge pain. Not due to ratelimiting or platform security but due to the lack thereof. Lacking consistency in response schemas, exploits allowing to upload arbitrary data, malformed JSON Objects and everything you can think of. Every time the tool hung, I would notice days later because I couldn't be bothered to actually add good monitoring and had to patch out all of the mistakes Amino had made. That is also why I decided to use MongoDB for this project. SQL is great for sure but if the schemas aren't consistent anywhere then uh... Where do you even go from there? To me it was more important to get as much as quickly as I could instead of wasting precious time creating schemas.

The problems didn't just end at bad design on Amino's end though. The main problem for me was: how do I find a list of all major communities? There was no way to enumerate every community on Amino besides trying to iterate over every single possible community id or stalking their sitemap.xml (I ended up doing a mix of both) which were both suboptimal as that led to including a bunch of small and or private and or deleted communities which due to lacking security were completely accessible by the API.

I decided to archive those but strip them from any public release, should I ever make one. I made a different choice in favor of privacy regarding chats. I like archiving, yes, but a lot of chats contained extremely private information which would be impossible for me to filter out. Despite the value of archiving them, I made the conscious decision of leaving chat logs as well as photo albums out of the archive and only retaining sparse chat room metadata, if at all.

Going on a small tangent here, but, did you know that private chats aren't actually private on Amino? As with private communities, private chats can be accessed solely if the chat id is known! No auth needed! It's kinda stupid that this works but hell, what do I know.

That leads us to today. Amino has now "officially" shut down. I'm using air quotes here, because there was no official statement, subscriptions weren't cancelled automatically and had to be refunded and stopped manually, there is no way to make GDPR requests anymore and a bunch of other stuff that is outright unlawful. The only thing that remains up is the image servers which are purely AWS S3 wrappers.

So, instead of archiving blogs etc. Let's set out on a different journey. Changing the scope from gigabytes to terabytes!

Since I don't have the storage necessary myself, I reached out to archiveteam as well a friend who will help me download what we can from the static media in order to bring amino back from the dead once more. Couple that archive with a project of mine (self hostable amino private server) and one day you'll have a fully functional archive of amino browsable in the app itself.

So consider this as a teaser for a larger project yet to come! Be prepared for Aminoid!

Preface

About two weeks ago I was faced with a pretty annoying problem. Whilst I was going home by train I have noticed that my server at home had been running hot and slowed down a lot. This prompted me to check my nginx logs, the only service that is indirectly available to the public (more on that later), which made me realize that - due to poor access control - someone had been sending me hundreds of thousands of huge DNS requests to my server, most likely testing for vulnerabilities. I added an iptables rule to drop all traffic from the aforementioned source and redirected remaining traffic to a backup NextDNS instance that I set up previously with the same overrides and custom records that my DNS had to not get any downtime for the service but also allow my server to cool down. I stopped the DNS service on my server at home and then used the remaining train ride to think. How would I stop this from happening in the future? I pondered multiple possible solutions for this problem, whether to use fail2ban, whether to just add better access control, or to just stick with the NextDNS instance.

I ended up going with a completely different option: making a solution, that's perfectly fit for my server, myself.

My Server Structure

So, I should probably explain how I host and why only nginx is public despite me hosting a bunch of services under the hood.

I have a public facing VPS that only allows traffic to nginx. That traffic then gets forwarded through a VPN connection to my home server so that I don't have to have any public facing ports on said home server. The VPS only really acts like the public interface for the home server with access control and logging sprinkled in throughout my configs to get more layers of security. Some Services can only be interacted with through the VPN or a local connection, such that not everything is actually forwarded - only what I need/want to be.

I actually do have fail2ban installed on both my VPS and home server, so why make another piece of software?

Tabarnak - Succeeding at Banning

I had a few requirements for what I wanted to do:

Only allow HTTP(S) traffic through Cloudflare

Only allow DNS traffic from given sources; (location filtering, explicit white-/blacklisting);

Webhook support for logging

Should be interactive (e.g. POST /api/ban/{IP})

Detect automated vulnerability scanning

Integration with the AbuseIPDB (for checking and reporting)

As I started working on this, I realized that this would soon become more complex than I had thought at first.

Webhooks for logging This was probably the easiest requirement to check off my list, I just wrote my own log() function that would call a webhook. Sadly, the rest wouldn't be as easy.

Allowing only Cloudflare traffic This was still doable, I only needed to add a filter in my nginx config for my domain to only allow Cloudflare IP ranges and disallow the rest. I ended up doing something slightly different. I added a new default nginx config that would just return a 404 on every route and log access to a different file so that I could detect connection attempts that would be made without Cloudflare and handle them in Tabarnak myself.

Integration with AbuseIPDB Also not yet the hard part, just call AbuseIPDB with the parsed IP and if the abuse confidence score is within a configured threshold, flag the IP, when that happens I receive a notification that asks me whether to whitelist or to ban the IP - I can also do nothing and let everything proceed as it normally would. If the IP gets flagged a configured amount of times, ban the IP unless it has been whitelisted by then.

Location filtering + Whitelist + Blacklist This is where it starts to get interesting. I had to know where the request comes from due to similarities of location of all the real people that would actually connect to the DNS. I didn't want to outright ban everyone else, as there could be valid requests from other sources. So for every new IP that triggers a callback (this would only be triggered after a certain amount of either flags or requests), I now need to get the location. I do this by just calling the ipinfo api and checking the supplied location. To not send too many requests I cache results (even though ipinfo should never be called twice for the same IP - same) and save results to a database. I made my own class that bases from collections.UserDict which when accessed tries to find the entry in memory, if it can't it searches through the DB and returns results. This works for setting, deleting, adding and checking for records. Flags, AbuseIPDB results, whitelist entries and blacklist entries also get stored in the DB to achieve persistent state even when I restart.

Detection of automated vulnerability scanning For this, I went through my old nginx logs, looking to find the least amount of paths I need to block to catch the biggest amount of automated vulnerability scan requests. So I did some data science magic and wrote a route blacklist. It doesn't just end there. Since I know the routes of valid requests that I would be receiving (which are all mentioned in my nginx configs), I could just parse that and match the requested route against that. To achieve this I wrote some really simple regular expressions to extract all location blocks from an nginx config alongside whether that location is absolute (preceded by an =) or relative. After I get the locations I can test the requested route against the valid routes and get back whether the request was made to a valid URL (I can't just look for 404 return codes here, because there are some pages that actually do return a 404 and can return a 404 on purpose). I also parse the request method from the logs and match the received method against the HTTP standard request methods (which are all methods that services on my server use). That way I can easily catch requests like:

XX.YYY.ZZZ.AA - - [25/Sep/2023:14:52:43 +0200] "145.ll|'|'|SGFjS2VkX0Q0OTkwNjI3|'|'|WIN-JNAPIER0859|'|'|JNapier|'|'|19-02-01|'|'||'|'|Win 7 Professional SP1 x64|'|'|No|'|'|0.7d|'|'|..|'|'|AA==|'|'|112.inf|'|'|SGFjS2VkDQoxOTIuMTY4LjkyLjIyMjo1NTUyDQpEZXNrdG9wDQpjbGllbnRhLmV4ZQ0KRmFsc2UNCkZhbHNlDQpUcnVlDQpGYWxzZQ==12.act|'|'|AA==" 400 150 "-" "-"

I probably over complicated this - by a lot - but I can't go back in time to change what I did.

Interactivity As I showed and mentioned earlier, I can manually white-/blacklist an IP. This forced me to add threads to my previously single-threaded program. Since I was too stubborn to use websockets (I have a distaste for websockets), I opted for probably the worst option I could've taken. It works like this: I have a main thread, which does all the log parsing, processing and handling and a side thread which watches a FIFO-file that is created on startup. I can append commands to the FIFO-file which are mapped to the functions they are supposed to call. When the FIFO reader detects a new line, it looks through the map, gets the function and executes it on the supplied IP. Doing all of this manually would be way too tedious, so I made an API endpoint on my home server that would append the commands to the file on the VPS. That also means, that I had to secure that API endpoint so that I couldn't just be spammed with random requests. Now that I could interact with Tabarnak through an API, I needed to make this user friendly - even I don't like to curl and sign my requests manually. So I integrated logging to my self-hosted instance of https://ntfy.sh and added action buttons that would send the request for me. All of this just because I refused to use sockets.

First successes and why I'm happy about this After not too long, the bans were starting to happen. The traffic to my server decreased and I can finally breathe again. I may have over complicated this, but I don't mind. This was a really fun experience to write something new and learn more about log parsing and processing. Tabarnak probably won't last forever and I could replace it with solutions that are way easier to deploy and way more general. But what matters is, that I liked doing it. It was a really fun project - which is why I'm writing this - and I'm glad that I ended up doing this. Of course I could have just used fail2ban but I never would've been able to write all of the extras that I ended up making (I don't want to take the explanation ad absurdum so just imagine that I added cool stuff) and I never would've learned what I actually did.

So whenever you are faced with a dumb problem and could write something yourself, I think you should at least try. This was a really fun experience and it might be for you as well.

Post Scriptum