Mae

Hi everyone! Devs have been cranking extra hard all week on user simulations, understanding the vulnerabilities we witnessed during the initial opening, and server strengthening. This week, we did the following:

Shifted the server provider to a larger, beefier variety.

Altered the architectural structure of server queries and ID+cat generation, which should nullify the ability for cats to lock up resources.

Identified the vulnerability in founder/follower creation which was causing illegal cat traits, and have built a fix for testing.

Ran successful tests which simulated 1791 users spamming onboarding and cat generation in the span of 5 minutes.

Fixed a backend bug on the tutorial which prevents users from spamming or skipping steps to break cat data.

We're feeling good, and we believe that, come Monday, after we have a much-needed breather, we should have a date and time to announce for re-open. We still want to preform a few rounds of internal testing while running the simulations. Monday, we return with full momentum to make the final decisions on the plan forward. Which brings us to the results of the intake form! Of 920 responses, near 80% voted for a complete wipe, with a brief early login window for Closed Beta testers. From hereon, we are closing the form. Thank you for participating! In addition, 74% voted for a 3-5 day notice, 42% for an afternoon time, and 49% for a Saturday reopen! We're sorry to those who may be disappointed by this decision, and thankful to everyone who voiced their opinions and perspectives. We are sympathetic that the situation is not ideal for everyone, and we do hope to make it up the best we can to our community. Have a good weekend, catfolk!

Hi all!

As a reminder, we experienced a nasty bottleneck bug while attempting to open the application for sign-ups. This was due to a last minute fix-push to the ID system, and it was ultimately shortsighted. We’ve been working on our fixes, enforcements, and rejuvenation of the server. As of 3/23, we have pushed a supposed fix for the locking issue, however, as promised, we are taking great considerations and efforts into further testing to ensure this does not happen again. 

As we said previously, our biggest mistake was not properly testing our mechanics and making negligent judgments based on rudimentary testing methods, and we apologize and take full accountability. Our greatest priorities right now are getting users back on as fast as possible, while taking the time necessary to ensure proper quality assurance. As a 3-person dev team with limited resources, we made some faulty calls, and it’s something we have recognized and learned from.

These efforts today include mass-simulating internal user processes, strengthening the architecture, and backing up all of our current data. We are in talks with how to proceed, what is the best path forward for ensuring this does not happen again while considering the happiness of our users, and we will know more in the coming days as we draft up our game-plan. Then, we will rally our users for the future!

You all have been exceedingly positive and patient in the face of this frustration, and we thank each and every one of you, including those who offered harsh feedback; that was important too.

To give some words of reassurance: while it is a disappointing setback, PawBorough is our promise to you. The team is here to correct, change course, and communicate with full transparency every step of the way. We will have everything up and running, and there is a light at the end! We still cannot wait to see it happen, and are working tirelessly!

Below is an intake form on how we should proceed for a re-open, and what we should do with the user data accrued during the bottleneck.

There are three options going forward, as we have looked over our data, the fix, and what is doable.

VIEW THE FORM HERE!

Here are the three options laid out:

We wipe all cat data, but keep user accounts. In this scenario, any user who got stuck on email verification would be automatically verified before opening, and would be able to immediately sign in. As well, Closed Beta testers will be offered a 5-10 minute headstart, which will be determined based on general lag times and ability for users to log in. This gated entry will be quick, as it would simulate the implicit difference between those with prior accounts who helped to fund the game, and those who would have needed to create an account. For full transparency and communication: this option is easiest and quickest for us as developers. → Pros: Those who were originally stuck (testers and new players alike) are given a fair shake at sign-on. Kickstarter backers are rewarded as originally promised while not being wholly unfair, and everyone will experience the onboarding process from start to finish as intended. → Cons: Those who enjoyed their numbers will not guarantee them again, sparking disappointment.

We wipe all cat data but keep founder IDs as is. In this scenario, everyone will be let in at the same time, expected to complete onboarding with the same founder IDs they were already assigned, and follower IDs will be accrued from there. → Pros: Those who got in are promised to keep what they got. → Cons: Those who were stuck are left in the lurch.

Frankenstein the cat data, re-assigning broken cats and releasing stuck IDs. → Pros: All valid data, including founders and followers, will be restored. Everyone who managed sign-on will keep what they received. → Cons: This option has the most unpredictability, as the efforts to restore the data may have unintended side effects, or shuffle numbers around. It will take the most time and energy to do, and those initially left in the lurch are not given a fair shake.

In all scenarios, users keep their current accounts.

In all scenarios, lost followers will be manually restored via ticketing.

To request your follower be restored:

Navigate to the site https://pawborough.zendesk.com/hc/en-us/requests/new  (Account and Rewards -> Administrative Requests).

Please include a full screenshot of the follower including colours, staff will not guess them from an image. This should be done BEFORE the next open beta date.

Staff will reach out after onboarding to ask which follower should be replaced. You are able to shuffle all followers and then decide which ones to replace. You may submit 1-3 followers.

You may decide not to change your followers. We can look up followers that were created but not screenshotted if needed. IDs will not be restored.

Cats containing the founder/g2 exclusive gradient colours Diamond, Puffin, Blood, Daybreak, Mango, Parakeet, Bismuth, Chrysalis, Stardust, and Pearl; will have to change these to a colour in its existing color hue/saturation/luminosity. A choice can be made after the site releases once more.

If no choice is made, the colour will be randomized after a site-wide reminder.

Cats generated with founder/g2 exclusive white markings such as the forehead crescent moon, forehead egg, and forehead star; will be generated without them while respecting the cat's other markings.

In addition, the form inquires on what days and times users would like to be invited back in, and how much prior notice is preferred.

Hi everyone!

On 3/21 at 6pm ET, we attempted to open the application to the wider public. We had performed a variety of stress and generative tests leading up to the open registration. The only issue we ran into at the last minute was the tutorial, where we found edge cases which could brick it, and which we cut down with the intent to prevent user frustration and revisit at a later time. Everything else looked good, fixing any issues found through staff testing. We expected to see some bugs cropping up here and there, as is customary in a beta.

However, there was a big bug which our tests didn’t reveal, and that was a locking issue with cat ID generation. We believe that our tests did not catch this because we did not simulate enough user processes at once, which is an error on our part. Our developers have been working all night on tests and further strengthening of the application.

The server itself was not under any significant strain by the user influx, but the locking created a Katamari-like effect, where each new cat added to the stuck queue, thus causing significant lag, strain, and a jumble of spiraling problems.

We were working to quickly fix the issue, but it was growing just as fast. In the end, we temporarily closed the application to prevent the issue from continuing to expand. We may have needed to make this call earlier, and we regret how this may have frustrated users.

We all owe you one heck of an apology.

And we do apologize that it took ‘til now to post this writeup. Our staff needed to rest for the night before returning! Users are entirely justified in their frustration and scorn, and here is our official accountability.

Where we need to improve:

We need to ensure more complex testing. Our technical architect has been working nonstop on building and running more tests, rapidly fixing any achilles heels like the locking issue.

What we did wrong:

We were short-sighted and negligent to allow this to happen. We feel any user upset is justified, and from hereon our testing process will be more advanced and thorough.

Our perspective:

We had to cry, then we picked ourselves up and dusted ourselves off! As sad as we are to see this happen, it’s all part of the punches with a small team and indie resources.

The positives:

The bug was a nasty bugger, but despite the query issue gunking the process for everyone, our server held up quite well under the influx of users. By peak traffic, we were using only 30% of our total CPU, and that’s great! It means that, once we address the bottleneck, the server will hold up to strain.

The community was majority positive and kind, even those who were disgruntled, bringing forth just how much goodness there is in our users. We had over 1,400 new users sign up! Another huge milestone, and we’re excited to have you all playing with a positive experience in the future, which brings us to the next point:

Our Response

First off, we will be rewarding everyone who created an account, including old and new users, an exclusive Fauna item. If you got stuck at the verification step, your account still has been created and will be rewarded the Fauna. This Fauna will only be available through this event, and we hope makes up for some of the disappointment and unrest.

Second, we are restoring lost followers. If you had a follower you liked, with either evidence through a picture/screenshot or the follower did get generated, we can and will restore it!

Before reopen, we will invite users to send a ticket of their follower evidence or request, and we will communicate with you in a multi-step process to fix your follower, no matter how we proceed! After we reopen all tickets sent will be followed up with a request to clarify which follower to change to the original. Further information on this will be released in the coming days.

We will also be requesting users to give input on the following:

What we should do with the data

The chief matter on our hands now is how we should handle the data going forward. We understand that users are very disappointed. Many cats were stuck with IDs while being invalid (ghost cats) and some users were not able to access cats at (the malformed cats bug, which was caused by the bottleneck, in which users could not generate anything). We want to ask which option going forward feels the fairest for users.

We will be releasing an official, clearly outlined form on two matters:

What is the userbase's most preferred time to reopen?

How do users want the data to be dispersed?

Whether that is through a data wipe, partial data wipe, or attempting to save and restore everything we already have, we are currently going through our options and assessing what is possible. We first made a Discord poll to see where users were at, but this was ultimately erroneous on our part and executed during a time where admin and devs were frantic, fatigued, addressing users, and attempting to damage control. We apologize if this caused any further upset or confusion.

All of the options provided remain possibilities, but we need to give a clearer and more explanative breakdown of what each option entails. For this, please allow us a few days!

In conclusion

Options include

Fix and redistribute everything to broken users, but keeping everything already confirmed!

Wipe it all for fairness, with a schedule that, upon polling and communication, majority works for users.

Keep current founder numbers and supplement them on users who were stuck, while wiping followers.

Another meowth, another update! This month we have a lot of announcements to make including color contest, mod applications, and Open Beta dates!

Hello, catfolk!

Pretty soon, we're going to host our official PawBorough Colorwheel expansion event! Where users will be invited to submit colors to add to the colorwheel as a contest! By the end of the event, we will see 50+ new colors added to the wheel by users and staff.

But we wanted to ask you a crucial thing:

How do you want to handle onboarding and new colors?

The new colors won't be here before Open Beta, and even if they would be, there's no telling if we may add even more later upon user feedback. We want to decide here and now how we'll approach new colors on our colorwheel with regards to onboarding and founder creation. And we figure the best way forward is to ask you! Here are the two options:

Add new colors to onboarding as we introduce them. This means new users will have a greater pool to pick from for founder creation than existing users. Existing users will only be able to add these colors to their founders with shufflestones or the future paid feature of founder editing.

Exclude all new colors from onboarding and founder customization by default. Have new colors included only by Shufflestone and breeding use, with new users unable to add them to their founders.

On our Open Beta roadmap, one feature we want to add is the ability to re-customize your founders, as you would during onboarding, as a premium feature. If we include new colors in onboarding, the colors would also be included in this feature. If we do not, they will not be included here either.

The pro of inclusion would be a broader opportunity for color economy, and better founder customization as a whole. However! We realize that this inclusion could feel deeply unfair to legacy users, and want to ask how you all feel! If you feel this is upsetting, we want to hear it!

RECORD YOUR RESPONSE HERE: https://forms.gle/mUCbPD8x13z6VDvo8