137Foundry Web & App Design

When we ship a chunked upload feature at 137Foundry, the part that takes the most design care is the upload session state. The chunked-protocol layer (whether Tus or S3 multipart) handles the wire-level resumability. The session state layer handles everything around it: authorization, quotas, progress tracking, cleanup, and the user-visible upload status across the application.

This is a walk-through of how we structure that state, what the tradeoffs are, and what we have changed over time.

Photo by Đào Hiếu on Unsplash

What "Session State" Actually Means

An upload session is the bookkeeping for one in-flight upload. The state record tracks:

The upload ID (a UUID we control).

The owning user ID (for authorization).

The destination path (where the file will live after assembly).

The declared content type and size from the client.

The current state (pending, uploading, processing, ready, rejected, expired).

Timestamps (created, last activity, completed).

A reference to the underlying protocol session (Tus session ID, S3 multipart upload ID, or equivalent).

The session record is what the application server uses to authorize chunk operations, report progress to the user, and clean up after abandoned uploads. It is separate from the actual file content, which lives in object storage.

Where the State Lives

We store session state in our primary application database (Postgres in most projects). The alternative is a key-value store (Redis), which is faster but loses the data on restart unless persistence is configured.

Postgres wins for us because the session state is small (a few hundred bytes per session), the query rate is low (one query per chunk on average, which is well within database capacity), and the consistency guarantees match what we want (no lost session records, transactional updates when state transitions).

For very high-throughput upload systems, Redis or a dedicated KV store can make sense. The session state becomes a hot path and the in-memory access pattern is much cheaper than database round-trips. Most of our projects do not reach that scale, so the Postgres pattern stays.

State Transitions

The session record moves through a small number of states. The transitions we use:

pending -> uploading: The client sends the first chunk. Once any chunk has been received, the session is actively uploading.

uploading -> processing: The client signals completion (or the protocol detects all chunks have arrived). The application server validates the chunk count, triggers post-upload processing (virus scan, content validation, metadata extraction), and the user sees a "processing" indicator.

processing -> ready: All post-upload processing has completed successfully. The file is now accessible to the user.

processing -> rejected: Post-upload processing flagged the file as invalid (virus scan failed, content type mismatch, format check failed). The file is moved to a quarantine location or deleted, the session is marked rejected, and the user is notified.

pending or uploading -> expired: No activity for the configured expiration window (we use 7 days). The session is marked expired and the partial data is cleaned up.

Each transition writes to the database with a timestamp. The audit trail is useful for debugging support tickets and for understanding upload patterns over time.

Per-User Quotas

The session state record is where per-user upload quotas are enforced. Before creating a new session, we check the user's current usage against their plan limits.

The check happens at two levels. Storage quota: the total size of all files this user owns. If the new upload would push the user past their limit, we refuse to create the session. Concurrent upload quota: the number of in-progress uploads for this user. We limit this (typically to 5 to 10 concurrent uploads per user) to prevent abuse and to keep our upload-session table from growing unbounded.

For users on enterprise plans with higher limits, the quota values are pulled from the user's plan tier. For users on free or trial plans, the limits are tighter to encourage upgrade. The session state layer is where these business rules attach to the underlying upload infrastructure.

Progress Reporting

The session state record tracks the current upload offset (in bytes). Every time the client successfully uploads a chunk, the offset advances. The application server can report progress to other clients (e.g., a multi-window user, a real-time UI on another tab) by reading the current offset.

For real-time progress display, we use server-sent events or WebSockets to push offset updates to interested clients. This is mostly used for collaboration features (showing another team member that an upload is in progress, displaying the progress in a shared workspace).

For single-user progress display, the upload progress is reported by the client itself based on the bytes it has sent. The server-side offset is mainly for cross-client visibility and for resume support.

Resume Support

The session record makes resume possible. When a client reconnects after a network drop and wants to resume an upload, the flow is:

Client queries the application server for the session's current state.

Application server checks authorization (this user owns this session) and returns the current offset.

Client resumes upload from the reported offset.

For Tus-based uploads, the resume protocol is built into the wire format (HEAD request returns the offset) per the Tus specification. For S3 multipart uploads, the equivalent is ListParts which returns the parts that have been uploaded so far.

The session state record allows resume across longer time gaps. A user who closes the browser and returns days later can have their upload resume because the session record persists. The Tus session and the S3 multipart upload may have a shorter lifetime than the session record, but the application database always has the source of truth on what was happening.

Cleanup of Abandoned Uploads

Without cleanup, the upload table and the storage location fill up with partial uploads from users who never finished. We run two cleanup jobs.

The first is a scheduled job that scans for sessions in pending or uploading state with no activity for more than 7 days. These get marked expired and the partial chunks are deleted from storage.

The second is a per-user limit. If a user has more than the maximum concurrent uploads (10 in our default config), the oldest stale session gets expired to make room. This catches users who repeatedly start uploads without finishing them.

The cleanup is observability-friendly. We log every expiration with the user ID, file size, and stale duration. The data is useful for identifying users with broken upload flows (frequent abandons) and for capacity planning.

Authorization at Every Step

The session record is the anchor for authorization. Every operation against the upload (chunk uploads, status queries, completion calls, cancellation) checks that the requesting user matches the session's owner.

For multi-tenant applications, the session also encodes the tenant. Cross-tenant access is impossible because the URL paths in object storage include the tenant ID, and the session record's owner check would reject any cross-tenant attempt before reaching the storage layer.

The OWASP file upload cheat sheet covers the broader authorization considerations. The pattern we use enforces ownership at the session level (one user owns a session) and tenant isolation at the storage path level (no two tenants can share storage paths).

Recovery From Server-Side Failures

If our application server crashes mid-upload, the session state record survives because it is in the database. When the server comes back up, in-progress sessions can continue from where they left off. The chunk-level resume protocol (Tus HEAD, S3 ListParts) recovers the actual upload state.

If the database itself fails, the upload sessions are unavailable until recovery completes. We use Postgres' standard replication and failover patterns for this. The mean time to recovery is low enough that most uploads can pause and resume successfully.

If the storage backend (S3) has an outage, the chunks cannot be uploaded. The session stays in uploading state, and the client retries with exponential backoff. Once S3 recovers, the uploads resume. This pattern survives multi-hour S3 outages cleanly, which has been useful during the few S3 region-wide events we have seen.

Photo by Christina Morillo on Pexels

Schema We Use Now

The Postgres schema for upload sessions, simplified for readability:

CREATE TABLE upload_sessions ( id UUID PRIMARY KEY, user_id UUID NOT NULL REFERENCES users(id), tenant_id UUID NOT NULL, destination_path TEXT NOT NULL, declared_size BIGINT NOT NULL, declared_content_type TEXT NOT NULL, current_offset BIGINT NOT NULL DEFAULT 0, state TEXT NOT NULL CHECK (state IN ('pending', 'uploading', 'processing', 'ready', 'rejected', 'expired')), protocol TEXT NOT NULL CHECK (protocol IN ('tus', 's3_multipart')), protocol_session_id TEXT, created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), last_activity_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), completed_at TIMESTAMPTZ ); CREATE INDEX upload_sessions_user_active ON upload_sessions (user_id, last_activity_at) WHERE state IN ('pending', 'uploading'); CREATE INDEX upload_sessions_cleanup ON upload_sessions (last_activity_at) WHERE state IN ('pending', 'uploading');

The two indices support the most common query patterns: finding active uploads for a user (for quota checks and progress reporting) and finding stale uploads for cleanup. Both are partial indices that only include the relevant states, which keeps them small.

What We Changed Over Time

Looking back at the upload-session schema we shipped in the first project versus now:

We added the tenant_id column once we built our first multi-tenant product. The original single-tenant schema was missing it and we had to migrate.

We added the protocol and protocol_session_id columns when we started using S3 multipart alongside Tus. The original schema only knew about Tus session IDs.

We added the processing state when we added asynchronous virus scanning. Before that, sessions went straight from uploading to ready once the bytes were assembled, and the scan results were not represented in the state machine.

We added the last_activity_at index for cleanup once the table grew enough that scanning it became slow. The original cleanup query was a full table scan that worked fine for the first year and started to hurt by month 18.

Each addition was driven by a specific operational pain point. Building this from the start would have saved some migration work, but we did not know about most of these pain points until we hit them.

Where 137Foundry Helps

Upload pipelines are one of the recurring engineering surfaces we ship for clients. The 137Foundry services page covers the broader engineering work that surrounds these. The 137Foundry web development services page is the specific landing point for upload work. The full architectural framing is in the longer guide on resumable file uploads.

A Closing Note on Boring Code

The upload session state code is some of the most boring code we write. It is also some of the most important. Mistakes here produce hard-to-debug bugs in production months after launch. Getting the state machine right, the indices right, the cleanup right, and the authorization right at the start saves a lot of pain later.

We have shipped file upload features for several SaaS products over the years at 137Foundry. The first one taught us most of the lessons. Every one since has refined the playbook a little, but the architecture and the testing approach have stayed remarkably stable since we got it right the first time.

This is a behind-the-scenes look at what changed between our first naive upload feature and the one we ship now by default.

Photo by Anna Shvets on Pexels

The First Version Was Wrong in the Obvious Ways

The first upload feature we shipped at 137foundry.com was a single fetch with the file as the request body. The endpoint accepted up to 50MB and most uploads worked. The ones that did not work, we treated as user error or transient issues. The support tickets started arriving within two weeks of launch.

The tickets clustered. Users on mobile networks had failed uploads about three times more often than users on wired connections. Users in regions with worse-quality internet had higher failure rates than users in metro areas with fiber. The pattern was obvious in hindsight: our upload system worked for users with our office network conditions, and not for anyone else.

We rebuilt it. The second version used chunked uploads with retry logic, but we still had a single application server in the chunk path. Bandwidth costs went up. We could see in the metrics that the upload-server pool was the bottleneck during traffic spikes.

The third version moved the actual chunk bytes directly to S3 via pre-signed URLs. The application server only handled session creation, authorization, and completion notification. Bandwidth costs dropped substantially because we were no longer running the upload bytes through our application infrastructure.

That third version has been the working pattern for all subsequent products.

The Lessons That Stuck

A few specific things we wish we had known on day one.

The network is more hostile than office testing suggests. Our office WiFi rarely dropped connections. Production users had connections drop multiple times per hour, and our upload system had to survive that. Test environments need to reproduce real-world network conditions, not just simulate slow speeds.

Single-request uploads have an upper bound that is much lower than the file size limit suggests. We had a 50MB upload cap because that is where our application server's request handling started to struggle, but real users were trying to upload files much larger than that. The right answer was not to raise the cap; it was to change the architecture.

Pre-signed URLs are the architectural move that makes uploads scale. Once the chunk bytes flow directly to S3 instead of through the application servers, the application server count becomes independent of the upload bandwidth. We could scale uploads without scaling application server capacity, which we could not do with the mediated pattern.

The resume protocol matters more than the chunked protocol. Splitting an upload into chunks is straightforward; designing the resume protocol to handle partial failures cleanly is the hard part. We use the Tus protocol when the application is not deeply tied to S3, and S3 multipart uploads when it is. Both handle resume natively.

Background scanning is the only pattern that works for virus checks. Inline scanning during upload makes the user experience painful (multi-minute waits after the bytes have transferred) and the operational story painful (CPU spikes during upload traffic). Asynchronous scanning with a "processing" state in the UI is the only approach that scales.

The Bugs That Surprised Us

A few specific bugs caught us off guard despite reading other teams' write-ups before we built.

The "completed but missing" bug. The client's perspective is that the upload finished (all bytes sent, success response received). The server's perspective is that the final completion call failed to record the upload in our database. The client thinks the file is uploaded; our system thinks no upload happened. The user logs in the next day and the file is missing.

The fix was to make the completion step idempotent and to let the client re-confirm via a HEAD request if the original response was lost. This is built into the Tus protocol but we had to add it explicitly to our own variant before we migrated to Tus.

The "wrong content type" bug. A user uploaded a file labeled as PDF that was actually executable content with a renamed extension. Our content validation only checked the declared content type, not the file's actual format. We added inspection of the file header bytes after a security review caught this. The OWASP file upload cheat sheet covers the broader pattern; we had not read it carefully enough on the first pass.

The "chunk size too small" bug. We initially used 1MB chunks because the math seemed to suggest smaller chunks meant faster recovery from drops. In practice, the HTTP overhead per chunk added up to substantial total upload time, especially on slower connections. We moved to 5MB chunks and the average upload time dropped 30 percent without making the failure recovery noticeably worse.

The "long-term partial upload" bug. Some users started uploads and never returned. Their partial uploads sat in our quarantine bucket forever, slowly consuming storage. We added a 7-day expiration on incomplete uploads with a daily cleanup job. The bucket size stabilized and the cost stopped creeping up month over month.

What We Test For Now

Before any new upload feature ships, our team at 137foundry.com runs through a specific test checklist that has caught most of the bugs we have seen historically.

The 1GB upload test. Upload a 1GB file from start to finish over a stable connection. The test confirms the chunking works, the assembly works, and the post-processing pipeline handles a real-world file size.

The throttled-connection test. Upload the same 1GB file with browser dev tools throttling the connection to 1 Mbps. The test confirms the upload completes within a reasonable time, the progress reporting stays accurate, and the system does not time out anywhere along the path.

The drop-mid-upload test. Start a large upload, disable the network mid-upload, wait a minute, re-enable the network. The upload should pause cleanly, surface a "connection lost" indicator to the user, and resume from where it stopped when the network returns.

The chunk-corruption test. Inject random bit flips into one chunk via a proxy that mangles the payload. The server should detect the corruption (via checksums or hash comparison) and request a retry of that specific chunk.

The browser-close test. Start an upload, close the browser tab, wait an hour, return to the application. The application should offer to resume the partial upload if the context allows it, or at minimum should not silently lose the partial data.

Passing those five tests covers the failure modes that produced almost all of our historical support tickets. A system that fails any of them will produce tickets within a month of launch.

The Stack We Use Now

For most projects we run, the upload stack looks like this.

Storage: S3 (or an S3-compatible object store for self-hosted deployments).

Wire protocol: Either Tus or S3 multipart, depending on the application's storage abstraction. Tus when storage is abstracted (we might move backends), S3 multipart when storage is fixed.

Authorization layer: Application server generates pre-signed URLs or upload session tokens. Per-user quotas and content restrictions enforced at this layer.

Validation pipeline: Asynchronous workers pick up completed uploads, run content-type validation, virus scanning, and any application-specific checks. Files transition from "pending" to "ready" or "rejected" based on the result.

UI: Real-time progress with continuous bytes-per-second updates, graceful "connection lost" states with auto-retry, and explicit "processing" states between upload completion and file availability.

The full architectural framing is in the longer guide on resumable file uploads at 137foundry.com.

Photo by Paul Seling on Pexels

When to Build vs Buy

We have built upload pipelines and we have also integrated third-party services (Uppy with a hosted Tus server, Filestack, Cloudinary). The decision usually comes down to a few variables.

Build when: uploads are central to the application's value, when scale is high enough that bandwidth costs through a third party would be significant, when the application needs custom validation or processing during upload, or when long-term operational simplicity matters more than initial time-to-launch.

Buy when: uploads are a minor feature, when scale is low or unpredictable, when the team has no prior experience with upload infrastructure, or when getting to market fast matters more than long-term cost optimization.

For most products we work on at 137foundry.com, the build path wins because the products tend to have meaningful upload usage and the operational cost of a self-managed stack is small once it is built. For smaller products or quick prototypes, the buy path is often the right choice and we recommend it without hesitation.

A Closing Note for Other Teams

If your team is about to build a file upload feature for the first time and you are reading this, the two pieces of advice that would have saved us the most time:

Skip the single-request upload pattern entirely. Even if your file size cap is modest, the network reality you cannot see is going to bite you. Chunked from day one is a small extra investment that prevents a substantial rebuild later.

Test the failure modes specifically. The happy path will work. The interesting bugs are in the failure recovery. Set aside time for the drop-mid-upload test and the chunk-corruption test before declaring the feature ready.

We have worked on enough SaaS onboarding redesigns over the last few years that a pattern has become hard to ignore. The instinct on every project is to add new screens, new tutorials, new tooltips, new prompts. The reality on every project is that the biggest activation lift comes from removing screens, not adding them.

This is a counterintuitive observation when you say it out loud. A team that has spent weeks discussing how to "improve" onboarding is rarely thinking about subtraction. The brief is usually some version of "make it better", which implicitly assumes additive change. The product manager pitches a guided tour. The designer mocks up a more sophisticated empty state. The growth lead wants a personalized welcome screen. Everyone is proposing additions.

Photo by Akshar Dave🌻 on Pexels

What the Data Actually Shows

When we instrument a SaaS onboarding funnel and look at where users drop off, the biggest losses are almost always at the screens that were added recently. The original signup-to-first-action flow was usually short and direct. The accumulated additions are usually the source of the drop-off.

Across the projects we have run at 137Foundry, the pattern is consistent:

Original flow at product launch: signup, password, dashboard, first action. Four screens.

Flow after 24 months of additions: signup, password, email verification, team setup, profile completion, plan selection, tutorial overlay, settings preview, integration prompts, dashboard, first action. Eleven screens.

Activation rate after 24 months of additions: roughly half what it was at launch.

The team adding each screen had a reasonable argument. The email verification was for security. The team setup was for collaboration. The profile completion was for personalization. Each addition, in isolation, was defensible.

In aggregate, they killed activation.

The Mental Model That Helps

The mental model we use is to ask, for every screen, "if a user closes the tab on this screen, what fraction of them will come back?" The answer is almost always lower than the team assumes. Most users who close the tab during onboarding never come back. The cost of every screen between signup and first useful action is the percentage of users who drop off at that screen, multiplied by the lifetime value of those users.

This makes additions much harder to justify. The team adding the email verification screen is implicitly accepting that some percentage of acquired users will be lost to it. The percentage is rarely measured. When we instrument it and show them the number, they are usually surprised.

When Additions Actually Help

We are not anti-addition. Some additions help, and some do not, and the difference is measurable rather than aesthetic.

Additions that help: sample data on first login, defaults on first-action forms, inline-first-action patterns that compress multiple steps into one, contextual prompts that surface only when the user has shown intent for the related action.

Additions that do not help: forced tutorial overlays, pre-task configuration wizards, email-gated unlocks, multi-step team setup wizards before the first useful action, mandatory profile fields that the product does not strictly need.

The pattern: additions that reduce the path to first useful action help. Additions that lengthen the path hurt. This sounds obvious. In practice, almost every addition lengthens the path, because the path-shortening additions tend to be invisible (a default value, an inline form, a sensible sample dataset) while the path-lengthening additions tend to look like progress (a new screen, a new feature, a new prompt).

The Practice We Use

For every onboarding addition we propose to a client, we ask three questions before we agree to ship it:

Does this addition shorten or lengthen the path from signup to first useful action?

What percentage of users do we expect to drop off because of this addition?

What value does the addition produce that justifies the drop-off cost?

If the addition lengthens the path, the answer to question 2 is non-zero. The question becomes whether the value in question 3 justifies the cost. Most of the time, the answer is no, and we propose a path-shortening alternative.

Email verification is the classic example. It lengthens the path. Some percentage of users drop off because of it. The value (preventing fake signups, enabling password reset, complying with anti-spam rules) is real. But the value can usually be captured without forcing verification on day one. The user can use the product immediately, and verification is required only when they take an action that needs it -- inviting a team member, configuring billing, contacting support.

The same logic applies to most other "required" additions. Configuration that matters for the user's third or fourth action can be deferred to the third or fourth action. The first action almost never needs more than sensible defaults.

What This Looks Like in Practice

At 137Foundry, our onboarding work for clients usually starts with a screen inventory. We list every screen between signup and first useful action and assign each one a cost (the percentage of users who drop off at it) and a value (the function it serves). The work that follows is mostly about cutting screens whose cost exceeds their value.

This is unglamorous work. The output is not a beautiful new design system or a clever new feature. The output is a shorter flow that the team built in the first place but bloated over time. The activation rate moves. The downstream metrics move with it. Nobody puts "removed 4 unnecessary onboarding screens" on their portfolio site.

For the framework we use, the 137Foundry guide on onboarding drop-off walks through the systematic redesign process. The services we offer include onboarding flow audits for SaaS and B2B products.

External design references we keep coming back to: the Nielsen Norman Group research on activation funnels, the Smashing Magazine editorial archive on UX case studies, and the broader W3C accessibility guidelines which intersect onboarding design in ways that most teams underweight.

The Mindset Shift

The hardest part of this work is the mindset shift. Teams who have been adding to the onboarding flow for two years find it counterintuitive to subtract. The screens that we propose cutting were added by people who are still on the team and remember the meeting where they argued for each addition.

The conversation about cutting them has to be framed against the activation data. "We are not saying your screen was a bad idea. We are saying the funnel data shows it is costing more than it is producing. Here is the number." When the conversation is framed this way, the team usually agrees to cut. When the conversation is framed as a design preference, the team usually defends the screen and the cut does not happen.

The Compounding Effect

The teams that get good at cutting onboarding screens tend to stay good at it. The discipline becomes part of how they ship product. Every new feature gets evaluated against the activation funnel. The accumulation of cruft slows. The activation rate stays stable or improves over time.

The teams that do not develop the discipline tend to drift in the other direction. The onboarding flow gets longer. The activation rate falls. The team blames upstream acquisition channels for the rising cost-per-activated-user, when the real problem is the flow that has accumulated 24 months of "improvements".

We have worked on both kinds of teams. The first kind is more fun to work with. The work is sharper and the results are easier to see. The second kind takes a few months to acclimate to the data-first mindset, but once they do, the same pattern emerges.

Across the SaaS onboarding redesigns we have run at 137Foundry, five patterns come up over and over as the dominant sources of activation drop-off. Each of these is technically defensible in isolation. In aggregate, they explain most of the gap between signup rate and activation rate.

If a product team is wondering why activation is below industry benchmarks, the answer is usually one or more of these five.

Photo by Walls.io on Pexels

1. The Empty Dashboard

The user signs up, lands on the main dashboard, and sees nothing. No projects. No data. No model for what success looks like. The dashboard has a "create your first project" button at the top, but the button leads to a blank form that expects the user to invent something from scratch.

A meaningful percentage of users close the tab here. They did not sign up to invent a project; they signed up because someone told them the product would help them with work they were already doing. The empty dashboard is asking them to do work the product was supposed to do for them.

The fix is sample data on first login, defaults that get the user to a populated state in one click, or an inline first-task flow that does not require navigating through the empty dashboard. The Nielsen Norman Group has documented this pattern across many product categories.

2. The Configuration Wizard Before First Use

The user signs up to send invoices, and the product asks them to configure currency, tax settings, payment methods, business hours, notification preferences, and team structure before they can send anything. The configuration wizard has 8 to 12 steps. The user came to send an invoice, not to configure a SaaS product.

The configuration wizard is a leftover from the enterprise-software era when products assumed users would deploy them carefully over weeks before using them. The pattern translates poorly to self-serve SaaS, where the user has 5 minutes to decide whether the product is worth more of their time.

The fix is to defer every configuration question until the moment it actually matters for the task at hand. The user can send an invoice with defaults. If the defaults are wrong for this specific invoice, surface the configuration question inline on the invoice screen. The full settings page is for users who have already gotten value from the product, not for users who are deciding whether to.

3. The Forced Tutorial Overlay

The user lands in the product and immediately sees an overlay that walks them through every feature on the screen. Each feature has a callout box with explanatory text and a "next" button. The user has 7 to 12 callouts to dismiss before they can interact with the product.

The forced tutorial is one of the most reliably hated onboarding patterns. Users skip them, dismiss them, and -- in the worst cases -- close the product entirely rather than dismiss them. The information in the tutorial is rarely retained because the user has no context for why the features matter.

The fix is to make tutorials optional and contextual. A small "show me around" button that the user can click when they want a tour, paired with inline hints that appear only when the user is about to take a related action, outperforms forced tutorials on every metric we have measured.

Smashing Magazine has published case studies on tutorial overlay performance. The consistent finding is that optional tutorials with contextual triggers outperform forced overlays by 20 to 40 percentage points on retention metrics.

4. The Email Verification Wall

The user signs up, provides their email, and is told to check their inbox to verify before they can use the product. They do not check the inbox. They close the tab. They never come back.

Email verification has legitimate purposes -- preventing fake signups, enabling password reset, complying with anti-spam rules. The mistake is requiring verification before first use, which turns the verification step into a major drop-off point.

The fix is to allow immediate access to the product while making verification a prerequisite for actions that actually require it. Inviting team members, configuring billing, contacting support -- these can require verification. Using the product for the first time should not.

5. The Profile Completion Demand

The user lands in the product and is shown a profile completion form. Photo, job title, company size, role, phone number, country, time zone. The form is 8 fields. The user is told they cannot use the product until they complete it.

The profile completion form is almost always for the company's benefit, not the user's. The company wants demographic data for sales segmentation, ICP analysis, or marketing. The user wants to use the product. The mismatch produces drop-off.

The fix is to make profile fields optional and surface the questions over time as they become relevant. The product can get most of the demographic data through behavior rather than self-report. The fields that are genuinely necessary for the product to function (timezone for scheduling, role for permissions) can be defaulted to sensible values and surfaced only when the defaults break.

Photo by Kawê Rodrigues on Pexels

What All Five Have in Common

The pattern across all five: each one prioritizes something the team wants over something the user wants. Configuration, verification, demographic data, education, and structured setup are all team priorities. The user priority is "let me use the product".

When the team priorities are forced on the user before they can use the product, the user leaves. This is not a UX problem in the narrow sense; it is a prioritization problem. The team is signaling that their priorities matter more than the user's, and the user is responding by going elsewhere.

The fix in every case is to defer the team priority until the user has gotten value from the product. Once the user has experienced what the product can do, the team priority becomes negotiable. Configuration, verification, demographic data, education, and structured setup all become acceptable once the user has a reason to want the product to keep working.

How We Find These Patterns

The diagnostic process is straightforward. At 137Foundry, we run a free onboarding flow audit on SaaS products where we sign up from scratch and walk through every screen, take notes on where the friction appears, and produce a prioritized list of cuts and changes.

The pattern is consistent. Most products have 3 to 4 of the five patterns above. The audit identifies which ones are most expensive in terms of activation impact and proposes specific changes. The implementation work is usually 4 to 8 weeks, and the activation improvement is usually 10 to 30 percentage points.

For the framework behind the audit, the 137Foundry guide on onboarding drop-off walks through the redesign process. The services we offer include onboarding flow audits for SaaS and B2B products.

External design references we keep coming back to: the Nielsen Norman Group research on activation funnels, the Smashing Magazine editorial archive on UX case studies, the W3C accessibility guidelines on progressive disclosure, and the MDN UI patterns reference for the technical implementation patterns.

Closing Thought

The five patterns are not a comprehensive list of every onboarding problem we see, but they account for most of the activation drop-off we have measured across client projects. Teams that fix these five usually move activation by enough to make the redesign work pay for itself within a few quarters.

The harder part is institutionalizing the discipline. The empty dashboard, the configuration wizard, the forced tutorial, the email verification wall, and the profile completion demand all came from someone proposing them in a meeting and the team agreeing. Avoiding the next round of accumulation requires building the activation-first habit into how the team evaluates new product proposals.

Writing regex without good tooling is like writing SQL without an EXPLAIN plan. You can do it, but you should not have to. Over years of building production validation and data ingestion layers at 137Foundry, a small toolkit of regex utilities has earned a permanent place in our workflow. This is a roundup of those tools, what each is good for, and which is wrong for which job.

The criteria for inclusion: actually used in our work, available for free (most have paid tiers but the free tiers are sufficient), and providing capabilities that meaningfully improve regex correctness or performance.

1. regex101

regex101 is the most-used regex tester in our toolkit. It supports PCRE, ECMAScript, Python, Java, and Go dialects, with side-by-side input testing, capture group highlighting, and a step-count metric that reveals catastrophic backtracking before it hits production.

The features that matter most for production work:

Step counter. Patterns with more than a few thousand steps on small inputs are likely vulnerable to ReDoS attacks. The step counter makes this visible before deployment.

Multi-dialect testing. The same pattern can behave differently across regex dialects. Being able to switch dialects and re-test catches subtle bugs.

Saved patterns. Patterns can be saved with URLs that include the test input, which makes them easy to share in code review or document in design docs.

Explanation panel. For complex patterns, the breakdown of what each token does is faster than reading the regex character by character.

The free tier is generous and covers nearly every realistic use case. Paid tier adds team features and more saved patterns.

2. regexr

regexr is an alternative regex tester with a slightly different feature set. It is faster for quick interactive testing (the UI is lighter) and includes a cheat sheet sidebar with quick references for common patterns.

Where it shines: rapid iteration on a regex while watching live highlighting of matches. The lighter UI makes it less distracting for short sessions.

Where regex101 wins: deeper analysis (step counting, dialect comparison), saved patterns with test input, and broader dialect support.

We use both. regexr for quick "does this pattern match what I think?" checks. regex101 for production-bound patterns and ReDoS analysis.

3. MDN Regex Reference

The MDN documentation on JavaScript regex is the canonical reference for ECMAScript regex behavior. It covers every token, every flag, browser-specific quirks, and the gradual evolution of the regex spec across ECMAScript versions.

For any regex destined to run in browser JavaScript or Node.js, MDN is the first stop when you are unsure of behavior. The lookbehind support in ES2018, the Unicode property escapes added in ES2018, and the dotAll flag are all documented there with examples.

For other languages, the analogous canonical references are: Python's re module docs, the PCRE documentation, and the Java regex Pattern class documentation.

4. Atom-Style Verbose Mode

Not a tool per se, but a practice worth highlighting. Python's re.VERBOSE flag (and PCRE's equivalent extended mode) let you write multi-line, commented regex:

EMAIL = re.compile(r""" ^ [A-Za-z0-9._%+-]+ # local part @ [A-Za-z0-9.-]+ # domain \. [A-Za-z]{2,} # TLD $ """, re.VERBOSE)

This is unrelated to a specific tool but transforms regex from a write-only language into a readable one. Languages that do not support verbose mode (JavaScript, sadly) get the same effect by composing the pattern from named string variables and constructing the final regex with new RegExp(parts.join('')).

5. Language-Specific Linters

Most modern linters can flag obvious regex problems before they reach production. ESLint has regex-specific rules. Bandit (a Python security linter) flags regex patterns with ReDoS risk. Spotbugs catches problematic regex in Java.

These rarely catch sophisticated bugs but consistently catch the obvious ones: unanchored patterns where anchors are expected, missing escapes, nested quantifiers in untrusted-input paths. Worth enabling.

6. A Test Corpus Approach

Not a tool but a discipline: maintain a test corpus of known-bad inputs alongside the regex patterns themselves. Every time a production incident reveals an input the regex should have rejected (or accepted), add the input to the corpus. The corpus becomes the institutional memory of "what we have learned about real input distributions."

The corpus is more valuable than any external tool because it captures the bugs that are specific to your domain. Generic regex testers catch generic problems. The corpus catches the problems that hit your users.

We track corpora as plain text files in the repos for each pipeline, with categories like valid_inputs.txt, invalid_inputs.txt, and historical_failures.txt. The free regex documentation by 137Foundry maintained in our internal docs includes example corpora for common input types, available on the 137Foundry site.

7. Compilation Profiling

For high-throughput regex (running thousands of times per second in a pipeline), compilation overhead matters. The general advice is "compile once, reuse the compiled pattern" in any language that supports it.

Profiling tools for the host language reveal whether you are actually doing this. In Python, cProfile shows time spent in re.compile() calls. If the same pattern is being compiled repeatedly, the fix is moving the compile out of the hot loop and into module-level or class-level constants.

In JavaScript, regex literals (/pattern/) are cached automatically. new RegExp(string) inside a loop is not. The fix is the same: build the regex once.

8. ReDoS Detection Tools

For regex patterns running on user-controlled input, ReDoS detection is worth automating. Several open-source tools analyze regex statically for backtracking risk:

rxxr2 for academic-grade analysis (slower but thorough)

safe-regex for Node.js codebases (npm-installable)

OWASP regex linter in various forms

These do not catch everything. They catch the patterns with obvious backtracking risk, which is the majority of ReDoS issues. For security-sensitive paths, running one of these against your regex inventory periodically is worth the time.

The OWASP ReDoS cheat sheet covers the underlying theory and the patterns most likely to be vulnerable. Worth reading once if you build regex into security-sensitive paths.

9. Live-Reloading Regex Test Suites

The fastest way to iterate on regex is a unit test file that runs on save. Add the test for the new input you want to handle, watch it fail, modify the regex, watch it pass, watch the previous tests still pass.

This pattern works in any language with a fast test runner: Jest in JavaScript, pytest in Python, JUnit with continuous testing in Java. The key is that the feedback loop is under a second; anything slower and you lose the interactive flow.

We pair this with the regex101 web tester for exploratory work and a unit test file for committed patterns. Exploratory: in the browser, fast iteration. Committed: in the test suite, permanent regression coverage.

10. Plain Old Documentation

The least-glamorous but most useful tool is plain documentation alongside the regex. Every pattern in our production code has a comment that includes:

What inputs the pattern accepts (examples)

What inputs the pattern rejects (examples)

What inputs it deliberately does not try to handle

Known assumptions about input format

This kind of documentation makes the next reader (often the original author six months later) able to understand what the pattern is doing without reverse-engineering it. The documentation is more valuable than the pattern itself when the assumptions need to change.

Putting the Toolkit Together

A practical workflow:

Sample real input. Inspect the variations.

Sketch a pattern in regex101. Watch it match.

Test against the boundary cases. Watch some of them fail.

Adjust. Test the step counter for backtracking risk.

Commit with documentation, test cases, and a corpus entry for any past failure mode.

Profile in production. Adjust compilation strategy if needed.

Periodically run static analysis on the regex inventory.

Each step uses one or two of the tools above. No single tool covers the whole flow. The combination is more useful than any single tool.

We see a lot of production data pipelines and validation code at 137Foundry. After enough postmortems, a pattern has emerged. The bugs everyone blames on regex are almost never actually regex bugs. They are assumption bugs. The regex did exactly what it was told to do. What it was told to do was based on a wrong understanding of what the input would look like.

This sounds pedantic, but it changes how you should think about regex in production. Here is what we mean.

The Pattern Behind the Pattern

A typical regex bug postmortem goes like this. A function is processing user input, validating it with a regex, and one day it crashes on a record that "should have been rejected." The team digs in. The regex is reviewed. Sure enough, the regex accepts the malformed input. Someone updates the regex. The bug is filed as a "regex bug."

But the regex was written against an assumption: that the input would always be ASCII, or always have at most three digits, or always end in .com. The assumption was wrong. The regex faithfully implemented the assumption. Calling it a regex bug is like calling a hammer a bad hammer because it does not drive screws.

The actual bug was upstream. Someone made an assumption about input distribution and codified it in a pattern. The assumption was incomplete. The fix is not just a better regex; the fix is a better process for noticing when assumptions about input do not hold.

How This Plays Out in Production

A few examples from the field, anonymized:

The phone number import. A regex validated US phone numbers in a customer onboarding flow. It worked for a year, then suddenly started rejecting one customer in a hundred. The customers were not malformed. The customers were Canadian (same format as US, but the regex accidentally required area codes starting with 2-9, which fails for some Canadian regions).

The regex was correct under the assumption "all customers are US-based." The assumption stopped being true. The "regex bug" was actually a product expansion bug.

The email validation incident. A regex rejected addresses with + characters in the local part, breaking signups for users on Gmail who used plus-tagging ([email protected]). The regex was wrong under any reasonable email validation rules. But the underlying problem was that nobody on the team had ever used Gmail plus addressing, so the assumption "valid emails do not contain plus signs" was unchecked.

The CSV parser explosion. A regex pattern was used to split CSV-like lines. It worked perfectly for two years. Then a customer uploaded a CSV from Excel on Windows that included a Unicode byte order mark () at the start of the file. The first field of the first row was now customer_id instead of customer_id, and every downstream lookup against the first field failed. The regex did not have a bug. The assumption "CSV files do not start with BOM characters" had a bug.

In each case, fixing the regex addressed the immediate problem. But the underlying issue was an assumption about input that had gone unverified.

What This Looks Like to Fix Properly

The fix that scales is not "write more elaborate regex." More elaborate regex codifies more elaborate assumptions, which means more elaborate failure modes when the assumptions are wrong. The fix that actually scales is a process for surfacing and testing assumptions about input.

A few concrete practices we use at https://137foundry.com:

Sample real input before writing regex. If the input is user-controlled, grab a sample of recent inputs from production and inspect them. If the input is from a partner system, request a representative file. The variations in real input are almost always wider than the variations you would have guessed.

Document what the regex assumes. A comment on the regex listing the input characteristics it assumes (ASCII only, no whitespace, US format, no fractional seconds) makes the assumption explicit. When the assumption stops holding, the comment lets the next reader see immediately what changed.

Test against pathological input. Empty strings, leading whitespace, trailing whitespace, mixed line endings, BOM characters, Unicode characters, very long strings. These are the inputs production will throw at you eventually. Test them now, not later.

Log rejections instead of silently dropping them. A regex that rejects a few percent of input over time tells you something about your assumptions. A regex that rejects silently lets the assumption drift be invisible until it causes a customer-visible issue. Log the rejected inputs with structured metadata. Periodically review the rejections to see if the assumptions still hold.

Test the assumption migration. If the regex's assumptions change (new country supported, new client integration with different format), update the test corpus first. Confirm the regex correctly handles the new shape before deploying. This catches more bugs than any amount of in-regex sophistication.

Why "Just Use a Library" Is Not Always the Answer

A common reflex when we make this argument is "well, then use a library instead of regex." This is sometimes right and sometimes wrong.

For email validation, parser libraries are generally not better than the pragmatic regex. They have the same fundamental limitation: format does not equal reachability. A library that returns "valid" for an unreachable address is no better than a regex that does the same.

For phone numbers, libraries (libphonenumber) are much better than regex because the rules are genuinely too complex for regex to encode correctly. The library handles every country's format quirks.

For URL parsing, the language's built-in parser is almost always better than regex. URL canonicalization, query parameter handling, and IDN decoding are too subtle for regex to get right.

The pattern: when the underlying rules are complex and well-specified, use a library. When the underlying rules are loose and just need a sanity check, regex is fine. The category mistake is using regex for genuinely complex parsing or using a heavy library for a one-line format check.

What This Means for Engineering Practice

If most regex bugs are actually assumption bugs, the engineering practice that prevents them is not "be better at regex." The practice is "be better at surfacing assumptions."

A few questions worth asking before shipping regex in any code path:

What input distribution does this pattern assume?

Where did the assumption come from (specification, intuition, "looked at one example")?

What changes in the upstream system would invalidate the assumption?

How would we notice if the assumption became wrong?

The pattern itself is a small part of the answer. The surrounding discipline is most of it.

Further Reading

We wrote a longer article covering the practical regex patterns we actually use in production validation layers at 137foundry.com, including the assumption notes that make them safe to deploy: Regex Code Snippets: Patterns for Common Validation and Parsing Problems. The patterns are useful. The discipline around the patterns is what makes them actually work.

For more on our approach to production data validation, the 137Foundry data integration service covers the architectural patterns. The general principle is the same one above: cheap format checks at the boundary, expensive verification later, structured logging of every rejection. The regex is one tool in the layer. The layer is the part that matters.

Useful tooling resources we point clients to: regex101.com for interactive testing and step counting, and MDN documentation for the canonical reference on JavaScript regex behavior across browsers. Both pair well with the kind of explicit-assumption practice described above.

We have built bi-directional data syncs across a range of client contexts at 137Foundry: CRM-to-billing syncs for SaaS companies, inventory syncs between warehouse management systems and e-commerce platforms, and customer data syncs between support platforms and backend databases.

The first integration always seems straightforward. It is the one running six months later that reveals what we got wrong at the start.

The conflict resolution question is a business question first

Every team we work with wants to talk about the technical implementation of conflict resolution before they have answered the business question: when both systems update the same record simultaneously, which system should win?

We have learned to force this question early. If the CRM team and the billing team disagree about which system owns customer_status, the sync layer cannot resolve that disagreement -- it will just consistently pick the wrong answer for half the stakeholders.

We now require a field authority document before any code is written: a spreadsheet mapping every shared field to an owner and a conflict rule, signed off by both sides. The PostgreSQL logical replication documentation describes the mechanics of change capture, but it does not tell you which system is authoritative for which fields. Getting the business question answered in writing first saves enormous pain downstream.

Sync loops are invisible until they are catastrophic

On every new bi-directional sync integration, we add sync loop detection before we add anything else. The detection is simple: tag changes made by the sync layer with an application identifier, filter those events out of the CDC stream before propagating them.

The reason this is the first thing we add: a sync loop does not cause obvious errors. The sync runs, data looks correct, metrics look fine. What accumulates is API call volume and change event volume -- both growing exponentially with every sync cycle. We have seen sync loops run for 18 hours before anyone noticed, generating millions of change events and exhausting the API quota for three downstream integrations.

The fix once it is running is to stop the sync, clear the change log, and restart. The fix before it runs is one conditional check in the CDC consumer. We treat this as a non-negotiable day-one requirement for every integration we ship.

Replication slot lag is a disk space problem waiting to happen

For syncs using PostgreSQL WAL-based CDC, the replication slot holds WAL segments until the consumer acknowledges them. If the consumer stops processing -- a deployment, a bug, an overloaded consumer -- the slot accumulates unreprocessed WAL segments. PostgreSQL will not clean up WAL segments referenced by an active replication slot.

In a high-write production database, an unmonitored lagging slot can fill disk in hours. We add a monitoring check on pg_replication_slots for lag_bytes within 48 hours of any WAL-based CDC setup. Alert at 1 GB lag, page at 5 GB. The two times this alert has fired in production, it prevented a disk-full outage.

The dead-letter queue needs a human interface before it has human traffic

Every bi-directional sync we ship now includes a dead-letter queue and a lightweight admin interface for reviewing DLQ entries. We build the interface before we have any DLQ traffic, not after.

The reason: the first time an unresolvable conflict lands in the DLQ, you want someone to be able to inspect it and decide whether to retry, manually resolve, or dismiss it. If the first DLQ entry arrives and there is no UI to review it, it sits in Redis accumulating siblings until someone builds the interface under pressure.

The interface does not need to be sophisticated. A table showing the record ID, the two conflicting versions, the conflict reason, and buttons to apply version A, apply version B, or dismiss is sufficient. What matters is that it exists before it is needed, not after the first incident that makes it necessary.

We documented the architecture and code patterns for all of this in How to Build a Bi-Directional Data Sync Between Business Applications. That guide reflects what we actually deploy, not the idealized version.

What Changed After We Built the Infrastructure First

After establishing this foundation -- field authority document before code, sync loop detection on day one, replication slot monitoring, DLQ with UI before first traffic -- our subsequent bi-directional sync projects have been substantially less eventful operationally.

The first incident on a new integration used to happen within 2 to 3 weeks: either a sync loop, a replication slot growing unnoticed, or a conflict resolution bug discovered by a user. Now the first incident on a new integration typically happens at month 2 or 3, is detected by monitoring rather than user report, and resolves in minutes rather than hours.

The cost of building the foundation upfront is roughly 20 to 30 percent more initial implementation time. The cost of not building it is one 4am incident call, a Sunday afternoon cleanup, and a retro where the team agrees to "add better monitoring" -- which then happens under time pressure and is therefore incomplete.

We have documented all of this in the implementation guide at 137Foundry for the teams who want to build it themselves: How to Build a Bi-Directional Data Sync Between Business Applications. For teams who want help designing or implementing the integration, the 137Foundry data integration team is available for architecture reviews and implementation work.

Why This Matters for Production Reliability

The failure modes of bi-directional sync are almost always discovered in production, not in testing. Test environments rarely replicate the exact conditions that cause clock skew conflicts -- clock synchronization on development machines is generally better than on production infrastructure. Test environments rarely replicate the specific bulk operation patterns that create consistency gaps. And test environments rarely run long enough to reveal the slow drift that accumulates when a field authority map is not updated after a schema change.

This is not an argument against testing -- it is an argument for investing in observability alongside testing. The monitoring patterns in this guide (sync lag, DLQ depth, conflict rate, record count parity) give you visibility into problems that tests will not catch before they affect users.

For teams building a bi-directional sync for the first time, the practical recommendation is: build the operational baseline (DLQ, monitoring, idempotency, loop prevention) before the first production deployment, not after the first production incident. The upfront cost is modest. The incident prevention value is significant.

For technical implementation guidance, see 137Foundry and the data integration resources. For production architecture review and implementation support, the 137Foundry services team works with teams across the integration lifecycle.

Why This Matters for Production Reliability

The failure modes of bi-directional sync are almost always discovered in production, not in testing. Test environments rarely replicate the exact conditions that cause clock skew conflicts -- clock synchronization on development machines is generally better than on production infrastructure. Test environments rarely replicate the specific bulk operation patterns that create consistency gaps. And test environments rarely run long enough to reveal the slow drift that accumulates when a field authority map is not updated after a schema change.

This is not an argument against testing -- it is an argument for investing in observability alongside testing. The monitoring patterns in this guide (sync lag, DLQ depth, conflict rate, record count parity) give you visibility into problems that tests will not catch before they affect users.

For teams building a bi-directional sync for the first time, the practical recommendation is: build the operational baseline (DLQ, monitoring, idempotency, loop prevention) before the first production deployment, not after the first production incident. The upfront cost is modest. The incident prevention value is significant.

Conflict resolution is the part of bi-directional sync that most guides skip. "Use last-write-wins" sounds reasonable until you are debugging why a customer's payment status keeps flipping between systems. Here is the process we actually follow at 137Foundry.

Step 1: Map every shared field to an owner before writing code

We start every bi-directional sync project by building a field authority matrix: a spreadsheet with every field in the shared data model, its source system, its destination system, and which system wins on conflict.

This document must be reviewed and approved by stakeholders on both sides before any code is written. The reason: conflict resolution is a policy decision, not a technical one. If the CRM team and the billing team disagree about which system owns customer_status, the sync layer cannot resolve that disagreement -- it will just consistently pick the wrong answer for half the stakeholders.

Step 2: Implement conflict resolution as configuration, not code

We implement the field authority map as a YAML configuration file:

field_authority: name: owner: crm rule: crm_wins payment_status: owner: billing rule: billing_wins last_activity: owner: shared rule: last_write_wins_with_tolerance tolerance_ms: 500 notes: owner: shared rule: dlq_on_conflict

The conflict resolution engine reads this configuration at startup. Adding a new field to the sync just requires updating the YAML -- no code deploy required.

Step 3: Build the resolver with an audit log

Every conflict resolution decision gets written to an audit log: the record ID, both versions, the resolution rule applied, and the outcome. We use SQLAlchemy for the audit rows and a Redis sorted set for DLQ entries.

The audit log serves two purposes: diagnosing incorrect conflict resolutions after the fact, and building the operational case that the rules are working correctly over time. Without the audit log, you cannot distinguish "the resolver is working as designed" from "the resolver is silently making bad decisions."

Step 4: Test with the actual conflict scenarios your system will see

We ask both system owners to describe the most common simultaneous write scenarios they expect. "CRM bulk import while billing processes end-of-month invoices" is a concrete conflict scenario. "Both systems update at the same time" is not useful.

For each described scenario, we write a test that simulates the exact timestamps and field values involved. These tests become regression tests when the configuration is updated.

Step 5: Monitor conflict rate as a first-class metric

We add conflict rate -- conflicts per 1,000 sync events -- to the primary operational dashboard for every integration we ship. A stable, low conflict rate (often well under 1 percent) is normal. A spike means something changed: a new bulk import process, a new application writing to the same fields, or a misconfigured integration that is creating duplicate writes.

The conflict rate dashboard catches problems early. An integration that starts generating 5 percent conflict rates three weeks in almost always traces to a new process added on one side of the integration without coordination with the other side.

For the full architecture and code patterns behind our bi-directional sync approach, see How to Build a Bi-Directional Data Sync Between Business Applications. To discuss the architecture for your specific integration, reach out to the 137Foundry data integration team.

What This Process Produces

The output of this five-step process is not just a conflict resolver that works. It is a conflict resolution system that is auditable, configurable without code deploys, tested against real-world scenarios, and monitored for drift.

When a user reports that "my data looks wrong" six months after a bi-directional sync is deployed, the audit log tells us exactly which record, which conflict rule was applied, and what both versions looked like at the time of resolution. That kind of auditability is the difference between a 20-minute investigation and a 3-day debugging session.

We also use the field authority document as a living specification. When either system adds a new field, the first step is to update the field authority YAML and get sign-off from both system owners before updating the sync consumer code. This disciplines the schema change process and prevents the "we didn't know the sync would be affected" conversation that comes up when fields are added without coordination.

The data integration work at 137Foundry applies this process across CRM, billing, support, and ERP integration contexts. The underlying patterns are the same regardless of which specific platforms are being connected. For the technical implementation details, see How to Build a Bi-Directional Data Sync Between Business Applications.

Why This Matters for Production Reliability

The failure modes of bi-directional sync are almost always discovered in production, not in testing. Test environments rarely replicate the exact conditions that cause clock skew conflicts -- clock synchronization on development machines is generally better than on production infrastructure. Test environments rarely replicate the specific bulk operation patterns that create consistency gaps. And test environments rarely run long enough to reveal the slow drift that accumulates when a field authority map is not updated after a schema change.

This is not an argument against testing -- it is an argument for investing in observability alongside testing. The monitoring patterns in this guide (sync lag, DLQ depth, conflict rate, record count parity) give you visibility into problems that tests will not catch before they affect users.

For teams building a bi-directional sync for the first time, the practical recommendation is: build the operational baseline (DLQ, monitoring, idempotency, loop prevention) before the first production deployment, not after the first production incident. The upfront cost is modest. The incident prevention value is significant.

For technical implementation guidance, see 137Foundry and the data integration resources. For production architecture review and implementation support, the 137Foundry services team works with teams across the integration lifecycle.

Why This Matters for Production Reliability

The failure modes of bi-directional sync are almost always discovered in production, not in testing. Test environments rarely replicate the exact conditions that cause clock skew conflicts -- clock synchronization on development machines is generally better than on production infrastructure. Test environments rarely replicate the specific bulk operation patterns that create consistency gaps. And test environments rarely run long enough to reveal the slow drift that accumulates when a field authority map is not updated after a schema change.

This is not an argument against testing -- it is an argument for investing in observability alongside testing. The monitoring patterns in this guide (sync lag, DLQ depth, conflict rate, record count parity) give you visibility into problems that tests will not catch before they affect users.

For teams building a bi-directional sync for the first time, the practical recommendation is: build the operational baseline (DLQ, monitoring, idempotency, loop prevention) before the first production deployment, not after the first production incident. The upfront cost is modest. The incident prevention value is significant.

For technical implementation guidance, see 137Foundry and the data integration resources. For production architecture review and implementation support, the 137Foundry services team works with teams across the integration lifecycle.

The underlying message is that bi-directional sync failures are rarely random -- they follow predictable patterns, and the patterns have known mitigations. A team that learns these patterns before deploying a bi-directional sync ships a more reliable integration. A team that learns them after the first production incident ships a better second integration. The choice between the two paths is mostly a function of whether the team invests in understanding the failure modes at design time.

We get asked regularly about our tooling stack. This is an honest answer: what we use, why we use it, and where we have moved away from tools we previously relied on. This is not a comprehensive list of everything available -- it is a specific list of what we reach for on real projects.

Native CSS Custom Properties for Theming

We use native CSS custom properties as the foundation for design tokens and theming on every project. We stopped using Sass variables as our primary theming mechanism because CSS custom properties solve the same problem with a significant advantage: they are live values that JavaScript can read and modify at runtime.

This matters specifically for dark mode. When theme switching depends on a JavaScript-toggled attribute, the CSS custom property layer handles the entire UI update with no JavaScript per element. Sass variables cannot do this -- they are resolved at compile time and baked into static values. For any application with user-controlled theming, runtime CSS variables are strictly more capable.

We still use PostCSS for certain things (nesting in older projects, autoprefixer for some target browser profiles), but the primary color and spacing token system is always CSS custom properties rather than preprocessor variables. The token layer also gives us a single place to audit design decisions -- every color and spacing value in the application is visible in one file rather than scattered across component stylesheets.

Vanilla JS for Theme Persistence

We do not use a library for localStorage-based theme persistence. The implementation is small enough that adding a dependency would increase complexity rather than reduce it:

function applyTheme(theme) { document.documentElement.setAttribute('data-theme', theme); } function getPreferredTheme() { return localStorage.getItem('theme') || (window.matchMedia('(prefers-color-scheme: dark)').matches ? 'dark' : 'light'); } applyTheme(getPreferredTheme());

This runs inline in the <head> before stylesheets paint, which prevents the flash of wrong theme. That placement is intentional and important -- the script blocks rendering briefly, but the blocking cost is negligible for a script this small, and it eliminates the visible theme flash that a deferred script produces. This is one of the few cases where a blocking script in the head is the correct choice.

The full implementation and context for this pattern is in our team's write-up: How to Add Dark Mode to a Web Application.

Stylelint for CSS Quality

Stylelint catches CSS issues that code review misses: unknown properties, selector specificity problems, use of deprecated values, and violations of team-defined conventions like requiring var() instead of hard-coded hex values.

We run Stylelint as part of CI on every project with a meaningful CSS surface area. The rule we use most is a custom rule that flags hard-coded hex values in component files -- any color value that is not accessed through a CSS custom property is a gap in the design token system, and Stylelint catches it before merge.

For teams starting with Stylelint, the standard config is a reasonable starting point. We layer custom rules on top based on the project's architecture decisions. The payoff is consistent: issues that previously surfaced in production or code review now surface at the lint stage, where they are cheapest to fix.

The W3C Contrast Checker for Accessibility Validation

We validate dark mode color palettes against WCAG contrast requirements before shipping. The minimum ratio for normal text is 4.5:1 against its background. This requirement does not relax in dark mode -- dark text on dark background failures are as common as they are in light mode, and they are easy to miss visually.

The most common failure we see in dark mode audits is secondary text -- labels, captions, placeholder text -- that falls below 4.5:1 because the muted text color was designed for a light background and was not recalibrated for dark. The fix is always to lighten the secondary text token for dark mode. We also avoid pure white on pure black combinations. Very high contrast between foreground text and dark background can cause halation on OLED displays -- a glow effect where bright text bleeds slightly into the surrounding dark area. A slightly off-white text value against a deep navy background reads more cleanly than maximum contrast extremes.

caniuse.com for Browser Support Decisions

caniuse.com is the reference we use for every browser support question. The questions we check most:

CSS custom properties -- supported across the board

prefers-color-scheme -- full modern browser support

The distinction between "supported in all modern browsers" and "supported in all browsers our specific client targets" matters. Enterprise applications targeting specific browser profiles often have different constraints than consumer web applications. caniuse gives us the data to make that judgment per project rather than applying a blanket rule.

What We Have Moved Away From

CSS-in-JS libraries for theming. We used CSS-in-JS for theming on early React projects when native CSS custom properties were less established. We have moved off it for new projects because the native CSS solution is faster (no runtime style injection), simpler (no library dependency), and works outside React components. The server-side rendering complexity of CSS-in-JS libraries was also a recurring source of friction that we no longer need to manage.

Global class-based dark mode toggles. The .dark-mode-enabled class on <body> approach requires overriding every component's styles in a matching .dark-mode-enabled .component selector block. This doubles the specificity of every dark mode rule and creates a maintenance burden proportional to the number of components. The data-theme attribute on <html> with CSS custom property overrides is cleaner and requires no per-component override rules.

Preprocessor variables as the primary theming mechanism. Already covered above -- compile-time variables cannot do what runtime custom properties do for theming. We keep PostCSS in the build pipeline for autoprefixer and occasionally for nesting in projects where native CSS nesting is not yet the target, but the token layer is always native custom properties.

How Tool Decisions Compound Over Time

Tooling decisions compound over the life of a project in both directions. A library that reduces friction in week one but requires major version migrations every 18 months creates a recurring cost that accumulates over a project's lifespan. A slightly more verbose native approach that is stable and maintainable without dependency upgrades often turns out to be cheaper over the same period when you account for the full cost of maintaining third-party dependencies.

Our general heuristic: if the native platform solves a problem adequately, use the native solution. Add a dependency only when there is a specific, named capability gap that the platform cannot fill. This is why we use vanilla JavaScript for theme persistence rather than a theming library, and why we use CSS custom properties rather than a CSS-in-JS runtime for token management. The resulting codebases are smaller, faster to onboard new developers onto, and cheaper to maintain over multi-year timelines.

"Tooling decisions compound over the life of a project. The tools we choose in week one are still there in year two, either serving us well or creating friction in every sprint. We are conservative about adding dependencies and aggressive about removing them when the platform catches up." -- Dennis Traina, founder of 137Foundry (view services)

At 137Foundry, we build and maintain web applications across a range of industries and complexity levels. After doing this for long enough, we have developed strong opinions about CSS architecture -- specifically about what works at scale and what creates maintenance problems that compound over time.

This is not a theoretical take. These are the decisions we make repeatedly on production codebases because they have consistently produced better outcomes than the alternatives.

Start With a Token System, Not Component Styles

The single most impactful structural decision in any web application's CSS is establishing a design token layer before writing any component styles. A token system is a set of named CSS custom properties that represent every design decision in the application: colors, spacing, border radii, font sizes, shadow values.

Without this layer, teams write component styles with hard-coded values. The button is #3b82f6. The card has padding: 16px. The heading is 1.125rem. These values are correct when they are written. Over time, the design evolves, requirements change, and someone needs to find every instance of #3b82f6 across two hundred CSS files to update the brand color. That search-and-replace exercise is the direct cost of skipping the abstraction layer.

With a token system, the brand color is var(--color-accent). Changing the brand color is a one-line edit. The same applies to dark mode: a semantic token set that maps cleanly to a dark variant makes theme switching a configuration change rather than a CSS surgery project.

We have shipped this architecture on every new project for the past three years. We have also retrofitted it onto existing codebases that did not have it, and the retrofit is always more expensive than starting with it. Start with the token layer.

The Cascade Is a Feature, Not a Bug

We see teams fight the CSS cascade constantly -- adding !important declarations, using inline styles to override stylesheet values, increasing specificity with deeply nested selectors. This is almost always a symptom of not trusting the cascade rather than a genuine limitation of it.

The cascade works correctly when selectors are written at the right specificity level and values are not overridden at the wrong layer. A component's base styles belong at low specificity. State variations (hover, focus, active) and modifier classes belong at the same specificity level as the base or slightly higher. The !important annotation should be genuinely rare in a well-architected stylesheet.

When we audit a codebase that has cascading issues, the most common cause is that component styles were written without considering the document structure they would appear in. A component written to work in isolation often has specificity problems when it is placed inside a more complex layout. The fix is usually not to add more specificity -- it is to remove the accidental specificity from the base component style.

Theming Is Infrastructure, Not a Feature

We treat dark mode and theme switching as infrastructure decisions rather than feature work. This distinction matters for timing: infrastructure decisions need to be made before component development begins, not added afterward.

When dark mode is retrofitted onto an existing stylesheet, the work is proportional to how many hard-coded color values were written before the retrofit. We have seen this process take weeks on mature codebases. When dark mode is part of the initial architecture, it costs almost nothing to implement -- the token layer handles it.

Our dark mode implementation guide, How to Add Dark Mode to a Web Application, reflects this infrastructure mindset. The article covers the CSS custom property architecture, the system preference detection, and the JavaScript toggle pattern that we use as a starting point on new projects.

We are not advocating for building dark mode before you know users want it. We are advocating for building the CSS architecture in a way that makes adding dark mode a configuration task rather than a refactoring project. The cost difference is significant.

Component-Level CSS Should Know Nothing About the Page Layout

A card component should not have margin or positioning rules. A button should not have margin-top or position: absolute. These layout properties belong in the parent element or in a composition layer that controls how components are arranged in the page.

When components include their own layout and spacing rules, they become fragile in new contexts. The card that assumes it is always followed by a sibling assumes a context that does not always hold. The button with a margin-top: 24px hardcoded into its stylesheet breaks every layout where that spacing is wrong.

We call this the layout-vs-component boundary, and we enforce it explicitly in code review. Components define their internal appearance -- colors, typography, borders, internal padding. Layout parents define how components are positioned relative to each other. Composition layers (using CSS Grid or Flexbox in a container element) define how multiple components are arranged in a larger structure. Each layer knows about its own concerns and nothing about the others.

This boundary is what makes components genuinely reusable. A card that has no layout assumptions about its context works correctly whether it is placed in a full-width content area, a narrow sidebar, a modal, or a grid of three.

Use New CSS Features When They Solve Real Problems

We adopted CSS custom properties for design tokens before they were universally recommended because they solved a genuine problem. We adopted container queries when they solved the "component that breaks in a narrow sidebar" problem that viewport media queries cannot address. We adopted native CSS nesting when it allowed us to remove a preprocessor dependency from projects that did not need other preprocessor features.

The pattern is consistent: adopt when the feature solves a real problem we were previously working around, not because it is new. We are currently evaluating prefers-color-scheme as the baseline for dark mode in client projects that previously used JavaScript-only approaches, because the media query approach eliminates the flash-of-wrong-theme issue at zero additional JavaScript cost.

CSS is improving fast enough that keeping a working knowledge of what the platform can do natively matters more than it did five years ago. The answer to "should we use a CSS-in-JS library for theming?" is now "almost certainly not" for most projects, because the CSS platform handles theming natively and with better performance characteristics.

"Our CSS architecture stance is simple: use the platform where it is capable, add tooling only for specific gaps, and design for the three-year maintenance window rather than the two-week shipping deadline. The teams that have the cleanest codebases two years later are the ones that resisted the temptation to add abstraction layers before understanding the problem they were solving." -- Dennis Traina, founder of 137Foundry (view services)

What This Means for Projects We Take On

When we start a new front-end engagement, the CSS architecture conversation happens before the first component is written. We establish the token layer, define the component-vs-layout boundary, and set up the theming infrastructure whether or not dark mode is a stated requirement. These decisions cost almost nothing to make at the beginning and a lot to retrofit later.

If you are starting a new web application or reaching a point where your existing CSS is creating friction, 137Foundry's web development team has done this front-end architecture work across enough projects to know where the common mistakes are and how to avoid them from the start. The about page has context on our approach if you want to understand how we work before reaching out.

We have built search into somewhere north of fifty client web applications at this point. Product catalogs, admin tools, documentation search, command palette interfaces, SaaS dashboards with global search, CMS record lookups. The backends vary enormously: Algolia, Typesense, PostgreSQL full-text, Fuse.js for client-side, Elasticsearch for a few of the larger projects, and more than a few SQL LIKE queries that were supposed to be replaced and have not been.

One thing is consistent across every single one of these: the first thing we implement, before we write a single line of backend code, is debounce on the search input.

Not because it is the most exciting part of building search. Because it is the part that fails silently if you skip it, and the failure costs more to diagnose after launch than it would have cost to build correctly in the first place.

The Problem That Hides Until Traffic Increases

Here is what happens without debounce. A user types "wireless keyboard" into the search box. That is sixteen characters. The search input fires sixteen separate API requests: one when they type "w", one for "wi", one for "wir", and so on through the full string.

At low traffic and with fast server responses, this looks fine in the browser. Results update smoothly as the user types. The backend team does not see a problem in the logs because the queries are fast and the traffic is low enough that sixteen requests per search interaction is not obviously visible as unusual.

Then traffic increases. A sale event, a feature launch, a product getting featured somewhere. Database query times stretch from 50ms to 300ms. Now the requests are in-flight simultaneously rather than completing before the next one starts. Whichever response arrives last overwrites the UI, regardless of whether it matches what the user actually typed.

A user who typed "wireless keyboard" might see the results for "wireles" if that query happened to be the last response to arrive. They typed nine more characters and the results got worse. This is confusing enough that users assume the search is broken -- because from their perspective, it is.

We have diagnosed this bug in client projects that had been live for months. The conversations are always the same: "search worked fine in testing." It did. The load was too low for the race condition to manifest. The only thing that changed was traffic, not the code.

Debounce eliminates this category of bug before it can happen.

What Debounce Actually Does

Debounce is a function wrapper that delays execution until after a pause. The classic use case is the search input: instead of firing a search request on every keystroke, debounce waits until the user stops typing -- for 250-350 milliseconds, typically -- and then fires exactly one request.

The user who types "wireless keyboard" still sees their results as intended. The search fires when they finish the word and pause. But instead of sixteen requests, the server receives one.

The implementation in vanilla JavaScript is small:

function debounce(fn, delay) { let timer; return (...args) => { clearTimeout(timer); timer = setTimeout(() => fn(...args), delay); }; } const handleSearch = debounce((query) => { fetchResults(query); }, 300); searchInput.addEventListener('input', (e) => { handleSearch(e.target.value); });

This is the entire pattern. No dependencies. No framework required. It composes with any search function: an Algolia API call, a Typesense request, a fetch to a Rails search endpoint, a Fuse.js local search. The function it wraps does not change. Only the timing changes.

For projects using React, we typically use the useMemo or useCallback hooks to memoize the debounced function so it is not recreated on every render. The underlying logic is identical.

Why 300ms and Not 150ms or 500ms

The 300ms default is calibrated to human typing speed, not picked arbitrarily. Average adult typing speed is 40-50 words per minute, which translates to roughly 200-250ms between keystrokes for normal words. A 300ms debounce fires when the user pauses -- between words, at the end of a term, when they hesitate mid-search. That pause is the signal that they have finished typing a meaningful unit and are ready for results.

At 150ms, debounce barely filters anything for users who type quickly. Two fast keystrokes land within 150ms of each other frequently at normal typing speed. The request volume reduction is minimal.

At 500ms, users who type quickly and then stop notice the delay. They have finished typing "keyboard" and wait half a second for anything to happen. At modern network speeds with a well-indexed database, 300ms is enough buffer to eliminate unnecessary requests without introducing visible latency.

The one exception is client-side search -- Fuse.js, Lunr.js, FlexSearch running entirely in the browser. When the search itself is synchronous and runs in under 2ms, there is no network to wait for. The debounce delay can drop to 150-200ms, because the goal is only preventing unnecessary DOM updates and the search result does not carry any latency penalty.

The Race Condition That Survives Debounce

Debounce reduces request volume. It does not eliminate all race conditions.

Even with a 300ms debounce, two requests can be in flight simultaneously: the user makes a query, pauses, makes a different query. Both are debounced, both fire. If the first query takes longer to complete than the second, its response arrives after the second query's results have already rendered. The stale results overwrite the current ones.

This is where AbortController comes in. Before each new request, cancel the previous one:

let controller; const handleSearch = debounce(async (query) => { if (controller) controller.abort(); controller = new AbortController(); try { const results = await fetchResults(query, { signal: controller.signal }); renderResults(results); } catch (err) { if (err.name === 'AbortError') return; throw err; } }, 300);

AbortErrors must be caught and treated as non-failures. They are intentional. The request was canceled because a newer query superseded it. That is correct behavior.

We use both patterns on every server-based search we build at 137Foundry: debounce to reduce request volume, AbortController to guarantee that stale responses never overwrite current results. They solve different problems, both are required.

The Backend Choice Comes Third

After debounce and AbortController, the third consideration is the loading state. After that, the backend architecture. We sequence it this way deliberately.

The frontend patterns -- debounce, race condition handling, loading feedback, empty state messaging -- are the same regardless of whether the search backend is Algolia or a PostgreSQL LIKE query. They determine whether the search feels right to users. A perfectly tuned search index behind a broken frontend UX does not feel like good search. A mediocre LIKE query behind a well-debounced, well-instrumented input feels surprisingly good.

When we scope search work with new clients, we scope the frontend work and the backend work separately. The frontend work is faster and its improvements are immediately visible. The backend work is slower and often requires real user search data before relevance tuning can be meaningfully done. We can ship a fast, well-behaved search input while the backend indexing strategy is still being finalized, and the users who encounter it first will not know the backend is not perfect yet.

One of the quietest bugs in search-as-you-type implementations is the stale response problem. A user types a query, the search fires, the user keeps typing before the first response arrives, a second search fires -- and then the first response arrives after the second. The UI renders results for the wrong query. The user sees results that do not match what they typed.

This bug is invisible in normal development conditions because the server is fast and local network latency is low. It surfaces in production under real-world latency, database load, or slower connections. Users report that search is "showing wrong results" or "flickering." The bug is real, it is predictable, and it is completely preventable.

At 137Foundry, we use AbortController to cancel in-flight requests before every new search fires. Here is how we implement it and what to watch out for.

Step 1: Understand the Race Condition

Before reaching for AbortController, it helps to be precise about what the race condition is.

Search-as-you-type fires a request as the user types. If the user types fast, or if network latency is variable, two requests can be in flight simultaneously. The responses arrive in an unpredictable order -- whichever backend query happens to complete first wins. If the first query (for "keyboa") completes after the second query (for "keyboard"), the UI ends up showing results for "keyboa" even though the user finished typing "keyboard."

This is the classic stale closure problem applied to asynchronous network requests. The solution is to cancel any in-flight request before starting a new one, so only one request is ever active at a time.

Step 2: Create the AbortController

The Web API for canceling fetch requests is AbortController. Each fetch call can be passed an AbortSignal. When the controller is aborted, any fetch using that signal is canceled immediately.

The setup is two lines:

const controller = new AbortController(); const { signal } = controller; const response = await fetch('/api/search?q=query', { signal });

Calling controller.abort() cancels the fetch. The promise rejects with a DOMException named AbortError.

Step 3: Wire It to the Search Input

In practice, we keep the controller in a module-level variable so we can cancel it from the next search call. The pattern looks like this:

let activeController = null; async function fetchResults(query) { if (activeController) { activeController.abort(); } activeController = new AbortController(); const response = await fetch(`/api/search?q=${encodeURIComponent(query)}`, { signal: activeController.signal, }); const data = await response.json(); return data.results; }

Every call to fetchResults cancels the previous request before starting a new one. Only the most recent request ever completes.

Step 4: Catch AbortErrors Without Treating Them as Failures

AbortErrors are intentional. Aborting a request is not an error in the application sense -- it is correct behavior. If you let AbortErrors propagate to your error handler, you will see false error logs, incorrect "search failed" states in the UI, and potentially broken analytics.

Catch AbortErrors separately and return silently:

async function handleSearch(query) { try { const results = await fetchResults(query); renderResults(results); } catch (err) { if (err.name === 'AbortError') return; showErrorState(); console.error('Search failed:', err); } }

The AbortError branch just returns. No UI update, no error log, no retry. The canceled request never had its results used, so there is nothing to clean up.

Step 5: Combine With Debounce

AbortController and debounce are complementary. They solve different problems.

Debounce reduces the number of requests fired. With a 300ms debounce, a user typing "keyboard" fires one request instead of eight. Debounce eliminates most of the race condition risk by simply not firing intermediate requests.

AbortController handles the race condition that survives debounce. Even with debounce, two requests can be in flight -- the debounce delay passes twice if the user types, pauses, then types again. AbortController ensures the first one is canceled when the second fires.

Together:

function debounce(fn, delay) { let timer; return (...args) => { clearTimeout(timer); timer = setTimeout(() => fn(...args), delay); }; } let activeController = null; async function search(query) { if (activeController) activeController.abort(); activeController = new AbortController(); try { const res = await fetch(`/api/search?q=${encodeURIComponent(query)}`, { signal: activeController.signal, }); const data = await res.json(); renderResults(data.results); } catch (err) { if (err.name === 'AbortError') return; showError(); } } const debouncedSearch = debounce(search, 300); searchInput.addEventListener('input', (e) => { const query = e.target.value.trim(); if (query.length < 2) { clearResults(); return; } debouncedSearch(query); });

This is the pattern we use for all server-side search implementations at 137Foundry. It is small, dependency-free, and provably correct for the race condition case.

Step 6: Test on a Slow Network

AbortController bugs and race conditions are invisible on fast local networks. To test that your implementation handles them correctly, use Chrome DevTools to simulate a slow connection.

Open DevTools, go to the Network tab, and change the throttling dropdown from "No throttling" to "Slow 3G" or a custom profile with 300-500ms latency. Then type a search query quickly. You will see the canceled requests appear in the network log with a red canceled status. Only the final request should resolve successfully and update the UI.

If the wrong results appear, or if the UI updates multiple times with different results as responses arrive, AbortController is not correctly wired up.

This test takes two minutes and makes the bug visible immediately. We run it on every search implementation before shipping.

When AbortController Is Not Needed

AbortController matters for server-based search. It is not needed for client-side search using Fuse.js, Lunr.js, or FlexSearch.

Client-side search runs synchronously in the browser. fuse.search(query) returns results immediately -- there is no network request, no in-flight state, and no possibility of a race condition. The debounce is still useful to avoid unnecessary DOM updates during fast typing, but AbortController adds nothing. If the dataset fits comfortably in browser memory, client-side search is often the right architecture: zero latency, no server load, and no concurrent request management needed.

If you switch from client-side search to a server-backed implementation later (because the dataset grew or you needed better relevance), adding AbortController at that point is the right time. It is a small addition and easy to retrofit.

What This Looks Like in a React Hook

For React projects, we encapsulate this in a custom hook:

function useSearch(fetchFn, delay = 300) { const [results, setResults] = useState([]); const controllerRef = useRef(null); const search = useCallback( debounce(async (query) => { if (controllerRef.current) controllerRef.current.abort(); controllerRef.current = new AbortController(); try { const data = await fetchFn(query, controllerRef.current.signal); setResults(data); } catch (err) { if (err.name === 'AbortError') return; setResults([]); } }, delay), [fetchFn, delay] ); return { results, search }; }

The hook manages the controller ref, handles cleanup, and exposes a search function that is already debounced and abort-safe. Components receive results and a search trigger without touching any of the concurrency logic.

Long-running scripts fail differently than web requests. A web request fails in milliseconds and the next request starts fresh. A script that processes 50,000 records fails hours in, and the question is not just what went wrong but how much work was lost and how much needs to be redone.

At 137Foundry, we work on data pipelines and automation scripts that process large batches of records. The patterns here come from building those systems and watching what breaks in production.

The Problem With Failing Fast in Long-Running Jobs

For short operations, fail fast is the correct default. If an API call fails at the start of a web request, raise immediately and let the framework return an error response. Clean and predictable.

For long-running batch jobs, fail-fast means losing all progress on a partially-completed batch. If you are processing 10,000 records and record 8,743 causes an unhandled exception, you have lost the results for records 1 through 8,742. You also have no idea which records failed and which succeeded.

The alternative is record-level error isolation: catch exceptions at the per-record level, log them with enough context to identify the problem record, and continue processing the remaining records.

results = {"success": [], "failed": []} for record in batch: try: output = transform(record) results["success"].append(output) except Exception: log.exception("Transform failed on record %s", record.id) results["failed"].append(record.id) log.info("Batch complete: %d succeeded, %d failed", len(results["success"]), len(results["failed"]))

This is a deliberate trade-off. You are choosing to continue processing over failing loudly. The correctness condition is that each record's failure or success is fully captured before moving to the next one.

Distinguishing Retryable from Fatal Failures

Not all failures are the same. A database timeout at record 4,000 might succeed on retry. A record with a malformed date field will fail every time. Treating both identically -- either always retry or always skip -- is wrong.

A custom exception hierarchy makes the distinction explicit:

class RetryableError(Exception): """The operation failed but may succeed on retry.""" pass class FatalError(Exception): """The operation will not succeed on retry. Escalate.""" pass

The service layer raises the appropriate type. The batch processor catches them differently:

for record in batch: for attempt in range(MAX_RETRIES): try: output = process(record) results["success"].append(output) break except RetryableError as err: if attempt == MAX_RETRIES - 1: log.error("Max retries reached for %s: %s", record.id, err) results["failed"].append(record.id) else: time.sleep(2 ** attempt) # Exponential backoff except FatalError: log.exception("Fatal failure on record %s, skipping", record.id) results["failed"].append(record.id) break

The Python documentation covers the exception hierarchy and how custom exceptions should be defined. The Python Enhancement Proposals on peps.python.org -- specifically PEP 3134 -- describe exception chaining, which is how you preserve the original cause when wrapping exceptions in your service layer.

Checkpointing to Avoid Reprocessing Completed Work

For very long jobs, error isolation is not enough. If the entire process crashes after 8 hours of work, you still lose all progress. Checkpointing lets you resume from the last successful point rather than starting over.

The simplest checkpoint is a file or database table that records which records have been processed:

import json CHECKPOINT_FILE = "progress.json" def load_checkpoint(): try: with open(CHECKPOINT_FILE) as f: return set(json.load(f)["completed"]) except FileNotFoundError: return set() def save_checkpoint(completed_ids): with open(CHECKPOINT_FILE, "w") as f: json.dump({"completed": list(completed_ids)}, f) completed = load_checkpoint() for record in batch: if record.id in completed: continue # Already processed on a previous run try: output = process(record) completed.add(record.id) save_checkpoint(completed) except Exception: log.exception("Failed on record %s", record.id)

The checkpoint write frequency is a trade-off: write after every record for maximum resume granularity, write less often for better throughput. For most batch jobs, writing every 100 records is a reasonable balance.

Structured Failure Reports

After a batch completes, the failure records are only useful if they are in a format that makes remediation possible. Logging exception messages is a minimum. Persisting the failed record IDs to a file or table is better, because it enables re-running just the failed subset without modifying the main script.

if results["failed"]: with open("failed_records.txt", "w") as f: for record_id in results["failed"]: f.write(f"{record_id}\n") log.warning("%d records failed. See failed_records.txt", len(results["failed"]))

For re-runs, the batch loading code accepts an optional filter:

def load_batch(rerun_ids=None): if rerun_ids: return [r for r in all_records if r.id in rerun_ids] return all_records

This pattern -- isolate failures, record them, enable targeted reruns -- is one of the consistent elements in the data pipeline work we do at 137Foundry. The same structure applies whether the batch is 1,000 records or 10 million.

When Record-Level Isolation Is Not Appropriate

Record-level isolation is not always the right pattern. It is specifically appropriate when:

Each record can be processed independently, with no side effects between records.

A failure on one record leaves the system in a clean state for the next record.

Partial completion is acceptable and resumable.

It is not appropriate when records have dependencies. If record B references the output of record A, and record A failed, record B may appear to succeed but produce corrupt output based on a missing or stale dependency. In these cases, the dependency graph needs explicit representation in the batch design, and failures need to propagate to dependent records.

Similarly, record-level isolation does not protect against failures in setup or teardown code outside the loop. If the database connection drops mid-batch and you are catching DatabaseConnectionError inside the loop, every remaining record will fail with the same error. The isolation masks what is actually a systemic failure. A connection health check before the loop and a circuit breaker pattern to abort when failure rate spikes prevents the isolation from hiding a systemic outage.

Production Monitoring Considerations

Error isolation means individual failures do not crash the job, but it also means failures can accumulate silently. A batch that completes with 200 failed records looks like a success from the job runner's perspective if you are only checking the exit code.

Three additions address this:

Exit with a non-zero code if failure rate exceeds a threshold. A 2% failure rate might be acceptable; a 40% failure rate indicates a systemic problem.

failure_rate = len(results["failed"]) / len(batch) if failure_rate > 0.05: log.error("Failure rate %.1f%% exceeds threshold", failure_rate * 100) sys.exit(1)

Send an alert when failure counts exceed a threshold. The Sentry error monitoring platform integrates with Python logging to aggregate error counts and alert when error rates spike above configured thresholds.

Log a structured summary at job end. A final log entry with total records, success count, failure count, and elapsed time makes it trivial to build dashboards and detect regressions across job runs without parsing individual record-level entries.

log.info( "Batch %s complete: total=%d success=%d failed=%d elapsed=%.1fs", batch.id, len(batch), len(results["success"]), len(results["failed"]), time.time() - start_time )

The full reference for Python exception handling patterns -- including exception chaining, service boundary design, and testing failure paths with pytest -- is in the Python Error Handling article on the 137Foundry blog.

Error handling is one of those things that looks like plumbing -- boring and invisible until it breaks. At 137Foundry, we have a consistent approach to exception handling that we apply to every Python data pipeline and web application we build. This post describes what that approach looks like and why we settled on it.

We Start With a Custom Exception Hierarchy

Every project that has a service layer gets a custom exception hierarchy in the first commit. Not at refactor time, not after the first incident -- in the initial setup.

The hierarchy follows a predictable shape:

class ProjectError(Exception): """Base for all application-layer failures in this project.""" pass class RetryableError(ProjectError): """The operation failed but may succeed on retry.""" pass class PermanentError(ProjectError): """The operation will not succeed on retry. Escalate.""" pass class ValidationError(PermanentError): """Input data failed validation rules.""" def __init__(self, message, field=None, value=None): super().__init__(message) self.field = field self.value = value

The motivation for starting here is that it forces a conversation early about which failure modes exist and which require different responses. A RetryableError and a ValidationError need completely different handling. If you do not make the distinction in the type system, you make it in scattered if/else logic that is harder to maintain and easier to get wrong.

We Use log.exception() at Service Boundaries

Every service boundary in our projects -- API handlers, background job entry points, CLI commands -- has a consistent logging pattern for caught exceptions:

try: result = service.process(request_data) except ProjectError: log.exception("Service operation failed for request %s", request.id) raise

log.exception() captures the full traceback automatically. We do not use log.error("Failed: %s", err) at boundaries because the error message alone rarely contains enough information to diagnose the problem. The traceback tells us the exact call chain and the exact line. Without it, an incident investigation starts from scratch.

We re-raise after logging so that callers at higher levels get the exception and can respond to it with their own logic -- returning an HTTP 500, scheduling a retry, or escalating to an alert.

We Preserve Tracebacks With Chaining

When we catch a low-level exception and raise a domain exception, we always use raise X from Y:

try: raw = db.query(sql) except DatabaseConnectionError as err: raise RetryableError("Database temporarily unavailable") from err

The from err clause preserves the original DatabaseConnectionError in the traceback. When a developer reads the log, they see both exceptions and can trace the failure back to its source. Without chaining, the original error disappears and the traceback shows only the RetryableError, which is less useful for diagnosis.

The Python documentation at docs.python.org covers exception chaining, and PEP 3134 on peps.python.org explains the design intent behind __cause__ and __context__.

We Write Tests for Failure Paths First

In data projects, failure path tests are often the ones that prevent the most expensive bugs. A test that verifies the service raises RetryableError on database timeout ensures that the retry logic in the background job processor actually fires on timeout. Without that test, a change that accidentally swallows the timeout and returns None instead would not be caught until timeout caused silent data loss in production.

Our standard pattern for failure path tests uses pytest and mocker:

def test_raises_retryable_on_db_timeout(mocker): mocker.patch("project.db.query", side_effect=TimeoutError) with pytest.raises(RetryableError): service.fetch_records(batch_id) def test_validation_error_carries_field_name(): with pytest.raises(ValidationError) as exc_info: service.process_record({"amount": "not_a_number"}) assert exc_info.value.field == "amount"

The pytest documentation at docs.pytest.org covers pytest.raises, the match parameter for asserting on exception messages, and mocker.patch for simulating failures.

How We Handle Failures in Background Jobs

Background job processors have different error handling requirements than API endpoints. A web request must respond within a timeout; a job can fail and be retried. But the failure must be recorded and the retry must actually happen -- silent failures are just as dangerous in a job queue as in an API.

Our background job processors follow a consistent structure:

The job function catches RetryableError and marks the job for retry with appropriate backoff.

The job function catches PermanentError and marks the job as permanently failed, triggering an alert.

Any unexpected exception propagates to the job runner's outer handler, which marks it as failed and sends an immediate alert.

def process_job(job): try: result = service.process(job.data) job.mark_complete(result) except RetryableError as err: delay = min(2 ** job.attempt_count, 3600) job.schedule_retry(delay=delay) log.warning("Retryable failure on job %s, retry in %ds: %s", job.id, delay, err) except PermanentError: log.exception("Permanent failure on job %s", job.id) job.mark_failed() alerts.send(f"Job {job.id} permanently failed")

The except PermanentError handler does not catch RetryableError because it is not a subclass of PermanentError. Adding a new failure mode to the hierarchy does not require changing this handler unless the new type needs a distinct response.

The Value of Consistent Patterns Across a Codebase

Exception handling inconsistency compounds as a codebase grows. When some services raise ValueError, others raise Exception("message"), some return None and log an error, and some return (None, error_message) tuples, adding a new feature requires understanding the error contract for every function you call, because none of them are the same.

The consistent approach described here is not complex. Each piece is a few lines. The value accumulates because every developer working on the codebase -- including developers joining the project later -- can read any service and immediately understand its failure modes. The exception type tells them whether to retry, escalate, or discard. The logging tells them where to look. The test tells them the retry logic actually fires.

We have found this consistency to be particularly valuable during handoffs. When we finish a project and transfer it to a client's internal team, the new developers spend far less time understanding the error handling because it follows the same pattern everywhere. That is the return on the investment in establishing the convention at project start.

We Apply the Same Pattern to All Project Types

The same exception hierarchy and logging pattern applies whether we are building a REST API, a batch data pipeline, or a scheduled automation script. The boundary changes -- the API handler is the boundary in a web app, the job entry function is the boundary in a pipeline -- but the design is the same.

This consistency has a practical benefit: when we hand a project off to a client's internal team, the error handling is predictable throughout. Every exception type communicates a specific failure mode. Every boundary logs with log.exception(). Every service-to-service exception is chained. Developers who are new to the codebase can read the exception type and know whether to retry, escalate, or discard.

The full reference collection of Python error handling code snippets -- covering try-except-else-finally, exception chaining, contextlib.suppress, and testing patterns -- is on the 137Foundry blog. It is the reference we send to developers starting on a new project.

We've built and operated event-driven data pipelines at 137Foundry across a range of client contexts: SaaS platforms that need to sync records as they're created, e-commerce backends that process order events in real time, and analytics pipelines that need to reflect data changes in near real-time rather than overnight.

Most of the hard-won lessons aren't in the initial implementation -- they're in what breaks after the first month of production load. Here's the honest version.

The thing that bites you first: consumers crashing silently

The first event-driven pipeline we deployed had a consumer that occasionally died without a trace. No error in the logs. No alert. Just a growing queue depth we noticed only because we happened to look at the Redis dashboard for an unrelated reason.

The root cause: an unhandled exception in a code path that was only reachable with a specific message format we hadn't tested. The consumer process exited, the supervisor restarted it, but the messages it was processing at the time were gone -- Redis had already popped them from the queue before the crash.

We added confirmed delivery (BRPOPLPUSH processing list pattern for Redis consumers), explicit exception handling with structured logging, and queue depth monitoring that alerts when depth exceeds a threshold for more than 5 minutes. None of this was in the initial implementation because none of it seemed necessary until it was.

The lesson: event-driven pipelines fail differently than scheduled jobs. A scheduled job that crashes is visible in the cron log immediately. A consumer that crashes is invisible until you notice the queue is growing.

The dead-letter queue gap

Our second major production incident came from a pipeline that was processing webhook events for a client. One category of event had a schema change that wasn't communicated to us -- the webhook payload added a required field that our schema validation code didn't expect, causing every event of that type to fail. Celery retried three times, hit max retries, marked the tasks as failed, and moved on.

We discovered this 48 hours later when the client noticed that three days of events weren't reflected in their data. The fix was straightforward once we knew about it, but the 48-hour gap was a problem.

The change we made: all Celery tasks now route failure entries to a DLQ Redis list on on_failure. We have a lightweight monitoring script that checks DLQ depth every 5 minutes and pages us if it exceeds 10 entries. We also have a requeue script that inspects DLQ entries and re-dispatches them after we've fixed the underlying issue.

The DLQ setup takes about two hours to implement properly. The operational visibility it gives you is worth significantly more than that.

Message volume assumptions that don't survive production

In development, we test with small message volumes: 10 events, 100 events. Production systems don't send 10 events. They send 10,000 events in a burst when something triggers a batch operation -- an import, a migration, a scheduled sync.

The first time we hit a burst like this, our single-consumer pipeline fell 6 hours behind real time before we noticed. The queue had backed up to 50,000 messages. Adding consumers helped, but the burst had already happened and the catch-up process was slow.

We now size consumer pools based on peak volume, not average volume. If the maximum realistic burst is 10,000 messages in 10 minutes, we configure enough consumers to handle that rate. This means running more workers than necessary most of the time, which costs money, but it's cheaper than a 6-hour lag when a client is watching.

We've also added circuit breakers on downstream dependencies (databases, APIs) so that a slow downstream doesn't cause the queue to back up indefinitely -- it causes the consumers to pause and retry, which is a different (and more manageable) failure mode.

The idempotency debt we paid later

Early in our event-driven pipeline development, we didn't implement idempotent processing because our broker (Celery with Redis) was delivering messages exactly once in practice. No duplicates, no issue.

Then we migrated one pipeline to RabbitMQ for durability. RabbitMQ's at-least-once delivery guarantee is not a theoretical concern -- it's actual behavior. Consumer acknowledgment after processing means that if the consumer crashes between processing and acknowledgment, the message is redelivered. We hadn't designed for this.

We spent a weekend adding deduplication to three pipelines. Each one required a "seen message IDs" set in Redis and a check at consumer entry. Not complex, but the refactor touched a lot of code paths we thought were stable.

The right approach is idempotent consumers from day one, regardless of which broker you're using. The cost of adding it at the start is much lower than retrofitting it.

The monitoring gap we kept running into

Each incident above taught us something about visibility. After the silent consumer crash, we added queue depth monitoring. After the DLQ incident, we added DLQ depth alerting. After the burst volume incident, we added consumer count dashboards. After the idempotency weekend, we added deduplication coverage to every new pipeline from day one.

What's notable in hindsight: every piece of monitoring we added came after a specific failure. None of it was in place before the first incident that made it necessary. That's a pattern we eventually broke by treating monitoring as a day-one requirement rather than a response to incidents.

Here is the baseline we now deploy with every pipeline before the first production traffic:

Queue depth with trend alerting. We scrape queue depth every 30 seconds. Alert when depth exceeds a threshold AND hasn't decreased in 5 minutes. Spikes that drain naturally are fine. Depth that grows and stays grown means something is wrong.

DLQ depth with escalating alerts. A script checks DLQ length every 5 minutes. Alert at 10 entries, page at 50. The threshold review happens every quarter -- pipelines that stabilize usually get tighter thresholds over time.

Consumer heartbeat monitoring. Celery Flower gives us a real-time view of worker status and last heartbeat time. A worker that hasn't checked in for twice the expected interval gets flagged automatically.

Structured consumer logs. We log task start, task complete, and task error as separate structured events, with task ID and event type in every entry. This makes "tasks that started but never completed" a queryable condition rather than something you discover by noticing the queue isn't draining three hours later.

Process supervisor alerts. Workers run under Supervisor or systemd. Crash loops -- more than three restarts in a short window -- generate alerts to the same channel as DLQ depth. The restart log tells us when the loop started and how frequently it happened.

None of this setup is complex. The overhead is a few hours per pipeline. The value is finding problems when they're small rather than after they've been accumulating unnoticed for hours.

What we'd do differently from the start

If we were starting a new event-driven pipeline today, the decisions we'd make differently:

DLQ from day one. Not after the first incident. Every task in Celery gets a base class with on_failure that routes to a DLQ Redis list.

Queue depth alerting in week one. Not as an afterthought after we've shipped the pipeline.

Idempotent consumers from the start. The at-most-once vs. at-least-once assumption is a time bomb when the broker or configuration changes.

Burst capacity planning. Size for peak, not average.

Schema validation at consumer entry. Malformed messages should fail fast with a clear error, not propagate through processing logic.

The technical implementation for these patterns is what we documented in "How to Build an Event-Driven Data Pipeline With Python and Message Queues" -- that guide reflects what we actually do, not what looked good in the initial design.

Queue monitoring is the thing you build after your first production incident, not before it. We learned this the hard way -- a consumer died silently, the queue backed up for two hours, and we found out from a client asking why their data wasn't updating. After that, monitoring went into every pipeline from day one.

Here is the exact setup we use.

Step 1: Track Queue Depth as Your Primary Metric

Queue depth is the most important health indicator for any event-driven pipeline. It's the number of messages waiting for a consumer. A healthy pipeline keeps depth near zero. Depth that grows without recovering means consumers cannot keep pace with producers.

For Redis-backed pipelines, get queue depth directly:

redis-cli LLEN your_queue_name

For Celery, active and reserved task counts come from the inspect API:

from celery.app.control import Inspect i = app.control.inspect() active = i.active() # tasks currently executing reserved = i.reserved() # tasks waiting to execute

We push these values to our metrics backend every 30 seconds. For paid client projects we use Datadog, which has a Redis integration that tracks list length natively. For self-hosted setups we use Grafana with the Prometheus Redis Exporter -- it scrapes Redis metrics and exposes them as Prometheus time series.

Both tools let you set threshold alerts: when queue depth exceeds N for more than X minutes, send a notification. We use a 5-minute window to reduce alert fatigue from short spikes that drain naturally.

The important threshold to set: not just "depth above N" but also "depth has not decreased in X minutes." A queue that has been at 5,000 entries for 20 minutes is more concerning than one that briefly hit 5,000 and is draining. The second alert -- no decrease over time -- catches stuck consumers that aren't crashing but also aren't making progress.

Step 2: Monitor DLQ Depth With Alerting

Every pipeline we build routes failed tasks to a dead-letter queue: a Redis list called celery_dlq where tasks end up after exhausting all retries. Monitoring that list is as important as monitoring the main queue.

Our DLQ depth check runs on a 5-minute interval:

import redis r = redis.Redis(host='your_redis_host', port=6379, db=0) def check_dlq(): depth = r.llen("celery_dlq") if depth > 10: severity = "critical" if depth >= 50 else "warning" send_alert( title=f"DLQ alert ({severity})", message=f"celery_dlq depth is {depth}", severity=severity ) return depth

The send_alert() implementation varies by client: Slack webhook, PagerDuty API, or email. The specifics don't matter. What matters is that a real person, who is empowered to fix the underlying issue, sees the alert and can act on it quickly.

We alert at 10 entries and escalate at 50. Ten entries usually means a category of messages is failing consistently -- a schema change, a dependency outage, a new code path that wasn't tested with production data. Fifty entries usually means the initial alert was missed or the issue has been going on long enough to accumulate significant failures.

A DLQ that is never checked is only slightly better than no DLQ. The monitoring has to loop back to a human who can identify the root cause, fix it, and requeue the affected messages.

Step 3: Check Consumer Health Beyond Queue Depth

An empty queue can mean two completely different things: all messages were processed (healthy), or consumers are down and no new messages are arriving (broken). Queue depth alone doesn't distinguish between them.

To detect "consumers are down," we track three things:

Consumer heartbeats. Celery workers emit heartbeats on a configurable interval. A worker that hasn't emitted a heartbeat in twice the expected interval is probably down. Flower, Celery's built-in monitoring interface, displays worker status and last-seen time. We run Flower on all production deployments and include it in our standard pipeline setup checklist.

Task completion rate. Even when queue depth is low, a drop in the rate at which tasks complete is worth investigating. If the completion rate drops to zero, something is wrong even if the queue looks healthy. Datadog's Celery integration tracks task events including success and failure rates by task name.

Process supervision. We run Celery workers under Supervisor or as systemd services. Both handle crash detection and automatic restarts. We configure Supervisor to alert after three consecutive restart attempts in a short window:

[program:celery_worker] command=celery -A myapp worker -l info -Q default autostart=true autorestart=true startretries=3 redirect_stderr=true stdout_logfile=/var/log/celery/worker.log

A worker in STOPPED state after hitting startretries is what generates the page-out. The restart log entries tell us when the crash loop started and how frequently it happened, which is usually enough to identify the cause.

Step 4: Add Structured Logging at Consumer Entry and Exit

Queue metrics tell you what is happening at the infrastructure level. Consumer logs tell you what is happening at the individual task level. Both matter, and they're measuring different things.

We add structured logging at three points in every consumer task:

import logging import time logger = logging.getLogger(__name__) @app.task(bind=True, max_retries=3) def process_event(self, event_data: dict): task_id = self.request.id start = time.monotonic() logger.info("task_start", extra={ "task_id": task_id, "event_type": event_data.get("type") }) try: result = do_processing(event_data) duration_ms = int((time.monotonic() - start) * 1000) logger.info("task_complete", extra={ "task_id": task_id, "duration_ms": duration_ms }) return result except Exception as exc: logger.exception("task_error", extra={ "task_id": task_id, "exception": str(exc) }) raise

This produces queryable log lines. You can find tasks that started but never completed -- which indicates a crash between entry and success. You can find tasks taking longer than expected, which indicates a slow downstream dependency. You can filter by event_type to identify which categories of events are failing.

We aggregate logs to Datadog Logs for paid client projects and to a self-hosted ELK stack (Elasticsearch, Logstash, Kibana) for lower-budget setups. The aggregation platform doesn't matter as much as the structured format: unstructured logs are much harder to query at 3am when you're debugging a production incident.

Step 5: Trim the DLQ Before It Grows Unbounded

A DLQ that grows indefinitely will eventually exhaust Redis memory for high-volume pipelines. We cap DLQ size with an LTRIM call after every write:

MAX_DLQ_ENTRIES = 1000 @task_failure.connect def handle_task_failure(sender=None, task_id=None, exception=None, args=None, kwargs=None, **kw): entry = build_dlq_entry(sender, task_id, exception, args, kwargs) r.rpush("celery_dlq", json.dumps(entry)) r.ltrim("celery_dlq", -MAX_DLQ_ENTRIES, -1)

LTRIM with -MAX_DLQ_ENTRIES, -1 keeps only the 1,000 most recent entries and discards older ones. If more than 1,000 tasks have failed without anyone reviewing the DLQ, the operational problem is the lack of response to alerts, not the DLQ capacity. The most recent entries are the most useful for diagnosing the current failure condition.

When you fix the underlying issue and want to requeue the DLQ contents, you inspect the entries and re-dispatch them as Celery tasks before clearing the list. The details of that requeue script are in the DLQ setup guide.

What This Monitoring Catches (and What It Doesn't)

The five checks above cover the most common production failures: consumer crashes, consumers falling behind burst volume, dead-letter buildup from data issues or service outages. In our experience these account for the majority of pipeline incidents.

What this monitoring does not catch: silent data corruption where the consumer completes successfully but writes incorrect values, producer failures that stop emitting events entirely, and downstream errors that are caught and swallowed quietly in application code rather than propagated. For those failure modes you need application-layer monitoring: data freshness checks that verify output is being produced at the expected rate, and integration tests that trace a synthetic event through the full pipeline end to end.

Queue monitoring is necessary but not sufficient. The setup described here is what 137Foundry's data automation team includes as a default on every new pipeline deployment and retrofits onto existing ones after a production incident or infrastructure migration. The full context for how these monitoring patterns connect to producer-consumer architecture decisions is in "How to Build an Event-Driven Data Pipeline With Python and Message Queues".

At 137Foundry, we have evaluated a lot of tools in the feature management category over the years. LaunchDarkly, Unleash, Flagsmith, Split -- we have used several of these with clients and built with them internally. After running both the managed-platform path and the DIY path on multiple production projects, we consistently land in the same place for most applications: the custom implementation wins until the team hits 30 to 50 engineers actively using flags.

Here is our reasoning, and the specific cases where we change the answer.

The Managed Platform Pitch Is Real But Oversold

The value proposition for platforms like LaunchDarkly is genuine. You get a management dashboard, audit logs, per-user targeting, multi-environment support, SDK integrations for every major language, and a reliable API. For an enterprise team shipping to production 20 times a day, that operational tooling has clear value.

The pitch oversells, though, when it is directed at small and mid-size teams with straightforward rollout needs. Most applications need three things from a feature flag system: turn new code on and off, roll out to a percentage of users, and do both without a redeploy. Those three things take one database table and 50 lines of application code. They do not require a SaaS contract.

The real cost of a managed platform is not just the subscription fee -- it is the dependency. When you adopt LaunchDarkly's SDK, flag evaluation happens inside their library and their network requests. The flag system stops being your infrastructure and becomes an external service your application depends on. For flags that control critical application paths, that dependency matters.

What We Built and Why It Was the Right Call

For a recent client project we built at 137Foundry -- a mid-size SaaS application with a team of 8 engineers -- we implemented a custom feature flag system as part of the core application infrastructure. The full implementation: one PostgreSQL table, a 60-line Node.js evaluator with hash-based percentage bucketing, a Redis cache layer, and a React hook for client-side flag consumption.

Total implementation time: about 6 hours including tests and code review.

Total ongoing maintenance burden over 12 months: low. The system works, flags are created and retired on a schedule, and the whole thing is visible in the application codebase rather than in an external dashboard.

The complete implementation we use for these projects is documented on the 137Foundry blog. It covers the data model, the evaluator pattern (including the deterministic hash that prevents inconsistent per-user flag state), and the React integration. If you are building a new application or adding flags to an existing one, that guide is the starting point we give our own engineers.

What 12 Months of Custom Flag Operations Actually Looked Like

After shipping the custom flag system for that mid-size SaaS client project, the team used it in production for over a year. Here is what the operational reality looked like.

Flag volume stayed manageable. At the 12-month mark, the flags table had 23 rows. Fourteen were actively used, six had been permanently enabled and were candidates for cleanup, and three were operational toggles that would stay indefinitely. Managing 23 rows in a database is trivially easy. A managed platform would not have changed this.

Flag creation took two minutes. A SQL INSERT, a comment in the Slack deploy channel, and a calendar reminder for cleanup. The absence of a UI dashboard was not friction at this team size.

One rollback saved significant user impact. In month eight, a flag controlled a new billing calculation. At 5 percent rollout, the team noticed an edge case producing incorrect invoice totals for monthly-to-annual subscription upgrades. Rolling back was a single SQL UPDATE. The whole incident -- detection to resolution -- took eleven minutes. Without the flag, that bug would have hit all users simultaneously and required an emergency deploy to fix.

The only real limitation appeared at month ten. A new engineer on the project wanted to enable a flag for testing in their development environment without touching the shared staging database. The custom system had no per-environment flag isolation -- flags were global across all environments sharing the same database. The workaround was a development-only flag override in a local .env file. It worked, but it was friction that a managed platform would have solved natively.

That one limitation -- per-environment flag isolation -- is the most common trigger we see for teams moving to Flagsmith or LaunchDarkly. It is a real operational need, and the workaround is less clean than a proper solution. For teams that hit it, Flagsmith self-hosted is the most efficient path to resolving it without committing to a SaaS subscription.

The Hash-Bucketing Detail That Most Tutorials Skip

One thing we see wrong in almost every DIY feature flag tutorial is the percentage rollout implementation. The common mistake: generate a random number between 0 and 99 on each evaluation and check if it falls below the rollout percentage. This means the same user might see a feature on page load and not see it on the next API call -- fundamentally broken behavior.

The correct approach is to hash a stable input (flag key + user ID) to produce a deterministic bucket. Same inputs, same output, same flag state -- every time, for every user.

function stableHash(input) { let h = 0; for (let i = 0; i < input.length; i++) h = (h * 31 + input.charCodeAt(i)) >>> 0; return h % 100; }

This is a small implementation detail that has a large user-experience impact. Without it, 10 percent rollouts produce inconsistent behavior that looks like a bug. We have debugged this exact issue on client projects that used naive random-number implementations before coming to us.

When the Calculus Changes

We change our recommendation from "build" to "buy" in two scenarios:

The team is large and ships frequently. At 30+ engineers shipping to production multiple times a day, the absence of a management dashboard becomes a real operational friction. Engineers need to create and modify flags without database access, flag history needs to be auditable, and per-user targeting requirements tend to become more sophisticated. That is when LaunchDarkly or Flagsmith (self-hosted, for cost reasons) earns its place.

Experimentation is a primary use case. If the team is running A/B tests with statistical significance measurement as part of their growth workflow, a purpose-built experimentation platform like Split.io is meaningfully better than a custom flags table. The experiment analysis tools alone justify the cost at that point.

For everyone else -- and that is most development teams we work with -- the custom approach is the right default. Start there, understand your system, and reach for a managed platform when the operational complexity genuinely justifies it.

The Recommendation

If you are building a new web application or adding feature flags to an existing one, implement the custom approach first. Use the build guide on our site as the starting point. Set a review date at 6 months. If the flag system is causing friction, evaluate managed options at that point.

If you are inheriting an existing application with no flag system and you are under time pressure, Flagsmith self-hosted is the fastest path to a working management UI with reasonable self-hosting costs. LaunchDarkly is the right call if the team is large, compliance requirements are strict, and the subscription cost is not a constraint.