Key Takeaways

• A monolith-to-microservices migration can succeed without a dedicated migration budget or multi-year plan by making every new feature, bug fix, or enhancement carve out a new domain service instead of touching the monolith, so the migration becomes a side effect of regular product work.
• The service-first approach adds roughly 50% to individual story times (a 4-to-6-hour tax in our case), which is affordable when the platform can provision a new service in minutes rather than weeks, and unaffordable when it cannot.
• Three platform investments make the approach work in production: self-service provisioning, an API gateway as the routing seam between old and new implementations, and a cached feature-flag layer that supports both gradual rollout and near-instant rollback.
• For workloads where losing data is not an option — payroll, timekeeping, transactions — every event should land in a durable store before any business logic runs, with a deterministic backfill stream so downstream failures never become data loss.
• Predictable traffic bursts, such as a shift-change spike, can be handled at roughly 70% lower cost than always-on peak capacity by combining scheduled pre-warming, reactive autoscaling, and serverless fan-out for the events pipeline.

Why the Funding Problem Isn't Really a Funding Problem

In May 2021, my team and I at Paycor (a human capital management platform, since acquired by Paychex) made a decision that ended up shaping the next five years of our platform. We stopped changing the monolith.

There were three monoliths at that point: large, multi-tenant, and deeply intertwined with the human capital management (HCM) product surface area we were responsible for. We had wanted to break them up for a long time. We had read the strangler-fig pattern. We had drafted multi-year migration plans.

The plans never got funded. Every quarter, the case for migration competed with revenue-generating features and lost.

So we tried something different. Instead of a separate program, the rule became simple: every new feature, every bug fix, every enhancement, anything that would have touched the monolith would instead carve out a new domain service.

The migration would happen as a side effect of regular roadmap work, paid for by the same budget that paid for everything else. A pull-based migration.

The cost was real. A change that would have taken four hours in the monolith now took six. We absorbed that cost across hundreds of stories instead of trying to package it into a CapEx ask that was unlikely to be approved.

Today, we run more than 120 domain microservices on Azure, with zero downtime across the entire migration. The three monoliths are still running for the small fraction of functionality no one has needed to touch. Everything else has moved.

This article covers how the approach actually worked: the platform investments that made it possible, the architectural choices it forced us to make, the cost engineering that kept it affordable, and the things I would do differently if I were starting again today.

Three Platform Unlocks That Made Pull-Based Migration Possible

Pull-based migration — where each product-driven code change carves its own service, rather than a separate program migrating on its own timeline — sounds simple in principle. In practice, it depends on three platform investments. We did not have all three when we started, and the migration was painful for the first quarter or so. Once those three were in place, the migration became routine.

Subscription-per-Domain on Azure

The first investment was structural. We adopted a pattern of one subscription per product domain. Time and attendance live in its own subscription. Payroll lives in another. Reporting, scheduling, integrations, each gets its own.

This sounds heavyweight on paper. In practice, it gave us three things that mattered more than we expected.

First, cost attribution. The bill for the time-and-attendance subscription tells us, every month, exactly what that domain is spending on cloud resources. We don’t allocate shared cloud costs across domains. Each team sees its own bill and is accountable for it.

It also gives us blast-radius isolation. A misconfiguration, a runaway script, an aggressive new IAM policy: all stay inside the domain’s subscription. We have not, in five years, had a single multi-domain outage caused by a configuration change.

And there’s a softer third effect: team accountability. When engineers can see their own resource consumption and their own platform footprint, they make different decisions in code review. That is a soft outcome, but it has shown up in the data.

The provisioning template is what makes the subscription pattern actually usable. When DevOps stood up a new service, the template included the Azure Kubernetes Service (AKS) namespace, the database, the Service Bus and Event Hub setup, the Key Vault, the observability hooks, and the network configuration. Provisioning that used to take two to four weeks of tickets dropped to minutes of self-service.

This was the biggest behavior change of the migration. Engineers will not pay a 50% upfront tax to build a new service if standing up the service takes three weeks. They will pay it if the new service takes ten minutes.

APIM as the Routing Seam

The second investment was operational. We put API Management (APIM) in front of every endpoint that we expected to migrate.

A monolith endpoint and a new domain service can both register with APIM under the same path. The traffic split is set by deployment configuration, not by client changes. We could send 1% of traffic to the new service, watch it for a day, then dial it up.

Clients never knew the cutover was happening. The mobile app didn't know. Partner public API consumers didn't know. APIM owned the contract. The new service implemented the contract. Traffic moved when we said it should move.

Feature Flags via Azure App Configuration with a Redis-cached Wrapper

The third investment was tactical. We needed a feature-flag layer we could trust at production volume.

Azure App Configuration has the right semantics for this. Calling it from every request, however, gets expensive at our traffic shape. So we built a thin wrapper that caches values in Redis with short TTLs and explicit invalidation hooks. From the application's point of view, a feature-flag check is a Redis lookup, which costs us nearly nothing.

The wrapper does two jobs. The first is gradual rollout: turn a flag on for 1% of tenants, watch the metrics, expand. The second is near-instant rollback. If a release misbehaves, flipping the flag in App Configuration propagates to every node within seconds.

The gradual-rollout work is the part most engineers think about. The near-instant rollback is the part that has saved us more than once.

Figure 1: Three platform unlocks that make pull-based migration possible. Source: Created by author.

The HCM SLOs and the Time-System Ingestion Path

Working in HCM, two SLOs shape almost every architectural decision we make.

The first is for the time-and-attendance system, which I’ll just call the time system from here on. When an hourly employee clocks in or out, we cannot lose that event. A lost punch becomes a missed paycheck for someone whose budget is already tight. There is no acceptable margin of error.

The second is for payroll. When payroll is supposed to run, it has to run. Downtime during the payroll window is not an inconvenience. It is a regulatory event with reporting consequences.

These two SLOs drove a specific set of architectural commitments.

Ten Ingestion Sources, One Durable Record

The time system has to accept punches from a long list of devices and channels. We currently ingest from ten sources:

• The employee self-service portal
• Physical time clocks at job sites
• Kiosks in break rooms
• Point-of-sale systems for retail and food-service customers
• iPad apps in field operations
• Mobile apps on iOS and Android
• The partner public API used by integrations
• Phone-based systems
• Manager-entered punches in the admin UI
• Edge devices for specific verticals

Each source has its own quirks. Connectivity is unreliable on physical clocks at remote sites. The partner API has rate limits we don't control. POS systems batch their submissions during quiet periods. Mobile apps queue offline and replay when connectivity returns.

Regardless of channel, the first thing every source does is land the punch into a Service Bus queue. From the queue, the punch is written to Table Storage as the durable record. Only after the durable write succeeds do we acknowledge the source.

This sequence (land in the queue, write the durable record, acknowledge) is the core of the SLO. Once a punch has a durable record, we can recover from any downstream failure. Until it has a durable record, we cannot say it has been received.

Event Hub Fan-Out

From the durable record, an Azure Event Hub broadcasts to four downstream consumers:

• Punch processing, which applies policy and calculates the wage event.
• The database of record, the system of truth for the punch.
• Reporting, for analytics and compliance.
• The recent-punch list, which is what the user sees in the UI immediately after they punch.

Each consumer reads from its own consumer group. They scale independently, fail independently, and none of them blocks the others.

The recent-punch list is the consumer that matters most for user perception. When an employee taps "clock in" on their phone, the UI confirms accept or reject within a fraction of a second. That confirmation is driven by the durable write, not by downstream completion. The phone does not wait for payroll calculations.

Deterministic Backfill

Downstream consumers fail. Not often, but they do. When they fail, we run a deterministic backfill stream that replays from the durable record forward, in order, into whichever downstream system is recovering.

The replay is idempotent at the consumer side: each downstream maintains its own watermark and skips events it has already processed.

The result is that a downstream outage never produces a lost punch. The punch is in the durable record from the moment it was acknowledged. Whatever happens downstream is recoverable.

That’s what landing every event durably actually buys you.

Payroll

Payroll runs multi-region active-active across two Azure regions in the United States (East and West). Same regulatory jurisdiction, low inter-region latency, and a shared operational team. The audit trail is preserved across both regions. Failover is tested quarterly with planned region drains.

We deliberately do not multi-region every other component. What we multi-region and what we don't is a budget choice. Most things tolerate single-region operation just fine, and over-replication would cost more than it saves.

Figure 2: The time-system ingestion path — ten sources land in Service Bus and Azure Table Storage as the durable record, with Event Hub fan-out to four downstream consumers and a deterministic backfill stream for recovery. Source: created by author.

Surviving the 8-9 AM Peak

The time system has a daily traffic shape that defines its cost structure. Between 8 and 9 AM local time, the punch volume per region runs into the millions, with the actual peak compressed into about a five-minute window as workforces clock in for first shift.

This shape is good news and bad news. Good news, because it is predictable: we know exactly when it is coming and how big it will be. Bad news, because naive cloud capacity for that shape gets expensive fast. The peak has a weekly shape too: weekdays look similar, weekends are much lower (retail and hospitality still punch, but not enough to change the aggregate). Our scheduled rules use both a daily and a weekly curve.

If we sized always-on capacity to the peak, we would be paying full price 24 hours a day for compute that runs at 5% utilization for 23 of them. We would also be over-paying for the majority of the year when workforce patterns are stable.

We do not do that. The combined approach we run today cuts burst-window spend by about 70% compared with always-on capacity sized to the peak. Here is how it works.

Time-Based Scaling

We have a scheduled rule that pre-warms capacity 15 minutes before each region's expected peak. The rule knows the local time-of-day curve for the region. It is not the same in every region, and the East and West coasts of North America hit at different absolute times.

The pre-warm exists for one reason: reactive autoscaling alone is too slow for a five-minute peak. By the time a reactive scaler notices the load, decides to add nodes, schedules them, and gets containers running, the peak is over. Pre-warming with a scheduled rule gives us the baseline. Reactive autoscaling absorbs the variance on top.

Reactive Autoscaling

The variance is real. Different customers have different shift-start patterns. Some workforces start at 8:00 sharp. Others at 8:15 or 8:30. Holidays shift everything. Daylight saving transitions throw off the scheduled rule by an hour twice a year.

For all of that variance, reactive autoscaling on CPU and queue depth handles the long tail. The pre-warm gives us the headroom. The reactive scaler tunes within it.

Serverless Fan-Out

The Event Hub consumers do not run on AKS. They run as Azure Functions in the consumption plan, billed per invocation.

Fan-out workloads are bursty and stateless. We do not want to pay for nodes when there are no events to process. Functions scale up to handle a punch surge in seconds and scale to zero when the surge is over. For a workload that runs hard for five minutes and then sits idle, the consumption-plan cost is dramatically lower than equivalent always-on compute. The consumption plan caps concurrent instances per function app, and we hit that ceiling during the peak. The fix was to split the fan-out into multiple function apps aligned with the downstream consumers, so each has its own concurrency envelope. The failure mode is silent queue growth, not a clear error — watch Event Hub lag. Cold starts were less painful than expected because the pre-warmed AKS layer absorbs the first few hundred milliseconds.

Caching

Caching matters at every layer. The most important caches are:

• Redis in front of App Configuration for feature flags and tenant-level settings.
• Tenant configuration: which clients use which features, with what limits.
• Schedule lookups: the schedule data that punches are matched against for policy.
• Policy data: the rules that determine whether a punch is accepted, late, early, overtime, and so on.

The feature-flag cache was the one we underestimated. App Configuration calls add up at our request volume. Without the Redis layer in front, we would be making millions of cross-network calls to App Configuration during the peak. With Redis caching with short TTLs, we make tens of thousands instead.

We did get bitten once by a cache stampede. A schedule data update invalidated a heavily-shared cache key, and the next request from each of several thousand processes simultaneously tried to repopulate the cache from the source of truth. The source of truth almost fell over. We added jittered TTLs and a single-flight pattern around the repopulate path.

APIM and Traffic Manager

At the regional boundary, APIM and Traffic Manager smooth saturation. If a region is approaching its capacity ceiling during the peak, traffic shifts to another region. We don't run active-active load-balancing for the time system at every moment, but the failover and overflow capacity is there when we need it.

The combined effect is that a workload designed around a known burst is dramatically cheaper than one designed for worst-case-always.

The Cutover Pattern, Repeated More Than 100 Times

Once the platform was in place, the migration mechanics became repeatable. Over five years we have run this cutover pattern more than 100 times:

• Build the domain service. A new service, in a new domain subscription, behind the standard provisioning template. It implements the same contract as the monolith endpoint it will replace.
• Dual-run behind a feature flag. The application code checks the flag. If the flag is on, it calls the new service. If the flag is off, it stays on the monolith path. Initially the flag is off in production for everyone.
• Turn the flag on for internal traffic. Our own QA traffic, our own dogfooding accounts, our own load tests. Compare the responses for parity. Fix the divergences.
• Shift APIM weight. Once the parity is clean, shift a small percentage of production traffic to the new service via APIM weight. Watch the SLOs.
• Dial up. If the SLOs hold, dial up the weight by an order of magnitude every few days: 1%, 10%, 50%, 100%.
• Retire the monolith path. Once the new service has owned 100% of the traffic for long enough to be confident, the monolith path is removed.

The "Migrated" Metric

We track an explicit metric for declaring an endpoint migrated. The metric is binary: either 100% of production traffic for the endpoint has been served by the new service for at least 30 days with the SLOs intact, or the endpoint isn't migrated yet. There is no in-between.

This sounds pedantic. It is critical. Without a hard definition of "migrated," teams keep poking at the new service indefinitely, hedge with monolith fallbacks, and never fully commit.

Rollbacks

Most cutovers go smoothly. Some don't. A representative list of what has gone wrong:

A new service had a hidden dependency on a monolith database column we hadn't realized was being read. The flag rollback got us back inside a second. The fix took two days.

An APIM weight shift to 50% revealed a connection-pool exhaustion in the new service that hadn't shown up at 10%. The flag rollback brought us back to 0% for the new service. We tuned the pool and re-ran the cutover.

A scheduled load test ran during a cutover window and skewed the SLO numbers. We learned to coordinate test schedules with cutover windows.

In every case, the feature-flag layer was the rollback mechanism. The flag flip propagates within seconds, which is the right scale for a production incident. APIM weight shifting is slower (minutes, not seconds) and is for traffic-shaping, not for emergency rollback. The two layers together cover our requirements.

What I Would Do Differently and a 90-Day Plan

Five years of pull-based migration have taught me what the approach is good for and where it breaks down.

Where It Breaks Down

The pull-based approach works because every change to a piece of code is an opportunity to migrate it. The flip side is that code nobody is changing doesn't get migrated. The cold corners are still in the monoliths: features we shipped years ago, that work, that no one has touched in a year, that customers depend on.

We used to think this would resolve itself. It hasn't. Five years in, the cold corners are still cold. They will not migrate themselves.

The pull-based approach handles roughly 80% of a system. The remaining 20% needs a different approach: either a deliberate one-off project or a willingness to leave it in the monolith forever. We are now doing some of both.

What to Standardize Before Allowing Service Number Two

If I were starting again today, I would not let the second domain service be created until five things were in place:

• The subscription-per-domain provisioning template, fully self-service.
• The feature-flag layer, App Configuration with the cached wrapper, exercised in staging.
• The APIM contract for the routing seam.
• The observability bundle, with metrics, traces, and logs flowing to a known destination.
• The durability pattern for any high-stakes ingestion: Service Bus, durable record, Event Hub fan-out, deterministic backfill.

Skip any of these, and the second service will need rework, and the third service will need rework. Standardize them first, and every subsequent service is cheap.

The Org-Side Moves That Mattered

Three non-technical moves did more for the migration than most of the platform work.

The first was a code-review gate that explicitly asked: "Does this change carve out a new domain service or extend the monolith?" If it extended the monolith, the reviewer asked why. Most of the time, there was a good reason. When there wasn't, the change got reshaped in review.

The second was an architecture review for the seam map (which endpoints belonged in which domain) that we ran early and revisited every six months. The map drifted between reviews, and the review caught the drift.

The third was tracking an engineer-experience metric for the upfront tax. We watched cycle time for stories that involved building a new domain service against monolith changes. As long as the gap stayed at the 4-hour-to-6-hour level we expected, we kept going. If it had widened, we would have stopped and looked for what was making the new path expensive. The metric was the check on whether the platform was delivering on its promise.

A 90-Day Plan

If you are an engineering leader sitting on a monolith with no migration budget, the following plan gets you to a place where you know whether the approach works for you:

• Days 1-30. Pick one domain. Build the provisioning template. Stand up APIM in front of one endpoint. Stand up the feature-flag layer.
• Days 31-60. Carve the first domain service. Cut over one endpoint. Measure the upfront tax.
• Days 61-90. Carve a second service. Standardize what you learned. Document the seam map for the next domain.

By day 90, you will know whether the approach works for your organization. You will also know whether your team has the org-side discipline to keep paying the upfront tax over the long run, which is the harder question.