My dispatcher cost 10 CU-hours a day. The database it kept awake cost 136.

For a month I thought I had already fixed the cost of our data refresh control panel. In September I moved its dispatcher notebook from Spark to the Python kernel and cut it from about 99 CU-hours a day to about 10.5. Then I sorted Capacity Metrics by item and found that the largest thing on the whole capacity was not a notebook or a pipeline.

It was the panel's small Fabric SQL database, at about 136 CU-hours a day. It was online about 97% of the time to do under two CPU-minutes of real query work. The dispatcher was cheap. The database it kept awake was not.

This post covers how I found that, the four changes I shipped in one day, and what I would check first on any Fabric app that polls.

The setup

The control panel is an internal web app for managing a nightly data refresh. People use it to run a pipeline on demand, pause the schedule, and change which sources and portfolios are in scope. It is deployed to Fabric through a shared app platform, and it has its own Fabric SQL database.

The panel never starts pipelines itself. Every click writes a request row into its database. A separate dispatcher notebook reads those requests, claims each one, triggers the right pipelines and writes the status back.

The catch was how the dispatcher found out there was work. It didn't. It ran on a cron schedule every minute, 1,440 times a day, and checked. A click in the panel just waited for the next tick, up to 60 seconds. The panel's UI also refreshed itself every 15 seconds, including in browser tabs nobody was looking at.

Where the capacity actually went

Capacity Metrics for the two weeks before the change, 29 September to 12 October, told a different story from the one I expected.

Capacity Metrics groups items by name, so the database row also includes its own SQL endpoint and the app's server-side functions.

The notebook I had already optimized was the smallest line.

The reason is how a Fabric SQL database bills. It bills while it is online, at roughly 150 to 170 CU-seconds per online minute, and it releases compute only after 15 idle minutes. A dispatcher that touches it every 60 seconds means it never gets those 15 minutes. Neither does a forgotten browser tab polling every 15 seconds. Each touch was tiny. The cost came from the gaps between touches never being long enough.

When I first saw that figure, billing while online explained only about half of it. At 150 to 170 CU-seconds a minute and 97% online, that comes to 58 to 66 CU-hours a day. The after numbers settled it: the item fell to about 1.2 CU-hours a day. So almost all of the 136 came from the constant touches, either as online time or as the reads and GraphQL calls themselves. I haven't split those two apart.

The September port did not cause this. Even at the old 5-minute cadence the database would never have reached 15 idle minutes. But it shows the trap: I measured the thing I had changed, and the thing I had changed was not the problem.

What I changed

Four changes, all shipped on the same day.

1. The panel wakes the dispatcher

I added a small server-side function to the app that starts the dispatcher through the Fabric jobs API (POST .../jobs/instances?jobType=RunNotebook). It runs as the app's own identity. The workspace and notebook IDs are fixed inside the function, so a caller cannot point it at anything else; the caller's reason string is only logged.

Before starting anything, the function lists the notebook's job instances. If one is already NotStarted or InProgress, it returns "already running" and does nothing. Any response other than HTTP 202 counts as not started.

On the client, every change already went through one helper, so wiring in the wake was one line. There is a 10-second cooldown, and the wake never throws: a save always succeeds, and a one-line note says whether the dispatcher started. The panel also wakes the dispatcher on refresh when the last sync is more than 30 minutes old, at most once every 5 minutes. That catches someone opening the panel after a quiet spell.

2. The dispatcher keeps itself going while there is work

One wake is not enough. A pipeline run takes minutes to hours, and the dispatcher has to keep checking its status and queue follow-up steps. So each pass now ends with one question: is anything still open? That means pending, claimed or running requests from the last 24 hours, plus any setting whose desired state does not match its actual state yet.

If work is open, the pass sleeps 45 seconds, then posts a new run of itself with chain_depth + 1. Three details mattered:

• Create the next job before the current one ends. The panel's "already running?" check never sees a gap, so a click mid-chain cannot start a second chain.
• Pass every parameter forward explicitly. Any notebook parameter you leave out reverts to its default.
• Cap it. A chain stops after 240 passes, about 5 hours. Anything still open is picked up by the next panel action or the daily run.

Each request is also claimed with a conditional update: UPDATE ... SET status = 'claimed' WHERE id = @id AND status = 'pending'. If no row changed, another pass already has it and this one skips it. Two overlapping passes cannot trigger the same pipeline twice.

The every-minute schedule became one run a day at 06:00 local time, a safety net for anything a missed wake leaves behind. That is also the main trade-off: if a wake fails, the change waits for the next panel action, the 30-minute stale-sync wake, or the daily run, and the panel says "Could not start the dispatcher".

3. The panel stops polling when nobody is using it

After the first two changes went live, Capacity Metrics still showed SQL usage every minute. The cause was one open panel tab, reading the database through GraphQL every 15 seconds.

The panel now pauses auto-refresh while its tab is hidden, and after 10 minutes with no pointer, key, wheel or touch event. A banner says why, and any click resumes it. It keeps refreshing while a run is in flight, using the same "is work open?" rules as the dispatcher, so a run being watched never looks frozen. An idle panel now lets the database sleep about 25 minutes after the last interaction: 10 idle minutes, then the database's own 15.

4. The source-launch notebook moved off Spark

The notebook the nightly refresh runs three times to launch the source systems was using about 28 CU-hours a day on Spark. Most of that was an idle session waiting on pipelines, which is not work Spark is needed for. It now runs on the Python 3.12 kernel, and now averages 3.9 CU-hours a day, 86% less, doing the same job.

What happened

The last every-minute run started at 15:42 UTC. From then to 21:10 the dispatcher ran 30 times, where the old schedule would have started about 328. Of those 30, 27 were one chain driving a real workbook refresh and publish end to end. Nothing at all ran after 17:38.

That chain is the clearest picture of the new design:

The last pass took 31 seconds instead of about 75, because nothing was open: it did not sleep or requeue.

Then the part I actually cared about. Capacity Metrics showed the database asleep at 18:07, 15 minutes after I closed the panel tabs. It woke for the next deploy and check from 18:10 to 18:34, then stayed asleep through at least 19:28, with zero interactive operations on the capacity.

There were two side benefits. A change now applies 7 to 9 seconds after Save instead of up to 60. And the dispatcher no longer starts during the 05:00 burst, when the nightly refresh starts about a dozen notebook sessions in the same minute.

Eleven days later, Capacity Metrics had the answer. This compares the two weeks before the change with the 11 days after it, leaving out the change day itself.

That is about 197 CU-hours a day back on the capacity, roughly 13% of an F64's 1,536 a day. The busiest day after the change, with real panel work in flight, came to about 12 CU-hours, still only 6% of a normal day before it.

One caveat: the lakehouse averaged about 20 CU-hours a day until early October, then climbed to 39 to 58 in the last five days before the change. Against the earlier level, its drop is closer to 93%.

What I would check on any Fabric app that polls

1. Sort Capacity Metrics by item, not by what you just changed. The biggest line on the capacity was one I had never looked at.
2. For anything that bills while online, count the gaps, not the work. A Fabric SQL database pauses only after 15 idle minutes. A touch every 60 seconds and a touch every 15 seconds cost the same, because neither ever lets it pause.
3. Count every client, including browser tabs. Fixing the server-side poller was not enough. One forgotten tab kept doing the same damage.
4. Event-driven does not have to mean one-shot. Wake the worker on the event, let it keep itself alive while work is open, and let it stop by itself when nothing is. Guard it with an "already running?" check, a conditional claim, and a cap.
5. Re-check schedules after every notebook import. On deploy day, run history showed every-minute runs coming back for a few minutes after the dispatcher was re-imported, until the extra schedule was removed. My reading is that the import recreated it. Now, after any import, I list the notebook's schedules and keep exactly one.
6. Deploy from merged main only. That afternoon, a deploy from a different feature branch overwrote the event-driven build and dropped the wake function. Two features that deploy to the same app item have to ship as one build.

The change that mattered most didn't make anything faster. It let one small database go to sleep, and 136 CU-hours a day became 1.2.

I rebuilt the pattern from scratch on synthetic data as a small open-source repo, with the wake function, the self-chaining dispatcher and the conditional claim: fabric-event-dispatcher.