One Electron app, two modes, one interface. What a two-day hackathon build taught us about on-device diarization, CoreML and DirectML, read against a production on-device engine.

At this year's internal hackathon, our esteemed colleagues Georgios Hadjiharalambous and Stuart Wood set out to build Inkwell, a dictation app centered around high-accuracy and speaker-aware note-taking, powered by Speechmatics speech-to-text.

Nothing too unusual there.

What makes Inkwell worth writing about is the decision they made early on: the app would run in two modes, cloud and fully local, behind one identical interface.

Flip a toggle, and the same Electron UI talks to either a realtime cloud endpoint or a speech recognition engine running entirely on the laptop in front of you, no network required after install.

That second mode is obviously the interesting one. "Local Whisper works" is a solved problem developers stopped being impressed by two years ago. What's still mostly undocumented is what it actually takes to get a production-grade, diarization-capable speech-to-text engine running as a real desktop feature.

A lot of accounts also tend to omit which parts of the process have to be a native binary, which parts of the accuracy story survive the trip off the cloud, and where GPU dependency stops being an optimization and starts being a hard requirement.

A 2-day hackathon project turns out to be a good lens for this gap, because none of the corners it cut were hidden by scale or budget. If something was hard, it stayed hard in the demo.

This post is an engineering read of Inkwell's local mode, not a product announcement. But the architecture decisions inside it line up closely with lessons we've already published from a much larger production effort.

Reading the two side by side is instructive: one is the fast, scrappy version of a decision; the other is what happens when you have to make that same decision hold up across millions of consumer devices.

The local path has one fewer hop than the cloud path

Inkwell is an Electron app, which means the renderer process (the UI) and the main process (Node.js) are already split the way most desktop apps are. For local mode, Georgios and Stuart added a third participant: a native C++ inference engine, spawned as a child process and talking over a loopback WebSocket.

That's a deliberate architectural choice. The renderer talks straight to the local engine over the WebSocket, bypassing the Electron IPC hop entirely, so audio goes from the browser layer to the recognition engine with one fewer hop than the cloud path takes.

The main process' job shrinks down to supervising the binary: spawn it, pass it a GPU backend flag (--coreml or --directml, plus a device index, so the client can pick which GPU to use if the machine has more than one, say a discrete NVIDIA card alongside integrated Intel graphics), and hand it a bundled language pack from disk. No API key touches this path at all, because secure, local authorization keeps you logged in and productive without waiting on internet-based authentication calls.

Worth flagging that the ideal for the underlying engine is diarization that updates continuously, so speaker labels are available at any instant rather than only once a stretch of audio finishes. That's the kind of detail that's easy to get wrong from the outside and worth checking with the team before this goes further.

Either way, it's a compact demonstration of something the report on HackerNoon's voice AI coverage flags as underserved territory: developers are told "on-device now works," but almost nobody publishes where, specifically, and why, it stops working without a GPU.

The part that looks easy and isn't: the backend flag is the tip of a months-long optimization chain

Spawning a native binary with a GPU backend flag sounds simple to describe. Getting that binary to actually be fast and accurate on ordinary hardware is where the real engineering effort goes, and it's exactly the ground we've already covered in detail.

When we rebuilt Speechmatics' on-device model to keep pace with Whisper for Adobe Premiere, quantization was the lever that made a cloud-grade model fit on a laptop at all: on a 2020 M1 MacBook Pro, our quantized model, ESL (Edge Speech Library), hit 47.2 s/s throughput on GPU/ANE using about 1.1 GB of RAM, against the closest Whisper-based tool's 11.7 s/s. On a Dell XPS 16, ESL reached 25.3 s/s multi-threaded on an RTX 4050 using 1.7 GB total memory, versus whisper.cpp's 22.1 s/s at nearly double the memory footprint.

The numbers were the easy part to report. The hard part, which we wrote up at length, was that quantization is one link in a brittle chain: export from PyTorch to ONNX, generic graph optimization, then hardware-specific optimization for CoreML on macOS or DirectML on Windows, with each stage capable of silently skipping an optimization or breaking the model outright if the inference graph doesn't match what the tooling expects. Reference-model benchmarks (ResNet, DistilBERT) don't tell you what will happen to an audio transformer's graph.

We ended up writing custom export and optimization scripts specifically because the standard tooling's pattern-matching wasn't reliable for our architecture.

Inkwell's --coreml/--directml/--cuda flag is the visible tip of exactly that iceberg. Choosing the backend at runtime instead of hardcoding one is the right call for a cross-platform desktop app, but it only works because there's a correctly optimized model behind each flag.

A hackathon team gets to treat that as a given, thanks to the engine already existing. Anyone building this from scratch, on top of an arbitrary model, is signing up for the export-and-optimize chain described above, not just a pip install.

Cloud vs local, line by line: diarization is GPU-only

The side-by-side comparison the team built into the app's own slide deck is a good, honest summary of what actually changes between modes:

| | Cloud mode | Local mode |
|---|---|---|
|
| wss://{region}.rt.speechmatics.com/v2 | ws://127.0.0.1:{port} |
|
| API key → short-lived JWT exchange | None (loopback only) |
|
| Server-side, always latest | Bundled language pack on disk |
|
| Required | Not required after install |
|
| Audio leaves the device | Audio never leaves the device |
|
| Built in | GPU only |
|
| API key | Language pack + GPU drivers |
|
| Network round-trip + engine | Engine only |

Nothing in that table is a surprise if you've built either kind of system before, but we think it’s rare to see it stated this plainly in one place, for one app, with both modes actually implemented rather than one being aspirational.

It's also a clean illustration of a point we think is under-argued in the industry right now: local versus cloud isn't a philosophical choice, it's a checklist of which specific trade-offs your users can tolerate. Inkwell's answer was "let the user pick," which is a reasonable answer for a note-taking app used in both offline meeting rooms and everyday cloud-connected work.

Four things to decide before you ship a local mode

If you're evaluating whether to add a local speech-to-text mode to an app that already has a cloud one, a few things from Inkwell (and the production work behind it) are worth internalizing:

• Treat GPU access as a feature gate.If your local mode does diarization, decide explicitly what happens on CPU-only hardware: degrade to transcription-only, refuse to run, or clearly warn the user. Don't assume it'll just be slower.

• The backend flag is downstream of months of optimization work,not a config option you get for free. If you're not starting from an already-quantized, already-graph-optimized model, budget real time for the export chain, not just the integration.

• A comparison table is a better spec than a feature list.Writing down exactly what changes between deployment modes, endpoint, auth, latency source, privacy posture, forces decisions that "it just works in both modes" quietly avoids.

• Hackathons are a good place to find real architecture decisions.The constraints (no time, no budget, no room to fake it) tend to surface the decision points that matter, even if the code around them doesn't survive past the demo.

What's next: what we'd need to make this generic

Inkwell isn't a shipped product but the local-mode architecture is a genuinely useful, small-scale case study in what on-device speech recognition costs to build properly, and it lines up with what we found doing this at Adobe's scale.

We’re carrying out active internal work exploring how to turn our current on-device engine from a customer-specific build into a generic, licensable, sellable on-device product.

If you want to try building something similar, the deployment options documentation covers cloud, on-prem, and on-device paths, and the realtime quickstart is the fastest way to get a cloud-mode WebSocket connection running in an afternoon. For runnable examples, the Speechmatics GitHub Academy has notebooks to start from.

And if you want the long version of the on-device optimization story referenced above, both Part One and Part Two of the Adobe story are worth the full read.