The observability tax
Every team eventually asks the same question: "Why is the checkout page slow — and is it the database, the cache, or that payment API?"
The usual answers come with a tax attached. SaaS APM agents (Datadog, New Relic) are excellent — and priced per host, per month, forever, while your traffic data flows to someone else's cloud. Homegrown solutions — logging microtime() into files, pushing counters to Redis — add code, add latency, and tend to die of neglect.
There's a third option that has quietly worked on high-load PHP farms since 2009: the Pinba extension. It's a PHP extension written in C that hooks the request lifecycle, gathers the numbers, and at request shutdown sends one small UDP packet to a Pinba server (a MySQL storage engine — see Part 1). Your code doesn't change. Your users don't wait. Your data stays on your servers.
Two lines, full visibility
; php.ini
pinba.enabled = 1
pinba.server = pinba-host:30002
Restart PHP-FPM. That's the whole integration. From this moment, every request automatically reports:
- request wall-clock time,
- peak memory usage and memory footprint,
- CPU time (user and system — so you can tell "slow code" from "waiting on I/O"),
- response size and HTTP status,
- hostname, virtual host, and script name.
Nothing to instrument, nothing to deploy inside the application. On the other end, you read it back with SQL:
-- The slowest endpoints across the whole farm, right now
SELECT script_name, req_count, req_time_total
FROM pinba.report_by_script_name
ORDER BY req_time_total DESC LIMIT 10;
Why it costs you nothing
The trick is the transport. The extension doesn't open connections, doesn't wait for acknowledgments, doesn't retry. It serializes the request stats into a compact Protocol Buffers message and fires a single UDP datagram as the request shuts down — after your user already got their response.
If the monitoring server is slow, down, or misconfigured — nothing happens to your site. No timeouts, no error pages, no cascading failure. The worst case is a lost packet, which at aggregation scale is statistical noise. This is monitoring designed on the assumption that monitoring must never become the incident.
Automatic request stats answer "which page is slow." Timers answer "*which part* of the page is slow." You wrap any suspicious block in a timer and label it with tags — arbitrary key-value pairs:
```
// How much of this request went to MySQL?
$t = pinba_timer_start(['group' => 'db', 'op' => 'select', 'table' => 'orders']);
$orders = $db->query($sql);pinba_timer_stop($t);
// How much went to the payment gateway?
$t = pinba_timer_start(['group' => 'http', 'service' => 'payments']);
$response = $client->post($url, $payload);
pinba_timer_stop($t);
```
If the timing already happened elsewhere (say, your DB client measured it), register it after the fact:
pinba_timer_add(['group' => 'db', 'op' => 'query'], $elapsedSeconds);
Request-level tags slice entire requests by business dimensions:
pinba_tag_set('api_version', 'v3');
pinba_tag_set('customer_tier', 'premium');
And then the payoff — on the server, tag reports aggregate all of this across the whole farm in real time. One SQL query tells you the total time spent in group=db versus group=http per virtual host, with request counts and percentiles. Deploy, run the query, see the diff in production behavior within seconds. No redeploys with extra logging, no "let's wait for tomorrow's log rotation."
Long-running workers are covered too — flush per job instead of per request:
while ($job = $queue->pop()) {
$t = pinba_timer_start(['queue' => $job->queue]);
process($job);
pinba_timer_stop($t);
pinba_script_name_set('worker/' . $job->type);
pinba_flush(); // send this unit of work now
pinba_reset(); // clean slate for the next one
}
The sleeping codebase
The extension's upstream stopped in 2020, at version 1.1.2. Inside, the code was a geological record of PHP history: #if PHP_VERSION_ID < 50400 branches, zend_bool, macros that existed to keep PHP 4 compiling. Meanwhile PHP 8 rewrote extension-facing internals repeatedly.
In the revival fork I cut away the archaeology and brought the code to current Zend idioms — with a support policy defined by operating system lifecycles, not just php.net's: if an OS I target ships a PHP by default, the extension supports it for that OS's life. That means everything from PHP 8.0 (AlmaLinux 9's default) through 8.5, on Ubuntu 24.04/26.04, Fedora 43/44, AlmaLinux 9/10 — with pre-built DEB packages in an Ubuntu PPA, RPMs in Fedora COPR (x86_64 and aarch64), and installation via PIE, PHP's new extension installer.
The 2008-era protobuf runtime vendored into the source tree went too — replaced by the system libprotobuf-c library, with the wire format verified byte-for-byte (same SHA-256 of the serialized packet before and after). And a CI "drift guard" now regenerates the protobuf bindings on every PR and fails if they disagree with the schema — because, fun fact, the checked-in .proto file had silently drifted from the actual wire format years ago, and regenerating from it would have quietly dropped per-timer CPU stats from every packet. The kind of landmine you disarm once and then make structurally impossible.
The bug that hid from every test
Now the promised detective story.
A C extension deserves memory-safety testing, so the plan was simple: run the test suite under AddressSanitizer, the compiler instrumentation that catches memory bugs at the exact line they happen. Compile the extension with ASan, load it into PHP, run tests. Done?
No. PHP loads extensions with RTLD_DEEPBIND, and ASan flatly refuses to work when only the extension is instrumented but the PHP binary isn't. The workarounds (preloading the ASan runtime) die on exactly the same flag. There was only one honest way forward: build PHP itself from source, instrumented with AddressSanitizer, in debug mode — a special minimal PHP whose only job is to interrogate this extension. That build is now cached in CI and runs on every pull request.
The very first run of that instrumented PHP found two bugs that had survived years of production and every conventional test run:
Bug one was an API declaration mismatch: pinba_timers_get() parsed an optional argument that its metadata claimed didn't exist. Regular PHP shrugs; a debug build treats it as the lie it is and fatals.
Bug two was the real prize: a use-after-free at request shutdown. During PHP's shutdown sequence, the extension auto-flushes data and then cleaned up its timer resources — but it set its "we're shutting down" flag after the flush instead of before. The flush deleted timer resources from PHP's resource list… and then PHP's executor, unaware, freed that list again. A double free, executed at the end of *every request with timers*, for years — silently corrupting memory in a way that happened not to crash. Until one day, on some future PHP version or allocator, it would have.
Both fixed the same week the instrumented build went live. Since then the suite also runs under Valgrind and against a thread-safe (ZTS) PHP build — which promptly exposed one more latent race in an error path, also fixed. Static analysis (clang-tidy) added its own trophy: a realloc misuse that leaked the buffer on out-of-memory.
The moral isn't "old code is bad." The moral is that test infrastructure is a feature. The bugs were always there; the difference between "mystery segfault once a quarter" and "red CI line with a file and line number" is the tooling you're willing to build.
What this unlocks
With the extension alive again, the whole free monitoring stack clicks together:
- production metrics with zero code changes and zero request latency,
- custom timers and tags for anything you care about — SQL, caches, queues, external APIs, feature flags,
- all of it queryable in real time with plain SQL, on your own hardware, with your data never leaving your network,
- installable from normal OS packages:
apt,dnf, PIE, or Docker.
```
Ubuntu / Debian-based
sudo add-apt-repository ppa:xolegator/packages
sudo apt install php-pinba
Fedora / AlmaLinux / Rocky
sudo dnf copr enable xolegator/pinba
sudo dnf install php-pinba
```
Next in the series: Pinboard — the dashboard that gives Pinba a memory: request-time percentiles, per-page history, e-mail alerts, and a Symfony-8 rewrite of a project first committed in 2013.