Why it ran hot

Part one ended with the phone working, and for a while that was the end of it. postmarketOS on a mainline kernel, Docker on top, a Tailscale address, one small service that needed nothing from me. It went on a shelf behind a monitor, and I stopped thinking about it, which is what I wanted from it.

The trouble with a machine you have stopped thinking about is that you also stop knowing anything about it. Mine was a list of running containers and an SSH prompt. Whether it was healthy, whether it was warm, whether it was quietly wearing out its battery — I had no way of saying. So I put a recorder on it, a few readings written to a file every five minutes, mostly to check that the service holding the battery at half charge was still doing its job.

It was. The same file also said the CPU had been sitting at 70 °C. Not while building something: that was the number it rested at, day and night, on a device with no fan, no heatsink, and nothing to move heat into but its own back cover.

My first reaction was to accept it. A 2016 Snapdragon sealed in plastic runs warm, 70 °C is well inside what the silicon tolerates, and nothing else about the machine was wrong.

The question I should have asked straight away is what it was doing to produce that.

Telemetry

The recorder

The reason I started recording anything was the battery. Part one ended with a small service holding the pack at about half charge: every twenty seconds it reads the current flowing in and out of the cell and nudges the charger's input-current limit until that current is near zero, so the battery neither charges nor discharges while the phone runs off the wall. I had watched it work while I was writing it, and had no way of knowing whether it was still holding or slowly charging the pack to full — which is the failure I wrote it to prevent.

Nothing on the machine could answer that. Portainer, the web interface over Docker I run on it, draws a CPU graph, but it graphs containers; the pack, the charger and the thermal zones sit underneath Docker and are invisible to it.

So I wrote a recorder, and decided up front that it would not be a daemon. Whatever I built was going to run on the machine whose power draw I eventually wanted to measure, and a resident process is part of what you are measuring. A systemd timer starts a shell script, which reads its files, appends one line and exits, at a cost of about 50 ms of CPU per run.

[Timer] OnBootSec=2min OnUnitActiveSec=5min AccuracySec=30s
The 30-second accuracy window lets systemd slide the firing to coalesce it with other timers, so the recorder mostly rides wake-ups the machine was going to have anyway.

Most of the line comes from the fuel gauge. The OnePlus 3T has a bq27541, a coulomb counter wired in series with the pack: rather than inferring the state of charge from terminal voltage, it integrates the current going in and out and reports what it has counted. Mainline exposes it as an ordinary power supply, so charge percentage, current, voltage and the pack's own temperature are four files under /sys/class/power_supply/bq27541-0/.

The rest I picked. The charger's online flag, so that a discharge reading could later be told apart from a meaningless one. The hottest of the cpu*-thermal zones rather than a fixed one, because which core is hottest keeps moving. And CPU busy percentage, computed as a delta against the previous run's /proc/stat, kept in a dotfile beside the CSV.

The viewer

Reading the file back needed a second process, and the same argument applied to it:

[Socket] ListenStream=8088 Accept=yes
With Accept=yes systemd hands the accepted connection to the program as its standard input and output, so the viewer can be an ordinary script that reads a request and writes a page. A Python process exists for the length of one page view and then stops.

The readings

Sampled at the same hour every day, the current in and out of the pack was 0 mA every time, and the charge drifted from 50 % to 46 % as the cell self-discharged and the loop left it alone. It was erring low, which is the side I wanted.

The temperature column beside it never left a daily average of 65.2–68.1 °C.

Nothing else in the file explained that. CPU busy averaged 2.3 % and never even touched 10 %, so nothing was burning a core in the background. The battery's own temperature sat at 24.7 °C, which puts the heat in the SoC rather than the battery.

The only reading taken with the charger unplugged is -610 mA at 3806 mV, which is 2.3 W for a machine whose four cores were idle 97 % of the time.

Nothing here looked like a fault. Every service was up and no log showed an error. That left the question of where 2.3 W goes on a machine that is doing nothing.

cpuidle

The first place to look was what the kernel does with a core that has nothing to run. That is cpuidle: a governor estimates how long the core will stay unoccupied, and a platform driver turns that estimate into one of the sleep states the hardware has, at a known wake-up cost.

On arm64 the platform driver cannot do the work itself. The registers that actually remove power from a core belong to firmware running at a higher privilege level than the kernel, and Linux is not allowed near them. What it gets instead is PSCI — the Power State Coordination Interface, an ARM standard that defines a handful of function numbers the kernel invokes with an smc instruction, a trap into that firmware. There is a call to bring a core online, one to take it offline, and one to suspend it. The suspend call carries a single argument, the suspend parameter, and that number is the whole of what the kernel gets to say about which state it wants.

The driver that speaks it is cpuidle-psci.

The configuration this kernel had been built with:

```
CONFIG_CPU_IDLE=y

CONFIG_CPU_IDLE_GOV_LADDER is not set

CONFIG_CPU_IDLE_GOV_MENU=y

CONFIG_CPU_IDLE_GOV_TEO is not set

ARM CPU Idle Drivers

CONFIG_ARM_PSCI_CPUIDLE is not set

end of ARM CPU Idle Drivers

`` Without a driver, cpuidle has no states to offer, and the idle loop falls through to the architecture default:wfi`, which stops the core fetching instructions until an interrupt arrives and leaves it powered throughout.

The kernel package

I had not written that configuration. postmarketOS builds one kernel package for the msm8996 family, and every device on that chip takes it — the OnePlus 3 and 3T, the Xiaomi Mi 5, Mi 5s Plus and Mi Note 2, the three Sony Xperia X Performance variants. Several of those device wikis mention lockups when CPU idle is enabled, which is presumably why it ships off.

Device tree

Sleep states on arm64 are described in the device tree, the data file that tells the kernel what hardware is present, since almost none of it can be discovered by probing. For this chip only msm8996.dtsi describes any, and none of the ten msm8996 boards in mainline overrides them, so all of those phones idle the same way:

CPU_SLEEP_0: cpu-sleep-0 { compatible = "arm,idle-state"; idle-state-name = "standalone-power-collapse"; arm,psci-suspend-param = <0x00000004>; entry-latency-us = <130>; exit-latency-us = <80>; min-residency-us = <300>; };
One state, for a core on its own. arm,psci-suspend-param is the value handed to firmware to say which state is wanted; the rest is the governor's arithmetic — 130 µs to go down, 80 µs to come back, and don't bother for an idle period shorter than 300 µs.

That entry latency is the one number here anybody ever went back and corrected. It started at 40 µs, and the correction derived it from the vendor kernel's own device tree, where a level is described not by an entry and an exit but by a wake-up latency and a total overhead:

entry-latency-us = qcom,time-overhead - qcom,latency-us = 210 - 80 = 130
That conversion is the only place mainline has ever taken a number for this chip from the kernel that ships on it.

msm8998

msm8998, the next year's Snapdragon, describes four states where msm8996 describes one — retention and power collapse for each of its two clusters, at 0x00000002 and 0x40000003. Its power-collapse states also carry a property msm8996's state does not:

local-timer-stop;
It tells the kernel that the core's own timer stops in that state, so the wake-up has to be handed over to a timer that keeps running.

The control experiment

It hangs

I flipped CONFIG_ARM_PSCI_CPUIDLE to y, changed nothing else, and loaded the kernel with fastboot boot so I wouldn't have to flash anything to try it. Nothing came up. No USB network device on the Mac, no address to reach, no way to tell whether the kernel had crashed or never got far enough to bring an interface up.

That matched what the wikis describe, so I went looking for a better description of the hardware rather than questioning the verdict.

The first thing I tried was more state, not less. I wired up genpd, the kernel's generic power-domain framework, which is how the last core going idle gets to request something bigger than a core-level state: a domain per CPU, the CPUs' domains under one per cluster, both clusters under a system domain, with cluster and system suspend parameters attached to each level. Nothing came up.

Then the parameter itself. 0x00000004 is suspect on its face, because a suspend parameter with the state-type bit clear tells firmware the core keeps its state, which is not what "power collapse" means. I set the bit and tried 0x40000004, which I believed at the time was what the vendor kernel composes for the same single-core state. Nothing came up. I added local-timer-stop on top of it, the property msm8996 is missing and every comparable SoC declares, on the theory that the core was going down with its wake-up left in a timer that stops with it. Nothing came up. I tried 0x00000002 — retention rather than power collapse, and the exact value msm8998 uses for it — on the grounds that the shallower state was the likelier one to work. Nothing came up. I tried 0x00000001, the most conservative value in the encoding, on the grounds that if anything at all was going to work it would be that. It behaved exactly like the rest.

The stock value and five variants, and every one failed the same way, which I read as consistency.

The explanations

Each failure produced an explanation, and all three were built out of something genuinely true about this chip.

The suspend parameter was one. What varies between Qualcomm SoCs of that generation is its low nibble, and everywhere else in the tree that nibble is 2 or 3 — msm8916, msm8976, msm8998, sdm630. msm8996 alone uses 4, which fits the symptom exactly: firmware refuses a state ID it does not recognise, and the core never comes back.

The interrupt controller was another. msm8996 is the only SoC in mainline carrying a quirk that disables part of the normal idle handling for it, and a core returning from a real power-down would plausibly come back with its own timer switched off.

The third was the layer underneath. The kernel reports CPU: All CPU(s) started at EL1, there is no /dev/kvm and no virtualisation support in the log, because EL2 — the exception level Linux normally occupies on arm64 — is held by Qualcomm's own hypervisor, and every PSCI call passes through it on the way to TrustZone. Mainline knows nothing about that layer, so "the hypervisor mishandles the call" explains a hang without requiring anything else to be wrong.

All three are tested properly later in this article. None of them had anything to do with what I was looking at here.

The control

The uniformity bothered me. Configurations that differ in what the firmware is being asked to do — a power collapse, a retention, a plain wait — should not fail in the same way, down to the same missing USB device.

So I stopped changing the kernel and changed the command line. I booted the same binary as the last failing test with cpuidle.off=1 added. With cpuidle disabled the kernel never enters an idle state, PSCI_CPU_SUSPEND is never called, and none of the three mechanisms above can occur. It failed exactly as before.

That disposed of all three at once, since they concerned a call that was no longer being made, and it meant every result I had was worthless: whatever was taking the device down was not the thing I was changing. The remaining suspect was the build itself, which gave a clean test. I rebuilt the exact commit the working, flashed kernel had come from, with its exact original configuration — cpuidle off, no edits of mine anywhere in it. It failed too.

The boot images

From there the device was not needed at all. The working kernel was still on disk as a boot image and so was every test build, including the one from the null experiment, and an Android boot image carries the kernel's own version string in its header:

WORKING flashed : "6.12.95-msm8996 ... #2-op3t" kernel=10484779 ramdisk=11639553 NULL EXPERIMENT : "6.12.95 ... #10-op3t" kernel=10664564 ramdisk=10428024
Same source, same configuration, and the initramfs has lost 1.2 MB. The release string is the thing to look at, though, and it is the one field every failing build shares: the working kernel calls itself 6.12.95-msm8996, and everything I had built called itself 6.12.95.

A kernel looks for its modules in /lib/modules/<release>. On the device that directory is /lib/modules/6.12.95-msm8996, so a kernel calling itself 6.12.95 looks in a directory that is not there and loads nothing: no USB gadget driver, no network over USB, no WiFi, no sshd. The kernel had booted every time and had no interface left to answer on.

The suffix comes from CONFIG_LOCALVERSION, and it was empty because the configuration file in my repository had been generated against kernel 7.2-rc1 while the tree I build is 6.12.95. make olddefconfig drops the roughly 450 symbols that do not exist in the older tree, fills in defaults for the rest, and exits 0. CONFIG_LOCALVERSION="-msm8996" went out with them.

Underneath that sat a second mistake. fastboot boot loads a kernel into RAM through the OEM bootloader, which bypasses lk2nd — and lk2nd is what applies this device's fixups before handing off, so a kernel booted that way never brings up the full system here even when it is perfectly healthy.

And a third. I had been judging each test on port 22 alone, while postmarketOS boots in two stages that answer on different ports: the initramfs brings up networking and offers a telnet shell on 23 well before there is an sshd.

Guards

Three scripts came out of this. check-kernel.sh refuses to let an image reach the device unless it matches a fingerprint of the known-good one: release string equal, initramfs size within 5 %, the configuration carrying CONFIG_LOCALVERSION="-msm8996" and no symbols from a newer kernel. Nothing on the device gets interpreted until that passes.

probe-device.sh replaced the port check:

```

So: ping + port 23 = booted, sitting in initramfs (usually rootfs trouble)

ping + port 22 = fully booted

ping, no ports = kernel alive, stalled before any service started

no ping at all = genuinely dead / no USB gadget

`` Andtelnet-device.shreads that initramfs shell non-interactively — uname, kernel command line, block devices, mounts, the modules directory,lsmod`, the tail of dmesg — which is the set of things that tells you why a boot stopped where it did.

The rule I took from this: rebuild the known-good configuration in the current environment and confirm it still works, before attributing any failure to the change under test.

With the configuration repaired and the guards in place, I built the same one-symbol change again and read the counters from the initramfs shell:

DRIVER=psci_idle GOVERNOR=menu state0 WFI usage=2942 time=2292104 us state1 cpu-sleep-0 usage=7742 time=22910277 us
The counters were moving and the kernel kept answering.

The temperature drops

Enabling the driver gave the kernel one state to enter, and it is the shallow one: 0x00000004, the state-type bit clear, meaning the core keeps its context. The obvious next step is the same state with the bit set, and firmware refuses it. Asking for 0x40000004 leaves the state's rejected counter climbing, 16797 of them, while usage stays at exactly zero. I did not yet know what that counter meant; what I could see was that the deeper standard state was not available to me.

The vendor SMC

The vendor kernel does not go deeper through PSCI either. For its default CPU power-collapse level it sets a device-tree property, qcom,hyp-psci, and instead of a suspend call it issues a fixed SMC:

__invoke_psci_fn_smc(0xC4000021, 0, 0, 0);
That function ID decodes as a vendor extension: owner field 4, which is the PSCI namespace, function number 0x21, past anything the PSCI specification allocates. It is serviced by the Qualcomm hypervisor at EL2 — the layer that had been my third explanation for the hangs, and which has an idle state of its own that the standard interface does not expose.

It takes no arguments, and it returns to its caller. That second part is what makes it usable: a PSCI suspend that really powers a core down never returns, which is why the kernel has to hand over a resume address and save everything first. This call comes back on the next instruction, so from the kernel's side there is nothing to hand over and nothing to save — whatever the core loses, the firmware restores before returning. It is called the way wfi is called, from the same place in the idle path, and the worst case if the firmware ignores it is a wasted trap rather than a core that never wakes.

So the driver learned to recognise the level, and the device tree gained a second state next to the one it already had:

CPU_SLEEP_1: cpu-sleep-1 { compatible = "qcom,idle-state-hyp-psci", "arm,idle-state"; idle-state-name = "hyp-power-collapse"; arm,psci-suspend-param = <0x00000000>; entry-latency-us = <40>; exit-latency-us = <120>; min-residency-us = <400>; };
The suspend parameter is ignored — the call takes no arguments — and is there only to satisfy the binding. The latencies come from the downstream level this mirrors, and they are what the governor chooses on.

Temperature

Both changes went onto the phone together, and this time they went on by flashing rather than by fastboot boot. A kernel loaded into RAM bypasses lk2nd and never brings the full system up on this device, so it can only be tested from the initramfs shell. To measure a server doing its actual work, the kernel has to be on the partition and the machine has to boot normally.

It did, and the recorder carried straight on across the change:

| CPU temperature, daily | mean | peak |
|---|---|---|
| before | 65.2 – 68.1 °C | 71.7 – 80.0 °C |
| after | 46.0 – 47.9 °C | 54.1 – 57.0 °C |

Two later days sit outside that second row, at peaks of 67 °C. Those are days I spent rebooting the phone into the bootloader to test the next kernel, which is the hottest state this device has, and they say nothing about the server. Otherwise nothing about the machine changed — same services, same workload, same charger.

Power

The temperature is the solid measurement here. The power figure is not, and it is worth saying why.

The fuel gauge sits in series with the battery, so it only sees what the board draws when the board is running off the battery. Plugged in, the charger feeds the system directly and the gauge never sees that current at all. A power number therefore only means anything with the cable out, and the recorder had caught one such moment before the change: -610 mA at 3806 mV, or 2.32 W.

After the change I unplugged the phone deliberately and let it run down to empty, which gives a proper discharge curve instead of a single point: an average of 366 mA across the whole run, 1.32 W at the voltages recorded along it.

One point against a full discharge is not a comparison to defend to a decimal place. What it supports is that the device now draws something like half of what it did, which is the same direction and roughly the same size as the temperature change.

Cluster and system states

The cores were as idle as they were going to get. Whatever was still drawing power sat underneath them, in the parts they share.

The levels

Everything from here on happens on one of three tiers, and it is worth having the shape in mind:

cpu0 cpu1 cpu2 cpu3 core wfi | hypervisor SMC | PSCI collapse \ / \ / L2 L2 cluster retention (gdhs) | power collapse (fpc) \ / L3 + coherent fabric system retention | power collapse
and one of two ways down:

governor -> cpuidle driver -> PSCI CPU_SUSPEND -> EL2 hypervisor -> EL3 firmware \-> vendor SMC 0xC4000021 -> EL2 hypervisor
The four cores are in two pairs, and each pair has its own L2 cache. Below the per-core states there is a level that acts on the pair: once both of its cores are asleep, the shared L2 can be put into retention, keeping its contents at reduced voltage, or powered off entirely and its contents discarded. The vendor kernel calls these l2-gdhs and l2-fpc, and has them separately for each cluster — the power cluster and the performance cluster, which differ only in their maximum frequency.

Above that there is a level that acts on both pairs at once. When every core in the machine is asleep, what the two clusters share can go down too: the L3, the coherent fabric that keeps the caches consistent, and the paths out to memory. The vendor kernel calls those system-ret and system-fpc.

It is the same PSCI call throughout, with a suspend parameter that names a higher affinity level instead of a core. Mainline has the machinery to compose one — the power-domain framework exists so that the last core going idle can make a request on behalf of a whole cluster. What mainline does not have for this chip is any of the parameters. The vendor kernel enters these states on this exact device; no mainline kernel ever has.

Every configuration I tried — flat per-core cluster states, the power-domain hierarchy, with and without the state-type bit — reset the device on the first collapse.

Post-mortem channels

To read what a dying kernel says on the way out you need a serial console, a crash dump, or at least a way to tell a panic from a reset. I had none of the three.

The debug UART is the same 1.8 V line on the USB connector that part one could not reach, and I still have no jig for it.

The kernel can be told to keep its last log in a reserved area of memory, and on this device the reserved area does not survive: the bootloader zeroes DRAM on every reset. I checked that for the region the SoC's own description reserves and for the one the board reserves, after a deliberate crash and after an ordinary systemctl reboot. /sys/fs/pstore was empty every time.

From the outside a panic and a firmware reset are the same event. The device disappears, comes back, and its logs start at boot.

The PMIC reset reason

The power-management chip keeps a register recording why the system last restarted, and it is readable over the regmap debug interface once the machine is up again:

```

grep -iE '^080a:' /sys/kernel/debug/regmap/0-00/registers

080a: 02
``0-00is the pm8994 and0x080a` is its warm-reset reason. The bits separate a software-requested reset from the PMIC's own watchdog, a hardware fault, or the power key being held:

0x01 SOFT 0x02 PS_HOLD 0x04 PMIC_WD 0x08 GP1 0x10 GP2 0x20 KPDPWR+RESIN 0x40 RESIN_N 0x80 KPDPWR_N
The cluster-state deaths read 0x02, PS_HOLD — the applications processor deliberately dropping the signal that holds the board powered. That rules out a hang, and it rules out the hypervisor or TrustZone faulting on its own. It does not separate a kernel panic calling for an emergency restart from a firmware-initiated reset, because both arrive there the same way, and it has to be read on the very next boot before another reboot overwrites it.

With no post-mortem channel, the only workable approach is to make the failure countable or non-fatal before it happens.

Reading the vendor kernel

The vendor kernel this phone shipped with enters all of these states, on this silicon, under this firmware. None of it is in mainline: it is a separate 3.18 tree with its own idle driver, lpm-levels.c, and its own device tree describing the levels, msm8996-pm.dtsi.

The arithmetic

The first thing that matters is that downstream does not carry a table of suspend parameters. It computes them, out of two macros and the level it has decided to enter:

```

define PSCI_POWER_STATE(reset) ((reset) << 30)

define PSCI_AFFINITY_LEVEL(lvl) (((lvl) & 0x3) << 24)

`` Each level in that device tree names a mode for the CPU, and, if it is a cluster level, a mode for the L2, and, if it is a system level, a mode for the system. The CPU's mode goes in the low nibble, the L2's in the second, the system's in the byte above those, the affinity level in bits 25:24, and bit 30 comes from a flag calledqcom,is-reset`.

The flag is the part I had wrong. The composed value takes is-reset from the CPU level being entered:

power_state = PSCI_POWER_STATE(cluster->cpu->levels[idx].is_reset);
and in the vendor device tree that property is set on cluster and system levels only. No CPU level carries it. Since the composed value reads the flag from the CPU level, bit 30 is never set at any depth, and what the vendor kernel actually hands to firmware is:

0x00000004 a core on its own 0x01000034 affinity 1, L2 into retention 0x01000044 affinity 1, L2 powered off 0x02002344 affinity 2, system retention 0x02003444 affinity 2, system power collapse
Mainline's value for the one state it declares is 0x00000004 — the same number. The cluster states I had been composing were 0x41000034 and 0x41000044: downstream's values with a bit set that downstream never sets.

Bit 30

That bit is not decoration on our side either. It is what mainline uses to decide whether the CPU is about to lose its context:

static inline bool psci_power_state_loses_context(u32 state) { const u32 mask = psci_has_ext_power_state() ? PSCI_1_0_EXT_POWER_STATE_TYPE_MASK : PSCI_0_2_POWER_STATE_TYPE_MASK; return state & mask; }
With the bit clear, the suspend call goes straight through, no entry point, nothing saved. With it set, the call goes through cpu_suspend() with a finisher, and the kernel saves and restores everything itself.

I had assumed that meant both of my options were broken: with the bit, firmware refuses the call; without it, firmware powers the cluster down while the kernel skips the save, and the core resumes into rubbish. The second half of that was wrong, and checking the vendor kernel's own suspend path is what showed it:

if (state_id & PSCI_POWER_STATE_BIT) return __cpu_suspend(state_id, psci_suspend_finisher); else return psci_ops.cpu_suspend(state_id, 0);
Since msm8996 never sets the bit, downstream always takes the second branch — entry point zero, no context saved — for cluster collapse as much as for a single core. The firmware preserves EL1 state itself and returns to the caller. Stock mainline already does exactly this for a parameter without the bit, so there was nothing to fix there, and the patch I had written to force a context save was removed.

The bisect

Knowing the right numbers did not make the device survive them, and there were two halves to suspect: everything the kernel does on the way into a cluster state, and the firmware call at the end of it.

Splitting them turned out to be easy, because only the last step needs to change. I left the entire kernel-side teardown running exactly as it was — the power-domain accounting, runtime PM suspending the domain, the wakeup controller being armed, the cluster PM notifiers firing — and changed one thing: the value handed to psci_cpu_suspend_enter() at the end, substituting a known-harmless per-core state for the composed cluster one.

The domain code composed 0x41000044 and ran its full teardown 1083 times in 82 seconds, and the device stayed up.

That clears the entire kernel side. It does not clear as much as I thought at the time: the safe value I substituted was 0x40000004, which this firmware refuses. So the call at the end failed on every one of those 1083 iterations, and no core ever went down. The experiment proves the teardown does not reset the box. It says nothing about whether the firmware can perform a collapse, which is what I took from it at first.

Two counters

Two of the numbers I was reading meant something other than what I thought.

The first was rejected, in the per-state cpuidle statistics. I read a rising rejected with usage stuck at zero as the timer broadcast failing — the kernel deciding it could not safely enter the state and backing out. It is the opposite. In cpuidle.c the counter is touched in exactly one place:

} else { dev->last_residency_ns = 0; dev->states_usage[index].rejected++; }
which is the branch taken when the state's enter callback returns a negative value. The state was entered, the call was made, and it came back with an error. A broadcast failure does not land here at all; it silently falls back to a shallower state and counts usage there. So those 16797 rejections were 16797 SMCs that firmware declined, which is a much more useful fact.

The second was -95. A suspend attempt carrying an entry point returned it, and I read it as firmware rejecting the resume address. It comes from arch/arm64/kernel/suspend.c:

/* * Never gets here, unless the suspend finisher fails. * Successful cpu_suspend() should return from cpu_resume(), * returning through this code path is considered an error * If the return value is set to 0 force ret = -EOPNOTSUPP * to make sure a proper error condition is propagated */ if (!ret) ret = -EOPNOTSUPP;
Linux produces that value, not firmware. It means the finisher returned instead of never coming back — that is, the firmware accepted the call, did not power the core down, and returned normally, which is correct behaviour for a state whose type bit says the core keeps running.

Accepted and refused requests

With both counters read correctly, the same set of requests says something coherent:

| request | result |
|---|---|
|
| accepted, thousands of entries, stable |
|
| returns success without powering down, reported as |
|
| device down immediately |
|
| device down immediately |

Firmware treats a core-level ID as a standby it can decline to act on, and acts on an affinity-level-1 ID with or without a resume address. Downstream's exact call shape was therefore not the missing piece: I was now sending the same values, the same way, and still losing the machine.

Wrong causes

The difference had to be somewhere other than the call.

min-child-idx

The first thing that survived was a constraint I had not implemented at all.

Our device tree gives each core two idle states, and the power-domain code lets each core choose between them independently. So the last core going idle could compose a cluster collapse while its sibling was parked inside a hypervisor SMC call — a state the firmware entered on that core's behalf, which the composing core knows nothing about.

Downstream forbids exactly this. Each of its cluster levels carries qcom,min-child-idx = <2>, which permits the cluster level only when every child core is at or below a given level, and never while one sits in the hypervisor state.

There is no mainline equivalent, but the constraint can be reproduced by hand: disable the vendor state on every core at runtime, so the only thing a core can be in is the PSCI one.

for c in /sys/devices/system/cpu/cpu[0-9]*/cpuidle/state1; do echo 1 > $c/disable; done
The device entered composed cluster states twenty times, returned from every one, stayed up for around a hundred seconds, and then died. Every attempt before that had died on its first collapse.

The budget

Twenty successful collapses followed by a death says something a death on its own does not: whatever goes wrong accumulates rather than striking on the first attempt. To measure that I gave the cluster path two knobs.

The first is a runtime gate, so the device boots with cluster states off and I turn them on from a shell — which means a fatal state kills a session I am watching rather than a boot I cannot read. The second is a budget: psci_cluster_budget, an atomic counter decremented on every composed entry, which closes the gate on its own when it reaches zero.

echo 5 > /sys/module/cpuidle_psci/parameters/psci_cluster_budget echo 1 > /sys/module/cpuidle_psci/parameters/psci_allow_cluster
With unratelimited printing around the call, a run reads:

cpu3 entering composed 0x1000034 (entry 1) cpu3 returned from 0x1000034 ret=2
A budget of one works, and the device stays up and readable afterwards. A budget of five kills it.

The GIC redistributor

The interrupt controller theory was still the best mechanism I had, and now there was a way to test it at a fixed number of collapses.

Each core has a private slice of the interrupt controller holding its own interrupt configuration — including the architected timer, which is what wakes an idle core. Mainline restores the CPU interface registers on idle exit and nothing else; the private slice is programmed only by a function that CPU hotplug calls. A core coming back from a genuine power-down would find its own timer disabled.

So I saved and restored it by hand around the composed entry: the group register, the enable set, the priorities and the configuration, with the enables restored last.

Budget of five survived — all five entries logged, all five returns, device alive and readable where every previous build had died — and then it went down a few seconds later.

That delayed death has an obvious cause once you see it: cluster PM notifiers fire only on the core that composed the state, while the power-down hits both cores of the cluster. The sibling was never restored. Moving the save and restore to the per-core notifiers, which do fire on both, made things worse, because the restore clears every enable before setting the saved ones:

writel_relaxed(~0, rbase + GICR_ICENABLER0);
On a cluster entry that is fine. On every ordinary idle exit, thousands of times a second, it briefly disabled the core's own timer interrupt.

What killed the theory outright was the vendor kernel. It has no save or restore of that block anywhere — and booting the recovery partition, which runs it, showed several hundred cluster collapses already counted on a system that had been sitting idle waiting for me. If the hardware lost that state across a collapse the vendor kernel could not work at all. The patch was treating a problem that does not exist and it went away.

The GICR_WAKER handshake

Underneath that sits a genuine documented difference, and it is msm8996-specific.

There is a register that takes a core's slice of the interrupt controller in and out of a low-power state, with a handshake: write the request, wait for the acknowledge. Downstream performs it on every idle entry and exit. On mainline, msm8996 is the only SoC in the tree with a quirk that turns the function performing it into a no-op — the commit that added that quirk reports a DB820c board rebooting when the register is touched, and attributes the restriction to the hypervisor.

That is a real difference on exactly the path in question, so I added a runtime switch to bypass the quirk and perform the handshake. On this board the write does not reboot anything and the handshake completes; the cluster collapse still killed the machine.

One cluster at a time

Downstream marks its deepest system level with a property that arms extra hardware before entering, and mainline has no equivalent hook. If the failure needed that hook, it would show up only when both clusters happened to be down at once, which nothing in the budget experiments prevented.

So I added a switch permitting at most one cluster in a composed state at a time and ran the staircase again, and it reached the same count and the same reset.

The Array Power Mux

Then a mechanism that fit better than anything before it.

Downstream carries a driver mainline has for no SoC at all: an Array Power Mux, which moves the CPU and L2 memory arrays between two supply rails. It switches them whenever the CPU voltage setpoint crosses a threshold, because below that voltage the arrays can no longer hold their contents on the main rail.

Cluster retention is precisely the state where the L2 has to keep its contents at reduced voltage. With the arrays left on the main rail and no driver to move them, whether a collapse survives would depend on the voltage in force at that instant — which is to say on the operating point. It even explains the probabilistic shape: one collapse usually fine, five usually not.

Better still, it suggested a test that needed no rebuild. Pin the CPU at its highest operating point, so the voltage stays above the threshold, and run the staircase.

Pinned frequencies

| pinned frequency | result |
|---|---|
| max (2188800 / 2342400) | 1120 collapses, no failures |
| 652800 | dies |
| 307200 (lowest) | dies within 8 s |

That is the largest effect in the entire investigation. Every build until then had died at twenty collapses regardless of what I changed; at the top frequency the same build did eleven hundred.

It is also not the Array Power Mux. Our device tree has no opp-microvolt and no cpu-supply, and mainline has no regulator driver for this SoC's CPU rails at all — it never changes the CPU voltage on this chip, at any frequency. The theory that produced the test cannot explain the test's result, because the mechanism it acts through is not present on our side.

Clocks and cpufreq

Two smaller candidates for the frequency effect went the same way.

The CPU clock driver switches its output path below 600 MHz, taking the clock off a divided leg instead of the PLL directly — a real mode change, on the boundary between a working and a failing frequency. Except that 652800 is above that boundary and 652800 dies.

Frequency transitions themselves were the other: a collapse racing a voltage or clock change would be intermittent in exactly this way. So I pinned minimum and maximum to the same value, 1593600, which leaves cpufreq nothing to do, and it still died.

Timing

What was left was the least specific explanation and the hardest to disprove: that nothing is missing at all, and some sequence in the collapse and restore path finishes in time when the cores are fast and does not when they are slow.

It could at least be tested without collapsing anything, which meant the device would survive being asked. I dumped the whole clock tree at the working frequency and at the failing one and joined the two by name:

| clock | max (works) | 1593600 (dies) |
|---|---|---|
|
| 2188.8 MHz | 1593.6 MHz |
|
| 2342.4 | 1593.6 |
|
| 1593.6 | 1382.4 |
| alternate PLLs, | unchanged | unchanged |

No mode change anywhere. The multiplexers stay on the same leg at both settings, the fabric clock does not cross its own threshold, the alternate PLLs and the auxiliary clock do not move. Only rates change, all in the same direction.

So the variable really was speed rather than configuration, and the question was not what mainline fails to program but what does not finish in time. Reading more source was not going to answer that. What would answer it was a machine that does this correctly, and I had one.

TWRP

The recovery partition had already answered one question in passing. This time I went back to it deliberately, with a list.

TWRP is a replacement recovery image — a small Android system used for flashing and repair — and it runs the vendor kernel. Same silicon, same firmware, same hypervisor, a kernel that does all of this correctly, and root over adb. I had been reading it as source. It is also a running system whose idle driver keeps counters.

lpm_stats

adb shell mount -t debugfs none /sys/kernel/debug adb shell cat /sys/kernel/debug/lpm_stats/stats
After a short while in recovery — a system doing nothing but waiting for me to press something:

| level | performance cluster | power cluster | total |
|---|---|---|---|
|
| 562 | 5 | 567 |
|
| 84 | 50 | 134 |
|
| — | — | 1 / 2 |
| per-core | 7522 / 3012 | 1447 / 958 | 12939 |
| per-core | 365 / 206 | 650 / 63 | 1284 |

Five hundred and sixty-seven cluster retentions and a hundred and thirty-four full L2 power-downs, on this device, on this firmware, while I watched. Whatever was wrong was on my side, not in the silicon, TrustZone or the hypervisor.

The bottom two rows also settle something I had assumed. The two per-core states coexist perfectly happily: the hypervisor level is entered about ten times as often as the PSCI one, and min-child-idx does not forbid that — it only blocks a cluster level while a core sits in the wrong state. Downstream reaches its cluster levels through the PSCI state while spending most of its entries in the other one.

Pinned frequencies, on a kernel that works

The measurement I came for was the frequency dependence, because that was the last live theory and it needed a machine that works to test against. The vendor kernel exposes the older cpufreq layout, so pinning is the same operation under a different path:

adb shell 'for c in /sys/devices/system/cpu/cpu[0-9]/cpufreq; do cat $c/scaling_max_freq > $c/scaling_min_freq; done'
Read the counters, wait, read them again. Against my own kernel's results at the same operating points:

| pinned frequency | vendor kernel | mine |
|---|---|---|
| 307200 (lowest) | +1193 gdhs, +1111 fpc in 30 s | dies |
| 1593600 | ~650 collapses in 25 s | dies |
| max | +251/+535 perf, +403/+665 pwr, | 1120 collapses, alive |

Over two thousand cluster collapses in half a minute at the lowest operating point, half of them with the L2 losing power entirely, at the frequency where my kernel died in eight seconds.

The last row has a number I had not expected. system-ret — both clusters down together, the state I had been treating as an exotic end goal — is entered by the stock kernel more than twenty times a second as a matter of routine.

The timing theory died here. If a sequence in the collapse path were failing to complete at lower clock rates, the vendor kernel would fail worst exactly where it is slowest, and instead it does the most collapses at the lowest frequency.

Clocks and rails

Two more readings while I had a working phone in front of me.

The clock tree, pinned at the same frequencies, compared against my own kernel's: at 1593600 the cluster clock topology is identical on both, rate for rate. At the lowest operating point the vendor kernel runs the cache-coherent fabric clock at 192 MHz — seven times slower than the rate my kernel had while failing — and collapses clusters constantly at that speed.

The CPU rails, from the regulator sysfs, at the three pinned points:

307200 1593600 max kryo0 798681 → 798681 → 798681 µV kryo1 745089 → 709361 → 834409 µV kryo0-retention 490527 → 490527 → 490527 µV kryo1-retention 490527 → 490527 → 490527 µV
The first two lines close the voltage question. One rail does not move at all and the other spans about 125 mV, non-monotonically, so "mainline wastes power by not scaling CPU voltage" was never true either.

The last two are a genuine gap, unrelated to the crash. Stock programs an explicit retention voltage of about 0.49 V for each CPU rail — the level the rail drops to while the cluster is in retention, which is exactly the state I was trying to enter. Mainline has no CPU rail driver for this SoC, so nothing on my side programs it and the hardware keeps whatever the bootloader left. That matters for how much these states actually save. It is not what was killing the machine, because the machine died at every depth, including the ones that remove power rather than retain it.

Three theories

The hardware does cluster collapse, at every operating point, with the fabric running slower than mine ever was. The Array Power Mux, the clock path and the timing margin all needed the chip to be the thing that could not do this, and it can.

So the frequency dependence I had measured was not a property of the chip. It was a property of my kernel, and I had been explaining it with the chip.

Residency

If the frequency dependence was mine, it was in something I had changed. So I went back through my own working tree looking for anything that could couple the length of an idle period to whether a collapse survives, and found it in the device tree, in numbers I had stopped thinking of as numbers.

Chasing an unrelated problem earlier — the deep state was being skipped because of a latency constraint, and I wanted to force the governor into it — I had cut the timings on both CPU idle states. Entry and exit latencies down to 20 and 25 µs, minimum residencies down to 150 and 250. It worked for that purpose, I moved on, and I never put them back. They were never committed either, which is why they survived so long: they lived in the working tree that every test kernel was built from, and no diff I read ever showed them to me.

The governor's arithmetic

Those three numbers are the entire basis on which the kernel picks a state. Entry and exit latency are what the transition costs. Minimum residency is the instruction not to enter this state unless the core is expected to stay idle at least that long, because below that the transition costs more than the state saves.

The part that makes this fatal here is how a cluster state is composed. The governor never picks a cluster level; it picks a CPU level, and the last core out adds the cluster's parameter to whatever the governor already chose. So the CPU state's minimum residency is the threshold that decides whether a full cluster collapse happens at all.

I had set that threshold to 250 µs. The value it should have carried is 800 µs, and the cluster operation it lets through is one the vendor kernel will not enter for anything shorter than 5000 µs — because that is what it costs to take both cores down, flush the shared cache, drop its power, and put all of it back. The governor was committing the pair to that for idle periods a twentieth of the length it needs.

This is the failure the msm8998 patch describes, in almost the same words: timings written for the wrong thing, a state entered too eagerly, and an SoC that reboots with nothing in the logs. I had read that patch and quoted it to myself as a reason to be careful, then produced my own version of it and explained it with interrupt controllers and power rails.

Restored

CPU_SLEEP_0 (hypervisor SMC) 40 / 80 / 300 CPU_SLEEP_1 (PSCI collapse) 80 / 130 / 800 CLUSTER_SLEEP_0 (L2 retention) 90 / 180 / 1000 CLUSTER_SLEEP_1 (L2 power off) 700 /1000 / 5000 entry / exit / min-residency, µs
One loose end in that table, which I would rather state than quietly ship: the entry and exit figures on the PSCI state are the wrong way round against the conversion rule from earlier — the vendor level's 80 µs is the wake-up cost, so it belongs in exit-latency-us, and 210 − 80 = 130 belongs in entry-latency-us, which is exactly what mainline had before I touched it. Their sum is what the governor's cost model uses and the residency is what gated this failure, so it did not change the result below; it is still wrong and still mine to fix.

Every operating point

The same experiment that had produced the frequency table, at full depth, with the budget effectively unlimited: pin the frequency, let it collapse clusters, step down, repeat.

| pinned frequency | result | cumulative entries |
|---|---|---|
| max (2188800 / 2342400) | alive | 4471 |
| 1593600 | alive | 11372 |
| 1132800 | alive | 20575 |
| 844800 | alive | 27047 |
| 614400 | alive | 33494 |
| 307200 (lowest) | alive | 39149 |

Thirty-nine thousand full cluster power collapses, down to the lowest operating point the chip has. Every earlier build had died at 1593600 without exception, most of them inside twenty collapses.

The frequency dependence explains itself once the residency is right. At the top of the range the machine finishes its work quickly, so the gaps between wake-ups are long, and even a nonsense threshold is satisfied by a genuinely long idle. Step the frequency down, the same work takes longer, the gaps get shorter, and a threshold set twenty times too low starts admitting collapses that cannot complete. Frequency was never the variable. It was a proxy for how long the cores were idle.

The earlier values

The values that shipped before I touched anything were also sane, and the device died then too. What is true is that in that earlier era it was carrying several other things at once: the power-domain hierarchy on the idle path, the wakeup controller parented above the clusters, suspend parameters with bit 30 set, and a forced context save. Those were removed one at a time over the preceding sections, each on its own evidence, and none of them alone made the device survive. The residency was the last one, and the one I had inflicted myself.

The rule I took from this: change one thing at a time, and control for the things you changed yourself. I had a working tree carrying an unrelated experiment, and I was measuring the symptoms it produced.

The driver

With the timings fixed, cluster collapse worked through the power-domain path. It also had that path's overheads, and by then I had read enough of the vendor driver to know how little it actually needs.

Why not genpd

The generic power domain framework is mainline's answer to the last-man-out problem, and on hardware built around it, it is the right answer. Each core gets a domain, domains nest, a core going idle suspends its domain, and when the last one leaves, the framework composes the parent's state and hands it down.

The cost is what runs on the way there. Suspending a domain goes through runtime PM, which walks the hierarchy, takes the framework's locks, and calls into per-domain callbacks — all of it on the idle path, with interrupts off, on a core that is about to stop. It is a lot of machinery to answer one question: is this the last core awake in this cluster?

The vendor kernel answers it with a spinlock and two cpumasks per cluster, and that is what I ended up writing: drivers/cpuidle/cpuidle-msm8996.c, about 260 lines, no genpd anywhere near the idle path.

Last man out

Each cluster has a mask of the cores that belong to it and a mask of the cores currently inside an idle state. A core marks itself on the way in; if the two masks are then equal it is the last one out, and it composes the cluster's suspend parameter onto its own. If every cluster is idle, it composes the system's instead:

raw_spin_lock(&msm8996_topo_lock); cpumask_set_cpu(dev->cpu, &cluster->in_sync); if (msm8996_cluster_depth && cpumask_equal(&cluster->in_sync, &cluster->child_cpus) && !msm8996_ipi_pending(&cluster->child_cpus)) { if (msm8996_system_depth && msm8996_all_clusters_idle()) { state |= (msm8996_system_depth >= 2) ? msm8996_system_fpc : msm8996_system_ret; system = true; } else { state |= (msm8996_cluster_depth >= 2) ? msm8996_cluster_pc : msm8996_cluster_gdhs; } composed = true; } raw_spin_unlock(&msm8996_topo_lock); if (composed) cpu_cluster_pm_enter(); ret = psci_cpu_suspend_enter(state) ? -1 : idx; if (composed) cpu_cluster_pm_exit();
One lock for the whole topology rather than one per cluster, because deciding that every cluster is idle means reading all of them and that has to be a single consistent view. The vendor kernel gets the same effect by recursing into the parent's lock.

msm8996_ipi_pending is a guard I ported faithfully from downstream: a per-core flag set when an inter-processor interrupt is raised for a core and cleared when that core takes it, so a cluster is never collapsed with one in flight. Keeping that flag means a twenty-line patch to arch/arm64/kernel/smp.c, which is a lot of generic architecture code to touch on behalf of one SoC. It is in there because downstream has it, it has never once changed a measured outcome on this device, and it is the piece of this work least likely to survive contact with a reviewer.

Cluster membership comes from the affinity field of each core's identity register rather than from anything in the device tree, and the driver prints what it found at attach:

cpuidle-msm8996: cluster 0: cpus 0-1 cpuidle-msm8996: cluster 1: cpus 2-3
which matches downstream's split into a power cluster and a performance cluster.

Attaching it takes three lines in the generic PSCI driver — when the per-core setup finds no power domain to attach to, hand the deepest state to this driver instead:

if (!data->dev) msm8996_cpuidle_attach(drv, state_count, psci_states[state_count - 1]);
The diagnostics I had added to that file are gone. What remains beside those three lines is the work from earlier sections: the vendor SMC state and its match entry, the cluster PM notifiers around a domain entry, and the change that lets the power-domain path attach on a device tree that declares domains without the firmware advertising the mode for them.

The device tree it ended up with

The states themselves moved around. The hypervisor level became the first of the two per-core states rather than the last, because the domain code overrides the enter callback of the deepest state, and with the vendor SMC sitting there that override replaced a call taking no arguments with a standard suspend carrying the placeholder parameter — which firmware refused, hundreds of thousands of times. The PSCI state went last and got local-timer-stop back, the property I had tried as a guess in the very first round of experiments: once the state really is a power-down composed with a cluster level, the core's timer really does stop, and the kernel really does need to know. And the cluster and system parameters moved out of the per-core list into a node of their own, since no core selects them directly:

domain-idle-states { CLUSTER_SLEEP_0: cluster-sleep-0 { arm,psci-suspend-param = <0x01000034>; /* L2 retention */ }; CLUSTER_SLEEP_1: cluster-sleep-1 { arm,psci-suspend-param = <0x01000044>; /* L2 powered off */ }; };

The ladder

Each level went in from a live shell, one at a time, with the tiers off at boot:

• cluster power collapse, 0x01000044, the shared L2 losing power: 3690 entries, alive
• system retention, 0x02002344, both clusters down and the L3 in retention: 712 entries, alive
• system power collapse, 0x02003444: 2079 entries in ten seconds, alive

That last one is the deepest state the vendor kernel has, and mainline msm8996 now reaches every rung of the same ladder: core, cluster retention, cluster power collapse, system retention, system power collapse.

The deepest of those levels also works without something downstream considers mandatory for it. System power collapse is the one level the vendor device tree marks qcom,notify-rpm: before entering it, the vendor kernel tells the resource-power manager that the applications side is going down and arms the always-on wakeup controller with the next timer expiry, on the reasoning that with everything else powered off that controller is what is left to wake the chip. System retention carries no such flag and needs no handshake on either kernel. Mainline has neither hook available: its wakeup-controller driver only exposes itself as a power domain, and there is no global "entering sleep" call to the resource manager at all. I entered the state without either, and the machine keeps coming back.

That demonstrates that on this firmware something below us arranges the wakeup for an idle-driven entry. It does not demonstrate that the handshake is unnecessary in general, and it is the first thing I would revisit if a longer or deeper sleep ever misbehaved.

An early build with both tiers enabled from boot never came up, and I recorded that as a safety property: the tiers must default off and be switched on after boot. Reproducing it refuted it — a kernel booted with both tiers on the command line comes up normally and survives a soak of fifty thousand cluster and thirteen thousand system entries. The build that died had been sitting in the bootloader beforehand, which is the hottest state this phone has, and it vanished from the USB bus entirely, which is how a power-off looks rather than how a hang looks. Almost certainly thermal, and not something I can prove after the fact.

Counters on the live server

The kernel that runs the server now has both tiers on by default. Reading its counters off the live machine:

cluster_entries 2150931 system_entries 3534083 cpu0 state2 usage=2633034 time=38688859635 us
Two million cluster collapses and three and a half million entries with both clusters down, and cpu0 spent 38 688 of the machine's 40 800 seconds in the deepest state it has — about 95 % residency.

The temperature moved again, by less than the first time and by more than I expected. Per-core idle alone had settled the daily average at 46.0 to 47.9 °C. With the cluster and system levels running it is 43.3 °C, and the coolest samples touch 40.3 — a floor around three degrees below anything the phone reached before, on the same shelf, in the same room, doing the same work.

Three degrees is the honest size of what the shared blocks were costing: the L2s, the L3 and the coherent fabric, sitting powered while four idle cores had nothing to ask of them. Measuring it in watts rather than degrees needs the charger physically out, because of where the fuel gauge sits, and the only unplugged run I have was taken before this work — so the number I can give is the temperature, and the direction is the one the design predicts.

The idle path on this SoC now does everything the vendor kernel's does, through the standard interface, and can be measured doing it.

What would have to change

None of this has been sent anywhere, and in its current shape none of it should be. That is not a verdict on the idea — it is that almost everything here was built to be tested, and the things that make a change easy to test are mostly the things that make it wrong to publish.

It is worth separating the two, because they get confused easily: what is prototype scaffolding, and what is genuinely unknown.

Scaffolding

The driver has module parameters for cluster depth and system depth, readable counters for how many times each level was entered, and a runtime gate. All of that exists because the failure mode was a phone that disappears off the USB bus. I needed to bring one level up at a time, from a shell, on a machine I was already watching, and to know afterwards how many times it had actually done the thing. None of it belongs in a finished driver: the available depths are a property of the SoC and belong in the device tree, and the counters are what /sys/devices/system/cpu/*/cpuidle/ already provides.

The vendor SMC lives in cpuidle-psci.c because that is where the table matching idle-state compatibles already is, and adding a row to an existing table was the shortest path to a testable kernel. A magic number and a vendor compatible string sitting in a generic driver is not where that belongs.

The flag tracking an interrupt raised but not yet taken is a faithful port of what downstream does, added because reproducing downstream exactly was the fastest way to rule it out as the difference. It costs a patch to arch/arm64/kernel/smp.c, which is a lot of generic code to touch on behalf of one SoC, and it does not need to be written that way: GICv3 already records pending inter-processor interrupts in a redistributor register a core can read about itself, so a core could simply decline to mark itself ready and let the last-man test come up short. That is a smaller and more local change than the one I made.

And the entry and exit latencies on the PSCI state are the wrong way round, which is a plain mistake rather than a design choice.

Genuinely open

The interesting question is the driver's existence.

The generic power-domain path is how mainline expects an SoC to do last-man-out accounting, and writing a second implementation for one chip needs a better justification than I currently have. What I have is historical: when I tried that path it failed. But at that point it was carrying suspend parameters with the power-down bit set, the wakeup controller parented above the clusters, a forced context save, and idle-state timings I had wrecked myself — and every one of those turned out to be a real defect on its own. Any of them could have been the reason it failed, and I never went back and checked once they were fixed.

So the experiment that decides the shape of all of this has not been run: put the corrected device tree back under the power-domain path and see what happens. If it works, most of what I wrote stops being necessary and this reduces to a device-tree change. If it does not, I get an isolated reason, which is a much stronger argument than a private driver — and quite possibly a fix belonging in the generic code rather than beside it.

Two smaller things are open in the same way. Whether the interrupt guard is needed at all is unknown, because it has never changed a measured outcome here and I have not tried removing it. And what the cluster and system levels save in watts is unmeasured; three degrees is real, but degrees are not what a power-management patch is judged on.

The part that stands

Underneath the scaffolding there is one change I would defend as it is: the suspend parameters and the idle states in the device tree.

They are wrong in mainline today in a way anyone can reproduce in a few minutes — enable the driver, watch rejected climb while usage stays at zero, drop one bit, watch it work. The levels they add are levels the hardware has and mainline has never described. And their timings are derived by the same conversion from the vendor tree that a Linaro maintainer already applied to this exact file, which makes it an established practice for this SoC rather than a private convention.

That is the piece worth preparing properly, and the order of work is clear enough: measure the power, run the power-domain experiment, remove what the answer makes unnecessary, and then see what is left to send.

In the end

The phone did its job the whole time. Every service stayed up, nothing crashed, no log ever said anything, and it was spending most of its power on nothing at all. What caused that was not a bug anyone introduced: a driver switched off deliberately for an entire SoC family because it was known to lock devices up, and a set of idle-state parameters derived on paper and never run on the hardware they describe. Both are ordinary conditions for an old device on a mainline kernel.

A port that boots, brings up WiFi and stays online is not a finished port. It is one whose remaining defects are all in places nothing reports on, and on this device those places were worth twenty-five degrees and something close to half the power draw. Measuring them took a shell script and a timer.

The kernel work is on a branch at github.com/arttttt/linux-op3t, and the tooling around it — the sampler, the viewer, the build and flash scripts — at github.com/arttttt/oneplus3t-pmos-server.