The Job That Ate Itself Every Minute

A silent timeout on shared hosting hid a near-cubic loop inside a backtesting engine. Fixing it took an algorithm rewrite, and a second implementation to prove the trades hadn't changed.

One morning my signals optimiser stopped returning results. Nothing had errored. The application log was clean. Jobs simply sat in the running state and never left it, and every sixty seconds, like clockwork, a fresh one began. The optimiser had become a zombie: dying, resurrecting, and dying again, burning CPU on a treadmill of failure that produced no output and raised no alarm.

I build and run Pipnotic, a trading-analytics platform, alone, PHP and MySQL, on shared hosting. The optimiser at the centre of this story runs a grid search over strategy parameters. For BASZ, a supply-and-demand zone strategy, that means thousands of parameter combinations, each requiring a full backtest over historical price bars. When it works, it produces a leaderboard of configurations. When it silently eats itself every minute, it produces nothing but a warm CPU.

Two separate failures were compounding. One was an infrastructure trap that hid the problem. The other was an algorithm that deserved to be caught. Neither was visible from the log.

The trap: fatal errors walk straight past your catch block

Shared hosts impose a hard ceiling on execution time, and they lock it down. A set_time_limit(0) inside your script does not override a limit the host has fixed above you; the script quietly runs on borrowed time until the ceiling arrives.

What happens then is the crux of the whole failure. When PHP hits that ceiling, it terminates with a fatal error, not an exception. A fatal error is not a throw. Control does not unwind back up the call stack looking for a handler, the script is torn down where it stands. So a try/catch never sees it. Neither does finally. Even catch (\Throwable $e), which in modern PHP catches a great many former fatals, does not catch this one: the execution-time timeout remains a genuine, uncatchable fatal. The cleanup I had carefully placed to flip the job row from running to failed simply never ran.

From there the loop writes itself. The job row stays running. The cron, seeing an unfinished job, assumes there is still work to do and launches another. That one times out too. And so on, once a minute, indefinitely, no crash report, no alert, just CPU quietly spent on an infinite loop of failure.

The robust fix does not depend on catching the timeout, because you often can't. It depends on keeping the recovery logic outside the process that dies:

• A shutdown handler catches the deaths PHP controls. register_shutdown_function does fire after PHP's own fatal errors, including the timeout, so a shutdown handler can mark the row failed on the way down. This is a good first line of defence.
• An attempts counter catches the deaths it doesn't. If the host kills the process from outside, a signal from a process reaper, a cgroup CPU or memory kill, PHP runs nothing at all on the way out, shutdown handlers included. An attempts counter on the job row, incremented before work begins and checked by the cron, caps the resurrections at a fixed number no matter how the process died.

It is worth knowing which layer is killing you, because it isn't always the one you assume. On Linux, PHP's own max_execution_time has historically measured CPU time, not wall-clock time, and it does not count time spent blocked in system calls, including waiting on MySQL. A query-heavy backtest can burn wall-clock minutes without accumulating much PHP-measured CPU time, which means the executioner may well be a different layer entirely, a FastCGI request_terminate_timeout, a web-server timeout, a host reaper, measuring wall time that PHP's own timer never sees. The attempts counter is the defence that holds regardless, because it lives in the database, not in the doomed process.

The timeout was the symptom, not the disease

Once I understood the jobs were timing out rather than erroring, the real question surfaced: why does a backtest that should take seconds take minutes?

The hot path was a function called precomputeAllZones(). BASZ works by finding "apex" points in price, building a zone around each one that meets certain criteria, and then tracking how subsequent bars consume that zone. The original code was written from the point of view of the current bar:

for each bar (currentIdx):
    for each apex within LOOKBACK of currentIdx:
        re-detect the zone geometry from scratch
        re-scan forward to measure how much has been consumed

The bug hides in plain sight. A zone's geometry is fixed the moment the zone is detected, it never changes afterward. Yet here it was recomputed on every bar. Worse, the forward consumption scan restarted from the zone's origin each time. A zone that lived for 500 bars had its geometry rebuilt 500 times and its consumption rescanned from scratch 500 times.

The cost of that is easy to underestimate. Take a single zone that stays alive for m bars. Each bar it is alive, the code rescans it from the origin up to the present, so the scan lengthens as the zone ages, running over roughly 1, then 2, then 3, up to m bars. Summed, that is work on the order of m²/2. One zone, quadratic in its own lifetime. Now sum that across every zone in the series, and multiply by thousands of grid combinations. The exact exponent depends on how the number and lifetime of zones scale with the length of the series, I'd call it cubic-ish in practice, but the precise power is beside the point. The useful fact is simpler: the work grew faster than linearly, and it did so by redoing, on every bar, work that had already been correct one bar earlier.

The uncomfortable part is that nothing looked wrong. Each individual computation was cheap. The code was clean and readable. This is the classic backtesting trap, writing from the perspective of "what does the world look like at bar N?" quietly re-derives everything from bars 0 through N−1 on every single step.

The fix: detect once, then move a cursor

The rewrite splits the work into two passes.

Pass 1, detect once. Walk the bars a single time. When an apex confirms, compute the zone's geometry there and then, and store it. Because the geometry is fixed at detection, computing it more than once is pure waste.

Pass 2, advance, don't rescan. Give each zone one small piece of state: a cursor marking how far its consumption scan has reached. When the backtest steps from bar k to bar k+1, each active zone advances its cursor by exactly one bar. No zone ever re-examines a bar it has already seen.

The saving is best understood as amortisation. In the naive version, a zone alive for m bars did O(m²) work, because it rescanned its lengthening history every bar. In the incremental version, that same zone's cursor moves forward a total of m times over its entire life, O(m), not O(m²). Each (zone, bar) pair is now touched exactly once. Across the whole series the total work falls to roughly O(n × A), where n is the number of bars and A is the average number of zones active at any one time, with constant work per step.

The 70× is not the point yet

The result was a roughly 70× speedup. The grid search that used to blow through the execution ceiling now finishes with room to spare.

And a 70× speedup would have been worthless if the fast version produced different trades.

This is the part that actually mattered. Backtest output feeds my leaderboard, and the leaderboard informs decisions people make with real money. Eyeballing charts and deciding it "looks right" is not a standard of proof. An optimisation that changes the answers is not an optimisation at all; it is a new, untested strategy wearing the old one's name.

Proving the answers didn't change

So before deploying, I wrote a third implementation of the algorithm, deliberately naive, deliberately slow, in Python, structurally unlike either PHP version. Then I ran the old logic and the new logic across 63-plus parameter configurations and compared every detected zone and every signal.

Zero mismatches.

The reason a different implementation is worth the effort comes down to how errors propagate. A second copy written in the same style, by the same hands, tends to reproduce the same mistakes, the shared assumption, the shared off-by-one. An independent implementation in another language, written the dumb, obvious way, is unlikely to share the fast version's bugs by coincidence. When two such implementations agree across dozens of configurations, that agreement is real evidence they compute the same function, rather than evidence they share the same blind spot.

I should be precise about what this establishes: empirical equivalence over the configurations tested, not a theorem for all possible inputs. Sixty-three configurations are a sample, not the population. But a broad differential test across the parameter space is a different order of confidence from looking at charts, and it caught nothing, which was exactly the outcome I needed. Writing that reference implementation took longer than writing the fix. It was the cheapest insurance I bought all week.

Three things worth carrying elsewhere

On shared hosting, assume cleanup won't run. Any job-state machine that only leaves running from inside a try/catch, or even a finally, will wedge the first time the process dies without unwinding, and on shared hosting it will. Put the recovery outside the process: a shutdown handler for the deaths PHP controls, an attempts counter for the ones it doesn't.

Per-bar loops hide superlinear cost beautifully. If anything inside your main bar loop "looks back" or "scans forward," ask whether it is recomputing something that was already true one bar ago. The answer is usually yes. The remedy is almost always the same shape: compute the fixed part once, then carry a cursor and advance it by one.

A speedup is a claim about performance, not correctness, verify the two separately. The cheapest way to trust an optimisation is an independent implementation and a differential test across many inputs. If the fast version is fast and wrong, all you have done is learn to be wrong more quickly.

The zombie job looked like an infrastructure problem, and in the narrow sense it was: fatals bypass catch. But the timeout was only the alarm. The fire underneath it was an inner loop redoing, on every bar, work that had been finished on the bar before, operations cheap enough to look harmless until there were millions of them. The fix was less cleverness than bookkeeping: detect once, remember where you are, move forward by one. And the number I'd underline isn't the 70×. It's the zero, zero mismatches, because in a trading system the figure that matters is never how fast the backtest runs. It's whether the trades come out the same.

I've built Pipnotic myself since 2011. If you've hit the same walls, PHP on constrained hosting, backtest engines, grid searches that melt under their own weight, I'd genuinely like to compare notes. And if you trade and want to see what BASZ and the other strategies actually produce, the platform is at pipnotic.com.

— Sarid Harper, founder of Pipnotic