Ten percent came back

September 4, 2026

This week I said I was going to test whether the operating rules of the company I run could be found by analysis instead of by memory, and that I would post the result whatever it was.

The number is ten percent. Sixty three hard operating constraints in a frozen answer key, the ones that make a trip invalid, unsafe or non-compliant if broken, and six recovered. Against the rules I could write down from memory alone, three of thirty five.

I said in advance what that would mean. If very few came back, what I have is judgment rather than a method. By my own stated criterion, that is the answer. Gear rules, nothing out of thirteen. Timing, nothing out of seven. Sequencing, nothing out of three. Not near misses. Whole categories the transaction data cannot express, because a rule somebody decided leaves no trace in a record of what happened.

The analysis is four probes over an export of the raw transaction records, with the rule tables on a denylist the loader refuses to read. One looks for the same resource committed twice and for the gaps between commitments. One looks for ceilings that are never exceeded. One looks for fields populated for some kinds of work and not others. One reads the log of manual overrides. Every candidate was scored by hand, one at a time, against a key that was frozen and hashed before any of it ran.

I know what follows reads like finding a consolation prize in a failed test. Here is why I do not think it is one.

The part I did not predict

The same run surfaced eighteen constraints that two years of building never wrote down. Not variants. New ones.

A bus can serve three trips in a day when the pickups line up. A guide driving their own trip’s shuttle is not a double booking. Certain launch days are exempt from a seat count because the guests shuttle themselves. The people who drive our buses do not use any of our systems, so every one of them has to be confirmed by text.

Fourteen of the eighteen came from one place: the log of times a person overruled the system.

I ran it again on a thinner export, the kind a normal business would produce, with the override log removed. Recall fell from ten percent to two. Findings fell from eighteen to three.

The thin run still had a relational database and thousands of assignments. So the claim is not that you need a database. It is that you need a record of the moments a person overruled the system.

Which creates an obvious problem. An override log only exists once there is something to override, and a business running on a whiteboard and a booking tool produces none. So the finding is circular for exactly the businesses that need it most.

So I went looking in the group chat

Every business has one. I mined nineteen days of ours from the 2025 season, on GroupMe: 235 messages, 170 operational. I wrote the prediction first, ten high confidence decision events per hundred operational messages, and below five it is not worth doing.

It came back at 5.3. Half what I predicted, and half a point above my own walk-away line. It also produced events without the structure that made them analysable. The analysis needs four things: which entity, which rule, who decided, and how long before the job. Chat gave the first two unreliably and the last two mostly not at all.

The obvious explanation was the tool. GroupMe has no threads and no channels, and the analysis specifically named the inability to order overlapping crises as a defeater. We moved to Slack for 2026. So I ran the same nineteen days of this season, all thirty two channels, twice: once on human messages only, once with every integration included.

I predicted fifteen events per hundred on the human pass and twenty five with the automation. I predicted the entity would be recoverable a fifth of the time.

Almost every prediction was wrong, in both directions

GroupMe 2025 from experiments/GROUPME-RESULT.md. Slack, humans from experiments/slack/pass_H/metrics_H.json. Slack, everything from experiments/slack/pass_A/metrics_A.json.

The density did not move. 5.3 to 5.7 across two platforms, two seasons, and a threefold change in message volume. I predicted fifteen and got 5.7. Whatever governs how often a person overrules a system, it is not the chat tool.

The fields moved enormously. Entity from unreliable to 96 percent against a prediction of 20. Lead time from rarely to 84.

So the same decisions are happening at the same rate, and the difference is entirely in how much of each one you can read afterwards.

And the cause is not the tool

This is the part worth the whole exercise.

The natural explanation is channel structure: a per-trip channel tells you which trip without anyone saying so. That is true where it applies, and it applied to twelve of forty nine events. Thirty three of the other thirty five were recoverable because of a writing habit.

People here open a message by naming the trip and the date. “Main Salmon launch logistics, week of July 21.” “2pm tomorrow updated from 4 people to 18.” “Friday, our most short staffed day.”

That habit, plus timestamps, is what took entity from unreadable to nearly free. Not the software. The most valuable instrumentation in the building turned out to be a sentence convention nobody was ever taught.

If you want your operation to be legible later, the cheapest thing you can do is start the message with what it is about. That costs nothing, works in any tool, and it moved three of my four fields.

Then I added the machines, and it got worse

The second Slack pass included every integration: the dispatch board, the booking feed, manifests, balance reports, forms, the relay that mirrors guides’ texts into a channel.

It added five hundred operational messages and five high confidence events. On the measure I pre-registered, density fell from 5.7 to 3.5.

Two things in that are worth more than the number.

The rule engine contributed almost nothing. Of fourteen new constraints, two came from the system that exists to enforce constraints. The other twelve came from integrations carrying human input: a text message relayed from a field phone, a vehicle issue form, a balance report, a manifest change note. The machine was not the source. The machine was the courier.

And automation made the decider harder to see. Actor recoverability fell from 84 percent to 76, and to 56 on the automation-only events. The shape is always the same: an integration delivers a question into the channel, and the answer goes back out through a phone, a form, or a face to face conversation the channel never sees. A form shows who logged the fault, not who decided to keep driving. A balance report shows the debt, not who chose to run the trip anyway.

Which produces a rule I did not expect to write down: the more you automate the asking, the less you capture the answering, unless the answer comes back through the same channel. Automation surfaced more open questions than it closed. Unresolved cases went from eight to twelve, and five of the twelve exist only because a machine asked something nobody answered in writing.

What the two sources actually contain

The override log gave scheduling, capacity, credentials, vehicles.

The chat gave trash disposal, propane storage, garbage bags, store coverage, never promise availability, move the cars before the city event, the roster freeze after a permit is submitted, the tip split, no sleeping in the yard.

And it gave a great many of them. Nineteen days of one season’s channels, read cold against a thirty nine rule baseline, came back with forty eight more. Nineteen days, one source, one season, and it listed more rules than the entire hard constraint key the main experiment was scored against.

So sixty three was never the size of the rule set. It is the size of the part that invalidates a trip. Below that line the gear rows alone are classes rather than line items, and a real checklist expands each of them across boat type, river, trip length and water level; add site practice and administrative policy and the working rule set runs to several hundred. I used to say that from impression. Now there is a floor under it.

There is almost no overlap between the two sources, and once you see why it is not a coincidence:

Nobody overrides a rule about propane inside a dispatch board, because a dispatch board has no propane in it.

Which generalises into the most useful thing I got out of any of this:

What you can rediscover is bounded by the schema of whatever system produced the record. You can only recover rules about things you already chose to model.

A system cannot find its own blind spots in its own logs. Build a dispatch board and you will get better at dispatch rules indefinitely while the store, the propane and the promises made to customers stay exactly where they have always been.

And a good part of that bound is correct

It sounds like a defect. Much of it is a decision.

We have a large propane cage. It runs empty perhaps twice a season. Modelling it would mean a table, a field somebody updates, a threshold, and a person remembering to look at all of it, in order to catch two events a year that are currently handled by somebody saying propane is low in the chat.

That is not a gap in the model. That is the right call.

Boats and vehicles are worth modelling because they collide constantly and getting one wrong is a trip that does not launch. Propane is not, because it collides with nothing and the failure mode is a phone call.

So the honest version is that every operating model is the product of somebody deciding what is worth writing down, and everything below that line lives in conversation permanently, on purpose.

Which makes the hardest part of building one of these not the modelling. It is knowing where the line goes. Model too little and the system is useless. Model too much and nobody maintains it and it rots inside a season. I do not know a way to compute that line and I have not seen anyone describe how they draw it.

The caveat that travels with all of it

This is the first thing I would attack if somebody else published it.

The analysis did not state eighteen rules. It found eighteen places where an undocumented rule was operating, and I supplied the rule. A probe said somebody overrode this seven times, what does the system keep getting wrong. I said that is the owner, he holds an outfitting license, he does not need a guide license.

Knowing where to ask is worth a great deal. It is not the same as knowing.

It is also one business. Every run here was on an operation that already had the thing being tested for, and that is the gap none of this closes.

What went wrong

Two loader bugs in the main run. The second read an empty field for trip crew while the real crew sat in five other fields, so no guide on any day trip reached the analysis at all. Caught mid judging, after the run had been committed. I stopped, fixed it, and re ran, with both runs kept and the reason recorded before anything was scored.

The confidence scale was applied by different sessions and never calibrated between them, so the 5.3 and the 5.7 are two judgements of the same rubric rather than one measurement.

The line between a human message and an automated one is a judgement call, because the manifests and balance reports post under my own account. A different line moves the rate by a few tenths.

And I was badly wrong about almost everything I predicted. Fifteen against 5.7. Twenty percent against 96. I predicted the override log would produce the fewest findings of the four probes and it produced almost all of them. Those numbers are only interesting because they were written down first.

The main run is checkable, including the mistakes: github.com/salmonriverops/constraint-battery. The chat experiment is there too, with its predictions, both reports, the per event scoring sheets and every count. What is not there is the row level record, one line per decision with the trip, the date, the vehicle and the person’s role attached. Those files carry things that are nobody else’s business, and reading them together identifies people. Their hashes are published, so the files are fixed and I cannot quietly revise them.

Where this leaves me

The method does not replace the work of getting rules out of a person. I tested that directly and it is not close.

What it does is tell you where to ask. It draws from two sources that cover different halves of a business, only one of which most operations have, and the readable half turned out to depend on a writing habit rather than on any software.

Underneath all of it sits a judgment nobody has automated: which parts of an operation are worth writing down at all. Get that line wrong in either direction and it does not matter how good the analysis is.

If you build software these businesses run on, the completeness bound and the disappearing decider are the parts worth arguing with me about. If you work on getting requirements out of people who cannot state them, I would like to compare notes.