A server that answers on port 22 is not a server you can use

A customer buys a server. Our worker asks the provider to create the instance, waits for it to come up, sets a root password, encrypts it, and shows it on their dashboard. Four steps, and the interesting part is entirely in the gaps between them.

The provider reports the instance as RUNNING about 23 seconds after the create call. That number is fast enough to be worth designing around, and it is also the source of the problem, because RUNNING is a claim about the instance and not a claim about anything a customer can do with it.

The check that looked correct

The obvious readiness probe is a TCP connect to port 22. The reasoning is a chain: if the port accepts, sshd is up, so the box has booted, so the credential works. It is one line, it needs no secrets to run, and it is what most provisioning code does.

It is also wrong in a way that only shows up in production. The chain breaks at the last link. The password reset returns SUCCESS, sshd restarts to pick up the new credential, and port 22 starts accepting connections while that restart is still in flight. The window is several seconds wide, which is small enough to never appear in a manual test and large enough to catch a worker every time. A TCP probe run inside it returns true and reports the instance ready.

The consequence is specific. We mark the instance running, encrypt the password, write it to the row, and put it on the dashboard of someone who paid a minute ago. They copy it, they SSH in, and they get Permission denied. The provisioning pipeline logged a clean success for that order. From where the customer sits, they bought a server they cannot log into.

Those are the same event with completely different outcomes, and the only thing separating them is which question the probe asked. “Is the port open” and “does this credential work” are both readiness checks. Only one of them is the readiness the customer cares about.

What the probe is now

The check asserts the exact credential authenticates, using the same credential we are about to store:

cfg := &ssh.ClientConfig{
	User:            user,
	Auth:            []ssh.AuthMethod{ssh.Password(password)},
	HostKeyCallback: ssh.InsecureIgnoreHostKey(),
	Timeout:         sshProbeTimeout,
}
conn, err := ssh.Dial("tcp", net.JoinHostPort(ip, "22"), cfg)

It runs in a retry loop with the same poll interval as the boot wait, and the credential is not sealed and written until that loop returns true. If it never returns true within the poll budget, the whole step returns a retry error rather than a failure, because “not yet” and “broken” need to be different outcomes.

The subtle part is what the probe proves. It does not prove the server is healthy, or that any particular service is running. It proves one narrow thing: the string we are about to hand a customer opens the door we are about to point them at. That is a smaller claim than “ready” and it is the only claim we actually need.

Why the password is a separate step at all

There is an upstream constraint underneath this that shaped the design before any of it was written. The create API will not accept a Linux password. There is no field for it. The password has to be set by a separate call after the instance exists and has booted.

We found that in the documentation before writing the state machine, which is the only reason it was cheap. Had we assumed create-and-configure was one atomic operation, we would have built a two-state flow and then discovered mid-implementation that it needed to be four, with a persistence boundary in the middle.

Because it is a separate call, it can fail on its own, and it does. The provider will refuse a password reset with OperationDenied.InstanceCreating on an instance it is simultaneously reporting as RUNNING with no operation in flight. Nothing in the instance view distinguishes that state from a settled one. The rejection is the only signal that exists.

That forced a distinction we now apply everywhere: errors that mean “too early” are not counted as failed attempts. If they were, a perfectly healthy instance could exhaust its attempt budget while it was merely booting slowly, and the pipeline would destroy the box and refund the order. The customer paid, the machine was fine, and we would have thrown it away for being slow. So each provisioning step carries its own retry counter and its own recorded state, and a retryable error advances neither.

The alternative we rejected

The cheap fix is a sleep. Pause for a fixed interval after the reset returns SUCCESS, then hand over the credential. It is one line, it needs no SSH client in the provisioning path, and on the day we measured the race it would have worked.

We rejected it because a fixed delay is a guess about a variable, and it fails in both directions. Too short on a slow day and you ship the broken credential anyway, which is the exact bug you set out to fix, now with a comment claiming it is handled. Too long on a fast day and every customer waits out a padding that only the worst case needed. And a delay that is long enough today will silently become too short when the provider changes something on their side, with no test failing to tell you.

More than the tuning, the shape is wrong. The sleep converts a correctness problem into a timing gamble. The system still does not know whether the credential works. It knows how long it waited, and it has been told to treat that as equivalent. Those stop being equivalent at the worst possible moment, which is under load, on the request that mattered.

The probe answers the question directly. It costs a real SSH handshake per provision and it needs a timeout so a black-holed host cannot hang the worker, and both of those are cheaper than being unsure.

The general shape

A success status from someone else’s system is a claim. The provider is not lying when it says RUNNING or SUCCESS. It is reporting the state of the object it manages, which is a different object from the one your customer interacts with. Those diverge for seconds at a time, and seconds are enough when the thing you emit at the end is a credential someone tries immediately.

The only readiness signal worth trusting is one you verified from the customer’s position, doing the thing the customer will do, with the artifact you are about to give them.

I would guess most provisioning code in the wild still probes the port. It works nearly always, and when it does not, the failure lands on a customer who assumes they mistyped the password.