A listening port is a wonderfully low bar. It proves that something accepted responsibility for an address. In local model infrastructure, that can happen well before the system is capable of doing the job that brought a caller there.
The distinction matters because model servers have expensive startup paths. A process can bind its socket, initialize an API, and return a friendly status response while weights are still loading. A supervisor sees a reachable service. The first real request sees a long wait, a rejected model name, or a memory allocation problem. Nothing contradicted the supervisor. It simply asked an easier question.
I separate reachability from readiness because they lead to different operating decisions. Reachability says a request can arrive. Readiness says the service can accept the kind of work I'm about to give it under the configuration I intend to use.
A socket only proves admission
The first layer is transport. Is the address bound? Can the caller establish a connection? Does the API answer at all? These checks catch a stopped process, a wrong bind address, a blocked path, or an obvious protocol mismatch. They're fast, cheap, and worth keeping.
They can't establish that the model exists, that its weights fit, or that the runtime finished initializing its execution backend. A generic status endpoint may be implemented by the web process while the model worker is unavailable. Even an endpoint that lists models can be backed by configuration rather than a loaded runtime.
This is why I don't rename a transport check to “healthy” and call the job complete. I label it as reachable. The narrow name keeps the next question visible.
Readiness belongs to a workload
There isn't one universal ready state for a model server. Readiness depends on the workload I expect to send.
If the caller needs a particular model, the check should request that model. If the workflow requires structured output, a minimal request should exercise that mode. If a large context window is essential, a tiny completion doesn't prove the system can handle the real allocation. Readiness is a claim about a class of work, not a personality trait the server acquires at startup.
I keep the probe small enough that it doesn't become production traffic in disguise. A short deterministic prompt can show that the selected model loads, token generation begins, and the response has the required shape. It doesn't judge answer quality. That belongs in evaluation. It answers a more operational question: can this configured path perform one representative unit of work now?
The request also needs a time budget. A model that eventually produces a token after callers have already abandoned their jobs isn't ready for those callers. The threshold should follow the workflow's tolerance, not whatever duration makes the status turn green.
Warmth is not the same as correctness
Local runtimes often unload models, rebuild caches, or share memory among workloads. That makes readiness change over time. A cold request can be valid but slow. A warm request can be fast while using the wrong model or stale parameters. I want to know which condition I measured.
For interactive work, cold-start latency may be acceptable once, then warmth becomes part of the expected experience. For scheduled automation, preloading may be necessary so a burst of jobs doesn't all discover the cold path together. Neither policy is inherently correct. The mistake is allowing a quick warm probe to represent a cold-start guarantee it never exercised.
Memory pressure complicates the story further. A lightweight model may answer while the model required by the workflow can't load beside other resident workloads. A server-wide ping stays cheerful because the lightweight path remains available. A model-specific readiness probe exposes the actual admission decision.
I record the selected model identity and whether the probe caused a load. That makes a slow success interpretable instead of mysterious. It also prevents a fallback model from satisfying the check unless fallback is explicitly part of the workflow.
Dependencies have to be usable, not present
A model API rarely works alone. It may read model files from local storage, use an accelerator backend, call an embedding service, or write results to another system. Checking that those dependencies exist isn't enough if the service can't use them with its current permissions and configuration.
The useful probe follows the minimum real path. It should force a read of the required weights rather than merely list a directory. It should exercise the configured execution backend rather than accept a process's claim that the backend initialized. When output has to be parsed downstream, the parser should at least see the probe response.
I don't make every dependency a hard readiness condition. An optional telemetry sink shouldn't block model work. A nonessential convenience can fail separately. Readiness conditions should match what must work for the promised request, or they become a bundle of unrelated opinions that no one wants to debug.
Ready needs an expiry
Readiness is an observation, not a lifetime award. Models can be unloaded. Storage can disappear. Memory can be consumed by another task. A successful probe needs a timestamp and a useful lifetime.
For a worker about to take an expensive queue item, I prefer a recent readiness result or a check performed as part of admission. For a human opening a local chat interface, it may be enough to show that the server is reachable and allow the first request to warm the model. Different callers can tolerate different uncertainty.
The operating choice is explicit: routing can begin only after the required model has answered a bounded representative request, and that answer remains valid for a defined interval. A transport ping still runs because it diagnoses a different failure cheaply. I keep both states because collapsing them saves one label and costs the exact distinction I need when a job is waiting.