What I built, what broke, and the evidence that changed the next decision.
No spam. No marketing. Just the writing.
A runbook becomes less useful when it tries to contain everything its author knows. Background, architecture, troubleshooting, credentials, and every remembered exception accumulate around the actual procedure. The document feels comprehensive, but the operator has to excavate the next safe step ...
Replacing a scheduled path is deceptively simple. The old job runs on a clock, the new worker reacts to a richer trigger, and both appear to perform the same task. It’s tempting to disable the old entry, enable the new one, and call the migration complete when the first run succeeds.
A work session usually ends before the work does. The clock wins, attention gets thin, or the next obligation arrives. That’s normal. What causes trouble is leaving an unfinished system in a state that only makes sense to the person who has been staring at it for three hours.
The most useful place for intelligence in a data tool is usually a small box labeled “these two records might be the same.” Everything around that box can be ordinary comparison code.
The rollback used to be the last part of my change notes. I’d describe what I wanted to alter, list the commands, think through validation, and then add a reassuring line about putting things back if necessary.
Decision: I’m going to let hobbies justify themselves by curiosity and learning. I’m going to judge responsibilities by whether I can make their total cost legible and accept the tradeoffs that cost creates.
The maintenance sequence I use for small systems contains an awkwardly manual step. After the reports are gathered and before any cleanup happens, I read the candidates and decide what not to touch.
Making another copy feels responsible. Storage is cheap, schedules are obedient, and a successful backup job offers a tidy little moment of relief. I know the feeling well. I also know how easily it lets me postpone the harder question: what, exactly, am I promising myself I can recover?
Configuration gets harder to trust as the distance grows between a service and the facts required to run it. The distance isn't measured only in directories. It shows up in extra repositories, dashboard-only settings, shell history, copied environment files, and instructions that say “remember to...
The first thing to fail in a homelab usually isn't the newest service. It's the assumption that the service is standing on something simple.