my-cd-webhook/docs/OPERATIONS.md

Operations

Runbook for the CD webhook receiver. See ../README.md for the design.

First question: is it actually working?

~/S/my-cd-webhook/bin/cd-status

An active service is not proof of anything on its own — the same lesson the uas-ng updater documents. cd-status shows the daemon state, whether the port is genuinely listening, the hook endpoints, and the outcome of the last deploy per target.

Where things are

What	Path
This repository	~/S/my-cd-webhook
Secrets (0600)	~/.config/cd-webhook/secrets.env
Rendered hooks (0600, generated)	~/.config/cd-webhook/hooks.json
Deploy logs and locks	~/.local/state/cd-webhook/<target>.log
systemd unit	~/.config/systemd/user/cd-webhook.service

journalctl --user -u cd-webhook.service -f        # receiver
tail -f ~/.local/state/cd-webhook/domaindingo-prod.log   # a deploy

Deploying by hand

The deploy script is the same one the webhook calls, so a manual deploy and an automatic one are identical:

cd ~/S/my-cd-webhook
./bin/cd-deploy --repo webdev/domaindingo \
                --ref refs/heads/prod \
                --sha <full 40-char commit sha>

Add --dry-run to see what it would do without touching anything.

The commit must be one CI has built, because the script deploys <branch>-sha-<short7> and verifies the image's revision label matches.

Rolling back

A failed health check rolls back automatically when the target has rollback_on_failure set. To roll back deliberately, deploy the previous commit:

./bin/cd-deploy --repo webdev/domaindingo --ref refs/heads/prod --sha <previous sha>

This works because every build is published under an immutable per-commit tag. Deploying "the previous version" never depends on a mutable tag still meaning what it meant yesterday — which in this fleet it once did not.

Rotating a webhook secret

1. Generate: openssl rand -hex 32
2. Update the value in Gitea (Repository → Settings → Webhooks → edit → Secret).
3. Update the matching line in ~/.config/cd-webhook/secrets.env.
4. Re-render and restart: ./install/install.sh

Order matters only in that there is a window between steps 2 and 3 where pushes are rejected. Nothing deploys with a bad signature, so the failure mode is a missed deploy, not a wrong one.

Adding a host

1. Add the host to "hosts" in targets.json with its WireGuard address and port 20090.
2. Add its targets.
3. Commit and push this repository.
4. On the host: clone, ./install/install.sh, fill in secrets.env, re-run.
5. Confirm Gitea (on s5) can reach the host's WireGuard address on that port.

Troubleshooting

Gitea shows the delivery as succeeded but nothing deployed. Expected when the push was to a branch this host does not deploy — the hook returns 200 by design so the delivery history stays readable. Check the receiver log for which rule did not match, and cd-target list for what this host owns.

Deliveries fail with a signature error. secrets.env and the Gitea webhook disagree. Re-run install/install.sh after fixing, and confirm the Gitea webhook's content type is application/json.

"image ... did not appear within 900s". CI never published the image. Check the repository's Actions run — this is almost always a failed build, not a deployment fault. The old container keeps serving throughout, so the site is unaffected.

"provenance mismatch: ... declares revision X, expected Y". The tag exists but was built from a different commit. This is the tag-drift failure mode, caught before anything changed. Do not work around it by deploying the tag directly; find out why the tag moved.

The receiver will not start.

systemctl --user status cd-webhook.service
journalctl --user -u cd-webhook.service -n 50 --no-pager

Usual causes: the webhook binary is missing, the bind address does not exist on this host (check ip addr show wg0), or the port is already taken.

Deploys stopped after a reboot and nobody was logged in. loginctl enable-linger should have been set by the installer. Verify with loginctl show-user "$(whoami)" -p Linger.

Deliberate non-features

• No docker compose down. up -d recreates only what changed. down -v in particular would destroy the data volumes these stacks depend on.
• No arbitrary SHA input over the webhook. The commit always comes from the signed push payload.
• No deploys from tags, branch deletions, or non-push events.
• No git operations. Compose stacks are uas-ng's responsibility and are refreshed by its own updater timer.