fisher
3a6fafb5c7
Replaces the per-project deploy scripts and self-contained webhook receivers with one manifest-driven implementation that lives outside the application repositories and can be updated independently of them. First targets: domaindingo test and prod on s5.
125 lines
4.6 KiB
Markdown
125 lines
4.6 KiB
Markdown
# Operations
|
|
|
|
Runbook for the CD webhook receiver. See [../README.md](../README.md) for the
|
|
design.
|
|
|
|
## First question: is it actually working?
|
|
|
|
```bash
|
|
~/S/my-cd-webhook/bin/cd-status
|
|
```
|
|
|
|
An active service is not proof of anything on its own — the same lesson the
|
|
`uas-ng` updater documents. `cd-status` shows the daemon state, whether the port
|
|
is genuinely listening, the hook endpoints, and the outcome of the last deploy
|
|
per target.
|
|
|
|
## Where things are
|
|
|
|
| What | Path |
|
|
|---|---|
|
|
| This repository | `~/S/my-cd-webhook` |
|
|
| Secrets (0600) | `~/.config/cd-webhook/secrets.env` |
|
|
| Rendered hooks (0600, generated) | `~/.config/cd-webhook/hooks.json` |
|
|
| Deploy logs and locks | `~/.local/state/cd-webhook/<target>.log` |
|
|
| systemd unit | `~/.config/systemd/user/cd-webhook.service` |
|
|
|
|
```bash
|
|
journalctl --user -u cd-webhook.service -f # receiver
|
|
tail -f ~/.local/state/cd-webhook/domaindingo-prod.log # a deploy
|
|
```
|
|
|
|
## Deploying by hand
|
|
|
|
The deploy script is the same one the webhook calls, so a manual deploy and an
|
|
automatic one are identical:
|
|
|
|
```bash
|
|
cd ~/S/my-cd-webhook
|
|
./bin/cd-deploy --repo webdev/domaindingo \
|
|
--ref refs/heads/prod \
|
|
--sha <full 40-char commit sha>
|
|
```
|
|
|
|
Add `--dry-run` to see what it would do without touching anything.
|
|
|
|
The commit must be one CI has built, because the script deploys
|
|
`<branch>-sha-<short7>` and verifies the image's revision label matches.
|
|
|
|
## Rolling back
|
|
|
|
A failed health check rolls back automatically when the target has
|
|
`rollback_on_failure` set. To roll back deliberately, deploy the previous
|
|
commit:
|
|
|
|
```bash
|
|
./bin/cd-deploy --repo webdev/domaindingo --ref refs/heads/prod --sha <previous sha>
|
|
```
|
|
|
|
This works because every build is published under an immutable per-commit tag.
|
|
Deploying "the previous version" never depends on a mutable tag still meaning
|
|
what it meant yesterday — which in this fleet it once did not.
|
|
|
|
## Rotating a webhook secret
|
|
|
|
1. Generate: `openssl rand -hex 32`
|
|
2. Update the value in Gitea (**Repository → Settings → Webhooks → edit → Secret**).
|
|
3. Update the matching line in `~/.config/cd-webhook/secrets.env`.
|
|
4. Re-render and restart: `./install/install.sh`
|
|
|
|
Order matters only in that there is a window between steps 2 and 3 where pushes
|
|
are rejected. Nothing deploys with a bad signature, so the failure mode is a
|
|
missed deploy, not a wrong one.
|
|
|
|
## Adding a host
|
|
|
|
1. Add the host to `"hosts"` in `targets.json` with its WireGuard address and
|
|
port `20090`.
|
|
2. Add its targets.
|
|
3. Commit and push this repository.
|
|
4. On the host: clone, `./install/install.sh`, fill in `secrets.env`, re-run.
|
|
5. Confirm Gitea (on s5) can reach the host's WireGuard address on that port.
|
|
|
|
## Troubleshooting
|
|
|
|
**Gitea shows the delivery as succeeded but nothing deployed.**
|
|
Expected when the push was to a branch this host does not deploy — the hook
|
|
returns 200 by design so the delivery history stays readable. Check the receiver
|
|
log for which rule did not match, and `cd-target list` for what this host owns.
|
|
|
|
**Deliveries fail with a signature error.**
|
|
`secrets.env` and the Gitea webhook disagree. Re-run `install/install.sh` after
|
|
fixing, and confirm the Gitea webhook's content type is `application/json`.
|
|
|
|
**"image ... did not appear within 900s".**
|
|
CI never published the image. Check the repository's Actions run — this is
|
|
almost always a failed build, not a deployment fault. The old container keeps
|
|
serving throughout, so the site is unaffected.
|
|
|
|
**"provenance mismatch: ... declares revision X, expected Y".**
|
|
The tag exists but was built from a different commit. This is the tag-drift
|
|
failure mode, caught before anything changed. Do not work around it by deploying
|
|
the tag directly; find out why the tag moved.
|
|
|
|
**The receiver will not start.**
|
|
```bash
|
|
systemctl --user status cd-webhook.service
|
|
journalctl --user -u cd-webhook.service -n 50 --no-pager
|
|
```
|
|
Usual causes: the `webhook` binary is missing, the bind address does not exist
|
|
on this host (check `ip addr show wg0`), or the port is already taken.
|
|
|
|
**Deploys stopped after a reboot and nobody was logged in.**
|
|
`loginctl enable-linger` should have been set by the installer. Verify with
|
|
`loginctl show-user "$(whoami)" -p Linger`.
|
|
|
|
## Deliberate non-features
|
|
|
|
- **No `docker compose down`.** `up -d` recreates only what changed. `down -v`
|
|
in particular would destroy the data volumes these stacks depend on.
|
|
- **No arbitrary SHA input over the webhook.** The commit always comes from the
|
|
signed push payload.
|
|
- **No deploys from tags, branch deletions, or non-push events.**
|
|
- **No git operations.** Compose stacks are `uas-ng`'s responsibility and are
|
|
refreshed by its own updater timer.
|