Generic host/repo/branch-aware CD on adnanh/webhook

Replaces the per-project deploy scripts and self-contained webhook
receivers with one manifest-driven implementation that lives outside the
application repositories and can be updated independently of them.

First targets: domaindingo test and prod on s5.
This commit is contained in:
fisher
2026-08-23 07:49:52 +00:00
commit 3a6fafb5c7
22 changed files with 1872 additions and 0 deletions
+124
View File
@@ -0,0 +1,124 @@
# Operations
Runbook for the CD webhook receiver. See [../README.md](../README.md) for the
design.
## First question: is it actually working?
```bash
~/S/my-cd-webhook/bin/cd-status
```
An active service is not proof of anything on its own — the same lesson the
`uas-ng` updater documents. `cd-status` shows the daemon state, whether the port
is genuinely listening, the hook endpoints, and the outcome of the last deploy
per target.
## Where things are
| What | Path |
|---|---|
| This repository | `~/S/my-cd-webhook` |
| Secrets (0600) | `~/.config/cd-webhook/secrets.env` |
| Rendered hooks (0600, generated) | `~/.config/cd-webhook/hooks.json` |
| Deploy logs and locks | `~/.local/state/cd-webhook/<target>.log` |
| systemd unit | `~/.config/systemd/user/cd-webhook.service` |
```bash
journalctl --user -u cd-webhook.service -f # receiver
tail -f ~/.local/state/cd-webhook/domaindingo-prod.log # a deploy
```
## Deploying by hand
The deploy script is the same one the webhook calls, so a manual deploy and an
automatic one are identical:
```bash
cd ~/S/my-cd-webhook
./bin/cd-deploy --repo webdev/domaindingo \
--ref refs/heads/prod \
--sha <full 40-char commit sha>
```
Add `--dry-run` to see what it would do without touching anything.
The commit must be one CI has built, because the script deploys
`<branch>-sha-<short7>` and verifies the image's revision label matches.
## Rolling back
A failed health check rolls back automatically when the target has
`rollback_on_failure` set. To roll back deliberately, deploy the previous
commit:
```bash
./bin/cd-deploy --repo webdev/domaindingo --ref refs/heads/prod --sha <previous sha>
```
This works because every build is published under an immutable per-commit tag.
Deploying "the previous version" never depends on a mutable tag still meaning
what it meant yesterday — which in this fleet it once did not.
## Rotating a webhook secret
1. Generate: `openssl rand -hex 32`
2. Update the value in Gitea (**Repository → Settings → Webhooks → edit → Secret**).
3. Update the matching line in `~/.config/cd-webhook/secrets.env`.
4. Re-render and restart: `./install/install.sh`
Order matters only in that there is a window between steps 2 and 3 where pushes
are rejected. Nothing deploys with a bad signature, so the failure mode is a
missed deploy, not a wrong one.
## Adding a host
1. Add the host to `"hosts"` in `targets.json` with its WireGuard address and
port `20090`.
2. Add its targets.
3. Commit and push this repository.
4. On the host: clone, `./install/install.sh`, fill in `secrets.env`, re-run.
5. Confirm Gitea (on s5) can reach the host's WireGuard address on that port.
## Troubleshooting
**Gitea shows the delivery as succeeded but nothing deployed.**
Expected when the push was to a branch this host does not deploy — the hook
returns 200 by design so the delivery history stays readable. Check the receiver
log for which rule did not match, and `cd-target list` for what this host owns.
**Deliveries fail with a signature error.**
`secrets.env` and the Gitea webhook disagree. Re-run `install/install.sh` after
fixing, and confirm the Gitea webhook's content type is `application/json`.
**"image ... did not appear within 900s".**
CI never published the image. Check the repository's Actions run — this is
almost always a failed build, not a deployment fault. The old container keeps
serving throughout, so the site is unaffected.
**"provenance mismatch: ... declares revision X, expected Y".**
The tag exists but was built from a different commit. This is the tag-drift
failure mode, caught before anything changed. Do not work around it by deploying
the tag directly; find out why the tag moved.
**The receiver will not start.**
```bash
systemctl --user status cd-webhook.service
journalctl --user -u cd-webhook.service -n 50 --no-pager
```
Usual causes: the `webhook` binary is missing, the bind address does not exist
on this host (check `ip addr show wg0`), or the port is already taken.
**Deploys stopped after a reboot and nobody was logged in.**
`loginctl enable-linger` should have been set by the installer. Verify with
`loginctl show-user "$(whoami)" -p Linger`.
## Deliberate non-features
- **No `docker compose down`.** `up -d` recreates only what changed. `down -v`
in particular would destroy the data volumes these stacks depend on.
- **No arbitrary SHA input over the webhook.** The commit always comes from the
signed push payload.
- **No deploys from tags, branch deletions, or non-push events.**
- **No git operations.** Compose stacks are `uas-ng`'s responsibility and are
refreshed by its own updater timer.
+17
View File
@@ -0,0 +1,17 @@
# Active context — knowledge routing index
Index of this project's knowledge-base entries. Read an entry **only when your
current task matches its trigger** — never preemptively. When an instruction has
been fully implemented and is no longer needed, delete its line. (An ai-context
update may re-add it; if it's still done, just delete it again.)
## Entries
- Working style — code/tool checklist, math/arithmetic options → `ai-context/style/working-style.md`
- Repository operations — Gitea project, `tea` usage, branch/PR flow, remote dev
bind addresses, package-manager notes → `docs/project-knowledge-base.md`
- Gitea workflow — branch flow (`main → test → dev → feature`) and posting comments/PRs with `tea``ai-context/infrastructure/git-instructions.md`
- Gitea CI on a rootless-Docker runner — Playwright/Buildx, socket mount errors → `ai-context/infrastructure/rootless-docker-gitea-runner.md`
- Publishing public releases to a separate GitHub repo → `ai-context/infrastructure/gitea-to-github-release.md`
- Uploading build artifacts to Cloudflare R2 from CI → `ai-context/services/cloudflare-r2.md`
- CI success/failure notifications via ntfy → `ai-context/services/ntfy.md`