Generic host/repo/branch-aware CD on adnanh/webhook
Replaces the per-project deploy scripts and self-contained webhook receivers with one manifest-driven implementation that lives outside the application repositories and can be updated independently of them. First targets: domaindingo test and prod on s5.
This commit is contained in:
@@ -0,0 +1,124 @@
|
||||
# Operations
|
||||
|
||||
Runbook for the CD webhook receiver. See [../README.md](../README.md) for the
|
||||
design.
|
||||
|
||||
## First question: is it actually working?
|
||||
|
||||
```bash
|
||||
~/S/my-cd-webhook/bin/cd-status
|
||||
```
|
||||
|
||||
An active service is not proof of anything on its own — the same lesson the
|
||||
`uas-ng` updater documents. `cd-status` shows the daemon state, whether the port
|
||||
is genuinely listening, the hook endpoints, and the outcome of the last deploy
|
||||
per target.
|
||||
|
||||
## Where things are
|
||||
|
||||
| What | Path |
|
||||
|---|---|
|
||||
| This repository | `~/S/my-cd-webhook` |
|
||||
| Secrets (0600) | `~/.config/cd-webhook/secrets.env` |
|
||||
| Rendered hooks (0600, generated) | `~/.config/cd-webhook/hooks.json` |
|
||||
| Deploy logs and locks | `~/.local/state/cd-webhook/<target>.log` |
|
||||
| systemd unit | `~/.config/systemd/user/cd-webhook.service` |
|
||||
|
||||
```bash
|
||||
journalctl --user -u cd-webhook.service -f # receiver
|
||||
tail -f ~/.local/state/cd-webhook/domaindingo-prod.log # a deploy
|
||||
```
|
||||
|
||||
## Deploying by hand
|
||||
|
||||
The deploy script is the same one the webhook calls, so a manual deploy and an
|
||||
automatic one are identical:
|
||||
|
||||
```bash
|
||||
cd ~/S/my-cd-webhook
|
||||
./bin/cd-deploy --repo webdev/domaindingo \
|
||||
--ref refs/heads/prod \
|
||||
--sha <full 40-char commit sha>
|
||||
```
|
||||
|
||||
Add `--dry-run` to see what it would do without touching anything.
|
||||
|
||||
The commit must be one CI has built, because the script deploys
|
||||
`<branch>-sha-<short7>` and verifies the image's revision label matches.
|
||||
|
||||
## Rolling back
|
||||
|
||||
A failed health check rolls back automatically when the target has
|
||||
`rollback_on_failure` set. To roll back deliberately, deploy the previous
|
||||
commit:
|
||||
|
||||
```bash
|
||||
./bin/cd-deploy --repo webdev/domaindingo --ref refs/heads/prod --sha <previous sha>
|
||||
```
|
||||
|
||||
This works because every build is published under an immutable per-commit tag.
|
||||
Deploying "the previous version" never depends on a mutable tag still meaning
|
||||
what it meant yesterday — which in this fleet it once did not.
|
||||
|
||||
## Rotating a webhook secret
|
||||
|
||||
1. Generate: `openssl rand -hex 32`
|
||||
2. Update the value in Gitea (**Repository → Settings → Webhooks → edit → Secret**).
|
||||
3. Update the matching line in `~/.config/cd-webhook/secrets.env`.
|
||||
4. Re-render and restart: `./install/install.sh`
|
||||
|
||||
Order matters only in that there is a window between steps 2 and 3 where pushes
|
||||
are rejected. Nothing deploys with a bad signature, so the failure mode is a
|
||||
missed deploy, not a wrong one.
|
||||
|
||||
## Adding a host
|
||||
|
||||
1. Add the host to `"hosts"` in `targets.json` with its WireGuard address and
|
||||
port `20090`.
|
||||
2. Add its targets.
|
||||
3. Commit and push this repository.
|
||||
4. On the host: clone, `./install/install.sh`, fill in `secrets.env`, re-run.
|
||||
5. Confirm Gitea (on s5) can reach the host's WireGuard address on that port.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**Gitea shows the delivery as succeeded but nothing deployed.**
|
||||
Expected when the push was to a branch this host does not deploy — the hook
|
||||
returns 200 by design so the delivery history stays readable. Check the receiver
|
||||
log for which rule did not match, and `cd-target list` for what this host owns.
|
||||
|
||||
**Deliveries fail with a signature error.**
|
||||
`secrets.env` and the Gitea webhook disagree. Re-run `install/install.sh` after
|
||||
fixing, and confirm the Gitea webhook's content type is `application/json`.
|
||||
|
||||
**"image ... did not appear within 900s".**
|
||||
CI never published the image. Check the repository's Actions run — this is
|
||||
almost always a failed build, not a deployment fault. The old container keeps
|
||||
serving throughout, so the site is unaffected.
|
||||
|
||||
**"provenance mismatch: ... declares revision X, expected Y".**
|
||||
The tag exists but was built from a different commit. This is the tag-drift
|
||||
failure mode, caught before anything changed. Do not work around it by deploying
|
||||
the tag directly; find out why the tag moved.
|
||||
|
||||
**The receiver will not start.**
|
||||
```bash
|
||||
systemctl --user status cd-webhook.service
|
||||
journalctl --user -u cd-webhook.service -n 50 --no-pager
|
||||
```
|
||||
Usual causes: the `webhook` binary is missing, the bind address does not exist
|
||||
on this host (check `ip addr show wg0`), or the port is already taken.
|
||||
|
||||
**Deploys stopped after a reboot and nobody was logged in.**
|
||||
`loginctl enable-linger` should have been set by the installer. Verify with
|
||||
`loginctl show-user "$(whoami)" -p Linger`.
|
||||
|
||||
## Deliberate non-features
|
||||
|
||||
- **No `docker compose down`.** `up -d` recreates only what changed. `down -v`
|
||||
in particular would destroy the data volumes these stacks depend on.
|
||||
- **No arbitrary SHA input over the webhook.** The commit always comes from the
|
||||
signed push payload.
|
||||
- **No deploys from tags, branch deletions, or non-push events.**
|
||||
- **No git operations.** Compose stacks are `uas-ng`'s responsibility and are
|
||||
refreshed by its own updater timer.
|
||||
@@ -0,0 +1,17 @@
|
||||
# Active context — knowledge routing index
|
||||
|
||||
Index of this project's knowledge-base entries. Read an entry **only when your
|
||||
current task matches its trigger** — never preemptively. When an instruction has
|
||||
been fully implemented and is no longer needed, delete its line. (An ai-context
|
||||
update may re-add it; if it's still done, just delete it again.)
|
||||
|
||||
## Entries
|
||||
|
||||
- Working style — code/tool checklist, math/arithmetic options → `ai-context/style/working-style.md`
|
||||
- Repository operations — Gitea project, `tea` usage, branch/PR flow, remote dev
|
||||
bind addresses, package-manager notes → `docs/project-knowledge-base.md`
|
||||
- Gitea workflow — branch flow (`main → test → dev → feature`) and posting comments/PRs with `tea` → `ai-context/infrastructure/git-instructions.md`
|
||||
- Gitea CI on a rootless-Docker runner — Playwright/Buildx, socket mount errors → `ai-context/infrastructure/rootless-docker-gitea-runner.md`
|
||||
- Publishing public releases to a separate GitHub repo → `ai-context/infrastructure/gitea-to-github-release.md`
|
||||
- Uploading build artifacts to Cloudflare R2 from CI → `ai-context/services/cloudflare-r2.md`
|
||||
- CI success/failure notifications via ntfy → `ai-context/services/ntfy.md`
|
||||
Reference in New Issue
Block a user