# Operations

Runbook for the CD webhook receiver. See [../README.md](../README.md) for the
design.

## First question: is it actually working?

```bash
~/S/my-cd-webhook/bin/cd-status
```

An active service is not proof of anything on its own — the same lesson the
`uas-ng` updater documents. `cd-status` shows the daemon state, whether the port
is genuinely listening, the hook endpoints, and how each target's last
deploy ended.

## Where things are

| What | Path |
|---|---|
| This repository | `~/S/my-cd-webhook` |
| Secrets (0600) | `~/.config/cd-webhook/secrets.env` |
| Rendered hooks (0600, generated) | `~/.config/cd-webhook/hooks.json` |
| Deploy logs and locks | `~/.local/state/cd-webhook/<target>.log` |
| systemd unit | `~/.config/systemd/user/cd-webhook.service` |

```bash
journalctl --user -u cd-webhook.service -f        # receiver
tail -f ~/.local/state/cd-webhook/domaindingo-prod.log   # a deploy
```

## Deploying by hand

The deploy script is the same one the webhook calls, so a manual deploy and an
automatic one are identical:

```bash
cd ~/S/my-cd-webhook
./bin/cd-deploy --repo webdev/domaindingo \
                --ref refs/heads/prod \
                --sha <full 40-char commit sha>
```

Add `--dry-run` to see what it would do without touching anything.

The commit must be one CI has built, because the script deploys
`<branch>-sha-<short7>` and verifies the image's revision label matches.

## Rolling back

A failed health check rolls back automatically when the target has
`rollback_on_failure` set. To roll back deliberately, deploy the previous
commit:

```bash
./bin/cd-deploy --repo webdev/domaindingo --ref refs/heads/prod --sha <previous sha>
```

This works because every build is published under an immutable per-commit tag.
Deploying "the previous version" never depends on a mutable tag still meaning
what it meant yesterday — which in this fleet it once did not.

## Rotating a webhook secret

1. Generate: `openssl rand -hex 32`
2. Update the value in Gitea (**Repository → Settings → Webhooks → edit → Secret**).
3. Update the matching line in `~/.config/cd-webhook/secrets.env`.
4. Re-render and restart: `./install/install.sh`

Order matters only in that there is a window between steps 2 and 3 where pushes
are rejected. Nothing deploys with a bad signature, so the failure mode is a
missed deploy, not a wrong one.

## Adding a host

1. Add the host to `"hosts"` in `targets.json` with its WireGuard address and
   port `21600`.
2. Add its targets.
3. Commit and push this repository.
4. On the host: clone, `./install/install.sh`, fill in `secrets.env`, re-run.
5. Confirm Gitea (on s5) can reach the host's WireGuard address on that port.

## Troubleshooting

**Gitea shows the delivery as succeeded but nothing deployed.**
Expected when the push was to a branch this host does not deploy — the hook
returns 200 by design so the delivery history stays readable. Check the receiver
log for which rule did not match, and `cd-target list` for what this host owns.

**Deliveries fail with a signature error.**
`secrets.env` and the Gitea webhook disagree. Re-run `install/install.sh` after
fixing, and confirm the Gitea webhook's content type is `application/json`.

**"image ... did not appear within 900s".**
CI never published the image. Check the repository's Actions run — this is
almost always a failed build, not a deployment fault. The old container keeps
serving throughout, so the site is unaffected.

**"provenance mismatch: ... declares revision X, expected Y".**
The tag exists but was built from a different commit. This is the tag-drift
failure mode, caught before anything changed. Do not work around it by deploying
the tag directly; find out why the tag moved.

**The receiver will not start.**
```bash
systemctl --user status cd-webhook.service
journalctl --user -u cd-webhook.service -n 50 --no-pager
```
Usual causes: the `webhook` binary is missing, the bind address does not exist
on this host (check `ip addr show wg0`), or the port is already taken.

**Deploys stopped after a reboot and nobody was logged in.**
`loginctl enable-linger` should have been set by the installer. Verify with
`loginctl show-user "$(whoami)" -p Linger`.

## Deliberate non-features

- **No `docker compose down`.** `up -d` recreates only what changed. `down -v`
  in particular would destroy the data volumes these stacks depend on.
- **No arbitrary SHA input over the webhook.** The commit always comes from the
  signed push payload.
- **No deploys from tags, branch deletions, or non-push events.**
- **No git operations.** Compose stacks are `uas-ng`'s responsibility and are
  refreshed by its own updater timer.