Procedure for pg-primary-unreachable alerts. Executed by the on-call database engineer.

Do NOT run promote on both replicas. Confirm with a second engineer before typing --confirm. Wrong-replica promotion is a 4-hour recovery.

Detect

PagerDuty fires pg-primary-unreachable for > 60s.
Confirm via rendra db:ping --primary from the bastion host.
Check the connections dashboard — open connections should be zero.

Mitigate

Identify the healthier replica:
```
rendra db:replicas --health
```

Promote the healthier replica:

rendra db:promote --replica=rendra-replica-b --confirm

Update DNS so application traffic flows to the new primary:
```
rendra dns:swap-primary rendra-replica-b
```

Verify

App-level error rate returns to baseline within 2 minutes.
Write traffic visible on the new primary's pg_stat_activity.
Replication lag from the surviving replicas is less than 5 seconds.

Postmortem

Open a postmortem ticket within 24 hours. Include time-to-detect, time-to-mitigate, customer impact window, and what we'd do differently. The template lives at /runbooks/postmortem-template.