detect and retry admin-terminated connections during healthcheck

[zacharyftw]

Fixes #614 - Detect and handle connections terminated by administrator command.

Changes

• Added detection for PostgreSQL error code 57P01 during healthcheck
• When detected, the connection is discarded and a fresh one is automatically fetched from the pool
• Added integration test to verify the retry behavior works correctly

[CLAassistant]

All committers have signed the CLA.

[codecov]

Codecov Report

❌ Patch coverage is 0% with 14 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
pgdog/src/backend/pool/healthcheck.rs	0.00%	14 Missing ⚠️

📢 Thoughts on this report? Let us know!

[levkk]

Hey! Thanks for the PR! I think this just logs the shutdown error, but the behavior of the connection pool remains the same, at least based on my reading. Happy to merge this in, it's definitely a logging/visibility improvement.

The test failure happens because the healthcheck doesn't run on every connection checkout - it runs every healthcheck_interval time, by default 30 seconds. If you want this behavior to be reproducible in tests, set that setting to zero for the duration of the test - it'll force PgDog to run a health check on each connection checkout.

You can do so by connecting to the admin database and running:

SET healthcheck_interval TO 0;

We have a utility function to connect to the admin database:

use crate::setup::admin_sqlx;

https://github.com/pgdogdev/pgdog/blob/67dc28be84fddbfdc77e5822b797f95904b69423/integration/rust/src/setup.rs#L108C14-L108C24

Don't forget to reset the settings at the end of the test (by running RELOAD) to make sure other tests are not affected.