nginx: separate the proxy, the application and the network before you change a setting

Three places a gateway error can start

A visitor who sees an error from a site behind nginx has met one of three different faults. The proxy can be misconfigured, with the wrong address, path or timeout. The application behind it can be down, crashing or slow. Or something in front of nginx, such as a CDN or load balancer, can be generating the error itself. The same status code can come from any of them. Working out which one is the first job, and it is cheap: the nginx error log, the application's own log and a direct request to the application, bypassing nginx, usually separate the three in minutes.

• Proxy fault: nginx's configuration or its network path to the application.
• Application fault: the process behind it is down, crashing or slow.
• Front-end fault: a CDN or load balancer is answering in nginx's place.

What a proxy repair can change

A repair on the nginx side can correct the upstream address or socket, the path nginx sends upstream, the timeout for a deliberately slow route, the retry conditions, the headers passed to the application and the redirect behaviour. It cannot make slow application code fast, resurrect a crashed process or remove a problem on a different machine. A trustworthy fix names which of these it is and shows the evidence, rather than raising numbers until the symptom hides.

• Address, port or socket nginx uses to reach the application.
• Path substitution and location choice.
• Timeouts and retry rules for one route, with the measured duration beside them.

Test, reload, prove

nginx has a configuration test that checks syntax and the files the configuration refers to, and a reload that brings in new workers while the old ones finish their current requests. If the new configuration is invalid, the reload is rolled back and the old configuration continues, so a failed reload is quiet. That is why a proper repair does not end at the reload command. It ends when the named URLs are requested repeatedly from outside and the error log is clean for the same window, and when the previous configuration can be restored with the written steps.

• Run the configuration test before every reload.
• Request each named URL many times, not once.
• Keep the previous configuration for a one-step revert.

Where the fixed repair job starts and stops

The fixed nginx repair covers one site on a server you run, up to three named URLs and an HTTP upstream reached by proxy_pass. It stops if the error page is not from your nginx, if the application is down for reasons such as a full disk, or if the errors cannot be reproduced. It does not promise uptime or speed, and it does not touch the application. Related work, such as a certificate that stopped renewing or a service that does not restart, has its own fixed job, listed with this page.