Graceful Gunicorn Timeouts
Update (2022-04-22): The described behaviour below is no longer accurate. Instead,
--graceful-timeout seems completely broken. Today --timeout sends a
SIGABRT as before but then 2 seconds later it sends a SIGKILL which you cannot
trap. A GracefulExit
handler may still be useful, just know you don't seem to be able
to modify the grace period anymore.
In case you're using Gunicorn and haven't read the manual, I
figured I'd let you know how your code can more gracefully handle timeouts.
Gunicorn timeouts are signaled using
the Unix SIGABRT signal.
Gunicorn provides an option called graceful-timeout which allows you to have it send a second timeout signal using SIGTERM.
This means you can trap the SIGABRT and have a few precious seconds to try and more
gracefully handle a timeout rather than just abandoning mid routine to return a generic timeout
error.
First, the config. While there are a couple ways to configure Gunicorn, this is how you'd configure it using command line arguments:
# Send SIGABRT at 5 seconds, send SIGTERM at 7 seconds
gunicorn --timeout 5 --graceful-timeout 7 …
# Send SIGABRT at 5 seconds followed by SIGKILL 2 seconds later
gunicorn --timeout 5 …
When it comes to writing Python code to use this config, here's an exception based system to handle graceful exits. First, we define a new interrupt object we can raise and a helper method to raise it.
import signal
import types
class GracefulExit(KeyboardInterrupt):
"""
An interrupt class to allow subroutines to try-except a block of code
that should have a chance to perform logic in the event of a graceful
exit signal (like keyboard interrupt or trapped Unix signal).
Properties:
signum: int
The Unix signal number trapped.
frame: frame
The python stack frame at the moment of interrupt.
"""
def __init__(self, signum: int, frame: types.FrameType):
"""
Store the values for handlers.
Arguments:
signum: int
The Unix signal number trapped.
frame: frame
Python stack frame of execution.
"""
super().__init__(f"{signal.Signals(signum).name} received.")
self.signum = signum
self.frame = frame
@staticmethod
def throw(signum: int, frame: types.FrameType):
"""
Pass to signal.signal() to raise a GracefulExit exception. Eg:
signal.signal(signal.SIGTERM, GracefulExit.throw)
Arguments:
signum: int
The Unix signal number trapped.
frame: frame
Python stack frame of execution.
Raises:
GracefulExit
This is the point…
"""
raise GracefulExit(signum, frame)
Next, when initializing everything we also need to register our handler for the signal:
signal.signal(signal.SIGABRT, GracefulExit.throw)
Finally, whenever we have a sensitive section of code, we can use a try-except block to provide a
graceful exit strategy. Our interrupt class inherits from KeyboardInterrupt
which
provides us three options:
- catch
KeyboardInterrupt
which will also handle our new interrupt; - catch
GracefulExit
to ignoreKeyboardInterrupt
issues; or - catch both in their own block to provide different clean up routines.
try:
… timeout code …
except GracefulExit:
… clean up code …
In case the inheritance from KeyboardInterrupt
doesn't make sense, you should know that
pressing Ctrl+D while using a
POSIX terminal sends the current process a
SIGINT. This includes
things running in CPython which by default
raises a KeyboardInterrupt
when this happens.