Example Application: NGINX Design and Implementation

NGINX is a multi-threaded web server where each thread performs both listen and accept functions. Because each thread listens, it uses the SO_​REUSEPORT socket option so that it can use the same listen port for each thread. Each application thread calls an accept syscall and then waits to be woken up by the kernel to handle an incoming connection.

By default, the kernel picks one of the waiting threads using JHASH-based selection. This is calculated using src IP + dest IP + src port + dest port. This mechanism for selecting a waiting thread does not guarantee equal load distribution of incoming connections to threads, and does not honor the concept of a receive queue for that connection.