SO_INCOMING_NAPI_ID Approach for Applications Written in C Programming Language

In this model, a server has a main application thread that performs a listen-accept connection and hands over accepted child connections to worker threads. There is one epoll object per worker thread, so when the server dispatches a connection to one of its worker threads, it essentially adds that child socket to the worker-thread-specific epoll object. From that point, the child socket is completely managed by the assigned worker thread.

To use ADQ technology effectively, the application thread selection logic must be modified so that a given worker thread is bound to a specific network adapter hardware queue. Each network adapter hardware queue is uniquely identified by a Linux New API ID (NAPI_​ID), which is a unique 32-bit number managed by the OS. The application can use this NAPI_​ID to align worker threads to specific network adapter queues instead of using simple round-robin-based thread selection. The application can query this information for each incoming connection using a socket option called SO_​INCOMING_​NAPI_​ID, which is supported on Linux kernel 4.12 and later. Once the connection is accepted, the application queries the NAPI_​ID on the child socket and uses the NAPI_​ID to select the worker thread that will handle the connection. All sockets with same NAPI_​ID are assigned to a specific worker thread (epoll object), establishing an application thread/network adapter hardware queue association.

Applications that have successfully used this model for ADQ enablement include Memcached and Aerospike.