System Setup

1. Perform general system OS install and setup.

   Complete the ADQ install and setup in ADQ System Under Test (SUT) Installation and General System Tuning on just the SUT system.

   Note:Many settings in General System Tuning do not persist between reboots and might need to be reapplied.
2. Install libevent-devel prerequisite package.

   Example on RHEL 8.x:

   yum install -y libevent-devel Note:For RHEL 7.x, libevent-devel is not packaged with the OS iso, it is available in the Server-optional package for the release.
3. Perform Memcached/rpc-perf build:
   1. Download the latest Memcached release.

      Example:

      wget https://memcached.org/files/memcached-1.6.9.tar.gz
   2. Untar the package. tar -xvzf memcached-1.6.9.tar.gz
   3. Install Memcached. i =${memcachedpath} ./configure -prefix=${memcachedpath}\ --with-libevent=${libeventdevelpath} make install

      After this, the Memcached server is found under ${memcachedpath}/bin.

   4. Download the rpc-perf release. cd /opt git clone https://github.com/twitter/rpc-perf.git
   5. Install rpc-perf. yum install cargo cd rpc-perf/ cargo build --release cp ../target/release/rpc-perf /usr/local/bin chmod +x /usr/local/bin/rpc-perf

Using ADQ Setup Script

The ADQ Setup script allows you to quickly and easily configure required ADQ parameters such as traffic classes, priority, filters, and ethtool parameters etc.

1. To configure ADQ, run the following command: adqsetup --dev=$iface –-priority=skbedit --busypoll=$bp --busyread=$br create $file_name queues \ $num_queues_tc1 ports $app_port addrs $addr

   See Notes below for customizing the ADQ configuration. Once ADQ is configured by adqsetup, start the Memcached server.

   Notes:
   1. The example command above creates both ingress (Rx) and egress (Tx) filters, and Linux cgroups are not needed to be created and can be skipped (cgroups are only needed if --priority=skbedit was NOT specified in adqsetup command).
   2. ADQ setup script handles symmetric queues and affinity.
   3. The setup script sets per-queue coalesce by default.
   4. The setup script sets per-TC inline flow director, for ice-1.8.x and later. For earlier versions of the ice driver, the global channel-inline-flow-director flag is used by default.
   5. Set the transmit and receive interrupt coalescing values to --rxadapt=off --rxusecs=0 --txadapt=off --txusecs=500 for improved ADQ performance.
   6. To configure independent pollers, add the -pollers=$pollers parameter in the adqsetup command (and optionally --pollers_​timeout), and remove the flags to set global --busypoll=$bp --busyread=$br.
   7. Use the cpu parameter in the command to bind the independent pollers to specific CPU cores. Refer to ADQ Setup Using ADQ Setup Script for more information on pinning pollers to specific CPU cores.
   8. The --debug parameter is optional, but it is useful for obtaining complete stack trace.
   9. For more information on how to use the script, refer to ADQ Setup Using ADQ Setup Script.
2. Seeding Memcached data prior to beginning benchmarking is best done with a python library for Memcached called pymemcache. Download and install from:

   https://github.com/pinterest/pymemcache

   This requires pip, which can be available when installing Python 3 setup tools.

3. Start Memcached server and seed data to prepare database.

   From the SUT, prepare the Memcached database for benchmarking by seeding the database, using a Python script to generate large number of keys and values. For simplicity, this example shows a single threaded pymemcache client, but this could be expanded easily to multiple threads.

   Note:The goal of seeding the database is just to write the key value pairs, so the number of threads used in the seeding process does not matter.

   For this example, key counts are 20000000, with key size of 8 bytes and value size of 64 bytes. To ensure 100% hit-rate so that every request has a value returned, each key has the format '00000001','00000002'...N. with N being the total number of keys. Each key value is the random string of length valsize.

   Example Memcached start script (calls pymemcache Python script):

   simple_memcached_start.sh iface="p1p2" num_queues_tc1=32 ipaddr=" 13.100.2.22" port=11211 mem=30000 keycount=20000000 keysize=8 valsize=64 server="${memcachedpath}/bin/memcached" client="./singlethread_seed.py -s ${ipaddr} -p ${port} -c ${keycount} -k ${keysize} -v ${valsize}" # seed database in the background (sleep 2; ${client}; echo "*** Done seeding, ready to start client side benchmark!") & # start server in the foreground ${server} -u root -p ${port} -c 8192 -- napi-ids=${num_queues_tc1} -t ${num_queues_tc1} -m ${mem} --disable-evictions Note:The --napi-ids option was added in Memcached version 1.6.9 to provide another thread selection option in the dispatch_​conn_​new code path. The default mechanism for thread selection is round-robin. The flag enables NAPI_​ID based thread selection that creates an association between socket(s) traffic that arrives on a given NIC (hardware) queue and the Memcached thread that consumes this traffic. This mapping between a Memcached thread and a HW NIC queue streamlines the flow of data from the NIC to the application. In addition, an optimal path with reduced context switches is possible, if epoll-based busy polling (sysctl -w net.core.busy_​poll = <non-zero value>) is also enabled.

   Example Python3 pymemcache script:

   singlethread_seed.py #!/usr/bin/env python3 import os import sys import argparse import random import string import psutil from pymemcache.client import base def set_key_range(ipaddr, port, keycount, keysize, valsize): client = base.PooledClient((ipaddr, port), connect_timeout=2) value = random_string(string_length=valsize) for i in range(1, keycount): key = str(i).zfill(keysize) #zero padding, '0000001' client.set(key, value) def random_string(string_length=64): letters = string.ascii_lowercase return ''.join(random.choice(letters) for i in range(string_length)) def main(argv): parser = argparse.ArgumentParser() parser.add_argument('-s', '--server', type=str, default="13.100.2.22", help="Server hostname or IP Address") parser.add_argument('-p', '--port', type=int, default=11211, help="TCP port server is listening on. Default is 11211.") parser.add_argument('-k', '--keysize', type=int, default=8, help="The keysize in bytes.") parser.add_argument('-c', '--keycount', type=int, default=20000000, help="Number of keys to produce.") parser.add_argument('-v', '--valsize', type=int, default=64, help="The value size in bytes.") parser.add_argument('-t', '--processs', type=int, default=44, help="Number of parallel thread creating keys and values.") args = parser.parse_args() set_key_range(args.server, args.port, args.keycount, args.keysize, args.valsize) if __name__ == "__main__": main(sys.argv)