Source: Riskfuel & Intel Analysis

4th Gen Xeon Scalable Processor: Test by Intel as of Nov, 2023. 2x Intel(R) Xeon(R) Platinum 8480+ CPU @ 2.00GHz, 56 cores/socket, 2 sockets, HT On, Turbo On, Total Memory 512 GB (16 slots/ 32 GB/ DDR5 4800 M Hz [run @ 4800 MHz] ), Inspur, NF5280-M7-A0-R0-00, 05.08.01, 0x2b000461, Red Hat Enterprise Linux release 8.8 (Ootpa)​, Kernel 4.18.0-477.21.1.el8_​8.x86_​64​ ​, Python: 3.10.13, OpenVINO: 2023.1.0 Workload:Riskfuel Model Inference, BF16, Batch Size: 1, 10, 50, 100, 1000, 2000, 3000, 4000, 5000, 6000, 7000, 8000, 9000, 10000, 11000, 12000, 13000, 14000, 15000, 16000, 17000, 18000, 19000, 20000, 80000

5th Gen Xeon Scalable Processor: Test by Intel as of Nov, 2023. 2x Intel(R) Xeon(R) Platinum 8592Y+ CPU @ 1.9GHz, 64 cores/socket, 2 sockets, HT On, Turbo On, Total Memory 512 GB (16 slots/ 32 GB/ DDR5 5600 M Hz [run @ 5600 MHz] ), Quanta Cloud Technology Inc., QuantaGrid D54Q-2U, 3B05.TEL4P1, 0x21000161, Red Hat Enterprise Linux release 8.8 (Ootpa)​, Kernel 4.18.0-477.21.1.el8_​8.x86_​64​, Python: 3.10.13, OpenVINO: 2023.1.0 Workload:Riskfuel Model Inference, BF16, Batch Size: 1, 10, 50, 100, 1000, 2000, 3000, 4000, 5000, 6000, 7000, 8000, 9000, 10000, 11000, 12000, 13000, 14000, 15000, 16000, 17000, 18000, 19000, 20000, 80000

Sub 100ms second token latency with 5th Gen Intel Xeon Platinum 8592+ on GPT-j use cases (BF16)

Up to 1.11x performance speedup and 1.18x higher performance/watt with 5th Gen Intel Xeon Scalable processor vs 4th Gen Intel Xeon on GPT-J first token latency (bfloat16)

Up to 1.17x performance speedup and 1.24x higher performance/watt with 5th Gen Intel Xeon Scalable processor vs 4th Gen Intel Xeon on GPT-J second token latency (bfloat16)

Up to 1.13x performance speedup and 1.20x higher performance/watt with 5th Gen Intel Xeon Scalable processor vs 4th Gen Intel Xeon on GPT-J first token latency (int8)

Up to 1.22x performance speedup and 1.30x higher performance/watt with 5th Gen Intel Xeon Scalable processor vs 4th Gen Intel Xeon on GPT-J second token latency (int8)

Up to 2.3x performance speedup and 1.81x higher performance/watt with 5th Gen Intel Xeon Scalable processor vs 3rd Gen Intel Xeon on GPT-J first token latency (int8)

Up to 1.64x performance speedup and 1.30x higher performance/watt with 5th Gen Intel Xeon Scalable processor vs 3rd Gen Intel Xeon on GPT-J second token latency (int8)

8592+: 1-node, 2x INTEL(R) XEON(R) PLATINUM 8592+, 64 cores, HT On, Turbo On, NUMA 2, Total Memory 1024GB (16x64GB DDR5 5600 MT/s [5600 MT/s]), BIOS 2.0, microcode 0x21000161, 2x Ethernet Controller 10-Gigabit X540-AT2, 1x 1.7T SAMSUNG MZQL21T9HCJR-00A07, Ubuntu 22.04.3 LTS, 5.15.0-83-generic, GPT-J 6B, gcc=12.3, python 3.9intel-extension-for-pytorch 2.2.0+git305fc52torch 2.2.0.dev20230914+cputransformers 4.31.0protobuf 3.20.3neural-compressor 2.2, Test on 1 socket, BS1 CPI 64. Test by Intel as of 09/21/23.

8480+: 1-node, 2x Intel(R) Xeon(R) Platinum 8480+, 56 cores, HT On, Turbo On, NUMA 2, Total Memory 1024GB (16x64GB DDR5 4800 MT/s [4800 MT/s]), BIOS 2.0, microcode 0x2b0004d0, 2x Ethernet Controller X710 for 10GBASE-T, 1x 1.7T SAMSUNG MZQL21T9HCJR-00A07, Ubuntu 22.04.3 LTS, 5.15.0-83-generic, GPT-J 6B, gcc=12.3, python 3.9intel-extension-for-pytorch 2.2.0+git305fc52torch 2.2.0.dev20230914+cputransformers 4.31.0protobuf 3.20.3neural-compressor 2.2, Test on 1 socket, BS1 CPI 56. Test by Intel as of 09/27/23.

8380: 1-node, 2x Intel(R) Xeon(R) Platinum 8380 CPU @ 2.30GHz, 40 cores, HT On, Turbo On, NUMA 2, Total Memory 1024GB (16x64GB DDR4 3200 MT/s [3200 MT/s]), BIOS 1.5, microcode 0xd0003a5, 2x Ethernet Controller 10-Gigabit X540-AT2, 1x 1.7T SAMSUNG MZQL21T9HCJR-00A07, Ubuntu 22.04.3 LTS, 5.15.0-83-generic, GPT-J 6B, gcc=12.3, python 3.9intel-extension-for-pytorch 2.2.0+git305fc52torch 2.2.0.dev20230914+cputransformers 4.31.0protobuf 3.20.3neural-compressor 2.2, Test on 1 socket, BS1 CPI 40. Test by Intel as of 09/26/23.

Up to 1.12x performance speedup and 1.17x higher performance/watt with 5th Gen Intel Xeon Scalable processor vs 4th Gen Intel Xeon on Llama 2 13B first token latency (bfloat16)

Up to 1.10x performance speedup and 1.16x higher performance/watt with 5th Gen Intel Xeon Scalable processor vs 4th Gen Intel Xeon on Llama 2 13B second token latency (bfloat16)

Up to 1.05x performance speedup and 1.10x higher performance/watt with 5th Gen Intel Xeon Scalable processor vs 4th Gen Intel Xeon on Llama 2 13B first token latency (int8)

Up to 1.08x performance speedup and 1.14x higher performance/watt with 5th Gen Intel Xeon Scalable processor vs 4th Gen Intel Xeon on Llama 2 13B second token latency (int8)

Up to 2.1x performance speedup and 1.58x higher performance/watt with 5th Gen Intel Xeon Scalable processor vs 3rd Gen Intel Xeon on Llama 2 13B first token latency (int8)

Up to 1.48x performance speedup and 1.11x higher performance/watt with 5th Gen Intel Xeon Scalable processor vs 3rd Gen Intel Xeon on Llama 2 13B second token latency (int8)

8592+: 1-node, 2x INTEL(R) XEON(R) PLATINUM 8592+, 64 cores, HT On, Turbo On, NUMA 2, Integrated Accelerators Available [used]: DLB 2 [0], DSA 2 [0], IAX 2 [0], QAT 2 [0], Total Memory 1024GB (16x64GB DDR5 5600 MT/s [5600 MT/s]), BIOS 2.0, microcode 0x21000161, 2x Ethernet Controller X710 for 10GBASE-T, 1x Ethernet interface, 1x 1.7T SAMSUNG MZQL21T9HCJR-00A07, Ubuntu 22.04.2 LTS, 5.15.0-78-generic, Test by Intel as of 10/10/23.

8480+ 1-node, 2x Intel(R) Xeon(R) Platinum 8480+, 56 cores, HT On, Turbo On, NUMA 2, Integrated Accelerators Available [used]: DLB 5 [0], DSA 5 [0], IAX 5 [0], QAT 5 [0], Total Memory 1024GB (16x64GB DDR5 4800 MT/s [4800 MT/s]), BIOS 1.3, microcode 0x2b0004b1, 2x Ethernet Controller X710 for 10GBASE-T, 1x Ethernet interface, 1x 1.7T SAMSUNG MZQL21T9HCJR-00A07, Ubuntu 22.04.2 LTS, 5.15.0-78-generic, Test by Intel as of 09/12/23.

8380: 1-node, 2x Intel(R) Xeon(R) Platinum 8380 CPU @ 2.30GHz, 40 cores, HT On, Turbo On, NUMA 2, Integrated Accelerators Available [used]: DLB 0 [0], DSA 0 [0], IAX 0 [0], QAT 0 [0], Total Memory 1024GB (16x64GB DDR4 3200 MT/s [3200 MT/s]), BIOS 1.5, microcode 0xd0003a5, 2x Ethernet Controller 10-Gigabit X540-AT2, 1x 1.7T SAMSUNG MZQL21T9HCJR-00A07, Ubuntu 22.04.2 LTS, 5.15.0-78-generic, Test by Intel as of 10/10/23. Llama2, gcc=12.3 gxx=12.3, python 3.9, intel-extension-for-pytorch 2.2.0+,git305fc52, torch 2.2.0.dev20230914+cpu, transformers 4.31.0, protobuf 3.20.3, neural-compressor 2.2, test on 1 socket, Batch Size(BS)=1, Corest per instance (CPI)=#cores/socket, search algorithm: BEAM4.

[A5] Up to 1.18x faster batch DLSA Bert-Large Finetuning for IMDB dataset with 5th Gen Intel Xeon Platinum 8592+ with AMX BF16 vs. prior generation with AMX BF16

Up to 3x faster batch DLSA Bert-Large Finetuning for IMDB dataset with 5th Gen Intel Xeon Platinum 8592+ with AMX BF16 compared to FP32

Up to 1.25x faster batch DLSA Bert-Large Finetuning for SST2 dataset with 5th Gen Intel Xeon Platinum 8592+ with AMX BF16 vs. prior generation with AMX BF16

Up to 3.4x faster batch DLSA Bert-Large Finetuning for SST2 dataset with 5th Gen Intel Xeon Platinum 8592+ with AMX BF16 compared to FP32

8592+: 1-node, 2x Intel(R) Xeon(R) Platinum 8592+, 64 cores, HT On, Turbo On, NUMA 2, Total Memory 1024GB (16x64GB DDR5 5600 MT/s [5600 MT/s]), BIOS 3B05.TEL4P1, microcode 0x21000161, 2x Ethernet Controller X710 for 10GBASE-T, 1x 1.8T SAMSUNG MZQL21T9HCJR-00A07, CentOS Stream 8, 5.18.14, Framework /Toolkit incl version -Torch 2.0.1+cpu, Transformers 4.35.0,IPEX 2.0.100,Oneccl 2.0.100,Test by Intel as of 10/05/23.

8480+: 1-node, 2x Intel(R) Xeon(R) Platinum 8480+, 56 cores, HT On, Turbo On, Total Memory 1024GB (16x64GB DDR5 4800 MT/s [4800 MT/s]), BIOS EGSDCRB1.SYS.0102.D37.2305081420, microcode 0x2b0004b1, 1x Ethernet Controller I225-LM, 1x 894.3G INTEL SSDSC2KG96, 1x 7.2G Cruzer Blade, 2x 3.7T INTEL SSDPE2KX040T8, CentOS Stream 8, 5.15.0-spr.bkc.pc.16.4.24.x86_​64, Framework /Toolkit incl version -Torch 2.0.1+cpu, Transformers 4.35.0,IPEX 2.0.100,Oneccl 2.0.100,Test by Intel as of 10/09/23

SST2 Dataset -

8592+: 1-node, 2x Intel(R) Xeon(R) Platinum 8592+, 64 cores, HT On, Turbo On, NUMA 2, Integrated Accelerators Available [used]: DLB 2 [0], DSA 2 [0], IAA 2 [0], QAT 2 [0], Total Memory 1024GB (16x64GB DDR5 5600 MT/s [5600 MT/s]), BIOS 3B05.TEL4P1, microcode 0x21000161, 2x Ethernet Controller X710 for 10GBASE-T, 1x 1.8T SAMSUNG MZQL21T9HCJR-00A07, CentOS Stream 8, 5.18.14, Framework /Toolkit incl version -Torch 2.0.1+cpu, Transformers 4.35.0,IPEX 2.0.100,Oneccl 2.0.100,Test by Intel as of 10/05/23.

8480+: 1-node, 2x Intel(R) Xeon(R) Platinum 8480+, 56 cores, HT On, Turbo On, Total Memory 1024GB (16x64GB DDR5 4800 MT/s [4800 MT/s]), BIOS EGSDCRB1.SYS.0102.D37.2305081420, microcode 0x2b0004b1, 1x Ethernet Controller I225-LM, 1x 894.3G INTEL SSDSC2KG96, 1x 7.2G Cruzer Blade, 2x 3.7T INTEL SSDPE2KX040T8, CentOS Stream 8, 5.15.0-spr.bkc.pc.16.4.24.x86_​64, Framework /Toolkit incl version -Torch 2.0.1+cpu, Transformers 4.35.0,IPEX 2.0.100,Oneccl 2.0.100,,Test by Intel as of 10/09/23

Up to 1.71x faster batch Vision Transfer Learning for Colorectal Dataset with 5th Gen Intel Xeon Platinum 8592+ with AMX BF16 compared to FP32

8480+: 1-node, 2x (1x used) Intel(R) Xeon(R) Platinum 8480+, 56 cores, HT On, Turbo On, Total Memory 1024GB (16x64GB DDR5 4800 MT/s [4800 MT/s]), BIOS EGSDCRB1.SYS.0102.D37.2305081420, microcode 0x2b0004b1, 1x Ethernet Controller I225-LM, 1x 894.3G INTEL SSDSC2KG96, 1x 7.2G Cruzer Blade, 2x 3.7T INTEL SSDPE2KX040T8, CentOS Stream 8, 5.15.0-spr.bkc.pc.16.4.24.x86_​64, Resnet 50 v1.5 , Intel Tensorflow AV512 2.14, Test by Intel as of 10/09/2023

8480+: 1-node, 2x Intel(R) Xeon(R) Platinum 8480+, 56 cores, HT On, Turbo On, NUMA 2, Total Memory 1024GB (16x64GB DDR5 4800 MT/s [4800 MT/s]), BIOS EGSDCRB1.SYS.0102.D37.2305081420, microcode 0x2b0004b1, 1x Ethernet Controller I225-LM, 1x 894.3G INTEL SSDSC2KG96, 1x 7.2G Cruzer Blade, 2x 3.7T INTEL SSDPE2KX040T8, CentOS Stream 8, 5.15.0-spr.bkc.pc.16.4.24.x86_​64, Spark 3.5.0, Intel Tensor-flow nightly 2.14 ww34,Python 3.10.12,Pyspark 3.5.0, export KMP_​BLOCKTIME=1,export KMP_​SETTINGS=1,granularity=fine,verbose,compact,1,0. Test by Intel as of 10/10/23.

8480+ : 1-node, 2x Intel® Xeon® Platinum 8480+, 56 cores, HT On, Turbo Off, Total Memory 1024 GB (16 slots/ 64 GB/ 4800 MHz [run @ 4800 MHz] ), EGSDCRB1.SYS.0102.D37.2305081420, 0x2b0004b1, CentOS 8 Stream, 5.15.0-spr.bkc.pc.16.4.24.x86_​64, 1x Ethernet Controller I225-LM, 1x 894.3G INTEL SSDSC2KG96, 1x 7.2G Cruzer Blade, 2x 3.7T INTEL SSDPE2KX040T8, E2E Census, XGBoost 1.7.6, Ridge Regression, Pandas 2.1.1, Modin 0.24.1, Scikit-Learn 1.3.1/2023.2.1 Numpy 1.26.0, FP32. Test by Intel on 10/12/2023.

8480+: -1 node, 2x Intel® Xeon® Platinum 8480+, 56 cores, HT On, Turbo On, Total Memory 1024GB (16x64GB DDR5 4800 MT/s [4800 MT/s]), BIOS 3A16.TEL0P1​, 0x2b0004b1, CentOS 8 Stream5.18.14, 2x Ethernet Controller X550, 1x 1.8T SAMSUNG MZWLR1T9HBJR-00007, PyTorch 2.0.1, IPEX 2.0.1, Transformers 4.31.0, Datasets 2.14.0, Peft 0.4.0, Evaluate 0.4.0​, Test by Intel as of 10/26/2023

8480+: 1-node, 2x Intel(R) Xeon(R) Platinum 8480+, 56 cores, HT On, Turbo On, NUMA 2, Integrated Accelerators Available [used]: DLB 2 [0], DSA 2 [0], IAA 2 [0], QAT 2 [0], Total Memory 1024GB (16x64GB DDR5 4800 MT/s [4800 MT/s]), BIOS EGSDCRB1.SYS.0102.D37.2305081420, microcode 0x2b0004b1, 1x Ethernet Controller I225-LM, 1x 7.2G Cruzer Blade, 2x 3.7T INTEL SSDPE2KX040T8, 1x 894.3G INTEL SSDSC2KG96, CentOS Stream 8, 5.15.0-spr.bkc.pc.16.4.24.x86_​64. TensorFlow TF: 2.14.0202334, OneDNN: v3.2 PyTorch: torch:2.1.0.dev20230825+cpu, IPEX 2.0.100, PEFT 0.3.0, BF16, Test by INTEL as of Tue Sep 5 08:04:39 UTC 2023

4.4x to 14.2x higher PyTorch Training performance on 5th Gen Intel Xeon Scalable processor with built in Intel AMX BF16 vs. 3rd Gen Intel Xeon (FP32) 6.2x higher average* PyTorch Training performance on 5th Gen Intel Xeon Scalable processor with built in Intel AMX BF16 vs. 3rd Gen Intel Xeon (FP32)

Up to 1.24x higher BERT-large training performance on 5th Gen Intel Xeon Platinum 8592+ with AMX BF16 vs. prior generation with AMX BF16

Up to 5.17x higher BERT-large training performance on 5th Gen Intel Xeon Platinum 8592+ with AMX BF16 vs. 3rd Gen Intel Xeon Platinum 8380 [fp32]

Up to 1.1x higher DLRM training performance on 5th Gen Intel Xeon Platinum 8592+ with AMX BF16 vs. prior generation with AMX BF16

Up to 4.4x higher DLRM training performance on 5th Gen Intel Xeon Platinum 8592+ with AMX BF16 vs. 3rd Gen Intel Xeon Platinum 8380 [fp32] Up to 1.29x higher MaskRCNN training performance on 5th Gen Intel Xeon Platinum 8592+ with AMX BF16 vs. prior generation with AMX BF16

Up to 6.3x higher MaskRCNN training performance on 5th Gen Intel Xeon Platinum 8592+ with AMX BF16 vs. 3rd Gen Intel Xeon Platinum 8380 [fp32]

Up to 1.2x higher ResNet50 v1.5 training performance on 5th Gen Intel Xeon Platinum 8592+ with AMX BF16 vs. prior generation with AMX BF16

Up to 4.5x higher ResNet50 v1.5 training performance on 5th Gen Intel Xeon Platinum 8592+ with AMX BF16 vs. 3rd Gen Intel Xeon Platinum 8380 [fp32] Up to 1.28x higher RNN-T training performance on 5th Gen Intel Xeon Platinum 8592+ with AMX BF16 vs. prior generation with AMX BF16

Up to 14.2x higher RNN-T training performance on 5th Gen Intel Xeon Platinum 8592+ with AMX BF16 vs. 3rd Gen Intel Xeon Platinum 8380 [fp32] Up to 1.17x higher SSD-ResNet34 training performance on 5th Gen Intel Xeon Platinum 8592+ with AMX BF16 vs. prior generation with AMX BF16

Up to 6.5x higher SSD-ResNet34 training performance on 5th Gen Intel Xeon Platinum 8592+ with AMX BF16 vs. 3rd Gen Intel Xeon Platinum 8380 [fp32]

8480+: 1-node, 2x (1x used) Intel(R) Xeon(R) Platinum 8480+, 56 cores, HT On, Turbo On, NUMA 2, Total Memory 1024GB (16x64GB DDR5 4800 MT/s [4800 MT/s]), BIOS EGSDCRB1.SYS.0102.D37.2305081420, microcode 0x2b0004b1, 1x Ethernet Controller I225-LM, 1x 7.2G Cruzer Blade, 2x 3.7T INTEL SSDPE2KX040T8, 1x 894.3G INTEL SSDSC2KG96, CentOS Stream 8, 5.15.0-spr.bkc.pc.16.4.24.x86_​64. PyTorch: torch:2.1.0.dev20230825+cpu oneDNN:v3.2.1 IPEX:2.1.0+git31b5ee1. Test by INTEL as of 09/05/2023.

8380: 1-node, 2x (1x used) Intel(R) Xeon(R) Platinum 8380 CPU @ 2.30GHz, 40 cores, HT On, Turbo On, Total Memory 1024GB (16x64GB DDR4 3200 MT/s [3200 MT/s]), BIOS SE5C620.86B.01.01.0008.2305172341, microcode 0xd0003a5, 2x Ethernet Controller 10G X550T, 3x 894.3G INTEL SSDSC2KG96, 1x 0B vmDisk, 1x 60G Flash Disk, 1x 1024M vmDisk-CD, 1x 3.6T INTEL SSDPE2KX040T8, 1x 931.5G INTEL SSDPE2KX010T8, Ubuntu 22.04 LTS, 5.15.0-78-generic, PyTorch: torch:2.1.0.dev20230825+cpu oneDNN:v3.2.1 IPEX:2.1.0+git31b5ee1. Test by INTEL as of 09/14/23.

BS=x, 1 instance/ numa node [One socket used]. Resnet50 v1.5: ImageNet (224 x224), SSD Resnet34: COCO 2017, BERT-large: Wikipedia 2020/01/01 ( seq len =512), DLRM: Criteo Terabyte Dataset, RNNT: LibriSpeech, Mask RCNN: COCO 2017 *Geomean of ResNet50 v1.5, Bert Large, SSD RN34, RNNT, MaskRCNN ,DLRM (PT:Training)

1.32x higher average* performance/watt with 5th Gen Intel Xeon Scalable processor vs. 4th Gen Intel Xeon Scalable processor with AMX BF16 on PyTorch Real-Time Inference

8.7x higher average* performance and 9.2x higher average* performance/watt with 5th Gen Intel Xeon Scalable processor with AMX INT8 vs FP32 on PyTorch Real-Time Inference

1.19x to 1.42x higher performance and 1.17x to 1.48x higher performance/watt on PyTorch Real Time Inference [AMX BF16] with 5th Gen Intel Xeon Scalable processor vs prior generation

1.3x higher average* performance and 1.32x higher average* performance/watt with 5th Gen Intel Xeon Scalable processor vs. 4th Gen Intel Xeon Scalable processor with AMX INT8 on PyTorch Real-Time Inference

5.1x higher average* Performance and 5.2x higher average* performance/watt with 5th Gen Intel Xeon Scalable processor with AMX BF16 vs. FP32 on PyTorch Real-Time Inference

2.9x to 6.7 higher performance and 2.3x to 4.8x higher performance/watt on PyTorch Real Time Inference [AMX INT8] with 5th Gen Intel Xeon Scalable processor vs 3rd Gen Intel Xeon Scalable processor

4.1x higher average* performance and 3.1x higher average* performance/watt with 5th Gen Intel Xeon Scalable processor with AMX INT8 vs. 3rd Gen Intel Xeon Scalable processor (VNNI) on PyTorch Real-Time Inference

7x to 14x higher performance and 5.3x to 9.5x higher performance/watt on PyTorch Real Time Inference [AMX BF16] with 5th Gen Intel Xeon Scalable processor vs 3rd Gen Intel Xeon Scalable processor

9x higher average* performance and 6.7x higher average* performance/watt with 5th Gen Intel Xeon Scalable processor with AMX BF16 vs. 3rd Gen Intel Xeon Scalable processor (FP32) on PyTorch Real-Time Inference

1.32x higher average* performance and 1.33x higher average* performance/watt with 5th Gen Intel Xeon Scalable processor vs. 4th Gen Intel Xeon Scalable processor with AMX INT8 on AI Batch Inference

11x higher average* performance and 11x higher average* performance/watt with 5th Gen Intel Xeon Scalable processor with AMX INT8 vs. FP32 on AI Batch Inference

1.24x to 1.44x higher performance and 1.31x to 1.48x higher performance/watt on PyTorch Batch Inference [AMX BF16] with 5th Gen Intel Xeon Scalable processor vs prior generation

1.35x higher average* performance and 1.3x higher average* performance/watt with 5th Gen Intel Xeon Scalable processor vs. 4th Gen Intel Xeon Scalable processor with AMX BF16 on AI Batch Inference

5.3x higher average* performance and 5.5x higher average* performance/watt with 5th Gen Intel Xeon Scalable processor with AMX BF16 vs. FP32 on AI Batch Inference

4x to 6.5x higher performance and 3x to 5x higher performance/watt on PyTorch Batch Inference [AMX INT8] with 5th Gen Intel Xeon Scalable processor vs 3rd Gen Intel Xeon Scalable processor

5.3x higher average* performance and 3.9x higher average* performance/watt with 5th Gen Intel Xeon Scalable processor with AMX INT8 vs. 3rd Gen Intel Xeon Scalable processor (VNNI) on PyTorch Batch Inference

7.7x to 13.8x higher performance and 5.8x to 10.3x higher performance/watt on PyTorch Batch Inference [AMX BF16] with 5th Gen Intel Xeon Scalable processor vs 3rd Gen Intel Xeon Scalable processor

9.9x higher average* performance and 7.3x higher average* performance/watt with 5th Gen Intel Xeon Scalable processor with AMX BF16 vs. 3rd Gen Intel Xeon Scalable processor (FP32) on PyTorch Batch Inference

Up to 1.23x higher real-time Natural Language Processing inference (BERT-large) performance and 1.20x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX INT8 vs. prior generation with AMX INT8

Up to 5.4x higher real-time Natural Language Processing inference performance (BERT-large) and 5.9x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX BF16 compared to FP32 Up to 9.2x higher real-time Natural Language Processing inference performance (BERT-large) and 10.2x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX INT8 compared to FP32

Up to 9.9x higher real-time Natural Language Processing inference (BERT-large) performance and 7.7x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX BF16 vs. 3rd gen Intel Xeon processor with FP32

Up to 3.4x higher real-time Natural Language Processing inference (BERT-large) performance and 2.5x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX INT8 vs. 3rd gen Intel Xeon processor with INT8 Batch Inference Performance

Up to 1.38x higher batch Natural Language Processing inference (BERT-large) performance and 1.22x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX BF16 vs. prior generation with AMX BF16

Up to 1.2x higher batch Natural Language Processing inference (BERT-large) performance and 1.26x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX INT8 vs. prior generation with AMX INT8

Up to 5.5x higher batch Natural Language Processing inference performance (BERT-large) and 5.3x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX BF16 compared to FP32

Up to 9x higher batch Natural Language Processing inference performance (BERT-large) and 9.3x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX INT8 compared to FP32

Up to 10x higher batch Natural Language Processing inference (BERT-large) performance and 7.1x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX BF16 vs. 3rd gen Intel Xeon processor with FP32

Up to 4x higher batch Natural Language Processing inference (BERT-large) performance and 3x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX INT8 vs. 3rd gen Intel Xeon processor with INT8

Up to 1.24x higher batch Recommendation System inference performance (DLRM) and 1.24x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX BF16 vs. prior generation with prior generation with AMX BF16

Up to 1.24x higher batch Recommendation System inference performance (DLRM) and 1.18x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX INT8 vs. prior generation with prior generation with AMX INT8

Up to 4.8x higher batch Recommendation System inference performance (DLRM) and 4.8x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX BF16 compared to FP32

Up to 10.6x higher batch Recommendation System inference performance (DLRM) and 10.6x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX INT8 compared to FP32

Up to 8.7x higher batch Recommendation System inference performance (DLRM) and 6.2x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX BF16 vs. 3rd Gen Intel Xeon processor with FP32

Up to 5.5x higher batch Recommendation System inference performance (DLRM) and 3.8x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX INT8 vs. 3rd Gen Intel Xeon processor with INT8

Up to 1.4x higher real time object detection inference performance (SSD-ResNet34) and 1.4x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX BF16 vs. prior generation with AMX BF16

Up to 1.34x higher real time object detection inference performance (SSD-ResNet34) and 1.42x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX INT8 vs. prior generation with AMX INT8 Up to 8x higher real time object detection inference performance (SSD-ResNet34) and 7.5x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX BF16 compared to FP32

Up to 15x higher real time object detection inference performance (SSD-ResNet34) and 15x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX INT8 compared to FP32

Up to 14x higher real time object detection inference performance (SSD-ResNet34) and 9.6x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX BF16 vs. 3rd Gen Intel Xeon processor with FP32

Up to 6.7x higher real time object detection inference performance (SSD-ResNet34) and 4.8x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX INT8 vs. 3rd Gen Intel Xeon processor with INT8 Batch Inference Performance

Up to 1.4x higher batch object detection inference performance (SSD-ResNet34) and 1.32x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX BF16 vs. prior generation with AMX BF16

Up to 1.43x higher batch object detection inference performance (SSD-ResNet34) and 1.42x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX INT8 vs. prior generation with AMX INT8 Up to 7.7x higher batch object detection inference performance (SSD-ResNet34) and 8.1x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX BF16 compared to FP32

Up to 14x higher batch object detection inference performance (SSD-ResNet34) and 17.5x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX INT8 compared to FP32

Up to 13.8x higher batch object detection inference performance (SSD-ResNet34) and 10x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX BF16 vs. 3rd Gen Intel Xeon processor with FP32

Up to 6.5x higher batch object detection inference performance (SSD-ResNet34) and 5x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX INT8 vs. 3rd Gen Intel Xeon processor with INT8

Up to 1.24x higher real time object classification inference performance (ResNeXT101_​32x16d) and 1.23x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX BF16 vs. prior generation with AMX BF16

Up to 1.3x higher real time object classification inference performance (ResNeXT101_​32x16d) and 1.28x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX INT8 vs. prior generation with AMX INT8

Up to 4.3x higher real time object classification inference performance (ResNeXT101_​32x16d) and 4.3x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX BF16 compared to FP32

Up to 9x higher real time object classification inference performance (ResNeXT101_​32x16d) and 9x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX INT8 compared to FP32

Up to 7.2x higher real time object classification inference performance (ResNeXT101_​32x16d) and 5.3x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX BF16 vs. 3rd Gen Intel Xeon processor with FP32

Up to 4.3x higher real time object classification inference performance (ResNeXT101_​32x16d) and 3.2x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX INT8 vs. 3rd Gen Intel Xeon processor with INT8 Batch Inference Performance

Up to 1.36x higher batch object classification inference performance (ResNeXT101_​32x16d) and 1.28x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX BF16 vs. prior generation with AMX BF16

Up to 1.29x higher batch object classification inference performance (ResNeXT101_​32x16d) and 1.4x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX INT8 vs. prior generation with AMX INT8

Up to 5.8x higher batch object classification inference performance (ResNeXT101_​32x16d) and 5.6x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX BF16 compared to FP32

Up to 12x higher batch object classification inference performance (ResNeXT101_​32x16d) and 13x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX INT8 compared to FP32

Up to 10x higher batch object classification inference performance (ResNeXT101_​32x16d) and 7x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX BF16 vs. 3rd Gen Intel Xeon processor with FP32

Up to 6x higher batch object classification inference performance (ResNeXT101_​32x16d) and 4.6x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX INT8 vs. 3rd Gen Intel Xeon processor with INT8

Up to 1.24x higher real time speech recognition inference performance (RNN-T) and 1.23x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX BF16 vs. prior generation with AMX BF16

Up to 4.2x higher real time speech recognition inference performance (RNN-T) and 4.2x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX BF16 compared to FP32

Up to 10.7x higher real time speech recognition inference performance (RNN-T) and 7.9x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX BF16 vs. 3rd Gen Intel Xeon processor with FP32 Batch Inference Performance

Up to 1.44x higher batch speech recognition inference performance (RNN-T) and 1.4x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX BF16 vs. prior generation with AMX BF16

Up to 3.7x higher batch speech recognition inference performance (RNN-T) and 3.9x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX BF16 compared to FP32

Up to 7.7x higher batch speech recognition inference performance (RNN-T) and 6x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX BF16 vs. 3rd Gen Intel Xeon processor with FP32

Up to 1.41x higher real-time Natural Language Processing inference (DistilBERT) performance and 1.48x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX BF16 vs. prior generation with AMX BF16

Up to 1.35x higher real-time Natural Language Processing inference (DistilBERT) performance and 1.38x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX INT8 vs. prior generation with AMX INT8

Up to 4.1x higher real-time Natural Language Processing inference performance (DistilBERT) and 4.4x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX BF16 compared to FP32

Up to 5.7x higher real-time Natural Language Processing inference performance (DistilBERT) and 6x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX INT8 compared to FP32

Up to 7x higher real-time Natural Language Processing inference (DistilBERT) performance and 5.6x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX BF16 vs. 3rd gen Intel Xeon processor with FP32

Up to 2.9x higher real-time Natural Language Processing inference (DistilBERT) performance and 2.3x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX INT8 vs. 3rd gen Intel Xeon processor with INT8 Batch Inference Performance

Up to 1.36x higher batch Natural Language Processing inference (DistilBERT) performance and 1.32x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX BF16 vs. prior generation with AMX BF16

Up to 1.38x higher batch Natural Language Processing inference (DistilBERT) performance and 1.35x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX INT8 vs. prior generation with AMX INT8

Up to 5.7x higher batch Natural Language Processing inference performance (DistilBERT) and 5.7x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX BF16 compared to FP32

Up to 8.8x higher batch Natural Language Processing inference performance (DistilBERT) and 9x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX INT8 compared to FP32

Up to 10x higher batch Natural Language Processing inference (DistilBERT) performance and 8x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX BF16 vs. 3rd gen Intel Xeon processor with FP32

Up to 4x higher batch Natural Language Processing inference (DistilBERT) performance and 3x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX INT8 vs. 3rd gen Intel Xeon processor with INT8

Up to 1.42x higher real time image segmentation inference performance (MaskRCNN) and 1.47x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX BF16 vs. prior generation with AMX BF16

Up to 5.3x higher real time image segmentation inference performance (MaskRCNN) and 5.4x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX BF16 compared to FP32

Up to 9x higher real time image segmentation inference performance (MaskRCNN) and 6.5x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX BF16 vs. 3rd Gen Intel Xeon processor with FP32 Batch Inference Performance

Up to 1.26x higher batch image segmentation inference performance (MaskRCNN) and 1.18x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX BF16 vs. prior generation with AMX BF16

Up to 4.6x higher batch image segmentation inference performance (MaskRCNN) and 4.9x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX BF16 compared to FP32

Up to 8.5x higher batch image segmentation inference performance (MaskRCNN) and 6x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX BF16 vs. 3rd Gen Intel Xeon processor with FP32

Up to 1.24x higher performance and 1.28x higher performance/watt on real-time image classification(ResNet50) inference (BF16) with 5th Gen Intel Xeon 8592+ (64c) vs. 4th Gen Intel Xeon 8480+

Up to 8x higher performance and 6.4x higher performance/watt on real-time image classification(ResNet50) inference (BF16) with 5th Gen Intel Xeon 8592+ (64c) vs. 3rd Gen Intel Xeon 8380 (fp32)

Up to 1.23x higher performance and 1.26x higher performance/watt on real-time image classification(ResNet50) inference (INT8) with 5th Gen Intel Xeon 8592+ (64c) vs. 4th Gen Intel Xeon 8480+

Up to 3.8x higher performance and 3x higher performance/watt on real-time image classification(ResNet50) inference (INT8) with 5th Gen Intel Xeon 8592+ (64c) vs. 3rd Gen Intel Xeon 8380

Batch Inference Performance:

Up to 1.21x higher performance and 1.29x higher performance/watt on batch image classification(ResNet50) inference (BF16) with 5th Gen Intel Xeon 8592+ (64c) 4th Gen Intel Xeon 8480+

Up to 8.8x higher performance and 7.4x higher performance/watt on batch image classification(ResNet50) inference (BF16) with 5th Gen Intel Xeon 8592+ (64c) vs. 3rd Gen Intel Xeon 8380 (fp32)

Up to 1.23x higher performance and 1.26x higher performance/watt on batch image classification(ResNet50) inference (INT8) with 5th Gen Intel Xeon 8592+ (64c) vs. 4th Gen Intel Xeon 8480+

Up to 3.8x higher performance and 3x higher performance/watt on batch image classification(ResNet50) inference (INT8) with 5th Gen Intel Xeon 8592+ (64c) vs. 3rd Gen Intel Xeon 8380

Up to 2.85x higher batched Recommendation System inference performance (DLRM) and 2.58x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX INT8 vs. AMD EPYC 9554 (Genoa)

Up to 2.34x higher batched Recommendation System inference performance (DLRM) and 2.26x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX INT8 vs. AMD EPYC 9654 (Genoa)

Up to 1.90x higher batched Recommendation System inference performance (DLRM) and 1.72x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX INT8 vs. AMD EPYC 9754 (Bergamo)

Intel Xeon 8592+: 1-node, 2x Intel(R) Xeon(R) Platinum 8592+, 64 cores, HT On, Turbo On, NUMA 2, Total Memory 1024GB (16x64GB DDR5 5600 MT/s [5600 MT/s]), BIOS 2.0, microcode 0x21000161, 2x Ethernet Controller X710 for 10GBASE-T, 1x 1.7T SAMSUNG MZQL21T9HCJR-00A07, Ubuntu 22.04.2 LTS, 5.15.0-78-generic. Framework=Pytorch 2.1, IPEX=2.1, oneDNN:v3.2.1, Python 3.8, AI Model= DLRM(https://github.com/IntelAI/models/) , Batched Results: best scores achieved using BS>1, Precision=INT8-AMX, Test by INTEL as of 10/10/2023.

AMD EPYC 9554: 1-node, 2x AMD EPYC 9554 64-Core Processor, 64 cores, SMT On, Turbo On, NUMA 2, Total Memory 1536GB (24x64GB DDR5 4800 MT/s [4800 MT/s]), BIOS 1.5, microcode 0xa10113e, 2x Ethernet Controller 10G X550T, 1x 1.7T SAMSUNG MZQL21T9HCJR-00A07, Ubuntu 22.04.3 LTS, 5.15.125-0515125-generic, Framework=Pytorch 2.1, IPEX=2.1, oneDNN: v3.2.1, Python 3.8, AI Model= DLRM(https://github.com/IntelAI/models/) , Batched Results: best scores achieved using BS>1, Precision=INT8. Test by INTEL as of 09/11/23.

AMD EPYC 9654: 1-node, 2x AMD EPYC 9654 96-Core Processor, 96 cores, SMT On, Turbo On, NUMA 2, Total Memory 1536GB (24x64GB DDR5 4800 MT/s [4800 MT/s]), BIOS 1.5, microcode 0xa10113e, 2x Ethernet Controller 10G X550T, 1x 1.7T SAMSUNG MZQL21T9HCJR-00A07, Ubuntu 22.04.3 LTS, 5.15.125-0515125-generic, Framework=Pytorch 2.1, IPEX=2.1, oneDNN: v3.2.1, Python 3.8, AI Model= DLRM(https://github.com/IntelAI/models/) , Batched Results: best scores achieved using BS>1, Precision=INT8. Test by INTEL as of 09/11/23.

AMD EPYC 9754: 1-node, 2x AMD EPYC 9754 128-Core Processor, 128 cores, SMT On, Turbo On, NUMA 2, Total Memory 1536GB (24x64GB DDR5 4800 MT/s [4800 MT/s]), BIOS 1.5, microcode 0xa10113e, 2x Ethernet Controller 10G X550T, 1x 1.7T SAMSUNG MZQL21T9HCJR-00A07, Ubuntu 22.04.3 LTS, 5.15.125-0515125-generic, Framework=Pytorch 2.1, IPEX=2.1, oneDNN: v3.2.1, Python 3.8, AI Model= DLRM(https://github.com/IntelAI/models/) , Batched Results: best scores achieved using BS>1, Precision=INT8. Test by INTEL as of 09/11/23.

Up to 1.57x higher real-time Natural Language Processing inference (DistilBERT) performance and 1.69x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX BF16 vs. AMD EPYC 9654 (Genoa)

Up to 2.57x higher real-time Natural Language Processing inference (DistilBERT) performance and 2.27x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX BF16 vs. AMD EPYC 9754 (Bergamo)

Up to 2.6x higher real-time Natural Language Processing inference (DistilBERT) performance and 2.34x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with INT8 vs. AMD EPYC 9654 (Genoa)

Up to 3.96x higher real-time Natural Language Processing inference (DistilBERT) performance and 3.39x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with INT8 vs. AMD EPYC 9754 (Bergamo)

Up to 2.25x higher batched Natural Language Processing inference (DistilBERT) performance and 2.58x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX BF16 vs. AMD EPYC 9654 (Genoa)

Up to 1.83x higher batched Natural Language Processing inference (DistilBERT) performance and 1.83x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX BF16 vs. AMD EPYC 9754 (Bergamo)

Up to 2.13x higher batched Natural Language Processing inference (DistilBERT) performance and 2.44x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with INT8 vs. AMD EPYC 9554 (Genoa)

Up to 1.74x higher batched Natural Language Processing inference (DistilBERT) performance and 1.89x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with INT8 vs. AMD EPYC 9654 (Genoa)

Up to 1.48x higher batched Natural Language Processing inference (DistilBERT) performance and 1.54x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with INT8 vs. AMD EPYC 9754 (Bergamo)

Claims with INT8:

Intel Xeon 8592+: 1-node, 2x Intel(R) Xeon(R) Platinum 8592+, 64 cores, HT On, Turbo On, NUMA 2, Total Memory 1024GB (16x64GB DDR5 5600 MT/s [5600 MT/s]), BIOS 2.0, microcode 0x21000161, 2x Ethernet Controller X710 for 10GBASE-T, 1x 1.7T SAMSUNG MZQL21T9HCJR-00A07, Ubuntu 22.04.2 LTS, 5.15.0-78-generic. Framework=Pytorch 2.0, IPEX=2.0, Python 3.8, AI Model=DistilBERT (https://github.com/IntelAI/models/), INT8-AMX, Real Time (BS=1) results while maintaining 5ms latency SLA, Test by INTEL  as of 10/10/2023.

AMD EPYC 9554: 1-node, 2x AMD EPYC 9554 64-Core Processor, 64 cores, SMT On, Turbo On, NUMA 2, Total Memory 1536GB (24x64GB DDR5 4800 MT/s [4800 MT/s]), BIOS 1.5,  microcode 0xa10113e, 2x Ethernet Controller 10G X550T, 1x 1.7T SAMSUNG MZQL21T9HCJR-00A07, Ubuntu 22.04.3 LTS, 5.15.125-0515125-generic, Framework=Pytorch 2.0, IPEX=2.0, Python 3.8, AI Model=DistilBERT (https://github.com/IntelAI/models/) , INT8 Real Time (BS=1) results while maintaining 5ms latency SLA.  Test by INTEL as of 09/11/23.

AMD EPYC 9654: 1-node, 2x AMD EPYC 9654 96-Core Processor, 96 cores, SMT On, Turbo On, NUMA 2, Total Memory 1536GB (24x64GB DDR5 4800 MT/s [4800 MT/s]), BIOS 1.5,  microcode 0xa10113e, 2x Ethernet Controller 10G X550T, 1x 1.7T SAMSUNG MZQL21T9HCJR-00A07, Ubuntu 22.04.3 LTS, 5.15.125-0515125-generic, Framework=Pytorch 2.0, IPEX=2.0, Python 3.8, AI Model=DistilBERT (https://github.com/IntelAI/models/) , INT8 Real Time (BS=1) results while maintaining 5ms latency SLA.  Test by INTEL as of 09/11/23.

AMD EPYC 9754: 1-node, 2x AMD EPYC 9754 128-Core Processor, 128 cores, SMT On, Turbo On, NUMA 2, Total Memory 1536GB (24x64GB DDR5 4800 MT/s [4800 MT/s]), BIOS 1.5,  microcode 0xaa00212, 2x Ethernet Controller 10G X550T, 1x 1.7T SAMSUNG MZQL21T9HCJR-00A07, Ubuntu 22.04.3 LTS, 5.15.125-0515125-generic, Framework=Pytorch 2.0, IPEX=2.0, Python 3.8, AI Model=DistilBERT (https://github.com/IntelAI/models/) , INT8 Real Time (BS=1) results while maintaining 5ms latency SLA.  Test by INTEL as of 10/26/23.

Batched Results: best scores achieved using (BS>1).

Claims with BF16:

Intel Xeon 8592+: 1-node, 2x Intel(R) Xeon(R) Platinum 8592+, 64 cores, HT On, Turbo On, NUMA 2, Total Memory 1024GB (16x64GB DDR5 5600 MT/s [5600 MT/s]), BIOS 2.0, microcode 0x21000161, 2x Ethernet Controller X710 for 10GBASE-T, 1x 1.7T SAMSUNG MZQL21T9HCJR-00A07, Ubuntu 22.04.2 LTS, 5.15.0-78-generic. Framework=Pytorch 2.0, IPEX=2.0, Python 3.8, AI Model=DistilBERT (https://github.com/IntelAI/models/), BF16-AMX, Real Time (BS=1) results while maintaining 5ms latency SLA, Test by INTEL  as of 10/10/2023.

AMD EPYC 9654: 1-node, 2x AMD EPYC 9654 96-Core Processor, 96 cores, SMT On, Turbo On, NUMA 2, Total Memory 1536GB (24x64GB DDR5 4800 MT/s [4800 MT/s]), BIOS 1.5,  microcode 0xa10113e, 2x Ethernet Controller 10G X550T, 1x 1.7T SAMSUNG MZQL21T9HCJR-00A07, Ubuntu 22.04.3 LTS, 5.15.125-0515125-generic, Framework=Pytorch 2.0, IPEX=2.0, Python 3.8, AI Model=DistilBERT (https://github.com/IntelAI/models/) , BF16 Real Time (BS=1) results while maintaining 5ms latency SLA.  Test by INTEL as of 09/11/23.

AMD EPYC 9754: 1-node, 2x AMD EPYC 9754 128-Core Processor, 128 cores, SMT On, Turbo On, NUMA 2, Total Memory 1536GB (24x64GB DDR5 4800 MT/s [4800 MT/s]), BIOS 1.5,  microcode 0xaa00212, 2x Ethernet Controller 10G X550T, 1x 1.7T SAMSUNG MZQL21T9HCJR-00A07, Ubuntu 22.04.3 LTS, 5.15.125-0515125-generic, Framework=Pytorch 2.0, IPEX=2.0, Python 3.8, AI Model=DistilBERT (https://github.com/IntelAI/models/) , BF16 Real Time (BS=1) results while maintaining 5ms latency SLA.  Test by INTEL as of 10/26/23.

Batched Results: best scores achieved using (BS>1).

Up to 2.13x higher real-time Natural Language Processing inference (BERT-Large) performance and 2.08x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX BF16 vs. AMD EPYC 9654 (Genoa)

Up to 1.83x higher real-time Natural Language Processing inference (BERT-Large) performance and 1.71x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX BF16 vs. AMD EPYC 9754 (Bergamo)

Up to 1.80x higher real-time Natural Language Processing inference (BERT-Large) performance and 1.65x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with INT8 vs. AMD EPYC 9654 (Genoa)

Up to 1.49x higher real-time Natural Language Processing inference (BERT-Large) performance and 1.30x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with INT8 vs. AMD EPYC 9754 (Bergamo)

Up to 2.69x higher batched Natural Language Processing inference (BERT-Large) performance and 2.96x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX BF16 vs. AMD EPYC 9654 (Genoa)

Up to 2.19x higher batched Natural Language Processing inference (BERT-Large) performance and 2.19x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX BF16 vs. AMD EPYC 9754 (Bergamo)

Up to 2.56x higher batched Natural Language Processing inference (BERT-Large) performance and 2.47x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with INT8 vs. AMD EPYC 9554 (Genoa)

Up to 2.10x higher batched Natural Language Processing inference (BERT-Large) performance and 2.13x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with INT8 vs. AMD EPYC 9654 (Genoa)

Up to 1.78x higher batched Natural Language Processing inference (BERT-Large) performance and 1.72x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with INT8 vs. AMD EPYC 9754 (Bergamo)

Claims with INT8

Intel Xeon 8592+: 1-node, 2x Intel(R) Xeon(R) Platinum 8592+, 64 cores, HT On, Turbo On, NUMA 2, Total Memory 1024GB (16x64GB DDR5 5600 MT/s [5600 MT/s]), BIOS 2.0, microcode 0x21000161, 2x Ethernet Controller X710 for 10GBASE-T, 1x 1.7T SAMSUNG MZQL21T9HCJR-00A07, Ubuntu 22.04.2 LTS, 5.15.0-78-generic. Framework=Pytorch 2.0, IPEX=2.0, Python 3.8, AI Model=BERT Large(https://github.com/IntelAI/models/) , INT8-AMX , Real Time (BS=1) results while maintaining 130ms latency SLA, Test by INTEL as of 10/10/2023.

AMD EPYC 9554: 1-node, 2x AMD EPYC 9554 64-Core Processor, 64 cores, SMT On, Turbo On, NUMA 2, Total Memory 1536GB (24x64GB DDR5 4800 MT/s [4800 MT/s]), BIOS 1.5, microcode 0xa10113e, 2x Ethernet Controller 10G X550T, 1x 1.7T SAMSUNG MZQL21T9HCJR-00A07, Ubuntu 22.04.3 LTS, 5.15.125-0515125-generic, Framework=Pytorch 2.0, IPEX=2.0, Python 3.8, AI Model=BERT Large(https://github.com/IntelAI/models/) , INT8, Real Time (BS=1) results while maintaining 130ms latency SLA. Test by INTEL as of 09/11/23.

AMD EPYC 9654: 1-node, 2x AMD EPYC 9654 96-Core Processor, 96 cores, SMT On, Turbo On, NUMA 2, Total Memory 1536GB (24x64GB DDR5 4800 MT/s [4800 MT/s]), BIOS 1.5, microcode 0xa10113e, 2x Ethernet Controller 10G X550T, 1x 1.7T SAMSUNG MZQL21T9HCJR-00A07, Ubuntu 22.04.3 LTS, 5.15.125-0515125-generic, Framework=Pytorch 2.0, IPEX=2.0, Python 3.8, AI Model=BERT Large(https://github.com/IntelAI/models/) , INT8, Real Time (BS=1) results while maintaining 130ms latency SLA. Test by INTEL as of 09/11/23.

AMD EPYC 9754: 1-node, 2x AMD EPYC 9754 128-Core Processor, 128 cores, SMT On, Turbo On, NUMA 2, Total Memory 1536GB (24x64GB DDR5 4800 MT/s [4800 MT/s]), BIOS 1.5, microcode 0xaa00212, 2x Ethernet Controller 10G X550T, 1x 1.7T SAMSUNG MZQL21T9HCJR-00A07, Ubuntu 22.04.3 LTS, 5.15.125-0515125-generic, Framework=Pytorch 2.0, IPEX=2.0, Python 3.8, AI Model=BERT Large(https://github.com/IntelAI/models/) , INT8, Real Time (BS=1) results while maintaining 130ms latency SLA. Test by INTEL as of 10/26/23.

Batched Results: best scores achieved using (BS >1).

Claims with BF16

Intel Xeon 8592+: 1-node, 2x Intel(R) Xeon(R) Platinum 8592+, 64 cores, HT On, Turbo On, NUMA 2, Total Memory 1024GB (16x64GB DDR5 5600 MT/s [5600 MT/s]), BIOS 2.0, microcode 0x21000161, 2x Ethernet Controller X710 for 10GBASE-T, 1x 1.7T SAMSUNG MZQL21T9HCJR-00A07, Ubuntu 22.04.2 LTS, 5.15.0-78-generic. Framework=Pytorch 2.0, IPEX=2.0, Python 3.8, AI Model=BERT Large(https://github.com/IntelAI/models/) , BF16-AMX , Real Time (BS=1) results while maintaining 130ms latency SLA, Test by INTEL as of 10/10/2023.

AMD EPYC 9654: 1-node, 2x AMD EPYC 9654 96-Core Processor, 96 cores, SMT On, Turbo On, NUMA 2, Total Memory 1536GB (24x64GB DDR5 4800 MT/s [4800 MT/s]), BIOS 1.5, microcode 0xa10113e, 2x Ethernet Controller 10G X550T, 1x 1.7T SAMSUNG MZQL21T9HCJR-00A07, Ubuntu 22.04.3 LTS, 5.15.125-0515125-generic, Framework=Pytorch 2.0, IPEX=2.0, Python 3.8, AI Model=BERT Large(https://github.com/IntelAI/models/) , BF16, Real Time (BS=1) results while maintaining 130ms latency SLA. Test by INTEL as of 09/11/23.

AMD EPYC 9754: 1-node, 2x AMD EPYC 9754 128-Core Processor, 128 cores, SMT On, Turbo On, NUMA 2, Total Memory 1536GB (24x64GB DDR5 4800 MT/s [4800 MT/s]), BIOS 1.5, microcode 0xaa00212, 2x Ethernet Controller 10G X550T, 1x 1.7T SAMSUNG MZQL21T9HCJR-00A07, Ubuntu 22.04.3 LTS, 5.15.125-0515125-generic, Framework=Pytorch 2.0, IPEX=2.0, Python 3.8, AI Model=BERT Large(https://github.com/IntelAI/models/) , BF16, Real Time (BS=1) results while maintaining 130ms latency SLA. Test by INTEL as of 10/26/23.

Batched Results: best scores achieved using (BS >1).

Up to 2.66x higher real-time Image Classification inference (ResNet50v1.5) performance and 2.57x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX BF16 vs. AMD EPYC 9654 (Genoa)

Up to 2.66x higher real-time Image Classification inference (ResNet50v1.5) performance and 2.36x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX BF16 vs. AMD EPYC 9754 (Bergamo)

Up to 2.44x higher real-time Classification inference (ResNet50v1.5) performance and 2.24x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with INT8 vs. AMD EPYC 9654 (Genoa)

Up to 2.15x higher real-time Image Classification inference (ResNet50v1.5) performance and 1.88x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with INT8 vs. AMD EPYC 9754 (Bergamo)

Up to 2.42x higher batched Image Classification inference (ResNet50v1.5) performance and 2.76x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX BF16 vs. AMD EPYC 9654 (Genoa)

Up to 2.01x higher real-time Image Classification inference (ResNet50v1.5) performance and 2.23x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with AMX BF16 vs. AMD EPYC 9754 (Bergamo)

Up to 2.71x higher batched Image Classification inference (ResNet50v1.5) performance and 2.9x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with INT8 vs. AMD EPYC 9554 (Genoa)

Up to 2.26x higher batched Image Classification inference (ResNet50v1.5) performance and 2.38x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with INT8 vs. AMD EPYC 9654 (Genoa)

Up to 1.86x higher batched Image Classification inference (ResNet50v1.5) performance and 1.95x higher performance/watt on 5th Gen Intel Xeon Platinum 8592+ with INT8 vs. AMD EPYC 9754 (Bergamo)

Claims with INT 8:

Intel Xeon 8592+: 1-node, 2x Intel(R) Xeon(R) Platinum 8592+, 64 cores, HT On, Turbo On, NUMA 2, Total Memory 1024GB (16x64GB DDR5 5600 MT/s [5600 MT/s]), BIOS 2.0, microcode 0x21000161, 2x Ethernet Controller X710 for 10GBASE-T, 1x 1.7T SAMSUNG MZQL21T9HCJR-00A07, Ubuntu 22.04.2 LTS, 5.15.0-78-generic. TensorFlow= Intel TF 2.13, OneDNN=3.2, Python 3.8, AI Model=ResNet50v1.5 (https://github.com/IntelAI/models/), INT8-AMX, Real Time (BS=1) results while maintaining 15ms latency SLA, Test by INTEL as of 10/10/2023.

AMD EPYC 9554: 1-node, 2x AMD EPYC 9554 64-Core Processor, 64 cores, SMT On, Turbo On, NUMA 2, Total Memory 1536GB (24x64GB DDR5 4800 MT/s [4800 MT/s]), BIOS 1.5, microcode 0xa10113e, 2x Ethernet Controller 10G X550T, 1x 1.7T SAMSUNG MZQL21T9HCJR-00A07, Ubuntu 22.04.3 LTS, 5.15.125-0515125-generic, ZenDNN 4.1, TensorFlow= 2.12.1, Python 3.8, AI Model=ResNet50v1.5 (https://github.com/IntelAI/models/) ,, INT8, Real Time (BS=1) results while maintaining 15ms latency SLA. Test by INTEL as of 09/11/23.

AMD EPYC 9654: 1-node, 2x AMD EPYC 9654 96-Core Processor, 96 cores, SMT On, Turbo On, NUMA 2, Total Memory 1536GB (24x64GB DDR5 4800 MT/s [4800 MT/s]), BIOS 1.5, microcode 0xa10113e, 2x Ethernet Controller 10G X550T, 1x 1.7T SAMSUNG MZQL21T9HCJR-00A07, Ubuntu 22.04.3 LTS, 5.15.125-0515125-generic, ZenDNN 4.1, TensorFlow= 2.12.1, Python 3.8, AI Model=ResNet50v1.5 (https://github.com/IntelAI/models/) ,, INT8, Real Time (BS=1) results while maintaining 15ms latency SLA. Test by INTEL as of 09/11/23.

AMD EPYC 9754: 1-node, 2x AMD EPYC 9754 128-Core Processor, 128 cores, SMT On, Turbo On, NUMA 2, Total Memory 1536GB (24x64GB DDR5 4800 MT/s [4800 MT/s]),BIOS 1.5., microcode 0xaa00212, 2x Ethernet Controller 10G X550T, 1x 1.7T SAMSUNG MZQL21T9HCJR-00A07, Ubuntu 22.04.3 LTS, 5.15.125-0515125-generic, ZenDNN 4.1, TensorFlow= 2.12.1, Python 3.8, AI Model=ResNet50v1.5 (https://github.com/IntelAI/models/) , INT8, Real Ti'me (BS=1) results while maintaining 15ms latency SLA. Test by INTEL as of 10/26/23.

Batched Results: best scores achieved using (BS>1).

Claims with BF16

Intel Xeon 8592+: 1-node, 2x Intel(R) Xeon(R) Platinum 8592+, 64 cores, HT On, Turbo On, NUMA 2, Total Memory 1024GB (16x64GB DDR5 5600 MT/s [5600 MT/s]), BIOS 2.0, microcode 0x21000161, 2x Ethernet Controller X710 for 10GBASE-T, 1x 1.7T SAMSUNG MZQL21T9HCJR-00A07, Ubuntu 22.04.2 LTS, 5.15.0-78-generic. TensorFlow= Intel TF 2.13, OneDNN=3.2, Python 3.8, AI Model=ResNet50v1.5 (https://github.com/IntelAI/models/), BF16-AMX, Real Time (BS=1) results while maintaining 15ms latency SLA, Test by INTEL as of 10/10/2023.

AMD EPYC 9654: 1-node, 2x AMD EPYC 9654 96-Core Processor, 96 cores, SMT On, Turbo On, NUMA 2, Total Memory 1536GB (24x64GB DDR5 4800 MT/s [4800 MT/s]), BIOS 1.5, microcode 0xa10113e, 2x Ethernet Controller 10G X550T, 1x 1.7T SAMSUNG MZQL21T9HCJR-00A07, Ubuntu 22.04.3 LTS, 5.15.125-0515125-generic, ZenDNN 4.1, TensorFlow= 2.12.1, Python 3.8, AI Model=ResNet50v1.5 (https://github.com/IntelAI/models/) , BF16, Real Time (BS=1) results while maintaining 15ms latency SLA. Test by INTEL as of 09/11/23.

AMD EPYC 9754: 1-node, 2x AMD EPYC 9754 128-Core Processor, 128 cores, SMT On, Turbo On, NUMA 2, Total Memory 1536GB (24x64GB DDR5 4800 MT/s [4800 MT/s]),BIOS 1.5., microcode 0xaa00212, 2x Ethernet Controller 10G X550T, 1x 1.7T SAMSUNG MZQL21T9HCJR-00A07, Ubuntu 22.04.3 LTS, 5.15.125-0515125-generic, ZenDNN 4.1, TensorFlow= 2.12.1, Python 3.8, AI Model=ResNet50v1.5 (https://github.com/IntelAI/models/) , BF16, Real Time (BS=1) results while maintaining 15ms latency SLA. Test by INTEL as of 10/26/23.

Batched Results: best scores achieved using (BS>1).

1.7x higher average batched AI performance and performance/watt on 5th Gen Intel Xeon Platinum 8592+ with Intel AMX supporting INT8 vs. AMD EPYC 9754 (Bergamo) for Industry Standard Frameworks including Classification, Natural Language Processing, and Recommender.

Up to 1.10x higher RocksDB performance and 1.12x higher performance/watt with 5th Gen Intel Xeon Platinum 8592+ using integrated Intel IAA vs. 4th Gen Intel Xeon Platinum 8480+ using integrated Intel IAA

Up to 3.7x higher RocksDB performance and 3.08x higher performance/watt with 5th Gen Intel Xeon Platinum 8592+ using integrated Intel IAA vs. 3rd Gen Intel Xeon Platinum 8380 using Zstd

Up to 5.6x higher RocksDB performance with 5th Gen Intel Xeon Platinum 8592+ using integrated Intel IAA vs. 1st Gen Intel Xeon Platinum 8180 using Zstd

Up to 52% latency reduction on RocksDB with 5th Gen Intel Xeon Platinum 8592+ using integrated Intel IAA vs. 1st Gen Intel Xeon Platinum 8180 using Zstd

Up to 3.4x higher RocksDB Perf/W with 5th Gen Intel Xeon Platinum 8592+ using integrated Intel IAA vs. 1st Gen Intel Xeon Platinum 8180 using Zstd

Up to 1.36x higher ClickHouse Perf/W with 5th Gen Intel Xeon Platinum 8592+ using integrated Intel IAA vs. 4th Gen Intel Xeon Platinum 8480+ using integrated Intel IAA

Up to 1.57x higher ClickHouse performance with 5th Gen Intel Xeon Platinum 8592+ using integrated Intel IAA vs. Zstd

Up to 1.46x higher ClickHouse Perf/W with 5th Gen Intel Xeon Platinum 8592+ using integrated Intel IAA vs. Zstd

Up to 2.49x higher ClickHouse performance with 5th Gen Intel Xeon Platinum 8592+ using integrated Intel IAA vs. 3rd Gen Intel Xeon Platinum 8380 using Zstd

Up to 2.08x higher ClickHouse Perf/W with 5th Gen Intel Xeon Platinum 8592+ using integrated Intel IAA vs. 3rd Gen Intel Xeon Platinum 8380 using Zstd

Up to 2.48x higher MS SQL+Backup performance with 5th Gen Intel Xeon Platinum 8592+ using integrated Intel QAT vs. default compression Up to 2.52x higher MS SQL+Backup Perf/W with 5th Gen Intel Xeon Platinum 8592+ using integrated Intel QAT vs. default compression

Up to 2.56x higher MS SQL+Backup performance with 5th Gen Intel Xeon Platinum 8592+ using integrated Intel QAT vs. 3rd Gen Intel Xeon Platinum 8380 using default compression

Up to 2.33x higher MS SQL+Backup Perf/W with 5th Gen Intel Xeon Platinum 8592+ using integrated Intel QAT vs. 3rd Gen Intel Xeon Platinum 8380 using default compression

Up to 2.72x speedup and 1.75x higher perf/W with 5th Gen Intel Xeon Platinum 8592+ using integrated Intel QAT vs. 1st Gen Intel Xeon Platinum 8180 with default compression

Up to 1.84x higher HammerDB MySQL OLTP performance and 1.53x higher HammerDB MySQL OLTP performance/watt with the 5th Gen Intel® Xeon® Platinum 8592+ processor vs. 3-year Refresh Platform

Up to 3.2x higher HammerDB MySQL OLTP performance and 1.9x higher HammerDB MySQL OLTP performance/watt with the 5th Gen Intel® Xeon® Platinum 8592+ processor vs. 5-year Refresh Platform.

Up to 32% latency reduction on Rocks DB with 5th Gen Intel Xeon Platinum 8592+ using integrated Intel IAA vs. 4th Gen AMD EPYC 9554 using Zstd

Up to 1.72x higher RocksDB Perf/W with 5th Gen Intel Xeon Platinum 8592+ using integrated Intel IAA vs. 4th Gen AMD EPYC 9554 using Zstd

Up to 1.11x higher ClickHouse Perf/W with 5th Gen Intel Xeon Platinum 8592+ using integrated Intel IAA vs. 4th Gen AMD EPYC 9554 using Zstd

Up to 1.43x higher HammerDB MySQL OLTP Perf/W with 5th Gen Intel Xeon Platinum 8592+ using integrated vs. 4th Gen AMD EPYC 9554.

1.84x average general compute performance gain with 5th Gen Intel Xeon Platinum 8592+ processor over 3rd Gen Intel Xeon Platinum 8380

2.8x average general compute performance gain with 5th Gen Intel Xeon Platinum 8592+ processor over Intel Xeon Platinum 8180

Up to 1.23x higher floating point throughput performance on 5th Gen Intel Xeon Platinum 8592+ processor vs. prior generation.

Up to 1.11x higher floating point speed performance on 5th Gen Intel Xeon Platinum 8592+ processor vs. prior generation.

Up to 1.19x higher average Industry Standard performance on 5th Gen Intel Xeon Scalable platform vs. prior gen

Up to 1.26x higher Earth System Model HPC performance on 5th Gen Intel Xeon Scalable platform vs. prior gen Up to 1.16x higher average FSI performance on 5th Gen Intel Xeon Scalable platform vs. prior gen

Up to 1.25x higher average Life & Material Science HPC performance on 5th Gen Intel Xeon Scalable platform vs. prior gen

Up to 1.3x higher average Manufacturing HPC performance on 5th Gen Intel Xeon Scalable platform vs. prior gen

Up to 2.14x higher average HPC Performance and 1.56x higher average HPC Performance/Watt with Intel® Xeon® Platinum 8592+ across a broad set of HPC codes vs. prior generation Intel Xeon Platinum 8380

Up to 1.71x higher Stream Triad performance on 5th Gen Intel Xeon Scalable platform vs. 3rd Gen Intel Xeon Platform

Up to 1.86x higher HPL performance on 5th Gen Intel Xeon Scalable platform vs. 3rd Gen Intel Xeon Platform

Up to 1.92x higher HPCG performance on 5th Gen Intel Xeon Scalable platform vs. 3rd Gen Intel Xeon Platform

Up to 2x higher OpenFOAM performance on 5th Gen Intel Xeon Scalable platform vs. 3rd Gen Intel Xeon Platform

Up to 2x higher ANSYS Fluent performance on 5th Gen Intel Xeon Scalable platform vs. 3rd Gen Intel Xeon Platform

Up to 2.36x higher Ansys LS-DYNA performance on 5th Gen Intel Xeon Scalable platform vs. 3rd Gen Intel Xeon Platform

Up to 1.98x higher WRF performance on 5th Gen Intel Xeon Scalable platform vs. 3rd Gen Inel Xeon Platform

Up to 2.3x higher GROMACS performance on 5th Gen Intel Xeon Scalable platform vs. 3rd Gen Intel Xeon Platform

Up to 3x higher NAMD performance on 5th Gen Intel Xeon Scalable platform vs. 3rd Gen Intel Xeon Platform

Up to 1.92x higher Binomial Options performance on 5th Gen Intel Xeon Scalable platform vs. 3rd Gen Intel Xeon Platform

Up to 1.66x higher Monte Carlo performance on 5th Gen Intel Xeon Scalable platform vs. 3rd Gen Intel Xeon Platform

Up to 2.31x higher Black Scholes performance on 5th Gen Intel Xeon Scalable platform vs. 3rd Gen Intel Xeon Platform

Up to 3x higher LAMMPS performance on 5th Gen Intel Xeon Scalable platform vs. 3rd Gen Intel Xeon Platform

Up to 1.20x higher FSI Perf/W with 5th Gen Intel Xeon Platinum 8592+ using integrated Intel IAA vs. 4th Gen AMD EPYC 9554.

Up to 1.43x higher performance with 2-socket 5th Gen Intel Xeon Gold 6538N vs. 3rd Gen Intel Xeon Processor on CDN Live with Varnish Enterprise [48k connections]

Up to 2.17x higher NGINX TLS Handshake performance per core with 5th Gen Intel Xeon Platinum 8592+ with integrated QAT vs OOB software

Up to 1.73x higher NGINX TLS Handshake performance with 5th Gen Intel Xeon Platinum 8592+vs. 3rd Gen Intel Xeon Platinum 8380

Up to 7.92x higher NGINX TLS Handshake performance with 5th Gen Intel Xeon Platinum 8592+ vs. 1st Gen Intel Xeon Platinum 8180

Up to 3.85x higher NGINX TLS Handshake performance per core with 5th Gen Intel Xeon Platinum 8592+ with integrated QAT vs. 1st Gen Intel Xeon Platinum 8180

Up to 1.24x higher Perf/W for small packet random read with 5th Gen Intel Xeon Platinum 8592+ using integrated Intel DSA vs. 4th Gen Intel Xeon Platinum 8480+ using integrated Intel DSA

Up to 2.26x perf and 2.15x higher Perf/W for large packet sequential read with 5th Gen Intel Xeon Platinum 8592+ using integrated Intel DSA vs. oob

Up to 1.90x perf and 1.80x higher Perf/W for small packet random read with 5th Gen Intel Xeon Platinum 8592+ using integrated Intel DSA vs. oob

Up to 2.85x higher IOPs and up to 65% latency reduction for large packet sequential read with 5th Gen Intel Xeon Platinum 8592+ using integrated Intel DSA vs. 3rd Gen Intel Xeon Platinum 8380 oob

Up to 2.06x higher Perf/W for large packet sequential read with 5th Gen Intel Xeon Platinum 8592+ using integrated Intel DSA vs. 3rd Gen Intel Xeon Platinum 8380 oob

Up to 2.45x higher IOPs and up to 59% latency reduction for small packet random read with 5th Gen Intel Xeon Platinum 8592+ using integrated Intel DSA vs. 3rd Gen Intel Xeon Platinum 8380 oob

Up to 1.82x higher Perf/W for small packet random read with 5th Gen Intel Xeon Platinum 8592+ using integrated Intel DSA vs. 3rd Gen Intel Xeon Platinum 8380

Up to 3.52x higher IOPs and up to 71% latency reduction for large packet sequential read with 5th Gen Intel Xeon Platinum 8592+ using integrated Intel DSA vs. 1st Gen Intel Xeon Platinum 8180 oob

Up to 3.19x higher IOPs and up to 69% latency reduction for small packet random read with 5th Gen Intel Xeon Platinum 8592+ using integrated Intel DSA vs. 1st Gen Intel Xeon Platinum 8180 oob

Up to 2.09x higher Perf/W for large packet sequential read with 5th Gen Intel Xeon Platinum 8592+ using integrated Intel DSA vs. 1st Gen Intel Xeon Platinum 8180 oob

Up to 1.68x higher Perf/W for small packet random read with 5th Gen Intel Xeon Platinum 8592+ using integrated Intel DSA vs. 1st Gen Intel Xeon Platinum 8180 oob

Up to 2.17x higher IOPs and up to 54% latency reduction for small packet random read with 5th Gen Intel Xeon Platinum 8592+ using integrated Intel DSA vs. 4th Gen AMD EPYC 9554 oob.

Up to 1.92x higher Perf/W for large packet sequential read with 5th Gen Intel Xeon Platinum 8592+ using integrated Intel DSA vs. 4th Gen AMD EPYC 9554 oob.

Up to 1.75x higher Perf/W for small packet random read with 5th Gen Intel Xeon Platinum 8592+ using integrated Intel DSA vs. 4th Gen AMD EPYC 9554 oob.

Up to 1.85x higher NGINX TLS Handshake performance per core with 5th Gen Intel Xeon Platinum 8592+ with integrated QAT vs. 4th Gen AMD EPYC 9554 OOB

Up to 11% higher virtual machine performance on 5th Gen Intel Xeon Scalable platform with TDX vs. 4th Gen Intel Xeon Scalable platform (no TDX) on integer, floating point and BERT-large

Up to 1.17x higher performance/watt at 40% load and up to 1.11x higher performance/watt at max load on 5th Gen Intel Xeon Scalable processors with OPM compared to 4th Gen Intel Xeon with OPM on power efficiency workload

Up to 2.53x higher performance/watt at 40% load and up to 1.8x higher performance/watt at max load on 5th Gen Intel Xeon Scalable processors with OPM compared to 3rd Gen Intel Xeon on power efficiency workload

Up to 1.38x higher performance/watt at ~50% load and up to 1.26x higher performance/watt at max load on 5th Gen Intel Xeon Scalable processors with OPM compared to 4th Gen Intel Xeon with OPM

Up to 1.66x higher performance/watt at 50% load and up to 1.5x higher performance/watt at max load on 5th Gen Intel Xeon Scalable processors with OPM compared to 3rd Gen Intel Xeon on MySQL database

Up to 1.41x higher performance/watt at ~45% load and up to 1.28x higher performance/watt at max load on 5th Gen Intel Xeon Scalable processors with OPM compared to 4th Gen Intel Xeon with OPM on ResNet50

Up to 4.4x higher performance/watt at 45% load and up to 3.4x higher performance/watt at max load on 5th Gen Intel Xeon Scalable processors with OPM compared to 3rd Gen Intel Xeon on MySQL database

Up to 1.18x higher performance/watt at ~50% load and up to 1.26x higher performance/watt at max load on 5th Gen Intel Xeon Scalable processors with OPM compared to 4th Gen Intel Xeon with OPM on WordPress

Up to 1.32x higher performance/watt at 50% load and up to 1.56x higher performance/watt at max load on 5th Gen Intel Xeon Scalable processors with OPM compared to 3rd Gen Intel Xeon on WordPress

Claim: 5th Gen Intel Xeon delivers up to 41% lower TCO than the 4th Gen AMD Epyc while running a real-time Natural Language Processing inference (BERT-Large) workload.

Based on 5th Gen Intel Xeon delivers up to a 2.22x faster than the 4th Gen AMD Epyc while running a real-time Natural Language Processing inference (BERT-Large) workload. This performance drives a fleet a reduction from 50 to 23 servers which, over 4 years, saves: 1205.2 MWH of energy, 510,912 kgCo2 emissions, and $883k of cost.

Intel Xeon 8592+: 1-node, 2x Intel(R) Xeon(R) Platinum 8592+, 64 cores, HT On, Turbo On, NUMA 2, Total Memory 1024GB (16x64GB DDR5 5600 MT/s [5600 MT/s]), BIOS 2.0, microcode 0x21000161, 2x Ethernet Controller X710 for 10GBASE-T, 1x 1.7T SAMSUNG MZQL21T9HCJR-00A07, Ubuntu 22.04.2 LTS, 5.15.0-78-generic. Framework=Pytorch 2.0, IPEX=2.0, Python 3.8, AI Model=BERT Large(https://github.com/IntelAI/models/) , INT8-AMX , Real Time (BS=1) results while maintaining 130ms latency SLA, Test by INTEL as of 10/10/2023.

1-node, 2x AMD EPYC 9554 64-Core Processor, 64 cores, SMT On, Turbo On, NUMA 2, Total Memory 1536GB (24x64GB DDR5 4800 MT/s [4800 MT/s]), BIOS 1.5, microcode 0xa10113e, 2x Ethernet Controller 10G X550T, 1x 1.7T SAMSUNG MZQL21T9HCJR-00A07, Ubuntu 22.04.3 LTS, 5.15.125-0515125-generic, Framework=Pytorch 2.0, IPEX=2.0, Python 3.8, AI Model=DistilBERT Large(https://github.com/IntelAI/models/) , INT8, Real Time (BS=1) results while maintaining 5ms latency SLA. Test by INTEL as of 09/11/23.

For a 50 server fleet of AMD EPYC 9554, estimated as of October 2023:

CapEx costs: $1.36M

OpEx costs (4 year, includes power and cooling utility costs, infrastructure and hardware maintenance costs): $766.9K

Energy use in kWh (4 year, per server): 50021, PUE 1.6

Other assumptions: utility cost $0.1/kWh, kWh to kg CO2 factor 0.42394

For a 23 server fleet of 5th Gen Xeon 8592+ as of October 2023

CapEx costs: $878K

OpEx costs (4 year, includes power and cooling utility costs, infrastructure and hardware maintenance costs): $367.3K

Energy use in kWh (4 year, per server): 56344, PUE 1.6

Other assumptions: utility cost $0.1/kWh, kWh to kg CO2 factor 0.42394

Costs based on Intel estimates and information from thinkmate.com as of October 2023.

Up to 1.3x higher Java Throughput within a given SLA with 5th Gen Intel Xeon Platinum 8592+ processor vs prior generation

Up to 1.18x higher Java Performance per watt with 5th Gen Intel Xeon Platinum 8592+ processor vs prior generation

Up to 1.33x higher Java Performance per watt within a given SLA with 5th Gen Intel Xeon Platinum 8592+ processor vs prior generation

Up to 1.85x higher Java Throughput with 5th Gen Intel Xeon Platinum 8592+ processor 3-year refresh platform

Up to 2x higher Java Throughput within a given SLA with 5th Gen Intel Xeon Platinum 8592+ processor vs 3-year refresh platform

Up to 1.54x higher Java Performance per watt with 5th Gen Intel Xeon Platinum 8592+ processor 3-year refresh platform

Up to 1.64x higher Java Performance per watt within a given SLA with 5th Gen Intel Xeon Platinum 8592+ processor vs 3-year refresh platform

Up to 2.9x higher Java Throughput with 5th Gen Intel Xeon Platinum 8592+ processor vs 5-year refresh platform

Up to 3x higher Java Throughput within a given SLA with 5th Gen Intel Xeon Platinum 8592+ processor vs 5-year refresh platform

Up to 1.8x higher Java Performance per watt with 5th Gen Intel Xeon Platinum 8592+ processor vs 5-year refresh platform

Up to 1.97x higher Java Performance per watt within a given SLA with 5th Gen Intel Xeon Platinum 8592+ processor vs 5-year refresh platform

Up to 1.75x higher performance on cloud native benchmark [CloudXPRT Web Microservices] with 5th gen Intel Xeon Platinum 8592+ processor vs. 3-year-old platform.

Up to 2.93x higher performance on cloud native benchmark [CloudXPRT Web Microservices] with 5th gen Intel Xeon Platinum 8592+ processor vs. 5-year-old platform.

Up to 1.04x higher performance-per-watt on cloud native benchmark [CloudXPRT Web Microservices] with 5th gen Intel Xeon Platinum 8592+ processor vs. prior gen.

Up to 1.09x higher performance-per-watt on cloud native benchmark [CloudXPRT Web Microservices] with 5th gen Intel Xeon Platinum 8592+ processor vs. 3-year-old platform.

Up to 1.6x higher performance-per-watt on cloud native benchmark [CloudXPRT Web Microservices] with 5th gen Intel Xeon Platinum 8592+ processor vs. 5-year-old platform.

5th Gen Intel Xeon Platinum 8592+ can process up to 1.76x higher transactions per second vs. 3rd Gen on Wordpress TLS 1.3 with CryptoNI

5th Gen Intel Xeon Platinum 8592+ can process up to 3.4x higher transactions per second vs. 5-year refresh platform on Wordpress TLS 1.3 with CryptoNI

Up to 1.73x higher virtualization performance with the 5th Gen Intel® Xeon® Platinum 8592+ processor vs. 3-year refresh platform

Up to 1.08x higher Server Side Java SLA throughput Perf/W with 5th Gen Intel Xeon Platinum 8592+ vs. 4th Gen AMD EPYC 9554.