Edge Developer Toolbox Developer Guide

ID 783775
Date 06/07/2024
Version 24.05
Confidential
Document Table of Contents
Document Table of Contents
Edge Developer Toolbox What You Can Do With Edge Developer Toolbox Get Started Create AI Model (Premium Functionality) Model Optimization Develop AI Application Benchmark Hardware Troubleshooting Guide 24.05 Release Notes Frequently Asked Questions Glossary Related Documentation Notices and Disclaimers
Edge Developer Toolbox
What You Can Do With Edge Developer Toolbox
Get Started
Get Started Access the Edge Developer Toolbox Edge Developer Toolbox Interface Overview The Edge Developer Toolbox Sample Applications
Access the Edge Developer Toolbox
Edge Developer Toolbox Interface Overview
Edge Developer Toolbox Interface Overview My Workspace My Dashboard
My Workspace
My Dashboard
The Edge Developer Toolbox Sample Applications
Create AI Model (Premium Functionality)
Create AI Model (Premium Functionality) Create and Train a New Machine Learning Model Validating a Model Using the Models
Create and Train a New Machine Learning Model
Validating a Model
Using the Models
Model Optimization
Model Optimization Launch Model Optimization and Import an AI Model Quantize Model Select Target Environment and Benchmarking Benchmark Another Model or Target Architecture and Compare Performance Use Optimized Model in Edge Developer Toolbox
Launch Model Optimization and Import an AI Model
Launch Model Optimization and Import an AI Model Import an AI Model from Open Model Zoo Import a Custom AI Model Import AI model from Hugging Face Convert and Optimize Model with OpenVINO™ Toolkit Import Test Dataset
Import an AI Model from Open Model Zoo
Import a Custom AI Model
Import AI model from Hugging Face
Convert and Optimize Model with OpenVINO™ Toolkit
Import Test Dataset
Quantize Model
Select Target Environment and Benchmarking
Benchmark Another Model or Target Architecture and Compare Performance
Use Optimized Model in Edge Developer Toolbox
Develop AI Application
Develop AI Application Learn Computer Vision and Generative AI App Development Quick Start with a Reference Implementation in VS Code Import Custom Artifacts for Modular Solution Development Import, Edit, and Build Solutions in VS Code
Learn Computer Vision and Generative AI App Development
Quick Start with a Reference Implementation in VS Code
Import Custom Artifacts for Modular Solution Development
Import Custom Artifacts for Modular Solution Development Bring Your Own Application Artifacts
Bring Your Own Application Artifacts
Import, Edit, and Build Solutions in VS Code
Benchmark Hardware
Benchmark Hardware Launch an Application from the Edge Developer Toolbox Portal Select the Processor to Launch the Application Access Streaming Telemetry Dashboards View Integrated Telemetry Dashboard for Processors with Graphics Accelerators View Start Time and Execution Time View Static Telemetry or Historic Telemetry Dashboard Metrics and Descriptions Large Language Model (LLM) Sample Application
Launch an Application from the Edge Developer Toolbox Portal
Select the Processor to Launch the Application
Access Streaming Telemetry Dashboards
View Integrated Telemetry Dashboard for Processors with Graphics Accelerators
View Start Time and Execution Time
View Static Telemetry or Historic Telemetry Dashboard
Metrics and Descriptions
Large Language Model (LLM) Sample Application
Troubleshooting Guide
24.05 Release Notes
Frequently Asked Questions
Glossary
Related Documentation
Notices and Disclaimers
Edge Developer Toolbox
What You Can Do With Edge Developer Toolbox
Get Started
Create AI Model (Premium Functionality)
Model Optimization
Develop AI Application
Benchmark Hardware
Troubleshooting Guide
24.05 Release Notes
Frequently Asked Questions
Glossary
Related Documentation
Notices and Disclaimers

Large Language Model (LLM) Sample Application

Note:

This feature is only available to premium customers, and it may not be shown on your page.

Tokens Per Second (TPS) metrics are now available in the Customer Support LLM sample application’s telemetry dashboard. To view these metrics, interact with the Personal Q&A Assistant. The TPS panels in the dashboard will only populate while using this feature.

  1. Navigate to the Develop AI Application section and choose the LLM Customer Support on the Edge sample and select Benchmark on Intel Hardware:

    Benchmark on Intel® Hardware

  2. Once launched you will see the sample running and under output choose the third icon (square with the arrow) and select the link that shows up. It will bring up the Assistant in a new tab:

    Customer Support on the Edge

  3. Click the Personal Q&A Assistant tab, enter a prompt, and submit it. Once the Assistant has answered the prompt, you can view the tokens per second that was obtained for the response.

    Personal Q&A Assistant

  4. Open the telemetry dashboard as you normally would, and you will see the TPS panels in addition to the system telemetry panels:

    TPS Panels
Troubleshooting Guide