Metro AI Suite Device Qualification Tool#

Media, AI and end-to-end pipeline benchmark tools for partners who wish to capture the video analytics performance of the system and co-market with Intel.

Overview#

Intel® Edge Software Device Qualification (Intel® ESDQ) for Metro AI Suite allows customers to run an Intel-provided test suite on the target system, enabling partners to qualify their platform as an Edge AI ready device.

The following information is specific to the Intel® ESDQ for Metro AI Suite package. For documentation on the Intel® ESDQ CLI binary, refer to Intel® Edge Software Device Qualification (Intel® ESDQ) CLI Overview.

Target System Requirements#

  • Intel® Core™:

    • 11th, 12th, or 13th Generation Embedded processors

    • Core™ Ultra Processors

    • 12th, or 13th Generation Desktop processors with Intel® Arc™ A380 Graphics

  • Intel Atom®: X7000 Series and N-series processors

  • Intel® Xeon®:

    • 4th, or 5th Generation Scalable Processors

    • Optional Pairing: Intel® Arc™ A380, Intel® Arc™ A550, Intel® Arc™ A770 Graphics

Other Prerequisites#

  • Operating System:

    • Ubuntu* 24.04.2 Desktop (fresh installation) for Intel® Core™ and Intel® Core™ Ultra platforms

    • Ubuntu* 24.04.2 Server (fresh installation) for Intel® Xeon® platform

  • System Requirements:

    • At least 160 GB of disk space

    • At least 16 GB of memory

  • Internet access

  • Superuser access (sudo)

How It Works#

The Metro AI Suite Test Module interacts with the Intel® ESDQ CLI through a common test module interface (TMI) layer, which is part of the Intel® ESDQ binary.

After installation, user can run the test cases with a single command and test results will be stored in the output file. Intel® ESDQ generates a complete test report in HTML format and detailed logs packaged as one ZIP file, which you can email to the Intel® ESDQ support team at edge.software.device.qualification@intel.com.

Metro AI Suite Test Module#

The Metro AI Suite test module is the validation framework for Metro AI Suite. This module validates the installation of software packages and measures the performance of the platform using the following benchmarks:

  • OpenVINO™ based neural network model benchmarks

  • Media performance benchmark

  • Video pipeline benchmark

  • Memory benchmarks

  • GPU AI frequency measurement

The following telemetry data will be collected when running benchmarks:

  • CPU Frequency

  • CPU Utilization

  • Memory Utilization

  • GPU Frequency

  • GPU EU Utilization

  • GPU VDBox Utilization

  • GPU Power

  • Package Power

OpenVINO Benchmark#

The following neural network models are benchmarked using the OpenVINO™ benchmark tool. Both latency and throughput are measured. Benchmark results are included in the ESDQ report.

  • resent-50-tf

  • ssdlite-mobilenet-v2

  • efficientnet-b0

  • yolo-v5 s

  • yolo-v8 s

  • mobilenet_v2

  • clip-vit-base-patch16

Media Performance Benchmark#

Media Performance Benchmark contains the following benchmarks:

  • Media Encode Benchmark: Encode video streams to different video codec (h264, h265) and resolution (1080p, 4K) combination using random noise as source. Measures max number of video streams can be encoded at 30 FPS.

  • Media Decode Benchmark: Decodes video streams with different video codec (h264, h265) and resolution (1080p, 4K) combination. Measures max number of video streams can be decoded at 30 FPS.

  • Media Decode + Compose Benchmark: Decode video streams with different video codec (h264, h265) and resolution (1080p, 4K) combination and composed into a video wall. Measures max number of video streams can be decoded and composed into video wall at 30 FPS

We recommend connecting 2x 4K monitors to the machine before running this benchmark, then the composed video in Media Decode + Compose benchmark will be driven to each monitor.

Video Pipeline Benchmark#

Video Pipeline Benchmarks include the following domain specific proxy pipeline benchmarks:

  • Smart NVR Pipeline: Measure the max number of AI-enabled video streams that the platform can support while keeping output frame rate equal to the input frame rate(20fps) out of a fixed number of input.

esdq_nvr

Figure 1: Smart NVR Proxy Pipeline

  • Headed Visual AI Pipeline: Measures the max number of channels the platform can run while keeping output frame rate equal to the input frame rate(30fps).

esdq_headed

Figure 2: Headed Visual AI Pipeline

  • VSaaS Gateway with Storage and AI Proxy Pipeline: Measures the max number of channels supported for the AI VSaaS Gateway pipeline while keeping the output frame rate equal to the input frame rate(30fps).

esdq_vsaas

Figure 3: VSaaS Gateway with Storage and AI Proxy Pipeline

We recommend connecting a 4K monitor to the machine before running Smart NVR pipeline and Headed Visual AI Pipeline, the composed video will be driven to the monitor.

Memory Benchmark#

The memory benchmark measures the sustained memory bandwidth based on STREAM.

AI Frequency Measurement#

The AI frequency benchmark was designed to stress the GPU for an extended period. The benchmark records the GPU frequency while it runs an inference workload using the OpenVINO™ Benchmark Tool.