Tutorials Advanced#
Get tensor vector data#
EVAM supports extracting tensor data (as python lists) from pipeline models by making use of DLStreamer’s add-tensor-data=true property for gvametaconvert element. Depending upon how gva elements are stacked and whether inference is done on entire frame or on ROIs (Region Of Interest), the metadata json is structured accordingly. Tensor outputs are vector representation of the frame/roi. It can be used by reference applications for various usecases such as image comparison, image description, image classification using custom model, etc. To learn more about the property, read here.
Change the pipeline as follows (edit the path to model xml and proc json to your needs) -
"pipeline": "{auto_source} name=source ! decodebin ! gvadetect model=/home/pipeline-server/omz/intel/person-vehicle-bike-detection-2004/FP32/person-vehicle-bike-detection-2004.xml model-proc=/opt/intel/dlstreamer/samples/gstreamer/model_proc/intel/person-vehicle-bike-detection-2004.json ! gvainference model=/home/pipeline-server/resources/models/classification/resnet50/FP16/resnet-50-pytorch.xml inference-region=1 ! queue ! gvametaconvert add-tensor-data=true name=metaconvert ! gvametapublish ! appsink name=destination ",
Notice the property add-tensor-data for the dlstreamer element gvametaconvert is set to true The. Add the following parameters [EVAM_WORKDIR]/configs/default/config.json as shown below to publish the tensor data along with all other metadata via MQTT
Add the below section in [EVAM_WORKDIR]/configs/default/config.json under pipelines section. Start EVAM with this configuration. Once started, hit a simple curl command as shown below
Note: Follow instruction in the Prerequisite section to create a sample configuration file.
"mqtt_publisher": {
"host": "<mqtt_broker_address>",
"port": 1883,
"publish_frame": false
},
curl localhost:8080/pipelines/user_defined_pipelines/pallet_defect_detection -X POST -H 'Content-Type: application/json' -d '{
"source": {
"uri": "file:///home/pipeline-server/resources/videos/person-bicycle-car-detection.mp4",
"type": "uri"
}
}'
You can check the vector output by subscribing to mqtt. You can check this document on how to configure and start mqtt subscriber.
Here’s what a sample metadata for a frame looks like (some data deleted to keep size small).
{
"objects": [
{
"detection": {
"bounding_box": {
"x_max": 0.6305969953536987,
"x_min": 0.38808196783065796,
"y_max": 0.8155133128166199,
"y_min": 0.5354097485542297
},
"confidence": 0.5702379941940308,
"label": "vehicle",
"label_id": 0
},
"h": 121,
"region_id": 146,
"roi_type": "vehicle",
"tensors": [
{
"confidence": 0.5702379941940308,
"label_id": 0,
"layer_name": "labels\\boxes",
"layout": "ANY",
"model_name": "torch-jit-export",
"name": "detection",
"precision": "UNSPECIFIED"
},
{
"data": [
1.1725661754608154,
-0.46770259737968445,
<omitted data>
-0.8607546091079712,
1.1693058013916016
],
"dims": [
1,
1000
],
"layer_name": "prob",
"layout": "ANY",
"model_name": "torch_jit",
"name": "inference_layer_name:prob",
"precision": "FP32"
}
],
"w": 186,
"x": 298,
"y": 231
},
{
"detection": {
"bounding_box": {
"x_max": 0.25753622874617577,
"x_min": 0.017545249313116074,
"y_max": 0.39748281240463257,
"y_min": 0.12764209508895874
},
"confidence": 0.5328243970870972,
"label": "vehicle",
"label_id": 0
},
"h": 117,
"region_id": 147,
"roi_type": "vehicle",
"tensors": [
{
"confidence": 0.5328243970870972,
"label_id": 0,
"layer_name": "labels\\boxes",
"layout": "ANY",
"model_name": "torch-jit-export",
"name": "detection",
"precision": "UNSPECIFIED"
},
{
"data": [
0.5690383911132813,
-0.5517100691795349,
<omitted data>
-0.8780728578567505,
1.1474417448043823
],
"dims": [
1,
1000
],
"layer_name": "prob",
"layout": "ANY",
"model_name": "torch_jit",
"name": "inference_layer_name:prob",
"precision": "FP32"
}
],
"w": 184,
"x": 13,
"y": 55
}
],
"resolution": {
"height": 432,
"width": 768
},
"timestamp": 0
}
Cross stream batching#
EVAM supports grouping multiple frames in single batch during model processing. batch-size is an optional parameter to be used which specifies the number of input frames grouped together in a single batch. In the below example, the model processes 4 frames at a time.
"pipeline": "{auto_source} name=source ! decodebin ! videoconvert ! gvadetect name=detection batch-size=4 model-instance-id=1 ! queue ! gvawatermark ! gvafpscounter ! gvametaconvert add-empty-results=true name=metaconvert ! gvametapublish name=destination ! appsink name=appsink",
Choosing the right batch size:
Real time applicationsKeep the batch-size small to minimize the latency. A larger batch size may cause the initial frames to wait until the batch is completely filled before the model begins processing. Also, large batch size means higher memory utilizationHigh throughputKeep the batch-size large to maximize the throughput. Some hardware are suited to process large number of frames in parallel, thus reducing overall time required to process all the frames.
Note In a multi stream pipeline with a shared model instance, frames can be grouped into a single batch either from multiple pipelines or exclusively from one pipeline, depending on the timing of arrival of frames from the pipelines.
To verify the effect of batch-size you can check the memory utilization of docker by using command docker stats. The memory utilization increases when we load multiple frames in one batch. The stats may vary depending on the underlying hardware.
You can use the following curl command to start the pipeline -
curl http://localhost:8080/pipelines/user_defined_pipelines/pallet_defect_detection -X POST -H 'Content-Type: application/json' -d '{
"source": {
"uri": "file:///home/pipeline-server/resources/videos/warehouse.avi",
"type": "uri"
},
"destination": {
"metadata": {
"type": "file",
"path": "/tmp/results.jsonl",
"format": "json-lines"
},
"frame": {
"type": "rtsp",
"path": "pallet-defect-detection"
}
},
"parameters": {
"detection-properties": {
"model": "/home/pipeline-server/resources/models/geti/pallet_defect_detection/deployment/Detection/model/model.xml",
"device": "CPU"
}
}
}'
docker stats with batch-size as 1, no of streams as 1
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
f4355ac7a42e edge-video-analytics-microservice 283.11% 322.6MiB / 31.18GiB 1.01% 42.8kB / 2.69kB 0B / 573kB 36
docker stats with batch-size as 16, no of streams as 1
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
6a3ccbc9fb44 edge-video-analytics-microservice 281.32% 811.7MiB / 31.18GiB 2.54% 42.5kB / 2.83kB 0B / 0B 37
docker stats with batch-size as 16, no of streams as 4
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
f842a3f617c8 edge-video-analytics-microservice 1169.10% 462.7MiB / 31.18GiB 1.45% 46.3kB / 4.18kB 0B / 352kB 55
docker stats with batch-size as 16, no of streams as 4
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
5b1c3b35ddfe edge-video-analytics-microservice 1170.64% 999.2MiB / 31.18GiB 3.13% 45.4kB / 4.05kB 0B / 123kB 55