UDF Writing Guide#

Overview
How-To Guide for Writing UDF

Overview#

An User Defined Function (UDF) is a chunk of user code that can transform video frames and/or manipulate metadata. For example, a UDF can act as filter, preprocessor, classifier or a detector. These User Defined Functions can be developed in C++ or Python. EVAM provides a GStreamer plugin - udfloader using which users can configure and load arbitrary UDFs. These UDFs are then called once for each video frame.

Gstreamer udfloader plugin supports loading and executing of native(C++) and python UDFs. Metadata/inference results generated by the UDFs are attached to GST buffer as GVAJSONMeta. A single instance of udfloader plugin can load and execute a sequence of UDFs, thus enabling chaining multiple UDFs together. Pipelines can also have multiple udfloader elements.

How-To Guide for Writing UDF#

This document describes step-wise instruction for writing a User defined Function (UDF) in C++/Python to be used with the gst udfloader element.

Ability to run arbitrary User Defined Function (UDFs) as a part of the GStreamer pipeline is one of the cardinal feature of EVAM. It enables users to adjoin any pre-processing or post-processing logic to the video analytics pipeline. Currently UDFs can be implemented using following languages.

C++ (It is also called Native UDF)
Python

The order in which the UDFs are defined in pipeline definition file is the order in which data will flow across them. Currently there is no support for demux/mux the data flow to/fro the UDFs.

Steps for Writing Native (C++) UDFs#

Every UDF writing has two major part to it.

Writing actual pre/post processing logic using EIS exposed APIs.
Adding configuration details to the gst udfloader element for deploying it

EIS APIs for Writing Native UDFs (C++)#

There are three APIs defined semantically to add the pre/post processing logic. These APIs must be implemented as method of a user defined class inherited from the Udf class named BaseUdf.

Initialization and DeInitialization

    class DummyUdf : public BaseUdf {
            public:
                DummyUdf(config_t* config) : BaseUdf(config) {
                    //initialization Code can be added here
                };

                ~DummyUdf() {
                    // Any de-initialization logic to be added here
                };
    };

The DummyUdf in the above code snippet is the user defined class and the constructor of the class initialize the UDF’s specific data-structure. The only argument passed to this function is config which depicts configuration details mentioned in the pipeline definition file.

Processing the Actual Data

Following is the API to utilize the ingested input:
```
UdfRetCode
process(cv::Mat& frame, cv::Mat& outputFrame, msg_envelope_t* meta) override {
    // Logic for processing the frame & returning the inference result.
}
```
This function is a override method which user need to define in its UDF file.

Following are the argument details:
- Argument 1(cv::Mat &frame): It represents the input frame for inference.
- Argument 2(cv::Mat &outputFrame): It represents the modified frame by the user. This can be used if user need to pass a modified frame forward.
- Argument 3(msg_envelope_t* meta): It represents the inference result returned by UDF. The user need to fill the msg_envelope_t structure.
Following are the return code details:
- UdfRetCode: Return the appropriate macro as mentioned.
- UDF_OK - UDF has processed the frame gracefully.
- UDF_DROP_FRAME - The frame passed to process function need to be dropped.
- UDF_ERROR - it should be returned for any kind of error in UDF.
Linking Udfloader and Custom UDFs

The initialize_udf() function need to defined as follows to create a link between UdfLoader module and respective UDF. This function ensure UdfLoader to call proper constructor and process() function of respective UDF.
```
extern "C" {
    void *initialize_udf(config_t *config) {
        DummyUdf *udf = new DummyUdf(config);
        return (void *)udf;
    }
}
```
The “DummyUdf” is the class name of the user defined custom UDF.

Steps for Writing Python UDFs#

This section describes the process of writing a Python UDF. As discussed in the aforementioned scenario, it also has two aspects to it.

Writing the actual UDF.
Adding configuration details to the gst udfloader element for deploying it

Python APIs for writing UDF#

Initialization

In case of Python the initialization callback is also same as the native case. User must create a Udf class and the __init__() function defined in class act as a initialization routine for custom UDF. Following is the dummy constructor code:
```
class Udf:
"""Example UDF
"""
def __init__(self):
    """Constructor
    """
    # Add the initialization code in this method.
```
Process Actual Data

The API used to process the actual frame is as follows:
```
process(self, frame, metadata):
    # Process the frame in this method
    # metadata can be used to return inference result
```
Argument:

frame: Image frame in numpy’s ndarray format

metadata: An empty dictionary. Inference results can be inserted in this data structure.

Return value:

This function returns three values.

1st Value : Represents if the frame Need to be dropped or Not. It is boolean in nature. In case of failure user can return True in this positional return value.

2nd Value : It represents the actual modified frame if at all it has been modified. Hence the type is numpy’s ndarray. If the frame is not modified user can return a None in this place.

3rd Value : Metadata is returned in this place. Hence the type is dict. In general user can return the passed argument as part of this function.