Data Store Microservice#

Data Store Microservice provides a robust on-prem storage solution offering persistent storage for metadata and Binary Large Object (BLOB) data

Overview#

Data Store Microservice provides on-prem persistent metadata stored in the InfluxDB* platform and Binary Large Object (BLOB) data stored in the MinIO* system. These storage solutions are of NoSQL nature and support video analytics and time-series analytics data store operations at the edge.

  • Programming Language: Golang*

How It Works#

Data Store Microservice provides stream-based pub-sub and req-resp filtering with retention policy enabled for data management at the edge.

The Microservice supports the following data storage types:

  • BLOBs: these are files in containers as BLOBs of data.

  • NoSQL data.

The Microservice supports the following data modalities:

  • Logs

  • Structured data: numerical and textual data with emphasis on data that is time-stamped. Intel maintains and manages the timestamps and universal time representation. Some of these timestamps could be adjusted (calibrated) using a universal clock that follows time-sensitive networking (TSN) directives.

  • Images

  • Videos

InfluxDB*#

InfluxDB is an open-source time-series data platform. It stores both Intel® Edge Insights System video and time-series analytics metadata.

MinIO*#

MinIO is a high-performance object storage system. The Edge Video Analytics Microservice data received over the EII message bus has both metadata and frame BLOB. The image handle (img_handle) in the metadata is extracted and used as a key with value, as frame BLOB. This key-value pair gets written to MinIO.

Data Store Microservice by default needs at least one database configuration (either the InfluxDB database or MinIO object storage) to be available to start executing and all database configurations specified must be available for connection. Data Store Microservice will create an instance for both the InfluxDB database handlers and MinIO database handles, and also initializes all interfaces like Pub/Sub and Req-Resp.

InfluxDB database and MinIO database server systems are started as separate containers.

Data Store Microservice requires at least one configuration, either the meta or blob configuration in the Data Store Microservice’s config.json file. For Data Store Microservice to proceed further, it requires all database servers to be up and running, which are configured in config.json. Data Store Microservice allows the database handlers to connect to a local or remote server. To change the IP of the server, add host (optional parameter) in the configuration with the host name or IP address and port of server, and port (optional parameter) with the respective port number in the configuration.

The InfluxDB database handlers and the MinIO database handlers use the “topics” key value in the “json” or “blob” keys under the “datatypes” key in the Data Store Microservice’s config.json file for each handler. This is done to fetch the metadata (video and time-series analytics) and frame details, and add them to the respective systems.

Figure 1: Architecture Diagram

Data Store Arch

Learn More#

  • Get started with the microservice using the Get Started Guide.

  • Follow step-by-step examples to become familiar with the core functionality of the microservice, in the Tutorials.