Data streams

Data streams can be used to produce and consume data to and from external data sources. This is useful for integrating with other systems, such as Kafka, to allow data synchronization between different instances of the Director or to read external selection input data.

Configuration

Currently, only Kafka data streams are supported. The addresses of the Kafka brokers to connect to are configured in integration.kafka.bootstrapServers:

confcli integration.kafka.bootstrapServers
{
    "bootstrapServers": [
        "kafka-broker-host:9096"
    ]
}

These Kafka brokers can then be interacted with by configuring data streams in the services.routing.dataStreams section of the configuration:

confcli services.routing.dataStreams
{
    "dataStreams": {
        "incoming": [],
        "outgoing": []
    }
}

Incoming data streams

incoming is a list of data streams that the Director will consume data from. An incoming data stream defines the following properties:

• name: The name of the data stream. This is used to identify the data stream in the configuration and in the logs.
• source: The source of the data stream. Currently, the only supported source is kafka, which means that the data will be consumed from the Kafka broker configured in integration.kafka.bootstrapServers.
• target: The target of the data consumed from the stream. Currently, the only supported target is selectionInput, which means that the consumed data will be stored as selection input data.
• kafkaTopics: A list of Kafka topics to consume data from.

The following configuration will make the Director consume data from the Kafka topic selection_input from the Kafka broker configured in integration.kafka.bootstrapServers and store it as selection input data.

confcli services.routing.dataStreams.incoming
{
    "incoming": [
        {
            "name": "incomingDataStream",
            "source": "kafka",
            "kafkaTopics": [
                "selection_input"
            ],
            "target": "selectionInput"
        }
    ]
}

Outgoing data streams

outgoing is a list of data streams that the Director will produce data to. An outgoing data stream defines the following properties:

• name: The name of the data stream. This is used to identify the data stream in the configuration, in a Lua context and in the logs.
• type: The type of the data stream. Currently, the only supported type is kafka, which means that the data will be produced to the Kafka broker configured in integration.kafka.bootstrapServers.
• kafkaTopic: The Kafka topic to produce data to.

Example of an outgoing data stream that produces to the Kafka topic selection_input:

confcli services.routing.dataStreams.outgoing
{
    "outgoing": [
        {
            "name": "outgoingDataStream",
            "type": "kafka",
            "kafkaTopic": "selection_input"
        }
    ]
}

Data can be sent to outgoing data streams from a Lua function, see Data stream related functions for more information.