Connections
Dmux Connections define Source to Sink connection. One Dmux instance can run multiple dmux connections.
Configuration options change per connection based on nature of Source and Sink.
Logging is a common configuration option across all connections.
The following Connections are supported:
kafka_http¶
This the most simplest connection that connects KafkaSource to HttpSink.
It expects KafkaSource to have KeydMessage - (Key + Value). ModuloHash will happen on Key if distributor type is Hash (default).
HttpSink will create the following url : http://endpoint/{topic}/{partition}/{key}/{offset}. The payload will be byte[] value from Kafka value. Note Dmux does no processing on value.
Any non 2xx response code will be considered as failure by HTTPSink and retried. Currently retry policy is fixed delay retry. You can configure retry_interval using config.
If batch_size is specified for batching the URL will change to:
http://endpoint/{topic}?batch=(partition1,key1,offset1~partition2,key2,offset2..) The query param batch was added to help make debugging easier.
Batching in Dmux is implemented differently, To understand check Batching Tab.
Payload in batch is encoded to convert 2d byte[] to 1d byte[], without any serialization.
Encoding Format¶
TODO
Java Lib¶
Java library exist to help decode : https://github.com/flipkart-incubator/go-dmux/tree/master/java/godmux-tools
You can add the below POM dependency. (TODO) move this to release version
<dependency>
<groupId>com.flipkart.godmux.tools</groupId>
<artifactId>godmux-tools</artifactId>
<version>1.0-SNAPSHOT</version>
</dependency>
kafka_foxtrot¶
This is an extension of kafka_http connection with customization specific to ingest to foxtrot. https://github.com/Flipkart/foxtrot
Across transact we use foxtrot for monitoring and debugging and have client lib which write to Kafka in a specific message format needed by Foxtrot.
TODO Add link to Foxtrot ingestion lib used in transact
This connection works on this structure of Kafka message where key = tableName and value is foxtrot payload.
Check https://github.com/Flipkart/foxtrot/wiki/Event-Ingestion to see foxtrot ingestion API.
URL expected to be configured http://foxtrot.com:10000/foxtrot/v1/document/KEY_NAME
KEY_NAME will be replaced by tableName.
batching is also supported and customised to format needed by foxtrot ingestion api.
query params are added to both Single URL and Batch URL which represent topicName, offset, partition to simplify debugging.