Installing Scalyr Kafka Sink Connector (Distributed)

Prereqs:

• Kafka (distributed)
• Kafka Connect
• Java 8+
• maven
• git
• unzip

Set Environment Variables

export $KAFKA_HOME = <installation directory of kafka>
export $GIT_PATH = <place where you will download scalyr sink git repo>
export $KAFKA_CONNECT_SCALYR_HOME = $GIT_PATH/kafka-connect-scalyr
export $MVN_BUILD_PATH = $KAFKA_CONNECT_SCALYR_HOME/target/components/packages
export $KAFKA_PLUGIN_PATH = usr/local/share/kafka/plugins
export $KAFKA_SCALYR_SINK_CONFIG = usr/local/share/kafka/plugins/scalyr-kafka-connect-scalyr-sink-0.1/etc

Extract Scalyr Plugin 

1. Clone the Kafka Connect Scalyr repository: 

cd $GIT_PATH

git clone https://github.com/scalyr/kafka-connect-scalyr 

2. Build the connector. The built connector is located in target/components/packages/scalyr-kafka-connect-scalyr-sink-0.1.zip.

cd $KAFKA_CONNECT_SCALYR_HOME
mvn clean package

3. Unzip and copy to the Kafka connect plugin path

mkdir -p $KAFKA_PLUGIN_PATH

unzip $MVN_BUILD_PATH/kafka-connect-scalyr-sink-0.1.zip -d $KAFKA_PLUGIN_PATH

We will configure the Scalyr plugin in the next section