Prerequisites

In this section, we will discuss what you will need to follow this tutorial and some assumptions that we make in terms of infrastructure. as well as some resources that help with Kafka  

High-level solution architecture

What you will need 

1. DataSet Account
   (If you don't have one, you can signup for a free trial here)
2. Distributed Kafka Cluster.
   (3 nodes, 1 Zookeeper instance. If you don't have one, you can see how to build a quick one in GCP here) You can also install one in Kubernetes here. 
3. An application running with Filebeat.
   (In this case, we are using a single Nginx server with FileBeat's Nginx plugin, but you can use these techniques to stream any type of data you want.)

Continue to Filebeat Configuration