Site icon IT Tutorial

Apache Kafka Producer Example With Java

Deniz Parlak

Hi, everyone in this tutorial we will make Kafka producer example with Java. There are multiple language options for writing code with Kafka producer. You can use Java, Scala, or python.

Kafka and Zookeeper must be installed on your computer before proceeding with writing the code.  You can download a virtual machine from Cloudera’s site with all of these packages installed. This eliminates the need for installation.

https://www.cloudera.com/downloads/hdp.html

 

First, let’s create a topic with CLI.

kafka-topics --create --zookeeper quickstart.cloudera:2181 --topic kafka_example --partitions 1 --replication-factor 1

Let’s check if it’s successful.

kafka-topics --list --zookeeper quickstart.cloudera:2181

I’m using Intellij to write code, but you can also use different IDEs.

First we need to create a Maven project.

 

Having Java 1.8 is important. You may see bugs in the upper versions.

Write groupID and etc. (The value you give does not matter) and click Next

Finally give a project name and click Finish.

The arrow shown in red is the xml file that is required to load dependencies. In the section indicated by the blue arrow, the package containing the classes where we will write the java codes

 

First of all, we need to set the required dependencies. To write the Kafka code, we need the kafka-client and slf4j jar files.

Let’s choose Kafka-client and go to the latest version.

Let’s take the xml code and paste it into our pom.xml file.

Next is slf4j

 

Let’s do the same process we did for kafka-client for slf4j and paste the xml code into our pom.xml file.

The final version of the xml file is as follows,

 

Let’s create a Java Class and start writing code,

 

We must create a properties for the Producer, there are 3 important conditions for these properties.

 

You can reach the detailed document for this, https://kafka.apache.org/documentation/#producerconfigs

In the code section, we make these settings with the Properties class.

String bootstrapServer="quickstart.cloudera:9092";
Properties properties =new Properties();


properties.setProperty(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,bootstrapServer);
properties.setProperty(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
properties.setProperty(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());

 

Bootstrap server, hostname and port number of kafka broker, ProducerConfig, the class we called from the kafka-client jar

After completing the settings, we need to create the object for the producer.

KafkaProducer<String, String > producer=new KafkaProducer<String, String>(properties);

We need to give topic and value for the data we will produce. We do this with ProducerRecord.

ProducerRecord<String,String> record = new ProducerRecord<String, String>("kafka_test","Hello Kafka Test Message!");

Finally, let’s send the data and close it.

producer.send(record);

producer.flush();
producer.close();

The final version of our code is as follows:

 

Before running, let’s run kafka-consumer through CLI and run the code and watch it.

kafka-console-consumer --zookeeper quickstart.cloudera --topic kafka_test

 

As shown in the picture, the data came to kafka-consumer as a result of the running code.

See you next article..

Exit mobile version