Spring Boot Kafka Tutorial

Introduction-

Kafka is an open-source distributed event streaming platform that allows you to publish and subscribe to streams of records. In this tutorial, we'll cover how to use Java with Kafka to send and receive messages.
First we will learn how to download and install Kafka in different operating systems, then we will write code in Java (Spring Boot framework) for producer and consumer. 

One of the core concepts in Kafka is the idea of a stream, which is a continuous flow of records. These records can be anything - log entries, sensor readings, user events, etc. - and are published by producers to one or more Kafka topics. Consumers can then subscribe to those topics and consume the records in real-time.

To get started with Kafka, you first need to download and install it on your machine. Kafka is available for different operating systems, including Windows, macOS, and Linux. Once you have Kafka installed, you can start using it to build real-time data pipelines.

To write code in Java for producing and consuming messages in Kafka, you can use the Spring Boot framework. Spring Boot provides a simple and easy-to-use interface for working with Kafka, making it a popular choice among developers.

For the producer, you will need to create a KafkaTemplate bean that allows you to send messages to a specific topic. You can then use this template to send messages to Kafka by calling the send() method.

For the consumer, you will need to create a KafkaListener bean that listens for messages on a specific topic. When a message is received, the listener method is called and you can process the message accordingly.

Download and Install Kafka

Download the Kafka binary from the official website: https://kafka.apache.org/downloads

Installing Kafka on Windows

1. Extract the downloaded file to a directory on your computer, such as C:\kafka

2. Open a command prompt and navigate to the Kafka directory using the cd command

3. Start the ZooKeeper server by running the following command:


bin\windows\zookeeper-server-start.bat config\zookeeper.properties

4. Open another command prompt window and start the Kafka server by running the following command: 


bin\windows\kafka-server-start.bat config\server.properties

Installing Kafka on macOS

1. Extract the downloaded file to a directory on your computer, such as /Users/your-username/kafka

2. Open a Terminal window and navigate to the Kafka directory using the cd command

3. Start the ZooKeeper server by running the following command:


bin/zookeeper-server-start.sh config/zookeeper.properties

4. Open another Terminal window and start the Kafka server by running the following command: 


bin/kafka-server-start.sh config/server.properties

Installing Kafka on Linux

1. Extract the downloaded file to a directory on your computer, such as /opt/kafka

2. Open a Terminal window and navigate to the Kafka directory using the cd command

3. Start the ZooKeeper server by running the following command:


bin/zookeeper-server-start.sh config/zookeeper.properties

4. Open another Terminal window and start the Kafka server by running the following command: 


bin/kafka-server-start.sh config/server.properties


Java Syntax and Steps to use kafka

Create a Topic

To create a topic in Kafka, you can use the following command:


bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic my_topic

This command creates a new topic called my_topic with one partition and a replication factor of 1.

Produce Messages

To produce messages in Kafka, you can use the following code:


Properties props = new Properties();

props.put("bootstrap.servers", "localhost:9092"); 
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer"); 

Producer<String, String> producer = new KafkaProducer<>(props); 

for (int i = 0; i < 10; i++) { 
	String key = "key-" + i; 
	String value = "value-" + i; 
	ProducerRecord<String, String> record = new ProducerRecord<>("my_topic", key, value); producer.send(record); 	
    } 
producer.close();

This code creates a Kafka producer and sends 10 messages to the my_topic topic.

Consume Messages

To consume messages in Kafka, you can use the following code:


Properties props = new Properties(); 

props.put("bootstrap.servers", "localhost:9092"); 
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer"); 
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer"); 
props.put("group.id", "my_group"); 

Consumer<String, String> consumer = new KafkaConsumer<>(props); consumer.subscribe(Collections.singletonList("my_topic")); 

while (true) { 

	ConsumerRecords<String, String> records = consumer.poll(Duration.ofSeconds(1)); 

	for (ConsumerRecord<String, String> record : records) {

 	System.out.println("offset = {}, key = {}, value = {}", record.offset(), record.key(), record.value()); 

	} 

}

This code creates a Kafka consumer that subscribes to the my_topic topic and prints out the messages it receives.

Close the Producer and Consumer

Finally, when you're finished with the producer and consumer, you should close them to free up resources: 


producer.close();

consumer.close();

This code closes the Kafka producer and consumer.


Spring boot with Kafka example

Spring Boot is a popular framework for building Java applications, and it has great support for working with Kafka. In this tutorial, we'll cover how to use Spring Boot with Kafka to create a simple example.


Step 1: Set up Kafka Environment

Before we start building our Spring Boot application, we need to set up a Kafka environment. You can follow the previous topic from this tutorial to install Kafka on your operating system.


Step 2: Adding Dependency in Spring Boot Project

To create a Spring Boot project, you can use Spring Initialiser or your preferred IDE. Here's an example of the dependencies you'll need to include in your pom.xml file: 


<dependency> 

    <groupId>org.springframework.boot</groupId>

     <artifactId>spring-boot-starter-web</artifactId>

</dependency>   

<dependency> 

    <groupId>org.springframework.kafka</groupId> 

    <artifactId>spring-kafka</artifactId> 

</dependency>

This configuration includes the spring-kafka dependency, which provides support for working with Kafka in Spring.

Step 3: Create Kafka Configuration

Next, we'll create a configuration file for Kafka. Here's an example:


@Configuration
@EnableKafka
public class KafkaConfig {
 
    @Value("${kafka.bootstrap-servers}")
    private String bootstrapServers;
 
    @Bean
    public Map<String, Object> producerConfigs() {
        Map<String, Object> props = new HashMap<>();
        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
        props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
        return props;
    }
 
    @Bean
    public ProducerFactory<String, String> producerFactory() {
        return new DefaultKafkaProducerFactory<>(producerConfigs());
    }
 
    @Bean
    public KafkaTemplate<String, String> kafkaTemplate() {
        return new KafkaTemplate<>(producerFactory());
    }
 
    @Bean
    public ConsumerFactory<String, String> consumerFactory() {
        Map<String, Object> props = new HashMap<>();
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
        props.put(ConsumerConfig.GROUP_ID_CONFIG, "group-id");
        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        return new DefaultKafkaConsumerFactory<>(props);
    }
 
    @Bean
    public ConcurrentKafkaListenerContainerFactory<String, String> kafkaListenerContainerFactory() {
        ConcurrentKafkaListenerContainerFactory<String, String> factory =
                new ConcurrentKafkaListenerContainerFactory<>();
        factory.setConsumerFactory(consumerFactory());
        return factory;
    }
}

This configuration creates the necessary beans to connect to Kafka and send and receive messages.


Step 4: Create a Producer

Now, we'll create a simple REST API that produces messages to Kafka. Here's an example: 


@RestController
public class KafkaController {
 
    @Autowired
    private KafkaTemplate<String, String> kafkaTemplate;
 
    @Value("${kafka.topic}")
    private String topic;
 
    @PostMapping("/message")
    public void sendMessage(@RequestBody String message) {
        kafkaTemplate.send(topic, message);
    }
}

Step 5: Create a Consumer

Finally, we'll create a simple consumer that listens for messages on the Kafka topic. Here's an example: 


@Service
public class KafkaConsumer {
 
    @KafkaListener(topics = "${kafka.topic}", groupId = "group-id")
    public void listen(String message) {
        System.out.println("Received Message in group - group-id: " + message);
    }
}

This code defines a Kafka listener that listens for messages on the Kafka topic specified in the application.properties file. When a message is received, it prints the message to the console.

Step 6: Configure Application Properties

Finally, we'll configure the application properties to specify the Kafka bootstrap servers and topic. Here's an example: 


    #KafkaConfiguration
  
    kafka.bootstrap-servers = localhost:9092
    
    kafka.topic = test-topic

Step 7: Run the Application

Now, we can run our Spring Boot application and test it out by sending a message to the /message endpoint using any REST client like Postman. You should see the message printed in the console.

This is a basic example of how to use Spring Boot with Kafka. You can extend this example to add more features like error handling, retry logic, and serialization/deserialization.

Hope this blog tutorial has been helpful to you. If you have any remaining doubts or suggestions, please feel free to share them in the comments below. We would be glad to receive your feedback.

💛 You can Click Here to connect with us. We will give our best for you.

Related categories:

Comments

Post a Comment

Popular Posts on Code Katha

Java Interview Questions for 10 Years Experience

Image
Welcome to the comprehensive guide on "Java Interview Questions for 10 Years Experience." Java has been one of the most popular programming languages for decades, and its widespread adoption means that the demand for skilled Java developers has never been higher. If you are someone who has been working with Java for the last decade, you have accumulated a wealth of experience and knowledge that you can showcase during interviews. Recommended  to read and brush up for your interview: Must know interview questions and concept for any Java Interview Spring boot interview Questions for 10 years experiance These simple and short articles help you to add AI to your resume and enhance the chance of selection: Java Spring Boot AI with Ollama Java Spring Boot AI with Open AI (Chat GPT) This blog post will provide you with a list of commonly asked interview questions for 10 years experienced Java developers. It will also include detailed answers to each question, with explanat...
Read more

Sql Interview Questions for 10 Years Experience

Image
As an experienced SQL professional, you might encounter challenging questions during a job interview. In this blog post, we will explore 10 common SQL interview questions and provide in-depth explanations along with relevant examples. Let's dive in! Contents [ hide ] What is SQL Injection, and how can you prevent it? SQL Injection is a malicious attack where an attacker manipulates input data to execute unintended SQL statements. It can lead to unauthorized access, data breaches, and data manipulation. To prevent SQL Injection, follow these best practices: 1.1. Use Prepared Statements: Prepared statements or parameterized queries separate SQL code from data input, preventing attackers from injecting malicious SQL code. Here's an example in PHP: $stmt = $pdo->prepare('SELECT * FROM users WHERE username = :username'); $stmt->bindParam(':username', $username); $stmt->execute(); 1.2. Sanitize User Input: Validate and sanitize user input ...
Read more

Spring Boot Interview Questions for 10 Years Experience

Image
Welcome to the comprehensive guide on "Spring Boot Interview Questions for 10 Years Experience". Spring Boot has become one of the most popular frameworks for developing enterprise-grade applications, and its widespread adoption means that the demand for skilled Spring Boot developers has never been higher. If you are someone who has been working with Spring Boot for the last decade, you have accumulated a wealth of experience and knowledge that you can showcase during interviews. Recommended  to read and brush up for your interview: Must know interview questions and concept for any Java Interview Java Interview Questions for 10 Years Experience These simple and short articles help you to add AI to your resume and enhance the chance of selection: Spring Boot AI with Ollama Spring Boot AI with Open AI (Chat GPT) This blog post will provide you with a list of commonly asked interview questions for 10 years expe...
Read more

Java interview questions - Must to know concepts

Contents [ hide ] How does Java achieve platform independence, and what is the underlying principle? Java programs can run on diverse operating systems and hardware setups without requiring modification. This feat is primarily achieved through the concept of the Java Virtual Machine (JVM). The Java Virtual Machine (JVM) The JVM acts as a bridge between the Java code you write and the underlying hardware and operating system . When you compile your Java code, it is translated into an intermediate form called bytecode, which is not tied to any specific platform. The JVM then comes into play. It takes this bytecode and dynamically translates it into machine-specific instructions that the host system can execute. This means that a Java program can be written once and executed on any system with the appropriate JVM for that system. Example: Running a Java Program on Different Platforms Consider a simple Java program that calculates the sum of two numbers: package codeKatha...
Read more

Visual Studio Code setup for Java and Spring with GitHub Copilot

Image
Introduction- In this tutorial blog, we will explore the process of setting up and utilising Visual Studio Code for Java programming, as well as incorporating GitHub Copilot as a valuable coding tool. Visual Studio Code is a powerful and versatile code editor that offers a wide range of features and extensions for Java developers. By using Visual Studio Code for Java programming, developers can benefit from features such as code completion, debugging, and live coding. Additionally, with the integration of GitHub Copilot, developers can streamline their coding process with AI-assisted suggestions for code snippets and functions. This blog will provide a comprehensive guide for setting up and using these tools for Java development. Contents [ hide ] Setup Visual Studio Code for Java and Spring Visual Studio Code (VS Code) is an open-source code editor that supports a wide range of programming languages, including Java. It is a powerful tool...
Read more

Spring Data JPA

Image
Introduction- We will learn here Spring Data JPA, perform CRUD operation, Clear concept of Relationship mapping like one-to-many, many-to-one with Example. Spring Data JPA is a powerful framework that simplifies the process of working with relational databases in Java applications. It provides an easy-to-use API for working with JPA, allowing developers to focus on their business logic instead of worrying about the details of how JPA works. With Spring Data JPA, you can easily define entities that map to database tables, and then use them to perform CRUD (create, read, update, delete) operations on those tables. In addition, Spring Data JPA provides support for more advanced features like querying and pagination, making it an excellent choice for both small and large projects. Contents [ hide ] In this blog, we will explore Spring Data JPA by building a sample application that uses it to manage users and packs. But, before let...
Read more

Spring AI with Ollama

Image
Introduction to Spring AI with Ollama Welcome to a comprehensive guide on leveraging Spring AI with Ollama to develop AI-driven applications using Java. This tutorial will cover everything from setting up Ollama locally, configuring your development environment, to creating an application that utilizes large language models for text generation. Contents [ hide ] Setting Up Ollama Installing Ollama To use Ollama locally, you must first install it on your machine. This involves downloading the Ollama software from the official repository and configuring it to run without Docker. Follow the installation instructions provided in the Ollama README file available on their official GitHub page . Ollama Download link Downloading the Mistral Model After setting up Ollama, you can download the Mistral model directly from the Ollama management interface. Mistral is designed for a broad range of applications, offering robust text generation capabilities. // Run this command in cl...
Read more

Data Structures & Algorithms Tutorial with Coding Interview Questions

Image
About the Course This course offers comprehensive coverage of coding questions that serve as a solid foundation for understanding DSA concepts. These questions are frequently asked in coding interviews, either directly or with some variations, by many companies, including Adobe, Facebook (Meta), Paytm, Amazon, Google, and others. Our primary objective is to enhance your problem-solving skills and enable you to devise optimal solutions. The course includes a wide range of topics, such as Arrays, Strings, Searching and Sorting, Greedy Algorithms, Trees, Graphs, Dynamic Programming, and more. Upon completion, you will possess the necessary skills to ace DSA interviews for typical companies or excel in the initial rounds of product-based companies such as Google, Microsoft and similar others. We are started with Graph data Structure and add others soon. Contents [ hide ] Approach Our course is  organised into chapters ...
Read more

Java interview questions for 5 years experience

Contents [ hide ] What is difference between Heap and Stack Memory in Java, and shed light on their utilization? In Java, memory management plays a crucial role in determining how objects are stored and accessed during program execution . Heap and Stack Memory are two distinct regions where different types of data are managed. Understanding their differences is essential for efficient memory utilisation and managing object lifecycles. Stack Memory Stack Memory is a region used for storing method calls, local variables, and references to objects . It operates in a Last-In-First-Out (LIFO) manner , resembling a stack of items. Each method call creates a new frame in the stack, containing variables specific to that method. Stack Memory is relatively fast for allocation and deallocation because it follows a strict order. However, it has limited space and is typically used for small, short-lived data. Primitive data types and references to objects are often stored here. Ex...
Read more

Spring AI with Open Ai

Image
Integrating OpenAI ChatGPT with Spring AI With the advent of AI-driven technologies, Spring AI now supports OpenAI's ChatGPT, a powerful language model that has significantly advanced the capabilities of text generation in software applications. In this blog, we will delve into the process of integrating ChatGPT with Spring AI, explaining key concepts and providing practical examples to help you get started. Contents [ hide ] Getting Started: Obtaining an OpenAI API Key The first step towards integrating ChatGPT into your Spring application is obtaining an API key from OpenAI, which serves as your secret key for accessing the API. Here’s how to do it: Visit the OpenAI website and sign up for an account. Once your account is set up, navigate to the API section and select the 'API Keys' page. Click on 'Create new key'. This will generate a new API key that you can use in your application. Copy this key and secure it. You will need to...
Read more