Introduction to kafka-node

Kafka-node is a popular Node.js library that provides efficient and reliable communication with Apache Kafka clusters. This library allows you to produce and consume messages with ease and offers dozens of useful APIs. In this comprehensive guide, we will explore the various APIs provided by kafka-node, complete with plenty of code snippets and an application example to help you get started.

Getting Started

Before diving into the APIs, you need to install kafka-node using npm:

  npm install kafka-node

Create Kafka Client

The first step is to create a Kafka client. The KafkaClient class allows you to connect to a Kafka broker.

  
    const kafka = require('kafka-node');
    const client = new kafka.KafkaClient({kafkaHost: 'localhost:9092'});
  

Producer API

The Producer class allows you to produce messages to a Kafka topic.

  
    const producer = new kafka.Producer(client);

    producer.on('ready', () => {
      const payloads = [
        { topic: 'test', messages: 'Hello Kafka' }
      ];

      producer.send(payloads, (err, data) => {
        console.log(data);
      });
    });

    producer.on('error', (err) => {
      console.error(err);
    });
  

Consumer API

The Consumer class allows you to consume messages from a Kafka topic.

  
    const consumer = new kafka.Consumer(
      client,
      [{ topic: 'test', partition: 0 }],
      { autoCommit: true }
    );

    consumer.on('message', (message) => {
      console.log(message);
    });

    consumer.on('error', (err) => {
      console.error(err);
    });
  

Admin API

The Admin class provides methods for managing topics and configurations.

  
    const admin = new kafka.Admin(client);

    admin.listGroups((err, res) => {
      console.log(res);
    });

    admin.describeConfigs({ resources: [{ type: 'topic', name: 'test' }] }, (err, res) => {
      console.log(res);
    });
  

HighLevelProducer API

The HighLevelProducer class provides additional features not available in the basic Producer class.

  
    const highLevelProducer = new kafka.HighLevelProducer(client);

    highLevelProducer.on('ready', () => {
      const payloads = [
        { topic: 'test', messages: 'Hello Kafka High Level' }
      ];

      highLevelProducer.send(payloads, (err, data) => {
        console.log(data);
      });
    });

    highLevelProducer.on('error', (err) => {
      console.error(err);
    });
  

Application Example

Here’s a complete example of a Kafka-node application that utilizes a producer and a consumer:

  
    const kafka = require('kafka-node');
    const client = new kafka.KafkaClient({ kafkaHost: 'localhost:9092' });
    const producer = new kafka.Producer(client);
    const consumer = new kafka.Consumer(
      client,
      [{ topic: 'test', partition: 0 }],
      { autoCommit: true }
    );

    producer.on('ready', () => {
      const payloads = [{ topic: 'test', messages: 'Hello Kafka World' }];
      producer.send(payloads, (err, data) => {
        console.log('Produced:', data);
      });
    });

    consumer.on('message', (message) => {
      console.log('Consumed:', message);
    });

    producer.on('error', console.error);
    consumer.on('error', console.error);
  

This example demonstrates a basic Kafka-node application where messages are produced to the ‘test’ topic and consumed from the same topic, showing the complete message lifecycle.

Hash: 8df1233f3378f749cfabafbd536d238a8ac6e4203a5339d99b8cdca56bc64844