Frameworks for kafka stream in nodejs?

DevOps Kafka
#1

Hi everyone,
May I ask what’s the best module perhaps framework for kafka stream in nodejs?
I’m having performance issue with the current config

#2

There is no Kafka Streams for Node. What are you currently using? From my experience for Kafka clients in Node, kafkajs is best.

#3

Hi, thanks for answering,
I’m having an issue with my consumer, at start it consume quickly but slowing down as I run a little stress test. Can you give me a suggestion how to config my consumer please?

#4

Not sure I can help, never run stress tests with Node, in our case the load was quit low.

#5

Can you suggest a consumer config that keep consumer stable? because the speed is different from 5 consecutive call, it’s getting bigger, doesn’t even qualify for a stress test

#6

Not sure, maybe you have a memory leak or something in what you do with the messages? I don’t think the problem is the consumer, but it could be.

#7 
        options: {
          client: {
            clientId: 'pragmatic-service',
            brokers: [process.env.KAFKA_HOST],
            ssl: {
                rejectUnauthorized: true,
                ca: [fs.readFileSync('ca.pem', 'utf-8')],
            },
            sasl: {
                mechanism: 'plain',
                username: process.env.KAFKA_USERNAME,
                password: process.env.KAFKA_PASSWORD,
            },
            connectionTimeout: 3000,
            authenticationTimeout: 1000,
            reauthenticationThreshold: 10000
          },
          run: {
            autoCommit: true
          },
          producer: {
            idempotent: false
          },
          consumer: {
            groupId: process.env.CONSUMER_GROUP,
            heartbeatInterval: 2000,
            sessionTimeout: 6000
          }
        }```
#8

The consumer just log the message, nothin else

#9

And if I give a little delay like 1s between request, it still fast