Aggregating each node's data on the primary node

DevOps Kafka
#1

Hello guys
I’m new to kafka, sorry for possible misunderstanding
My usecase: 9 server cluster (3 nodes quorum-replicated, 6 others just parallel nodes); all the nodes write some data to postgres locally (separate postgres clusters); I’d like to aggregate each node’s data on the primary node, taking the summary from each node’s data

Is this an adequate usecase from kafka/CDC/debezium/whatever or are they an overkill for my usecase? If they are, why? And what better tools to use then?

Thanks in advance

#2

What do you mean by “3 nodes quorum-replicated, 6 others just parallel nodes” ?

#3

1 primary node, 2 replicas of it
other 6 nodes are just basically in the same network and doing the same work as the ‘main’ 3 ones, but are not technically part of a quorum cluster

#4

You might want to read up a bit on how Kafka works.

#5

You setup Kafka clusters. It is not a master replica kinda thing like databases.

#6

Can’t I just use a single-node kafka cluster on a primary node?

#7

Like I said, there is no primary node. You talk to the cluster.

#8

You can have a single broker cluster but I would defn not recommend that for production.

#9

Well, I’m bounded by a single primary node…
But anyway, thanks for suggestions :slightly_smiling_face:

#10

Once you create topics, their replicas are created on other brokers in that cluster, assuming you are using replication-factor 3 and have 3 or more brokers in that cluster.

#11

I see that for it to work it’s best to have several nodes for kafka and several for the target store (data wharehouse etc.) and maybe several for the source

#12

Apparently I have no free room for a separate kafka cluster :smiley:

#13

What is your objective? is this a PoC?

#14

If its a PoC you can do everything on one machine. just dont expect data resiliency in case something goes wrong.

#15

I have a storage system that is pretty much a rack of 9 nodes, 3 management nodes, 1 of them primary at a time
It has usage statistics on each node
I’d like to aggregate those stats in real-time to the primary (and then from there send them somewhere else or show in UI or …)

#16

Why not use something like prometheus? dont need kafka in that case

#17

I should correct myself. I have no clue what storage system you are using.

#18

If it’s application logging your after, I once used Logstash to get them to Kafka. Not sure that’s still the best. There might be a Kafka Connect Connector for it these days. Something like https://docs.confluent.io/kafka-connect-spooldir/current/connectors/elf_source_connector.html