Updating messages in Kafka(Java)

DevOps Kafka
#1

Hi,
How can I update the message in kafka via java? I ve producer and consumer where POST working fine but how should I use PATCH with kafka

#2

Clarify? Kafka isn’t an HTTP protocol. Also messages cannot be modified once sent to the broker

#3

So let me ask in another way… How do I consume and overwrite only changes for related object

#4

What I am trying to ask is, let suppose I sent this object to Kafka and consume it… Then I would like to update the price… So how consumer can identify this

    "title": "Thsirt",
    "img": "img1",
    "categories": [
        "Fitness",
        "Men"
    ],
    "size": "40",
    "price": 16,
    "countInStock": 20
}```
#5

One option is to send a whole new event

    "title": "Thsirt",
    "img": "img1",
    "categories": [
        "Fitness",
        "Men"
    ],
    "size": "40",
    "price": 42,
    "countInStock": 20
}```
#6

Other option would be to build a KTable that holds a “itemUUID” as a key, and then you have a stream of fields that change {"UUID": "xyz", "fields": {"price": 42}}. Then you have a stream processor that parses these events, and updates the record according to the ordered events, by each key.

#7

The first option is more common if you use Kafka Connect from databases like Mongo or an RDBMS, particularly with Debezium

#8

This pages covers some more details https://stackoverflow.blog/2022/07/21/event-driven-topic-design-using-kafka/

#9

Might be noob question but how consumer can understand its the same product

#10

That’s exactly the issue. Your producer needs to generate some identifier. Then you need to maintain state somewhere

#11

Thus why any shopping store has product SKUs

#12

What kind of identifier, something like key ? What is SKUs

#13

Yes, a key. A SKU is a “Stock keeping unit”. Your example shows a TShirt. What sets that shirt apart from any other shirt?

#14

Or in a physical store, a barcode is associated with a SKU and barcodes map to some numeric value.

#15

Sorry for bothering but can you please share one example about this key

#16

Can we say its not a good practice to use kafka for PATCH

#17

Most programming languages have UUID generator functions. Start with that when you have any “new” shirt event. Then you need to “lookup” the same value when you perform any “update”.

Like I said, Kafka isn’t an HTTP protocol. Updates/Upserts work fine if you design your event payloads properly.

#18

Thanks a lot