More of a general Kafka question, I know, but anyone knows of a strategic initiative or KIP to introduce reverse consuming topics?
Could you elaborate on what you mean by reverse? As in from tail to head?
Yup. The ability to start with the latest offset and go back. Ideally something you define when creating your consumer group.
I know you can do a seek, but it feels wrong imho.
There’s been talk of this but so far no one has really started a kip or any work that I know of.
The main benefit of this would be in an operational kafka use-case (high retention) where you want to replay data to perform analysis (trend, aggregating, …)
What I’ve seen Is people sorta hack it. And grab offsets 100-90, then 89-80 for example
Yeah, was thinking of that as well, some micro-batching kinda approach
Yes. It’s not fun. But sorta works. Really if you’re after analytics Kafka and the clients are a really expensive place to keep your data
Cheaper to offload to S3 (or object store) and partition by whatever date ranges you need
Well, we are aiming to keep 160Pb within kafka globally, explicitly not wanting to rely on S3 directly (allthough, we are relying on tiered storage a lot)
Yeah, I know, a lot of data, and most people freak out if I start talking about it. But there is more context to it why we are doing it this way.
That’s a lot of data to keep in Kafka and a seriously limited query pattern on that data
(I say this as a huge fan of Kafka)
I know, but it actually makes sense for us
Collecting market data ticks?
We use a tiered reporting/querying strategy using statestores and replaying strategies for long-term reports and analysis use-cases
Close 
. building a global Trade Repository
Nice. Fun use case, worked on a couple of those
Ah nice. Would be very interested in sharing war stories
