Prepare to overwhelm partitions on your state stores.
Yup, already anticipating that to some extend
Even writing custom statestores to be able to deal with it to some extend
Offloading some of the work to kdb/druid/* worked out better for the ones I’ve worked on. Someone always comes along and wants a count()
Well, we are not a high throughput, low latency usecase, but high-retention (regulatory reporting usecases)
But in the end, offloading it towards KDB or something else would just be another materialized state view, right?
Yes. But far cheaper and more flexible
Unless you need to be able to host different data models to the same data, like graph models, search indexes etc.
But I get your point. Flexibility for us is the ability to choose the data models that make sense to offload data to depending on domain or enduser usecase. Therefor we handle state as a consequence and use our events within kafka as truth
