How common is it to just write data to a durable log like Kafka if you know it doesn’t need to be strongly consistent, even if the end destination is an RDBMS. I.e. some request comes in, you write it to a durable log before returning a success
Most of the DBMS you can imagine do that at least for durability. NoSQL sometimes omit that a bit in the face of presented replication and tunable “consistency” level. not sure if they flush memtable (LSM) every X seconds… others NewSQL based on Consensus protocols have not much choice, they have to keep RSM log consistent and durable (they fsync)
However, even Scylla/Cassandra has WAL
Having a database write to WAL/journal/whatever as it acks a write is a bit different than having a service shove into kafka and respond, then worry about inserting into a database later.
both are super common, though.
using kafka as a buffer to smooth out spikey api patterns so you can effectively rate limit / throttle underlying databases is exceptionally common, as is using kafka in front of fan out consumers (write into one kafka topic, but consume it multiple times, each consumer writing into different storage, oltp/olap/cold, etc)
Ty folks that’s very helpful
I was definitely thinking about it with respect to rate limiting
