Does log.flush.interval.messages have an affect on ___consumer_offsets_ ?
It seems that with this set the consumer offset topic is being delayed flush to disk.
It does. But more importantly how are you measuring this perceived lag?
I noticed that the filesize on the consumer partition was only increasing in batches of 18MB
That’s the flush.size when it’s fsync()'d kafka only flushes to disk at a size or time interval. Doesn’t really mean the data is LOST if the machine crashes before that, since replication.factor=3
Toss in journalled file system(ext4) and you get weird things like files being written too, but only increase in batches
