[Kafka cluster] Disk usage across servers is not equal

Hi all, I have three cluster kafka on msk (amazon). something weird I found, I see that disk usage of those three server is not equal.
the first server 95% and two others only 5% seems that kafka topic only exist on server. I already check that all of the topic already set to 3 replication. any body has an experience on this problem ?

You need to check which topics are taking all this space - either explore the log size metric (kafka.log<type=Log, name=Size …>), or explore the data directory directly on disk.

MSK does not implement self-balancing Kafka like Confluent Cloud does. It’s just not a feature in the open source version of Kafka. You have to run your own batch balancing tools like Cruise Control if you want to get the cluster back into balance

Also if the RF=3 on all topics (including the sometimes hidden system topics) then a three node cluster should not get that out of since unless you are storing more than 1 replica on the same broker. That’s clearly a big problem for reliability, Rebalancing should be AZ aware an place one replica is each of the three brokers if they are evenly spread across all three AZs