Kafka - Migrating from Zookeeper to KRaft - Still waiting for ZK brokers

DevOps Kafka
#1

I’m having a play with trying to migrate a zookeeper backed 3.4 cluster to KRaft, using confluent platform 7.4. This is just using a local docker cluster. I’ve had the cluster running in zookeeper mode, and also in KRaft (no migration) mode, all seems fine. When trying to migrate the cluster from zookeeper to KRaft though it seems to be waiting for the brokers to join the cluster. The setup is 2 brokers, one controller and one zookeeper node. After everything comes up I see this constantly every second on the controller:

docker-kafka-app_3-1      | [2023-06-29 07:47:15,397] INFO Still waiting for ZK brokers [1, 2] to register with KRaft. (org.apache.kafka.metadata.migration.KRaftMigrationDriver)
docker-kafka-app_3-1      | [2023-06-29 07:47:15,610] INFO [MetadataLoader 3] handleSnapshot: generated a metadata delta from a snapshot at offset 1109 in 6 us. (org.apache.kafka.image.loader.MetadataLoader)```
The output of the following commands seems to indicate all is ok though?
```/usr/bin/kafka-metadata-quorum --bootstrap-server 127.0.0.1:9092 describe --status
ClusterId:              tS8_4J0QRf6GTTI0BMxABw
LeaderId:               3
LeaderEpoch:            3
HighWatermark:          509
MaxFollowerLag:         0
MaxFollowerLagTimeMs:   0
CurrentVoters:          [3]
CurrentObservers:       [1,2]```
```/usr/bin/kafka-metadata-quorum --bootstrap-server 127.0.0.1:9092 describe --replication
NodeId	LogEndOffset	Lag	LastFetchTimestamp	LastCaughtUpTimestamp	Status  	
3     	601         	0  	1688024580554     	1688024580554        	Leader  	
1     	601         	0  	1688024580142     	1688024580142        	Observer	
2     	601         	0  	1688024580140     	1688024580140        	Observer	```
#2

Are you following the migration process as shown here? The “waiting for brokers to register with KRaft” is an expected output when first starting the KRaft controller in migration mode.
https://kafka.apache.org/documentation/#kraft_zk_migration

In general the migration process is as follows:
• Start a KRaft controller in migration mode
• Update broker configs to be in migration mode with the correct KRaft controller quorum configuration
• Restart the broker (at this point your KRaft controller output should show migration success)
• Update the broker config to be taken out of migration mode
• Restart broker
• Update KRaft controller config to be taken out of migration mode
• Restart KRaft controller