Hi all, I got a private GKE cluster, from where I deployed a Kafka (port 9092), and enabled with SASL only, there is firewall rule that only 80 and 443 of my GKE cluster can be access by outside, but I want teams can access my Kafka from private GKE, so I deployed a nginx as proxy between users and my Kafka in private GKE, everything looks fine, but it looks like nginx is trying to establish connection with Kafka but failed, as nginx doesn’t have SASL info:
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2024/06/13 14:45:03 [error] 29#29: *31 upstream timed out (110: Connection timed out) while connecting to upstream, client: 240.224.129.1, server: 0.0.0.0:80, upstream: "240.224.129.28:9092", bytes from/to client:0/0, bytes from/to upstream:0/0
2024/06/13 14:45:13 [error] 29#29: *31 upstream timed out (110: Connection timed out) while connecting to upstream, client: 240.224.129.1, server: 0.0.0.0:80, upstream: "240.224.131.21:9092", bytes from/to client:0/0, bytes from/to upstream:0/0
2024/06/13 14:45:13 [error] 29#29: *34 upstream timed out (110: Connection timed out) while connecting to upstream, client: 10.31.130.233, server: 0.0.0.0:80, upstream: "240.224.128.16:9092", bytes from/to client:0/0, bytes from/to upstream:0/0
2024/06/13 14:45:13 [error] 29#29: *34 no live upstreams while connecting to upstream, client: 10.31.130.233, server: 0.0.0.0:80, upstream: "kafka", bytes from/to client:0/0, bytes from/to upstream:0/0```
on the other side, I try to list topic from local, I’m not sure if below command successfully go through nginx proxy and reach to Kafka in my private GKE cluster? I double check I’m sure I have setup client.properties correctly….
```pingc bin> ./kafka-topics.sh --bootstrap-server 10.31.130.226:80 --command-config /Users/pingc/Documents/VmShared/temp/client.properties --list
[2024-06-13 22:45:10,036] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (10.31.130.226/10.31.130.226:80) terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue. (org.apache.kafka.clients.NetworkClient)
Error while executing topic command : Timed out waiting for a node assignment. Call: listTopics
[2024-06-13 22:45:59,286] ERROR org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: listTopics
(kafka.admin.TopicCommand$)
pingc bin>```