Deploying nginx proxy to connect to Kafka in private GKE cluster

Hi all, I got a private GKE cluster, from where I deployed a Kafka (port 9092), and enabled with SASL only, there is firewall rule that only 80 and 443 of my GKE cluster can be access by outside, but I want teams can access my Kafka from private GKE, so I deployed a nginx as proxy between users and my Kafka in private GKE, everything looks fine, but it looks like nginx is trying to establish connection with Kafka but failed, as nginx doesn’t have SASL info:

/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2024/06/13 14:45:03 [error] 29#29: *31 upstream timed out (110: Connection timed out) while connecting to upstream, client: 240.224.129.1, server: 0.0.0.0:80, upstream: "240.224.129.28:9092", bytes from/to client:0/0, bytes from/to upstream:0/0
2024/06/13 14:45:13 [error] 29#29: *31 upstream timed out (110: Connection timed out) while connecting to upstream, client: 240.224.129.1, server: 0.0.0.0:80, upstream: "240.224.131.21:9092", bytes from/to client:0/0, bytes from/to upstream:0/0
2024/06/13 14:45:13 [error] 29#29: *34 upstream timed out (110: Connection timed out) while connecting to upstream, client: 10.31.130.233, server: 0.0.0.0:80, upstream: "240.224.128.16:9092", bytes from/to client:0/0, bytes from/to upstream:0/0
2024/06/13 14:45:13 [error] 29#29: *34 no live upstreams while connecting to upstream, client: 10.31.130.233, server: 0.0.0.0:80, upstream: "kafka", bytes from/to client:0/0, bytes from/to upstream:0/0```


on the other side, I try to list topic from local, I’m not sure if below command successfully go through nginx proxy and reach to Kafka in my private GKE cluster? I double check I’m sure I have setup client.properties correctly….


```pingc bin> ./kafka-topics.sh --bootstrap-server 10.31.130.226:80 --command-config /Users/pingc/Documents/VmShared/temp/client.properties --list
[2024-06-13 22:45:10,036] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (10.31.130.226/10.31.130.226:80) terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue. (org.apache.kafka.clients.NetworkClient)
Error while executing topic command : Timed out waiting for a node assignment. Call: listTopics
[2024-06-13 22:45:59,286] ERROR org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: listTopics
 (kafka.admin.TopicCommand$)
pingc bin>```

anyone has experience of setup a proxy for Kafka in a private K8S cluster? is nginx a good one? normally how do we setup the proxy, thanks!

I have not done it in k8s, but here is one I have done in docker compose

https://github.com/kineticedge/dev-local/tree/main/kafka-lb

  1. need to do it for each broker as clients need to talk to each broker
  2. the advertised listeners of kafka must match what the IP/DNS are for NGINX proxy
  3. Config must be for tcp (stream) not for http.

I would recommend reading through this blog series (at least the “Introduction”, “Load Balancers”, and “Ingress” entries).

I know you’re not using Strimzi but the blogs are fantastically written and explain a lot of the concepts that you need to understand in order to make it work

https://strimzi.io/blog/2019/04/17/accessing-kafka-part-1/

Thanks and , sorry to get back later, those info are really helpful, thanks!
Currently networking team is discussing to open port for my private GKE, in the meantime, I will see if I can setup needed config for Kafka proxy, as I don’t have full permission in my private GKE cluster……