Redis cluster design

DevOps
#1

Hey everyone!

I have set up 3 Sentinel Redis servers where I have one main Redis instance and 2 secondary Redis instances. Sentinel enables Redis secondary instance to get promoted to main on when the main instance goes down. I have two questions around this setup:

  1. My application should always be communicating with the main instance right?
  2. All three Redis servers have different IP addresses. If my main redis server goes down, I will need to do several checks:
    a. The first one will acknowledge that the redis server is down and can’t process requests (i.e my request will fail)
    b. I will need to hit one of the other two servers to check which one is the main one.
    c. Send the request to the main redis instance
    Is there a simpler way around this?
#2

According to the documentation you need to get the new leader address(https://redis.io/docs/manual/sentinel/#obtaining-the-address-of-the-current-master), this doesn’t add much complexity since they expose an API to do so. If a write fails then you just call the “Get Master Addr By Name” API. Most clients available have sentinel support. For example: https://github.com/go-redis/redis/blob/master/sentinel.go#L315

Hope this helps

#3

Hey !

Thanks for the reply.

Yeah, if a write fails, I need to call this API “Get Master Addr By Name” and for this I would need to ping one of the other servers. Once I get the the Address, I would then send the request to the Address. These are three network calls to the Redis cluster. Is there a simpler way?

#4

You could load balance your reads across replicas with something like ‘read.my-redis-cluster01’ and your writes with ‘write.my-redis-cluster01’ where the load balancer is responsible for routing to the active leader. Sentinel can notify about problems with redis instances.
But if you want to talk directly with the cluster I think your only option is to query for the active leader either before you make a write request or after write failure and retry.

#5

It seems like I would need to setup HAProxy

#6

It depends on your environment, setting up proxy/LBs might be overkill, if you are running containers then an ambassador might be nice but could also be overkill as well, the simplest lowest effort would probably be a decorator for your write function if your Redis client doesn’t provide any leader discovery features.

#7

I found something: https://redis.readthedocs.io/en/stable/connections.html#id1

Maybe this could be of help. Let’s see.

#8

Thanks for all your help!