AWS RDS Proxy for Aurora MySQL cluster connection issue

DevOps AWS
#1

Hi all, I’m having a weird issue using AWS RDS proxy with an Aurora MySQL cluster, where, if I check the status of the targets, I’m getting an AVAILABLE state in of of the instances but the other is unhealthy, this is the output of aws rds describe-db-proxy-targets

    "Targets": [
        {
            "Endpoint": "endpoint-instance-0.region.rds.amazonaws.com",
            "TrackedClusterId": "db-cluster",
            "RdsResourceId": "db-instance-0",
            "Port": 3306,
            "Type": "RDS_INSTANCE",
            "Role": "READ_WRITE",
            "TargetHealth": {
                "State": "AVAILABLE"
            }
        },
        {
            "Endpoint": "endpoint-instance-1.region.rds.amazonaws.com",
            "TrackedClusterId": "db-cluster",
            "RdsResourceId": "db-instance-1",
            "Port": 3306,
            "Type": "RDS_INSTANCE",
            "Role": "UNKNOWN",
            "TargetHealth": {
                "State": "endpoint-instance-0.region.rds.amazonaws.com",
                "Reason": "UNREACHABLE",
                "Description": "Timeout connecting to the database"
            }
        },
        {
            "RdsResourceId": "db-cluster",
            "Port": 3306,
            "Type": "TRACKED_CLUSTER"
        }
    ]
}```


And this is what I see of the instance logs:



```[Server] Access denied for user 'rdsadmin'@'localhost' (using password: YES) (sql_authentication.cc:1412)```
#2

I mean, trust the error message, bad password - do you have automated password rotation via secrets manager?

#3

Tho the one error says timeout

#4

I don’t use secrets manager, the password is self managed and I’ve created the cluster and instances via terraform. I can also connect to the instances and the cluster endpoints with dbeaver

#5

So you can connect to all nodes in the cluster?

#6

No, individually with the master password. Connecting to the proxy also works but it is probably using the available RW endpoint

#7

Gotcha, you can’t go through to some specific endpoint?

#8

I assume if all the instances are good it’s a proxy misconfiguration somehow, like it’s trying to route to an old instance that was rebuilt or something

#9

Or at least the health check is doing something different than you are when you can connect to the instance successfully

#10

That’s the thread I would pull on but unfortunately I don’t have like an aha here’s the problem

#11

Are you logging in with the same account as the proxy health check?

#12

The fact that you got a bad password error makes me think that something changed

#13

Yes, same account, the health check gets the credentials from a secret but that secret is replicating the master pwd

#14

Can you just look at the values and make sure that they’re right

#15

Or make sure that they match I guess

#16

I believe something is wrong with the instance but can’t pinpoint it… the bad instance is on us-east-1c and the good one on us-east-1a

#17

You can take drastic measures but I would do all of the reasonable sanity checks first

#18

thinking my next step should be having them both on the same AZ, as I’m not going to need multi AZ at this point, but I wonder why because multi AZ is supported

#19

That seems even more drastic than I was thinking lol

#20

You have some threads to pull on, you’re not in outage