CloudFormation race condition: ECS service attaching to LoadBalancer with TargetGroup creation issue

Anyone have had came across annoying race condition within CloudFormation when you are attaching ECS::Service to LoadBalancer, where TargetGroup is being created in the same Stack, and ListenerRule as well ?

Despite adding TargetGroup to DependsOn list to ECS::Service, it produces: The target group with targetGroupArn does not have an associated load balancer.

Any tricks on this or I really need yet another Lambda for this ? :smile:

Use ListenerRule in DependsOn list.

Well, that’s the problem, ListenerRule is present already on ECS::Service DependsOn list

which why this drives me crazy :smile:

from crhelper import CfnResource
import logging as log
import os
import time

LOG_LEVEL = os.getenv("LOG_LEVEL", "DEBUG")
client = boto3.client('elbv2')

helper = CfnResource(
    json_logging=False, log_level="DEBUG", boto_level="CRITICAL", sleep_on_delete=120

def wait_attachment(event, context):
    properties = event["ResourceProperties"]
    target_group_arn = properties["TargetGroupArn"]
    while True:
        response = client.describe_target_health(
        for group in response["TargetGroups"]:
            if group["TargetGroupArn"] == target_group_arn:
                if len(group["LoadBalancerArns"]) > 0:
def lambda_handler(event, context):
    log.debug(f"EVENT: {event}")
    helper(event, context)```

baked quick lambda, let’s see if it helps

Hmm, it should work. I used ListenerRule (only) with DependsOn for multiple services.

Well, further testing makes this even more bizzare - I have multiple ECS::Service in single Stack, each having its own ListenerRule, and only on of them is failing

Target group is already a dependency for ListenerRule to be created, so it doesn’t need to be listed for service.

That’s my expectation as well, and not the first time I am dealing with ECS::Service + ELB, but seems that there is room for race condition anyway

let’s see does Lambda help here

if it does, then it’s clearly to be reported as bug

        Type: AWS::ECS::Service
            -   PublicListener
            -   AttachRegionalCertificate
            -   AlertmanagerByHostname
            -   PublicTargetGroupAlertmanager
            Cluster: !Ref ServiceCluster
            DesiredCount: 1
            LaunchType: FARGATE
                    AssignPublicIp: DISABLED
                        - !Ref AlertmanagerSecurityGroup
                        - !Ref EFSSecurityGroup
                        - !ImportValue RegionalPrivateSubnet0
            TaskDefinition: !Ref AlertmanagerTaskDefinition
                MaximumPercent: 200
                MinimumHealthyPercent: 100
            SchedulingStrategy: REPLICA
            EnableExecuteCommand: true
                -   Key: Name
                    Value: observability-public-alertmanager
                -   ContainerName: alertmanager
                    ContainerPort: 9093
                    TargetGroupArn: !Ref PublicTargetGroupAlertmanager
                -   RegistryArn: !GetAtt CloudmapServicealertmanager.Arn
                    ContainerName: alertmanager
                    ContainerPort: 9093

        Type: AWS::ElasticLoadBalancingV2::TargetGroup
            Name: alertmanager-public
            Port: 9093
            Protocol: HTTP
            VpcId: !ImportValue RegionalVPC
            TargetType: ip
            HealthCheckIntervalSeconds: 10
            HealthCheckPath: /metrics
            HealthCheckProtocol: HTTP
            HealthCheckTimeoutSeconds: 5
            HealthyThresholdCount: 2
                -   Key: Name
                    Value: observability-public-alertmanager

        Type: AWS::ElasticLoadBalancingV2::ListenerRule
                -   Type: forward
                    TargetGroupArn: !Ref PublicTargetGroupVMAgent
                -   Field: host-header
                            - !Sub "alerts.${PublicZoneName}"
                            - !Sub "alerts.${AWS::Region}.${PublicZoneName}"
            ListenerArn: !Ref PublicListener
            Priority: 3```

Alternatively you can try to use ListenerRule as dependency for TaskDefinition, to give it a bit more time before it tries to create the service. :shrug:

We think very alike, as that I tried already :smile:

Oh, I know why I am getting this race condition and it’s stupid reason and clearly race condition :smile:

Here, I have ECS::Service already defined/deployed, and I am extending it via Update with LoadBalancer association, whilst rest of the services when I was deploying, I was deploying ECS::Service with Loadbalancer assoc. via Create, aka ECS::Service was to be created as well, here only Updated.

So, attaching loadbalancer to existing ECS::Service will throw you given alert

CF bug reported: