Configuring Firehose to Assume Role for Dynamic S3 Prefix Generation

DevOps AWS
#1

Hi all. I have a Firehose DeliveryStream with an S3 destination that uses dynamic partitioning to generate the S3 prefix based on a customer ID that is included in the PutRecord body.

I would like Firehose to assume the role that is calling PutRecord, so that I can apply an S3 IAM condition using {$aws.PrincipalTag} that prevents customers from uploading records with an incorrect customer ID. Rather than just letting Firehose assume a role which can write to any prefix.

Is that possible? Is there a better way to do this? My customers are authenticated via a Cognito Identity pool, and my identity provider returns the customer ID, so it’s available as a session tag. I’m just not sure how to leverage that with Firehose in the middle.

Thanks!

#2

Hey. I don’t think there is such thing as assume role. First what comes to mind is to have this logic as part of Data Transformation by invoking Lambda, which will check ID before put to the S3. https://docs.aws.amazon.com/firehose/latest/dev/data-transformation.html

#3

Thanks . Looks like a lambda would definitely work so that is an option at least. Would be great if I could accomplish it with IAM only though. There are a lot of options there that seem like they could maybe work, like sts:SourceIdentity or sts:RoleSessionName. But I’m not sure which are actually compatible with web identity federation.