Filtering SNS Messages Based on Time and Key Value in SQS

DevOps AWS
#1

Hi friends, im looking for someone to help me with SNS Message Body Filtering. I have a simple setup - S3 → SNS → SQS → Lambda. SNS is listening to new objects created, I then have SQS subscribing to the SNS topic and then a lambda function processing the message. Here is what the SNS message looks like

  "Type" : "Notification",
  "MessageId" : "ce0e42bc-b989-531e-8414-1ceb21785db5",
  "TopicArn" : "arn:aws:sns:us-east-2:123:game-fetcher-collector-sns",
  "Message" : "{\"version\":\"0\",\"id\":\"3747a21f-d642-5ee7-0b5a-5396dae172fa\",\"detail-type\":\"Object Created\",\"source\":\"aws.s3\",\"account\":\"123\",\"time\":\"2024-02-27T08:00:48Z\",\"region\":\"us-east-2\",\"resources\":[\"arn:aws:s3:::my-data-lake\"],\"detail\":{\"version\":\"0\",\"bucket\":{\"name\":\"my-data-lake\"},\"object\":{\"key\":\"bronze/v1/games/2024/02/27/08/3558957217.json\",\"size\":1651,\"etag\":\"3e1bab9d2ca4e2bd353b2352d0449172\",\"version-id\":\"2p00Xqw2UCjyMvRNlNiIIb7OJtjjCi6B\",\"sequencer\":\"0065DD96B096A408D8\"},\"request-id\":\"KGY6D8HZBSETMC7Y\",\"requester\":\"123\",\"source-ip-address\":\"3.140.166.200\",\"reason\":\"PutObject\"}}",
  "Timestamp" : "2024-02-27T08:00:49.532Z",
  "SignatureVersion" : "1",
  "Signature" : "Wl7O9qfOFr7BWsjh0cO0fiMYQlVp0FG0DoOLNG0M19dEw6F6iaBhTszC+bBM4DNm3+GylDw+zvDzs2cD4fDNCSm0O50arnOQdOZ/SDiTiER9atoDINEJ7pbZ2HlzwwbWLq3i1BiZPEkZmuhGs4fMnOxXTJzBcy7rP6pNZsdcjLpiH5gqtG2mK/sjFIdKFpJ7eNJ8jRWcwXJu0KCzrTpTgowsPKRF7l2ixTlYAbWtZbMJkmH2g3xILzxE7uve1BURkEGDhpjH+C/lsBzwh8i+DFDL8b3tDoP0ANMkVW8+vyNRXVt2sAyGCycT8tVQdLA5yTyhsg5nw1DJWWJCe0cvcg==",
  "SigningCertURL" : "<https://sns.us-east-2.amazonaws.com/SimpleNotificationService-60eadc530605d63b8e62a523676ef735.pem>",
  "UnsubscribeURL" : "<https://sns.us-east-2.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-east-2:700849607999:game-fetcher-collector-sns:3f7b3758-4312-42ae-b174-f34c77f3741b>"
}```


I want to filter out the messages that the SQS receives from the SNS to only messages where the time is at 1:00. Or if the value of the key contains 10. Is this possible?
#2

I’m not 100% sure about whether you can do it in SQS or SNS, but you could definitely do it in the Lambda?

#3

ye probably, was looking for a solution using the existing sns message filtering policy. The main issue is that you can do “prefix” and “suffix” but not “contains” or some other regex kind of pattern

#4

Ah, yeah, I see — especially because you can’t match the end of the time string because of the seconds!

Looks like from reading the documentation that there’s no way to do it in SNS.

Do you have a support plan with entitlement to use-case guidance? I’m happy to raise a case using our plan if not, as this feels like a shortcoming in the product!

#5

Ye you get it! We don’t have aws support at all as we are a startup.

I do believe a substr or regex pattern matching feature here would be super useful.

#6

I’ll DM you now and we can get a case opened. :slightly_smiling_face: