Hello! I was trying to use Pub/Sub with go and discovered, much to my surprise, that pubsub go sdk does not support batch pulling from a subscription. Am I missing something, or is this intended design?
interesting, I think it supports the stream pull where many other language sdks don’t? surprising that it doesn’t support pulling multiple at a time at all. you sure?
I couldn’t find a reference to a function like that at all. It only has a Receive method, but no Pull
yeah just looking at the docs they have examples of how to use streamingpull for go and docs for other langs to use the fixed msg pull but no Go example. weird!
just curious because I’ve always wanted to use streaming pull method but my lang didn’t support it easily, why do you want the fixed msg route?
because i want to batch sql queries and external api calls. So that instead of 100 calls i can make one. Would reduce the load a lot and optimize costs
i think you have two options, you could either make the request yourself, or accumulate the messages using the stream locally, then batch them with a timer or count
but how do i ack them?
just add a waitgroup until the corresponding batch processes?
well i’m not a go user but have used the library on some other langs, usually you have a reference to a message and there’s an api to acknowledge them, but maybe not with the stream api? it might do it internally somehow
basically, they have stream processing through callbacks. So i would need to hang the callback until i’ve processed the message, which seems like a major pain…
i guess raw rpc is the way, but it’s so very annoying
well just looking at the docs it seems it wants you to call m.Ack(), it’s not clear to me if it blocks receiving on that stream until you ack it, but it also says it uses 10 streams at a time
i wonder if you could send them over a channel to some other code running logic to batch them, then ack them once you’ve built your batch insert or whatever
not sure how that would work with go concurrency model
but i agree pretty obtuse that they dont even offer the ability
i did notice that they have deprecated the ability to immediately return whatever messages are in the sub on the fixed pull route, so maybe some sort of push to make it more efficient
like “handle it yourself” for us
yes, the receives are concurrent and the goroutine is blocked until you exit the function, but other messages can be processed at the same time. The way you would do it, is by calling some Process() chan error function, and then wait on the channel. In the function, you would add the messages with corresponding channels to a queue, and then once processed write to the channel for all
but it’s like using a controller, to steer a wheel, which moves a mouse, which clicks buttons on a virtual controller
i threw this together with AI-assistance and rudimentary knowledge of Go. seems to work but obviously there’s plenty of uncovered corner cases. one I noticed already was that sometimes receive may get a previously ack’ed message again
