Hi, had some doubts on a google cloud dataflow pipeline that I want to run.
I want to take data from a topic and perform some minimal level of processing on it before storing it in firestore. I am using dataflow as it provides a streaming option and want to leverage the parallel processing used by google cloud platform. Is this the best way or can unbounded data be handled in the same way by another runner?
Also I have written a pipeline but when I run it, I get a pipeline where nothing happens. If I comment out the second windowing step and run the code again on dataflow, I get a runtime error with some grpc constructor that I cant understand. Please help me as I am a beginner
My question with code is here on stack-overflow: https://stackoverflow.com/questions/78834161/facing-runtime-error-in-gcp-dataflow-pipeline
please note that i am using a gcp trial subscription to experiment with gcp
I recommend a cloud function or cloud run application for such a simple task. dataflow feels like overkill for this
cloud run and functions with autoscale up to meet demand too
Hi Preston, thanks for replying! I thought of using cloud functions but want to practice dataflow for this use case in case I need to do more complex processing later
this is really just a proof of concept for later
i gotcha, if you are dead set on using dataflow streaming then it will work, and you can use autoscaling to have it automatically scale up
I still think building your own system with cloud fn and run could be a good option though even if it’s more complex. dataflow streaming is kinda niche but it’s definitely nice to have if you really need it
I really have no idea what the issue is with your code
i would suggest going through a dataflow tutorial and get the most basic streaming example working first. then customize to your needs once you’ve validated for sanity
Yes that seems like a good idea will start in the meantime. If you are able to solve the issue/make the pipeline code cleaner please let me know
