Troubleshooting dataflow batch pipelines not starting as scheduled daily

DevOps Google Cloud
1

Hey, I have several dataflow batch pipelines running once per day to read data from MongoDB and ingest it into BigQuery. But for some reason sometimes some pipelines do not start at all. I just don’t see any information about running jobs on that day (it’s neither Succeeded, not Failed, they just didn’t start). Please advise on how to troubleshoot that

Upd: for scheduling I am using daily dataflow type. Maybe it’s buggy and makes sense to try custom one

2

Check the Cloud Monitoring logs. There should be some scheduler jobs in there that you can see if there’s an error or info message that was logged that wasn’t returned in the API for some reason.

3

Hm, I don’t see anything there either. Have no idea what happens