What are the ways to update Glue catalog table from Glue job? I don’t want to use Glue crawler due to some limitations. One of the way is to use configure this, https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-hive-metastore-glue.html but it’s not efficient.
Hi What issue you are having with crawlers?
Let’s say I have this folder structure
• root_path
◦ table1
◦ table2
◦ table3
I have set root_path in crawler. When I run it creates 3 tables : table1, table2, table3
But If I add new folder table4 then It won’t recognise it and table4 won’t be created on next run.
I have to re-create the glue crawler and crawl all the folders
Your new folder is inside root folder?
Would you be able to create separate schemas and target those schemas for auto table creation?
Sorry, What you’re asking?
What’s your Data source?
