r/snowflake
My team and I built a EL tool especially for snowflake, not sure if we are reinventing the wheels or it's worth sharing with the community. let me know what you think.
- upvotes
- 8
- comments
- 11
Post
Hi Community, I'll try not to sound like a sales person here. My team and I built this tool for us to use to move data from major databases(mysql, postgres, sql server mainly) to snowflake(I know it's pretty limited, but that's enough for us for now). Just want to know if the community thinks it's a good idea to open it up to the community for free or it's just unique cases just for us. It's mainly to solve some of the pains you might have encountered as well when doing EL to snowflake as a data engineer or architect. Anyway, things that bothers me and my team a lot were as follows, you can comment on how you solve this instead of writing a tool to do the syncing(and make us look stupid of course) Big database table, 1+ billion records syncing - if you've done this scale before, you know what I'm talking about, it's super slow. Export existing views instead of tables!!! - I don't see why the existing tools would not do that. Schedule per table! - Why do I have to run the syncing to the WHOLE database at scheduled time.... it doesn't make sense. Delete detections - Seriously, why this is so hard to achieve... SOX Audit - it's a perfect use case for the EL tool IMO, why isn't there one already. etc... These are some main pain points we faced during our operations, wanna see your thoughts as to how you solve these problems, and if our in house tool that solved these...
Keep reading with a free account
The rest of this post, and every signal for Fivetran, is in your free account.
Extracted from these lines
[comment u/Muted_Jellyfish_6784] There is room even with Fivetran and Airbyte out there, the gap is usually schema drift handling and how cleanly the tool recovers from a source changing shape mid-sync without dropping columns.
[comment u/Muted_Jellyfish_6784] Stress test idempotency, what happens on a replay after a failed load, duplicate rows from a bad retry are the top complaint about EL tools in production.