>Once you need something complicated, you're on your own, while being stuck in a proprietary environment that you cannot extend.
Snowflake has spark connector too. So I don't know what the difference would be writing a spark job against deltalake vs snowflake.
> 80% of the things you will need - which is great for some newbie stuff or for sales presentations.
This is obviously wrong.
> You will have to develop some kind of data lake to store unstructured data anyway. You will end up with a Snowflake data warehouse and a data lake. Why not just go with data lake first then.
We store unstructured data in snowflake. I don't understand why you need a datalake on top of it.
EXACTLY. You absolutely can store unstructured and semi structured data in Snowflake. I find it baffling and at this point a bit irritating that there is this community of people insisting that is not allowed for...some unspecified reason.
Why would I store eg. bunch of html files as string columns in Snowflake, only to download them down, process them in python and load back into some other string table.
Because it actually costs the same, and if you process them in Snowflake using SQL or UDFs, you will get your results in seconds and you won't have to manage any of the underlying infrastructure.
Snowflake has spark connector too. So I don't know what the difference would be writing a spark job against deltalake vs snowflake.
> 80% of the things you will need - which is great for some newbie stuff or for sales presentations.
This is obviously wrong.
> You will have to develop some kind of data lake to store unstructured data anyway. You will end up with a Snowflake data warehouse and a data lake. Why not just go with data lake first then.
We store unstructured data in snowflake. I don't understand why you need a datalake on top of it.