> A data warehouse on the other hand is an OLAP database and is optimized to work on columns
A bit of a pedantic nit here: a data warehouse is a usage pattern. It’s not necessarily tied to any specific technology, however it is commonly implemented with OLAP systems like Snowflake, BigQuery, etc. But there’s nothing stopping you from building out your data warehouse in Postgres or MySQL. If you’re stitching together disparate datasets to build a unified model for analytics, you’ve got yourself a data warehouse no matter what system it lives on.
MNeverOff 10 minutes ago [-]
It's a good all-round primer, well written.
Would love to hear more about larger-than-memory tasks and running local Dask clusters. I processed many-a-dataset that way that would normally make pandas choke.
michaepf 4 minutes ago [-]
This was great, thanks for writing it up. Even as someone in the data space for a long time, I learned quite a bit.
jpitz 16 minutes ago [-]
Apache Avro has 2 encodings: binary AND json.
macintux 10 minutes ago [-]
Are you sure? I can't find any reference indicating that.
Now I'll be thinking of "L" in ETL as "Land" and not "Load".
Although the article doesn't propose that but uses a lot of "Land" terminology.
"Load" => "load where? or FROM where?" - ambiguous
"Land" => "land where?" - clear
madsaylor 1 hours ago [-]
Data is the new oil
Rendered at 16:56:41 GMT+0000 (Coordinated Universal Time) with Vercel.
A bit of a pedantic nit here: a data warehouse is a usage pattern. It’s not necessarily tied to any specific technology, however it is commonly implemented with OLAP systems like Snowflake, BigQuery, etc. But there’s nothing stopping you from building out your data warehouse in Postgres or MySQL. If you’re stitching together disparate datasets to build a unified model for analytics, you’ve got yourself a data warehouse no matter what system it lives on.
Update: Huh, TIL https://avro.apache.org/docs/%2B%2Bversion%2B%2B/specificati...