Converting Huge CSV Files to Parquet with Dask, DuckDB, Polars, Pandas. | by Mariusz Kujawski | Medium
Parquet reading/writing is non-deterministic for multi-index frames · Issue #6484 · dask/dask · GitHub
![Python and Parquet performance optimization using Pandas, PySpark, PyArrow, Dask, fastparquet and AWS S3 | Data Syndrome Blog Python and Parquet performance optimization using Pandas, PySpark, PyArrow, Dask, fastparquet and AWS S3 | Data Syndrome Blog](https://miro.medium.com/v2/resize:fit:1400/1*9xqMqJyfaJ2NP64xbuRWGw.png)
Python and Parquet performance optimization using Pandas, PySpark, PyArrow, Dask, fastparquet and AWS S3 | Data Syndrome Blog
![python - Using set_index() on a Dask Dataframe and writing to parquet causes memory explosion - Stack Overflow python - Using set_index() on a Dask Dataframe and writing to parquet causes memory explosion - Stack Overflow](https://i.stack.imgur.com/tNStl.png)
python - Using set_index() on a Dask Dataframe and writing to parquet causes memory explosion - Stack Overflow
Writing to parquet with `.set_index("col", drop=False)` yields: `ValueError(f"cannot insert {column}, already exists")` · Issue #9328 · dask /dask · GitHub
Converting Huge CSV Files to Parquet with Dask, DuckDB, Polars, Pandas. | by Mariusz Kujawski | Medium
![Converting Huge CSV Files to Parquet with Dask, DuckDB, Polars, Pandas. | by Mariusz Kujawski | Medium Converting Huge CSV Files to Parquet with Dask, DuckDB, Polars, Pandas. | by Mariusz Kujawski | Medium](https://miro.medium.com/v2/resize:fit:806/1*BFeoK5UaWHB4cfE-YhL0PQ.png)
Converting Huge CSV Files to Parquet with Dask, DuckDB, Polars, Pandas. | by Mariusz Kujawski | Medium
Python and Parquet performance optimization using Pandas, PySpark, PyArrow, Dask, fastparquet and AWS S3 | Data Syndrome Blog
![Python and Parquet performance optimization using Pandas, PySpark, PyArrow, Dask, fastparquet and AWS S3 | Data Syndrome Blog Python and Parquet performance optimization using Pandas, PySpark, PyArrow, Dask, fastparquet and AWS S3 | Data Syndrome Blog](https://miro.medium.com/v2/resize:fit:1400/1*dwGGDr7HA76J6sMYt0A8iQ.png)