You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "djouallah (via GitHub)" <gi...@apache.org> on 2023/01/28 22:47:37 UTC
[GitHub] [arrow-datafusion-python] djouallah opened a new issue, #157: Out of memory when sorting
djouallah opened a new issue, #157:
URL: https://github.com/apache/arrow-datafusion-python/issues/157
**Describe the bug**
try a sort and export a parquet file using Colab generate an Out of memory error
**To Reproduce**
```
!curl -L 'https://drive.google.com/uc?export=download&id=18gv0Yd_a-Zc7CSolol8qeYVAAzSthnSN&confirm=t' > lineitem.parquet
from datafusion import SessionContext
ctx = SessionContext()
ctx.register_parquet('lineitem', 'lineitem.parquet')
df = ctx.sql("select * from lineitem order by l_shipdate")
df.write_parquet("lineitem_Datafusion.parquet")
```
**Expected behavior**
I expected to use only the available memory
here is the link comparing the same using Polars and DuckDB
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion-python] andygrove commented on issue #157: Out of memory when sorting
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #157:
URL: https://github.com/apache/arrow-datafusion-python/issues/157#issuecomment-1411203031
We need to expose the memory/disk config in the Python bindings so that they can be set when creating the SessionContext. Here is Rust code for reference.
I will try and look at this over the weekend unless someone beats me to it.
```rust
let runtime_config = crate::execution::runtime_env::RuntimeConfig::new()
.with_memory_pool(Arc::new(crate::execution::memory_pool::GreedyMemoryPool::new(1024*1024*1024)))
.with_disk_manager(crate::execution::disk_manager::DiskManagerConfig::new_specified(vec!["/Users/a/spill/".into()]));
let runtime = Arc::new(crate::execution::runtime_env::RuntimeEnv::new(runtime_config).unwrap());
let ctx = SessionContext::with_config_rt(SessionConfig::new(), runtime);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion-python] comphead commented on issue #157: Out of memory when sorting
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on issue #157:
URL: https://github.com/apache/arrow-datafusion-python/issues/157#issuecomment-1410637293
@djouallah please check some advices in https://github.com/apache/arrow-datafusion/issues/5108 on how to enable memory pool with spilling, to use only allocated memory instead of all available
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion-python] andygrove commented on issue #157: Out of memory when sorting
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #157:
URL: https://github.com/apache/arrow-datafusion-python/issues/157#issuecomment-1416777294
@djouallah Hopefully this helps: https://github.com/apache/arrow-datafusion-python/pull/165
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion-python] andygrove closed issue #157: Out of memory when sorting
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed issue #157: Out of memory when sorting
URL: https://github.com/apache/arrow-datafusion-python/issues/157
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion-python] andygrove commented on issue #157: Out of memory when sorting
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #157:
URL: https://github.com/apache/arrow-datafusion-python/issues/157#issuecomment-1407679309
I filed an issue against the core DataFusion repo - https://github.com/apache/arrow-datafusion/issues/5108
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org