You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "buhrmann (via GitHub)" <gi...@apache.org> on 2024/02/02 16:43:45 UTC
[I] Dataset-like interface for "columnar" partitioning [arrow]
buhrmann opened a new issue, #39915:
URL: https://github.com/apache/arrow/issues/39915
### Describe the enhancement requested
Hi, I was wondering whether there is a trick, or plans for a future feature, to have a Dataset interface to multiple ipc/parquet files on disk, where the files contain not different rows, but different columns? I know I can combine them cheaply in memory, but the idea would be for this interface to work everywhere that existing tools expect an arrow dataset, and to be able to access the data in streaming mode, e.g. via ibis, polars, duckdb etc.
Or perhaps there is a trick to cheaply concatenate various datasets? I can see the SQL-like `dataset.join()`, but not a simple horizontal concatenation ("join on row numbers").
### Component(s)
C++, Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Dataset-like interface for "columnar" partitioning [arrow]
Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic closed issue #39915: Dataset-like interface for "columnar" partitioning
URL: https://github.com/apache/arrow/issues/39915
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Dataset-like interface for "columnar" partitioning [arrow]
Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic commented on issue #39915:
URL: https://github.com/apache/arrow/issues/39915#issuecomment-3921319498
This issue has been closed because there has been no activity for the 14 days since the stale warning. Feel free to reopen if this has been closed in error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Dataset-like interface for "columnar" partitioning [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #39915:
URL: https://github.com/apache/arrow/issues/39915#issuecomment-3547084244
This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this improvement is still desired but has no current owner, please add the 'Status: needs champion' label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org