You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/03/18 16:33:27 UTC
[GitHub] [arrow] Arttii opened a new issue #12667: Dataset not supporting RecordBatchReader
Arttii opened a new issue #12667:
URL: https://github.com/apache/arrow/issues/12667
Hi, looking at the [code](https://github.com/apache/arrow/blob/master/python/pyarrow/dataset.py#L511) and the docs, the dataset should support using a RecordBatchReader, but it does not seem to be possible at the moment. Is this not implemented, or am I missing something?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on issue #12667: Dataset not supporting RecordBatchReader
Posted by GitBox <gi...@apache.org>.
wjones127 commented on issue #12667:
URL: https://github.com/apache/arrow/issues/12667#issuecomment-1072614292
That should be working just like you wrote, and is definitely an intended us case. Could you give more detail on the error you are getting?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on issue #12667: Dataset not supporting RecordBatchReader
Posted by GitBox <gi...@apache.org>.
wjones127 commented on issue #12667:
URL: https://github.com/apache/arrow/issues/12667#issuecomment-1072708793
Actually I think is change in API is intentional, given https://github.com/apache/arrow/pull/10070. We just forgot to update the docstring (which I'll create a Jira for).
Instead, you should convert the RecordBatchReader directly into a scanner using `Scanner.from_batches(reader)`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] Arttii commented on issue #12667: Dataset not supporting RecordBatchReader
Posted by GitBox <gi...@apache.org>.
Arttii commented on issue #12667:
URL: https://github.com/apache/arrow/issues/12667#issuecomment-1072673994
Running this:
```
pyarrow.__version__
df = pd.DataFrame({"a": [1, 2, 3]})
table = pyarrow.Table.from_pandas(df)
userdata_parquet_dataset= pyarrow.dataset.dataset(table)
batches=[r for r in userdata_parquet_dataset.to_batches()]
reader=pyarrow.dataset.Scanner.from_batches(batches,userdata_parquet_dataset.schema).to_reader()
dataset=pyarrow.dataset.dataset(reader)
```
gets me this:
```
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
Input In [15], in <cell line: 7>()
5 batches=[r for r in userdata_parquet_dataset.to_batches()]
6 reader=pyarrow.dataset.Scanner.from_batches(batches,userdata_parquet_dataset.schema).to_reader()
----> 7 dataset=pyarrow.dataset.dataset(reader)
File ~/workdir/duckdb/.venv/lib/python3.9/site-packages/pyarrow/dataset.py:687, in dataset(source, schema, format, filesystem, partitioning, partition_base_dir, exclude_invalid_files, ignore_prefixes)
685 return _in_memory_dataset(source, **kwargs)
686 else:
--> 687 raise TypeError(
688 'Expected a path-like, list of path-likes or a list of Datasets '
689 'instead of the given type: {}'.format(type(source).__name__)
690 )
TypeError: Expected a path-like, list of path-likes or a list of Datasets instead of the given type: RecordBatchReader
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on issue #12667: Dataset not supporting RecordBatchReader
Posted by GitBox <gi...@apache.org>.
wjones127 commented on issue #12667:
URL: https://github.com/apache/arrow/issues/12667#issuecomment-1073307721
> Is there no way to have a rereadable one-shot source?
If you want to make a one-shot source re-readable, you'll have to collect it into a table. That might work for you if it's not larger than memory.
But I think the idea of a (re-readable) dataset that wraps a flight data source has been something we've thought of, but haven't yet implemented. Here is the relevant Jira issue: https://issues.apache.org/jira/browse/ARROW-10524.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] Arttii closed issue #12667: Dataset not supporting RecordBatchReader
Posted by GitBox <gi...@apache.org>.
Arttii closed issue #12667:
URL: https://github.com/apache/arrow/issues/12667
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] Arttii commented on issue #12667: Dataset not supporting RecordBatchReader
Posted by GitBox <gi...@apache.org>.
Arttii commented on issue #12667:
URL: https://github.com/apache/arrow/issues/12667#issuecomment-1073202683
I see, thanks for clarifying. Is there no way to have a rereadable one-shot source? I was working on reading a returned reader from a Flight server in [DuckDB](https://github.com/duckdb/duckdb/pull/3257) and I there was no other good way from the API to support this at the moment. Or perhaps the data needs to be reread from the server.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] Arttii commented on issue #12667: Dataset not supporting RecordBatchReader
Posted by GitBox <gi...@apache.org>.
Arttii commented on issue #12667:
URL: https://github.com/apache/arrow/issues/12667#issuecomment-1073592479
Oke thanks for the help! I will look into the issue and see if i could try implementing it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on issue #12667: Dataset not supporting RecordBatchReader
Posted by GitBox <gi...@apache.org>.
wjones127 commented on issue #12667:
URL: https://github.com/apache/arrow/issues/12667#issuecomment-1072596427
Given what you linked, this is in PyArrow, right? It should be working, at least in 7.0.0. We have unit tests for writing with RecordBatchReaders.
What code did you try? And what version of PyArrow?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on issue #12667: Dataset not supporting RecordBatchReader
Posted by GitBox <gi...@apache.org>.
wjones127 commented on issue #12667:
URL: https://github.com/apache/arrow/issues/12667#issuecomment-1072699569
Sorry, I read your question wrong initially; I thought you were referring to `write_dataset`, which definitely does accept a RBR.
It sounds like the lines that handled RBR (introduced in [this PR](https://github.com/apache/arrow/pull/9802)) went missing. 😕 I'll file a Jira issue to get those readded.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] Arttii commented on issue #12667: Dataset not supporting RecordBatchReader
Posted by GitBox <gi...@apache.org>.
Arttii commented on issue #12667:
URL: https://github.com/apache/arrow/issues/12667#issuecomment-1072603158
@wjones127 thanks for the reply, and sorry for the lack of detail. I am using 7.0.0, but I wanted to create a dataset to read. Basically, I was trying to get a dataset to read from a Flight response. Basically:
```
reader = client.do_get(fl.Ticket('nyc/2015')).to_reader()
dataset = ds.dataset(reader)
```
Is something like this planned, or I am just misunderstanding it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org