You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2023/01/18 16:43:40 UTC
[GitHub] [arrow] akshaysu12 opened a new pull request, #33761: GH-14932: Add python bindings for JSON streaming reader
akshaysu12 opened a new pull request, #33761:
URL: https://github.com/apache/arrow/pull/33761
### What changes are included in this PR?
This PR adds a new python function open_json() that allows for opening a streaming reader to a json file. Arguments for open_json() are the same as for read_json().
open_json() is the python binding for the streaming json reader implemented in this PR: https://github.com/apache/arrow/pull/14355
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #33761: GH-14932: Add python bindings for JSON streaming reader
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #33761:
URL: https://github.com/apache/arrow/pull/33761#issuecomment-1387371991
* Closes: #14932
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] AlenkaF commented on pull request #33761: GH-14932: [Python] Add python bindings for JSON streaming reader
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on PR #33761:
URL: https://github.com/apache/arrow/pull/33761#issuecomment-1404700904
> #14355 indicates multiple threads can be used but there's no explicit documentation at `docs/source/cpp/json.rst`.
I think the reference docs have all the info:
https://arrow.apache.org/docs/dev/cpp/api/formats.html#_CPPv4N5arrow4json11ReadOptionsE
https://arrow.apache.org/docs/dev/cpp/api/formats.html#_CPPv4N5arrow4json15StreamingReaderE
It makes sense for the CSV stream reader to have [notes about only single-threaded reader](https://arrow.apache.org/docs/python/csv.html#incremental-reading) and that there are no additional notes on `use_threads` in [Reading JSON files](https://arrow.apache.org/docs/dev/cpp/json.html#reading-json-files ) as [ReadOptions](https://arrow.apache.org/docs/dev/cpp/api/formats.html#_CPPv4N5arrow4json11ReadOptionsE) are linked from there.
> The tests I've written in test_json.py pass if use_threads is False or True. It looks like `cpp/src/arrow/json/reader_test.cc` has more specific tests for the AsyncStreamingReader.
Oh yes, That is true.
> 1. Do I need to write additional tests more specific to read_options.use_threads = True?
I think you can add something similar to `TestSerialJSONRead` and `TestParallelJSONRead`. @jorisvandenbossche what do you think?
> 2. Do I need to include documentation on read_options.use_threads?
I think it is well documented in [pa.json.ReadOptions](https://arrow.apache.org/docs/python/generated/pyarrow.json.ReadOptions.html#pyarrow.json.ReadOptions). But you can definitely mention the possibility of using multiple threads in JSON incremental reading [in the general docs](https://arrow.apache.org/docs/dev/python/json.html).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-14932: [Python] Add python bindings for JSON streaming reader [arrow]
Posted by "pan-x-c (via GitHub)" <gi...@apache.org>.
pan-x-c commented on PR #33761:
URL: https://github.com/apache/arrow/pull/33761#issuecomment-2550243670
Hi, I'm currently encountering issues that could be resolved by the changes proposed in this PR.
The current PR has not progressed for a long time. Is there a plan to continue moving forward? I can help if needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #33761: GH-14932: Add python bindings for JSON streaming reader
Posted by GitBox <gi...@apache.org>.
pitrou commented on PR #33761:
URL: https://github.com/apache/arrow/pull/33761#issuecomment-1387376655
@AlenkaF @jorisvandenbossche would one of you like to help @akshaysu12 on this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] AlenkaF commented on pull request #33761: GH-14932: Add python bindings for JSON streaming reader
Posted by GitBox <gi...@apache.org>.
AlenkaF commented on PR #33761:
URL: https://github.com/apache/arrow/pull/33761#issuecomment-1397108286
Great work so far @akshaysu12 !
Looking at the code from the binding for `CSVStreamingReader` and the work done on the C++ side for the JSON stream reader I would say this PR is already in a good shape.
As you mentioned, you should add the documentation to the https://arrow.apache.org/docs/dev/python/api/formats.html (docs/source/python/api/formats.rst)and https://arrow.apache.org/docs/dev/python/json.html (docs/source/python/json.rst).
To build the documentation locally see: https://arrow.apache.org/docs/dev/developers/documentation.html
You should also correct the linter error:
```python
--- original//arrow/python/pyarrow/tests/test_json.py
+++ fixed//arrow/python/pyarrow/tests/test_json.py
@@ -106,6 +106,7 @@
check_options_class_pickling(cls, explicit_schema=schema,
newlines_in_values=False,
unexpected_field_behavior="ignore")
+
class BaseTestJSON(abc.ABC):
@abc.abstractmethod
@@ -296,11 +297,12 @@
# Better error output
assert table.to_pydict() == expected.to_pydict()
+
class BaseTestJSONRead(BaseTestJSON):
def read_bytes(self, b, **kwargs):
return self.read_json(pa.py_buffer(b), **kwargs)
-
+
def test_file_object(self):
data = b'{"a": 1, "b": 2}\n'
expected_data = {'a': [1], 'b': [2]}
@@ -311,7 +313,7 @@
sio = io.StringIO(data.decode())
with pytest.raises(TypeError):
self.read_json(sio)
-
+
def test_reconcile_accross_blocks(self):
# ARROW-12065: reconciling inferred types across blocks
first_row = b'{ }\n'
```
To run the linter locally see: https://arrow.apache.org/docs/dev/developers/python.html#coding-style
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] AlenkaF commented on pull request #33761: GH-14932: Add python bindings for JSON streaming reader
Posted by GitBox <gi...@apache.org>.
AlenkaF commented on PR #33761:
URL: https://github.com/apache/arrow/pull/33761#issuecomment-1387464145
> @AlenkaF @jorisvandenbossche would one of you like to help @akshaysu12 on this?
Sure, I would love to help! Will look at the issue and the current state of the PR tomorrow and hope I get through it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-14932: [Python] Add python bindings for JSON streaming reader [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF closed pull request #33761: GH-14932: [Python] Add python bindings for JSON streaming reader
URL: https://github.com/apache/arrow/pull/33761
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #33761: GH-14932: Add python bindings for JSON streaming reader
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #33761:
URL: https://github.com/apache/arrow/pull/33761#issuecomment-1387372055
:warning: GitHub issue #14932 **has been automatically assigned in GitHub** to PR creator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] akshaysu12 commented on pull request #33761: GH-14932: [Python] Add python bindings for JSON streaming reader
Posted by "akshaysu12 (via GitHub)" <gi...@apache.org>.
akshaysu12 commented on PR #33761:
URL: https://github.com/apache/arrow/pull/33761#issuecomment-1402072674
@AlenkaF thanks for the feedback! Sorry I've been sitting on this a bit because I'm not sure what to do about the read_options.use_threads. It looks like the one major difference from the CSV streaming reader.
https://github.com/apache/arrow/pull/14355 indicates multiple threads can be used but there's no explicit documentation at `docs/source/cpp/json.rst`. The tests I've written in test_json.py pass if use_threads is False or True. It looks like `cpp/src/arrow/json/reader_test.cc` has more specific tests for the AsyncStreamingReader.
1. Do I need to write additional tests more specific to read_options.use_threads = True?
2. Do I need to include documentation on read_options.use_threads?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-14932: [Python] Add python bindings for JSON streaming reader [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on PR #33761:
URL: https://github.com/apache/arrow/pull/33761#issuecomment-2673384273
I will be closing this PR as the issue was already closed by the following: https://github.com/apache/arrow/pull/45084.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org