You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by GitBox <gi...@apache.org> on 2021/02/03 15:24:23 UTC
[GitHub] [arrow-testing] pitrou opened a new pull request #56: ARROW-11417: Add integration files for buffer compression
pitrou opened a new pull request #56:
URL: https://github.com/apache/arrow-testing/pull/56
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-testing] pitrou commented on pull request #56: ARROW-11417: Add integration files for buffer compression
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #56:
URL: https://github.com/apache/arrow-testing/pull/56#issuecomment-772617773
Ok, I edited the JSON files by hand to remove the offending field and now Java manages to read them (**cringe**), though it fails to instantiate the compression:
```
java.lang.IllegalArgumentException: Compression type not supported: 0
at org.apache.arrow.vector.compression.CompressionUtil.createCodec(CompressionUtil.java:57)
at org.apache.arrow.vector.VectorLoader.load(VectorLoader.java:59)
at org.apache.arrow.vector.ipc.ArrowReader.loadRecordBatch(ArrowReader.java:205)
at org.apache.arrow.vector.ipc.ArrowFileReader.loadNextBatch(ArrowFileReader.java:153)
at org.apache.arrow.vector.ipc.ArrowFileReader.loadRecordBatch(ArrowFileReader.java:184)
at org.apache.arrow.tools.Integration$Command$3.execute(Integration.java:205)
at org.apache.arrow.tools.Integration.run(Integration.java:118)
at org.apache.arrow.tools.Integration.main(Integration.java:69)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-testing] pitrou merged pull request #56: ARROW-11417: Add integration files for buffer compression
Posted by GitBox <gi...@apache.org>.
pitrou merged pull request #56:
URL: https://github.com/apache/arrow-testing/pull/56
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-testing] pitrou edited a comment on pull request #56: ARROW-11417: Add integration files for buffer compression
Posted by GitBox <gi...@apache.org>.
pitrou edited a comment on pull request #56:
URL: https://github.com/apache/arrow-testing/pull/56#issuecomment-772592633
I'm not sure what the naming scheme should be. I chose "1.0.0" for the format version, but is it the Arrow library version that generated the files instead? (in which case it should be "2.0.0")
@wesm
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-testing] pitrou commented on pull request #56: ARROW-11417: Add integration files for buffer compression
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #56:
URL: https://github.com/apache/arrow-testing/pull/56#issuecomment-772601749
The JSON files are refused by the Java Arrow integration test:
```
Incompatible files
Expected END_OBJECT but got FIELD_NAME
16:30:12.557 [main] ERROR org.apache.arrow.tools.Integration - Incompatible files
java.lang.IllegalStateException: Expected END_OBJECT but got FIELD_NAME
at org.apache.arrow.vector.ipc.JsonFileReader.readToken(JsonFileReader.java:779)
at org.apache.arrow.vector.ipc.JsonFileReader.readFromJsonIntoVector(JsonFileReader.java:740)
at org.apache.arrow.vector.ipc.JsonFileReader.read(JsonFileReader.java:219)
at org.apache.arrow.tools.Integration$Command$3.execute(Integration.java:203)
at org.apache.arrow.tools.Integration.run(Integration.java:118)
at org.apache.arrow.tools.Integration.main(Integration.java:69)
Suppressed: java.lang.IllegalStateException: Memory was leaked by query. Memory leaked: (16648)
Allocator(ROOT) 0/16648/49416/2147483647 (res/actual/peak/limit)
at org.apache.arrow.memory.BaseAllocator.close(BaseAllocator.java:437)
at org.apache.arrow.memory.RootAllocator.close(RootAllocator.java:29)
at org.apache.arrow.tools.Integration$Command$3.execute(Integration.java:185)
... 2 common frames omitted
```
It looks like the problem is that it contains an empty "children" field for primitive batches, and that displeases the Java reader (which for some unknown reason uses a low-level token-by-token parsing technique).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-testing] emkornfield commented on pull request #56: ARROW-11417: Add integration files for buffer compression
Posted by GitBox <gi...@apache.org>.
emkornfield commented on pull request #56:
URL: https://github.com/apache/arrow-testing/pull/56#issuecomment-772638909
Strange, I wonder why main integration tests aren't broken for the JSON issue. Regarding versioning, I used the library version that generated them. It seems like a bug that Java can't even read uncompressed buffers? Or maybe the java error message is bad. Either way something for Java contributors to figure out. Thanks for adding these.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-testing] pitrou commented on pull request #56: ARROW-11417: Add integration files for buffer compression
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #56:
URL: https://github.com/apache/arrow-testing/pull/56#issuecomment-772592633
I'm not sure what the naming scheme should be. I chose "1.0.0" for the format version, but is it the Arrow library version that generated the files instead? (in which case it should be "2.0.0")
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-testing] pitrou commented on pull request #56: ARROW-11417: Add integration files for buffer compression
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #56:
URL: https://github.com/apache/arrow-testing/pull/56#issuecomment-772679327
I renamed to "2.0.0-compression", will merge.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-testing] pitrou commented on pull request #56: ARROW-11417: Add integration files for buffer compression
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #56:
URL: https://github.com/apache/arrow-testing/pull/56#issuecomment-772679092
> Strange, I wonder why main integration tests aren't broken for the JSON issue.
Because the JSON files that are used in the integration tests are generated by the Python datagen harness (in `archery.integration`).
The `ARROW_TO_JSON` mode of operation of native integration runners isn't used in the integration tests (as opposed to the `JSON_TO_ARROW` and `VALIDATE` mdoes).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org