You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/06/20 23:50:57 UTC
[GitHub] [arrow] adzcai opened a new issue, #13405: [Java] Loading complex types from a ArrowRecordBatch
adzcai opened a new issue, #13405:
URL: https://github.com/apache/arrow/issues/13405
Hello, I've been having some issues with loading a Parquet file with complex types into Arrow for Java, and I was hoping here would be a good place to ask.
I've written the following table to a Parquet file:
```none
lists: list<item: int8>
child 0, item: int8
structs: struct<x: int64, y: bool>
child 0, x: int64
child 1, y: bool
----
lists: [[[3,4,5],[6,7]]]
structs: [
-- is_valid: all not null
-- child 0 type: int64
[1,3]
-- child 1 type: bool
[false,true]]
```
Then, I'm trying to load this file into a `VectorSchemaRoot` in Java with a `VectorLoader` using [this code](https://gist.github.com/adzcai/4e1a9c130341ad8073cb1976174b1301) according to [the Java Arrow documentation](https://arrow.apache.org/docs/java/vector_schema_root.html). However, it breaks with the following error:
```none
Exception in thread "main" java.util.NoSuchElementException
at java.util.ArrayList$Itr.next(ArrayList.java:864)
at org.apache.arrow.vector.VectorLoader.loadBuffers(VectorLoader.java:104)
at org.apache.arrow.vector.VectorLoader.loadBuffers(VectorLoader.java:133)
at org.apache.arrow.vector.VectorLoader.load(VectorLoader.java:84)
at Scratch.main(scratch.java:57)
Suppressed: java.lang.IllegalStateException: Memory was leaked by query. Memory leaked: (41998)
Allocator(ROOT) 0/41998/41998/9223372036854775807 (res/actual/peak/limit)
at org.apache.arrow.memory.BaseAllocator.close(BaseAllocator.java:437)
at org.apache.arrow.memory.RootAllocator.close(RootAllocator.java:29)
at Scratch.main(scratch.java:61)
```
I'm not super familiar with Arrow internals, but after some quick debugging, it seems that in some places, a List field takes up two buffers, while in other places it only takes up one. But I could be mistaken. The code works fine for non-complex types.
Any advice would be greatly appreciated! Thanks for your time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lwhite1 commented on issue #13405: [Java] Loading complex types from a ArrowRecordBatch
Posted by GitBox <gi...@apache.org>.
lwhite1 commented on issue #13405:
URL: https://github.com/apache/arrow/issues/13405#issuecomment-1182004773
Hi @adzcai, I looked into this a bit more.
The method ScanTask#batchIterator that you're using was removed in Arrow 8, when the Arrow Datasets Parquet reader was re-implemented to use the c-data-interface. While the Parquet reader has no tests to demonstrate that complex type supports work, the c-data-interface code does test for all types.
With the new implementation complex types should be supported. If you're able to update to the latest Arrow version you should be ok, but some code modifications will be required.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] adzcai commented on issue #13405: [Java] Loading complex types from a ArrowRecordBatch
Posted by GitBox <gi...@apache.org>.
adzcai commented on issue #13405:
URL: https://github.com/apache/arrow/issues/13405#issuecomment-1182892598
Hello, yes, I updated to `arrow-dataset` v8.0.0 and everything works now! Thanks for the clarification.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] adzcai closed issue #13405: [Java] Loading complex types from a ArrowRecordBatch
Posted by GitBox <gi...@apache.org>.
adzcai closed issue #13405: [Java] Loading complex types from a ArrowRecordBatch
URL: https://github.com/apache/arrow/issues/13405
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lwhite1 commented on issue #13405: [Java] Loading complex types from a ArrowRecordBatch
Posted by GitBox <gi...@apache.org>.
lwhite1 commented on issue #13405:
URL: https://github.com/apache/arrow/issues/13405#issuecomment-1180836703
Hi @adzcai, could you say which version of Arrow you are using? Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lwhite1 commented on issue #13405: [Java] Loading complex types from a ArrowRecordBatch
Posted by GitBox <gi...@apache.org>.
lwhite1 commented on issue #13405:
URL: https://github.com/apache/arrow/issues/13405#issuecomment-1187642568
I think this ticket can be closed by anyone with permission. Cheers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org