You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "tisonkun (via GitHub)" <gi...@apache.org> on 2024/12/26 13:24:48 UTC
[I] How to iterate over a record batch? [arrow-java]
tisonkun opened a new issue, #468:
URL: https://github.com/apache/arrow-java/issues/468
### Describe the usage question you have. Please include as many useful details as possible.
Now I have a `VectorSchemaRoot`. I can see to iterate over the batch with `getVector` and then `getObject`.
But the return value is of type `Obejct`. And I wonder how I can downcast it for some useful class I can retrieve the real value (string, int, float, etc.).
I know we have the field info of each vetcor, but I don't know the mapping between field type to real Java class. It looks over challenge to remember all the mapping by reverse engineering the code, and it may change as version evolves.
I checked https://arrow.apache.org/docs/java/index.html but all the pages tell about constructing a batch and how to move it from one place to another, rather than tell about how to read and dump a batch to a typed two-dimensional matrix.
The most trivial usage, `contentToTSVString`, call `Object::toString` on each cell. But I don't thing we should convert all the values to String and reparse it to concrete type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] How to iterate over a record batch? [arrow-java]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #468:
URL: https://github.com/apache/arrow-java/issues/468#issuecomment-2568585745
In general I would like to see several documentation improvements:
- Replace or update the cookbook's way of handling Java recipes so we aren't embedding giant chunks of Java code inside reST (makes them a pain to write/test/update)
- Add more usage documentation and cookbook examples,
- Add general documentation on how the library is structured (Java doesn't follow the C++ convention of effectively dataframes, instead valuing memory management and streaming data)
- Use the sphinx-javadoc bridge I wrote for arrow-adbc so we can cross-link Sphinx and Javadocs more easily,
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] How to iterate over a record batch? [arrow-java]
Posted by "ParthChonkar (via GitHub)" <gi...@apache.org>.
ParthChonkar commented on issue #468:
URL: https://github.com/apache/arrow-java/issues/468#issuecomment-2564645204
The patterns I've seen/used revolved around down-casting/type matching on the FieldVector subclasses:
1. Downcast the returned FieldVector to the concrete class (IntVector, VarCharVector, etc) and then use the corresponding typed `get*` method from them
2. Use visitor pattern/method signature overloading
VectorSchemaRoot stores a schema and a corresponding bag of [FieldVectors](https://arrow.apache.org/docs/java/reference/org.apache.arrow.vector/org/apache/arrow/vector/FieldVector.html). The actual subclass is tied to the arrow type for that field. (This is gets a more tricky if you are using [nested types](https://arrow.apache.org/docs/java/vector.html#building-listvector))
For (1) you do need to know the mapping between your schema's arrow field types <-> ValueVectors subclasses and have logic to cast accordingly based on the field type. This is reflected a bit in [this example](https://arrow.apache.org/docs/java/quickstartguide.html#create-a-vectorschemaroot) the vectors have to be downcasted in order to write the values properly, this also applies to reading their values in a typed manner.
Agree that it's a bit tricky at first to map the simple types to their ValueVector subclasses (basically need to look [here](https://github.com/apache/arrow-java/blob/main/vector/src/main/java/org/apache/arrow/vector/types/Types.java)) - would be nice documentation add. It looks like there's already a stub for a table with this mapping [here](https://arrow.apache.org/docs/java/vector.html#building-listvector). ("Table with non-intuitive names").
As an aside it seems safer to cast the vector to its typed vector first rather than casting the type directly from the `FieldVector#getObject` (in case you want to fail loudly before accidentally coercing doubles/ints via a downcast).
For (2) this is letting java multiple dispatch route the FieldVector from your VectorSchemaRoot to a function that accepts its concrete subclass.
Curious what other folks approaches are/if there are conventions or patterns I might be missing here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] How to iterate over a record batch? [arrow-java]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #468:
URL: https://github.com/apache/arrow-java/issues/468#issuecomment-2568584798
The mapping should stay fixed. Unfortunately I don't think there's a way in Java to do some type-level metaprogramming like we can in the C++ library (in C++ the vector type is effectively an associated type of the...type type, so you can write `typename TypeTraits<StringType>::ArrayType`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] How to iterate over a record batch? [arrow-java]
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on issue #468:
URL: https://github.com/apache/arrow-java/issues/468#issuecomment-2564741023
@ParthChonkar Thank you! These two links help:
* https://github.com/apache/arrow-java/blob/main/vector/src/main/java/org/apache/arrow/vector/types/Types.java
* https://arrow.apache.org/docs/java/vector.html#building-listvector
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org