You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "igor-suhorukov (via GitHub)" <gi...@apache.org> on 2023/03/23 05:27:51 UTC
[GitHub] [arrow] igor-suhorukov opened a new issue, #34699: [Java] VectorSchemaRoot convertion into ArrowArrayStream
igor-suhorukov opened a new issue, #34699:
URL: https://github.com/apache/arrow/issues/34699
### Describe the enhancement requested
DuckDB support ArrowArrayStream as virtual table. JDBC driver of DuckDB has "register" ArrowStream method on DuckDBConnection.
VectorSchemaRoot convertion into ArrowArrayStream object will allows perform complex SQL transformation with CTE,window function on existing Arrow data from java process without data format transformation and also as workaround to write parquet file from VectorSchemaRoot.
@lidavidm do you have idea how to implement such conversion?
### Component(s)
Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on issue #34699: [Java] VectorSchemaRoot convertion into ArrowArrayStream
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #34699:
URL: https://github.com/apache/arrow/issues/34699#issuecomment-1481075603
I think you'd have to write an ArrowReader implementation that wraps a VectorSchemaRoot, like this: https://github.com/apache/arrow-adbc/blob/main/java/driver/flight-sql/src/main/java/org/apache/arrow/adbc/driver/flightsql/RootArrowReader.java
@davisusanibar @lwhite1 this might be the kind of thing to just add to arrow-vector (C++/Python have similar adapters)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] davisusanibar commented on issue #34699: [Java] VectorSchemaRoot convertion into ArrowArrayStream
Posted by "davisusanibar (via GitHub)" <gi...@apache.org>.
davisusanibar commented on issue #34699:
URL: https://github.com/apache/arrow/issues/34699#issuecomment-1520495025
> I think you'd have to write an ArrowReader implementation that wraps a VectorSchemaRoot, like this: https://github.com/apache/arrow-adbc/blob/main/java/driver/flight-sql/src/main/java/org/apache/arrow/adbc/driver/flightsql/RootArrowReader.java
>
> @davisusanibar @lwhite1 this might be the kind of thing to just add to arrow-vector (C++/Python have similar adapters)
Please @lidavidm could you help me to provide a link for adapters on C++/Python?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on issue #34699: [Java] VectorSchemaRoot convertion into ArrowArrayStream
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #34699:
URL: https://github.com/apache/arrow/issues/34699#issuecomment-1591073586
Ah, yes, it's redundant
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on issue #34699: [Java] VectorSchemaRoot convertion into ArrowArrayStream
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #34699:
URL: https://github.com/apache/arrow/issues/34699#issuecomment-1520931626
Table.to_batches
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] igor-suhorukov commented on issue #34699: [Java] VectorSchemaRoot convertion into ArrowArrayStream
Posted by "igor-suhorukov (via GitHub)" <gi...@apache.org>.
igor-suhorukov commented on issue #34699:
URL: https://github.com/apache/arrow/issues/34699#issuecomment-1481911931
Thank you @lidavidm. It works for me with copy of [RootArrowReader.java](https://github.com/apache/arrow-adbc/blob/main/java/driver/flight-sql/src/main/java/org/apache/arrow/adbc/driver/flightsql/RootArrowReader.java) in my classpath.
Good idea add it to arrow-vector module!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ssvendsen commented on issue #34699: [Java] VectorSchemaRoot convertion into ArrowArrayStream
Posted by "ssvendsen (via GitHub)" <gi...@apache.org>.
ssvendsen commented on issue #34699:
URL: https://github.com/apache/arrow/issues/34699#issuecomment-1590745970
To help with my understanding: In the referred RootArrowReader, is there a reason why a new VectorLoader is created in loadNextBatch() at [line 52](https://github.com/apache/arrow-adbc/blob/main/java/driver/flight-sql/src/main/java/org/apache/arrow/adbc/driver/flightsql/RootArrowReader.java#LL52C46-L52C46)?
Would it not work to just call loadRecordBatch to let the VectorLoader of the parent ArrowReader handle the loading?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org