You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@hop.apache.org by "usbrandon (via GitHub)" <gi...@apache.org> on 2024/12/29 20:06:48 UTC
[D] Apache Arrow Support (hop)
GitHub user usbrandon created a discussion: Apache Arrow Support
Greetings. Please for renewed consideration, https://github.com/apache/hop/issues/2556 could we evaluate using Apache Arrow as the bridge between the JVM and Python process in the CPython transform?
More broadly there are probably connectivity efficiency gains to be had in connecting to more modern data sources that support ADBC
https://arrow.apache.org/docs/format/ADBC.html
Feb 2024 - Snowflake, BigQuery, Postgres, SQLite, and Pandas (think CPython 30x to 80x faster) transporting data using Arrow.
https://voltrondata.com/blog/go-inside-the-arrow-database-connectivity-roadmap-background-and-community?utm_source=chatgpt.com
Perhaps just border to border level changes might make it easy to take incremental steps towards Arrow. Being a pragmatist, I would try to prioritize the enhancement for the Cpython step to get rid of the "Server" process it spins up that can die / hang if any data is not escaped in the dataframe or variables passed between Hop and the Python process running outside the JVM. This is a matter of stability, not just data transport. A company that will not be named used py4j that came out before Arrow. It uses sockets and ports to facilitate transfers between JVM / Python Processes, but I feel like Arrow will be more portable and is not such an outlier project like VFS etc. I sense greater longevity and process with Apache Arrow.
GitHub link: https://github.com/apache/hop/discussions/4749
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Apache Arrow Support (hop)
Posted by "mattcasters (via GitHub)" <gi...@apache.org>.
GitHub user mattcasters added a comment to the discussion: Apache Arrow Support
I looked at Apache Arrow during my time at Neo4j, and I think this is quite doable.
Are you talking specifically about writing the data to an [IPC streaming Arrow file](https://arrow.apache.org/docs/java/quickstartguide.html#interprocess-communication-ipc )?
GitHub link: https://github.com/apache/hop/discussions/4749#discussioncomment-11746981
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org