You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@age.apache.org by "lmeyerov (via GitHub)" <gi...@apache.org> on 2023/02/13 15:44:11 UTC
[GitHub] [age] lmeyerov opened a new issue, #669: Bulk import/export using Arrow ?
lmeyerov opened a new issue, #669:
URL: https://github.com/apache/age/issues/669
We are thinking about using AGE as part of some compute-tier systems, so fast ingest/export matters. Think 10k-100M edges. If we have parquet or arrow format data, is there a good path here, and any doc pointers? (I could not find)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@age.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [age] rafsun42 commented on issue #669: Bulk import/export using Arrow ?
Posted by "rafsun42 (via GitHub)" <gi...@apache.org>.
rafsun42 commented on issue #669:
URL: https://github.com/apache/age/issues/669#issuecomment-1430109679
The python driver is a wrapper of psycopg2, which is a wrapper of libpq. Libpq is more specific to Postgres so you may get better performance. Generally, the driver builds SQL query string and executes through psychopg2.
I don't have exact performance metrics for large data sets. If you want to run benchmark test, I may be able to assist you here.
Link to the driver: https://github.com/apache/age/tree/master/drivers/python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@age.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [age] eyab commented on issue #669: Bulk import/export using Arrow ?
Posted by "eyab (via GitHub)" <gi...@apache.org>.
eyab commented on issue #669:
URL: https://github.com/apache/age/issues/669#issuecomment-1690451379
@lmeyerov do you have any updates on this? Did you find a solution?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@age.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [age] rafsun42 commented on issue #669: Bulk import/export using Arrow ?
Posted by "rafsun42 (via GitHub)" <gi...@apache.org>.
rafsun42 commented on issue #669:
URL: https://github.com/apache/age/issues/669#issuecomment-1429993031
Have you considered using AGE's python driver with PyArrow?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@age.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [age] lmeyerov commented on issue #669: Bulk import/export using Arrow ?
Posted by "lmeyerov (via GitHub)" <gi...@apache.org>.
lmeyerov commented on issue #669:
URL: https://github.com/apache/age/issues/669#issuecomment-1430009287
Is there an example here? I'm worried if it just converts to odbc underneath
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@age.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [age] lmeyerov commented on issue #669: Bulk import/export using Arrow ?
Posted by "lmeyerov (via GitHub)" <gi...@apache.org>.
lmeyerov commented on issue #669:
URL: https://github.com/apache/age/issues/669#issuecomment-1690546114
No, we are still curious about bulk ingest/export for use with analytics pipelines, esp subsecond interactive & GPU at 100MB-10GB scales
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@age.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [age] lmeyerov commented on issue #669: Bulk import/export using Arrow ?
Posted by "lmeyerov (via GitHub)" <gi...@apache.org>.
lmeyerov commented on issue #669:
URL: https://github.com/apache/age/issues/669#issuecomment-1430271897
Thanks -- yes, odbc for postgres is too slow for our interactive compute workloads. I think arrow community is making a new protocol w postgres support, but I haven't tracked that nor know whether it'd combine well with AGE
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@age.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [age] lmeyerov commented on issue #669: Bulk import/export using Arrow ?
Posted by "lmeyerov (via GitHub)" <gi...@apache.org>.
lmeyerov commented on issue #669:
URL: https://github.com/apache/age/issues/669#issuecomment-1690547667
I think ADBC is the relevant protocol snowflake and others are adopting here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@age.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org