You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@age.apache.org by GitBox <gi...@apache.org> on 2022/12/27 00:57:48 UTC
[GitHub] [age] marodins opened a new issue, #432: Allow load_labels_from_file and load_edges_from_file to accept file data directly
marodins opened a new issue, #432:
URL: https://github.com/apache/age/issues/432
It seems that there is a limit to the quantity of inserts using 'CREATE' in cypher with PG due to the nature of the query. Most importantly creating and inserting a large quantity of nodes when generated by reading a csv file client side with CREATE queries is much slower than utilizing load_labels_from_file and load_edges_from_file functions implemented as a part of the extension.
I understand that from the age documentation when creating multiple vertices we can create a query like so
SELECT *
FROM cypher('graph_name', $$
CREATE (n), (m), (e), ...
$$) as (v agtype);
when creating any number of nodes over 1664 an error is thrown by PG stating
“target lists can have at most 1664 entries” error code 54011. The solution in this case is to create individual queries for each node. Individual create queries for each node take a very long time to process by the backend. Whereas the functions stated above "load_labels_from_file" e.g. is much faster but only works with a direct path to a file.
Possible Solution:
Implement load_labels_from_file and load_edges_from_file to accept a sequence of bytes rather than just a file path to parse csv formatted data. This will allow quick client side initialization of data from any client side application.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@age.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Allow load_labels_from_file and load_edges_from_file to accept file data directly [age]
Posted by "Ameer2896 (via GitHub)" <gi...@apache.org>.
Ameer2896 commented on issue #432:
URL: https://github.com/apache/age/issues/432#issuecomment-3315951065
@jrgemignani
Any plan to incorporate client side loading that is similar to load_labels_from_file or load_edges_from_file?
I have been testing loading directly from PG tables using similar code as these functions, but am not sure of the usefulness of it. The idea is to load data into a table from the client using COPY, and call AGE functions to create vertex/labels similarly to the csv loading.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@age.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Allow load_labels_from_file and load_edges_from_file to accept file data directly [age]
Posted by "jrgemignani (via GitHub)" <gi...@apache.org>.
jrgemignani commented on issue #432:
URL: https://github.com/apache/age/issues/432#issuecomment-3320711152
@DavideDelVecchio @Ameer2896 This tool might be a better choice going forward, **in the future**, and it works from the client side. I say, **in the future**, as there is a bug with the edge load creating duplicates. However, there is a fix in the works that should be available shortly.
[https://github.com/rioriost/agefreighter](https://github.com/rioriost/agefreighter)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@age.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Allow load_labels_from_file and load_edges_from_file to accept file data directly [age]
Posted by "Ameer2896 (via GitHub)" <gi...@apache.org>.
Ameer2896 commented on issue #432:
URL: https://github.com/apache/age/issues/432#issuecomment-3321590311
@jrgemignani
Looking into AGEFreighter, I can see it is using COPY for fast loading but bypassing the normal flow of node/edge creation in AGE (for example manually inserting rows and updating label sequences).
Seems like a good solution for now. Though it may leave room for error in the future, mainly due to AGE not supporting this operation natively.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@age.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Allow load_labels_from_file and load_edges_from_file to accept file data directly [age]
Posted by "DavideDelVecchio (via GitHub)" <gi...@apache.org>.
DavideDelVecchio commented on issue #432:
URL: https://github.com/apache/age/issues/432#issuecomment-2656128184
I think this can be helpful for loading data from a local client not working directly on the server machine. In a way similar to how the COPY command works, that lets data to be copied from a client running on a different machine than the PG node: Both Edge Loading (ag_load_edges.c) and Label Loading (ag_load_labels.c) use fopen to read the file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@age.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org