You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@age.apache.org by "rafsun42 (via GitHub)" <gi...@apache.org> on 2023/07/25 22:05:28 UTC

[GitHub] [age] rafsun42 opened a new issue, #1090: [Label Redesign] Update age_load

rafsun42 opened a new issue, #1090:
URL: https://github.com/apache/age/issues/1090

   (no comment)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@age.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [age] Zainab-Saad commented on issue #1090: [Label Redesign] Update age_load

Posted by "Zainab-Saad (via GitHub)" <gi...@apache.org>.
Zainab-Saad commented on issue #1090:
URL: https://github.com/apache/age/issues/1090#issuecomment-1657875288

   @rafsun42 Which field of the vertex table or edges csv should we sort? 
   The primary performance issue is that I need to bring vertex tables into memory before creating the edges (to get the id of the vertex). This is the point that needs optimization.
   I am not sure how sorting would help reduce the run time in this case.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@age.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [age] rafsun42 commented on issue #1090: [Label Redesign] Update age_load

Posted by "rafsun42 (via GitHub)" <gi...@apache.org>.
rafsun42 commented on issue #1090:
URL: https://github.com/apache/age/issues/1090#issuecomment-1658810204

   @Zainab-Saad 
   I was thinking about an algorithm that merges two sorted array. In our case, the vertex table and edge csv file. And, we are not actually merging them. We are just scanning the vertex table to get the new id. 
   
   Sort the vertex table by `id`. Sort the edge csv file by `start_id` and `end_id`. You can set up a cursor to scan the vertex table row by row. As you read the edge csv file, you can get the next vertex from the cursor, instead of scanning the entire vertex table. Because these tables are sorted, you need to scan vertex table and csv file only once. 
   
   This is just an idea. One drawback is sorting will cost time, but may be worth it for large csv file.
   
   Another idea that Shoaib suggested is to use the IDs in the csv file as actual ID (instead of getting it from the sequence), while importing vertices. Once imported, set the sequence to reflect the max ID in CSV + 1.
   
   Both of these are just ideas, you may need to think about their correctness and any potential edge cases.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@age.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [age] Zainab-Saad commented on issue #1090: [Label Redesign] Update age_load

Posted by "Zainab-Saad (via GitHub)" <gi...@apache.org>.
Zainab-Saad commented on issue #1090:
URL: https://github.com/apache/age/issues/1090#issuecomment-1658975105

   @rafsun42 I have tried analyzing the two ideas
   
   Regarding @muhammadshoaib 's idea, we can have the same performance as before (as we wont be scanning any table) but there is a loop hole what if two different csv files have the same `id` column value, this will cause clashes in vertex id. If we can find a way to go about this issue, then this idea has potential because previously too the function depended on `id` column as it used this as the `entry id` for generating `graphid`. Another issue could be that we might waste sequence numbers (I am not sure if this would be a big issue as the max sequence number is `9223372036854775807` )
   
   Regarding your idea, the only issue might be the vertex table scan (which will take the most time even if done once)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@age.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [age] panosfol commented on issue #1090: [Label Redesign] Update age_load

Posted by "panosfol (via GitHub)" <gi...@apache.org>.
panosfol commented on issue #1090:
URL: https://github.com/apache/age/issues/1090#issuecomment-1664763841

   assign me please


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@age.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [age] Zainab-Saad commented on issue #1090: [Label Redesign] Update age_load

Posted by "Zainab-Saad (via GitHub)" <gi...@apache.org>.
Zainab-Saad commented on issue #1090:
URL: https://github.com/apache/age/issues/1090#issuecomment-1660823663

   @rafsun42 
   After step 1, if we reset the sequence starting number to `max CSV ID + 1` and then create a vertex as in step 2, there would not be an issue till this point. But as we move to the step 3, if the vertex2 csv has an id that we have already used in step 2 in `CREATE(x: vertex2)` then this will cause id clash and a possible break.
   PS: I am considering here the id clash from the CREATE clause and the vertex2 csv not across multiple vertex csvs (which we are ignoring for now as per your comment)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@age.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [age] mohayu22 commented on issue #1090: [Label Redesign] Update age_load

Posted by "mohayu22 (via GitHub)" <gi...@apache.org>.
mohayu22 commented on issue #1090:
URL: https://github.com/apache/age/issues/1090#issuecomment-1651354399

   Assign this to me


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@age.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [age] rafsun42 commented on issue #1090: [Label Redesign] Update age_load

Posted by "rafsun42 (via GitHub)" <gi...@apache.org>.
rafsun42 commented on issue #1090:
URL: https://github.com/apache/age/issues/1090#issuecomment-1656099757

   @Zainab-Saad Does sorting the vertex tables and edge csv file help?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@age.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [age] Zainab-Saad commented on issue #1090: [Label Redesign] Update age_load

Posted by "Zainab-Saad (via GitHub)" <gi...@apache.org>.
Zainab-Saad commented on issue #1090:
URL: https://github.com/apache/age/issues/1090#issuecomment-1665873324

   @rafsun42 
   I have created a [PR](https://github.com/rafsun42/age/pull/23) that updates age_load and the relevant regression tests (tests pass).
   Once this is approved, we can start working on supporting multiple labels with age_load.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@age.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [age] Zainab-Saad commented on issue #1090: [Label Redesign] Update age_load

Posted by "Zainab-Saad (via GitHub)" <gi...@apache.org>.
Zainab-Saad commented on issue #1090:
URL: https://github.com/apache/age/issues/1090#issuecomment-1660841060

   @rafsun42 
   That solves the id clash but will break the edge csv (as it still has the previous ids)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@age.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [age] Zainab-Saad commented on issue #1090: [Label Redesign] Update age_load

Posted by "Zainab-Saad (via GitHub)" <gi...@apache.org>.
Zainab-Saad commented on issue #1090:
URL: https://github.com/apache/age/issues/1090#issuecomment-1659143969

   @rafsun42 
   When creating vertices from the csv, we can cater duplicate ids (for example by adding timestamps to id) but that will break the edge csv because edge csv holds the ids that were originally present in the vertex csv
   One approach could be to not allow such id duplication altogether.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@age.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [age] rafsun42 commented on issue #1090: [Label Redesign] Update age_load

Posted by "rafsun42 (via GitHub)" <gi...@apache.org>.
rafsun42 commented on issue #1090:
URL: https://github.com/apache/age/issues/1090#issuecomment-1659060463

   @Zainab-Saad 
   
   Feel free to think about any addition you would like to make to this idea to go around the issue. I and Shoaib will discuss it after.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@age.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [age] rafsun42 commented on issue #1090: [Label Redesign] Update age_load

Posted by "rafsun42 (via GitHub)" <gi...@apache.org>.
rafsun42 commented on issue #1090:
URL: https://github.com/apache/age/issues/1090#issuecomment-1660831620

   @Zainab-Saad 
   Does it solve the problem if `actual ID = next available ID in the sequence + ID in csv`?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@age.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [age] CapnSpek commented on issue #1090: [Label Redesign] Update age_load

Posted by "CapnSpek (via GitHub)" <gi...@apache.org>.
CapnSpek commented on issue #1090:
URL: https://github.com/apache/age/issues/1090#issuecomment-1719698740

   Assign this to me please.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@age.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [age] Zainab-Saad commented on issue #1090: [Label Redesign] Update age_load

Posted by "Zainab-Saad (via GitHub)" <gi...@apache.org>.
Zainab-Saad commented on issue #1090:
URL: https://github.com/apache/age/issues/1090#issuecomment-1650849928

   Assign me please


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@age.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [age] rafsun42 commented on issue #1090: [Label Redesign] Update age_load

Posted by "rafsun42 (via GitHub)" <gi...@apache.org>.
rafsun42 commented on issue #1090:
URL: https://github.com/apache/age/issues/1090#issuecomment-1660803027

   @Zainab-Saad 
   
   Let's say there is no duplication in vertex csv files. And, we are planning to use the csv IDs as actual ID.
   
   1. `load_vertex_from_csv (vertex1)`
   2. `CREATE (x:vertex1)
   3. `load_vertex_from_csv (vertex2)`
   4. `load_edge_from_csv(edge1)`
   
   Would the `CREATE` clause break this solution?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@age.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org