You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@graphar.apache.org by "yangxk1 (via GitHub)" <gi...@apache.org> on 2025/09/11 09:58:24 UTC
[I] [DISCUSSION][java] Architecture Discussion for Java Repository [incubator-graphar]
yangxk1 opened a new issue, #756:
URL: https://github.com/apache/incubator-graphar/issues/756
### Describe the enhancement requested
We need to have a focused discussion on the overall architecture of the Java repository. While we’ve had preliminary conversations in comments (#738, #739), those discussions lack a consolidated summary and clear direction.
To move forward, we should revisit key architectural details, and ultimately produce a design document that will guide the development of remaining work.
### Component(s)
Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For additional commands, e-mail: commits-help@graphar.apache.org
Re: [I] [DISCUSSION][java] Architecture Discussion for Java Repository [incubator-graphar]
Posted by "keksmd (via GitHub)" <gi...@apache.org>.
keksmd commented on issue #756:
URL: https://github.com/apache/incubator-graphar/issues/756#issuecomment-5255377427
Issue-first delivery map, derived from this discussion: edge layout #943 (implementation PR #946) -> metadata parity #944 -> storage #947 -> IO contract #948 -> Parquet #949 -> ordered-source reader #950 -> writer/validator #951. The independent C++ devcontainer/toolchain blocker is #945 and gates the cross-language acceptance portions of #944, #949, and #951.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For additional commands, e-mail: commits-help@graphar.apache.org
Re: [I] [DISCUSSION][java] Architecture Discussion for Java Repository [incubator-graphar]
Posted by "SemyonSinchenko (via GitHub)" <gi...@apache.org>.
SemyonSinchenko commented on issue #756:
URL: https://github.com/apache/incubator-graphar/issues/756#issuecomment-3298863279
Or we can postpone it but leave a place for it in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For additional commands, e-mail: commits-help@graphar.apache.org
Re: [I] [DISCUSSION][java] Architecture Discussion for Java Repository [incubator-graphar]
Posted by "yangxk1 (via GitHub)" <gi...@apache.org>.
yangxk1 commented on issue #756:
URL: https://github.com/apache/incubator-graphar/issues/756#issuecomment-3299423061
I think the AST is highly related to query, but at the moment we only have simple traversal query requirements. Therefore, I think it can be built later as an "optimization" feature once the high-level functions are completed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For additional commands, e-mail: commits-help@graphar.apache.org
Re: [I] [DISCUSSION][java] Architecture Discussion for Java Repository [incubator-graphar]
Posted by "SemyonSinchenko (via GitHub)" <gi...@apache.org>.
SemyonSinchenko commented on issue #756:
URL: https://github.com/apache/incubator-graphar/issues/756#issuecomment-3299824576
My point is let's leave a place for it? At the moment we can have a dummy placeholders.
And even for simple traversal queries! For example, in the graph `people likes movies`, if I want to do something `MATCH (:people {id: [1, 2, 3]} -> [likes] -> (:movies {name: [Termninator, Matrx]} return people`, it is definitely not a bad idea to skip all the chunks from `poople` that does not contains vertices with id `1, 2, 3`. The same is true for edges `likes`: we can simply skip most of parquet chunks and read to memory only chunks, related to edges, that starts from `1, 2, 3`
etc.
Otherwise we may read to memory millions of nodes and billions of edges just to scan them and filter out most of them...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For additional commands, e-mail: commits-help@graphar.apache.org
Re: [I] [DISCUSSION][java] Architecture Discussion for Java Repository [incubator-graphar]
Posted by "yangxk1 (via GitHub)" <gi...@apache.org>.
yangxk1 commented on issue #756:
URL: https://github.com/apache/incubator-graphar/issues/756#issuecomment-3298274153
If we could pushdwon this would be very good, and we did implement it in C++ library. But we may have many storage places and formats, for example, s3 stores chunks in csv format, so it seems that we cannot read the specified rows through offset.
I'm thinking about a more general interface for `java-io-common` module.
Or if we can't pushdown, we can downgrade to memory processing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For additional commands, e-mail: commits-help@graphar.apache.org
Re: [I] [DISCUSSION][java] Architecture Discussion for Java Repository [incubator-graphar]
Posted by "yangxk1 (via GitHub)" <gi...@apache.org>.
yangxk1 commented on issue #756:
URL: https://github.com/apache/incubator-graphar/issues/756#issuecomment-3301058342
I agree to leave a place for it.
I think it's enough to leave an `interface`/`abstract class` in the `api-level`, we don't need to add level/module.
The main job of GAR is to provide the management format of storage rather than the computing engine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For additional commands, e-mail: commits-help@graphar.apache.org
Re: [I] [DISCUSSION][java] Architecture Discussion for Java Repository [incubator-graphar]
Posted by "yangxk1 (via GitHub)" <gi...@apache.org>.
yangxk1 commented on issue #756:
URL: https://github.com/apache/incubator-graphar/issues/756#issuecomment-3301534711
For Q2, I think we can provide an abstract interface, and different memory types can implement this interface.
```
interface Row{
public<T> T getValue(Long columnIndex);
}
class ArrowProxy implements Row{
public<T> T getValue(Long columnIndex) {
//use arrow library to decoding
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For additional commands, e-mail: commits-help@graphar.apache.org
Re: [I] [DISCUSSION][java] Architecture Discussion for Java Repository [incubator-graphar]
Posted by "yangxk1 (via GitHub)" <gi...@apache.org>.
yangxk1 commented on issue #756:
URL: https://github.com/apache/incubator-graphar/issues/756#issuecomment-3296672877
I have a question: Is the I/O granularity at the chunk level?
For example, in `java-io-parquet`, is each I/O operation reading one entire chunk, and then filtering and computation within the chunk are done in memory?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For additional commands, e-mail: commits-help@graphar.apache.org
Re: [I] [DISCUSSION][java] Architecture Discussion for Java Repository [incubator-graphar]
Posted by "SemyonSinchenko (via GitHub)" <gi...@apache.org>.
SemyonSinchenko commented on issue #756:
URL: https://github.com/apache/incubator-graphar/issues/756#issuecomment-3298535793
Like Array[Filter] pushDownPredicates, Array[String] pushDownProjections and concrete implementation of io should decide what do with them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For additional commands, e-mail: commits-help@graphar.apache.org
Re: [I] [DISCUSSION][java] Architecture Discussion for Java Repository [incubator-graphar]
Posted by "SemyonSinchenko (via GitHub)" <gi...@apache.org>.
SemyonSinchenko commented on issue #756:
URL: https://github.com/apache/incubator-graphar/issues/756#issuecomment-3298495013
I think "common" should pushdown to "io" as much as possible and "io" will do or not do optimizations. io-parquet can scan headers and use index table, but csv can only get offsets of rows from index table and connot do min-max skipping.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For additional commands, e-mail: commits-help@graphar.apache.org
Re: [I] [DISCUSSION][java] Architecture Discussion for Java Repository [incubator-graphar]
Posted by "SemyonSinchenko (via GitHub)" <gi...@apache.org>.
SemyonSinchenko commented on issue #756:
URL: https://github.com/apache/incubator-graphar/issues/756#issuecomment-3297328060
> I have some questions:
> Q1: Is the I/O granularity at the chunk level?
> For example, in `java-io-parquet`, is each I/O operation reading one entire chunk, and then filtering and computation within the chunk are done in memory?
In that case we lost all the benfits of GAR (except better compression). Because GAR data is sorted by IDs and we have also index table, we can befit of it doing pushdows and in case, for example, user wants to read only 2-hop neighborhoud of small subset of IDs in the Graph, we must check first index table and min-max statistics per column stored in headers of parquet files to skip most of chunks. Reading and filtering in memory sounds crazy for me, like why?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For additional commands, e-mail: commits-help@graphar.apache.org
Re: [I] [DISCUSSION][java] Architecture Discussion for Java Repository [incubator-graphar]
Posted by "keksmd (via GitHub)" <gi...@apache.org>.
keksmd commented on issue #756:
URL: https://github.com/apache/incubator-graphar/issues/756#issuecomment-5254632668
I have started the implementation with a small, format-compatible first slice: PR #942 fixes the pure-Java `EdgeInfo` path contract and adds fixture-backed tests against the existing Parquet LDBC data.
The accompanying architecture and compatibility documents make these decisions explicit:
1. The logical request is range-oriented, not "read a whole chunk then filter in memory." The graph layer resolves an offset-derived edge range and sends URI, projection, range, predicate, and limit hints to the physical reader.
2. Each backend reports which hints it used. Parquet can push projection/range/predicate work to row groups; a weaker backend may decline a hint without changing result semantics.
3. `graphar-info` stays dependency-light. The initial reader vertical is local storage + Parquet + `ordered_by_source`; it does not make Arrow mandatory and does not introduce a general query AST.
4. The API boundary leaves a place for future planning/AST work without making a query engine a prerequisite for useful traversal reads.
The first code gate is deliberately metadata-only: C++-compatible edge count, offset, topology, and property locations, including the required `(adjacency type, vertex chunk, edge chunk)` tuple. It passes `mvn clean verify` in the project devcontainer (124 tests, no failures). This gives the IO layer an unambiguous physical-layout contract before new storage or Parquet modules are added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For additional commands, e-mail: commits-help@graphar.apache.org
Re: [I] [DISCUSSION][java] Architecture Discussion for Java Repository [incubator-graphar]
Posted by "shivendra-dev54 (via GitHub)" <gi...@apache.org>.
shivendra-dev54 commented on issue #756:
URL: https://github.com/apache/incubator-graphar/issues/756#issuecomment-3755361118
Hi @SemyonSinchenko @yangxk1 @Thespica ,
I’ve been following the discussion in #756 and drafted a proposal for a pure Java GraphAr reader/writer using Arrow.
Before finalizing, I wanted to confirm whether any active implementation work has already begun in this direction.
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For additional commands, e-mail: commits-help@graphar.apache.org
Re: [I] [DISCUSSION][java] Architecture Discussion for Java Repository [incubator-graphar]
Posted by "yangxk1 (via GitHub)" <gi...@apache.org>.
yangxk1 commented on issue #756:
URL: https://github.com/apache/incubator-graphar/issues/756#issuecomment-3296811843
Q2. After the data is read into memory, what memory structure can be used, whether to use arrow or create a new class, such as GarTable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For additional commands, e-mail: commits-help@graphar.apache.org
Re: [I] [DISCUSSION][java] Architecture Discussion for Java Repository [incubator-graphar]
Posted by "yangxk1 (via GitHub)" <gi...@apache.org>.
yangxk1 commented on issue #756:
URL: https://github.com/apache/incubator-graphar/issues/756#issuecomment-3298678162
Do you think io level is a proxy for io? That is, an interface similar to the following is provided at io level:
```
interface FileReader{
Array<Row> readFile(URI fileUri,Fileters filters,Projections projections,...);
}
class ParquetReaderProxy implements FileReader{
public Array<Row> readFile(URI fileUri,Fileters filters,Projections projections,...) {
//use apache-parquet library to read file
}
}
```
and in higher levels, we provide access interface:
```
class PorpertyGroupReader{
FileReader fileReader;
Array<Row> readFromFile(int chunkIndex, PropertyGroup pg, URI baseUri){
//use baseUri to openFile
//use chunkIndex to pushDown
//use pg to decoding properties
return fileReader.readFile(...,...,...)
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For additional commands, e-mail: commits-help@graphar.apache.org
Re: [I] [DISCUSSION][java] Architecture Discussion for Java Repository [incubator-graphar]
Posted by "SemyonSinchenko (via GitHub)" <gi...@apache.org>.
SemyonSinchenko commented on issue #756:
URL: https://github.com/apache/incubator-graphar/issues/756#issuecomment-3298862351
What do you think about building a whole AST of the user query in a top level `PorpertyGroupReader` (or even on a level above)? It allows us to determine filters/projection and push them to the `Array<Row> readFile(URI fileUri,Fileters filters,Projections projections,...);`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For additional commands, e-mail: commits-help@graphar.apache.org
Re: [I] [DISCUSSION][java] Architecture Discussion for Java Repository [incubator-graphar]
Posted by "VidyaTrimbake (via GitHub)" <gi...@apache.org>.
VidyaTrimbake commented on issue #756:
URL: https://github.com/apache/incubator-graphar/issues/756#issuecomment-3321423127
I/O Layer
1. readFile(fileUri, filters, projections, chunkIndex) – read data from a file.
2. supportsPushdown() – check if filters/projections can be applied.
High-Level Layer
1. readFromFile(chunkIndex, propertyGroup, baseUri) – read a specific chunk.
2. readAll(propertyGroup, baseUri) – read all data.
Memory Layer
1. getValue(columnIndex) – get value from a row.
2. getSchema() – get column types.
Future Optimization (placeholder)
1. buildAST(query) – plan for query optimization.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For additional commands, e-mail: commits-help@graphar.apache.org
Re: [I] [DISCUSSION][java] Architecture Discussion for Java Repository [incubator-graphar]
Posted by "keksmd (via GitHub)" <gi...@apache.org>.
keksmd commented on issue #756:
URL: https://github.com/apache/incubator-graphar/issues/756#issuecomment-5254816201
Execution is now tracked issue-first: #943 covers C++-compatible edge layout and #944 covers remaining metadata parity. The earlier out-of-order PR #942 was closed; future code PRs will link their completed issue and its acceptance evidence.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For additional commands, e-mail: commits-help@graphar.apache.org