You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@graphar.apache.org by "keksmd (via GitHub)" <gi...@apache.org> on 2026/08/19 12:37:59 UTC

[PR] feat(java): let the storage API reach the local filesystem [incubator-graphar]

keksmd opened a new pull request, #960:
URL: https://github.com/apache/incubator-graphar/pull/960

   ### Reason for this PR
   
   Closes the remaining half of #953. The storage API merged in #958 defines
   `Storage`, `InputFile`, `OutputFile`, `SeekableInput`, and `PositionOutput`,
   but no implementation ships with it, so nothing in the Java tree can open a
   GraphAr file yet. Every later layer of #947 (physical IO, Parquet backend,
   reader, writer) needs at least one working adapter to be testable.
   
   ### What changes are included in this PR?
   
   A local filesystem adapter in a new `graphar-storage-local` module:
   
   - `LocalStorage` resolves a `file:` URI (and a bare path) to a `java.nio.Path`
     and hands back input and output files.
   - `LocalSeekableInput` reads at an absolute position and reports the end of the
     stream instead of a short buffer.
   - `LocalOutputFile` / `LocalPositionOutput` create parent directories on demand,
     refuse an existing target unless the caller asked to replace it, and report
     the number of bytes written so far.
   
   The module depends only on `graphar-storage-api`. It carries no GraphAr layout,
   format, projection, or query concern, per the boundary agreed in #953.
   
   ### Are these changes tested?
   
   Yes. `LocalStorageTest` covers the round trip, positional reads, end-of-stream
   behaviour, directory creation, and the refusal to overwrite an existing target.
   
   ```
   mvn --no-transfer-progress -pl storage-local -am clean verify
   Tests run: 4, Failures: 0, Errors: 0, Skipped: 0
   BUILD SUCCESS
   ```
   
   ### Are there any user-facing changes?
   
   A new module and package `org.apache.graphar.storage.local`. Nothing existing
   changes.
   
   ## Checklist
   
   - [x] I have performed a self-review of my own code.
   - [ ] I have formatted my own code using `make cpplint` before submitting when changed files are in the `cpp` directory.
   - [x] I have performed `pre-commit run` before commit the changed files.
   - [x] I have added tests to prove my changes are effective.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For additional commands, e-mail: commits-help@graphar.apache.org


Re: [PR] feat(java): let the storage API reach the local filesystem [incubator-graphar]

Posted by "SemyonSinchenko (via GitHub)" <gi...@apache.org>.
SemyonSinchenko commented on PR #960:
URL: https://github.com/apache/incubator-graphar/pull/960#issuecomment-5375519898

   @SYaoJun @yangxk1 Hi! Let's merge this one? Java module is a work-in-progress so I see no problems at all.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For additional commands, e-mail: commits-help@graphar.apache.org


Re: [PR] feat(java): let the storage API reach the local filesystem [incubator-graphar]

Posted by "SemyonSinchenko (via GitHub)" <gi...@apache.org>.
SemyonSinchenko commented on code in PR #960:
URL: https://github.com/apache/incubator-graphar/pull/960#discussion_r3831756295


##########
maven-projects/storage-local/src/main/java/org/apache/graphar/storage/local/LocalPositionOutput.java:
##########
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.graphar.storage.local;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.ByteBuffer;
+import org.apache.graphar.storage.PositionOutput;
+
+final class LocalPositionOutput implements PositionOutput {
+    private static final int BUFFER_SIZE = 8192;
+
+    private final OutputStream output;
+    private long position;
+
+    LocalPositionOutput(OutputStream output) {
+        this.output = output;
+    }
+
+    @Override
+    public long position() {
+        return position;
+    }
+
+    @Override
+    public void write(ByteBuffer source) throws IOException {
+        if (source.hasArray()) {
+            int length = source.remaining();
+            write(source.array(), source.arrayOffset() + source.position(), length);
+            source.position(source.position() + length);
+            return;
+        }
+
+        byte[] buffer = new byte[Math.min(source.remaining(), BUFFER_SIZE)];

Review Comment:
   Should we create and reuse the instance' own buffer (of the size `BUFFER_SIZE`) instead of the new allocation per call? It is synchronous (not thread safe dues to `position`), so I see no problem at all.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For additional commands, e-mail: commits-help@graphar.apache.org


Re: [PR] feat(java): let the storage API reach the local filesystem [incubator-graphar]

Posted by "SYaoJun (via GitHub)" <gi...@apache.org>.
SYaoJun merged PR #960:
URL: https://github.com/apache/incubator-graphar/pull/960


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For additional commands, e-mail: commits-help@graphar.apache.org


Re: [PR] feat(java): let the storage API reach the local filesystem [incubator-graphar]

Posted by "keksmd (via GitHub)" <gi...@apache.org>.
keksmd commented on code in PR #960:
URL: https://github.com/apache/incubator-graphar/pull/960#discussion_r3832722654


##########
maven-projects/storage-local/src/main/java/org/apache/graphar/storage/local/LocalPositionOutput.java:
##########
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.graphar.storage.local;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.ByteBuffer;
+import org.apache.graphar.storage.PositionOutput;
+
+final class LocalPositionOutput implements PositionOutput {
+    private static final int BUFFER_SIZE = 8192;
+
+    private final OutputStream output;
+    private long position;
+
+    LocalPositionOutput(OutputStream output) {
+        this.output = output;
+    }
+
+    @Override
+    public long position() {
+        return position;
+    }
+
+    @Override
+    public void write(ByteBuffer source) throws IOException {
+        if (source.hasArray()) {
+            int length = source.remaining();
+            write(source.array(), source.arrayOffset() + source.position(), length);
+            source.position(source.position() + length);
+            return;
+        }
+
+        byte[] buffer = new byte[Math.min(source.remaining(), BUFFER_SIZE)];

Review Comment:
   Done in 3dde49e. The staging array is now one instance field, allocated on first use so an output that only ever sees heap-backed buffers never pays for it.
   
   While writing it I found the direct-buffer path had no test that crossed a chunk boundary (the existing one wrote 2 bytes), so I added one for a 20000-byte direct buffer. With the copy loop reduced to a single pass it fails with `expected:<20000> but was:<8192>`, so it does hold the loop down.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For additional commands, e-mail: commits-help@graphar.apache.org