You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "wwj6591812 (via GitHub)" <gi...@apache.org> on 2026/06/12 08:30:09 UTC

[PR] [Blob] Support blob-write-null-on-missing-file for HTTP URLs. [paimon]

wwj6591812 opened a new pull request, #8219:
URL: https://github.com/apache/paimon/pull/8219

   ### Background
   
   `blob-write-null-on-missing-file` already allows Flink writes to store `NULL` instead of failing when a descriptor BLOB points to a missing file on filesystem-based URIs (`file://`, `hdfs://`, `oss://`, etc.).
   
   However, for `http://` / `https://` URIs, `UriReaderFactory.exists()` previously always returned `true` for HTTP readers, so the pre-check never ran. As a result, missing HTTP resources (e.g. CDN image URLs returning 404) were only detected during the actual blob read in `BlobFormatWriter`, causing the whole write task to fail.
   
   This is a common issue when ingesting external image URLs into Paimon blob tables.
   
   ### Purpose
   
   This PR makes `blob-write-null-on-missing-file` work for HTTP/HTTPS descriptor URLs.
   
   **Primary path (pre-check):**
   - Add real HTTP existence checks via `HttpClientUtils.exists()` (HEAD first, fallback to lightweight `Range: bytes=0-0` GET when HEAD is not supported)
   - Wire `HttpUriReader.exists()` into `UriReaderFactory.exists()`
   - Reuse the existing Flink write path in `FlinkRowWrapper.isNullAt()` so missing HTTP descriptors are treated as `NULL` before write
   
   **Fallback path (write-time):**
   - In `BlobFormatWriter`, when `writeNullOnMissingFile` is enabled, catch HTTP 404 during blob read and write `NULL` with a warning instead of failing the task
   - Thread the option through `BlobFileContext` → `MultipleBlobFileWriter` / `ExternalStorageBlobWriter` → `BlobFileFormat`
   
   With this change, bad HTTP URLs are skipped gracefully while the job continues, and the failing URL is logged for troubleshooting.
   
   ### Tests
   
   - `HttpClientUtilsTest`
     - HTTP 200 → `exists() == true`
     - HTTP 404 → `exists() == false`
     - HEAD not allowed (405) → fallback GET check
     - `isNotFoundError()` helper
   - `UriReaderFactoryTest`
     - HTTP missing resource → `exists() == false`
     - HTTP available resource → `exists() == true`
   - `BlobFormatWriterTest`
     - write-path 404 fallback writes `NULL` when enabled
     - 404 still fails when option is disabled
   - `FlinkRowWrapperTest`
     - HTTP 404 descriptor → `isNullAt() == true` when checking enabled
     - HTTP 200 descriptor → `isNullAt() == false` when checking enabled


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Blob] Support blob-write-null-on-missing-file for HTTP URLs. [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #8219:
URL: https://github.com/apache/paimon/pull/8219#discussion_r3445301688


##########
paimon-api/src/main/java/org/apache/paimon/rest/HttpClientUtils.java:
##########
@@ -86,9 +88,69 @@ private static HttpClientConnectionManager configureConnectionManager() {
     public static InputStream getAsInputStream(String uri) throws IOException {
         HttpGet httpGet = new HttpGet(uri);
         CloseableHttpResponse response = DEFAULT_HTTP_CLIENT.execute(httpGet);
-        if (response.getCode() != 200) {
-            throw new RuntimeException("HTTP error code: " + response.getCode());
+        int statusCode = response.getCode();
+        if (statusCode != HttpStatus.SC_OK) {
+            try {
+                throw httpError(statusCode);
+            } finally {
+                response.close();
+            }
         }
         return response.getEntity().getContent();
     }
+
+    /**
+     * Checks whether an HTTP resource exists. HEAD is attempted first; when HEAD does not return
+     * 200, a lightweight GET with {@code Range: bytes=0-0} is used to verify readability. This
+     * avoids treating signed or GET-only URLs as missing when HEAD is rejected or returns a
+     * different status than GET.
+     */
+    public static boolean exists(String uri) throws IOException {
+        int headStatusCode = headStatusCode(uri);
+        if (headStatusCode == HttpStatus.SC_OK) {
+            return true;
+        }
+        int rangeStatusCode = getRangeStatusCode(uri);
+        if (rangeStatusCode == HttpStatus.SC_OK

Review Comment:
   The range fallback still treats a valid empty HTTP resource as an error. If HEAD is rejected or unavailable, `Range: bytes=0-0` against a zero-length object commonly returns 416 because the requested range is unsatisfiable, even though a normal GET would succeed with an empty body. In that case `exists()` throws here, so Flink writes using `blob-write-null-on-missing-file` can fail for existing empty HTTP blobs. Could we handle 416 as an existing resource for this probe (or fall back to a full GET) and add a test with HEAD disabled plus an empty response?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Blob] Support blob-write-null-on-missing-file for HTTP URLs. [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #8219:
URL: https://github.com/apache/paimon/pull/8219#discussion_r3408926391


##########
paimon-format/src/main/java/org/apache/paimon/format/blob/BlobFormatWriter.java:
##########
@@ -97,21 +114,35 @@ public void addElement(InternalRow element) throws IOException {
         }
 
         long previousPos = out.getPos();
-        crc32.reset();
-
-        write(MAGIC_NUMBER_BYTES);
-
-        long blobPos = out.getPos();
+        ByteArrayOutputStream stagedPayload = new ByteArrayOutputStream();

Review Comment:
   Could we avoid staging the entire payload in memory here? This writer is used for normal blob writes too, so the new `ByteArrayOutputStream` + `toByteArray()` path can allocate roughly 2x the blob size before writing. Since BLOBs are expected to be large and `blob.target-file-size` can be much larger than task memory, a valid large blob can now OOM even when `writeNullOnMissingFile` is false. One safer approach is to open the source stream before appending anything to `out`; if opening hits 404, write NULL, otherwise keep the previous streaming copy path.



##########
paimon-api/src/main/java/org/apache/paimon/rest/HttpClientUtils.java:
##########
@@ -86,9 +88,61 @@ private static HttpClientConnectionManager configureConnectionManager() {
     public static InputStream getAsInputStream(String uri) throws IOException {
         HttpGet httpGet = new HttpGet(uri);
         CloseableHttpResponse response = DEFAULT_HTTP_CLIENT.execute(httpGet);
-        if (response.getCode() != 200) {
-            throw new RuntimeException("HTTP error code: " + response.getCode());
+        int statusCode = response.getCode();
+        if (statusCode != HttpStatus.SC_OK) {
+            throw httpError(statusCode);
         }
         return response.getEntity().getContent();
     }
+
+    /**
+     * Checks whether an HTTP resource exists. HEAD is attempted first; when the server does not
+     * support HEAD, a lightweight GET with {@code Range: bytes=0-0} is used as fallback.
+     */
+    public static boolean exists(String uri) throws IOException {
+        int statusCode = headStatusCode(uri);

Review Comment:
   Is it safe to make the HEAD result authoritative here? Some HTTP resources, including signed or GET-only URLs, can reject HEAD or return a different status than GET even though the payload is readable. With this change, a HEAD 404/403 would make `exists()` return false or throw before trying the range GET, so an existing blob could be treated as missing or fail the write. Could we fall back to `Range: bytes=0-0` GET for non-success HEAD statuses as well, at least for 403/404, and only return false when the GET also returns 404?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Blob] Support blob-write-null-on-missing-file for HTTP URLs. [paimon]

Posted by "wwj6591812 (via GitHub)" <gi...@apache.org>.
wwj6591812 commented on PR #8219:
URL: https://github.com/apache/paimon/pull/8219#issuecomment-4776737826

   @JingsongLi 
   Hi,Please CC, Thx


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Blob] Support blob-write-null-on-missing-file for HTTP URLs. [paimon]

Posted by "wwj6591812 (via GitHub)" <gi...@apache.org>.
wwj6591812 commented on code in PR #8219:
URL: https://github.com/apache/paimon/pull/8219#discussion_r3445759716


##########
paimon-api/src/main/java/org/apache/paimon/rest/HttpClientUtils.java:
##########
@@ -86,9 +88,69 @@ private static HttpClientConnectionManager configureConnectionManager() {
     public static InputStream getAsInputStream(String uri) throws IOException {
         HttpGet httpGet = new HttpGet(uri);
         CloseableHttpResponse response = DEFAULT_HTTP_CLIENT.execute(httpGet);
-        if (response.getCode() != 200) {
-            throw new RuntimeException("HTTP error code: " + response.getCode());
+        int statusCode = response.getCode();
+        if (statusCode != HttpStatus.SC_OK) {
+            try {
+                throw httpError(statusCode);
+            } finally {
+                response.close();
+            }
         }
         return response.getEntity().getContent();
     }
+
+    /**
+     * Checks whether an HTTP resource exists. HEAD is attempted first; when HEAD does not return
+     * 200, a lightweight GET with {@code Range: bytes=0-0} is used to verify readability. This
+     * avoids treating signed or GET-only URLs as missing when HEAD is rejected or returns a
+     * different status than GET.
+     */
+    public static boolean exists(String uri) throws IOException {
+        int headStatusCode = headStatusCode(uri);
+        if (headStatusCode == HttpStatus.SC_OK) {
+            return true;
+        }
+        int rangeStatusCode = getRangeStatusCode(uri);
+        if (rangeStatusCode == HttpStatus.SC_OK

Review Comment:
   Thanks for catching this, @JingsongLi . 
   
   You're right — for a zero-length HTTP resource, Range: bytes=0-0 can legitimately return 416 even though a normal GET would succeed with an empty body, and treating that as an error breaks exists() for existing empty blobs.
   
   I've updated HttpClientUtils.exists() to treat HTTP 416 from the range probe as an existing resource, and added testExistsTreatsEmptyResourceAsExistingWhenRangeReturns416 (HEAD disabled + empty response + range GET returns 416). Please take another look when you have a chance.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Blob] Support blob-write-null-on-missing-file for HTTP URLs. [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #8219:
URL: https://github.com/apache/paimon/pull/8219#discussion_r3407205482


##########
paimon-format/src/main/java/org/apache/paimon/format/blob/BlobFormatWriter.java:
##########
@@ -108,6 +124,17 @@ public void addElement(InternalRow element) throws IOException {
                 write(tmpBuffer, bytesRead);
                 bytesRead = in.read(tmpBuffer);
             }
+        } catch (IOException | RuntimeException e) {
+            if (writeNullOnMissingFile && isNotFoundError(e)) {
+                LOG.warn(
+                        "Failed to read blob from {}, writing NULL for BLOB field {}.",
+                        blobUri(blob),
+                        blobFieldName,
+                        e);
+                writeNullElement();

Review Comment:
   The missing-file fallback can leave stale bytes in the blob file. `addElement` writes `MAGIC_NUMBER_BYTES` before opening/reading the referenced blob, then on a 404 it records the row as NULL and returns without rewinding/truncating `out`. A 404 from `newInputStream()` already leaves the magic bytes behind; if a later non-null blob is written to the same file, `BlobFileMeta` computes that later blob's offset as 0 because the NULL length does not advance offsets, while the actual bytes start after the orphan magic bytes. That will make subsequent blob descriptors/readback point at the wrong offset and corrupt data.
   
   Could we check existence before writing, or stage the blob into a temp buffer/file and only append to `out` after the read succeeds? It would also be good to add a test that writes a missing HTTP descriptor followed by a valid blob and reads the valid blob back.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Blob] Support blob-write-null-on-missing-file for HTTP URLs. [paimon]

Posted by "wwj6591812 (via GitHub)" <gi...@apache.org>.
wwj6591812 commented on PR #8219:
URL: https://github.com/apache/paimon/pull/8219#issuecomment-4757135950

   @JingsongLi 
   Hi,Please CC, Thx


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Blob] Support blob-write-null-on-missing-file for HTTP URLs. [paimon]

Posted by "wwj6591812 (via GitHub)" <gi...@apache.org>.
wwj6591812 commented on code in PR #8219:
URL: https://github.com/apache/paimon/pull/8219#discussion_r3410638645


##########
paimon-format/src/main/java/org/apache/paimon/format/blob/BlobFormatWriter.java:
##########
@@ -96,22 +112,35 @@ public void addElement(InternalRow element) throws IOException {
             return;
         }
 
-        long previousPos = out.getPos();
-        crc32.reset();
+        SeekableInputStream in;
+        try {
+            in = blob.newInputStream();
+        } catch (IOException | RuntimeException e) {
+            if (writeNullOnMissingFile && isNotFoundError(e)) {
+                LOG.warn(
+                        "Failed to open blob from {}, writing NULL for BLOB field {}.",
+                        blobUri(blob),
+                        blobFieldName,
+                        e);
+                writeNullElement();
+                return;
+            }
+            throw e;
+        }
 
         write(MAGIC_NUMBER_BYTES);

Review Comment:
   @JingsongLi 
   
   Good catch — you're right. crc32.reset() was accidentally dropped when I removed the in-memory staging path.
   
   I've restored crc32.reset() before writing MAGIC_NUMBER_BYTES for each non-null blob, so every per-record CRC is computed only over that record's bytes again (matching the old streaming behavior). I also added testTwoConsecutiveBlobsPreserveReadback to cover writing two consecutive blobs in the same file.
   
   Thanks again for the careful review!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Flink][Blob] Support blob-write-null-on-missing-file for HTTP URLs. [paimon]

Posted by "wwj6591812 (via GitHub)" <gi...@apache.org>.
wwj6591812 commented on PR #8219:
URL: https://github.com/apache/paimon/pull/8219#issuecomment-4828849816

   > Please carefully consider my question about forcefully judging in Writer. Choosing a static method may seem like a very perfunctory answer and solution. Please switch from AI to human state.
   
   Thanks for the feedback. @JingsongLi 
   
   I removed the Writer-side 404 detection, reverted all batch/core changes, and limited this PR to the existing Flink path (FlinkRowWrapper + UriReaderFactory.exists()), with HTTP support added at the URI layer only. Sorry for the earlier over-scoped diff.
   
   On master, blob-write-null-on-missing-file already applies only to Flink writes: FlinkSinkBuilder → FlinkRowWrapper.isNullAt() → UriReaderFactory.exists() for descriptor BLOB fields (local files only; HTTP was always treated as existing). Batch/core writes do not use this option. This PR keeps the same scope — Flink only — and just extends that existing exists() check to HTTP URIs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Blob] Support blob-write-null-on-missing-file for HTTP URLs. [paimon]

Posted by "wwj6591812 (via GitHub)" <gi...@apache.org>.
wwj6591812 commented on code in PR #8219:
URL: https://github.com/apache/paimon/pull/8219#discussion_r3457788787


##########
paimon-core/src/main/java/org/apache/paimon/append/ExternalStorageBlobWriter.java:
##########
@@ -125,20 +121,19 @@ public InternalRow transformRow(InternalRow row) throws IOException {
             return row;
         }
 
-        // clear all override positions so non-overridden fields fall back to delegate
-        for (ExternalStorageBlobFieldWriter fw : fieldWriters) {
-            overrideRow.setField(fw.fieldIndex(), null);
-        }
+        GenericRow result = (GenericRow) copyInternalRow(row, writeSchema);

Review Comment:
   @JingsongLi 
   
   Thanks for pointing this out — you're right that copyInternalRow() preserves the concrete row type, so casting its result to GenericRow is unsafe for BinaryRow / NestedRow inputs.
   
   I've fixed this by introducing InternalRowUtils.copyToGenericRow(), which always materializes the input into a mutable GenericRow before applying field overrides. ExternalStorageBlobWriter.transformRow() now uses this helper instead of casting copyInternalRow().
   
   I also added testCopyToGenericRowFromBinaryRow to cover the InternalRowSerializer.toBinaryRow(...) path and verify the result remains mutable via setField().
   
   Please take another look when you have a chance. Thanks again!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Blob] Support blob-write-null-on-missing-file for HTTP URLs. [paimon]

Posted by "wwj6591812 (via GitHub)" <gi...@apache.org>.
wwj6591812 commented on code in PR #8219:
URL: https://github.com/apache/paimon/pull/8219#discussion_r3417638753


##########
paimon-core/src/main/java/org/apache/paimon/append/ExternalStorageBlobWriter.java:
##########
@@ -201,12 +205,14 @@ private static ExternalStorageBlobFieldWriter createFieldWriter(
             FileSource fileSource,
             boolean asyncFileWrite,
             boolean statsDenseStore,
-            long targetFileSize) {
+            long targetFileSize,
+            boolean writeNullOnMissingFile) {
         int fieldIndex = writeSchema.getFieldIndex(fieldName);
         ExternalStorageBlobFieldWriter fieldWriter = new ExternalStorageBlobFieldWriter(fieldIndex);
 
         BlobFileFormat blobFileFormat = new BlobFileFormat();
         blobFileFormat.setWriteConsumer(fieldWriter);
+        blobFileFormat.setWriteNullOnMissingFile(writeNullOnMissingFile);

Review Comment:
   @JingsongLi 
   
   Thanks for catching this — you're absolutely right. When the external-storage copy hit the 404 fallback, BlobConsumer.accept(field, null) made writeAndReplace return null, and FallbackMappingRow treated that as "no override", so the inline descriptor field still kept the original HTTP URL instead of NULL.
   
   I've fixed this by copying the input row first and applying explicit field outcomes afterward: missing-file fallback now sets an explicit NULL on the descriptor field, while a successful copy still replaces it with the new external-storage descriptor. I also added testExternalStorageMissingHttpBlobWritesNull, which writes an external-storage descriptor pointing to a missing HTTP URL and verifies the stored field is NULL on readback.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Blob] Support blob-write-null-on-missing-file for HTTP URLs. [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #8219:
URL: https://github.com/apache/paimon/pull/8219#discussion_r3417540350


##########
paimon-api/src/main/java/org/apache/paimon/rest/HttpClientUtils.java:
##########
@@ -86,9 +88,65 @@ private static HttpClientConnectionManager configureConnectionManager() {
     public static InputStream getAsInputStream(String uri) throws IOException {
         HttpGet httpGet = new HttpGet(uri);
         CloseableHttpResponse response = DEFAULT_HTTP_CLIENT.execute(httpGet);
-        if (response.getCode() != 200) {
-            throw new RuntimeException("HTTP error code: " + response.getCode());
+        int statusCode = response.getCode();
+        if (statusCode != HttpStatus.SC_OK) {
+            throw httpError(statusCode);

Review Comment:
   This path leaks the HTTP response on non-200 statuses. `getAsInputStream` now turns a 404 into the signal that `BlobFormatWriter` catches for `blob-write-null-on-missing-file`, so repeated missing HTTP blobs can leave `CloseableHttpResponse`s open until the connection pool is exhausted. Could we close the response before throwing (for example with try-with-resources on the error branch, while still returning a stream that owns the response on the 200 branch)? A test that repeatedly opens missing URLs would catch this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Blob] Support blob-write-null-on-missing-file for HTTP URLs. [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #8219:
URL: https://github.com/apache/paimon/pull/8219#discussion_r3457550766


##########
paimon-core/src/main/java/org/apache/paimon/append/ExternalStorageBlobWriter.java:
##########
@@ -125,20 +121,19 @@ public InternalRow transformRow(InternalRow row) throws IOException {
             return row;
         }
 
-        // clear all override positions so non-overridden fields fall back to delegate
-        for (ExternalStorageBlobFieldWriter fw : fieldWriters) {
-            overrideRow.setField(fw.fieldIndex(), null);
-        }
+        GenericRow result = (GenericRow) copyInternalRow(row, writeSchema);

Review Comment:
   `copyInternalRow` is not guaranteed to return a `GenericRow`: for inputs such as `BinaryRow` (and `NestedRow`) it returns the same concrete row type copy. This writer accepts the generic `InternalRow` path, so an external-storage blob table write can reach here with a binary row (for example, `InternalRowSerializer.toBinaryRow(...)`) and then fails with `ClassCastException: BinaryRow cannot be cast to GenericRow` before writing. Please build a mutable `GenericRow` explicitly, or otherwise avoid assuming the copied row type before calling `setField`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Blob] Support blob-write-null-on-missing-file for HTTP URLs. [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #8219:
URL: https://github.com/apache/paimon/pull/8219#discussion_r3465146721


##########
paimon-format/src/main/java/org/apache/paimon/format/blob/BlobFormatWriter.java:
##########
@@ -127,6 +157,33 @@ public void addElement(InternalRow element) throws IOException {
         }
     }
 
+    private void writeNullElement() throws IOException {
+        lengths.add(NULL_LENGTH);
+        if (writeConsumer != null) {
+            writeConsumer.accept(blobFieldName, null);
+        }
+    }
+
+    private static String blobUri(Blob blob) {
+        if (blob instanceof BlobRef) {
+            return ((BlobRef) blob).toDescriptor().uri();
+        }
+        return "unknown";
+    }
+
+    private static boolean isNotFoundError(Throwable throwable) {
+        Throwable current = throwable;
+        while (current != null) {
+            if (current instanceof RuntimeException
+                    && current.getMessage() != null
+                    && current.getMessage().startsWith("HTTP error code: 404")) {

Review Comment:
   This looks quite hacky here. Is it possible not to modify it, or is it possible to throw a specific exception in HttpUriReader?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Blob] Support blob-write-null-on-missing-file for HTTP URLs. [paimon]

Posted by "wwj6591812 (via GitHub)" <gi...@apache.org>.
wwj6591812 commented on code in PR #8219:
URL: https://github.com/apache/paimon/pull/8219#discussion_r3407477095


##########
paimon-format/src/main/java/org/apache/paimon/format/blob/BlobFormatWriter.java:
##########
@@ -108,6 +124,17 @@ public void addElement(InternalRow element) throws IOException {
                 write(tmpBuffer, bytesRead);
                 bytesRead = in.read(tmpBuffer);
             }
+        } catch (IOException | RuntimeException e) {
+            if (writeNullOnMissingFile && isNotFoundError(e)) {
+                LOG.warn(
+                        "Failed to read blob from {}, writing NULL for BLOB field {}.",
+                        blobUri(blob),
+                        blobFieldName,
+                        e);
+                writeNullElement();

Review Comment:
   @JingsongLi 
   
   Thanks for catching this — you are absolutely right.
   
   The write-time 404 fallback previously wrote `MAGIC_NUMBER_BYTES` to `out` before opening the referenced blob. When the read failed, we recorded a NULL entry but left orphan magic bytes in the file. Because NULL entries do not advance blob offsets, a later non-null blob could get a descriptor offset of 0 while the actual payload started after those stale bytes, which would corrupt subsequent reads.
   
   I have fixed this by staging the blob payload (magic number + bytes) in memory first and only appending to `out` after the referenced blob is fully read successfully. If the read fails with a 404 and `blob-write-null-on-missing-file` is enabled, we now write NULL without touching `out` at all.
   
   I also added `testMissingHttpBlobFollowedByValidBlobPreservesReadback`, which writes a missing HTTP descriptor followed by a valid blob and verifies that the valid blob can be read back correctly via `BlobFileMeta` / `BlobFormatReader`.
   
   The primary pre-check path (`UriReaderFactory.exists()` for HTTP + `FlinkRowWrapper.isNullAt()`) remains unchanged and should prevent most missing URLs from reaching the write path; the staged write is now a safe fallback for the remaining cases.
   
   Please take another look when you have a chance. Thanks again!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Blob] Support blob-write-null-on-missing-file for HTTP URLs. [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #8219:
URL: https://github.com/apache/paimon/pull/8219#discussion_r3410550754


##########
paimon-format/src/main/java/org/apache/paimon/format/blob/BlobFormatWriter.java:
##########
@@ -96,22 +112,35 @@ public void addElement(InternalRow element) throws IOException {
             return;
         }
 
-        long previousPos = out.getPos();
-        crc32.reset();
+        SeekableInputStream in;
+        try {
+            in = blob.newInputStream();
+        } catch (IOException | RuntimeException e) {
+            if (writeNullOnMissingFile && isNotFoundError(e)) {
+                LOG.warn(
+                        "Failed to open blob from {}, writing NULL for BLOB field {}.",
+                        blobUri(blob),
+                        blobFieldName,
+                        e);
+                writeNullElement();
+                return;
+            }
+            throw e;
+        }
 
         write(MAGIC_NUMBER_BYTES);

Review Comment:
   Looks like `crc32.reset()` was dropped while removing the in-memory staging path. The CRC field is still written per blob record below, and the old streaming implementation reset the CRC before writing each record. Without that reset, the second and later non-null blobs will write a checksum that also includes bytes from previous blobs, so the blob file contains invalid per-record CRC values. Could we reset the CRC before writing `MAGIC_NUMBER_BYTES` for each non-null blob?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Blob] Support blob-write-null-on-missing-file for HTTP URLs. [paimon]

Posted by "wwj6591812 (via GitHub)" <gi...@apache.org>.
wwj6591812 commented on PR #8219:
URL: https://github.com/apache/paimon/pull/8219#issuecomment-4714110059

   @JingsongLi 
   Hi,Please CC, Thx


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Blob] Support blob-write-null-on-missing-file for HTTP URLs. [paimon]

Posted by "wwj6591812 (via GitHub)" <gi...@apache.org>.
wwj6591812 commented on code in PR #8219:
URL: https://github.com/apache/paimon/pull/8219#discussion_r3409252231


##########
paimon-format/src/main/java/org/apache/paimon/format/blob/BlobFormatWriter.java:
##########
@@ -97,21 +114,35 @@ public void addElement(InternalRow element) throws IOException {
         }
 
         long previousPos = out.getPos();
-        crc32.reset();
-
-        write(MAGIC_NUMBER_BYTES);
-
-        long blobPos = out.getPos();
+        ByteArrayOutputStream stagedPayload = new ByteArrayOutputStream();

Review Comment:
   @JingsongLi 
   
   Thanks for the follow-up — you're right that staging the full payload in memory is not acceptable for normal large blob writes.
   
   I've removed the ByteArrayOutputStream path and switched to the approach you suggested: open the source stream before appending anything to out. If opening fails with HTTP 404 and blob-write-null-on-missing-file is enabled, we record NULL without touching out; otherwise we keep the original streaming copy path (magic number + chunked read/write). This avoids the orphan-magic-byte issue for the common HTTP 404-at-open case without adding extra memory overhead for valid blobs.
   
   testMissingHttpBlobFollowedByValidBlobPreservesReadback still passes. Please take another look when you have a chance.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Blob] Support blob-write-null-on-missing-file for HTTP URLs. [paimon]

Posted by "wwj6591812 (via GitHub)" <gi...@apache.org>.
wwj6591812 commented on PR #8219:
URL: https://github.com/apache/paimon/pull/8219#issuecomment-4787774893

   @JingsongLi
   Hi,Please CC, Thx


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Flink][Blob] Support blob-write-null-on-missing-file for HTTP URLs. [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #8219:
URL: https://github.com/apache/paimon/pull/8219


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Blob] Support blob-write-null-on-missing-file for HTTP URLs. [paimon]

Posted by "wwj6591812 (via GitHub)" <gi...@apache.org>.
wwj6591812 commented on code in PR #8219:
URL: https://github.com/apache/paimon/pull/8219#discussion_r3417640044


##########
paimon-api/src/main/java/org/apache/paimon/rest/HttpClientUtils.java:
##########
@@ -86,9 +88,65 @@ private static HttpClientConnectionManager configureConnectionManager() {
     public static InputStream getAsInputStream(String uri) throws IOException {
         HttpGet httpGet = new HttpGet(uri);
         CloseableHttpResponse response = DEFAULT_HTTP_CLIENT.execute(httpGet);
-        if (response.getCode() != 200) {
-            throw new RuntimeException("HTTP error code: " + response.getCode());
+        int statusCode = response.getCode();
+        if (statusCode != HttpStatus.SC_OK) {
+            throw httpError(statusCode);

Review Comment:
   @JingsongLi 
   
   Good catch — getAsInputStream was throwing on non-200 responses without closing the CloseableHttpResponse, which could leak connections when many missing HTTP blobs are processed.
   
   I've updated the error branch to close the response in a finally block before throwing, while keeping the 200 path unchanged. I also added testGetAsInputStreamDoesNotLeakConnectionsOnRepeatedNotFound, which issues many consecutive 404 requests and then verifies a subsequent successful GET still works.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Blob] Support blob-write-null-on-missing-file for HTTP URLs. [paimon]

Posted by "wwj6591812 (via GitHub)" <gi...@apache.org>.
wwj6591812 commented on PR #8219:
URL: https://github.com/apache/paimon/pull/8219#issuecomment-4705005834

   @JingsongLi 
   Hi,Please CC, Thx


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Blob] Support blob-write-null-on-missing-file for HTTP URLs. [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #8219:
URL: https://github.com/apache/paimon/pull/8219#discussion_r3417538887


##########
paimon-core/src/main/java/org/apache/paimon/append/ExternalStorageBlobWriter.java:
##########
@@ -201,12 +205,14 @@ private static ExternalStorageBlobFieldWriter createFieldWriter(
             FileSource fileSource,
             boolean asyncFileWrite,
             boolean statsDenseStore,
-            long targetFileSize) {
+            long targetFileSize,
+            boolean writeNullOnMissingFile) {
         int fieldIndex = writeSchema.getFieldIndex(fieldName);
         ExternalStorageBlobFieldWriter fieldWriter = new ExternalStorageBlobFieldWriter(fieldIndex);
 
         BlobFileFormat blobFileFormat = new BlobFileFormat();
         blobFileFormat.setWriteConsumer(fieldWriter);
+        blobFileFormat.setWriteNullOnMissingFile(writeNullOnMissingFile);

Review Comment:
   This still does not make missing external-storage blobs become NULL in the data file. When `BlobFormatWriter` hits the 404 fallback it calls `BlobConsumer.accept(blobFieldName, null)`, so `writeAndReplace` returns `null`. But `transformRow` only sets `overrideRow` when the descriptor is non-null, and the final `FallbackMappingRow` treats a null override as "fall back to the original row". As a result, after the external copy writes a NULL entry, the normal writer still stores the original descriptor bytes in the inline descriptor field; reads will later point back to the missing HTTP URL instead of seeing NULL. We need to distinguish "no override" from "explicit NULL override" here, and add a test for an external-storage descriptor field where the source blob disappears/404s during the external copy.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Blob] Support blob-write-null-on-missing-file for HTTP URLs. [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #8219:
URL: https://github.com/apache/paimon/pull/8219#discussion_r3484963302


##########
docs/generated/core_configuration.html:
##########
@@ -903,6 +903,24 @@
             <td>Integer</td>
             <td>Level threshold of lookup to generate remote lookup files. Level files below this threshold will not generate remote lookup files.</td>
         </tr>
+        <tr>
+            <td><h5>manifest-sort.enabled</h5></td>

Review Comment:
   This generated-doc change looks unrelated to the HTTP BLOB missing-file handling. The PR only changes the blob option text in `CoreOptions`, but this file also moves the `manifest-sort.*` rows, which adds noise and can create unnecessary conflicts with other manifest-sort work. Could you regenerate/drop this unrelated reordering so the diff only contains the blob option description update?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Blob] Support blob-write-null-on-missing-file for HTTP URLs. [paimon]

Posted by "wwj6591812 (via GitHub)" <gi...@apache.org>.
wwj6591812 commented on code in PR #8219:
URL: https://github.com/apache/paimon/pull/8219#discussion_r3465981368


##########
paimon-format/src/main/java/org/apache/paimon/format/blob/BlobFormatWriter.java:
##########
@@ -127,6 +157,33 @@ public void addElement(InternalRow element) throws IOException {
         }
     }
 
+    private void writeNullElement() throws IOException {
+        lengths.add(NULL_LENGTH);
+        if (writeConsumer != null) {
+            writeConsumer.accept(blobFieldName, null);
+        }
+    }
+
+    private static String blobUri(Blob blob) {
+        if (blob instanceof BlobRef) {
+            return ((BlobRef) blob).toDescriptor().uri();
+        }
+        return "unknown";
+    }
+
+    private static boolean isNotFoundError(Throwable throwable) {
+        Throwable current = throwable;
+        while (current != null) {
+            if (current instanceof RuntimeException
+                    && current.getMessage() != null
+                    && current.getMessage().startsWith("HTTP error code: 404")) {

Review Comment:
   Thanks for the review @JingsongLi. 
   
   Agreed — matching the 404 error by message string in BlobFormatWriter was too hacky.
   
   I've introduced a dedicated HttpNotFoundException in HttpClientUtils (thrown when getAsInputStream() receives HTTP 404, and propagated through HttpUriReader). HttpClientUtils.isNotFoundError() now checks the exception type in the cause chain instead of parsing messages. BlobFormatWriter reuses that helper and no longer contains its own string-matching logic.
   
   Please take another look when you have time.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Blob] Support blob-write-null-on-missing-file for HTTP URLs. [paimon]

Posted by "wwj6591812 (via GitHub)" <gi...@apache.org>.
wwj6591812 commented on code in PR #8219:
URL: https://github.com/apache/paimon/pull/8219#discussion_r3488768424


##########
docs/generated/core_configuration.html:
##########
@@ -903,6 +903,24 @@
             <td>Integer</td>
             <td>Level threshold of lookup to generate remote lookup files. Level files below this threshold will not generate remote lookup files.</td>
         </tr>
+        <tr>
+            <td><h5>manifest-sort.enabled</h5></td>

Review Comment:
   Thanks @JingsongLi . 
   
   You're right — the generated doc diff accidentally reordered the manifest-sort.* rows. I've restored the original ordering in core_configuration.html so this PR only updates the blob-write-null-on-missing-file description.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Blob] Support blob-write-null-on-missing-file for HTTP URLs. [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #8219:
URL: https://github.com/apache/paimon/pull/8219#discussion_r3460468185


##########
paimon-flink/paimon-flink-common/src/test/java/org/apache/paimon/flink/FlinkRowWrapperTest.java:
##########
@@ -59,6 +83,36 @@ public void testExistingBlobDescriptorIsReadableWhenCheckingEnabled() throws Exc
         assertThat(wrapper.getBlob(0).toData()).isEqualTo(bytes);
     }
 
+    @Test
+    public void testMissingHttpBlobDescriptorIsNullWhenCheckingEnabled() throws Exception {

Review Comment:
   These new tests still only exercise a single-column descriptor row, but the enabled path is used on full Flink input rows. With `blob-write-null-on-missing-file=true`, `FlinkRowWrapper.isNullAt` calls `row.getBinary(pos)` for every non-null field; a table like `(id INT, b BLOB)` will therefore probe `id` with `getBinary(0)` and can fail before the blob field is written. Please add a regression test with at least one non-BLOB column before the descriptor column and fix the wrapper to check only known descriptor/BLOB field indexes.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Blob] Support blob-write-null-on-missing-file for HTTP URLs. [paimon]

Posted by "wwj6591812 (via GitHub)" <gi...@apache.org>.
wwj6591812 commented on code in PR #8219:
URL: https://github.com/apache/paimon/pull/8219#discussion_r3409252578


##########
paimon-api/src/main/java/org/apache/paimon/rest/HttpClientUtils.java:
##########
@@ -86,9 +88,61 @@ private static HttpClientConnectionManager configureConnectionManager() {
     public static InputStream getAsInputStream(String uri) throws IOException {
         HttpGet httpGet = new HttpGet(uri);
         CloseableHttpResponse response = DEFAULT_HTTP_CLIENT.execute(httpGet);
-        if (response.getCode() != 200) {
-            throw new RuntimeException("HTTP error code: " + response.getCode());
+        int statusCode = response.getCode();
+        if (statusCode != HttpStatus.SC_OK) {
+            throw httpError(statusCode);
         }
         return response.getEntity().getContent();
     }
+
+    /**
+     * Checks whether an HTTP resource exists. HEAD is attempted first; when the server does not
+     * support HEAD, a lightweight GET with {@code Range: bytes=0-0} is used as fallback.
+     */
+    public static boolean exists(String uri) throws IOException {
+        int statusCode = headStatusCode(uri);

Review Comment:
   @JingsongLi 
   
   Good point — making HEAD authoritative is unsafe for signed or GET-only URLs.
   
   I've updated exists() so that only a successful HEAD (200) returns true immediately. For any non-200 HEAD status (including 403/404, as well as 405/501), we now fall back to a lightweight Range: bytes=0-0 GET and only return false when that GET also returns 404. Added tests for HEAD 404/403 with a readable range GET, and for HEAD 404 + GET 404.
   
   Thanks again for the careful review!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Blob] Support blob-write-null-on-missing-file for HTTP URLs. [paimon]

Posted by "wwj6591812 (via GitHub)" <gi...@apache.org>.
wwj6591812 commented on PR #8219:
URL: https://github.com/apache/paimon/pull/8219#issuecomment-4702094248

   @JingsongLi
   Hi,Please CC, Thx


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Flink][Blob] Support blob-write-null-on-missing-file for HTTP URLs. [paimon]

Posted by "wwj6591812 (via GitHub)" <gi...@apache.org>.
wwj6591812 commented on PR #8219:
URL: https://github.com/apache/paimon/pull/8219#issuecomment-4829681076

   @JingsongLi
   Hi,Please CC, Thx


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Blob] Support blob-write-null-on-missing-file for HTTP URLs. [paimon]

Posted by "wwj6591812 (via GitHub)" <gi...@apache.org>.
wwj6591812 commented on code in PR #8219:
URL: https://github.com/apache/paimon/pull/8219#discussion_r3463983092


##########
paimon-flink/paimon-flink-common/src/test/java/org/apache/paimon/flink/FlinkRowWrapperTest.java:
##########
@@ -59,6 +83,36 @@ public void testExistingBlobDescriptorIsReadableWhenCheckingEnabled() throws Exc
         assertThat(wrapper.getBlob(0).toData()).isEqualTo(bytes);
     }
 
+    @Test
+    public void testMissingHttpBlobDescriptorIsNullWhenCheckingEnabled() throws Exception {

Review Comment:
   Thanks for pointing this out. You're right — with blob-write-null-on-missing-file=true, FlinkRowWrapper.isNullAt() was calling row.getBinary(pos) on every non-null field, so a row like (id INT, b BLOB) could fail on getBinary(0) before reaching the blob column.
   
   I've fixed this by tracking known BLOB field indexes (derived from the row schema in FlinkSinkBuilder.mapToInternalRow()) and only running the descriptor existence check on those positions. I also added testMissingHttpBlobDescriptorWithNonBlobColumnBefore as a regression test with a non-BLOB column before the descriptor column. Please take another look when you have a chance.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Blob] Support blob-write-null-on-missing-file for HTTP URLs. [paimon]

Posted by "wwj6591812 (via GitHub)" <gi...@apache.org>.
wwj6591812 commented on PR #8219:
URL: https://github.com/apache/paimon/pull/8219#issuecomment-4785054480

   @JingsongLi 
   Hi,Please CC, Thx
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org