You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "vinothchandar (via GitHub)" <gi...@apache.org> on 2026/07/04 04:54:45 UTC
[PR] fix(spark): read Lance BLOB columns in <=512-row chunks to avoid lance-core FFI abort [hudi]
vinothchandar opened a new pull request, #19181:
URL: https://github.com/apache/hudi/pull/19181
### Describe the issue this Pull Request addresses
lance-core 4.0.0 aborts the JVM in its Arrow C-stream export (arrow_array::ffi_stream::get_next: "range end index N out of range for slice of length 0") whenever a single readAll stream crosses Lance's internal BLOB page boundary (512 rows); the requested batchSize does not help because Lance re-chunks BLOB columns at 512 internally. As a result CoW Lance tables with BLOB columns could not be read past 512 rows: OUT_OF_LINE reads threw an Arrow "should have as many children as in the schema" error and INLINE CONTENT reads crashed the JVM (SIGABRT).
Read BLOB-containing Lance files in <=512-row row-range chunks, issuing a fresh readAll per chunk so each FFI stream stays within a single BLOB page and the buggy second-page export is never reached. LanceRecordIterator gains a chunkedBlobReader(...) factory driven by an ArrowReaderSupplier; SparkLanceReaderBase (SQL/DataFrame read) and HoodieSparkLanceReader (internal CONTENT path) route to it when the projection contains a BLOB field. Non-BLOB reads keep the single streamed reader and are unchanged; the per-row hot path is untouched, with only small fixed per-chunk allocations.
Also document hoodie.read.blob.inline.mode option placement (read option for both DataFrame and SQL) and add batch-scale regression tests: INLINE CONTENT, VECTOR + OUT_OF_LINE BLOB at n=100/1000 (crossing the 512-row boundary), with vector top-k, projection/filter, and read_blob byte + SHA-256 checks.
### Summary and Changelog
lance reads work reliably without crashes for base files with larger number of records.
### Impact
none
### Risk Level
none
### Documentation Update
none
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Enough context is provided in the sections above
- [ ] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): read Lance BLOB columns in <=512-row chunks to avoid lance-core FFI abort [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on PR #19181:
URL: https://github.com/apache/hudi/pull/19181#issuecomment-4897548358
@vinothchandar
> @rahil-c @voonhous does this ring a bell. have you all seen this in your testing
The closest thing I could find was the following github issue: https://github.com/apache/hudi/pull/18613. Will take a look at this pr sometime.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): read Lance BLOB columns in <=512-row chunks to avoid lance-core FFI abort [hudi]
Posted by "hudi-agent (via GitHub)" <gi...@apache.org>.
hudi-agent commented on code in PR #19181:
URL: https://github.com/apache/hudi/pull/19181#discussion_r3522623363
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/LanceRecordIterator.java:
##########
@@ -32,39 +32,54 @@
import org.apache.spark.sql.types.StructType;
import org.apache.spark.sql.vectorized.ColumnVector;
import org.apache.spark.sql.vectorized.ColumnarBatch;
+import org.lance.file.FileReadOptions;
import org.lance.file.LanceFileReader;
import org.lance.spark.vectorized.LanceArrowColumnVector;
+import org.lance.util.Range;
import java.io.IOException;
+import java.util.Collections;
import java.util.HashMap;
import java.util.Iterator;
import java.util.List;
import java.util.Map;
/**
- * Iterator for reading Lance files and converting Arrow batches to Spark {@link UnsafeRow}s.
- * Used by both Hudi's internal Lance reader and Spark datasource integration.
+ * Iterator over a Lance file that converts Arrow batches to Spark {@link UnsafeRow}s, owning the
+ * {@link BufferAllocator}, {@link LanceFileReader}, {@link ArrowReader}(s) and current
+ * {@link ColumnarBatch}. Used by both Hudi's internal Lance reader and the Spark datasource. An
+ * optional {@link BlobDescriptorTransform} rewrites BLOB columns for DESCRIPTOR-mode reads.
*
- * <p>The iterator manages the lifecycle of:
- * <ul>
- * <li>BufferAllocator - Arrow memory management</li>
- * <li>LanceFileReader - Lance file handle</li>
- * <li>ArrowReader - Arrow batch reader</li>
- * <li>ColumnarBatch - Current batch being iterated</li>
- * </ul>
- *
- * <p>An optional {@link BlobDescriptorTransform} can be composed in to rewrite BLOB columns
- * in DESCRIPTOR mode.
+ * <p>BLOB chunked reading: lance-core 4.0.0 aborts the JVM in its Arrow C-stream export whenever a
+ * single {@code readAll} stream crosses Lance's internal BLOB page boundary (512 rows), and the
+ * requested {@code batchSize} does not change that. So BLOB reads issue one {@code readAll} per
+ * row-range chunk of {@link #BLOB_READ_CHUNK_ROWS} rows; non-BLOB reads use one streamed reader.
*/
public final class LanceRecordIterator implements ClosableIterator<UnsafeRow> {
+
+ /** Rows per {@code readAll} for BLOB reads. Must not exceed Lance's 512-row BLOB page size. */
+ public static final int BLOB_READ_CHUNK_ROWS = 512;
+
+ /**
+ * Supplies the sequence of {@link ArrowReader}s to drain, one after another. Single-reader
+ * mode yields one reader; BLOB chunked mode yields one fresh reader per row-range chunk. Each
+ * returned reader is owned (and closed) by {@link LanceRecordIterator}.
+ */
Review Comment:
π€ nit: `ArrowReaderSupplier` borrows the `Supplier<T>` name from `java.util.function`, which conventionally has a `get()` method and is usable as a lambda β this interface throws `IOException` and uses `next()`, so it doesn't fit that mental model. Something like `ArrowReaderSequence` or `ReaderChain` might avoid the false association for future readers of this class.
<sub><i>β οΈ AI-generated; verify before applying. React π/π to flag quality.</i></sub>
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/execution/datasources/lance/SparkLanceReaderBase.scala:
##########
@@ -148,8 +147,18 @@ class SparkLanceReaderBase(enableVectorizedReader: Boolean) extends SparkColumna
} else {
null
}
- lanceIterator = new LanceRecordIterator(
- allocator, lanceReader, arrowReader, iteratorSchema, filePath, blobTransform)
+ // lance-core 4.0.0 aborts the JVM when a single readAll stream crosses Lance's internal
+ // BLOB page boundary (512 rows). For BLOB-containing reads, drain the file in <=512-row
+ // range chunks (one fresh readAll each); non-BLOB reads keep the single streamed reader.
+ lanceIterator = if (!blobFieldNames.isEmpty) {
Review Comment:
π€ nit: `blobFieldNames.nonEmpty` is the idiomatic Scala spelling β reads as a positive predicate without the leading negation to parse.
<sub><i>β οΈ AI-generated; verify before applying. React π/π to flag quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): read Lance BLOB columns in <=512-row chunks to avoid lance-core FFI abort [hudi]
Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
vinothchandar commented on PR #19181:
URL: https://github.com/apache/hudi/pull/19181#issuecomment-4880732222
@yihua could you shepherd this? I discovered the issue by replicating some lance tests using pySpark/Hudi/Lance. Locally validated the FFI crash is gone.
I suggest looking at the versions for lance and see if a later version resolves the issue (separate PR)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): read Lance BLOB columns in <=512-row chunks to avoid lance-core FFI abort [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19181:
URL: https://github.com/apache/hudi/pull/19181#issuecomment-4923059726
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "63b524f2aa25df0976f440385e32d52f25796173",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15206",
"triggerID" : "63b524f2aa25df0976f440385e32d52f25796173",
"triggerType" : "PUSH"
}, {
"hash" : "42a7e0d3c92f39c83b3b4a4ba0ab808102a8dfb7",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15405",
"triggerID" : "42a7e0d3c92f39c83b3b4a4ba0ab808102a8dfb7",
"triggerType" : "PUSH"
}, {
"hash" : "f3c076b3293f362371f8b469bf18af5b827ad416",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f3c076b3293f362371f8b469bf18af5b827ad416",
"triggerType" : "PUSH"
}, {
"hash" : "46409202e7e679c9e31af9b98fea6cd1c6ddffd6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15406",
"triggerID" : "46409202e7e679c9e31af9b98fea6cd1c6ddffd6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 42a7e0d3c92f39c83b3b4a4ba0ab808102a8dfb7 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15405)
* f3c076b3293f362371f8b469bf18af5b827ad416 UNKNOWN
* 46409202e7e679c9e31af9b98fea6cd1c6ddffd6 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15406)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): read Lance BLOB columns in <=512-row chunks to avoid lance-core FFI abort [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19181:
URL: https://github.com/apache/hudi/pull/19181#issuecomment-4923648061
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "63b524f2aa25df0976f440385e32d52f25796173",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15206",
"triggerID" : "63b524f2aa25df0976f440385e32d52f25796173",
"triggerType" : "PUSH"
}, {
"hash" : "42a7e0d3c92f39c83b3b4a4ba0ab808102a8dfb7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15405",
"triggerID" : "42a7e0d3c92f39c83b3b4a4ba0ab808102a8dfb7",
"triggerType" : "PUSH"
}, {
"hash" : "f3c076b3293f362371f8b469bf18af5b827ad416",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f3c076b3293f362371f8b469bf18af5b827ad416",
"triggerType" : "PUSH"
}, {
"hash" : "46409202e7e679c9e31af9b98fea6cd1c6ddffd6",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15406",
"triggerID" : "46409202e7e679c9e31af9b98fea6cd1c6ddffd6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f3c076b3293f362371f8b469bf18af5b827ad416 UNKNOWN
* 46409202e7e679c9e31af9b98fea6cd1c6ddffd6 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15406)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): read Lance BLOB columns in <=512-row chunks to avoid lance-core FFI abort [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19181:
URL: https://github.com/apache/hudi/pull/19181#issuecomment-4922432718
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "63b524f2aa25df0976f440385e32d52f25796173",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15206",
"triggerID" : "63b524f2aa25df0976f440385e32d52f25796173",
"triggerType" : "PUSH"
}, {
"hash" : "42a7e0d3c92f39c83b3b4a4ba0ab808102a8dfb7",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15405",
"triggerID" : "42a7e0d3c92f39c83b3b4a4ba0ab808102a8dfb7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 63b524f2aa25df0976f440385e32d52f25796173 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15206)
* 42a7e0d3c92f39c83b3b4a4ba0ab808102a8dfb7 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15405)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): read Lance BLOB columns in <=512-row chunks to avoid lance-core FFI abort [hudi]
Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on PR #19181:
URL: https://github.com/apache/hudi/pull/19181#issuecomment-4912902316
The above bug highlighted is per-basefile, not per query, IIUC, it only triggers if:
1. A single .lance base file holds >512 rows, and
2. Β the read projects the BLOB column, and
3. one readAll FFI stream drains past row 512 (crossing Lance's internal BLOB page boundary).
So, total rows ingested is irrelevant. What matters is rows-per-base-file.
I think the demos spread >500 rows across multiple base files. No single file's readAll ever crossed the page boundary, which is why it never triggered the bug above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): read Lance BLOB columns in <=512-row chunks to avoid lance-core FFI abort [hudi]
Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #19181:
URL: https://github.com/apache/hudi/pull/19181#discussion_r3542673414
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/LanceRecordIterator.java:
##########
@@ -32,39 +32,54 @@
import org.apache.spark.sql.types.StructType;
import org.apache.spark.sql.vectorized.ColumnVector;
import org.apache.spark.sql.vectorized.ColumnarBatch;
+import org.lance.file.FileReadOptions;
import org.lance.file.LanceFileReader;
import org.lance.spark.vectorized.LanceArrowColumnVector;
+import org.lance.util.Range;
import java.io.IOException;
+import java.util.Collections;
import java.util.HashMap;
import java.util.Iterator;
import java.util.List;
import java.util.Map;
/**
- * Iterator for reading Lance files and converting Arrow batches to Spark {@link UnsafeRow}s.
- * Used by both Hudi's internal Lance reader and Spark datasource integration.
+ * Iterator over a Lance file that converts Arrow batches to Spark {@link UnsafeRow}s, owning the
+ * {@link BufferAllocator}, {@link LanceFileReader}, {@link ArrowReader}(s) and current
+ * {@link ColumnarBatch}. Used by both Hudi's internal Lance reader and the Spark datasource. An
+ * optional {@link BlobDescriptorTransform} rewrites BLOB columns for DESCRIPTOR-mode reads.
*
- * <p>The iterator manages the lifecycle of:
- * <ul>
- * <li>BufferAllocator - Arrow memory management</li>
- * <li>LanceFileReader - Lance file handle</li>
- * <li>ArrowReader - Arrow batch reader</li>
- * <li>ColumnarBatch - Current batch being iterated</li>
- * </ul>
- *
- * <p>An optional {@link BlobDescriptorTransform} can be composed in to rewrite BLOB columns
- * in DESCRIPTOR mode.
+ * <p>BLOB chunked reading: lance-core 4.0.0 aborts the JVM in its Arrow C-stream export whenever a
+ * single {@code readAll} stream crosses Lance's internal BLOB page boundary (512 rows), and the
+ * requested {@code batchSize} does not change that. So BLOB reads issue one {@code readAll} per
+ * row-range chunk of {@link #BLOB_READ_CHUNK_ROWS} rows; non-BLOB reads use one streamed reader.
*/
public final class LanceRecordIterator implements ClosableIterator<UnsafeRow> {
+
+ /** Rows per {@code readAll} for BLOB reads. Must not exceed Lance's 512-row BLOB page size. */
+ public static final int BLOB_READ_CHUNK_ROWS = 512;
Review Comment:
Two silent couplings that would each bring the JVM abort back with no test signal - non-blocking, but "silent" is what made this one expensive, so cheap loud guards feel worth it:
**1. This constant hard-codes lance-core 4.0.0's internal BLOB page size.** A lance bump that changes it re-introduces straddling chunks silently - nothing fails unless a >512-row fixture happens to catch it. Worth deriving/asserting it against lance, or at least pinning a comment to the lance version so the next upgrader revalidates.
**2. Blob detection (`isBlobField()` over top-level fields) is top-level-only.** Correct today since the writer only supports top-level BLOB columns, but the schema layer already models nested blobs (`nestedNullableBlob` / `mapWithNestedBlob` / `arrayWithNestedBlob` in `TestHoodieSchema`, plus the nested-record case in `TestHoodieSchemaCompatibility`). If the writer ever gains nested-blob support, a schema like:
```sql
CREATE TABLE docs (
id INT,
document STRUCT<source_url: STRING, content_type: STRING, raw_bytes: BLOB>
);
```
silently skips chunking on read and aborts again once a base file crosses 512 rows: the top-level field `document` is a STRUCT, not a blob / array-of-blob / map-of-blob, so `isBlobField()` returns false and the non-chunked path is taken. `HoodieSchema.containsBlobType()` already does the recursive check as a drop-in, or add an assertion that a projected blob isn't nested so a future writer change fails loud instead of aborting the JVM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): read Lance BLOB columns in <=512-row chunks to avoid lance-core FFI abort [hudi]
Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #19181:
URL: https://github.com/apache/hudi/pull/19181#discussion_r3549673096
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/HoodieSparkLanceReader.java:
##########
@@ -208,8 +208,17 @@ private ClosableIterator<UnsafeRow> getUnsafeRowIterator(HoodieSchema requestedS
// Pinned to CONTENT: compaction/merge/log-replay need actual bytes to rewrite.
// The user-facing `hoodie.read.blob.inline.mode` is honored by SparkLanceReaderBase.
FileReadOptions readOpts = FileReadOptions.builder().blobReadMode(BlobReadMode.CONTENT).build();
- ArrowReader arrowReader = lanceReader.readAll(columnNames, null, DEFAULT_BATCH_SIZE, readOpts);
+ // BLOB reads must be chunked to dodge a lance-core FFI abort (see LanceRecordIterator).
+ // Unwrap a nullable UNION wrapper before checking whether the projection has a BLOB field.
+ HoodieSchema recordSchema = requestedSchema.getNonNullType();
+ boolean hasBlob = recordSchema.hasFields()
+ && recordSchema.getFields().stream().anyMatch(f -> f.schema().isBlobField());
+ if (hasBlob) {
+ return LanceRecordIterator.chunkedBlobReader(allocator, lanceReader, columnNames, readOpts,
Review Comment:
tried adding this and it surfaced a pre-existing bug: compaction nulls inline blob bytes of untouched rows in the rewritten file group (repros on master without this PR, even at 6 rows, once the base file holds untouched rows -- our existing test masks it because untouched rows never sat in a compacted file group). filing a separate JIRA with the repro; the >512 compaction test belongs with that fix. not blocking this PR on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): read Lance BLOB columns in <=512-row chunks to avoid lance-core FFI abort [hudi]
Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #19181:
URL: https://github.com/apache/hudi/pull/19181#discussion_r3549713634
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/LanceRecordIterator.java:
##########
@@ -32,39 +32,54 @@
import org.apache.spark.sql.types.StructType;
import org.apache.spark.sql.vectorized.ColumnVector;
import org.apache.spark.sql.vectorized.ColumnarBatch;
+import org.lance.file.FileReadOptions;
import org.lance.file.LanceFileReader;
import org.lance.spark.vectorized.LanceArrowColumnVector;
+import org.lance.util.Range;
import java.io.IOException;
+import java.util.Collections;
import java.util.HashMap;
import java.util.Iterator;
import java.util.List;
import java.util.Map;
/**
- * Iterator for reading Lance files and converting Arrow batches to Spark {@link UnsafeRow}s.
- * Used by both Hudi's internal Lance reader and Spark datasource integration.
+ * Iterator over a Lance file that converts Arrow batches to Spark {@link UnsafeRow}s, owning the
+ * {@link BufferAllocator}, {@link LanceFileReader}, {@link ArrowReader}(s) and current
+ * {@link ColumnarBatch}. Used by both Hudi's internal Lance reader and the Spark datasource. An
+ * optional {@link BlobDescriptorTransform} rewrites BLOB columns for DESCRIPTOR-mode reads.
*
- * <p>The iterator manages the lifecycle of:
- * <ul>
- * <li>BufferAllocator - Arrow memory management</li>
- * <li>LanceFileReader - Lance file handle</li>
- * <li>ArrowReader - Arrow batch reader</li>
- * <li>ColumnarBatch - Current batch being iterated</li>
- * </ul>
- *
- * <p>An optional {@link BlobDescriptorTransform} can be composed in to rewrite BLOB columns
- * in DESCRIPTOR mode.
+ * <p>BLOB chunked reading: lance-core 4.0.0 aborts the JVM in its Arrow C-stream export whenever a
+ * single {@code readAll} stream crosses Lance's internal BLOB page boundary (512 rows), and the
+ * requested {@code batchSize} does not change that. So BLOB reads issue one {@code readAll} per
+ * row-range chunk of {@link #BLOB_READ_CHUNK_ROWS} rows; non-BLOB reads use one streamed reader.
*/
public final class LanceRecordIterator implements ClosableIterator<UnsafeRow> {
+
+ /** Rows per {@code readAll} for BLOB reads. Must not exceed Lance's 512-row BLOB page size. */
+ public static final int BLOB_READ_CHUNK_ROWS = 512;
Review Comment:
both addressed: blob detection now recurses (made containsBlobType() public and used it in HoodieSparkLanceReader, recursive walk on the datasource side), so a nested blob would still route through the chunked path. for the 512: lance exposes no API for its blob page size, so pinned the constant javadoc to lance-core 4.0.0 with a note to revalidate against the batch-scale tests on any lance bump.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): read Lance BLOB columns in <=512-row chunks to avoid lance-core FFI abort [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19181:
URL: https://github.com/apache/hudi/pull/19181#issuecomment-4880754432
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "63b524f2aa25df0976f440385e32d52f25796173",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15206",
"triggerID" : "63b524f2aa25df0976f440385e32d52f25796173",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 63b524f2aa25df0976f440385e32d52f25796173 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15206)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): read Lance BLOB columns in <=512-row chunks to avoid lance-core FFI abort [hudi]
Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #19181:
URL: https://github.com/apache/hudi/pull/19181#discussion_r3549673096
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/HoodieSparkLanceReader.java:
##########
@@ -208,8 +208,17 @@ private ClosableIterator<UnsafeRow> getUnsafeRowIterator(HoodieSchema requestedS
// Pinned to CONTENT: compaction/merge/log-replay need actual bytes to rewrite.
// The user-facing `hoodie.read.blob.inline.mode` is honored by SparkLanceReaderBase.
FileReadOptions readOpts = FileReadOptions.builder().blobReadMode(BlobReadMode.CONTENT).build();
- ArrowReader arrowReader = lanceReader.readAll(columnNames, null, DEFAULT_BATCH_SIZE, readOpts);
+ // BLOB reads must be chunked to dodge a lance-core FFI abort (see LanceRecordIterator).
+ // Unwrap a nullable UNION wrapper before checking whether the projection has a BLOB field.
+ HoodieSchema recordSchema = requestedSchema.getNonNullType();
+ boolean hasBlob = recordSchema.hasFields()
+ && recordSchema.getFields().stream().anyMatch(f -> f.schema().isBlobField());
+ if (hasBlob) {
+ return LanceRecordIterator.chunkedBlobReader(allocator, lanceReader, columnNames, readOpts,
Review Comment:
tried adding this and it surfaced a pre-existing bug: compaction nulls inline blob bytes of untouched rows in the rewritten file group (repros on master without this PR, even at 6 rows, once the base file holds untouched rows -- our existing test masks it because untouched rows never sat in a compacted file group). filed #19232 with the repro; the >512 compaction test belongs with that fix. not blocking this PR on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): read Lance BLOB columns in <=512-row chunks to avoid lance-core FFI abort [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19181:
URL: https://github.com/apache/hudi/pull/19181#issuecomment-4922789985
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "63b524f2aa25df0976f440385e32d52f25796173",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15206",
"triggerID" : "63b524f2aa25df0976f440385e32d52f25796173",
"triggerType" : "PUSH"
}, {
"hash" : "42a7e0d3c92f39c83b3b4a4ba0ab808102a8dfb7",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15405",
"triggerID" : "42a7e0d3c92f39c83b3b4a4ba0ab808102a8dfb7",
"triggerType" : "PUSH"
}, {
"hash" : "f3c076b3293f362371f8b469bf18af5b827ad416",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f3c076b3293f362371f8b469bf18af5b827ad416",
"triggerType" : "PUSH"
}, {
"hash" : "46409202e7e679c9e31af9b98fea6cd1c6ddffd6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "46409202e7e679c9e31af9b98fea6cd1c6ddffd6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 63b524f2aa25df0976f440385e32d52f25796173 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15206)
* 42a7e0d3c92f39c83b3b4a4ba0ab808102a8dfb7 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15405)
* f3c076b3293f362371f8b469bf18af5b827ad416 UNKNOWN
* 46409202e7e679c9e31af9b98fea6cd1c6ddffd6 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): read Lance BLOB columns in <=512-row chunks to avoid lance-core FFI abort [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19181:
URL: https://github.com/apache/hudi/pull/19181#issuecomment-4922401948
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "63b524f2aa25df0976f440385e32d52f25796173",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15206",
"triggerID" : "63b524f2aa25df0976f440385e32d52f25796173",
"triggerType" : "PUSH"
}, {
"hash" : "42a7e0d3c92f39c83b3b4a4ba0ab808102a8dfb7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "42a7e0d3c92f39c83b3b4a4ba0ab808102a8dfb7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 63b524f2aa25df0976f440385e32d52f25796173 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15206)
* 42a7e0d3c92f39c83b3b4a4ba0ab808102a8dfb7 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): read Lance BLOB columns in <=512-row chunks to avoid lance-core FFI abort [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19181:
URL: https://github.com/apache/hudi/pull/19181#issuecomment-4922753853
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "63b524f2aa25df0976f440385e32d52f25796173",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15206",
"triggerID" : "63b524f2aa25df0976f440385e32d52f25796173",
"triggerType" : "PUSH"
}, {
"hash" : "42a7e0d3c92f39c83b3b4a4ba0ab808102a8dfb7",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15405",
"triggerID" : "42a7e0d3c92f39c83b3b4a4ba0ab808102a8dfb7",
"triggerType" : "PUSH"
}, {
"hash" : "f3c076b3293f362371f8b469bf18af5b827ad416",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f3c076b3293f362371f8b469bf18af5b827ad416",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 63b524f2aa25df0976f440385e32d52f25796173 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15206)
* 42a7e0d3c92f39c83b3b4a4ba0ab808102a8dfb7 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15405)
* f3c076b3293f362371f8b469bf18af5b827ad416 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): read Lance BLOB columns in <=512-row chunks to avoid lance-core FFI abort [hudi]
Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #19181:
URL: https://github.com/apache/hudi/pull/19181#discussion_r3549337051
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/LanceRecordIterator.java:
##########
@@ -32,39 +32,54 @@
import org.apache.spark.sql.types.StructType;
import org.apache.spark.sql.vectorized.ColumnVector;
import org.apache.spark.sql.vectorized.ColumnarBatch;
+import org.lance.file.FileReadOptions;
import org.lance.file.LanceFileReader;
import org.lance.spark.vectorized.LanceArrowColumnVector;
+import org.lance.util.Range;
import java.io.IOException;
+import java.util.Collections;
import java.util.HashMap;
import java.util.Iterator;
import java.util.List;
import java.util.Map;
/**
- * Iterator for reading Lance files and converting Arrow batches to Spark {@link UnsafeRow}s.
- * Used by both Hudi's internal Lance reader and Spark datasource integration.
+ * Iterator over a Lance file that converts Arrow batches to Spark {@link UnsafeRow}s, owning the
+ * {@link BufferAllocator}, {@link LanceFileReader}, {@link ArrowReader}(s) and current
+ * {@link ColumnarBatch}. Used by both Hudi's internal Lance reader and the Spark datasource. An
+ * optional {@link BlobDescriptorTransform} rewrites BLOB columns for DESCRIPTOR-mode reads.
*
- * <p>The iterator manages the lifecycle of:
- * <ul>
- * <li>BufferAllocator - Arrow memory management</li>
- * <li>LanceFileReader - Lance file handle</li>
- * <li>ArrowReader - Arrow batch reader</li>
- * <li>ColumnarBatch - Current batch being iterated</li>
- * </ul>
- *
- * <p>An optional {@link BlobDescriptorTransform} can be composed in to rewrite BLOB columns
- * in DESCRIPTOR mode.
+ * <p>BLOB chunked reading: lance-core 4.0.0 aborts the JVM in its Arrow C-stream export whenever a
+ * single {@code readAll} stream crosses Lance's internal BLOB page boundary (512 rows), and the
+ * requested {@code batchSize} does not change that. So BLOB reads issue one {@code readAll} per
+ * row-range chunk of {@link #BLOB_READ_CHUNK_ROWS} rows; non-BLOB reads use one streamed reader.
*/
public final class LanceRecordIterator implements ClosableIterator<UnsafeRow> {
+
+ /** Rows per {@code readAll} for BLOB reads. Must not exceed Lance's 512-row BLOB page size. */
+ public static final int BLOB_READ_CHUNK_ROWS = 512;
+
+ /**
+ * Supplies the sequence of {@link ArrowReader}s to drain, one after another. Single-reader
+ * mode yields one reader; BLOB chunked mode yields one fresh reader per row-range chunk. Each
+ * returned reader is owned (and closed) by {@link LanceRecordIterator}.
+ */
Review Comment:
renamed to ArrowReaderSequence
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/execution/datasources/lance/SparkLanceReaderBase.scala:
##########
@@ -148,8 +147,18 @@ class SparkLanceReaderBase(enableVectorizedReader: Boolean) extends SparkColumna
} else {
null
}
- lanceIterator = new LanceRecordIterator(
- allocator, lanceReader, arrowReader, iteratorSchema, filePath, blobTransform)
+ // lance-core 4.0.0 aborts the JVM when a single readAll stream crosses Lance's internal
+ // BLOB page boundary (512 rows). For BLOB-containing reads, drain the file in <=512-row
+ // range chunks (one fresh readAll each); non-BLOB reads keep the single streamed reader.
+ lanceIterator = if (!blobFieldNames.isEmpty) {
Review Comment:
fixed. it was a java.util.Set (no nonEmpty), switched to a scala Set + asJava at the ctor
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): read Lance BLOB columns in <=512-row chunks to avoid lance-core FFI abort [hudi]
Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
vinothchandar commented on PR #19181:
URL: https://github.com/apache/hudi/pull/19181#issuecomment-4897512733
@rahil-c @voonhous does this ring a bell. have you all seen this in your testing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): read Lance BLOB columns in <=512-row chunks to avoid lance-core FFI abort [hudi]
Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous closed pull request #19181: fix(spark): read Lance BLOB columns in <=512-row chunks to avoid lance-core FFI abort
URL: https://github.com/apache/hudi/pull/19181
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): read Lance BLOB columns in <=512-row chunks to avoid lance-core FFI abort [hudi]
Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #19181:
URL: https://github.com/apache/hudi/pull/19181#discussion_r3542085557
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/HoodieSparkLanceReader.java:
##########
@@ -208,8 +208,17 @@ private ClosableIterator<UnsafeRow> getUnsafeRowIterator(HoodieSchema requestedS
// Pinned to CONTENT: compaction/merge/log-replay need actual bytes to rewrite.
// The user-facing `hoodie.read.blob.inline.mode` is honored by SparkLanceReaderBase.
FileReadOptions readOpts = FileReadOptions.builder().blobReadMode(BlobReadMode.CONTENT).build();
- ArrowReader arrowReader = lanceReader.readAll(columnNames, null, DEFAULT_BATCH_SIZE, readOpts);
+ // BLOB reads must be chunked to dodge a lance-core FFI abort (see LanceRecordIterator).
+ // Unwrap a nullable UNION wrapper before checking whether the projection has a BLOB field.
+ HoodieSchema recordSchema = requestedSchema.getNonNullType();
+ boolean hasBlob = recordSchema.hasFields()
+ && recordSchema.getFields().stream().anyMatch(f -> f.schema().isBlobField());
+ if (hasBlob) {
+ return LanceRecordIterator.chunkedBlobReader(allocator, lanceReader, columnNames, readOpts,
Review Comment:
This branch is the internal CONTENT path (via `HoodieSparkFileReaderFactory`), so it is only reached by compaction/merge/log-replay, not datasource reads. Both batch-scale tests read through `SparkLanceReaderBase`, so nothing exercises this chunked branch above 512 rows. A MOR blob table with >512 rows in a base file would run it during compaction and cross the boundary untested. Can we add a >512-row MOR test that forces inline compaction and reads back, so this path is covered at the boundary too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): read Lance BLOB columns in <=512-row chunks to avoid lance-core FFI abort [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19181:
URL: https://github.com/apache/hudi/pull/19181#issuecomment-4880739734
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "63b524f2aa25df0976f440385e32d52f25796173",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "63b524f2aa25df0976f440385e32d52f25796173",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 63b524f2aa25df0976f440385e32d52f25796173 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): read Lance BLOB columns in <=512-row chunks to avoid lance-core FFI abort [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19181:
URL: https://github.com/apache/hudi/pull/19181#issuecomment-4881153224
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "63b524f2aa25df0976f440385e32d52f25796173",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15206",
"triggerID" : "63b524f2aa25df0976f440385e32d52f25796173",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 63b524f2aa25df0976f440385e32d52f25796173 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15206)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): read Lance BLOB columns in <=512-row chunks to avoid lance-core FFI abort [hudi]
Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous merged PR #19181:
URL: https://github.com/apache/hudi/pull/19181
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): read Lance BLOB columns in <=512-row chunks to avoid lance-core FFI abort [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19181:
URL: https://github.com/apache/hudi/pull/19181#issuecomment-4922900053
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "63b524f2aa25df0976f440385e32d52f25796173",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15206",
"triggerID" : "63b524f2aa25df0976f440385e32d52f25796173",
"triggerType" : "PUSH"
}, {
"hash" : "42a7e0d3c92f39c83b3b4a4ba0ab808102a8dfb7",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15405",
"triggerID" : "42a7e0d3c92f39c83b3b4a4ba0ab808102a8dfb7",
"triggerType" : "PUSH"
}, {
"hash" : "f3c076b3293f362371f8b469bf18af5b827ad416",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f3c076b3293f362371f8b469bf18af5b827ad416",
"triggerType" : "PUSH"
}, {
"hash" : "46409202e7e679c9e31af9b98fea6cd1c6ddffd6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15406",
"triggerID" : "46409202e7e679c9e31af9b98fea6cd1c6ddffd6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 63b524f2aa25df0976f440385e32d52f25796173 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15206)
* 42a7e0d3c92f39c83b3b4a4ba0ab808102a8dfb7 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15405)
* f3c076b3293f362371f8b469bf18af5b827ad416 UNKNOWN
* 46409202e7e679c9e31af9b98fea6cd1c6ddffd6 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15406)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org