You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "rahil-c (via GitHub)" <gi...@apache.org> on 2025/12/17 16:08:51 UTC
[PR] feat: Add HoodieBaseLanceFileWriter and implementation for SparkFileW… [hudi]
rahil-c opened a new pull request, #17629:
URL: https://github.com/apache/hudi/pull/17629
…riter
### Describe the issue this Pull Request addresses
Issue: https://github.com/apache/hudi/issues/17622
### Summary and Changelog
* Introduced a HoodieLanceBaseWriter that takes in some generic representation of data.
* Added a implementation of the above with HoodieSparkLanceWriter
* Leverage Lance java apis which require on Apache Arrow in order to represent underlying data
* Added unit tests for this class to write data from spark internal row to lance files works correctly
### Impact
none
### Risk Level
low
### Documentation Update
none
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Enough context is provided in the sections above
- [ ] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add HoodieBaseLanceFileWriter and implementation for SparkFileW… [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17629:
URL: https://github.com/apache/hudi/pull/17629#discussion_r2638197982
##########
hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/io/storage/TestHoodieSparkLanceWriter.java:
##########
@@ -0,0 +1,503 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.io.storage;
+
+import com.lancedb.lance.file.LanceFileReader;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.memory.RootAllocator;
+import org.apache.arrow.vector.BigIntVector;
+import org.apache.arrow.vector.BitVector;
+import org.apache.arrow.vector.Float4Vector;
+import org.apache.arrow.vector.Float8Vector;
+import org.apache.arrow.vector.IntVector;
+import org.apache.arrow.vector.VarBinaryVector;
+import org.apache.arrow.vector.VarCharVector;
+import org.apache.arrow.vector.VectorSchemaRoot;
+import org.apache.arrow.vector.ipc.ArrowReader;
+import org.apache.hudi.client.SparkTaskContextSupplier;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.testutils.HoodieTestUtils;
+import org.apache.hudi.storage.HoodieStorage;
+import org.apache.hudi.storage.StoragePath;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.catalyst.expressions.GenericInternalRow;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.unsafe.types.UTF8String;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.condition.DisabledIfSystemProperty;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.COMMIT_SEQNO_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.COMMIT_TIME_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.FILENAME_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.PARTITION_PATH_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.RECORD_KEY_METADATA_FIELD;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * Tests for {@link HoodieSparkLanceWriter}.
+ */
+@DisabledIfSystemProperty(named = "lance.skip.tests", matches = "true")
+public class TestHoodieSparkLanceWriter {
+
+ @TempDir
+ File tempDir;
+
+ private HoodieStorage storage;
+ private SparkTaskContextSupplier taskContextSupplier;
+ private String instantTime;
+
+ @BeforeEach
+ public void setUp() throws IOException {
+ storage = HoodieTestUtils.getStorage(tempDir.getAbsolutePath());
+ taskContextSupplier = new SparkTaskContextSupplier();
+ instantTime = "20251201120000000";
+ }
+
+ @AfterEach
+ public void tearDown() throws Exception {
+ if (storage != null) {
+ storage.close();
+ }
+ }
+
+ @Test
+ public void testWriteRowWithMetadataPopulation() throws Exception {
+ // Schema WITH meta fields (writer expects this when populateMetaFields=true)
+ StructType schema = createSchemaWithMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_with_metadata.lance");
+ HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, true);
+
+ // Create row with PLACEHOLDER meta fields + user data
+ InternalRow row = createRowWithMetaFields(1, "Alice", 30L);
+ HoodieKey key = new HoodieKey("key1", "partition1");
+
+ writer.writeRowWithMetadata(key, row);
+ writer.close();
+
+ // Verify using LanceFileReader
+ assertTrue(storage.exists(path), "Lance file should exist");
+
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator)) {
+
+ assertEquals(1, reader.numRows(), "Should have 1 record");
+
+ // Schema should have 5 meta fields + 3 user fields = 8 fields
+ assertEquals(8, reader.schema().getFields().size(), "Should have 8 fields");
+
+ // Read and verify data
+ ArrowReader arrowReader = reader.readAll(null, null, Integer.MAX_VALUE);
+ assertTrue(arrowReader.loadNextBatch(), "Should load batch");
+ VectorSchemaRoot root = arrowReader.getVectorSchemaRoot();
+
+ // Verify meta fields were populated
+ VarCharVector commitTimeVector = (VarCharVector) root.getVector(COMMIT_TIME_METADATA_FIELD.getFieldName());
+ assertNotNull(commitTimeVector);
+ assertEquals(instantTime, new String(commitTimeVector.get(0)), "Commit time should match");
+
+ VarCharVector recordKeyVector = (VarCharVector) root.getVector(RECORD_KEY_METADATA_FIELD.getFieldName());
+ assertEquals("key1", new String(recordKeyVector.get(0)), "Record key should match");
+
+ VarCharVector partitionPathVector = (VarCharVector) root.getVector(PARTITION_PATH_METADATA_FIELD.getFieldName());
+ assertEquals("partition1", new String(partitionPathVector.get(0)), "Partition path should match");
+
+ VarCharVector fileNameVector = (VarCharVector) root.getVector(FILENAME_METADATA_FIELD.getFieldName());
+ assertEquals(path.getName(), new String(fileNameVector.get(0)), "File name should match");
+
+ VarCharVector seqNoVector = (VarCharVector) root.getVector(COMMIT_SEQNO_METADATA_FIELD.getFieldName());
+ String seqNo = new String(seqNoVector.get(0));
+ assertTrue(seqNo.startsWith(instantTime), "Sequence number should start with instant time");
+
+ // Verify user data fields
+ IntVector idVector = (IntVector) root.getVector("id");
+ assertEquals(1, idVector.get(0), "ID should match");
+
+ VarCharVector nameVector = (VarCharVector) root.getVector("name");
+ assertEquals("Alice", new String(nameVector.get(0)), "Name should match");
+
+ BigIntVector ageVector = (BigIntVector) root.getVector("age");
+ assertEquals(30L, ageVector.get(0), "Age should match");
+
+ arrowReader.close();
+ }
+ }
+
+ @Test
+ public void testWriteRowWithoutMetadataPopulation() throws Exception {
+ // Schema WITHOUT meta fields
+ StructType schema = createSchemaWithoutMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_without_metadata.lance");
+ HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false);
+
+ // Create row with just user data (no meta fields)
+ InternalRow row = createRow(1, "Bob", 25L);
+ HoodieKey key = new HoodieKey("key2", "partition2");
+
+ writer.writeRowWithMetadata(key, row);
+ writer.close();
+
+ // Verify using LanceFileReader
+ assertTrue(storage.exists(path));
+
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator)) {
+
+ assertEquals(1, reader.numRows());
+
+ // Schema should have ONLY 3 user fields (no meta fields)
+ assertEquals(3, reader.schema().getFields().size(), "Should have only user fields");
+
+ // Read and verify data
+ ArrowReader arrowReader = reader.readAll(null, null, Integer.MAX_VALUE);
+ assertTrue(arrowReader.loadNextBatch());
+ VectorSchemaRoot root = arrowReader.getVectorSchemaRoot();
+
+ // Verify NO meta fields exist
+ assertFalse(hasField(root, COMMIT_TIME_METADATA_FIELD.getFieldName()), "Should not have commit time field");
+ assertFalse(hasField(root, RECORD_KEY_METADATA_FIELD.getFieldName()), "Should not have record key field");
+
+ // Verify user data
+ IntVector idVector = (IntVector) root.getVector("id");
+ assertEquals(1, idVector.get(0));
+
+ VarCharVector nameVector = (VarCharVector) root.getVector("name");
+ assertEquals("Bob", new String(nameVector.get(0)));
+
+ arrowReader.close();
+ }
+ }
+
+ @Test
+ public void testWriteRowSimple() throws Exception {
+ StructType schema = createSchemaWithoutMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_simple_write.lance");
+ HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false);
+
+ InternalRow row = createRow(1, "Charlie", 35L);
+ writer.writeRow("key3", row);
+ writer.close();
+
+ // Verify file exists and has correct record count
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator)) {
+ assertEquals(1, reader.numRows());
+ }
+ }
+
+ @Test
+ public void testSequenceIdGeneration() throws Exception {
+ StructType schema = createSchemaWithMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_sequence_id.lance");
+ HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, true);
+
+ // Write multiple records
+ for (int i = 0; i < 3; i++) {
+ InternalRow row = createRowWithMetaFields(i, "User" + i, 20L + i);
+ HoodieKey key = new HoodieKey("key" + i, "partition1");
+ writer.writeRowWithMetadata(key, row);
+ }
+ writer.close();
+
+ // Verify sequence IDs are unique and properly formatted
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator)) {
+
+ assertEquals(3, reader.numRows());
+
+ ArrowReader arrowReader = reader.readAll(null, null, Integer.MAX_VALUE);
+ arrowReader.loadNextBatch();
+ VectorSchemaRoot root = arrowReader.getVectorSchemaRoot();
+
+ VarCharVector seqNoVector = (VarCharVector) root.getVector(COMMIT_SEQNO_METADATA_FIELD.getFieldName());
Review Comment:
Will consolidate the logic of this sequence test and include it in testWriteRowWithMetadataPopulation, so we dont have two separate ones
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add HoodieBaseLanceFileWriter and implementation for SparkFileW… [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17629:
URL: https://github.com/apache/hudi/pull/17629#discussion_r2641907772
##########
hudi-hadoop-common/src/main/java/org/apache/hudi/io/lance/HoodieBaseLanceWriter.java:
##########
@@ -0,0 +1,185 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.io.lance;
+
+import com.lancedb.lance.file.LanceFileWriter;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.memory.RootAllocator;
+import org.apache.arrow.vector.VectorSchemaRoot;
+import org.apache.arrow.vector.types.pojo.Schema;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.storage.HoodieStorage;
+import org.apache.hudi.storage.StoragePath;
+
+import javax.annotation.concurrent.NotThreadSafe;
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * Base class for Hudi Lance file writers supporting different record types.
+ *
+ * This class handles common Lance file operations including:
+ * - LanceFileWriter lifecycle management
+ * - BufferAllocator management
+ * - Record buffering and batch flushing
+ * - File size checks
+ *
+ * Subclasses must implement type-specific conversion to Arrow format.
+ *
+ * @param <R> The record type (e.g., GenericRecord, InternalRow)
+ */
+@NotThreadSafe
+public abstract class HoodieBaseLanceWriter<R> implements Closeable {
+ protected static final int DEFAULT_BATCH_SIZE = 1000;
+ protected final HoodieStorage storage;
+ protected final StoragePath path;
+ protected final BufferAllocator allocator;
+ protected final List<R> bufferedRecords;
+ protected final int batchSize;
+ protected final long maxFileSize;
+ protected long writtenRecordCount = 0;
+ protected VectorSchemaRoot root;
+
+ private LanceFileWriter writer;
+
+ /**
+ * Constructor for base Lance writer.
+ *
+ * @param storage HoodieStorage instance
+ * @param path Path where Lance file will be written
+ * @param batchSize Number of records to buffer before flushing to Lance
+ * @param maxFileSize Maximum file size in bytes before rolling over to new file
Review Comment:
I think for now can remove, this might have been originally for the `canWrite()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add HoodieBaseLanceFileWriter and implementation for SparkFileW… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17629:
URL: https://github.com/apache/hudi/pull/17629#issuecomment-3686818638
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9633e6bc476078ebf78b401e00ccbf43cdab67e8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10326",
"triggerID" : "9633e6bc476078ebf78b401e00ccbf43cdab67e8",
"triggerType" : "PUSH"
}, {
"hash" : "6adcb20a0c841eb4164570bed7e3daa28349be35",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10335",
"triggerID" : "6adcb20a0c841eb4164570bed7e3daa28349be35",
"triggerType" : "PUSH"
}, {
"hash" : "f0f38ca10ad7bd5db724f5568aa560e0c2af8539",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10401",
"triggerID" : "f0f38ca10ad7bd5db724f5568aa560e0c2af8539",
"triggerType" : "PUSH"
}, {
"hash" : "3e18561eb435a7b4e5452c2df03793db7e70ad82",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10420",
"triggerID" : "3e18561eb435a7b4e5452c2df03793db7e70ad82",
"triggerType" : "PUSH"
}, {
"hash" : "33cfffa5175585e24bd00025a01f3bb4f5abf3fb",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10424",
"triggerID" : "33cfffa5175585e24bd00025a01f3bb4f5abf3fb",
"triggerType" : "PUSH"
}, {
"hash" : "b169123159877a93ac7a88862484211c1fa4e4f8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b169123159877a93ac7a88862484211c1fa4e4f8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 33cfffa5175585e24bd00025a01f3bb4f5abf3fb Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10424)
* b169123159877a93ac7a88862484211c1fa4e4f8 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add HoodieBaseLanceFileWriter and implementation for SparkFileW… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17629:
URL: https://github.com/apache/hudi/pull/17629#issuecomment-3679921401
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9633e6bc476078ebf78b401e00ccbf43cdab67e8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10326",
"triggerID" : "9633e6bc476078ebf78b401e00ccbf43cdab67e8",
"triggerType" : "PUSH"
}, {
"hash" : "6adcb20a0c841eb4164570bed7e3daa28349be35",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10335",
"triggerID" : "6adcb20a0c841eb4164570bed7e3daa28349be35",
"triggerType" : "PUSH"
}, {
"hash" : "f0f38ca10ad7bd5db724f5568aa560e0c2af8539",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10401",
"triggerID" : "f0f38ca10ad7bd5db724f5568aa560e0c2af8539",
"triggerType" : "PUSH"
}, {
"hash" : "3e18561eb435a7b4e5452c2df03793db7e70ad82",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10420",
"triggerID" : "3e18561eb435a7b4e5452c2df03793db7e70ad82",
"triggerType" : "PUSH"
}, {
"hash" : "33cfffa5175585e24bd00025a01f3bb4f5abf3fb",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10424",
"triggerID" : "33cfffa5175585e24bd00025a01f3bb4f5abf3fb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 33cfffa5175585e24bd00025a01f3bb4f5abf3fb Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10424)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add HoodieBaseLanceFileWriter and implementation for SparkFileW… [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on PR #17629:
URL: https://github.com/apache/hudi/pull/17629#issuecomment-3682101189
<img width="319" height="578" alt="Screenshot 2025-12-22 at 8 28 28 AM" src="https://github.com/user-attachments/assets/fd8393df-61c1-4688-8ebe-346da1cd33ec" />
Azure CI is passing on second rerun can we merge this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add HoodieBaseLanceFileWriter and implementation for SparkFileW… [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17629:
URL: https://github.com/apache/hudi/pull/17629#discussion_r2641726793
##########
hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/io/storage/TestHoodieSparkLanceWriter.java:
##########
@@ -0,0 +1,454 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.io.storage;
+
+import com.lancedb.lance.file.LanceFileReader;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.memory.RootAllocator;
+import org.apache.arrow.vector.BigIntVector;
+import org.apache.arrow.vector.BitVector;
+import org.apache.arrow.vector.Float4Vector;
+import org.apache.arrow.vector.Float8Vector;
+import org.apache.arrow.vector.IntVector;
+import org.apache.arrow.vector.VarBinaryVector;
+import org.apache.arrow.vector.VarCharVector;
+import org.apache.arrow.vector.VectorSchemaRoot;
+import org.apache.arrow.vector.ipc.ArrowReader;
+import org.apache.hudi.client.SparkTaskContextSupplier;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.testutils.HoodieTestUtils;
+import org.apache.hudi.storage.HoodieStorage;
+import org.apache.hudi.storage.StoragePath;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.catalyst.expressions.GenericInternalRow;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.unsafe.types.UTF8String;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.condition.DisabledIfSystemProperty;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.COMMIT_SEQNO_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.COMMIT_TIME_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.FILENAME_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.PARTITION_PATH_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.RECORD_KEY_METADATA_FIELD;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * Tests for {@link HoodieSparkLanceWriter}.
+ */
+@DisabledIfSystemProperty(named = "lance.skip.tests", matches = "true")
+public class TestHoodieSparkLanceWriter {
+
+ @TempDir
+ File tempDir;
+
+ private HoodieStorage storage;
+ private SparkTaskContextSupplier taskContextSupplier;
+ private String instantTime;
+
+ @BeforeEach
+ public void setUp() throws IOException {
+ storage = HoodieTestUtils.getStorage(tempDir.getAbsolutePath());
+ taskContextSupplier = new SparkTaskContextSupplier();
+ instantTime = "20251201120000000";
+ }
+
+ @AfterEach
+ public void tearDown() throws Exception {
+ if (storage != null) {
+ storage.close();
+ }
+ }
+
+ @Test
+ public void testWriteRowWithMetadataPopulation() throws Exception {
+ // Schema WITH meta fields (writer expects this when populateMetaFields=true)
+ StructType schema = createSchemaWithMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_with_metadata.lance");
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, true)) {
+ // Write multiple records to test metadata population and sequence ID generation
+ for (int i = 0; i < 3; i++) {
+ InternalRow row = createRowWithMetaFields(i, "User" + i, 20L + i);
+ HoodieKey key = new HoodieKey("key" + i, "partition1");
+ writer.writeRowWithMetadata(key, row);
+ }
+ }
+
+ // Verify using LanceFileReader
+ assertTrue(storage.exists(path), "Lance file should exist");
+
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator);
+ ArrowReader arrowReader = reader.readAll(null, null, Integer.MAX_VALUE)) {
+
+ assertEquals(3, reader.numRows(), "Should have 3 records");
+
+ // Schema should have 5 meta fields + 3 user fields = 8 fields
+ assertEquals(8, reader.schema().getFields().size(), "Should have 8 fields");
+
+ // Read and verify data
+ assertTrue(arrowReader.loadNextBatch(), "Should load batch");
+ VectorSchemaRoot root = arrowReader.getVectorSchemaRoot();
+
+ // Verify meta fields were populated for first record
+ VarCharVector commitTimeVector = (VarCharVector) root.getVector(COMMIT_TIME_METADATA_FIELD.getFieldName());
+ assertNotNull(commitTimeVector);
+ assertEquals(instantTime, new String(commitTimeVector.get(0)), "Commit time should match");
+
+ VarCharVector recordKeyVector = (VarCharVector) root.getVector(RECORD_KEY_METADATA_FIELD.getFieldName());
+ assertEquals("key0", new String(recordKeyVector.get(0)), "Record key should match");
+
+ VarCharVector partitionPathVector = (VarCharVector) root.getVector(PARTITION_PATH_METADATA_FIELD.getFieldName());
+ assertEquals("partition1", new String(partitionPathVector.get(0)), "Partition path should match");
+
+ VarCharVector fileNameVector = (VarCharVector) root.getVector(FILENAME_METADATA_FIELD.getFieldName());
+ assertEquals(path.getName(), new String(fileNameVector.get(0)), "File name should match");
+
+ // Verify sequence IDs are unique and properly formatted
+ VarCharVector seqNoVector = (VarCharVector) root.getVector(COMMIT_SEQNO_METADATA_FIELD.getFieldName());
+ List<String> seqNos = new ArrayList<>();
+ for (int i = 0; i < 3; i++) {
+ String seqNo = new String(seqNoVector.get(i));
+ assertTrue(seqNo.startsWith(instantTime + "_"), "Sequence number should start with instant time");
+ seqNos.add(seqNo);
+ }
+ assertEquals(3, seqNos.stream().distinct().count(), "All sequence IDs should be unique");
+
+ // Verify user data fields for first record
+ IntVector idVector = (IntVector) root.getVector("id");
+ assertEquals(0, idVector.get(0), "ID should match");
+
+ VarCharVector nameVector = (VarCharVector) root.getVector("name");
+ assertEquals("User0", new String(nameVector.get(0)), "Name should match");
+
+ BigIntVector ageVector = (BigIntVector) root.getVector("age");
+ assertEquals(20L, ageVector.get(0), "Age should match");
+ }
+ }
+
+ @Test
+ public void testWriteRowWithoutMetadataPopulation() throws Exception {
+ // Schema WITHOUT meta fields
+ StructType schema = createSchemaWithoutMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_without_metadata.lance");
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false)) {
+ // Create row with just user data (no meta fields)
+ InternalRow row = createRow(1, "Bob", 25L);
+ HoodieKey key = new HoodieKey("key2", "partition2");
+
+ writer.writeRowWithMetadata(key, row);
+ }
+
+ // Verify using LanceFileReader
+ assertTrue(storage.exists(path));
+
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator);
+ ArrowReader arrowReader = reader.readAll(null, null, Integer.MAX_VALUE)) {
+
+ assertEquals(1, reader.numRows());
+
+ // Schema should have ONLY 3 user fields (no meta fields)
+ assertEquals(3, reader.schema().getFields().size(), "Should have only user fields");
+
+ // Read and verify data
+ assertTrue(arrowReader.loadNextBatch());
+ VectorSchemaRoot root = arrowReader.getVectorSchemaRoot();
+
+ // Verify NO meta fields exist
+ assertFalse(hasField(root, COMMIT_TIME_METADATA_FIELD.getFieldName()), "Should not have commit time field");
+ assertFalse(hasField(root, RECORD_KEY_METADATA_FIELD.getFieldName()), "Should not have record key field");
+
+ // Verify user data
+ IntVector idVector = (IntVector) root.getVector("id");
+ assertEquals(1, idVector.get(0));
+
+ VarCharVector nameVector = (VarCharVector) root.getVector("name");
+ assertEquals("Bob", new String(nameVector.get(0)));
+ }
+ }
+
+ @Test
+ public void testWriteRowSimple() throws Exception {
+ StructType schema = createSchemaWithoutMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_simple_write.lance");
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false)) {
+ InternalRow row = createRow(1, "Charlie", 35L);
+ writer.writeRow("key3", row);
+ }
+
+ // Verify file exists and has correct record count
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator)) {
+ assertEquals(1, reader.numRows());
+ }
+ }
+
+ @Test
+ public void testBatchFlushing() throws Exception {
+ StructType schema = createSchemaWithoutMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_batch_flush.lance");
+ // Write more than DEFAULT_BATCH_SIZE (1000) records
+ int recordCount = 2500;
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false)) {
+ for (int i = 0; i < recordCount; i++) {
+ InternalRow row = createRow(i, "User" + i, 20L + i);
+ writer.writeRow("key" + i, row);
+ }
+ }
+
+ // Verify all records were written
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator)) {
+ assertEquals(recordCount, reader.numRows(), "All records should be written");
+ }
+ }
+
+ @Test
+ public void testPrimitiveTypes() throws Exception {
+ StructType schema = new StructType()
+ .add("int_field", DataTypes.IntegerType, false)
+ .add("long_field", DataTypes.LongType, false)
+ .add("float_field", DataTypes.FloatType, false)
+ .add("double_field", DataTypes.DoubleType, false)
+ .add("bool_field", DataTypes.BooleanType, false)
+ .add("string_field", DataTypes.StringType, false)
+ .add("binary_field", DataTypes.BinaryType, false);
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_primitives.lance");
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false)) {
+ GenericInternalRow row = new GenericInternalRow(new Object[]{
+ 42, // int
+ 123456789L, // long
+ 3.14f, // float
+ 2.71828, // double
+ true, // boolean
+ UTF8String.fromString("test"), // string
+ new byte[]{1, 2, 3, 4} // binary
+ });
+
+ writer.writeRow("key1", row);
+ }
+
+ // Verify all types were written correctly
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator);
+ ArrowReader arrowReader = reader.readAll(null, null, Integer.MAX_VALUE)) {
+
+ assertEquals(1, reader.numRows());
+ assertEquals(7, reader.schema().getFields().size());
+
+ arrowReader.loadNextBatch();
+ VectorSchemaRoot root = arrowReader.getVectorSchemaRoot();
+
+ assertEquals(42, ((IntVector) root.getVector("int_field")).get(0));
+ assertEquals(123456789L, ((BigIntVector) root.getVector("long_field")).get(0));
+ assertEquals(3.14f, ((Float4Vector) root.getVector("float_field")).get(0), 0.001);
+ assertEquals(2.71828, ((Float8Vector) root.getVector("double_field")).get(0), 0.00001);
+ assertEquals(1, ((BitVector) root.getVector("bool_field")).get(0));
+ assertEquals("test", new String(((VarCharVector) root.getVector("string_field")).get(0)));
+
+ byte[] binary = ((VarBinaryVector) root.getVector("binary_field")).get(0);
+ assertEquals(4, binary.length);
+ }
+ }
+
+ @Test
+ public void testNullValues() throws Exception {
+ StructType schema = new StructType()
+ .add("id", DataTypes.IntegerType, false)
+ .add("name", DataTypes.StringType, true)
+ .add("age", DataTypes.LongType, true);
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_nulls.lance");
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false)) {
+ // Write rows with null values
+ writer.writeRow("key1", createRow(1, "Alice", 30L));
+ writer.writeRow("key2", createRow(2, null, 25L)); // null name
+ writer.writeRow("key3", createRow(3, "Charlie", null)); // null age
+ }
+
+ // Verify nulls are preserved
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator);
+ ArrowReader arrowReader = reader.readAll(null, null, Integer.MAX_VALUE)) {
+
+ assertEquals(3, reader.numRows());
+
+ arrowReader.loadNextBatch();
+ VectorSchemaRoot root = arrowReader.getVectorSchemaRoot();
+
+ VarCharVector nameVector = (VarCharVector) root.getVector("name");
+ assertFalse(nameVector.isNull(0), "First name should not be null");
+ assertTrue(nameVector.isNull(1), "Second name should be null");
+ assertFalse(nameVector.isNull(2), "Third name should not be null");
+
+ BigIntVector ageVector = (BigIntVector) root.getVector("age");
+ assertFalse(ageVector.isNull(0), "First age should not be null");
+ assertFalse(ageVector.isNull(1), "Second age should not be null");
+ assertTrue(ageVector.isNull(2), "Third age should be null");
+ }
+ }
+
+ @Test
+ public void testWriteEmptyDataset() throws Exception {
+ StructType schema = new StructType()
+ .add("id", DataTypes.IntegerType, false);
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_empty.lance");
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false)) {
+ // Close without writing any rows
+ }
+
+ // Lance doesn't create a file if no data is written
+ assertFalse(storage.exists(path), "Lance file should not exist when no data is written");
Review Comment:
@the-other-tim-brown @yihua When looking at the Hudi base parquet writer constructor, I noticed that the in the underlying parquet writer's build https://github.com/apache/hudi/blob/master/hudi-hadoop-common/src/main/java/org/apache/hudi/io/hadoop/HoodieBaseParquetWriter.java#L82 that it has the following code
```
ParquetFileWriter fileWriter = new ParquetFileWriter(
file, schema, mode, rowGroupSize, maxPaddingSize,
encodingProps.getColumnIndexTruncateLength(), encodingProps.getStatisticsTruncateLength(),
encodingProps.getPageWriteChecksumEnabled(), encryptionProperties);
fileWriter.start();
```
So i am not sure if the default behavior of parquet files is that it will create an empty file on intilization of the constructor, regardless if there is any actual data to write?
```
/**
* start the file
* @throws IOException if there is an error while writing
*/
public void start() throws IOException {
state = state.start();
LOG.debug("{}: start", out.getPos());
byte[] magic = MAGIC;
if (null != fileEncryptor && fileEncryptor.isFooterEncrypted()) {
magic = EFMAGIC;
}
out.write(magic);
}
```
https://github.com/apache/parquet-java/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileWriter.java#L596
Unless you guys were referring to another code path, im missing that we should be looking at.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add HoodieBaseLanceFileWriter and implementation for SparkFileW… [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17629:
URL: https://github.com/apache/hudi/pull/17629#discussion_r2641898018
##########
hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/io/storage/TestHoodieSparkLanceWriter.java:
##########
@@ -0,0 +1,454 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.io.storage;
+
+import com.lancedb.lance.file.LanceFileReader;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.memory.RootAllocator;
+import org.apache.arrow.vector.BigIntVector;
+import org.apache.arrow.vector.BitVector;
+import org.apache.arrow.vector.Float4Vector;
+import org.apache.arrow.vector.Float8Vector;
+import org.apache.arrow.vector.IntVector;
+import org.apache.arrow.vector.VarBinaryVector;
+import org.apache.arrow.vector.VarCharVector;
+import org.apache.arrow.vector.VectorSchemaRoot;
+import org.apache.arrow.vector.ipc.ArrowReader;
+import org.apache.hudi.client.SparkTaskContextSupplier;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.testutils.HoodieTestUtils;
+import org.apache.hudi.storage.HoodieStorage;
+import org.apache.hudi.storage.StoragePath;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.catalyst.expressions.GenericInternalRow;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.unsafe.types.UTF8String;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.condition.DisabledIfSystemProperty;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.COMMIT_SEQNO_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.COMMIT_TIME_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.FILENAME_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.PARTITION_PATH_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.RECORD_KEY_METADATA_FIELD;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * Tests for {@link HoodieSparkLanceWriter}.
+ */
+@DisabledIfSystemProperty(named = "lance.skip.tests", matches = "true")
+public class TestHoodieSparkLanceWriter {
+
+ @TempDir
+ File tempDir;
+
+ private HoodieStorage storage;
+ private SparkTaskContextSupplier taskContextSupplier;
+ private String instantTime;
+
+ @BeforeEach
+ public void setUp() throws IOException {
+ storage = HoodieTestUtils.getStorage(tempDir.getAbsolutePath());
+ taskContextSupplier = new SparkTaskContextSupplier();
+ instantTime = "20251201120000000";
+ }
+
+ @AfterEach
+ public void tearDown() throws Exception {
+ if (storage != null) {
+ storage.close();
+ }
+ }
+
+ @Test
+ public void testWriteRowWithMetadataPopulation() throws Exception {
+ // Schema WITH meta fields (writer expects this when populateMetaFields=true)
+ StructType schema = createSchemaWithMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_with_metadata.lance");
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, true)) {
+ // Write multiple records to test metadata population and sequence ID generation
+ for (int i = 0; i < 3; i++) {
+ InternalRow row = createRowWithMetaFields(i, "User" + i, 20L + i);
+ HoodieKey key = new HoodieKey("key" + i, "partition1");
+ writer.writeRowWithMetadata(key, row);
+ }
+ }
+
+ // Verify using LanceFileReader
+ assertTrue(storage.exists(path), "Lance file should exist");
+
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator);
+ ArrowReader arrowReader = reader.readAll(null, null, Integer.MAX_VALUE)) {
+
+ assertEquals(3, reader.numRows(), "Should have 3 records");
+
+ // Schema should have 5 meta fields + 3 user fields = 8 fields
+ assertEquals(8, reader.schema().getFields().size(), "Should have 8 fields");
+
+ // Read and verify data
+ assertTrue(arrowReader.loadNextBatch(), "Should load batch");
+ VectorSchemaRoot root = arrowReader.getVectorSchemaRoot();
+
+ // Verify meta fields were populated for first record
+ VarCharVector commitTimeVector = (VarCharVector) root.getVector(COMMIT_TIME_METADATA_FIELD.getFieldName());
+ assertNotNull(commitTimeVector);
+ assertEquals(instantTime, new String(commitTimeVector.get(0)), "Commit time should match");
+
+ VarCharVector recordKeyVector = (VarCharVector) root.getVector(RECORD_KEY_METADATA_FIELD.getFieldName());
+ assertEquals("key0", new String(recordKeyVector.get(0)), "Record key should match");
+
+ VarCharVector partitionPathVector = (VarCharVector) root.getVector(PARTITION_PATH_METADATA_FIELD.getFieldName());
+ assertEquals("partition1", new String(partitionPathVector.get(0)), "Partition path should match");
+
+ VarCharVector fileNameVector = (VarCharVector) root.getVector(FILENAME_METADATA_FIELD.getFieldName());
+ assertEquals(path.getName(), new String(fileNameVector.get(0)), "File name should match");
+
+ // Verify sequence IDs are unique and properly formatted
+ VarCharVector seqNoVector = (VarCharVector) root.getVector(COMMIT_SEQNO_METADATA_FIELD.getFieldName());
+ List<String> seqNos = new ArrayList<>();
+ for (int i = 0; i < 3; i++) {
+ String seqNo = new String(seqNoVector.get(i));
+ assertTrue(seqNo.startsWith(instantTime + "_"), "Sequence number should start with instant time");
+ seqNos.add(seqNo);
+ }
+ assertEquals(3, seqNos.stream().distinct().count(), "All sequence IDs should be unique");
+
+ // Verify user data fields for first record
+ IntVector idVector = (IntVector) root.getVector("id");
+ assertEquals(0, idVector.get(0), "ID should match");
+
+ VarCharVector nameVector = (VarCharVector) root.getVector("name");
+ assertEquals("User0", new String(nameVector.get(0)), "Name should match");
+
+ BigIntVector ageVector = (BigIntVector) root.getVector("age");
+ assertEquals(20L, ageVector.get(0), "Age should match");
+ }
+ }
+
+ @Test
+ public void testWriteRowWithoutMetadataPopulation() throws Exception {
+ // Schema WITHOUT meta fields
+ StructType schema = createSchemaWithoutMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_without_metadata.lance");
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false)) {
+ // Create row with just user data (no meta fields)
+ InternalRow row = createRow(1, "Bob", 25L);
+ HoodieKey key = new HoodieKey("key2", "partition2");
+
+ writer.writeRowWithMetadata(key, row);
+ }
+
+ // Verify using LanceFileReader
+ assertTrue(storage.exists(path));
+
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator);
+ ArrowReader arrowReader = reader.readAll(null, null, Integer.MAX_VALUE)) {
+
+ assertEquals(1, reader.numRows());
+
+ // Schema should have ONLY 3 user fields (no meta fields)
+ assertEquals(3, reader.schema().getFields().size(), "Should have only user fields");
+
+ // Read and verify data
+ assertTrue(arrowReader.loadNextBatch());
+ VectorSchemaRoot root = arrowReader.getVectorSchemaRoot();
+
+ // Verify NO meta fields exist
+ assertFalse(hasField(root, COMMIT_TIME_METADATA_FIELD.getFieldName()), "Should not have commit time field");
+ assertFalse(hasField(root, RECORD_KEY_METADATA_FIELD.getFieldName()), "Should not have record key field");
+
+ // Verify user data
+ IntVector idVector = (IntVector) root.getVector("id");
+ assertEquals(1, idVector.get(0));
+
+ VarCharVector nameVector = (VarCharVector) root.getVector("name");
+ assertEquals("Bob", new String(nameVector.get(0)));
+ }
+ }
+
+ @Test
+ public void testWriteRowSimple() throws Exception {
+ StructType schema = createSchemaWithoutMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_simple_write.lance");
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false)) {
+ InternalRow row = createRow(1, "Charlie", 35L);
+ writer.writeRow("key3", row);
+ }
+
+ // Verify file exists and has correct record count
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator)) {
+ assertEquals(1, reader.numRows());
+ }
+ }
+
+ @Test
+ public void testBatchFlushing() throws Exception {
+ StructType schema = createSchemaWithoutMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_batch_flush.lance");
+ // Write more than DEFAULT_BATCH_SIZE (1000) records
+ int recordCount = 2500;
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false)) {
+ for (int i = 0; i < recordCount; i++) {
+ InternalRow row = createRow(i, "User" + i, 20L + i);
+ writer.writeRow("key" + i, row);
+ }
+ }
+
+ // Verify all records were written
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator)) {
+ assertEquals(recordCount, reader.numRows(), "All records should be written");
+ }
+ }
+
+ @Test
+ public void testPrimitiveTypes() throws Exception {
+ StructType schema = new StructType()
+ .add("int_field", DataTypes.IntegerType, false)
+ .add("long_field", DataTypes.LongType, false)
+ .add("float_field", DataTypes.FloatType, false)
+ .add("double_field", DataTypes.DoubleType, false)
+ .add("bool_field", DataTypes.BooleanType, false)
+ .add("string_field", DataTypes.StringType, false)
+ .add("binary_field", DataTypes.BinaryType, false);
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_primitives.lance");
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false)) {
+ GenericInternalRow row = new GenericInternalRow(new Object[]{
+ 42, // int
+ 123456789L, // long
+ 3.14f, // float
+ 2.71828, // double
+ true, // boolean
+ UTF8String.fromString("test"), // string
+ new byte[]{1, 2, 3, 4} // binary
+ });
+
+ writer.writeRow("key1", row);
+ }
+
+ // Verify all types were written correctly
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator);
+ ArrowReader arrowReader = reader.readAll(null, null, Integer.MAX_VALUE)) {
+
+ assertEquals(1, reader.numRows());
+ assertEquals(7, reader.schema().getFields().size());
+
+ arrowReader.loadNextBatch();
+ VectorSchemaRoot root = arrowReader.getVectorSchemaRoot();
+
+ assertEquals(42, ((IntVector) root.getVector("int_field")).get(0));
+ assertEquals(123456789L, ((BigIntVector) root.getVector("long_field")).get(0));
+ assertEquals(3.14f, ((Float4Vector) root.getVector("float_field")).get(0), 0.001);
+ assertEquals(2.71828, ((Float8Vector) root.getVector("double_field")).get(0), 0.00001);
+ assertEquals(1, ((BitVector) root.getVector("bool_field")).get(0));
+ assertEquals("test", new String(((VarCharVector) root.getVector("string_field")).get(0)));
+
+ byte[] binary = ((VarBinaryVector) root.getVector("binary_field")).get(0);
+ assertEquals(4, binary.length);
+ }
+ }
+
+ @Test
+ public void testNullValues() throws Exception {
+ StructType schema = new StructType()
+ .add("id", DataTypes.IntegerType, false)
+ .add("name", DataTypes.StringType, true)
+ .add("age", DataTypes.LongType, true);
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_nulls.lance");
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false)) {
+ // Write rows with null values
+ writer.writeRow("key1", createRow(1, "Alice", 30L));
+ writer.writeRow("key2", createRow(2, null, 25L)); // null name
+ writer.writeRow("key3", createRow(3, "Charlie", null)); // null age
+ }
+
+ // Verify nulls are preserved
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator);
+ ArrowReader arrowReader = reader.readAll(null, null, Integer.MAX_VALUE)) {
+
+ assertEquals(3, reader.numRows());
+
+ arrowReader.loadNextBatch();
+ VectorSchemaRoot root = arrowReader.getVectorSchemaRoot();
+
+ VarCharVector nameVector = (VarCharVector) root.getVector("name");
+ assertFalse(nameVector.isNull(0), "First name should not be null");
+ assertTrue(nameVector.isNull(1), "Second name should be null");
+ assertFalse(nameVector.isNull(2), "Third name should not be null");
+
+ BigIntVector ageVector = (BigIntVector) root.getVector("age");
+ assertFalse(ageVector.isNull(0), "First age should not be null");
+ assertFalse(ageVector.isNull(1), "Second age should not be null");
+ assertTrue(ageVector.isNull(2), "Third age should be null");
+ }
+ }
+
+ @Test
+ public void testWriteEmptyDataset() throws Exception {
+ StructType schema = new StructType()
+ .add("id", DataTypes.IntegerType, false);
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_empty.lance");
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false)) {
+ // Close without writing any rows
+ }
+
+ // Lance doesn't create a file if no data is written
+ assertFalse(storage.exists(path), "Lance file should not exist when no data is written");
Review Comment:
Following @the-other-tim-brown recommendation I tried writing an equivalent of this test above, but using the `HoodieSparkParquetWriter`
https://gist.github.com/rahil-c/f1251f999df7674077153acdf37e610d
However when doing so i ran into the following issue before seeing an empty file created
```
org.apache.hudi.exception.HoodieException: Unable to instantiate class org.apache.hudi.io.storage.row.HoodieRowParquetWriteSupport
at org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:75)
at org.apache.hudi.io.storage.row.HoodieRowParquetWriteSupport.getHoodieRowParquetWriteSupport(HoodieRowParquetWriteSupport.java:416)
at org.apache.hudi.io.storage.TestHoodieSparkParquetWriter.testWriteEmptyDataset(TestHoodieSparkParquetWriter.java:87)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
Caused by: java.lang.reflect.InvocationTargetException
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
at org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:73)
... 5 more
Caused by: java.lang.IllegalArgumentException: Schema string cannot be null or empty
at org.apache.hudi.common.util.ValidationUtils.checkArgument(ValidationUtils.java:42)
at org.apache.hudi.common.schema.HoodieSchema$Parser.parse(HoodieSchema.java:893)
at org.apache.hudi.common.schema.HoodieSchema.parse(HoodieSchema.java:138)
at org.apache.hudi.io.storage.row.HoodieRowParquetWriteSupport.lambda$new$2(HoodieRowParquetWriteSupport.java:138)
at org.apache.hudi.common.util.Option.orElseGet(Option.java:153)
at org.apache.hudi.io.storage.row.HoodieRowParquetWriteSupport.<init>(HoodieRowParquetWriteSupport.java:136)
... 10 more
```
I then discussed with @yihua offline, who pointed me to see if `HoodieAvroParquetWriter` creates an empty file. When writing a similar test in test `TestHoodieAvroParquetWriter` : https://gist.github.com/rahil-c/0b719717f72eb59dacdd646c2802309b
was able to confirm that an empty file is created despite no actual data being rows being written.
When inspecting the underlying parquet file that was created from the test I see the following, where the empty file persists the schema.
<img width="1548" height="274" alt="Screenshot 2025-12-22 at 11 27 32 PM" src="https://github.com/user-attachments/assets/648138ec-6d46-45dd-91fd-a071ea0a5481" />
@the-other-tim-brown @yihua
So to confirm is the behavior we want to achieve, when there is no data present that we just want to write the schema to a lance base file? The lance java file writer does not have any apis writing to the footer similar to parquet, https://github.com/lance-format/lance/blob/main/java/src/main/java/org/lance/file/LanceFileWriter.java#L31 so need to investigate this more.
Update: I think i might have fix for this in our hudi lance integration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add HoodieBaseLanceFileWriter and implementation for SparkFileW… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17629:
URL: https://github.com/apache/hudi/pull/17629#issuecomment-3666080465
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9633e6bc476078ebf78b401e00ccbf43cdab67e8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10326",
"triggerID" : "9633e6bc476078ebf78b401e00ccbf43cdab67e8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9633e6bc476078ebf78b401e00ccbf43cdab67e8 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10326)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add HoodieBaseLanceFileWriter and implementation for SparkFileW… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17629:
URL: https://github.com/apache/hudi/pull/17629#issuecomment-3667585405
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9633e6bc476078ebf78b401e00ccbf43cdab67e8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10326",
"triggerID" : "9633e6bc476078ebf78b401e00ccbf43cdab67e8",
"triggerType" : "PUSH"
}, {
"hash" : "6adcb20a0c841eb4164570bed7e3daa28349be35",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10335",
"triggerID" : "6adcb20a0c841eb4164570bed7e3daa28349be35",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6adcb20a0c841eb4164570bed7e3daa28349be35 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10335)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add HoodieBaseLanceFileWriter and implementation for SparkFileW… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17629:
URL: https://github.com/apache/hudi/pull/17629#issuecomment-3679748079
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9633e6bc476078ebf78b401e00ccbf43cdab67e8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10326",
"triggerID" : "9633e6bc476078ebf78b401e00ccbf43cdab67e8",
"triggerType" : "PUSH"
}, {
"hash" : "6adcb20a0c841eb4164570bed7e3daa28349be35",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10335",
"triggerID" : "6adcb20a0c841eb4164570bed7e3daa28349be35",
"triggerType" : "PUSH"
}, {
"hash" : "f0f38ca10ad7bd5db724f5568aa560e0c2af8539",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10401",
"triggerID" : "f0f38ca10ad7bd5db724f5568aa560e0c2af8539",
"triggerType" : "PUSH"
}, {
"hash" : "3e18561eb435a7b4e5452c2df03793db7e70ad82",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10420",
"triggerID" : "3e18561eb435a7b4e5452c2df03793db7e70ad82",
"triggerType" : "PUSH"
}, {
"hash" : "33cfffa5175585e24bd00025a01f3bb4f5abf3fb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "33cfffa5175585e24bd00025a01f3bb4f5abf3fb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3e18561eb435a7b4e5452c2df03793db7e70ad82 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10420)
* 33cfffa5175585e24bd00025a01f3bb4f5abf3fb UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add HoodieBaseLanceFileWriter and implementation for SparkFileW… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17629:
URL: https://github.com/apache/hudi/pull/17629#issuecomment-3677838408
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9633e6bc476078ebf78b401e00ccbf43cdab67e8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10326",
"triggerID" : "9633e6bc476078ebf78b401e00ccbf43cdab67e8",
"triggerType" : "PUSH"
}, {
"hash" : "6adcb20a0c841eb4164570bed7e3daa28349be35",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10335",
"triggerID" : "6adcb20a0c841eb4164570bed7e3daa28349be35",
"triggerType" : "PUSH"
}, {
"hash" : "f0f38ca10ad7bd5db724f5568aa560e0c2af8539",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f0f38ca10ad7bd5db724f5568aa560e0c2af8539",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6adcb20a0c841eb4164570bed7e3daa28349be35 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10335)
* f0f38ca10ad7bd5db724f5568aa560e0c2af8539 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add HoodieBaseLanceFileWriter and implementation for SparkFileW… [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #17629:
URL: https://github.com/apache/hudi/pull/17629#discussion_r2641717532
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/HoodieSparkLanceWriter.java:
##########
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.io.storage;
+
+import com.lancedb.lance.spark.arrow.LanceArrowWriter;
+import org.apache.arrow.vector.types.pojo.Schema;
+import org.apache.hudi.common.engine.TaskContextSupplier;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.io.lance.HoodieBaseLanceWriter;
+import org.apache.hudi.storage.HoodieStorage;
+import org.apache.hudi.storage.StoragePath;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.util.LanceArrowUtils;
+import org.apache.spark.unsafe.types.UTF8String;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.function.Function;
+
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.COMMIT_SEQNO_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.COMMIT_TIME_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.FILENAME_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.PARTITION_PATH_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.RECORD_KEY_METADATA_FIELD;
+
+/**
+ * Spark Lance file writer implementing {@link HoodieSparkFileWriter}.
+ *
+ * This writer integrates with Hudi's storage I/O layer and supports:
+ * - Hudi metadata field population
+ * - Record key tracking (for bloom filters - TODO https://github.com/apache/hudi/issues/17664)
+ * - Sequence ID generation
+ */
+public class HoodieSparkLanceWriter extends HoodieBaseLanceWriter<InternalRow> implements HoodieSparkFileWriter {
+
+ private static final long DEFAULT_MAX_FILE_SIZE = 120 * 1024 * 1024; // 120MB
Review Comment:
Let's make sure there is a task to make this configurable as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add HoodieBaseLanceFileWriter and implementation for SparkFileW… [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17629:
URL: https://github.com/apache/hudi/pull/17629#discussion_r2643368749
##########
hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/io/storage/TestHoodieSparkLanceWriter.java:
##########
@@ -0,0 +1,454 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.io.storage;
+
+import com.lancedb.lance.file.LanceFileReader;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.memory.RootAllocator;
+import org.apache.arrow.vector.BigIntVector;
+import org.apache.arrow.vector.BitVector;
+import org.apache.arrow.vector.Float4Vector;
+import org.apache.arrow.vector.Float8Vector;
+import org.apache.arrow.vector.IntVector;
+import org.apache.arrow.vector.VarBinaryVector;
+import org.apache.arrow.vector.VarCharVector;
+import org.apache.arrow.vector.VectorSchemaRoot;
+import org.apache.arrow.vector.ipc.ArrowReader;
+import org.apache.hudi.client.SparkTaskContextSupplier;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.testutils.HoodieTestUtils;
+import org.apache.hudi.storage.HoodieStorage;
+import org.apache.hudi.storage.StoragePath;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.catalyst.expressions.GenericInternalRow;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.unsafe.types.UTF8String;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.condition.DisabledIfSystemProperty;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.COMMIT_SEQNO_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.COMMIT_TIME_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.FILENAME_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.PARTITION_PATH_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.RECORD_KEY_METADATA_FIELD;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * Tests for {@link HoodieSparkLanceWriter}.
+ */
+@DisabledIfSystemProperty(named = "lance.skip.tests", matches = "true")
+public class TestHoodieSparkLanceWriter {
+
+ @TempDir
+ File tempDir;
+
+ private HoodieStorage storage;
+ private SparkTaskContextSupplier taskContextSupplier;
+ private String instantTime;
+
+ @BeforeEach
+ public void setUp() throws IOException {
+ storage = HoodieTestUtils.getStorage(tempDir.getAbsolutePath());
+ taskContextSupplier = new SparkTaskContextSupplier();
+ instantTime = "20251201120000000";
+ }
+
+ @AfterEach
+ public void tearDown() throws Exception {
+ if (storage != null) {
+ storage.close();
+ }
+ }
+
+ @Test
+ public void testWriteRowWithMetadataPopulation() throws Exception {
+ // Schema WITH meta fields (writer expects this when populateMetaFields=true)
+ StructType schema = createSchemaWithMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_with_metadata.lance");
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, true)) {
+ // Write multiple records to test metadata population and sequence ID generation
+ for (int i = 0; i < 3; i++) {
+ InternalRow row = createRowWithMetaFields(i, "User" + i, 20L + i);
+ HoodieKey key = new HoodieKey("key" + i, "partition1");
+ writer.writeRowWithMetadata(key, row);
+ }
+ }
+
+ // Verify using LanceFileReader
+ assertTrue(storage.exists(path), "Lance file should exist");
+
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator);
+ ArrowReader arrowReader = reader.readAll(null, null, Integer.MAX_VALUE)) {
+
+ assertEquals(3, reader.numRows(), "Should have 3 records");
+
+ // Schema should have 5 meta fields + 3 user fields = 8 fields
+ assertEquals(8, reader.schema().getFields().size(), "Should have 8 fields");
+
+ // Read and verify data
+ assertTrue(arrowReader.loadNextBatch(), "Should load batch");
+ VectorSchemaRoot root = arrowReader.getVectorSchemaRoot();
+
+ // Verify meta fields were populated for first record
+ VarCharVector commitTimeVector = (VarCharVector) root.getVector(COMMIT_TIME_METADATA_FIELD.getFieldName());
+ assertNotNull(commitTimeVector);
+ assertEquals(instantTime, new String(commitTimeVector.get(0)), "Commit time should match");
+
+ VarCharVector recordKeyVector = (VarCharVector) root.getVector(RECORD_KEY_METADATA_FIELD.getFieldName());
+ assertEquals("key0", new String(recordKeyVector.get(0)), "Record key should match");
+
+ VarCharVector partitionPathVector = (VarCharVector) root.getVector(PARTITION_PATH_METADATA_FIELD.getFieldName());
+ assertEquals("partition1", new String(partitionPathVector.get(0)), "Partition path should match");
+
+ VarCharVector fileNameVector = (VarCharVector) root.getVector(FILENAME_METADATA_FIELD.getFieldName());
+ assertEquals(path.getName(), new String(fileNameVector.get(0)), "File name should match");
+
+ // Verify sequence IDs are unique and properly formatted
+ VarCharVector seqNoVector = (VarCharVector) root.getVector(COMMIT_SEQNO_METADATA_FIELD.getFieldName());
+ List<String> seqNos = new ArrayList<>();
+ for (int i = 0; i < 3; i++) {
+ String seqNo = new String(seqNoVector.get(i));
+ assertTrue(seqNo.startsWith(instantTime + "_"), "Sequence number should start with instant time");
+ seqNos.add(seqNo);
+ }
+ assertEquals(3, seqNos.stream().distinct().count(), "All sequence IDs should be unique");
+
+ // Verify user data fields for first record
+ IntVector idVector = (IntVector) root.getVector("id");
+ assertEquals(0, idVector.get(0), "ID should match");
+
+ VarCharVector nameVector = (VarCharVector) root.getVector("name");
+ assertEquals("User0", new String(nameVector.get(0)), "Name should match");
+
+ BigIntVector ageVector = (BigIntVector) root.getVector("age");
+ assertEquals(20L, ageVector.get(0), "Age should match");
+ }
+ }
+
+ @Test
+ public void testWriteRowWithoutMetadataPopulation() throws Exception {
+ // Schema WITHOUT meta fields
+ StructType schema = createSchemaWithoutMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_without_metadata.lance");
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false)) {
+ // Create row with just user data (no meta fields)
+ InternalRow row = createRow(1, "Bob", 25L);
+ HoodieKey key = new HoodieKey("key2", "partition2");
+
+ writer.writeRowWithMetadata(key, row);
+ }
+
+ // Verify using LanceFileReader
+ assertTrue(storage.exists(path));
+
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator);
+ ArrowReader arrowReader = reader.readAll(null, null, Integer.MAX_VALUE)) {
+
+ assertEquals(1, reader.numRows());
+
+ // Schema should have ONLY 3 user fields (no meta fields)
+ assertEquals(3, reader.schema().getFields().size(), "Should have only user fields");
+
+ // Read and verify data
+ assertTrue(arrowReader.loadNextBatch());
+ VectorSchemaRoot root = arrowReader.getVectorSchemaRoot();
+
+ // Verify NO meta fields exist
+ assertFalse(hasField(root, COMMIT_TIME_METADATA_FIELD.getFieldName()), "Should not have commit time field");
+ assertFalse(hasField(root, RECORD_KEY_METADATA_FIELD.getFieldName()), "Should not have record key field");
+
+ // Verify user data
+ IntVector idVector = (IntVector) root.getVector("id");
+ assertEquals(1, idVector.get(0));
+
+ VarCharVector nameVector = (VarCharVector) root.getVector("name");
+ assertEquals("Bob", new String(nameVector.get(0)));
+ }
+ }
+
+ @Test
+ public void testWriteRowSimple() throws Exception {
+ StructType schema = createSchemaWithoutMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_simple_write.lance");
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false)) {
+ InternalRow row = createRow(1, "Charlie", 35L);
+ writer.writeRow("key3", row);
+ }
+
+ // Verify file exists and has correct record count
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator)) {
+ assertEquals(1, reader.numRows());
+ }
+ }
+
+ @Test
+ public void testBatchFlushing() throws Exception {
+ StructType schema = createSchemaWithoutMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_batch_flush.lance");
+ // Write more than DEFAULT_BATCH_SIZE (1000) records
+ int recordCount = 2500;
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false)) {
+ for (int i = 0; i < recordCount; i++) {
+ InternalRow row = createRow(i, "User" + i, 20L + i);
+ writer.writeRow("key" + i, row);
+ }
+ }
+
+ // Verify all records were written
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator)) {
+ assertEquals(recordCount, reader.numRows(), "All records should be written");
+ }
+ }
+
+ @Test
+ public void testPrimitiveTypes() throws Exception {
+ StructType schema = new StructType()
+ .add("int_field", DataTypes.IntegerType, false)
+ .add("long_field", DataTypes.LongType, false)
+ .add("float_field", DataTypes.FloatType, false)
+ .add("double_field", DataTypes.DoubleType, false)
+ .add("bool_field", DataTypes.BooleanType, false)
+ .add("string_field", DataTypes.StringType, false)
+ .add("binary_field", DataTypes.BinaryType, false);
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_primitives.lance");
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false)) {
+ GenericInternalRow row = new GenericInternalRow(new Object[]{
+ 42, // int
+ 123456789L, // long
+ 3.14f, // float
+ 2.71828, // double
+ true, // boolean
+ UTF8String.fromString("test"), // string
+ new byte[]{1, 2, 3, 4} // binary
+ });
+
+ writer.writeRow("key1", row);
+ }
+
+ // Verify all types were written correctly
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator);
+ ArrowReader arrowReader = reader.readAll(null, null, Integer.MAX_VALUE)) {
+
+ assertEquals(1, reader.numRows());
+ assertEquals(7, reader.schema().getFields().size());
+
+ arrowReader.loadNextBatch();
+ VectorSchemaRoot root = arrowReader.getVectorSchemaRoot();
+
+ assertEquals(42, ((IntVector) root.getVector("int_field")).get(0));
+ assertEquals(123456789L, ((BigIntVector) root.getVector("long_field")).get(0));
+ assertEquals(3.14f, ((Float4Vector) root.getVector("float_field")).get(0), 0.001);
+ assertEquals(2.71828, ((Float8Vector) root.getVector("double_field")).get(0), 0.00001);
+ assertEquals(1, ((BitVector) root.getVector("bool_field")).get(0));
+ assertEquals("test", new String(((VarCharVector) root.getVector("string_field")).get(0)));
+
+ byte[] binary = ((VarBinaryVector) root.getVector("binary_field")).get(0);
+ assertEquals(4, binary.length);
+ }
+ }
+
+ @Test
+ public void testNullValues() throws Exception {
+ StructType schema = new StructType()
+ .add("id", DataTypes.IntegerType, false)
+ .add("name", DataTypes.StringType, true)
+ .add("age", DataTypes.LongType, true);
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_nulls.lance");
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false)) {
+ // Write rows with null values
+ writer.writeRow("key1", createRow(1, "Alice", 30L));
+ writer.writeRow("key2", createRow(2, null, 25L)); // null name
+ writer.writeRow("key3", createRow(3, "Charlie", null)); // null age
+ }
+
+ // Verify nulls are preserved
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator);
+ ArrowReader arrowReader = reader.readAll(null, null, Integer.MAX_VALUE)) {
+
+ assertEquals(3, reader.numRows());
+
+ arrowReader.loadNextBatch();
+ VectorSchemaRoot root = arrowReader.getVectorSchemaRoot();
+
+ VarCharVector nameVector = (VarCharVector) root.getVector("name");
+ assertFalse(nameVector.isNull(0), "First name should not be null");
+ assertTrue(nameVector.isNull(1), "Second name should be null");
+ assertFalse(nameVector.isNull(2), "Third name should not be null");
+
+ BigIntVector ageVector = (BigIntVector) root.getVector("age");
+ assertFalse(ageVector.isNull(0), "First age should not be null");
+ assertFalse(ageVector.isNull(1), "Second age should not be null");
+ assertTrue(ageVector.isNull(2), "Third age should be null");
+ }
+ }
+
+ @Test
+ public void testWriteEmptyDataset() throws Exception {
+ StructType schema = new StructType()
+ .add("id", DataTypes.IntegerType, false);
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_empty.lance");
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false)) {
+ // Close without writing any rows
+ }
+
+ // Lance doesn't create a file if no data is written
+ assertFalse(storage.exists(path), "Lance file should not exist when no data is written");
Review Comment:
@the-other-tim-brown @yihua
Have pushed a change to address this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add HoodieBaseLanceFileWriter and implementation for SparkFileW… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17629:
URL: https://github.com/apache/hudi/pull/17629#issuecomment-3666089620
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9633e6bc476078ebf78b401e00ccbf43cdab67e8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10326",
"triggerID" : "9633e6bc476078ebf78b401e00ccbf43cdab67e8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9633e6bc476078ebf78b401e00ccbf43cdab67e8 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10326)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add HoodieBaseLanceFileWriter and implementation for SparkFileW… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17629:
URL: https://github.com/apache/hudi/pull/17629#issuecomment-3666071587
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9633e6bc476078ebf78b401e00ccbf43cdab67e8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9633e6bc476078ebf78b401e00ccbf43cdab67e8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9633e6bc476078ebf78b401e00ccbf43cdab67e8 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add HoodieBaseLanceFileWriter and implementation for SparkFileW… [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #17629:
URL: https://github.com/apache/hudi/pull/17629#discussion_r2641194296
##########
hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/io/storage/TestHoodieSparkLanceWriter.java:
##########
@@ -0,0 +1,454 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.io.storage;
+
+import com.lancedb.lance.file.LanceFileReader;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.memory.RootAllocator;
+import org.apache.arrow.vector.BigIntVector;
+import org.apache.arrow.vector.BitVector;
+import org.apache.arrow.vector.Float4Vector;
+import org.apache.arrow.vector.Float8Vector;
+import org.apache.arrow.vector.IntVector;
+import org.apache.arrow.vector.VarBinaryVector;
+import org.apache.arrow.vector.VarCharVector;
+import org.apache.arrow.vector.VectorSchemaRoot;
+import org.apache.arrow.vector.ipc.ArrowReader;
+import org.apache.hudi.client.SparkTaskContextSupplier;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.testutils.HoodieTestUtils;
+import org.apache.hudi.storage.HoodieStorage;
+import org.apache.hudi.storage.StoragePath;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.catalyst.expressions.GenericInternalRow;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.unsafe.types.UTF8String;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.condition.DisabledIfSystemProperty;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.COMMIT_SEQNO_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.COMMIT_TIME_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.FILENAME_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.PARTITION_PATH_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.RECORD_KEY_METADATA_FIELD;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * Tests for {@link HoodieSparkLanceWriter}.
+ */
+@DisabledIfSystemProperty(named = "lance.skip.tests", matches = "true")
+public class TestHoodieSparkLanceWriter {
+
+ @TempDir
+ File tempDir;
+
+ private HoodieStorage storage;
+ private SparkTaskContextSupplier taskContextSupplier;
+ private String instantTime;
+
+ @BeforeEach
+ public void setUp() throws IOException {
+ storage = HoodieTestUtils.getStorage(tempDir.getAbsolutePath());
+ taskContextSupplier = new SparkTaskContextSupplier();
+ instantTime = "20251201120000000";
+ }
+
+ @AfterEach
+ public void tearDown() throws Exception {
+ if (storage != null) {
+ storage.close();
+ }
+ }
+
+ @Test
+ public void testWriteRowWithMetadataPopulation() throws Exception {
+ // Schema WITH meta fields (writer expects this when populateMetaFields=true)
+ StructType schema = createSchemaWithMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_with_metadata.lance");
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, true)) {
+ // Write multiple records to test metadata population and sequence ID generation
+ for (int i = 0; i < 3; i++) {
+ InternalRow row = createRowWithMetaFields(i, "User" + i, 20L + i);
+ HoodieKey key = new HoodieKey("key" + i, "partition1");
+ writer.writeRowWithMetadata(key, row);
+ }
+ }
+
+ // Verify using LanceFileReader
+ assertTrue(storage.exists(path), "Lance file should exist");
+
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator);
+ ArrowReader arrowReader = reader.readAll(null, null, Integer.MAX_VALUE)) {
+
+ assertEquals(3, reader.numRows(), "Should have 3 records");
+
+ // Schema should have 5 meta fields + 3 user fields = 8 fields
+ assertEquals(8, reader.schema().getFields().size(), "Should have 8 fields");
+
+ // Read and verify data
+ assertTrue(arrowReader.loadNextBatch(), "Should load batch");
+ VectorSchemaRoot root = arrowReader.getVectorSchemaRoot();
+
+ // Verify meta fields were populated for first record
+ VarCharVector commitTimeVector = (VarCharVector) root.getVector(COMMIT_TIME_METADATA_FIELD.getFieldName());
+ assertNotNull(commitTimeVector);
+ assertEquals(instantTime, new String(commitTimeVector.get(0)), "Commit time should match");
+
+ VarCharVector recordKeyVector = (VarCharVector) root.getVector(RECORD_KEY_METADATA_FIELD.getFieldName());
+ assertEquals("key0", new String(recordKeyVector.get(0)), "Record key should match");
+
+ VarCharVector partitionPathVector = (VarCharVector) root.getVector(PARTITION_PATH_METADATA_FIELD.getFieldName());
+ assertEquals("partition1", new String(partitionPathVector.get(0)), "Partition path should match");
+
+ VarCharVector fileNameVector = (VarCharVector) root.getVector(FILENAME_METADATA_FIELD.getFieldName());
+ assertEquals(path.getName(), new String(fileNameVector.get(0)), "File name should match");
+
+ // Verify sequence IDs are unique and properly formatted
+ VarCharVector seqNoVector = (VarCharVector) root.getVector(COMMIT_SEQNO_METADATA_FIELD.getFieldName());
+ List<String> seqNos = new ArrayList<>();
+ for (int i = 0; i < 3; i++) {
+ String seqNo = new String(seqNoVector.get(i));
+ assertTrue(seqNo.startsWith(instantTime + "_"), "Sequence number should start with instant time");
+ seqNos.add(seqNo);
+ }
+ assertEquals(3, seqNos.stream().distinct().count(), "All sequence IDs should be unique");
+
+ // Verify user data fields for first record
+ IntVector idVector = (IntVector) root.getVector("id");
+ assertEquals(0, idVector.get(0), "ID should match");
+
+ VarCharVector nameVector = (VarCharVector) root.getVector("name");
+ assertEquals("User0", new String(nameVector.get(0)), "Name should match");
+
+ BigIntVector ageVector = (BigIntVector) root.getVector("age");
+ assertEquals(20L, ageVector.get(0), "Age should match");
+ }
+ }
+
+ @Test
+ public void testWriteRowWithoutMetadataPopulation() throws Exception {
+ // Schema WITHOUT meta fields
+ StructType schema = createSchemaWithoutMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_without_metadata.lance");
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false)) {
+ // Create row with just user data (no meta fields)
+ InternalRow row = createRow(1, "Bob", 25L);
+ HoodieKey key = new HoodieKey("key2", "partition2");
+
+ writer.writeRowWithMetadata(key, row);
+ }
+
+ // Verify using LanceFileReader
+ assertTrue(storage.exists(path));
+
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator);
+ ArrowReader arrowReader = reader.readAll(null, null, Integer.MAX_VALUE)) {
+
+ assertEquals(1, reader.numRows());
+
+ // Schema should have ONLY 3 user fields (no meta fields)
+ assertEquals(3, reader.schema().getFields().size(), "Should have only user fields");
+
+ // Read and verify data
+ assertTrue(arrowReader.loadNextBatch());
+ VectorSchemaRoot root = arrowReader.getVectorSchemaRoot();
+
+ // Verify NO meta fields exist
+ assertFalse(hasField(root, COMMIT_TIME_METADATA_FIELD.getFieldName()), "Should not have commit time field");
+ assertFalse(hasField(root, RECORD_KEY_METADATA_FIELD.getFieldName()), "Should not have record key field");
+
+ // Verify user data
+ IntVector idVector = (IntVector) root.getVector("id");
+ assertEquals(1, idVector.get(0));
+
+ VarCharVector nameVector = (VarCharVector) root.getVector("name");
+ assertEquals("Bob", new String(nameVector.get(0)));
+ }
+ }
+
+ @Test
+ public void testWriteRowSimple() throws Exception {
+ StructType schema = createSchemaWithoutMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_simple_write.lance");
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false)) {
+ InternalRow row = createRow(1, "Charlie", 35L);
+ writer.writeRow("key3", row);
+ }
+
+ // Verify file exists and has correct record count
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator)) {
+ assertEquals(1, reader.numRows());
+ }
+ }
+
+ @Test
+ public void testBatchFlushing() throws Exception {
+ StructType schema = createSchemaWithoutMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_batch_flush.lance");
+ // Write more than DEFAULT_BATCH_SIZE (1000) records
+ int recordCount = 2500;
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false)) {
+ for (int i = 0; i < recordCount; i++) {
+ InternalRow row = createRow(i, "User" + i, 20L + i);
+ writer.writeRow("key" + i, row);
+ }
+ }
+
+ // Verify all records were written
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator)) {
+ assertEquals(recordCount, reader.numRows(), "All records should be written");
+ }
+ }
+
+ @Test
+ public void testPrimitiveTypes() throws Exception {
+ StructType schema = new StructType()
+ .add("int_field", DataTypes.IntegerType, false)
+ .add("long_field", DataTypes.LongType, false)
+ .add("float_field", DataTypes.FloatType, false)
+ .add("double_field", DataTypes.DoubleType, false)
+ .add("bool_field", DataTypes.BooleanType, false)
+ .add("string_field", DataTypes.StringType, false)
+ .add("binary_field", DataTypes.BinaryType, false);
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_primitives.lance");
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false)) {
+ GenericInternalRow row = new GenericInternalRow(new Object[]{
+ 42, // int
+ 123456789L, // long
+ 3.14f, // float
+ 2.71828, // double
+ true, // boolean
+ UTF8String.fromString("test"), // string
+ new byte[]{1, 2, 3, 4} // binary
+ });
+
+ writer.writeRow("key1", row);
+ }
+
+ // Verify all types were written correctly
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator);
+ ArrowReader arrowReader = reader.readAll(null, null, Integer.MAX_VALUE)) {
+
+ assertEquals(1, reader.numRows());
+ assertEquals(7, reader.schema().getFields().size());
+
+ arrowReader.loadNextBatch();
+ VectorSchemaRoot root = arrowReader.getVectorSchemaRoot();
+
+ assertEquals(42, ((IntVector) root.getVector("int_field")).get(0));
+ assertEquals(123456789L, ((BigIntVector) root.getVector("long_field")).get(0));
+ assertEquals(3.14f, ((Float4Vector) root.getVector("float_field")).get(0), 0.001);
+ assertEquals(2.71828, ((Float8Vector) root.getVector("double_field")).get(0), 0.00001);
+ assertEquals(1, ((BitVector) root.getVector("bool_field")).get(0));
+ assertEquals("test", new String(((VarCharVector) root.getVector("string_field")).get(0)));
+
+ byte[] binary = ((VarBinaryVector) root.getVector("binary_field")).get(0);
+ assertEquals(4, binary.length);
+ }
+ }
+
+ @Test
+ public void testNullValues() throws Exception {
+ StructType schema = new StructType()
+ .add("id", DataTypes.IntegerType, false)
+ .add("name", DataTypes.StringType, true)
+ .add("age", DataTypes.LongType, true);
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_nulls.lance");
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false)) {
+ // Write rows with null values
+ writer.writeRow("key1", createRow(1, "Alice", 30L));
+ writer.writeRow("key2", createRow(2, null, 25L)); // null name
+ writer.writeRow("key3", createRow(3, "Charlie", null)); // null age
+ }
+
+ // Verify nulls are preserved
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator);
+ ArrowReader arrowReader = reader.readAll(null, null, Integer.MAX_VALUE)) {
+
+ assertEquals(3, reader.numRows());
+
+ arrowReader.loadNextBatch();
+ VectorSchemaRoot root = arrowReader.getVectorSchemaRoot();
+
+ VarCharVector nameVector = (VarCharVector) root.getVector("name");
+ assertFalse(nameVector.isNull(0), "First name should not be null");
+ assertTrue(nameVector.isNull(1), "Second name should be null");
+ assertFalse(nameVector.isNull(2), "Third name should not be null");
+
+ BigIntVector ageVector = (BigIntVector) root.getVector("age");
+ assertFalse(ageVector.isNull(0), "First age should not be null");
+ assertFalse(ageVector.isNull(1), "Second age should not be null");
+ assertTrue(ageVector.isNull(2), "Third age should be null");
+ }
+ }
+
+ @Test
+ public void testWriteEmptyDataset() throws Exception {
+ StructType schema = new StructType()
+ .add("id", DataTypes.IntegerType, false);
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_empty.lance");
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false)) {
+ // Close without writing any rows
+ }
+
+ // Lance doesn't create a file if no data is written
+ assertFalse(storage.exists(path), "Lance file should not exist when no data is written");
Review Comment:
Hudi Parquet writer should still write a file if there is no row in the file group as I remember. @rahil-c could you check if that's case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add HoodieBaseLanceFileWriter and implementation for SparkFileW… [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17629:
URL: https://github.com/apache/hudi/pull/17629#discussion_r2641726793
##########
hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/io/storage/TestHoodieSparkLanceWriter.java:
##########
@@ -0,0 +1,454 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.io.storage;
+
+import com.lancedb.lance.file.LanceFileReader;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.memory.RootAllocator;
+import org.apache.arrow.vector.BigIntVector;
+import org.apache.arrow.vector.BitVector;
+import org.apache.arrow.vector.Float4Vector;
+import org.apache.arrow.vector.Float8Vector;
+import org.apache.arrow.vector.IntVector;
+import org.apache.arrow.vector.VarBinaryVector;
+import org.apache.arrow.vector.VarCharVector;
+import org.apache.arrow.vector.VectorSchemaRoot;
+import org.apache.arrow.vector.ipc.ArrowReader;
+import org.apache.hudi.client.SparkTaskContextSupplier;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.testutils.HoodieTestUtils;
+import org.apache.hudi.storage.HoodieStorage;
+import org.apache.hudi.storage.StoragePath;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.catalyst.expressions.GenericInternalRow;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.unsafe.types.UTF8String;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.condition.DisabledIfSystemProperty;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.COMMIT_SEQNO_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.COMMIT_TIME_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.FILENAME_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.PARTITION_PATH_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.RECORD_KEY_METADATA_FIELD;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * Tests for {@link HoodieSparkLanceWriter}.
+ */
+@DisabledIfSystemProperty(named = "lance.skip.tests", matches = "true")
+public class TestHoodieSparkLanceWriter {
+
+ @TempDir
+ File tempDir;
+
+ private HoodieStorage storage;
+ private SparkTaskContextSupplier taskContextSupplier;
+ private String instantTime;
+
+ @BeforeEach
+ public void setUp() throws IOException {
+ storage = HoodieTestUtils.getStorage(tempDir.getAbsolutePath());
+ taskContextSupplier = new SparkTaskContextSupplier();
+ instantTime = "20251201120000000";
+ }
+
+ @AfterEach
+ public void tearDown() throws Exception {
+ if (storage != null) {
+ storage.close();
+ }
+ }
+
+ @Test
+ public void testWriteRowWithMetadataPopulation() throws Exception {
+ // Schema WITH meta fields (writer expects this when populateMetaFields=true)
+ StructType schema = createSchemaWithMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_with_metadata.lance");
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, true)) {
+ // Write multiple records to test metadata population and sequence ID generation
+ for (int i = 0; i < 3; i++) {
+ InternalRow row = createRowWithMetaFields(i, "User" + i, 20L + i);
+ HoodieKey key = new HoodieKey("key" + i, "partition1");
+ writer.writeRowWithMetadata(key, row);
+ }
+ }
+
+ // Verify using LanceFileReader
+ assertTrue(storage.exists(path), "Lance file should exist");
+
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator);
+ ArrowReader arrowReader = reader.readAll(null, null, Integer.MAX_VALUE)) {
+
+ assertEquals(3, reader.numRows(), "Should have 3 records");
+
+ // Schema should have 5 meta fields + 3 user fields = 8 fields
+ assertEquals(8, reader.schema().getFields().size(), "Should have 8 fields");
+
+ // Read and verify data
+ assertTrue(arrowReader.loadNextBatch(), "Should load batch");
+ VectorSchemaRoot root = arrowReader.getVectorSchemaRoot();
+
+ // Verify meta fields were populated for first record
+ VarCharVector commitTimeVector = (VarCharVector) root.getVector(COMMIT_TIME_METADATA_FIELD.getFieldName());
+ assertNotNull(commitTimeVector);
+ assertEquals(instantTime, new String(commitTimeVector.get(0)), "Commit time should match");
+
+ VarCharVector recordKeyVector = (VarCharVector) root.getVector(RECORD_KEY_METADATA_FIELD.getFieldName());
+ assertEquals("key0", new String(recordKeyVector.get(0)), "Record key should match");
+
+ VarCharVector partitionPathVector = (VarCharVector) root.getVector(PARTITION_PATH_METADATA_FIELD.getFieldName());
+ assertEquals("partition1", new String(partitionPathVector.get(0)), "Partition path should match");
+
+ VarCharVector fileNameVector = (VarCharVector) root.getVector(FILENAME_METADATA_FIELD.getFieldName());
+ assertEquals(path.getName(), new String(fileNameVector.get(0)), "File name should match");
+
+ // Verify sequence IDs are unique and properly formatted
+ VarCharVector seqNoVector = (VarCharVector) root.getVector(COMMIT_SEQNO_METADATA_FIELD.getFieldName());
+ List<String> seqNos = new ArrayList<>();
+ for (int i = 0; i < 3; i++) {
+ String seqNo = new String(seqNoVector.get(i));
+ assertTrue(seqNo.startsWith(instantTime + "_"), "Sequence number should start with instant time");
+ seqNos.add(seqNo);
+ }
+ assertEquals(3, seqNos.stream().distinct().count(), "All sequence IDs should be unique");
+
+ // Verify user data fields for first record
+ IntVector idVector = (IntVector) root.getVector("id");
+ assertEquals(0, idVector.get(0), "ID should match");
+
+ VarCharVector nameVector = (VarCharVector) root.getVector("name");
+ assertEquals("User0", new String(nameVector.get(0)), "Name should match");
+
+ BigIntVector ageVector = (BigIntVector) root.getVector("age");
+ assertEquals(20L, ageVector.get(0), "Age should match");
+ }
+ }
+
+ @Test
+ public void testWriteRowWithoutMetadataPopulation() throws Exception {
+ // Schema WITHOUT meta fields
+ StructType schema = createSchemaWithoutMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_without_metadata.lance");
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false)) {
+ // Create row with just user data (no meta fields)
+ InternalRow row = createRow(1, "Bob", 25L);
+ HoodieKey key = new HoodieKey("key2", "partition2");
+
+ writer.writeRowWithMetadata(key, row);
+ }
+
+ // Verify using LanceFileReader
+ assertTrue(storage.exists(path));
+
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator);
+ ArrowReader arrowReader = reader.readAll(null, null, Integer.MAX_VALUE)) {
+
+ assertEquals(1, reader.numRows());
+
+ // Schema should have ONLY 3 user fields (no meta fields)
+ assertEquals(3, reader.schema().getFields().size(), "Should have only user fields");
+
+ // Read and verify data
+ assertTrue(arrowReader.loadNextBatch());
+ VectorSchemaRoot root = arrowReader.getVectorSchemaRoot();
+
+ // Verify NO meta fields exist
+ assertFalse(hasField(root, COMMIT_TIME_METADATA_FIELD.getFieldName()), "Should not have commit time field");
+ assertFalse(hasField(root, RECORD_KEY_METADATA_FIELD.getFieldName()), "Should not have record key field");
+
+ // Verify user data
+ IntVector idVector = (IntVector) root.getVector("id");
+ assertEquals(1, idVector.get(0));
+
+ VarCharVector nameVector = (VarCharVector) root.getVector("name");
+ assertEquals("Bob", new String(nameVector.get(0)));
+ }
+ }
+
+ @Test
+ public void testWriteRowSimple() throws Exception {
+ StructType schema = createSchemaWithoutMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_simple_write.lance");
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false)) {
+ InternalRow row = createRow(1, "Charlie", 35L);
+ writer.writeRow("key3", row);
+ }
+
+ // Verify file exists and has correct record count
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator)) {
+ assertEquals(1, reader.numRows());
+ }
+ }
+
+ @Test
+ public void testBatchFlushing() throws Exception {
+ StructType schema = createSchemaWithoutMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_batch_flush.lance");
+ // Write more than DEFAULT_BATCH_SIZE (1000) records
+ int recordCount = 2500;
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false)) {
+ for (int i = 0; i < recordCount; i++) {
+ InternalRow row = createRow(i, "User" + i, 20L + i);
+ writer.writeRow("key" + i, row);
+ }
+ }
+
+ // Verify all records were written
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator)) {
+ assertEquals(recordCount, reader.numRows(), "All records should be written");
+ }
+ }
+
+ @Test
+ public void testPrimitiveTypes() throws Exception {
+ StructType schema = new StructType()
+ .add("int_field", DataTypes.IntegerType, false)
+ .add("long_field", DataTypes.LongType, false)
+ .add("float_field", DataTypes.FloatType, false)
+ .add("double_field", DataTypes.DoubleType, false)
+ .add("bool_field", DataTypes.BooleanType, false)
+ .add("string_field", DataTypes.StringType, false)
+ .add("binary_field", DataTypes.BinaryType, false);
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_primitives.lance");
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false)) {
+ GenericInternalRow row = new GenericInternalRow(new Object[]{
+ 42, // int
+ 123456789L, // long
+ 3.14f, // float
+ 2.71828, // double
+ true, // boolean
+ UTF8String.fromString("test"), // string
+ new byte[]{1, 2, 3, 4} // binary
+ });
+
+ writer.writeRow("key1", row);
+ }
+
+ // Verify all types were written correctly
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator);
+ ArrowReader arrowReader = reader.readAll(null, null, Integer.MAX_VALUE)) {
+
+ assertEquals(1, reader.numRows());
+ assertEquals(7, reader.schema().getFields().size());
+
+ arrowReader.loadNextBatch();
+ VectorSchemaRoot root = arrowReader.getVectorSchemaRoot();
+
+ assertEquals(42, ((IntVector) root.getVector("int_field")).get(0));
+ assertEquals(123456789L, ((BigIntVector) root.getVector("long_field")).get(0));
+ assertEquals(3.14f, ((Float4Vector) root.getVector("float_field")).get(0), 0.001);
+ assertEquals(2.71828, ((Float8Vector) root.getVector("double_field")).get(0), 0.00001);
+ assertEquals(1, ((BitVector) root.getVector("bool_field")).get(0));
+ assertEquals("test", new String(((VarCharVector) root.getVector("string_field")).get(0)));
+
+ byte[] binary = ((VarBinaryVector) root.getVector("binary_field")).get(0);
+ assertEquals(4, binary.length);
+ }
+ }
+
+ @Test
+ public void testNullValues() throws Exception {
+ StructType schema = new StructType()
+ .add("id", DataTypes.IntegerType, false)
+ .add("name", DataTypes.StringType, true)
+ .add("age", DataTypes.LongType, true);
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_nulls.lance");
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false)) {
+ // Write rows with null values
+ writer.writeRow("key1", createRow(1, "Alice", 30L));
+ writer.writeRow("key2", createRow(2, null, 25L)); // null name
+ writer.writeRow("key3", createRow(3, "Charlie", null)); // null age
+ }
+
+ // Verify nulls are preserved
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator);
+ ArrowReader arrowReader = reader.readAll(null, null, Integer.MAX_VALUE)) {
+
+ assertEquals(3, reader.numRows());
+
+ arrowReader.loadNextBatch();
+ VectorSchemaRoot root = arrowReader.getVectorSchemaRoot();
+
+ VarCharVector nameVector = (VarCharVector) root.getVector("name");
+ assertFalse(nameVector.isNull(0), "First name should not be null");
+ assertTrue(nameVector.isNull(1), "Second name should be null");
+ assertFalse(nameVector.isNull(2), "Third name should not be null");
+
+ BigIntVector ageVector = (BigIntVector) root.getVector("age");
+ assertFalse(ageVector.isNull(0), "First age should not be null");
+ assertFalse(ageVector.isNull(1), "Second age should not be null");
+ assertTrue(ageVector.isNull(2), "Third age should be null");
+ }
+ }
+
+ @Test
+ public void testWriteEmptyDataset() throws Exception {
+ StructType schema = new StructType()
+ .add("id", DataTypes.IntegerType, false);
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_empty.lance");
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false)) {
+ // Close without writing any rows
+ }
+
+ // Lance doesn't create a file if no data is written
+ assertFalse(storage.exists(path), "Lance file should not exist when no data is written");
Review Comment:
@the-other-tim-brown @yihua When looking at the Hudi base parquet writer constructor, I noticed in the underlying parquet writer's `build` https://github.com/apache/hudi/blob/master/hudi-hadoop-common/src/main/java/org/apache/hudi/io/hadoop/HoodieBaseParquetWriter.java#L82 that it has the following code
```
ParquetFileWriter fileWriter = new ParquetFileWriter(
file, schema, mode, rowGroupSize, maxPaddingSize,
encodingProps.getColumnIndexTruncateLength(), encodingProps.getStatisticsTruncateLength(),
encodingProps.getPageWriteChecksumEnabled(), encryptionProperties);
fileWriter.start();
```
So i am not sure if the default behavior of parquet files is that it will create an empty file on initialization of the constructor, regardless if there is any actual data to write?
```
/**
* start the file
* @throws IOException if there is an error while writing
*/
public void start() throws IOException {
state = state.start();
LOG.debug("{}: start", out.getPos());
byte[] magic = MAGIC;
if (null != fileEncryptor && fileEncryptor.isFooterEncrypted()) {
magic = EFMAGIC;
}
out.write(magic);
}
```
https://github.com/apache/parquet-java/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileWriter.java#L596
Unless you guys were referring to another code path in hudi that does this empty file creation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add HoodieBaseLanceFileWriter and implementation for SparkFileW… [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17629:
URL: https://github.com/apache/hudi/pull/17629#discussion_r2641907939
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/HoodieSparkLanceWriter.java:
##########
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.io.storage;
+
+import com.lancedb.lance.spark.arrow.LanceArrowWriter;
+import org.apache.arrow.vector.types.pojo.Schema;
+import org.apache.hudi.common.engine.TaskContextSupplier;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.io.lance.HoodieBaseLanceWriter;
+import org.apache.hudi.storage.HoodieStorage;
+import org.apache.hudi.storage.StoragePath;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.util.LanceArrowUtils;
+import org.apache.spark.unsafe.types.UTF8String;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.function.Function;
+
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.COMMIT_SEQNO_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.COMMIT_TIME_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.FILENAME_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.PARTITION_PATH_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.RECORD_KEY_METADATA_FIELD;
+
+/**
+ * Spark Lance file writer implementing {@link HoodieSparkFileWriter}.
+ *
+ * This writer integrates with Hudi's storage I/O layer and supports:
+ * - Hudi metadata field population
+ * - Record key tracking (for bloom filters - TODO https://github.com/apache/hudi/issues/17664)
+ * - Sequence ID generation
+ */
+public class HoodieSparkLanceWriter extends HoodieBaseLanceWriter<InternalRow> implements HoodieSparkFileWriter {
+
+ private static final long DEFAULT_MAX_FILE_SIZE = 120 * 1024 * 1024; // 120MB
Review Comment:
will make a task, and link it in the canWrite(), since this max file size was originally planned to be used there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add HoodieBaseLanceFileWriter and implementation for SparkFileW… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17629:
URL: https://github.com/apache/hudi/pull/17629#issuecomment-3667578542
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9633e6bc476078ebf78b401e00ccbf43cdab67e8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10326",
"triggerID" : "9633e6bc476078ebf78b401e00ccbf43cdab67e8",
"triggerType" : "PUSH"
}, {
"hash" : "6adcb20a0c841eb4164570bed7e3daa28349be35",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6adcb20a0c841eb4164570bed7e3daa28349be35",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9633e6bc476078ebf78b401e00ccbf43cdab67e8 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10326)
* 6adcb20a0c841eb4164570bed7e3daa28349be35 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add HoodieBaseLanceFileWriter and implementation for SparkFileW… [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #17629:
URL: https://github.com/apache/hudi/pull/17629#discussion_r2638157209
##########
hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/io/storage/TestHoodieSparkLanceWriter.java:
##########
@@ -0,0 +1,503 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.io.storage;
+
+import com.lancedb.lance.file.LanceFileReader;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.memory.RootAllocator;
+import org.apache.arrow.vector.BigIntVector;
+import org.apache.arrow.vector.BitVector;
+import org.apache.arrow.vector.Float4Vector;
+import org.apache.arrow.vector.Float8Vector;
+import org.apache.arrow.vector.IntVector;
+import org.apache.arrow.vector.VarBinaryVector;
+import org.apache.arrow.vector.VarCharVector;
+import org.apache.arrow.vector.VectorSchemaRoot;
+import org.apache.arrow.vector.ipc.ArrowReader;
+import org.apache.hudi.client.SparkTaskContextSupplier;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.testutils.HoodieTestUtils;
+import org.apache.hudi.storage.HoodieStorage;
+import org.apache.hudi.storage.StoragePath;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.catalyst.expressions.GenericInternalRow;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.unsafe.types.UTF8String;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.condition.DisabledIfSystemProperty;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.COMMIT_SEQNO_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.COMMIT_TIME_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.FILENAME_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.PARTITION_PATH_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.RECORD_KEY_METADATA_FIELD;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * Tests for {@link HoodieSparkLanceWriter}.
+ */
+@DisabledIfSystemProperty(named = "lance.skip.tests", matches = "true")
+public class TestHoodieSparkLanceWriter {
+
+ @TempDir
+ File tempDir;
+
+ private HoodieStorage storage;
+ private SparkTaskContextSupplier taskContextSupplier;
+ private String instantTime;
+
+ @BeforeEach
+ public void setUp() throws IOException {
+ storage = HoodieTestUtils.getStorage(tempDir.getAbsolutePath());
+ taskContextSupplier = new SparkTaskContextSupplier();
+ instantTime = "20251201120000000";
+ }
+
+ @AfterEach
+ public void tearDown() throws Exception {
+ if (storage != null) {
+ storage.close();
+ }
+ }
+
+ @Test
+ public void testWriteRowWithMetadataPopulation() throws Exception {
+ // Schema WITH meta fields (writer expects this when populateMetaFields=true)
+ StructType schema = createSchemaWithMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_with_metadata.lance");
+ HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, true);
+
+ // Create row with PLACEHOLDER meta fields + user data
+ InternalRow row = createRowWithMetaFields(1, "Alice", 30L);
+ HoodieKey key = new HoodieKey("key1", "partition1");
+
+ writer.writeRowWithMetadata(key, row);
+ writer.close();
+
+ // Verify using LanceFileReader
+ assertTrue(storage.exists(path), "Lance file should exist");
+
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator)) {
+
+ assertEquals(1, reader.numRows(), "Should have 1 record");
+
+ // Schema should have 5 meta fields + 3 user fields = 8 fields
+ assertEquals(8, reader.schema().getFields().size(), "Should have 8 fields");
+
+ // Read and verify data
+ ArrowReader arrowReader = reader.readAll(null, null, Integer.MAX_VALUE);
+ assertTrue(arrowReader.loadNextBatch(), "Should load batch");
+ VectorSchemaRoot root = arrowReader.getVectorSchemaRoot();
+
+ // Verify meta fields were populated
+ VarCharVector commitTimeVector = (VarCharVector) root.getVector(COMMIT_TIME_METADATA_FIELD.getFieldName());
+ assertNotNull(commitTimeVector);
+ assertEquals(instantTime, new String(commitTimeVector.get(0)), "Commit time should match");
+
+ VarCharVector recordKeyVector = (VarCharVector) root.getVector(RECORD_KEY_METADATA_FIELD.getFieldName());
+ assertEquals("key1", new String(recordKeyVector.get(0)), "Record key should match");
+
+ VarCharVector partitionPathVector = (VarCharVector) root.getVector(PARTITION_PATH_METADATA_FIELD.getFieldName());
+ assertEquals("partition1", new String(partitionPathVector.get(0)), "Partition path should match");
+
+ VarCharVector fileNameVector = (VarCharVector) root.getVector(FILENAME_METADATA_FIELD.getFieldName());
+ assertEquals(path.getName(), new String(fileNameVector.get(0)), "File name should match");
+
+ VarCharVector seqNoVector = (VarCharVector) root.getVector(COMMIT_SEQNO_METADATA_FIELD.getFieldName());
+ String seqNo = new String(seqNoVector.get(0));
+ assertTrue(seqNo.startsWith(instantTime), "Sequence number should start with instant time");
+
+ // Verify user data fields
+ IntVector idVector = (IntVector) root.getVector("id");
+ assertEquals(1, idVector.get(0), "ID should match");
+
+ VarCharVector nameVector = (VarCharVector) root.getVector("name");
+ assertEquals("Alice", new String(nameVector.get(0)), "Name should match");
+
+ BigIntVector ageVector = (BigIntVector) root.getVector("age");
+ assertEquals(30L, ageVector.get(0), "Age should match");
+
+ arrowReader.close();
+ }
+ }
+
+ @Test
+ public void testWriteRowWithoutMetadataPopulation() throws Exception {
+ // Schema WITHOUT meta fields
+ StructType schema = createSchemaWithoutMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_without_metadata.lance");
+ HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false);
+
+ // Create row with just user data (no meta fields)
+ InternalRow row = createRow(1, "Bob", 25L);
+ HoodieKey key = new HoodieKey("key2", "partition2");
+
+ writer.writeRowWithMetadata(key, row);
+ writer.close();
+
+ // Verify using LanceFileReader
+ assertTrue(storage.exists(path));
+
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator)) {
+
+ assertEquals(1, reader.numRows());
+
+ // Schema should have ONLY 3 user fields (no meta fields)
+ assertEquals(3, reader.schema().getFields().size(), "Should have only user fields");
+
+ // Read and verify data
+ ArrowReader arrowReader = reader.readAll(null, null, Integer.MAX_VALUE);
+ assertTrue(arrowReader.loadNextBatch());
+ VectorSchemaRoot root = arrowReader.getVectorSchemaRoot();
+
+ // Verify NO meta fields exist
+ assertFalse(hasField(root, COMMIT_TIME_METADATA_FIELD.getFieldName()), "Should not have commit time field");
+ assertFalse(hasField(root, RECORD_KEY_METADATA_FIELD.getFieldName()), "Should not have record key field");
+
+ // Verify user data
+ IntVector idVector = (IntVector) root.getVector("id");
+ assertEquals(1, idVector.get(0));
+
+ VarCharVector nameVector = (VarCharVector) root.getVector("name");
+ assertEquals("Bob", new String(nameVector.get(0)));
+
+ arrowReader.close();
+ }
+ }
+
+ @Test
+ public void testWriteRowSimple() throws Exception {
+ StructType schema = createSchemaWithoutMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_simple_write.lance");
+ HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false);
+
+ InternalRow row = createRow(1, "Charlie", 35L);
+ writer.writeRow("key3", row);
+ writer.close();
+
+ // Verify file exists and has correct record count
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator)) {
+ assertEquals(1, reader.numRows());
+ }
+ }
+
+ @Test
+ public void testSequenceIdGeneration() throws Exception {
+ StructType schema = createSchemaWithMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_sequence_id.lance");
+ HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, true);
+
+ // Write multiple records
+ for (int i = 0; i < 3; i++) {
+ InternalRow row = createRowWithMetaFields(i, "User" + i, 20L + i);
+ HoodieKey key = new HoodieKey("key" + i, "partition1");
+ writer.writeRowWithMetadata(key, row);
+ }
+ writer.close();
+
+ // Verify sequence IDs are unique and properly formatted
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator)) {
+
+ assertEquals(3, reader.numRows());
+
+ ArrowReader arrowReader = reader.readAll(null, null, Integer.MAX_VALUE);
+ arrowReader.loadNextBatch();
+ VectorSchemaRoot root = arrowReader.getVectorSchemaRoot();
+
+ VarCharVector seqNoVector = (VarCharVector) root.getVector(COMMIT_SEQNO_METADATA_FIELD.getFieldName());
Review Comment:
Then in that case, do we really need a separate test for this or should we combine?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add HoodieBaseLanceFileWriter and implementation for SparkFileW… [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #17629:
URL: https://github.com/apache/hudi/pull/17629#discussion_r2638158059
##########
hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/io/storage/TestHoodieSparkLanceWriter.java:
##########
@@ -0,0 +1,503 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.io.storage;
+
+import com.lancedb.lance.file.LanceFileReader;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.memory.RootAllocator;
+import org.apache.arrow.vector.BigIntVector;
+import org.apache.arrow.vector.BitVector;
+import org.apache.arrow.vector.Float4Vector;
+import org.apache.arrow.vector.Float8Vector;
+import org.apache.arrow.vector.IntVector;
+import org.apache.arrow.vector.VarBinaryVector;
+import org.apache.arrow.vector.VarCharVector;
+import org.apache.arrow.vector.VectorSchemaRoot;
+import org.apache.arrow.vector.ipc.ArrowReader;
+import org.apache.hudi.client.SparkTaskContextSupplier;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.testutils.HoodieTestUtils;
+import org.apache.hudi.storage.HoodieStorage;
+import org.apache.hudi.storage.StoragePath;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.catalyst.expressions.GenericInternalRow;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.unsafe.types.UTF8String;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.condition.DisabledIfSystemProperty;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.COMMIT_SEQNO_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.COMMIT_TIME_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.FILENAME_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.PARTITION_PATH_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.RECORD_KEY_METADATA_FIELD;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * Tests for {@link HoodieSparkLanceWriter}.
+ */
+@DisabledIfSystemProperty(named = "lance.skip.tests", matches = "true")
+public class TestHoodieSparkLanceWriter {
+
+ @TempDir
+ File tempDir;
+
+ private HoodieStorage storage;
+ private SparkTaskContextSupplier taskContextSupplier;
+ private String instantTime;
+
+ @BeforeEach
+ public void setUp() throws IOException {
+ storage = HoodieTestUtils.getStorage(tempDir.getAbsolutePath());
+ taskContextSupplier = new SparkTaskContextSupplier();
+ instantTime = "20251201120000000";
+ }
+
+ @AfterEach
+ public void tearDown() throws Exception {
+ if (storage != null) {
+ storage.close();
+ }
+ }
+
+ @Test
+ public void testWriteRowWithMetadataPopulation() throws Exception {
+ // Schema WITH meta fields (writer expects this when populateMetaFields=true)
+ StructType schema = createSchemaWithMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_with_metadata.lance");
+ HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, true);
+
+ // Create row with PLACEHOLDER meta fields + user data
+ InternalRow row = createRowWithMetaFields(1, "Alice", 30L);
+ HoodieKey key = new HoodieKey("key1", "partition1");
+
+ writer.writeRowWithMetadata(key, row);
+ writer.close();
Review Comment:
@rahil-c let's do this throughout the test class
##########
hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/io/storage/TestHoodieSparkLanceWriter.java:
##########
@@ -0,0 +1,503 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.io.storage;
+
+import com.lancedb.lance.file.LanceFileReader;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.memory.RootAllocator;
+import org.apache.arrow.vector.BigIntVector;
+import org.apache.arrow.vector.BitVector;
+import org.apache.arrow.vector.Float4Vector;
+import org.apache.arrow.vector.Float8Vector;
+import org.apache.arrow.vector.IntVector;
+import org.apache.arrow.vector.VarBinaryVector;
+import org.apache.arrow.vector.VarCharVector;
+import org.apache.arrow.vector.VectorSchemaRoot;
+import org.apache.arrow.vector.ipc.ArrowReader;
+import org.apache.hudi.client.SparkTaskContextSupplier;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.testutils.HoodieTestUtils;
+import org.apache.hudi.storage.HoodieStorage;
+import org.apache.hudi.storage.StoragePath;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.catalyst.expressions.GenericInternalRow;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.unsafe.types.UTF8String;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.condition.DisabledIfSystemProperty;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.COMMIT_SEQNO_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.COMMIT_TIME_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.FILENAME_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.PARTITION_PATH_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.RECORD_KEY_METADATA_FIELD;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * Tests for {@link HoodieSparkLanceWriter}.
+ */
+@DisabledIfSystemProperty(named = "lance.skip.tests", matches = "true")
+public class TestHoodieSparkLanceWriter {
+
+ @TempDir
+ File tempDir;
+
+ private HoodieStorage storage;
+ private SparkTaskContextSupplier taskContextSupplier;
+ private String instantTime;
+
+ @BeforeEach
+ public void setUp() throws IOException {
+ storage = HoodieTestUtils.getStorage(tempDir.getAbsolutePath());
+ taskContextSupplier = new SparkTaskContextSupplier();
+ instantTime = "20251201120000000";
+ }
+
+ @AfterEach
+ public void tearDown() throws Exception {
+ if (storage != null) {
+ storage.close();
+ }
+ }
+
+ @Test
+ public void testWriteRowWithMetadataPopulation() throws Exception {
+ // Schema WITH meta fields (writer expects this when populateMetaFields=true)
+ StructType schema = createSchemaWithMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_with_metadata.lance");
+ HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, true);
+
+ // Create row with PLACEHOLDER meta fields + user data
+ InternalRow row = createRowWithMetaFields(1, "Alice", 30L);
+ HoodieKey key = new HoodieKey("key1", "partition1");
+
+ writer.writeRowWithMetadata(key, row);
+ writer.close();
+
+ // Verify using LanceFileReader
+ assertTrue(storage.exists(path), "Lance file should exist");
+
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator)) {
+
+ assertEquals(1, reader.numRows(), "Should have 1 record");
+
+ // Schema should have 5 meta fields + 3 user fields = 8 fields
+ assertEquals(8, reader.schema().getFields().size(), "Should have 8 fields");
+
+ // Read and verify data
+ ArrowReader arrowReader = reader.readAll(null, null, Integer.MAX_VALUE);
Review Comment:
@rahil-c do this throughout the test class
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add HoodieBaseLanceFileWriter and implementation for SparkFileW… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17629:
URL: https://github.com/apache/hudi/pull/17629#issuecomment-3677861829
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9633e6bc476078ebf78b401e00ccbf43cdab67e8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10326",
"triggerID" : "9633e6bc476078ebf78b401e00ccbf43cdab67e8",
"triggerType" : "PUSH"
}, {
"hash" : "6adcb20a0c841eb4164570bed7e3daa28349be35",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10335",
"triggerID" : "6adcb20a0c841eb4164570bed7e3daa28349be35",
"triggerType" : "PUSH"
}, {
"hash" : "f0f38ca10ad7bd5db724f5568aa560e0c2af8539",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10401",
"triggerID" : "f0f38ca10ad7bd5db724f5568aa560e0c2af8539",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f0f38ca10ad7bd5db724f5568aa560e0c2af8539 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10401)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add HoodieBaseLanceFileWriter and implementation for SparkFileW… [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on PR #17629:
URL: https://github.com/apache/hudi/pull/17629#issuecomment-3678014877
@the-other-tim-brown @yihua @voonhous can you guys rerun the java ci
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add HoodieBaseLanceFileWriter and implementation for SparkFileW… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17629:
URL: https://github.com/apache/hudi/pull/17629#issuecomment-3687547597
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b169123159877a93ac7a88862484211c1fa4e4f8",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10474",
"triggerID" : "b169123159877a93ac7a88862484211c1fa4e4f8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b169123159877a93ac7a88862484211c1fa4e4f8 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10474)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add HoodieBaseLanceFileWriter and implementation for SparkFileW… [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17629:
URL: https://github.com/apache/hudi/pull/17629#discussion_r2638204974
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/HoodieSparkLanceWriter.java:
##########
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.io.storage;
+
+import com.lancedb.lance.spark.arrow.LanceArrowWriter;
+import org.apache.arrow.vector.types.pojo.Schema;
+import org.apache.hudi.common.engine.TaskContextSupplier;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.io.lance.HoodieBaseLanceWriter;
+import org.apache.hudi.storage.HoodieStorage;
+import org.apache.hudi.storage.StoragePath;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.util.LanceArrowUtils;
+import org.apache.spark.unsafe.types.UTF8String;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.function.Function;
+
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.COMMIT_SEQNO_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.COMMIT_TIME_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.FILENAME_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.PARTITION_PATH_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.RECORD_KEY_METADATA_FIELD;
+
+/**
+ * Spark Lance file writer implementing {@link HoodieSparkFileWriter}.
+ *
+ * This writer integrates with Hudi's storage I/O layer and supports:
+ * - Hudi metadata field population
+ * - Record key tracking (for bloom filters - TODO)
Review Comment:
will add a link
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add HoodieBaseLanceFileWriter and implementation for SparkFileW… [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17629:
URL: https://github.com/apache/hudi/pull/17629#discussion_r2638197982
##########
hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/io/storage/TestHoodieSparkLanceWriter.java:
##########
@@ -0,0 +1,503 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.io.storage;
+
+import com.lancedb.lance.file.LanceFileReader;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.memory.RootAllocator;
+import org.apache.arrow.vector.BigIntVector;
+import org.apache.arrow.vector.BitVector;
+import org.apache.arrow.vector.Float4Vector;
+import org.apache.arrow.vector.Float8Vector;
+import org.apache.arrow.vector.IntVector;
+import org.apache.arrow.vector.VarBinaryVector;
+import org.apache.arrow.vector.VarCharVector;
+import org.apache.arrow.vector.VectorSchemaRoot;
+import org.apache.arrow.vector.ipc.ArrowReader;
+import org.apache.hudi.client.SparkTaskContextSupplier;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.testutils.HoodieTestUtils;
+import org.apache.hudi.storage.HoodieStorage;
+import org.apache.hudi.storage.StoragePath;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.catalyst.expressions.GenericInternalRow;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.unsafe.types.UTF8String;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.condition.DisabledIfSystemProperty;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.COMMIT_SEQNO_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.COMMIT_TIME_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.FILENAME_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.PARTITION_PATH_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.RECORD_KEY_METADATA_FIELD;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * Tests for {@link HoodieSparkLanceWriter}.
+ */
+@DisabledIfSystemProperty(named = "lance.skip.tests", matches = "true")
+public class TestHoodieSparkLanceWriter {
+
+ @TempDir
+ File tempDir;
+
+ private HoodieStorage storage;
+ private SparkTaskContextSupplier taskContextSupplier;
+ private String instantTime;
+
+ @BeforeEach
+ public void setUp() throws IOException {
+ storage = HoodieTestUtils.getStorage(tempDir.getAbsolutePath());
+ taskContextSupplier = new SparkTaskContextSupplier();
+ instantTime = "20251201120000000";
+ }
+
+ @AfterEach
+ public void tearDown() throws Exception {
+ if (storage != null) {
+ storage.close();
+ }
+ }
+
+ @Test
+ public void testWriteRowWithMetadataPopulation() throws Exception {
+ // Schema WITH meta fields (writer expects this when populateMetaFields=true)
+ StructType schema = createSchemaWithMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_with_metadata.lance");
+ HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, true);
+
+ // Create row with PLACEHOLDER meta fields + user data
+ InternalRow row = createRowWithMetaFields(1, "Alice", 30L);
+ HoodieKey key = new HoodieKey("key1", "partition1");
+
+ writer.writeRowWithMetadata(key, row);
+ writer.close();
+
+ // Verify using LanceFileReader
+ assertTrue(storage.exists(path), "Lance file should exist");
+
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator)) {
+
+ assertEquals(1, reader.numRows(), "Should have 1 record");
+
+ // Schema should have 5 meta fields + 3 user fields = 8 fields
+ assertEquals(8, reader.schema().getFields().size(), "Should have 8 fields");
+
+ // Read and verify data
+ ArrowReader arrowReader = reader.readAll(null, null, Integer.MAX_VALUE);
+ assertTrue(arrowReader.loadNextBatch(), "Should load batch");
+ VectorSchemaRoot root = arrowReader.getVectorSchemaRoot();
+
+ // Verify meta fields were populated
+ VarCharVector commitTimeVector = (VarCharVector) root.getVector(COMMIT_TIME_METADATA_FIELD.getFieldName());
+ assertNotNull(commitTimeVector);
+ assertEquals(instantTime, new String(commitTimeVector.get(0)), "Commit time should match");
+
+ VarCharVector recordKeyVector = (VarCharVector) root.getVector(RECORD_KEY_METADATA_FIELD.getFieldName());
+ assertEquals("key1", new String(recordKeyVector.get(0)), "Record key should match");
+
+ VarCharVector partitionPathVector = (VarCharVector) root.getVector(PARTITION_PATH_METADATA_FIELD.getFieldName());
+ assertEquals("partition1", new String(partitionPathVector.get(0)), "Partition path should match");
+
+ VarCharVector fileNameVector = (VarCharVector) root.getVector(FILENAME_METADATA_FIELD.getFieldName());
+ assertEquals(path.getName(), new String(fileNameVector.get(0)), "File name should match");
+
+ VarCharVector seqNoVector = (VarCharVector) root.getVector(COMMIT_SEQNO_METADATA_FIELD.getFieldName());
+ String seqNo = new String(seqNoVector.get(0));
+ assertTrue(seqNo.startsWith(instantTime), "Sequence number should start with instant time");
+
+ // Verify user data fields
+ IntVector idVector = (IntVector) root.getVector("id");
+ assertEquals(1, idVector.get(0), "ID should match");
+
+ VarCharVector nameVector = (VarCharVector) root.getVector("name");
+ assertEquals("Alice", new String(nameVector.get(0)), "Name should match");
+
+ BigIntVector ageVector = (BigIntVector) root.getVector("age");
+ assertEquals(30L, ageVector.get(0), "Age should match");
+
+ arrowReader.close();
+ }
+ }
+
+ @Test
+ public void testWriteRowWithoutMetadataPopulation() throws Exception {
+ // Schema WITHOUT meta fields
+ StructType schema = createSchemaWithoutMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_without_metadata.lance");
+ HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false);
+
+ // Create row with just user data (no meta fields)
+ InternalRow row = createRow(1, "Bob", 25L);
+ HoodieKey key = new HoodieKey("key2", "partition2");
+
+ writer.writeRowWithMetadata(key, row);
+ writer.close();
+
+ // Verify using LanceFileReader
+ assertTrue(storage.exists(path));
+
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator)) {
+
+ assertEquals(1, reader.numRows());
+
+ // Schema should have ONLY 3 user fields (no meta fields)
+ assertEquals(3, reader.schema().getFields().size(), "Should have only user fields");
+
+ // Read and verify data
+ ArrowReader arrowReader = reader.readAll(null, null, Integer.MAX_VALUE);
+ assertTrue(arrowReader.loadNextBatch());
+ VectorSchemaRoot root = arrowReader.getVectorSchemaRoot();
+
+ // Verify NO meta fields exist
+ assertFalse(hasField(root, COMMIT_TIME_METADATA_FIELD.getFieldName()), "Should not have commit time field");
+ assertFalse(hasField(root, RECORD_KEY_METADATA_FIELD.getFieldName()), "Should not have record key field");
+
+ // Verify user data
+ IntVector idVector = (IntVector) root.getVector("id");
+ assertEquals(1, idVector.get(0));
+
+ VarCharVector nameVector = (VarCharVector) root.getVector("name");
+ assertEquals("Bob", new String(nameVector.get(0)));
+
+ arrowReader.close();
+ }
+ }
+
+ @Test
+ public void testWriteRowSimple() throws Exception {
+ StructType schema = createSchemaWithoutMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_simple_write.lance");
+ HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false);
+
+ InternalRow row = createRow(1, "Charlie", 35L);
+ writer.writeRow("key3", row);
+ writer.close();
+
+ // Verify file exists and has correct record count
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator)) {
+ assertEquals(1, reader.numRows());
+ }
+ }
+
+ @Test
+ public void testSequenceIdGeneration() throws Exception {
+ StructType schema = createSchemaWithMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_sequence_id.lance");
+ HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, true);
+
+ // Write multiple records
+ for (int i = 0; i < 3; i++) {
+ InternalRow row = createRowWithMetaFields(i, "User" + i, 20L + i);
+ HoodieKey key = new HoodieKey("key" + i, "partition1");
+ writer.writeRowWithMetadata(key, row);
+ }
+ writer.close();
+
+ // Verify sequence IDs are unique and properly formatted
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator)) {
+
+ assertEquals(3, reader.numRows());
+
+ ArrowReader arrowReader = reader.readAll(null, null, Integer.MAX_VALUE);
+ arrowReader.loadNextBatch();
+ VectorSchemaRoot root = arrowReader.getVectorSchemaRoot();
+
+ VarCharVector seqNoVector = (VarCharVector) root.getVector(COMMIT_SEQNO_METADATA_FIELD.getFieldName());
Review Comment:
Will consolidate
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add HoodieBaseLanceFileWriter and implementation for SparkFileW… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17629:
URL: https://github.com/apache/hudi/pull/17629#issuecomment-3678801985
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9633e6bc476078ebf78b401e00ccbf43cdab67e8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10326",
"triggerID" : "9633e6bc476078ebf78b401e00ccbf43cdab67e8",
"triggerType" : "PUSH"
}, {
"hash" : "6adcb20a0c841eb4164570bed7e3daa28349be35",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10335",
"triggerID" : "6adcb20a0c841eb4164570bed7e3daa28349be35",
"triggerType" : "PUSH"
}, {
"hash" : "f0f38ca10ad7bd5db724f5568aa560e0c2af8539",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10401",
"triggerID" : "f0f38ca10ad7bd5db724f5568aa560e0c2af8539",
"triggerType" : "PUSH"
}, {
"hash" : "3e18561eb435a7b4e5452c2df03793db7e70ad82",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10420",
"triggerID" : "3e18561eb435a7b4e5452c2df03793db7e70ad82",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f0f38ca10ad7bd5db724f5568aa560e0c2af8539 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10401)
* 3e18561eb435a7b4e5452c2df03793db7e70ad82 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10420)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add HoodieBaseLanceFileWriter and implementation for SparkFileW… [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #17629:
URL: https://github.com/apache/hudi/pull/17629#discussion_r2641714800
##########
hudi-hadoop-common/src/main/java/org/apache/hudi/io/lance/HoodieBaseLanceWriter.java:
##########
@@ -0,0 +1,185 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.io.lance;
+
+import com.lancedb.lance.file.LanceFileWriter;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.memory.RootAllocator;
+import org.apache.arrow.vector.VectorSchemaRoot;
+import org.apache.arrow.vector.types.pojo.Schema;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.storage.HoodieStorage;
+import org.apache.hudi.storage.StoragePath;
+
+import javax.annotation.concurrent.NotThreadSafe;
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * Base class for Hudi Lance file writers supporting different record types.
+ *
+ * This class handles common Lance file operations including:
+ * - LanceFileWriter lifecycle management
+ * - BufferAllocator management
+ * - Record buffering and batch flushing
+ * - File size checks
+ *
+ * Subclasses must implement type-specific conversion to Arrow format.
+ *
+ * @param <R> The record type (e.g., GenericRecord, InternalRow)
+ */
+@NotThreadSafe
+public abstract class HoodieBaseLanceWriter<R> implements Closeable {
+ protected static final int DEFAULT_BATCH_SIZE = 1000;
+ protected final HoodieStorage storage;
+ protected final StoragePath path;
+ protected final BufferAllocator allocator;
+ protected final List<R> bufferedRecords;
+ protected final int batchSize;
+ protected final long maxFileSize;
+ protected long writtenRecordCount = 0;
+ protected VectorSchemaRoot root;
+
+ private LanceFileWriter writer;
+
+ /**
+ * Constructor for base Lance writer.
+ *
+ * @param storage HoodieStorage instance
+ * @param path Path where Lance file will be written
+ * @param batchSize Number of records to buffer before flushing to Lance
+ * @param maxFileSize Maximum file size in bytes before rolling over to new file
Review Comment:
I'm not seeing where this value is used, is there a follow up to use it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add HoodieBaseLanceFileWriter and implementation for SparkFileW… [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17629:
URL: https://github.com/apache/hudi/pull/17629#discussion_r2641898018
##########
hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/io/storage/TestHoodieSparkLanceWriter.java:
##########
@@ -0,0 +1,454 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.io.storage;
+
+import com.lancedb.lance.file.LanceFileReader;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.memory.RootAllocator;
+import org.apache.arrow.vector.BigIntVector;
+import org.apache.arrow.vector.BitVector;
+import org.apache.arrow.vector.Float4Vector;
+import org.apache.arrow.vector.Float8Vector;
+import org.apache.arrow.vector.IntVector;
+import org.apache.arrow.vector.VarBinaryVector;
+import org.apache.arrow.vector.VarCharVector;
+import org.apache.arrow.vector.VectorSchemaRoot;
+import org.apache.arrow.vector.ipc.ArrowReader;
+import org.apache.hudi.client.SparkTaskContextSupplier;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.testutils.HoodieTestUtils;
+import org.apache.hudi.storage.HoodieStorage;
+import org.apache.hudi.storage.StoragePath;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.catalyst.expressions.GenericInternalRow;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.unsafe.types.UTF8String;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.condition.DisabledIfSystemProperty;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.COMMIT_SEQNO_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.COMMIT_TIME_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.FILENAME_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.PARTITION_PATH_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.RECORD_KEY_METADATA_FIELD;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * Tests for {@link HoodieSparkLanceWriter}.
+ */
+@DisabledIfSystemProperty(named = "lance.skip.tests", matches = "true")
+public class TestHoodieSparkLanceWriter {
+
+ @TempDir
+ File tempDir;
+
+ private HoodieStorage storage;
+ private SparkTaskContextSupplier taskContextSupplier;
+ private String instantTime;
+
+ @BeforeEach
+ public void setUp() throws IOException {
+ storage = HoodieTestUtils.getStorage(tempDir.getAbsolutePath());
+ taskContextSupplier = new SparkTaskContextSupplier();
+ instantTime = "20251201120000000";
+ }
+
+ @AfterEach
+ public void tearDown() throws Exception {
+ if (storage != null) {
+ storage.close();
+ }
+ }
+
+ @Test
+ public void testWriteRowWithMetadataPopulation() throws Exception {
+ // Schema WITH meta fields (writer expects this when populateMetaFields=true)
+ StructType schema = createSchemaWithMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_with_metadata.lance");
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, true)) {
+ // Write multiple records to test metadata population and sequence ID generation
+ for (int i = 0; i < 3; i++) {
+ InternalRow row = createRowWithMetaFields(i, "User" + i, 20L + i);
+ HoodieKey key = new HoodieKey("key" + i, "partition1");
+ writer.writeRowWithMetadata(key, row);
+ }
+ }
+
+ // Verify using LanceFileReader
+ assertTrue(storage.exists(path), "Lance file should exist");
+
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator);
+ ArrowReader arrowReader = reader.readAll(null, null, Integer.MAX_VALUE)) {
+
+ assertEquals(3, reader.numRows(), "Should have 3 records");
+
+ // Schema should have 5 meta fields + 3 user fields = 8 fields
+ assertEquals(8, reader.schema().getFields().size(), "Should have 8 fields");
+
+ // Read and verify data
+ assertTrue(arrowReader.loadNextBatch(), "Should load batch");
+ VectorSchemaRoot root = arrowReader.getVectorSchemaRoot();
+
+ // Verify meta fields were populated for first record
+ VarCharVector commitTimeVector = (VarCharVector) root.getVector(COMMIT_TIME_METADATA_FIELD.getFieldName());
+ assertNotNull(commitTimeVector);
+ assertEquals(instantTime, new String(commitTimeVector.get(0)), "Commit time should match");
+
+ VarCharVector recordKeyVector = (VarCharVector) root.getVector(RECORD_KEY_METADATA_FIELD.getFieldName());
+ assertEquals("key0", new String(recordKeyVector.get(0)), "Record key should match");
+
+ VarCharVector partitionPathVector = (VarCharVector) root.getVector(PARTITION_PATH_METADATA_FIELD.getFieldName());
+ assertEquals("partition1", new String(partitionPathVector.get(0)), "Partition path should match");
+
+ VarCharVector fileNameVector = (VarCharVector) root.getVector(FILENAME_METADATA_FIELD.getFieldName());
+ assertEquals(path.getName(), new String(fileNameVector.get(0)), "File name should match");
+
+ // Verify sequence IDs are unique and properly formatted
+ VarCharVector seqNoVector = (VarCharVector) root.getVector(COMMIT_SEQNO_METADATA_FIELD.getFieldName());
+ List<String> seqNos = new ArrayList<>();
+ for (int i = 0; i < 3; i++) {
+ String seqNo = new String(seqNoVector.get(i));
+ assertTrue(seqNo.startsWith(instantTime + "_"), "Sequence number should start with instant time");
+ seqNos.add(seqNo);
+ }
+ assertEquals(3, seqNos.stream().distinct().count(), "All sequence IDs should be unique");
+
+ // Verify user data fields for first record
+ IntVector idVector = (IntVector) root.getVector("id");
+ assertEquals(0, idVector.get(0), "ID should match");
+
+ VarCharVector nameVector = (VarCharVector) root.getVector("name");
+ assertEquals("User0", new String(nameVector.get(0)), "Name should match");
+
+ BigIntVector ageVector = (BigIntVector) root.getVector("age");
+ assertEquals(20L, ageVector.get(0), "Age should match");
+ }
+ }
+
+ @Test
+ public void testWriteRowWithoutMetadataPopulation() throws Exception {
+ // Schema WITHOUT meta fields
+ StructType schema = createSchemaWithoutMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_without_metadata.lance");
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false)) {
+ // Create row with just user data (no meta fields)
+ InternalRow row = createRow(1, "Bob", 25L);
+ HoodieKey key = new HoodieKey("key2", "partition2");
+
+ writer.writeRowWithMetadata(key, row);
+ }
+
+ // Verify using LanceFileReader
+ assertTrue(storage.exists(path));
+
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator);
+ ArrowReader arrowReader = reader.readAll(null, null, Integer.MAX_VALUE)) {
+
+ assertEquals(1, reader.numRows());
+
+ // Schema should have ONLY 3 user fields (no meta fields)
+ assertEquals(3, reader.schema().getFields().size(), "Should have only user fields");
+
+ // Read and verify data
+ assertTrue(arrowReader.loadNextBatch());
+ VectorSchemaRoot root = arrowReader.getVectorSchemaRoot();
+
+ // Verify NO meta fields exist
+ assertFalse(hasField(root, COMMIT_TIME_METADATA_FIELD.getFieldName()), "Should not have commit time field");
+ assertFalse(hasField(root, RECORD_KEY_METADATA_FIELD.getFieldName()), "Should not have record key field");
+
+ // Verify user data
+ IntVector idVector = (IntVector) root.getVector("id");
+ assertEquals(1, idVector.get(0));
+
+ VarCharVector nameVector = (VarCharVector) root.getVector("name");
+ assertEquals("Bob", new String(nameVector.get(0)));
+ }
+ }
+
+ @Test
+ public void testWriteRowSimple() throws Exception {
+ StructType schema = createSchemaWithoutMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_simple_write.lance");
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false)) {
+ InternalRow row = createRow(1, "Charlie", 35L);
+ writer.writeRow("key3", row);
+ }
+
+ // Verify file exists and has correct record count
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator)) {
+ assertEquals(1, reader.numRows());
+ }
+ }
+
+ @Test
+ public void testBatchFlushing() throws Exception {
+ StructType schema = createSchemaWithoutMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_batch_flush.lance");
+ // Write more than DEFAULT_BATCH_SIZE (1000) records
+ int recordCount = 2500;
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false)) {
+ for (int i = 0; i < recordCount; i++) {
+ InternalRow row = createRow(i, "User" + i, 20L + i);
+ writer.writeRow("key" + i, row);
+ }
+ }
+
+ // Verify all records were written
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator)) {
+ assertEquals(recordCount, reader.numRows(), "All records should be written");
+ }
+ }
+
+ @Test
+ public void testPrimitiveTypes() throws Exception {
+ StructType schema = new StructType()
+ .add("int_field", DataTypes.IntegerType, false)
+ .add("long_field", DataTypes.LongType, false)
+ .add("float_field", DataTypes.FloatType, false)
+ .add("double_field", DataTypes.DoubleType, false)
+ .add("bool_field", DataTypes.BooleanType, false)
+ .add("string_field", DataTypes.StringType, false)
+ .add("binary_field", DataTypes.BinaryType, false);
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_primitives.lance");
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false)) {
+ GenericInternalRow row = new GenericInternalRow(new Object[]{
+ 42, // int
+ 123456789L, // long
+ 3.14f, // float
+ 2.71828, // double
+ true, // boolean
+ UTF8String.fromString("test"), // string
+ new byte[]{1, 2, 3, 4} // binary
+ });
+
+ writer.writeRow("key1", row);
+ }
+
+ // Verify all types were written correctly
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator);
+ ArrowReader arrowReader = reader.readAll(null, null, Integer.MAX_VALUE)) {
+
+ assertEquals(1, reader.numRows());
+ assertEquals(7, reader.schema().getFields().size());
+
+ arrowReader.loadNextBatch();
+ VectorSchemaRoot root = arrowReader.getVectorSchemaRoot();
+
+ assertEquals(42, ((IntVector) root.getVector("int_field")).get(0));
+ assertEquals(123456789L, ((BigIntVector) root.getVector("long_field")).get(0));
+ assertEquals(3.14f, ((Float4Vector) root.getVector("float_field")).get(0), 0.001);
+ assertEquals(2.71828, ((Float8Vector) root.getVector("double_field")).get(0), 0.00001);
+ assertEquals(1, ((BitVector) root.getVector("bool_field")).get(0));
+ assertEquals("test", new String(((VarCharVector) root.getVector("string_field")).get(0)));
+
+ byte[] binary = ((VarBinaryVector) root.getVector("binary_field")).get(0);
+ assertEquals(4, binary.length);
+ }
+ }
+
+ @Test
+ public void testNullValues() throws Exception {
+ StructType schema = new StructType()
+ .add("id", DataTypes.IntegerType, false)
+ .add("name", DataTypes.StringType, true)
+ .add("age", DataTypes.LongType, true);
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_nulls.lance");
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false)) {
+ // Write rows with null values
+ writer.writeRow("key1", createRow(1, "Alice", 30L));
+ writer.writeRow("key2", createRow(2, null, 25L)); // null name
+ writer.writeRow("key3", createRow(3, "Charlie", null)); // null age
+ }
+
+ // Verify nulls are preserved
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator);
+ ArrowReader arrowReader = reader.readAll(null, null, Integer.MAX_VALUE)) {
+
+ assertEquals(3, reader.numRows());
+
+ arrowReader.loadNextBatch();
+ VectorSchemaRoot root = arrowReader.getVectorSchemaRoot();
+
+ VarCharVector nameVector = (VarCharVector) root.getVector("name");
+ assertFalse(nameVector.isNull(0), "First name should not be null");
+ assertTrue(nameVector.isNull(1), "Second name should be null");
+ assertFalse(nameVector.isNull(2), "Third name should not be null");
+
+ BigIntVector ageVector = (BigIntVector) root.getVector("age");
+ assertFalse(ageVector.isNull(0), "First age should not be null");
+ assertFalse(ageVector.isNull(1), "Second age should not be null");
+ assertTrue(ageVector.isNull(2), "Third age should be null");
+ }
+ }
+
+ @Test
+ public void testWriteEmptyDataset() throws Exception {
+ StructType schema = new StructType()
+ .add("id", DataTypes.IntegerType, false);
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_empty.lance");
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false)) {
+ // Close without writing any rows
+ }
+
+ // Lance doesn't create a file if no data is written
+ assertFalse(storage.exists(path), "Lance file should not exist when no data is written");
Review Comment:
Following @the-other-tim-brown recommendation I tried writing an equivalent of this test above, but using the `HoodieSparkParquetWriter`
https://gist.github.com/rahil-c/f1251f999df7674077153acdf37e610d
However when doing so i ran into the following issue before seeing an empty file created
```
org.apache.hudi.exception.HoodieException: Unable to instantiate class org.apache.hudi.io.storage.row.HoodieRowParquetWriteSupport
at org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:75)
at org.apache.hudi.io.storage.row.HoodieRowParquetWriteSupport.getHoodieRowParquetWriteSupport(HoodieRowParquetWriteSupport.java:416)
at org.apache.hudi.io.storage.TestHoodieSparkParquetWriter.testWriteEmptyDataset(TestHoodieSparkParquetWriter.java:87)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
Caused by: java.lang.reflect.InvocationTargetException
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
at org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:73)
... 5 more
Caused by: java.lang.IllegalArgumentException: Schema string cannot be null or empty
at org.apache.hudi.common.util.ValidationUtils.checkArgument(ValidationUtils.java:42)
at org.apache.hudi.common.schema.HoodieSchema$Parser.parse(HoodieSchema.java:893)
at org.apache.hudi.common.schema.HoodieSchema.parse(HoodieSchema.java:138)
at org.apache.hudi.io.storage.row.HoodieRowParquetWriteSupport.lambda$new$2(HoodieRowParquetWriteSupport.java:138)
at org.apache.hudi.common.util.Option.orElseGet(Option.java:153)
at org.apache.hudi.io.storage.row.HoodieRowParquetWriteSupport.<init>(HoodieRowParquetWriteSupport.java:136)
... 10 more
```
I then discussed with @yihua offline, who pointed me to see if `HoodieAvroParquetWriter` creates an empty file. When writing a similar test in test `TestHoodieAvroParquetWriter` : https://gist.github.com/rahil-c/0b719717f72eb59dacdd646c2802309b
was able to confirm that an empty file is created despite no actual data being rows being written.
When inspecting the underlying parquet file that was created from the test I see the following, where the empty file persists the schema.
<img width="1548" height="274" alt="Screenshot 2025-12-22 at 11 27 32 PM" src="https://github.com/user-attachments/assets/648138ec-6d46-45dd-91fd-a071ea0a5481" />
@the-other-tim-brown @yihua
So to confirm is the behavior we want to achieve, when there is no data present that we just want to write the schema to a lance base file? The lance java file writer does not have any apis writing to the footer similar to parquet, https://github.com/lance-format/lance/blob/main/java/src/main/java/org/lance/file/LanceFileWriter.java#L31 so need to investigate this more.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add HoodieBaseLanceFileWriter and implementation for SparkFileW… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17629:
URL: https://github.com/apache/hudi/pull/17629#issuecomment-3667581901
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9633e6bc476078ebf78b401e00ccbf43cdab67e8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10326",
"triggerID" : "9633e6bc476078ebf78b401e00ccbf43cdab67e8",
"triggerType" : "PUSH"
}, {
"hash" : "6adcb20a0c841eb4164570bed7e3daa28349be35",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10335",
"triggerID" : "6adcb20a0c841eb4164570bed7e3daa28349be35",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9633e6bc476078ebf78b401e00ccbf43cdab67e8 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10326)
* 6adcb20a0c841eb4164570bed7e3daa28349be35 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10335)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add HoodieBaseLanceFileWriter and implementation for SparkFileW… [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17629:
URL: https://github.com/apache/hudi/pull/17629#discussion_r2627681400
##########
pom.xml:
##########
@@ -238,6 +238,11 @@
<springboot.version>2.7.10</springboot.version>
<spring.shell.version>2.1.1</spring.shell.version>
<snappy.version>1.1.10.7</snappy.version>
+ <arrow.version>18.3.0</arrow.version>
+ <lance.version>0.38.0</lance.version>
+ <lance.spark.connector.version>0.0.13</lance.spark.connector.version>
+ <lance.spark.artifact>lance-spark-3.5_${scala.binary.version}</lance.spark.artifact>
Review Comment:
need to think how we can have spark 3.5 and 4.0 work
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add HoodieBaseLanceFileWriter and implementation for SparkFileW… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17629:
URL: https://github.com/apache/hudi/pull/17629#issuecomment-3686825137
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9633e6bc476078ebf78b401e00ccbf43cdab67e8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10326",
"triggerID" : "9633e6bc476078ebf78b401e00ccbf43cdab67e8",
"triggerType" : "PUSH"
}, {
"hash" : "6adcb20a0c841eb4164570bed7e3daa28349be35",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10335",
"triggerID" : "6adcb20a0c841eb4164570bed7e3daa28349be35",
"triggerType" : "PUSH"
}, {
"hash" : "f0f38ca10ad7bd5db724f5568aa560e0c2af8539",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10401",
"triggerID" : "f0f38ca10ad7bd5db724f5568aa560e0c2af8539",
"triggerType" : "PUSH"
}, {
"hash" : "3e18561eb435a7b4e5452c2df03793db7e70ad82",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10420",
"triggerID" : "3e18561eb435a7b4e5452c2df03793db7e70ad82",
"triggerType" : "PUSH"
}, {
"hash" : "33cfffa5175585e24bd00025a01f3bb4f5abf3fb",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10424",
"triggerID" : "33cfffa5175585e24bd00025a01f3bb4f5abf3fb",
"triggerType" : "PUSH"
}, {
"hash" : "b169123159877a93ac7a88862484211c1fa4e4f8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10474",
"triggerID" : "b169123159877a93ac7a88862484211c1fa4e4f8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 33cfffa5175585e24bd00025a01f3bb4f5abf3fb Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10424)
* b169123159877a93ac7a88862484211c1fa4e4f8 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10474)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add HoodieBaseLanceFileWriter and implementation for SparkFileW… [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on PR #17629:
URL: https://github.com/apache/hudi/pull/17629#issuecomment-3687503731
On rerun the azure ci passes, only the bundle validation is failing for flaky reasons (even though it was passing fine in prior commits)
<img width="353" height="593" alt="Screenshot 2025-12-23 at 1 01 07 PM" src="https://github.com/user-attachments/assets/8f297245-cade-424f-a4fe-b8aa304e8b02" />
@the-other-tim-brown @yihua wondering if we can merge this at this point since it will make it easier for me to rebase the other stacked prs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add HoodieBaseLanceFileWriter and implementation for SparkFileW… [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #17629:
URL: https://github.com/apache/hudi/pull/17629#discussion_r2637383050
##########
hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/io/storage/TestHoodieSparkLanceWriter.java:
##########
@@ -0,0 +1,503 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.io.storage;
+
+import com.lancedb.lance.file.LanceFileReader;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.memory.RootAllocator;
+import org.apache.arrow.vector.BigIntVector;
+import org.apache.arrow.vector.BitVector;
+import org.apache.arrow.vector.Float4Vector;
+import org.apache.arrow.vector.Float8Vector;
+import org.apache.arrow.vector.IntVector;
+import org.apache.arrow.vector.VarBinaryVector;
+import org.apache.arrow.vector.VarCharVector;
+import org.apache.arrow.vector.VectorSchemaRoot;
+import org.apache.arrow.vector.ipc.ArrowReader;
+import org.apache.hudi.client.SparkTaskContextSupplier;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.testutils.HoodieTestUtils;
+import org.apache.hudi.storage.HoodieStorage;
+import org.apache.hudi.storage.StoragePath;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.catalyst.expressions.GenericInternalRow;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.unsafe.types.UTF8String;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.condition.DisabledIfSystemProperty;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.COMMIT_SEQNO_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.COMMIT_TIME_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.FILENAME_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.PARTITION_PATH_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.RECORD_KEY_METADATA_FIELD;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * Tests for {@link HoodieSparkLanceWriter}.
+ */
+@DisabledIfSystemProperty(named = "lance.skip.tests", matches = "true")
+public class TestHoodieSparkLanceWriter {
+
+ @TempDir
+ File tempDir;
+
+ private HoodieStorage storage;
+ private SparkTaskContextSupplier taskContextSupplier;
+ private String instantTime;
+
+ @BeforeEach
+ public void setUp() throws IOException {
+ storage = HoodieTestUtils.getStorage(tempDir.getAbsolutePath());
+ taskContextSupplier = new SparkTaskContextSupplier();
+ instantTime = "20251201120000000";
+ }
+
+ @AfterEach
+ public void tearDown() throws Exception {
+ if (storage != null) {
+ storage.close();
+ }
+ }
+
+ @Test
+ public void testWriteRowWithMetadataPopulation() throws Exception {
+ // Schema WITH meta fields (writer expects this when populateMetaFields=true)
+ StructType schema = createSchemaWithMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_with_metadata.lance");
+ HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, true);
+
+ // Create row with PLACEHOLDER meta fields + user data
+ InternalRow row = createRowWithMetaFields(1, "Alice", 30L);
+ HoodieKey key = new HoodieKey("key1", "partition1");
+
+ writer.writeRowWithMetadata(key, row);
+ writer.close();
+
+ // Verify using LanceFileReader
+ assertTrue(storage.exists(path), "Lance file should exist");
+
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator)) {
+
+ assertEquals(1, reader.numRows(), "Should have 1 record");
+
+ // Schema should have 5 meta fields + 3 user fields = 8 fields
+ assertEquals(8, reader.schema().getFields().size(), "Should have 8 fields");
+
+ // Read and verify data
+ ArrowReader arrowReader = reader.readAll(null, null, Integer.MAX_VALUE);
+ assertTrue(arrowReader.loadNextBatch(), "Should load batch");
+ VectorSchemaRoot root = arrowReader.getVectorSchemaRoot();
+
+ // Verify meta fields were populated
+ VarCharVector commitTimeVector = (VarCharVector) root.getVector(COMMIT_TIME_METADATA_FIELD.getFieldName());
+ assertNotNull(commitTimeVector);
+ assertEquals(instantTime, new String(commitTimeVector.get(0)), "Commit time should match");
+
+ VarCharVector recordKeyVector = (VarCharVector) root.getVector(RECORD_KEY_METADATA_FIELD.getFieldName());
+ assertEquals("key1", new String(recordKeyVector.get(0)), "Record key should match");
+
+ VarCharVector partitionPathVector = (VarCharVector) root.getVector(PARTITION_PATH_METADATA_FIELD.getFieldName());
+ assertEquals("partition1", new String(partitionPathVector.get(0)), "Partition path should match");
+
+ VarCharVector fileNameVector = (VarCharVector) root.getVector(FILENAME_METADATA_FIELD.getFieldName());
+ assertEquals(path.getName(), new String(fileNameVector.get(0)), "File name should match");
+
+ VarCharVector seqNoVector = (VarCharVector) root.getVector(COMMIT_SEQNO_METADATA_FIELD.getFieldName());
+ String seqNo = new String(seqNoVector.get(0));
+ assertTrue(seqNo.startsWith(instantTime), "Sequence number should start with instant time");
+
+ // Verify user data fields
+ IntVector idVector = (IntVector) root.getVector("id");
+ assertEquals(1, idVector.get(0), "ID should match");
+
+ VarCharVector nameVector = (VarCharVector) root.getVector("name");
+ assertEquals("Alice", new String(nameVector.get(0)), "Name should match");
+
+ BigIntVector ageVector = (BigIntVector) root.getVector("age");
+ assertEquals(30L, ageVector.get(0), "Age should match");
+
+ arrowReader.close();
+ }
+ }
+
+ @Test
+ public void testWriteRowWithoutMetadataPopulation() throws Exception {
+ // Schema WITHOUT meta fields
+ StructType schema = createSchemaWithoutMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_without_metadata.lance");
+ HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false);
+
+ // Create row with just user data (no meta fields)
+ InternalRow row = createRow(1, "Bob", 25L);
+ HoodieKey key = new HoodieKey("key2", "partition2");
+
+ writer.writeRowWithMetadata(key, row);
+ writer.close();
+
+ // Verify using LanceFileReader
+ assertTrue(storage.exists(path));
+
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator)) {
+
+ assertEquals(1, reader.numRows());
+
+ // Schema should have ONLY 3 user fields (no meta fields)
+ assertEquals(3, reader.schema().getFields().size(), "Should have only user fields");
+
+ // Read and verify data
+ ArrowReader arrowReader = reader.readAll(null, null, Integer.MAX_VALUE);
+ assertTrue(arrowReader.loadNextBatch());
+ VectorSchemaRoot root = arrowReader.getVectorSchemaRoot();
+
+ // Verify NO meta fields exist
+ assertFalse(hasField(root, COMMIT_TIME_METADATA_FIELD.getFieldName()), "Should not have commit time field");
+ assertFalse(hasField(root, RECORD_KEY_METADATA_FIELD.getFieldName()), "Should not have record key field");
+
+ // Verify user data
+ IntVector idVector = (IntVector) root.getVector("id");
+ assertEquals(1, idVector.get(0));
+
+ VarCharVector nameVector = (VarCharVector) root.getVector("name");
+ assertEquals("Bob", new String(nameVector.get(0)));
+
+ arrowReader.close();
+ }
+ }
+
+ @Test
+ public void testWriteRowSimple() throws Exception {
+ StructType schema = createSchemaWithoutMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_simple_write.lance");
+ HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false);
+
+ InternalRow row = createRow(1, "Charlie", 35L);
+ writer.writeRow("key3", row);
+ writer.close();
+
+ // Verify file exists and has correct record count
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator)) {
+ assertEquals(1, reader.numRows());
+ }
+ }
+
+ @Test
+ public void testSequenceIdGeneration() throws Exception {
+ StructType schema = createSchemaWithMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_sequence_id.lance");
+ HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, true);
+
+ // Write multiple records
+ for (int i = 0; i < 3; i++) {
+ InternalRow row = createRowWithMetaFields(i, "User" + i, 20L + i);
+ HoodieKey key = new HoodieKey("key" + i, "partition1");
+ writer.writeRowWithMetadata(key, row);
+ }
+ writer.close();
+
+ // Verify sequence IDs are unique and properly formatted
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator)) {
+
+ assertEquals(3, reader.numRows());
+
+ ArrowReader arrowReader = reader.readAll(null, null, Integer.MAX_VALUE);
+ arrowReader.loadNextBatch();
+ VectorSchemaRoot root = arrowReader.getVectorSchemaRoot();
+
+ VarCharVector seqNoVector = (VarCharVector) root.getVector(COMMIT_SEQNO_METADATA_FIELD.getFieldName());
Review Comment:
Will the seqno also be populated in `testWriteRowWithMetadataPopulation`?
##########
hudi-hadoop-common/src/main/java/org/apache/hudi/io/lance/HoodieBaseLanceWriter.java:
##########
@@ -0,0 +1,196 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.io.lance;
+
+import com.lancedb.lance.file.LanceFileWriter;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.memory.RootAllocator;
+import org.apache.arrow.vector.VectorSchemaRoot;
+import org.apache.arrow.vector.types.pojo.Schema;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.storage.HoodieStorage;
+import org.apache.hudi.storage.StoragePath;
+
+import javax.annotation.concurrent.NotThreadSafe;
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * Base class for Hudi Lance file writers supporting different record types.
+ *
+ * This class handles common Lance file operations including:
+ * - LanceFileWriter lifecycle management
+ * - BufferAllocator management
+ * - Record buffering and batch flushing
+ * - File size checks
+ *
+ * Subclasses must implement type-specific conversion to Arrow format.
+ *
+ * @param <R> The record type (e.g., GenericRecord, InternalRow)
+ */
+@NotThreadSafe
+public abstract class HoodieBaseLanceWriter<R> implements Closeable {
+ protected static final int DEFAULT_BATCH_SIZE = 1000;
+ protected final HoodieStorage storage;
+ protected final StoragePath path;
+ protected final BufferAllocator allocator;
+ protected final List<R> bufferedRecords;
+ protected final int batchSize;
+ protected final long maxFileSize;
+ protected long writtenRecordCount = 0;
+ protected VectorSchemaRoot root;
+
+ private LanceFileWriter writer;
+
+ /**
+ * Constructor for base Lance writer.
+ *
+ * @param storage HoodieStorage instance
+ * @param path Path where Lance file will be written
+ * @param batchSize Number of records to buffer before flushing to Lance
+ * @param maxFileSize Maximum file size in bytes before rolling over to new file
+ */
+ protected HoodieBaseLanceWriter(HoodieStorage storage, StoragePath path, int batchSize, long maxFileSize) {
+ this.storage = storage;
+ this.path = path;
+ this.allocator = new RootAllocator(Long.MAX_VALUE);
+ this.bufferedRecords = new ArrayList<>(batchSize);
+ this.batchSize = batchSize;
+ this.maxFileSize = maxFileSize;
+ }
+
+ /**
+ * Populate the VectorSchemaRoot with buffered records.
+ * Subclasses must implement type-specific conversion logic.
+ * The VectorSchemaRoot field is reused across batches and managed by this base class.
+ *
+ * @param records List of records to convert
+ */
+ protected abstract void populateVectorSchemaRoot(List<R> records);
+
+ /**
+ * Get the Arrow schema for this writer.
+ * Subclasses must provide the Arrow schema corresponding to their record type.
+ *
+ * @return Arrow schema
+ */
+ protected abstract Schema getArrowSchema();
+
+ /**
+ * Write a single record. Records are buffered and flushed in batches.
+ *
+ * @param record Record to write
+ * @throws IOException if write fails
+ */
+ public void write(R record) throws IOException {
+ bufferedRecords.add(record);
+ writtenRecordCount++;
+
+ if (bufferedRecords.size() >= batchSize) {
+ flushBatch();
+ }
+ }
+
+ /**
+ * Check if writer can accept more records based on file size.
+ * Uses filesystem-based size checking (similar to ORC/HFile approach).
+ *
+ * @return true if writer can accept more records, false if file size limit reached
+ */
+ public boolean canWrite() {
Review Comment:
Since this is part of the `HoodieSparkFileWriter` interface, let's move it to the `HoodieSparkLanceWriter`?
##########
pom.xml:
##########
@@ -240,6 +240,11 @@
<springboot.version>2.7.10</springboot.version>
<spring.shell.version>2.1.1</spring.shell.version>
<snappy.version>1.1.10.7</snappy.version>
+ <arrow.version>18.3.0</arrow.version>
+ <lance.version>0.38.0</lance.version>
Review Comment:
Let's upgrade to 0.39.0
##########
pom.xml:
##########
@@ -240,6 +240,11 @@
<springboot.version>2.7.10</springboot.version>
<spring.shell.version>2.1.1</spring.shell.version>
<snappy.version>1.1.10.7</snappy.version>
+ <arrow.version>18.3.0</arrow.version>
+ <lance.version>0.38.0</lance.version>
+ <lance.spark.connector.version>0.0.13</lance.spark.connector.version>
Review Comment:
Looks like 0.15.0 is the latest now
##########
hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/io/storage/TestHoodieSparkLanceWriter.java:
##########
@@ -0,0 +1,503 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.io.storage;
+
+import com.lancedb.lance.file.LanceFileReader;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.memory.RootAllocator;
+import org.apache.arrow.vector.BigIntVector;
+import org.apache.arrow.vector.BitVector;
+import org.apache.arrow.vector.Float4Vector;
+import org.apache.arrow.vector.Float8Vector;
+import org.apache.arrow.vector.IntVector;
+import org.apache.arrow.vector.VarBinaryVector;
+import org.apache.arrow.vector.VarCharVector;
+import org.apache.arrow.vector.VectorSchemaRoot;
+import org.apache.arrow.vector.ipc.ArrowReader;
+import org.apache.hudi.client.SparkTaskContextSupplier;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.testutils.HoodieTestUtils;
+import org.apache.hudi.storage.HoodieStorage;
+import org.apache.hudi.storage.StoragePath;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.catalyst.expressions.GenericInternalRow;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.unsafe.types.UTF8String;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.condition.DisabledIfSystemProperty;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.COMMIT_SEQNO_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.COMMIT_TIME_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.FILENAME_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.PARTITION_PATH_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.RECORD_KEY_METADATA_FIELD;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * Tests for {@link HoodieSparkLanceWriter}.
+ */
+@DisabledIfSystemProperty(named = "lance.skip.tests", matches = "true")
+public class TestHoodieSparkLanceWriter {
+
+ @TempDir
+ File tempDir;
+
+ private HoodieStorage storage;
+ private SparkTaskContextSupplier taskContextSupplier;
+ private String instantTime;
+
+ @BeforeEach
+ public void setUp() throws IOException {
+ storage = HoodieTestUtils.getStorage(tempDir.getAbsolutePath());
+ taskContextSupplier = new SparkTaskContextSupplier();
+ instantTime = "20251201120000000";
+ }
+
+ @AfterEach
+ public void tearDown() throws Exception {
+ if (storage != null) {
+ storage.close();
+ }
+ }
+
+ @Test
+ public void testWriteRowWithMetadataPopulation() throws Exception {
+ // Schema WITH meta fields (writer expects this when populateMetaFields=true)
+ StructType schema = createSchemaWithMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_with_metadata.lance");
+ HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, true);
+
+ // Create row with PLACEHOLDER meta fields + user data
+ InternalRow row = createRowWithMetaFields(1, "Alice", 30L);
+ HoodieKey key = new HoodieKey("key1", "partition1");
+
+ writer.writeRowWithMetadata(key, row);
+ writer.close();
+
+ // Verify using LanceFileReader
+ assertTrue(storage.exists(path), "Lance file should exist");
+
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator)) {
+
+ assertEquals(1, reader.numRows(), "Should have 1 record");
+
+ // Schema should have 5 meta fields + 3 user fields = 8 fields
+ assertEquals(8, reader.schema().getFields().size(), "Should have 8 fields");
+
+ // Read and verify data
+ ArrowReader arrowReader = reader.readAll(null, null, Integer.MAX_VALUE);
Review Comment:
nitpick: can we move this up to the try-with-resources?
##########
hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/io/storage/TestHoodieSparkLanceWriter.java:
##########
@@ -0,0 +1,503 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.io.storage;
+
+import com.lancedb.lance.file.LanceFileReader;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.memory.RootAllocator;
+import org.apache.arrow.vector.BigIntVector;
+import org.apache.arrow.vector.BitVector;
+import org.apache.arrow.vector.Float4Vector;
+import org.apache.arrow.vector.Float8Vector;
+import org.apache.arrow.vector.IntVector;
+import org.apache.arrow.vector.VarBinaryVector;
+import org.apache.arrow.vector.VarCharVector;
+import org.apache.arrow.vector.VectorSchemaRoot;
+import org.apache.arrow.vector.ipc.ArrowReader;
+import org.apache.hudi.client.SparkTaskContextSupplier;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.testutils.HoodieTestUtils;
+import org.apache.hudi.storage.HoodieStorage;
+import org.apache.hudi.storage.StoragePath;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.catalyst.expressions.GenericInternalRow;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.unsafe.types.UTF8String;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.condition.DisabledIfSystemProperty;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.COMMIT_SEQNO_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.COMMIT_TIME_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.FILENAME_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.PARTITION_PATH_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.RECORD_KEY_METADATA_FIELD;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * Tests for {@link HoodieSparkLanceWriter}.
+ */
+@DisabledIfSystemProperty(named = "lance.skip.tests", matches = "true")
+public class TestHoodieSparkLanceWriter {
+
+ @TempDir
+ File tempDir;
+
+ private HoodieStorage storage;
+ private SparkTaskContextSupplier taskContextSupplier;
+ private String instantTime;
+
+ @BeforeEach
+ public void setUp() throws IOException {
+ storage = HoodieTestUtils.getStorage(tempDir.getAbsolutePath());
+ taskContextSupplier = new SparkTaskContextSupplier();
+ instantTime = "20251201120000000";
+ }
+
+ @AfterEach
+ public void tearDown() throws Exception {
+ if (storage != null) {
+ storage.close();
+ }
+ }
+
+ @Test
+ public void testWriteRowWithMetadataPopulation() throws Exception {
+ // Schema WITH meta fields (writer expects this when populateMetaFields=true)
+ StructType schema = createSchemaWithMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_with_metadata.lance");
+ HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, true);
+
+ // Create row with PLACEHOLDER meta fields + user data
+ InternalRow row = createRowWithMetaFields(1, "Alice", 30L);
+ HoodieKey key = new HoodieKey("key1", "partition1");
+
+ writer.writeRowWithMetadata(key, row);
+ writer.close();
Review Comment:
nit: Let's use try-with-resources here
##########
hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/io/storage/TestHoodieSparkLanceWriter.java:
##########
@@ -0,0 +1,503 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.io.storage;
+
+import com.lancedb.lance.file.LanceFileReader;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.memory.RootAllocator;
+import org.apache.arrow.vector.BigIntVector;
+import org.apache.arrow.vector.BitVector;
+import org.apache.arrow.vector.Float4Vector;
+import org.apache.arrow.vector.Float8Vector;
+import org.apache.arrow.vector.IntVector;
+import org.apache.arrow.vector.VarBinaryVector;
+import org.apache.arrow.vector.VarCharVector;
+import org.apache.arrow.vector.VectorSchemaRoot;
+import org.apache.arrow.vector.ipc.ArrowReader;
+import org.apache.hudi.client.SparkTaskContextSupplier;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.testutils.HoodieTestUtils;
+import org.apache.hudi.storage.HoodieStorage;
+import org.apache.hudi.storage.StoragePath;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.catalyst.expressions.GenericInternalRow;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.unsafe.types.UTF8String;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.condition.DisabledIfSystemProperty;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.COMMIT_SEQNO_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.COMMIT_TIME_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.FILENAME_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.PARTITION_PATH_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.RECORD_KEY_METADATA_FIELD;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * Tests for {@link HoodieSparkLanceWriter}.
+ */
+@DisabledIfSystemProperty(named = "lance.skip.tests", matches = "true")
+public class TestHoodieSparkLanceWriter {
+
+ @TempDir
+ File tempDir;
+
+ private HoodieStorage storage;
+ private SparkTaskContextSupplier taskContextSupplier;
+ private String instantTime;
+
+ @BeforeEach
+ public void setUp() throws IOException {
+ storage = HoodieTestUtils.getStorage(tempDir.getAbsolutePath());
+ taskContextSupplier = new SparkTaskContextSupplier();
+ instantTime = "20251201120000000";
+ }
+
+ @AfterEach
+ public void tearDown() throws Exception {
+ if (storage != null) {
+ storage.close();
+ }
+ }
+
+ @Test
+ public void testWriteRowWithMetadataPopulation() throws Exception {
+ // Schema WITH meta fields (writer expects this when populateMetaFields=true)
+ StructType schema = createSchemaWithMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_with_metadata.lance");
+ HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, true);
+
+ // Create row with PLACEHOLDER meta fields + user data
+ InternalRow row = createRowWithMetaFields(1, "Alice", 30L);
+ HoodieKey key = new HoodieKey("key1", "partition1");
+
+ writer.writeRowWithMetadata(key, row);
+ writer.close();
+
+ // Verify using LanceFileReader
+ assertTrue(storage.exists(path), "Lance file should exist");
+
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator)) {
+
+ assertEquals(1, reader.numRows(), "Should have 1 record");
+
+ // Schema should have 5 meta fields + 3 user fields = 8 fields
+ assertEquals(8, reader.schema().getFields().size(), "Should have 8 fields");
+
+ // Read and verify data
+ ArrowReader arrowReader = reader.readAll(null, null, Integer.MAX_VALUE);
+ assertTrue(arrowReader.loadNextBatch(), "Should load batch");
+ VectorSchemaRoot root = arrowReader.getVectorSchemaRoot();
+
+ // Verify meta fields were populated
+ VarCharVector commitTimeVector = (VarCharVector) root.getVector(COMMIT_TIME_METADATA_FIELD.getFieldName());
+ assertNotNull(commitTimeVector);
+ assertEquals(instantTime, new String(commitTimeVector.get(0)), "Commit time should match");
+
+ VarCharVector recordKeyVector = (VarCharVector) root.getVector(RECORD_KEY_METADATA_FIELD.getFieldName());
+ assertEquals("key1", new String(recordKeyVector.get(0)), "Record key should match");
+
+ VarCharVector partitionPathVector = (VarCharVector) root.getVector(PARTITION_PATH_METADATA_FIELD.getFieldName());
+ assertEquals("partition1", new String(partitionPathVector.get(0)), "Partition path should match");
+
+ VarCharVector fileNameVector = (VarCharVector) root.getVector(FILENAME_METADATA_FIELD.getFieldName());
+ assertEquals(path.getName(), new String(fileNameVector.get(0)), "File name should match");
+
+ VarCharVector seqNoVector = (VarCharVector) root.getVector(COMMIT_SEQNO_METADATA_FIELD.getFieldName());
+ String seqNo = new String(seqNoVector.get(0));
+ assertTrue(seqNo.startsWith(instantTime), "Sequence number should start with instant time");
+
+ // Verify user data fields
+ IntVector idVector = (IntVector) root.getVector("id");
+ assertEquals(1, idVector.get(0), "ID should match");
+
+ VarCharVector nameVector = (VarCharVector) root.getVector("name");
+ assertEquals("Alice", new String(nameVector.get(0)), "Name should match");
+
+ BigIntVector ageVector = (BigIntVector) root.getVector("age");
+ assertEquals(30L, ageVector.get(0), "Age should match");
+
+ arrowReader.close();
+ }
+ }
+
+ @Test
+ public void testWriteRowWithoutMetadataPopulation() throws Exception {
+ // Schema WITHOUT meta fields
+ StructType schema = createSchemaWithoutMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_without_metadata.lance");
+ HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false);
+
+ // Create row with just user data (no meta fields)
+ InternalRow row = createRow(1, "Bob", 25L);
+ HoodieKey key = new HoodieKey("key2", "partition2");
+
+ writer.writeRowWithMetadata(key, row);
+ writer.close();
+
+ // Verify using LanceFileReader
+ assertTrue(storage.exists(path));
+
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator)) {
+
+ assertEquals(1, reader.numRows());
+
+ // Schema should have ONLY 3 user fields (no meta fields)
+ assertEquals(3, reader.schema().getFields().size(), "Should have only user fields");
+
+ // Read and verify data
+ ArrowReader arrowReader = reader.readAll(null, null, Integer.MAX_VALUE);
+ assertTrue(arrowReader.loadNextBatch());
+ VectorSchemaRoot root = arrowReader.getVectorSchemaRoot();
+
+ // Verify NO meta fields exist
+ assertFalse(hasField(root, COMMIT_TIME_METADATA_FIELD.getFieldName()), "Should not have commit time field");
+ assertFalse(hasField(root, RECORD_KEY_METADATA_FIELD.getFieldName()), "Should not have record key field");
+
+ // Verify user data
+ IntVector idVector = (IntVector) root.getVector("id");
+ assertEquals(1, idVector.get(0));
+
+ VarCharVector nameVector = (VarCharVector) root.getVector("name");
+ assertEquals("Bob", new String(nameVector.get(0)));
+
+ arrowReader.close();
+ }
+ }
+
+ @Test
+ public void testWriteRowSimple() throws Exception {
+ StructType schema = createSchemaWithoutMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_simple_write.lance");
+ HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false);
+
+ InternalRow row = createRow(1, "Charlie", 35L);
+ writer.writeRow("key3", row);
+ writer.close();
+
+ // Verify file exists and has correct record count
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator)) {
+ assertEquals(1, reader.numRows());
+ }
+ }
+
+ @Test
+ public void testSequenceIdGeneration() throws Exception {
+ StructType schema = createSchemaWithMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_sequence_id.lance");
+ HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, true);
+
+ // Write multiple records
+ for (int i = 0; i < 3; i++) {
+ InternalRow row = createRowWithMetaFields(i, "User" + i, 20L + i);
+ HoodieKey key = new HoodieKey("key" + i, "partition1");
+ writer.writeRowWithMetadata(key, row);
+ }
+ writer.close();
+
+ // Verify sequence IDs are unique and properly formatted
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator)) {
+
+ assertEquals(3, reader.numRows());
+
+ ArrowReader arrowReader = reader.readAll(null, null, Integer.MAX_VALUE);
+ arrowReader.loadNextBatch();
+ VectorSchemaRoot root = arrowReader.getVectorSchemaRoot();
+
+ VarCharVector seqNoVector = (VarCharVector) root.getVector(COMMIT_SEQNO_METADATA_FIELD.getFieldName());
+
+ List<String> seqNos = new ArrayList<>();
+ for (int i = 0; i < 3; i++) {
+ String seqNo = new String(seqNoVector.get(i));
+ assertTrue(seqNo.startsWith(instantTime + "_"), "Sequence ID should start with instantTime");
+ seqNos.add(seqNo);
+ }
+
+ // All sequence IDs should be unique
+ assertEquals(3, seqNos.stream().distinct().count(), "All sequence IDs should be unique");
+
+ arrowReader.close();
+ }
+ }
+
+ @Test
+ public void testBatchFlushing() throws Exception {
+ StructType schema = createSchemaWithoutMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_batch_flush.lance");
+ HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false);
+
+ // Write more than DEFAULT_BATCH_SIZE (1000) records
+ int recordCount = 2500;
+ for (int i = 0; i < recordCount; i++) {
+ InternalRow row = createRow(i, "User" + i, 20L + i);
+ writer.writeRow("key" + i, row);
+ }
+ writer.close();
+
+ // Verify all records were written
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator)) {
+ assertEquals(recordCount, reader.numRows(), "All records should be written");
+ }
+ }
+
+ @Test
+ public void testPrimitiveTypes() throws Exception {
+ StructType schema = new StructType()
+ .add("int_field", DataTypes.IntegerType, false)
+ .add("long_field", DataTypes.LongType, false)
+ .add("float_field", DataTypes.FloatType, false)
+ .add("double_field", DataTypes.DoubleType, false)
+ .add("bool_field", DataTypes.BooleanType, false)
+ .add("string_field", DataTypes.StringType, false)
+ .add("binary_field", DataTypes.BinaryType, false);
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_primitives.lance");
+ HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false);
+
+ GenericInternalRow row = new GenericInternalRow(new Object[]{
+ 42, // int
+ 123456789L, // long
+ 3.14f, // float
+ 2.71828, // double
+ true, // boolean
+ UTF8String.fromString("test"), // string
+ new byte[]{1, 2, 3, 4} // binary
+ });
+
+ writer.writeRow("key1", row);
+ writer.close();
+
+ // Verify all types were written correctly
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator)) {
+
+ assertEquals(1, reader.numRows());
+ assertEquals(7, reader.schema().getFields().size());
+
+ ArrowReader arrowReader = reader.readAll(null, null, Integer.MAX_VALUE);
+ arrowReader.loadNextBatch();
+ VectorSchemaRoot root = arrowReader.getVectorSchemaRoot();
+
+ assertEquals(42, ((IntVector) root.getVector("int_field")).get(0));
+ assertEquals(123456789L, ((BigIntVector) root.getVector("long_field")).get(0));
+ assertEquals(3.14f, ((Float4Vector) root.getVector("float_field")).get(0), 0.001);
+ assertEquals(2.71828, ((Float8Vector) root.getVector("double_field")).get(0), 0.00001);
+ assertEquals(1, ((BitVector) root.getVector("bool_field")).get(0));
+ assertEquals("test", new String(((VarCharVector) root.getVector("string_field")).get(0)));
+
+ byte[] binary = ((VarBinaryVector) root.getVector("binary_field")).get(0);
+ assertEquals(4, binary.length);
+
+ arrowReader.close();
+ }
+ }
+
+ @Test
+ public void testNullValues() throws Exception {
+ StructType schema = new StructType()
+ .add("id", DataTypes.IntegerType, false)
+ .add("name", DataTypes.StringType, true)
+ .add("age", DataTypes.LongType, true);
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_nulls.lance");
+ HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false);
+
+ // Write rows with null values
+ writer.writeRow("key1", createRow(1, "Alice", 30L));
+ writer.writeRow("key2", createRow(2, null, 25L)); // null name
+ writer.writeRow("key3", createRow(3, "Charlie", null)); // null age
+ writer.close();
+
+ // Verify nulls are preserved
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator)) {
+
+ assertEquals(3, reader.numRows());
+
+ ArrowReader arrowReader = reader.readAll(null, null, Integer.MAX_VALUE);
+ arrowReader.loadNextBatch();
+ VectorSchemaRoot root = arrowReader.getVectorSchemaRoot();
+
+ VarCharVector nameVector = (VarCharVector) root.getVector("name");
+ assertFalse(nameVector.isNull(0), "First name should not be null");
+ assertTrue(nameVector.isNull(1), "Second name should be null");
+ assertFalse(nameVector.isNull(2), "Third name should not be null");
+
+ BigIntVector ageVector = (BigIntVector) root.getVector("age");
+ assertFalse(ageVector.isNull(0), "First age should not be null");
+ assertFalse(ageVector.isNull(1), "Second age should not be null");
+ assertTrue(ageVector.isNull(2), "Third age should be null");
+
+ arrowReader.close();
+ }
+ }
+
+ @Test
+ public void testWriteEmptyDataset() throws Exception {
+ StructType schema = new StructType()
+ .add("id", DataTypes.IntegerType, false);
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_empty.lance");
+ HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false);
+ writer.close(); // Close without writing any rows
+
+ // Lance doesn't create a file if no data is written
+ assertFalse(storage.exists(path), "Lance file should not exist when no data is written");
+ }
+
+ @Test
+ public void testWriteStructType() throws Exception {
+ // Create schema with nested struct
+ StructType addressSchema = new StructType()
+ .add("street", DataTypes.StringType, true)
+ .add("city", DataTypes.StringType, true)
+ .add("zipcode", DataTypes.IntegerType, true);
+
+ StructType schema = new StructType()
+ .add("id", DataTypes.IntegerType, false)
+ .add("name", DataTypes.StringType, true)
+ .add("address", addressSchema, true);
+
+ // Create test data with nested struct
+ GenericInternalRow address1 = new GenericInternalRow(new Object[]{
+ UTF8String.fromString("123 Main St"),
+ UTF8String.fromString("New York"),
+ 10001
+ });
+
+ GenericInternalRow address2 = new GenericInternalRow(new Object[]{
+ UTF8String.fromString("456 Oak Ave"),
+ UTF8String.fromString("Los Angeles"),
+ 90001
+ });
+
+ List<InternalRow> rows = new ArrayList<>();
+ rows.add(new GenericInternalRow(new Object[]{1, UTF8String.fromString("Alice"), address1}));
+ rows.add(new GenericInternalRow(new Object[]{2, UTF8String.fromString("Bob"), address2}));
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_struct.lance");
+ HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false);
+
+ for (InternalRow row : rows) {
+ writer.writeRow("key" + rows.indexOf(row), row);
+ }
+ writer.close();
+
+ assertTrue(storage.exists(path), "Lance file with struct type should exist");
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator)) {
+ assertEquals(rows.size(), reader.numRows(), "Row count should match");
+ assertEquals(3, reader.schema().getFields().size(), "Should have 3 top-level fields (id, name, address)");
+ }
+ }
+
+ // Helper methods
+
+ private StructType createSchemaWithMetaFields() {
+ return new StructType()
+ .add(COMMIT_TIME_METADATA_FIELD.getFieldName(), DataTypes.StringType, false)
+ .add(COMMIT_SEQNO_METADATA_FIELD.getFieldName(), DataTypes.StringType, false)
+ .add(RECORD_KEY_METADATA_FIELD.getFieldName(), DataTypes.StringType, false)
+ .add(PARTITION_PATH_METADATA_FIELD.getFieldName(), DataTypes.StringType, false)
+ .add(FILENAME_METADATA_FIELD.getFieldName(), DataTypes.StringType, false)
+ .add("id", DataTypes.IntegerType, false)
+ .add("name", DataTypes.StringType, true)
+ .add("age", DataTypes.LongType, true);
+ }
+
+ private StructType createSchemaWithoutMetaFields() {
+ return new StructType()
+ .add("id", DataTypes.IntegerType, false)
+ .add("name", DataTypes.StringType, true)
+ .add("age", DataTypes.LongType, true);
+ }
+
+ private InternalRow createRowWithMetaFields(Object... userValues) {
+ // Create row with PLACEHOLDER meta fields (will be updated by writer) + user data
+ Object[] allValues = new Object[5 + userValues.length];
+
+ // Meta fields - use empty strings as placeholders
+ allValues[0] = UTF8String.fromString(""); // commit_time
+ allValues[1] = UTF8String.fromString(""); // commit_seqno
+ allValues[2] = UTF8String.fromString(""); // record_key
+ allValues[3] = UTF8String.fromString(""); // partition_path
+ allValues[4] = UTF8String.fromString(""); // file_name
+
+ // Copy user values
+ for (int i = 0; i < userValues.length; i++) {
+ allValues[5 + i] = processValue(userValues[i]);
+ }
+
+ return new GenericInternalRow(allValues);
+ }
+
+ private InternalRow createRow(Object... values) {
+ Object[] processedValues = new Object[values.length];
+ for (int i = 0; i < values.length; i++) {
+ processedValues[i] = processValue(values[i]);
+ }
+ return new GenericInternalRow(processedValues);
+ }
+
+ private Object processValue(Object value) {
+ if (value instanceof String) {
+ return UTF8String.fromString((String) value);
+ }
+ return value;
+ }
+
+ private boolean hasField(VectorSchemaRoot root, String fieldName) {
+ try {
+ return root.getVector(fieldName) != null;
+ } catch (Exception e) {
+ return false;
+ }
+ }
+}
Review Comment:
nitpick: add trailing newline
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add HoodieBaseLanceFileWriter and implementation for SparkFileW… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17629:
URL: https://github.com/apache/hudi/pull/17629#issuecomment-3678800888
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9633e6bc476078ebf78b401e00ccbf43cdab67e8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10326",
"triggerID" : "9633e6bc476078ebf78b401e00ccbf43cdab67e8",
"triggerType" : "PUSH"
}, {
"hash" : "6adcb20a0c841eb4164570bed7e3daa28349be35",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10335",
"triggerID" : "6adcb20a0c841eb4164570bed7e3daa28349be35",
"triggerType" : "PUSH"
}, {
"hash" : "f0f38ca10ad7bd5db724f5568aa560e0c2af8539",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10401",
"triggerID" : "f0f38ca10ad7bd5db724f5568aa560e0c2af8539",
"triggerType" : "PUSH"
}, {
"hash" : "3e18561eb435a7b4e5452c2df03793db7e70ad82",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3e18561eb435a7b4e5452c2df03793db7e70ad82",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f0f38ca10ad7bd5db724f5568aa560e0c2af8539 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10401)
* 3e18561eb435a7b4e5452c2df03793db7e70ad82 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add HoodieBaseLanceFileWriter and implementation for SparkFileW… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17629:
URL: https://github.com/apache/hudi/pull/17629#issuecomment-3679749400
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9633e6bc476078ebf78b401e00ccbf43cdab67e8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10326",
"triggerID" : "9633e6bc476078ebf78b401e00ccbf43cdab67e8",
"triggerType" : "PUSH"
}, {
"hash" : "6adcb20a0c841eb4164570bed7e3daa28349be35",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10335",
"triggerID" : "6adcb20a0c841eb4164570bed7e3daa28349be35",
"triggerType" : "PUSH"
}, {
"hash" : "f0f38ca10ad7bd5db724f5568aa560e0c2af8539",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10401",
"triggerID" : "f0f38ca10ad7bd5db724f5568aa560e0c2af8539",
"triggerType" : "PUSH"
}, {
"hash" : "3e18561eb435a7b4e5452c2df03793db7e70ad82",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10420",
"triggerID" : "3e18561eb435a7b4e5452c2df03793db7e70ad82",
"triggerType" : "PUSH"
}, {
"hash" : "33cfffa5175585e24bd00025a01f3bb4f5abf3fb",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10424",
"triggerID" : "33cfffa5175585e24bd00025a01f3bb4f5abf3fb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3e18561eb435a7b4e5452c2df03793db7e70ad82 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10420)
* 33cfffa5175585e24bd00025a01f3bb4f5abf3fb Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10424)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add HoodieBaseLanceFileWriter and implementation for SparkFileW… [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #17629:
URL: https://github.com/apache/hudi/pull/17629#discussion_r2638157554
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/HoodieSparkLanceWriter.java:
##########
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.io.storage;
+
+import com.lancedb.lance.spark.arrow.LanceArrowWriter;
+import org.apache.arrow.vector.types.pojo.Schema;
+import org.apache.hudi.common.engine.TaskContextSupplier;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.io.lance.HoodieBaseLanceWriter;
+import org.apache.hudi.storage.HoodieStorage;
+import org.apache.hudi.storage.StoragePath;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.util.LanceArrowUtils;
+import org.apache.spark.unsafe.types.UTF8String;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.function.Function;
+
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.COMMIT_SEQNO_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.COMMIT_TIME_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.FILENAME_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.PARTITION_PATH_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.RECORD_KEY_METADATA_FIELD;
+
+/**
+ * Spark Lance file writer implementing {@link HoodieSparkFileWriter}.
+ *
+ * This writer integrates with Hudi's storage I/O layer and supports:
+ * - Hudi metadata field population
+ * - Record key tracking (for bloom filters - TODO)
Review Comment:
For any TODO, let's include a link to the GH issue for the task
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add HoodieBaseLanceFileWriter and implementation for SparkFileW… [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17629:
URL: https://github.com/apache/hudi/pull/17629#discussion_r2627679786
##########
pom.xml:
##########
@@ -238,6 +238,11 @@
<springboot.version>2.7.10</springboot.version>
<spring.shell.version>2.1.1</spring.shell.version>
<snappy.version>1.1.10.7</snappy.version>
+ <arrow.version>18.3.0</arrow.version>
Review Comment:
can check current versions for arrow and lance now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add HoodieBaseLanceFileWriter and implementation for SparkFileW… [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #17629:
URL: https://github.com/apache/hudi/pull/17629#discussion_r2638291844
##########
hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/io/storage/TestHoodieSparkLanceWriter.java:
##########
@@ -0,0 +1,454 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.io.storage;
+
+import com.lancedb.lance.file.LanceFileReader;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.memory.RootAllocator;
+import org.apache.arrow.vector.BigIntVector;
+import org.apache.arrow.vector.BitVector;
+import org.apache.arrow.vector.Float4Vector;
+import org.apache.arrow.vector.Float8Vector;
+import org.apache.arrow.vector.IntVector;
+import org.apache.arrow.vector.VarBinaryVector;
+import org.apache.arrow.vector.VarCharVector;
+import org.apache.arrow.vector.VectorSchemaRoot;
+import org.apache.arrow.vector.ipc.ArrowReader;
+import org.apache.hudi.client.SparkTaskContextSupplier;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.testutils.HoodieTestUtils;
+import org.apache.hudi.storage.HoodieStorage;
+import org.apache.hudi.storage.StoragePath;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.catalyst.expressions.GenericInternalRow;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.unsafe.types.UTF8String;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.condition.DisabledIfSystemProperty;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.COMMIT_SEQNO_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.COMMIT_TIME_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.FILENAME_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.PARTITION_PATH_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.RECORD_KEY_METADATA_FIELD;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * Tests for {@link HoodieSparkLanceWriter}.
+ */
+@DisabledIfSystemProperty(named = "lance.skip.tests", matches = "true")
+public class TestHoodieSparkLanceWriter {
+
+ @TempDir
+ File tempDir;
+
+ private HoodieStorage storage;
+ private SparkTaskContextSupplier taskContextSupplier;
+ private String instantTime;
+
+ @BeforeEach
+ public void setUp() throws IOException {
+ storage = HoodieTestUtils.getStorage(tempDir.getAbsolutePath());
+ taskContextSupplier = new SparkTaskContextSupplier();
+ instantTime = "20251201120000000";
+ }
+
+ @AfterEach
+ public void tearDown() throws Exception {
+ if (storage != null) {
+ storage.close();
+ }
+ }
+
+ @Test
+ public void testWriteRowWithMetadataPopulation() throws Exception {
+ // Schema WITH meta fields (writer expects this when populateMetaFields=true)
+ StructType schema = createSchemaWithMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_with_metadata.lance");
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, true)) {
+ // Write multiple records to test metadata population and sequence ID generation
+ for (int i = 0; i < 3; i++) {
+ InternalRow row = createRowWithMetaFields(i, "User" + i, 20L + i);
+ HoodieKey key = new HoodieKey("key" + i, "partition1");
+ writer.writeRowWithMetadata(key, row);
+ }
+ }
+
+ // Verify using LanceFileReader
+ assertTrue(storage.exists(path), "Lance file should exist");
+
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator);
+ ArrowReader arrowReader = reader.readAll(null, null, Integer.MAX_VALUE)) {
+
+ assertEquals(3, reader.numRows(), "Should have 3 records");
+
+ // Schema should have 5 meta fields + 3 user fields = 8 fields
+ assertEquals(8, reader.schema().getFields().size(), "Should have 8 fields");
+
+ // Read and verify data
+ assertTrue(arrowReader.loadNextBatch(), "Should load batch");
+ VectorSchemaRoot root = arrowReader.getVectorSchemaRoot();
+
+ // Verify meta fields were populated for first record
+ VarCharVector commitTimeVector = (VarCharVector) root.getVector(COMMIT_TIME_METADATA_FIELD.getFieldName());
+ assertNotNull(commitTimeVector);
+ assertEquals(instantTime, new String(commitTimeVector.get(0)), "Commit time should match");
+
+ VarCharVector recordKeyVector = (VarCharVector) root.getVector(RECORD_KEY_METADATA_FIELD.getFieldName());
+ assertEquals("key0", new String(recordKeyVector.get(0)), "Record key should match");
+
+ VarCharVector partitionPathVector = (VarCharVector) root.getVector(PARTITION_PATH_METADATA_FIELD.getFieldName());
+ assertEquals("partition1", new String(partitionPathVector.get(0)), "Partition path should match");
+
+ VarCharVector fileNameVector = (VarCharVector) root.getVector(FILENAME_METADATA_FIELD.getFieldName());
+ assertEquals(path.getName(), new String(fileNameVector.get(0)), "File name should match");
+
+ // Verify sequence IDs are unique and properly formatted
+ VarCharVector seqNoVector = (VarCharVector) root.getVector(COMMIT_SEQNO_METADATA_FIELD.getFieldName());
+ List<String> seqNos = new ArrayList<>();
+ for (int i = 0; i < 3; i++) {
+ String seqNo = new String(seqNoVector.get(i));
+ assertTrue(seqNo.startsWith(instantTime + "_"), "Sequence number should start with instant time");
+ seqNos.add(seqNo);
+ }
+ assertEquals(3, seqNos.stream().distinct().count(), "All sequence IDs should be unique");
+
+ // Verify user data fields for first record
+ IntVector idVector = (IntVector) root.getVector("id");
+ assertEquals(0, idVector.get(0), "ID should match");
+
+ VarCharVector nameVector = (VarCharVector) root.getVector("name");
+ assertEquals("User0", new String(nameVector.get(0)), "Name should match");
+
+ BigIntVector ageVector = (BigIntVector) root.getVector("age");
+ assertEquals(20L, ageVector.get(0), "Age should match");
+ }
+ }
+
+ @Test
+ public void testWriteRowWithoutMetadataPopulation() throws Exception {
+ // Schema WITHOUT meta fields
+ StructType schema = createSchemaWithoutMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_without_metadata.lance");
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false)) {
+ // Create row with just user data (no meta fields)
+ InternalRow row = createRow(1, "Bob", 25L);
+ HoodieKey key = new HoodieKey("key2", "partition2");
+
+ writer.writeRowWithMetadata(key, row);
+ }
+
+ // Verify using LanceFileReader
+ assertTrue(storage.exists(path));
+
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator);
+ ArrowReader arrowReader = reader.readAll(null, null, Integer.MAX_VALUE)) {
+
+ assertEquals(1, reader.numRows());
+
+ // Schema should have ONLY 3 user fields (no meta fields)
+ assertEquals(3, reader.schema().getFields().size(), "Should have only user fields");
+
+ // Read and verify data
+ assertTrue(arrowReader.loadNextBatch());
+ VectorSchemaRoot root = arrowReader.getVectorSchemaRoot();
+
+ // Verify NO meta fields exist
+ assertFalse(hasField(root, COMMIT_TIME_METADATA_FIELD.getFieldName()), "Should not have commit time field");
+ assertFalse(hasField(root, RECORD_KEY_METADATA_FIELD.getFieldName()), "Should not have record key field");
+
+ // Verify user data
+ IntVector idVector = (IntVector) root.getVector("id");
+ assertEquals(1, idVector.get(0));
+
+ VarCharVector nameVector = (VarCharVector) root.getVector("name");
+ assertEquals("Bob", new String(nameVector.get(0)));
+ }
+ }
+
+ @Test
+ public void testWriteRowSimple() throws Exception {
+ StructType schema = createSchemaWithoutMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_simple_write.lance");
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false)) {
+ InternalRow row = createRow(1, "Charlie", 35L);
+ writer.writeRow("key3", row);
+ }
+
+ // Verify file exists and has correct record count
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator)) {
+ assertEquals(1, reader.numRows());
+ }
+ }
+
+ @Test
+ public void testBatchFlushing() throws Exception {
+ StructType schema = createSchemaWithoutMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_batch_flush.lance");
+ // Write more than DEFAULT_BATCH_SIZE (1000) records
+ int recordCount = 2500;
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false)) {
+ for (int i = 0; i < recordCount; i++) {
+ InternalRow row = createRow(i, "User" + i, 20L + i);
+ writer.writeRow("key" + i, row);
+ }
+ }
+
+ // Verify all records were written
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator)) {
+ assertEquals(recordCount, reader.numRows(), "All records should be written");
+ }
+ }
+
+ @Test
+ public void testPrimitiveTypes() throws Exception {
+ StructType schema = new StructType()
+ .add("int_field", DataTypes.IntegerType, false)
+ .add("long_field", DataTypes.LongType, false)
+ .add("float_field", DataTypes.FloatType, false)
+ .add("double_field", DataTypes.DoubleType, false)
+ .add("bool_field", DataTypes.BooleanType, false)
+ .add("string_field", DataTypes.StringType, false)
+ .add("binary_field", DataTypes.BinaryType, false);
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_primitives.lance");
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false)) {
+ GenericInternalRow row = new GenericInternalRow(new Object[]{
+ 42, // int
+ 123456789L, // long
+ 3.14f, // float
+ 2.71828, // double
+ true, // boolean
+ UTF8String.fromString("test"), // string
+ new byte[]{1, 2, 3, 4} // binary
+ });
+
+ writer.writeRow("key1", row);
+ }
+
+ // Verify all types were written correctly
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator);
+ ArrowReader arrowReader = reader.readAll(null, null, Integer.MAX_VALUE)) {
+
+ assertEquals(1, reader.numRows());
+ assertEquals(7, reader.schema().getFields().size());
+
+ arrowReader.loadNextBatch();
+ VectorSchemaRoot root = arrowReader.getVectorSchemaRoot();
+
+ assertEquals(42, ((IntVector) root.getVector("int_field")).get(0));
+ assertEquals(123456789L, ((BigIntVector) root.getVector("long_field")).get(0));
+ assertEquals(3.14f, ((Float4Vector) root.getVector("float_field")).get(0), 0.001);
+ assertEquals(2.71828, ((Float8Vector) root.getVector("double_field")).get(0), 0.00001);
+ assertEquals(1, ((BitVector) root.getVector("bool_field")).get(0));
+ assertEquals("test", new String(((VarCharVector) root.getVector("string_field")).get(0)));
+
+ byte[] binary = ((VarBinaryVector) root.getVector("binary_field")).get(0);
+ assertEquals(4, binary.length);
+ }
+ }
+
+ @Test
+ public void testNullValues() throws Exception {
+ StructType schema = new StructType()
+ .add("id", DataTypes.IntegerType, false)
+ .add("name", DataTypes.StringType, true)
+ .add("age", DataTypes.LongType, true);
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_nulls.lance");
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false)) {
+ // Write rows with null values
+ writer.writeRow("key1", createRow(1, "Alice", 30L));
+ writer.writeRow("key2", createRow(2, null, 25L)); // null name
+ writer.writeRow("key3", createRow(3, "Charlie", null)); // null age
+ }
+
+ // Verify nulls are preserved
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator);
+ ArrowReader arrowReader = reader.readAll(null, null, Integer.MAX_VALUE)) {
+
+ assertEquals(3, reader.numRows());
+
+ arrowReader.loadNextBatch();
+ VectorSchemaRoot root = arrowReader.getVectorSchemaRoot();
+
+ VarCharVector nameVector = (VarCharVector) root.getVector("name");
+ assertFalse(nameVector.isNull(0), "First name should not be null");
+ assertTrue(nameVector.isNull(1), "Second name should be null");
+ assertFalse(nameVector.isNull(2), "Third name should not be null");
+
+ BigIntVector ageVector = (BigIntVector) root.getVector("age");
+ assertFalse(ageVector.isNull(0), "First age should not be null");
+ assertFalse(ageVector.isNull(1), "Second age should not be null");
+ assertTrue(ageVector.isNull(2), "Third age should be null");
+ }
+ }
+
+ @Test
+ public void testWriteEmptyDataset() throws Exception {
+ StructType schema = new StructType()
+ .add("id", DataTypes.IntegerType, false);
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_empty.lance");
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false)) {
+ // Close without writing any rows
+ }
+
+ // Lance doesn't create a file if no data is written
+ assertFalse(storage.exists(path), "Lance file should not exist when no data is written");
Review Comment:
@yihua can you confirm that this is the expected behavior? I remember some case where we produce an empty file when all records are deleted in a compaction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add HoodieBaseLanceFileWriter and implementation for SparkFileW… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17629:
URL: https://github.com/apache/hudi/pull/17629#issuecomment-3678880636
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9633e6bc476078ebf78b401e00ccbf43cdab67e8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10326",
"triggerID" : "9633e6bc476078ebf78b401e00ccbf43cdab67e8",
"triggerType" : "PUSH"
}, {
"hash" : "6adcb20a0c841eb4164570bed7e3daa28349be35",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10335",
"triggerID" : "6adcb20a0c841eb4164570bed7e3daa28349be35",
"triggerType" : "PUSH"
}, {
"hash" : "f0f38ca10ad7bd5db724f5568aa560e0c2af8539",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10401",
"triggerID" : "f0f38ca10ad7bd5db724f5568aa560e0c2af8539",
"triggerType" : "PUSH"
}, {
"hash" : "3e18561eb435a7b4e5452c2df03793db7e70ad82",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10420",
"triggerID" : "3e18561eb435a7b4e5452c2df03793db7e70ad82",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3e18561eb435a7b4e5452c2df03793db7e70ad82 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10420)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add HoodieBaseLanceFileWriter and implementation for SparkFileW… [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17629:
URL: https://github.com/apache/hudi/pull/17629#discussion_r2641898018
##########
hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/io/storage/TestHoodieSparkLanceWriter.java:
##########
@@ -0,0 +1,454 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.io.storage;
+
+import com.lancedb.lance.file.LanceFileReader;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.memory.RootAllocator;
+import org.apache.arrow.vector.BigIntVector;
+import org.apache.arrow.vector.BitVector;
+import org.apache.arrow.vector.Float4Vector;
+import org.apache.arrow.vector.Float8Vector;
+import org.apache.arrow.vector.IntVector;
+import org.apache.arrow.vector.VarBinaryVector;
+import org.apache.arrow.vector.VarCharVector;
+import org.apache.arrow.vector.VectorSchemaRoot;
+import org.apache.arrow.vector.ipc.ArrowReader;
+import org.apache.hudi.client.SparkTaskContextSupplier;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.testutils.HoodieTestUtils;
+import org.apache.hudi.storage.HoodieStorage;
+import org.apache.hudi.storage.StoragePath;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.catalyst.expressions.GenericInternalRow;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.unsafe.types.UTF8String;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.condition.DisabledIfSystemProperty;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.COMMIT_SEQNO_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.COMMIT_TIME_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.FILENAME_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.PARTITION_PATH_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.RECORD_KEY_METADATA_FIELD;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * Tests for {@link HoodieSparkLanceWriter}.
+ */
+@DisabledIfSystemProperty(named = "lance.skip.tests", matches = "true")
+public class TestHoodieSparkLanceWriter {
+
+ @TempDir
+ File tempDir;
+
+ private HoodieStorage storage;
+ private SparkTaskContextSupplier taskContextSupplier;
+ private String instantTime;
+
+ @BeforeEach
+ public void setUp() throws IOException {
+ storage = HoodieTestUtils.getStorage(tempDir.getAbsolutePath());
+ taskContextSupplier = new SparkTaskContextSupplier();
+ instantTime = "20251201120000000";
+ }
+
+ @AfterEach
+ public void tearDown() throws Exception {
+ if (storage != null) {
+ storage.close();
+ }
+ }
+
+ @Test
+ public void testWriteRowWithMetadataPopulation() throws Exception {
+ // Schema WITH meta fields (writer expects this when populateMetaFields=true)
+ StructType schema = createSchemaWithMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_with_metadata.lance");
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, true)) {
+ // Write multiple records to test metadata population and sequence ID generation
+ for (int i = 0; i < 3; i++) {
+ InternalRow row = createRowWithMetaFields(i, "User" + i, 20L + i);
+ HoodieKey key = new HoodieKey("key" + i, "partition1");
+ writer.writeRowWithMetadata(key, row);
+ }
+ }
+
+ // Verify using LanceFileReader
+ assertTrue(storage.exists(path), "Lance file should exist");
+
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator);
+ ArrowReader arrowReader = reader.readAll(null, null, Integer.MAX_VALUE)) {
+
+ assertEquals(3, reader.numRows(), "Should have 3 records");
+
+ // Schema should have 5 meta fields + 3 user fields = 8 fields
+ assertEquals(8, reader.schema().getFields().size(), "Should have 8 fields");
+
+ // Read and verify data
+ assertTrue(arrowReader.loadNextBatch(), "Should load batch");
+ VectorSchemaRoot root = arrowReader.getVectorSchemaRoot();
+
+ // Verify meta fields were populated for first record
+ VarCharVector commitTimeVector = (VarCharVector) root.getVector(COMMIT_TIME_METADATA_FIELD.getFieldName());
+ assertNotNull(commitTimeVector);
+ assertEquals(instantTime, new String(commitTimeVector.get(0)), "Commit time should match");
+
+ VarCharVector recordKeyVector = (VarCharVector) root.getVector(RECORD_KEY_METADATA_FIELD.getFieldName());
+ assertEquals("key0", new String(recordKeyVector.get(0)), "Record key should match");
+
+ VarCharVector partitionPathVector = (VarCharVector) root.getVector(PARTITION_PATH_METADATA_FIELD.getFieldName());
+ assertEquals("partition1", new String(partitionPathVector.get(0)), "Partition path should match");
+
+ VarCharVector fileNameVector = (VarCharVector) root.getVector(FILENAME_METADATA_FIELD.getFieldName());
+ assertEquals(path.getName(), new String(fileNameVector.get(0)), "File name should match");
+
+ // Verify sequence IDs are unique and properly formatted
+ VarCharVector seqNoVector = (VarCharVector) root.getVector(COMMIT_SEQNO_METADATA_FIELD.getFieldName());
+ List<String> seqNos = new ArrayList<>();
+ for (int i = 0; i < 3; i++) {
+ String seqNo = new String(seqNoVector.get(i));
+ assertTrue(seqNo.startsWith(instantTime + "_"), "Sequence number should start with instant time");
+ seqNos.add(seqNo);
+ }
+ assertEquals(3, seqNos.stream().distinct().count(), "All sequence IDs should be unique");
+
+ // Verify user data fields for first record
+ IntVector idVector = (IntVector) root.getVector("id");
+ assertEquals(0, idVector.get(0), "ID should match");
+
+ VarCharVector nameVector = (VarCharVector) root.getVector("name");
+ assertEquals("User0", new String(nameVector.get(0)), "Name should match");
+
+ BigIntVector ageVector = (BigIntVector) root.getVector("age");
+ assertEquals(20L, ageVector.get(0), "Age should match");
+ }
+ }
+
+ @Test
+ public void testWriteRowWithoutMetadataPopulation() throws Exception {
+ // Schema WITHOUT meta fields
+ StructType schema = createSchemaWithoutMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_without_metadata.lance");
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false)) {
+ // Create row with just user data (no meta fields)
+ InternalRow row = createRow(1, "Bob", 25L);
+ HoodieKey key = new HoodieKey("key2", "partition2");
+
+ writer.writeRowWithMetadata(key, row);
+ }
+
+ // Verify using LanceFileReader
+ assertTrue(storage.exists(path));
+
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator);
+ ArrowReader arrowReader = reader.readAll(null, null, Integer.MAX_VALUE)) {
+
+ assertEquals(1, reader.numRows());
+
+ // Schema should have ONLY 3 user fields (no meta fields)
+ assertEquals(3, reader.schema().getFields().size(), "Should have only user fields");
+
+ // Read and verify data
+ assertTrue(arrowReader.loadNextBatch());
+ VectorSchemaRoot root = arrowReader.getVectorSchemaRoot();
+
+ // Verify NO meta fields exist
+ assertFalse(hasField(root, COMMIT_TIME_METADATA_FIELD.getFieldName()), "Should not have commit time field");
+ assertFalse(hasField(root, RECORD_KEY_METADATA_FIELD.getFieldName()), "Should not have record key field");
+
+ // Verify user data
+ IntVector idVector = (IntVector) root.getVector("id");
+ assertEquals(1, idVector.get(0));
+
+ VarCharVector nameVector = (VarCharVector) root.getVector("name");
+ assertEquals("Bob", new String(nameVector.get(0)));
+ }
+ }
+
+ @Test
+ public void testWriteRowSimple() throws Exception {
+ StructType schema = createSchemaWithoutMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_simple_write.lance");
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false)) {
+ InternalRow row = createRow(1, "Charlie", 35L);
+ writer.writeRow("key3", row);
+ }
+
+ // Verify file exists and has correct record count
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator)) {
+ assertEquals(1, reader.numRows());
+ }
+ }
+
+ @Test
+ public void testBatchFlushing() throws Exception {
+ StructType schema = createSchemaWithoutMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_batch_flush.lance");
+ // Write more than DEFAULT_BATCH_SIZE (1000) records
+ int recordCount = 2500;
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false)) {
+ for (int i = 0; i < recordCount; i++) {
+ InternalRow row = createRow(i, "User" + i, 20L + i);
+ writer.writeRow("key" + i, row);
+ }
+ }
+
+ // Verify all records were written
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator)) {
+ assertEquals(recordCount, reader.numRows(), "All records should be written");
+ }
+ }
+
+ @Test
+ public void testPrimitiveTypes() throws Exception {
+ StructType schema = new StructType()
+ .add("int_field", DataTypes.IntegerType, false)
+ .add("long_field", DataTypes.LongType, false)
+ .add("float_field", DataTypes.FloatType, false)
+ .add("double_field", DataTypes.DoubleType, false)
+ .add("bool_field", DataTypes.BooleanType, false)
+ .add("string_field", DataTypes.StringType, false)
+ .add("binary_field", DataTypes.BinaryType, false);
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_primitives.lance");
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false)) {
+ GenericInternalRow row = new GenericInternalRow(new Object[]{
+ 42, // int
+ 123456789L, // long
+ 3.14f, // float
+ 2.71828, // double
+ true, // boolean
+ UTF8String.fromString("test"), // string
+ new byte[]{1, 2, 3, 4} // binary
+ });
+
+ writer.writeRow("key1", row);
+ }
+
+ // Verify all types were written correctly
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator);
+ ArrowReader arrowReader = reader.readAll(null, null, Integer.MAX_VALUE)) {
+
+ assertEquals(1, reader.numRows());
+ assertEquals(7, reader.schema().getFields().size());
+
+ arrowReader.loadNextBatch();
+ VectorSchemaRoot root = arrowReader.getVectorSchemaRoot();
+
+ assertEquals(42, ((IntVector) root.getVector("int_field")).get(0));
+ assertEquals(123456789L, ((BigIntVector) root.getVector("long_field")).get(0));
+ assertEquals(3.14f, ((Float4Vector) root.getVector("float_field")).get(0), 0.001);
+ assertEquals(2.71828, ((Float8Vector) root.getVector("double_field")).get(0), 0.00001);
+ assertEquals(1, ((BitVector) root.getVector("bool_field")).get(0));
+ assertEquals("test", new String(((VarCharVector) root.getVector("string_field")).get(0)));
+
+ byte[] binary = ((VarBinaryVector) root.getVector("binary_field")).get(0);
+ assertEquals(4, binary.length);
+ }
+ }
+
+ @Test
+ public void testNullValues() throws Exception {
+ StructType schema = new StructType()
+ .add("id", DataTypes.IntegerType, false)
+ .add("name", DataTypes.StringType, true)
+ .add("age", DataTypes.LongType, true);
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_nulls.lance");
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false)) {
+ // Write rows with null values
+ writer.writeRow("key1", createRow(1, "Alice", 30L));
+ writer.writeRow("key2", createRow(2, null, 25L)); // null name
+ writer.writeRow("key3", createRow(3, "Charlie", null)); // null age
+ }
+
+ // Verify nulls are preserved
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator);
+ ArrowReader arrowReader = reader.readAll(null, null, Integer.MAX_VALUE)) {
+
+ assertEquals(3, reader.numRows());
+
+ arrowReader.loadNextBatch();
+ VectorSchemaRoot root = arrowReader.getVectorSchemaRoot();
+
+ VarCharVector nameVector = (VarCharVector) root.getVector("name");
+ assertFalse(nameVector.isNull(0), "First name should not be null");
+ assertTrue(nameVector.isNull(1), "Second name should be null");
+ assertFalse(nameVector.isNull(2), "Third name should not be null");
+
+ BigIntVector ageVector = (BigIntVector) root.getVector("age");
+ assertFalse(ageVector.isNull(0), "First age should not be null");
+ assertFalse(ageVector.isNull(1), "Second age should not be null");
+ assertTrue(ageVector.isNull(2), "Third age should be null");
+ }
+ }
+
+ @Test
+ public void testWriteEmptyDataset() throws Exception {
+ StructType schema = new StructType()
+ .add("id", DataTypes.IntegerType, false);
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_empty.lance");
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false)) {
+ // Close without writing any rows
+ }
+
+ // Lance doesn't create a file if no data is written
+ assertFalse(storage.exists(path), "Lance file should not exist when no data is written");
Review Comment:
Following @the-other-tim-brown recommendation I tried writing an equivalent of this test above, but using the `HoodieSparkParquetWriter`
https://gist.github.com/rahil-c/f1251f999df7674077153acdf37e610d
However when doing so i ran into the following issue before seeing an empty file created
```
org.apache.hudi.exception.HoodieException: Unable to instantiate class org.apache.hudi.io.storage.row.HoodieRowParquetWriteSupport
at org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:75)
at org.apache.hudi.io.storage.row.HoodieRowParquetWriteSupport.getHoodieRowParquetWriteSupport(HoodieRowParquetWriteSupport.java:416)
at org.apache.hudi.io.storage.TestHoodieSparkParquetWriter.testWriteEmptyDataset(TestHoodieSparkParquetWriter.java:87)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
Caused by: java.lang.reflect.InvocationTargetException
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
at org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:73)
... 5 more
Caused by: java.lang.IllegalArgumentException: Schema string cannot be null or empty
at org.apache.hudi.common.util.ValidationUtils.checkArgument(ValidationUtils.java:42)
at org.apache.hudi.common.schema.HoodieSchema$Parser.parse(HoodieSchema.java:893)
at org.apache.hudi.common.schema.HoodieSchema.parse(HoodieSchema.java:138)
at org.apache.hudi.io.storage.row.HoodieRowParquetWriteSupport.lambda$new$2(HoodieRowParquetWriteSupport.java:138)
at org.apache.hudi.common.util.Option.orElseGet(Option.java:153)
at org.apache.hudi.io.storage.row.HoodieRowParquetWriteSupport.<init>(HoodieRowParquetWriteSupport.java:136)
... 10 more
```
I then discussed with @yihua offline, who pointed me to see if `HoodieAvroParquetWriter` creates an empty file. When writing a similar test in test `TestHoodieAvroParquetWriter` : https://gist.github.com/rahil-c/0b719717f72eb59dacdd646c2802309b
was able to confirm that an empty file is created despite no actual data being rows being written.
When inspecting the underlying parquet file that was created from the test I see the following, where the empty file persists the schema.
<img width="1548" height="274" alt="Screenshot 2025-12-22 at 11 27 32 PM" src="https://github.com/user-attachments/assets/648138ec-6d46-45dd-91fd-a071ea0a5481" />
@the-other-tim-brown @yihua
So to confirm is the behavior we want that we want to achieve with an empty lance file in case there is no data present is that we just want to write the schema? The lance java file writer does not have any apis writing to the footer similar to parquet, https://github.com/lance-format/lance/blob/main/java/src/main/java/org/lance/file/LanceFileWriter.java#L31 so need to investigate this more.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add HoodieBaseLanceFileWriter and implementation for SparkFileW… [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17629:
URL: https://github.com/apache/hudi/pull/17629#discussion_r2641898018
##########
hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/io/storage/TestHoodieSparkLanceWriter.java:
##########
@@ -0,0 +1,454 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.io.storage;
+
+import com.lancedb.lance.file.LanceFileReader;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.memory.RootAllocator;
+import org.apache.arrow.vector.BigIntVector;
+import org.apache.arrow.vector.BitVector;
+import org.apache.arrow.vector.Float4Vector;
+import org.apache.arrow.vector.Float8Vector;
+import org.apache.arrow.vector.IntVector;
+import org.apache.arrow.vector.VarBinaryVector;
+import org.apache.arrow.vector.VarCharVector;
+import org.apache.arrow.vector.VectorSchemaRoot;
+import org.apache.arrow.vector.ipc.ArrowReader;
+import org.apache.hudi.client.SparkTaskContextSupplier;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.testutils.HoodieTestUtils;
+import org.apache.hudi.storage.HoodieStorage;
+import org.apache.hudi.storage.StoragePath;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.catalyst.expressions.GenericInternalRow;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.unsafe.types.UTF8String;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.condition.DisabledIfSystemProperty;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.COMMIT_SEQNO_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.COMMIT_TIME_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.FILENAME_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.PARTITION_PATH_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.RECORD_KEY_METADATA_FIELD;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * Tests for {@link HoodieSparkLanceWriter}.
+ */
+@DisabledIfSystemProperty(named = "lance.skip.tests", matches = "true")
+public class TestHoodieSparkLanceWriter {
+
+ @TempDir
+ File tempDir;
+
+ private HoodieStorage storage;
+ private SparkTaskContextSupplier taskContextSupplier;
+ private String instantTime;
+
+ @BeforeEach
+ public void setUp() throws IOException {
+ storage = HoodieTestUtils.getStorage(tempDir.getAbsolutePath());
+ taskContextSupplier = new SparkTaskContextSupplier();
+ instantTime = "20251201120000000";
+ }
+
+ @AfterEach
+ public void tearDown() throws Exception {
+ if (storage != null) {
+ storage.close();
+ }
+ }
+
+ @Test
+ public void testWriteRowWithMetadataPopulation() throws Exception {
+ // Schema WITH meta fields (writer expects this when populateMetaFields=true)
+ StructType schema = createSchemaWithMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_with_metadata.lance");
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, true)) {
+ // Write multiple records to test metadata population and sequence ID generation
+ for (int i = 0; i < 3; i++) {
+ InternalRow row = createRowWithMetaFields(i, "User" + i, 20L + i);
+ HoodieKey key = new HoodieKey("key" + i, "partition1");
+ writer.writeRowWithMetadata(key, row);
+ }
+ }
+
+ // Verify using LanceFileReader
+ assertTrue(storage.exists(path), "Lance file should exist");
+
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator);
+ ArrowReader arrowReader = reader.readAll(null, null, Integer.MAX_VALUE)) {
+
+ assertEquals(3, reader.numRows(), "Should have 3 records");
+
+ // Schema should have 5 meta fields + 3 user fields = 8 fields
+ assertEquals(8, reader.schema().getFields().size(), "Should have 8 fields");
+
+ // Read and verify data
+ assertTrue(arrowReader.loadNextBatch(), "Should load batch");
+ VectorSchemaRoot root = arrowReader.getVectorSchemaRoot();
+
+ // Verify meta fields were populated for first record
+ VarCharVector commitTimeVector = (VarCharVector) root.getVector(COMMIT_TIME_METADATA_FIELD.getFieldName());
+ assertNotNull(commitTimeVector);
+ assertEquals(instantTime, new String(commitTimeVector.get(0)), "Commit time should match");
+
+ VarCharVector recordKeyVector = (VarCharVector) root.getVector(RECORD_KEY_METADATA_FIELD.getFieldName());
+ assertEquals("key0", new String(recordKeyVector.get(0)), "Record key should match");
+
+ VarCharVector partitionPathVector = (VarCharVector) root.getVector(PARTITION_PATH_METADATA_FIELD.getFieldName());
+ assertEquals("partition1", new String(partitionPathVector.get(0)), "Partition path should match");
+
+ VarCharVector fileNameVector = (VarCharVector) root.getVector(FILENAME_METADATA_FIELD.getFieldName());
+ assertEquals(path.getName(), new String(fileNameVector.get(0)), "File name should match");
+
+ // Verify sequence IDs are unique and properly formatted
+ VarCharVector seqNoVector = (VarCharVector) root.getVector(COMMIT_SEQNO_METADATA_FIELD.getFieldName());
+ List<String> seqNos = new ArrayList<>();
+ for (int i = 0; i < 3; i++) {
+ String seqNo = new String(seqNoVector.get(i));
+ assertTrue(seqNo.startsWith(instantTime + "_"), "Sequence number should start with instant time");
+ seqNos.add(seqNo);
+ }
+ assertEquals(3, seqNos.stream().distinct().count(), "All sequence IDs should be unique");
+
+ // Verify user data fields for first record
+ IntVector idVector = (IntVector) root.getVector("id");
+ assertEquals(0, idVector.get(0), "ID should match");
+
+ VarCharVector nameVector = (VarCharVector) root.getVector("name");
+ assertEquals("User0", new String(nameVector.get(0)), "Name should match");
+
+ BigIntVector ageVector = (BigIntVector) root.getVector("age");
+ assertEquals(20L, ageVector.get(0), "Age should match");
+ }
+ }
+
+ @Test
+ public void testWriteRowWithoutMetadataPopulation() throws Exception {
+ // Schema WITHOUT meta fields
+ StructType schema = createSchemaWithoutMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_without_metadata.lance");
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false)) {
+ // Create row with just user data (no meta fields)
+ InternalRow row = createRow(1, "Bob", 25L);
+ HoodieKey key = new HoodieKey("key2", "partition2");
+
+ writer.writeRowWithMetadata(key, row);
+ }
+
+ // Verify using LanceFileReader
+ assertTrue(storage.exists(path));
+
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator);
+ ArrowReader arrowReader = reader.readAll(null, null, Integer.MAX_VALUE)) {
+
+ assertEquals(1, reader.numRows());
+
+ // Schema should have ONLY 3 user fields (no meta fields)
+ assertEquals(3, reader.schema().getFields().size(), "Should have only user fields");
+
+ // Read and verify data
+ assertTrue(arrowReader.loadNextBatch());
+ VectorSchemaRoot root = arrowReader.getVectorSchemaRoot();
+
+ // Verify NO meta fields exist
+ assertFalse(hasField(root, COMMIT_TIME_METADATA_FIELD.getFieldName()), "Should not have commit time field");
+ assertFalse(hasField(root, RECORD_KEY_METADATA_FIELD.getFieldName()), "Should not have record key field");
+
+ // Verify user data
+ IntVector idVector = (IntVector) root.getVector("id");
+ assertEquals(1, idVector.get(0));
+
+ VarCharVector nameVector = (VarCharVector) root.getVector("name");
+ assertEquals("Bob", new String(nameVector.get(0)));
+ }
+ }
+
+ @Test
+ public void testWriteRowSimple() throws Exception {
+ StructType schema = createSchemaWithoutMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_simple_write.lance");
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false)) {
+ InternalRow row = createRow(1, "Charlie", 35L);
+ writer.writeRow("key3", row);
+ }
+
+ // Verify file exists and has correct record count
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator)) {
+ assertEquals(1, reader.numRows());
+ }
+ }
+
+ @Test
+ public void testBatchFlushing() throws Exception {
+ StructType schema = createSchemaWithoutMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_batch_flush.lance");
+ // Write more than DEFAULT_BATCH_SIZE (1000) records
+ int recordCount = 2500;
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false)) {
+ for (int i = 0; i < recordCount; i++) {
+ InternalRow row = createRow(i, "User" + i, 20L + i);
+ writer.writeRow("key" + i, row);
+ }
+ }
+
+ // Verify all records were written
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator)) {
+ assertEquals(recordCount, reader.numRows(), "All records should be written");
+ }
+ }
+
+ @Test
+ public void testPrimitiveTypes() throws Exception {
+ StructType schema = new StructType()
+ .add("int_field", DataTypes.IntegerType, false)
+ .add("long_field", DataTypes.LongType, false)
+ .add("float_field", DataTypes.FloatType, false)
+ .add("double_field", DataTypes.DoubleType, false)
+ .add("bool_field", DataTypes.BooleanType, false)
+ .add("string_field", DataTypes.StringType, false)
+ .add("binary_field", DataTypes.BinaryType, false);
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_primitives.lance");
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false)) {
+ GenericInternalRow row = new GenericInternalRow(new Object[]{
+ 42, // int
+ 123456789L, // long
+ 3.14f, // float
+ 2.71828, // double
+ true, // boolean
+ UTF8String.fromString("test"), // string
+ new byte[]{1, 2, 3, 4} // binary
+ });
+
+ writer.writeRow("key1", row);
+ }
+
+ // Verify all types were written correctly
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator);
+ ArrowReader arrowReader = reader.readAll(null, null, Integer.MAX_VALUE)) {
+
+ assertEquals(1, reader.numRows());
+ assertEquals(7, reader.schema().getFields().size());
+
+ arrowReader.loadNextBatch();
+ VectorSchemaRoot root = arrowReader.getVectorSchemaRoot();
+
+ assertEquals(42, ((IntVector) root.getVector("int_field")).get(0));
+ assertEquals(123456789L, ((BigIntVector) root.getVector("long_field")).get(0));
+ assertEquals(3.14f, ((Float4Vector) root.getVector("float_field")).get(0), 0.001);
+ assertEquals(2.71828, ((Float8Vector) root.getVector("double_field")).get(0), 0.00001);
+ assertEquals(1, ((BitVector) root.getVector("bool_field")).get(0));
+ assertEquals("test", new String(((VarCharVector) root.getVector("string_field")).get(0)));
+
+ byte[] binary = ((VarBinaryVector) root.getVector("binary_field")).get(0);
+ assertEquals(4, binary.length);
+ }
+ }
+
+ @Test
+ public void testNullValues() throws Exception {
+ StructType schema = new StructType()
+ .add("id", DataTypes.IntegerType, false)
+ .add("name", DataTypes.StringType, true)
+ .add("age", DataTypes.LongType, true);
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_nulls.lance");
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false)) {
+ // Write rows with null values
+ writer.writeRow("key1", createRow(1, "Alice", 30L));
+ writer.writeRow("key2", createRow(2, null, 25L)); // null name
+ writer.writeRow("key3", createRow(3, "Charlie", null)); // null age
+ }
+
+ // Verify nulls are preserved
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator);
+ ArrowReader arrowReader = reader.readAll(null, null, Integer.MAX_VALUE)) {
+
+ assertEquals(3, reader.numRows());
+
+ arrowReader.loadNextBatch();
+ VectorSchemaRoot root = arrowReader.getVectorSchemaRoot();
+
+ VarCharVector nameVector = (VarCharVector) root.getVector("name");
+ assertFalse(nameVector.isNull(0), "First name should not be null");
+ assertTrue(nameVector.isNull(1), "Second name should be null");
+ assertFalse(nameVector.isNull(2), "Third name should not be null");
+
+ BigIntVector ageVector = (BigIntVector) root.getVector("age");
+ assertFalse(ageVector.isNull(0), "First age should not be null");
+ assertFalse(ageVector.isNull(1), "Second age should not be null");
+ assertTrue(ageVector.isNull(2), "Third age should be null");
+ }
+ }
+
+ @Test
+ public void testWriteEmptyDataset() throws Exception {
+ StructType schema = new StructType()
+ .add("id", DataTypes.IntegerType, false);
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_empty.lance");
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false)) {
+ // Close without writing any rows
+ }
+
+ // Lance doesn't create a file if no data is written
+ assertFalse(storage.exists(path), "Lance file should not exist when no data is written");
Review Comment:
Following @the-other-tim-brown recommendation I tried writing an equivalent of this test above, but using the `HoodieSparkParquetWriter`
https://gist.github.com/rahil-c/f1251f999df7674077153acdf37e610d
However when doing so i ran into the following issue before seeing an empty file created
```
org.apache.hudi.exception.HoodieException: Unable to instantiate class org.apache.hudi.io.storage.row.HoodieRowParquetWriteSupport
at org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:75)
at org.apache.hudi.io.storage.row.HoodieRowParquetWriteSupport.getHoodieRowParquetWriteSupport(HoodieRowParquetWriteSupport.java:416)
at org.apache.hudi.io.storage.TestHoodieSparkParquetWriter.testWriteEmptyDataset(TestHoodieSparkParquetWriter.java:87)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
Caused by: java.lang.reflect.InvocationTargetException
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
at org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:73)
... 5 more
Caused by: java.lang.IllegalArgumentException: Schema string cannot be null or empty
at org.apache.hudi.common.util.ValidationUtils.checkArgument(ValidationUtils.java:42)
at org.apache.hudi.common.schema.HoodieSchema$Parser.parse(HoodieSchema.java:893)
at org.apache.hudi.common.schema.HoodieSchema.parse(HoodieSchema.java:138)
at org.apache.hudi.io.storage.row.HoodieRowParquetWriteSupport.lambda$new$2(HoodieRowParquetWriteSupport.java:138)
at org.apache.hudi.common.util.Option.orElseGet(Option.java:153)
at org.apache.hudi.io.storage.row.HoodieRowParquetWriteSupport.<init>(HoodieRowParquetWriteSupport.java:136)
... 10 more
```
I then discussed with @yihua offline, who pointed me to see if `HoodieAvroParquetWriter` creates an empty file. When writing a similar test in test `TestHoodieAvroParquetWriter` : https://gist.github.com/rahil-c/0b719717f72eb59dacdd646c2802309b
was able to confirm that an empty file is created despite no actual data being rows being written.
When inspecting the underlying parquet file that was created from the test I see the following, where the empty file persists the schema.
<img width="1548" height="274" alt="Screenshot 2025-12-22 at 11 27 32 PM" src="https://github.com/user-attachments/assets/648138ec-6d46-45dd-91fd-a071ea0a5481" />
@the-other-tim-brown @yihua
So to confirm is the behavior we want that we want to achieve with an empty lance file in case there is no data present? The lance java file writer does not have any apis writing to the footer similar to parquet, https://github.com/lance-format/lance/blob/main/java/src/main/java/org/lance/file/LanceFileWriter.java#L31
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add HoodieBaseLanceFileWriter and implementation for SparkFileW… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17629:
URL: https://github.com/apache/hudi/pull/17629#issuecomment-3687108935
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9633e6bc476078ebf78b401e00ccbf43cdab67e8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10326",
"triggerID" : "9633e6bc476078ebf78b401e00ccbf43cdab67e8",
"triggerType" : "PUSH"
}, {
"hash" : "6adcb20a0c841eb4164570bed7e3daa28349be35",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10335",
"triggerID" : "6adcb20a0c841eb4164570bed7e3daa28349be35",
"triggerType" : "PUSH"
}, {
"hash" : "f0f38ca10ad7bd5db724f5568aa560e0c2af8539",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10401",
"triggerID" : "f0f38ca10ad7bd5db724f5568aa560e0c2af8539",
"triggerType" : "PUSH"
}, {
"hash" : "3e18561eb435a7b4e5452c2df03793db7e70ad82",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10420",
"triggerID" : "3e18561eb435a7b4e5452c2df03793db7e70ad82",
"triggerType" : "PUSH"
}, {
"hash" : "33cfffa5175585e24bd00025a01f3bb4f5abf3fb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10424",
"triggerID" : "33cfffa5175585e24bd00025a01f3bb4f5abf3fb",
"triggerType" : "PUSH"
}, {
"hash" : "b169123159877a93ac7a88862484211c1fa4e4f8",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10474",
"triggerID" : "b169123159877a93ac7a88862484211c1fa4e4f8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b169123159877a93ac7a88862484211c1fa4e4f8 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10474)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add HoodieBaseLanceFileWriter and implementation for SparkFileW… [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17629:
URL: https://github.com/apache/hudi/pull/17629#discussion_r2641726793
##########
hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/io/storage/TestHoodieSparkLanceWriter.java:
##########
@@ -0,0 +1,454 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.io.storage;
+
+import com.lancedb.lance.file.LanceFileReader;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.memory.RootAllocator;
+import org.apache.arrow.vector.BigIntVector;
+import org.apache.arrow.vector.BitVector;
+import org.apache.arrow.vector.Float4Vector;
+import org.apache.arrow.vector.Float8Vector;
+import org.apache.arrow.vector.IntVector;
+import org.apache.arrow.vector.VarBinaryVector;
+import org.apache.arrow.vector.VarCharVector;
+import org.apache.arrow.vector.VectorSchemaRoot;
+import org.apache.arrow.vector.ipc.ArrowReader;
+import org.apache.hudi.client.SparkTaskContextSupplier;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.testutils.HoodieTestUtils;
+import org.apache.hudi.storage.HoodieStorage;
+import org.apache.hudi.storage.StoragePath;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.catalyst.expressions.GenericInternalRow;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.unsafe.types.UTF8String;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.condition.DisabledIfSystemProperty;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.COMMIT_SEQNO_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.COMMIT_TIME_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.FILENAME_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.PARTITION_PATH_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.RECORD_KEY_METADATA_FIELD;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * Tests for {@link HoodieSparkLanceWriter}.
+ */
+@DisabledIfSystemProperty(named = "lance.skip.tests", matches = "true")
+public class TestHoodieSparkLanceWriter {
+
+ @TempDir
+ File tempDir;
+
+ private HoodieStorage storage;
+ private SparkTaskContextSupplier taskContextSupplier;
+ private String instantTime;
+
+ @BeforeEach
+ public void setUp() throws IOException {
+ storage = HoodieTestUtils.getStorage(tempDir.getAbsolutePath());
+ taskContextSupplier = new SparkTaskContextSupplier();
+ instantTime = "20251201120000000";
+ }
+
+ @AfterEach
+ public void tearDown() throws Exception {
+ if (storage != null) {
+ storage.close();
+ }
+ }
+
+ @Test
+ public void testWriteRowWithMetadataPopulation() throws Exception {
+ // Schema WITH meta fields (writer expects this when populateMetaFields=true)
+ StructType schema = createSchemaWithMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_with_metadata.lance");
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, true)) {
+ // Write multiple records to test metadata population and sequence ID generation
+ for (int i = 0; i < 3; i++) {
+ InternalRow row = createRowWithMetaFields(i, "User" + i, 20L + i);
+ HoodieKey key = new HoodieKey("key" + i, "partition1");
+ writer.writeRowWithMetadata(key, row);
+ }
+ }
+
+ // Verify using LanceFileReader
+ assertTrue(storage.exists(path), "Lance file should exist");
+
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator);
+ ArrowReader arrowReader = reader.readAll(null, null, Integer.MAX_VALUE)) {
+
+ assertEquals(3, reader.numRows(), "Should have 3 records");
+
+ // Schema should have 5 meta fields + 3 user fields = 8 fields
+ assertEquals(8, reader.schema().getFields().size(), "Should have 8 fields");
+
+ // Read and verify data
+ assertTrue(arrowReader.loadNextBatch(), "Should load batch");
+ VectorSchemaRoot root = arrowReader.getVectorSchemaRoot();
+
+ // Verify meta fields were populated for first record
+ VarCharVector commitTimeVector = (VarCharVector) root.getVector(COMMIT_TIME_METADATA_FIELD.getFieldName());
+ assertNotNull(commitTimeVector);
+ assertEquals(instantTime, new String(commitTimeVector.get(0)), "Commit time should match");
+
+ VarCharVector recordKeyVector = (VarCharVector) root.getVector(RECORD_KEY_METADATA_FIELD.getFieldName());
+ assertEquals("key0", new String(recordKeyVector.get(0)), "Record key should match");
+
+ VarCharVector partitionPathVector = (VarCharVector) root.getVector(PARTITION_PATH_METADATA_FIELD.getFieldName());
+ assertEquals("partition1", new String(partitionPathVector.get(0)), "Partition path should match");
+
+ VarCharVector fileNameVector = (VarCharVector) root.getVector(FILENAME_METADATA_FIELD.getFieldName());
+ assertEquals(path.getName(), new String(fileNameVector.get(0)), "File name should match");
+
+ // Verify sequence IDs are unique and properly formatted
+ VarCharVector seqNoVector = (VarCharVector) root.getVector(COMMIT_SEQNO_METADATA_FIELD.getFieldName());
+ List<String> seqNos = new ArrayList<>();
+ for (int i = 0; i < 3; i++) {
+ String seqNo = new String(seqNoVector.get(i));
+ assertTrue(seqNo.startsWith(instantTime + "_"), "Sequence number should start with instant time");
+ seqNos.add(seqNo);
+ }
+ assertEquals(3, seqNos.stream().distinct().count(), "All sequence IDs should be unique");
+
+ // Verify user data fields for first record
+ IntVector idVector = (IntVector) root.getVector("id");
+ assertEquals(0, idVector.get(0), "ID should match");
+
+ VarCharVector nameVector = (VarCharVector) root.getVector("name");
+ assertEquals("User0", new String(nameVector.get(0)), "Name should match");
+
+ BigIntVector ageVector = (BigIntVector) root.getVector("age");
+ assertEquals(20L, ageVector.get(0), "Age should match");
+ }
+ }
+
+ @Test
+ public void testWriteRowWithoutMetadataPopulation() throws Exception {
+ // Schema WITHOUT meta fields
+ StructType schema = createSchemaWithoutMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_without_metadata.lance");
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false)) {
+ // Create row with just user data (no meta fields)
+ InternalRow row = createRow(1, "Bob", 25L);
+ HoodieKey key = new HoodieKey("key2", "partition2");
+
+ writer.writeRowWithMetadata(key, row);
+ }
+
+ // Verify using LanceFileReader
+ assertTrue(storage.exists(path));
+
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator);
+ ArrowReader arrowReader = reader.readAll(null, null, Integer.MAX_VALUE)) {
+
+ assertEquals(1, reader.numRows());
+
+ // Schema should have ONLY 3 user fields (no meta fields)
+ assertEquals(3, reader.schema().getFields().size(), "Should have only user fields");
+
+ // Read and verify data
+ assertTrue(arrowReader.loadNextBatch());
+ VectorSchemaRoot root = arrowReader.getVectorSchemaRoot();
+
+ // Verify NO meta fields exist
+ assertFalse(hasField(root, COMMIT_TIME_METADATA_FIELD.getFieldName()), "Should not have commit time field");
+ assertFalse(hasField(root, RECORD_KEY_METADATA_FIELD.getFieldName()), "Should not have record key field");
+
+ // Verify user data
+ IntVector idVector = (IntVector) root.getVector("id");
+ assertEquals(1, idVector.get(0));
+
+ VarCharVector nameVector = (VarCharVector) root.getVector("name");
+ assertEquals("Bob", new String(nameVector.get(0)));
+ }
+ }
+
+ @Test
+ public void testWriteRowSimple() throws Exception {
+ StructType schema = createSchemaWithoutMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_simple_write.lance");
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false)) {
+ InternalRow row = createRow(1, "Charlie", 35L);
+ writer.writeRow("key3", row);
+ }
+
+ // Verify file exists and has correct record count
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator)) {
+ assertEquals(1, reader.numRows());
+ }
+ }
+
+ @Test
+ public void testBatchFlushing() throws Exception {
+ StructType schema = createSchemaWithoutMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_batch_flush.lance");
+ // Write more than DEFAULT_BATCH_SIZE (1000) records
+ int recordCount = 2500;
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false)) {
+ for (int i = 0; i < recordCount; i++) {
+ InternalRow row = createRow(i, "User" + i, 20L + i);
+ writer.writeRow("key" + i, row);
+ }
+ }
+
+ // Verify all records were written
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator)) {
+ assertEquals(recordCount, reader.numRows(), "All records should be written");
+ }
+ }
+
+ @Test
+ public void testPrimitiveTypes() throws Exception {
+ StructType schema = new StructType()
+ .add("int_field", DataTypes.IntegerType, false)
+ .add("long_field", DataTypes.LongType, false)
+ .add("float_field", DataTypes.FloatType, false)
+ .add("double_field", DataTypes.DoubleType, false)
+ .add("bool_field", DataTypes.BooleanType, false)
+ .add("string_field", DataTypes.StringType, false)
+ .add("binary_field", DataTypes.BinaryType, false);
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_primitives.lance");
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false)) {
+ GenericInternalRow row = new GenericInternalRow(new Object[]{
+ 42, // int
+ 123456789L, // long
+ 3.14f, // float
+ 2.71828, // double
+ true, // boolean
+ UTF8String.fromString("test"), // string
+ new byte[]{1, 2, 3, 4} // binary
+ });
+
+ writer.writeRow("key1", row);
+ }
+
+ // Verify all types were written correctly
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator);
+ ArrowReader arrowReader = reader.readAll(null, null, Integer.MAX_VALUE)) {
+
+ assertEquals(1, reader.numRows());
+ assertEquals(7, reader.schema().getFields().size());
+
+ arrowReader.loadNextBatch();
+ VectorSchemaRoot root = arrowReader.getVectorSchemaRoot();
+
+ assertEquals(42, ((IntVector) root.getVector("int_field")).get(0));
+ assertEquals(123456789L, ((BigIntVector) root.getVector("long_field")).get(0));
+ assertEquals(3.14f, ((Float4Vector) root.getVector("float_field")).get(0), 0.001);
+ assertEquals(2.71828, ((Float8Vector) root.getVector("double_field")).get(0), 0.00001);
+ assertEquals(1, ((BitVector) root.getVector("bool_field")).get(0));
+ assertEquals("test", new String(((VarCharVector) root.getVector("string_field")).get(0)));
+
+ byte[] binary = ((VarBinaryVector) root.getVector("binary_field")).get(0);
+ assertEquals(4, binary.length);
+ }
+ }
+
+ @Test
+ public void testNullValues() throws Exception {
+ StructType schema = new StructType()
+ .add("id", DataTypes.IntegerType, false)
+ .add("name", DataTypes.StringType, true)
+ .add("age", DataTypes.LongType, true);
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_nulls.lance");
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false)) {
+ // Write rows with null values
+ writer.writeRow("key1", createRow(1, "Alice", 30L));
+ writer.writeRow("key2", createRow(2, null, 25L)); // null name
+ writer.writeRow("key3", createRow(3, "Charlie", null)); // null age
+ }
+
+ // Verify nulls are preserved
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator);
+ ArrowReader arrowReader = reader.readAll(null, null, Integer.MAX_VALUE)) {
+
+ assertEquals(3, reader.numRows());
+
+ arrowReader.loadNextBatch();
+ VectorSchemaRoot root = arrowReader.getVectorSchemaRoot();
+
+ VarCharVector nameVector = (VarCharVector) root.getVector("name");
+ assertFalse(nameVector.isNull(0), "First name should not be null");
+ assertTrue(nameVector.isNull(1), "Second name should be null");
+ assertFalse(nameVector.isNull(2), "Third name should not be null");
+
+ BigIntVector ageVector = (BigIntVector) root.getVector("age");
+ assertFalse(ageVector.isNull(0), "First age should not be null");
+ assertFalse(ageVector.isNull(1), "Second age should not be null");
+ assertTrue(ageVector.isNull(2), "Third age should be null");
+ }
+ }
+
+ @Test
+ public void testWriteEmptyDataset() throws Exception {
+ StructType schema = new StructType()
+ .add("id", DataTypes.IntegerType, false);
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_empty.lance");
+ try (HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false)) {
+ // Close without writing any rows
+ }
+
+ // Lance doesn't create a file if no data is written
+ assertFalse(storage.exists(path), "Lance file should not exist when no data is written");
Review Comment:
@the-other-tim-brown @yihua When looking at the Hudi base parquet writer constructor, I noticed in the underlying parquet writer's `build` https://github.com/apache/hudi/blob/master/hudi-hadoop-common/src/main/java/org/apache/hudi/io/hadoop/HoodieBaseParquetWriter.java#L82 that it has the following code
```
ParquetFileWriter fileWriter = new ParquetFileWriter(
file, schema, mode, rowGroupSize, maxPaddingSize,
encodingProps.getColumnIndexTruncateLength(), encodingProps.getStatisticsTruncateLength(),
encodingProps.getPageWriteChecksumEnabled(), encryptionProperties);
fileWriter.start();
```
So i am not sure if the default behavior of parquet files is that it will create an empty file on initialization of the constructor, regardless if there is any actual data to write?
```
/**
* start the file
* @throws IOException if there is an error while writing
*/
public void start() throws IOException {
state = state.start();
LOG.debug("{}: start", out.getPos());
byte[] magic = MAGIC;
if (null != fileEncryptor && fileEncryptor.isFooterEncrypted()) {
magic = EFMAGIC;
}
out.write(magic);
}
```
https://github.com/apache/parquet-java/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileWriter.java#L596
Unless you guys were referring to another code path in hudi or parquet? If so would be helpful if you can link.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add HoodieBaseLanceFileWriter and implementation for SparkFileW… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17629:
URL: https://github.com/apache/hudi/pull/17629#issuecomment-3687543481
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b169123159877a93ac7a88862484211c1fa4e4f8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b169123159877a93ac7a88862484211c1fa4e4f8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b169123159877a93ac7a88862484211c1fa4e4f8 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add HoodieBaseLanceFileWriter and implementation for SparkFileWriter [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua merged PR #17629:
URL: https://github.com/apache/hudi/pull/17629
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add HoodieBaseLanceFileWriter and implementation for SparkFileW… [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17629:
URL: https://github.com/apache/hudi/pull/17629#discussion_r2627681400
##########
pom.xml:
##########
@@ -238,6 +238,11 @@
<springboot.version>2.7.10</springboot.version>
<spring.shell.version>2.1.1</spring.shell.version>
<snappy.version>1.1.10.7</snappy.version>
+ <arrow.version>18.3.0</arrow.version>
+ <lance.version>0.38.0</lance.version>
+ <lance.spark.connector.version>0.0.13</lance.spark.connector.version>
+ <lance.spark.artifact>lance-spark-3.5_${scala.binary.version}</lance.spark.artifact>
Review Comment:
need to think how we can have spark 3.5 and 4.0 work
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add HoodieBaseLanceFileWriter and implementation for SparkFileW… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17629:
URL: https://github.com/apache/hudi/pull/17629#issuecomment-3677839390
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9633e6bc476078ebf78b401e00ccbf43cdab67e8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10326",
"triggerID" : "9633e6bc476078ebf78b401e00ccbf43cdab67e8",
"triggerType" : "PUSH"
}, {
"hash" : "6adcb20a0c841eb4164570bed7e3daa28349be35",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10335",
"triggerID" : "6adcb20a0c841eb4164570bed7e3daa28349be35",
"triggerType" : "PUSH"
}, {
"hash" : "f0f38ca10ad7bd5db724f5568aa560e0c2af8539",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10401",
"triggerID" : "f0f38ca10ad7bd5db724f5568aa560e0c2af8539",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6adcb20a0c841eb4164570bed7e3daa28349be35 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10335)
* f0f38ca10ad7bd5db724f5568aa560e0c2af8539 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10401)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add HoodieBaseLanceFileWriter and implementation for SparkFileW… [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17629:
URL: https://github.com/apache/hudi/pull/17629#discussion_r2637823057
##########
hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/io/storage/TestHoodieSparkLanceWriter.java:
##########
@@ -0,0 +1,503 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.io.storage;
+
+import com.lancedb.lance.file.LanceFileReader;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.memory.RootAllocator;
+import org.apache.arrow.vector.BigIntVector;
+import org.apache.arrow.vector.BitVector;
+import org.apache.arrow.vector.Float4Vector;
+import org.apache.arrow.vector.Float8Vector;
+import org.apache.arrow.vector.IntVector;
+import org.apache.arrow.vector.VarBinaryVector;
+import org.apache.arrow.vector.VarCharVector;
+import org.apache.arrow.vector.VectorSchemaRoot;
+import org.apache.arrow.vector.ipc.ArrowReader;
+import org.apache.hudi.client.SparkTaskContextSupplier;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.testutils.HoodieTestUtils;
+import org.apache.hudi.storage.HoodieStorage;
+import org.apache.hudi.storage.StoragePath;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.catalyst.expressions.GenericInternalRow;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.unsafe.types.UTF8String;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.condition.DisabledIfSystemProperty;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.COMMIT_SEQNO_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.COMMIT_TIME_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.FILENAME_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.PARTITION_PATH_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.RECORD_KEY_METADATA_FIELD;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * Tests for {@link HoodieSparkLanceWriter}.
+ */
+@DisabledIfSystemProperty(named = "lance.skip.tests", matches = "true")
+public class TestHoodieSparkLanceWriter {
+
+ @TempDir
+ File tempDir;
+
+ private HoodieStorage storage;
+ private SparkTaskContextSupplier taskContextSupplier;
+ private String instantTime;
+
+ @BeforeEach
+ public void setUp() throws IOException {
+ storage = HoodieTestUtils.getStorage(tempDir.getAbsolutePath());
+ taskContextSupplier = new SparkTaskContextSupplier();
+ instantTime = "20251201120000000";
+ }
+
+ @AfterEach
+ public void tearDown() throws Exception {
+ if (storage != null) {
+ storage.close();
+ }
+ }
+
+ @Test
+ public void testWriteRowWithMetadataPopulation() throws Exception {
+ // Schema WITH meta fields (writer expects this when populateMetaFields=true)
+ StructType schema = createSchemaWithMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_with_metadata.lance");
+ HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, true);
+
+ // Create row with PLACEHOLDER meta fields + user data
+ InternalRow row = createRowWithMetaFields(1, "Alice", 30L);
+ HoodieKey key = new HoodieKey("key1", "partition1");
+
+ writer.writeRowWithMetadata(key, row);
+ writer.close();
+
+ // Verify using LanceFileReader
+ assertTrue(storage.exists(path), "Lance file should exist");
+
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator)) {
+
+ assertEquals(1, reader.numRows(), "Should have 1 record");
+
+ // Schema should have 5 meta fields + 3 user fields = 8 fields
+ assertEquals(8, reader.schema().getFields().size(), "Should have 8 fields");
+
+ // Read and verify data
+ ArrowReader arrowReader = reader.readAll(null, null, Integer.MAX_VALUE);
+ assertTrue(arrowReader.loadNextBatch(), "Should load batch");
+ VectorSchemaRoot root = arrowReader.getVectorSchemaRoot();
+
+ // Verify meta fields were populated
+ VarCharVector commitTimeVector = (VarCharVector) root.getVector(COMMIT_TIME_METADATA_FIELD.getFieldName());
+ assertNotNull(commitTimeVector);
+ assertEquals(instantTime, new String(commitTimeVector.get(0)), "Commit time should match");
+
+ VarCharVector recordKeyVector = (VarCharVector) root.getVector(RECORD_KEY_METADATA_FIELD.getFieldName());
+ assertEquals("key1", new String(recordKeyVector.get(0)), "Record key should match");
+
+ VarCharVector partitionPathVector = (VarCharVector) root.getVector(PARTITION_PATH_METADATA_FIELD.getFieldName());
+ assertEquals("partition1", new String(partitionPathVector.get(0)), "Partition path should match");
+
+ VarCharVector fileNameVector = (VarCharVector) root.getVector(FILENAME_METADATA_FIELD.getFieldName());
+ assertEquals(path.getName(), new String(fileNameVector.get(0)), "File name should match");
+
+ VarCharVector seqNoVector = (VarCharVector) root.getVector(COMMIT_SEQNO_METADATA_FIELD.getFieldName());
+ String seqNo = new String(seqNoVector.get(0));
+ assertTrue(seqNo.startsWith(instantTime), "Sequence number should start with instant time");
+
+ // Verify user data fields
+ IntVector idVector = (IntVector) root.getVector("id");
+ assertEquals(1, idVector.get(0), "ID should match");
+
+ VarCharVector nameVector = (VarCharVector) root.getVector("name");
+ assertEquals("Alice", new String(nameVector.get(0)), "Name should match");
+
+ BigIntVector ageVector = (BigIntVector) root.getVector("age");
+ assertEquals(30L, ageVector.get(0), "Age should match");
+
+ arrowReader.close();
+ }
+ }
+
+ @Test
+ public void testWriteRowWithoutMetadataPopulation() throws Exception {
+ // Schema WITHOUT meta fields
+ StructType schema = createSchemaWithoutMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_without_metadata.lance");
+ HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false);
+
+ // Create row with just user data (no meta fields)
+ InternalRow row = createRow(1, "Bob", 25L);
+ HoodieKey key = new HoodieKey("key2", "partition2");
+
+ writer.writeRowWithMetadata(key, row);
+ writer.close();
+
+ // Verify using LanceFileReader
+ assertTrue(storage.exists(path));
+
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator)) {
+
+ assertEquals(1, reader.numRows());
+
+ // Schema should have ONLY 3 user fields (no meta fields)
+ assertEquals(3, reader.schema().getFields().size(), "Should have only user fields");
+
+ // Read and verify data
+ ArrowReader arrowReader = reader.readAll(null, null, Integer.MAX_VALUE);
+ assertTrue(arrowReader.loadNextBatch());
+ VectorSchemaRoot root = arrowReader.getVectorSchemaRoot();
+
+ // Verify NO meta fields exist
+ assertFalse(hasField(root, COMMIT_TIME_METADATA_FIELD.getFieldName()), "Should not have commit time field");
+ assertFalse(hasField(root, RECORD_KEY_METADATA_FIELD.getFieldName()), "Should not have record key field");
+
+ // Verify user data
+ IntVector idVector = (IntVector) root.getVector("id");
+ assertEquals(1, idVector.get(0));
+
+ VarCharVector nameVector = (VarCharVector) root.getVector("name");
+ assertEquals("Bob", new String(nameVector.get(0)));
+
+ arrowReader.close();
+ }
+ }
+
+ @Test
+ public void testWriteRowSimple() throws Exception {
+ StructType schema = createSchemaWithoutMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_simple_write.lance");
+ HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false);
+
+ InternalRow row = createRow(1, "Charlie", 35L);
+ writer.writeRow("key3", row);
+ writer.close();
+
+ // Verify file exists and has correct record count
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator)) {
+ assertEquals(1, reader.numRows());
+ }
+ }
+
+ @Test
+ public void testSequenceIdGeneration() throws Exception {
+ StructType schema = createSchemaWithMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_sequence_id.lance");
+ HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, true);
+
+ // Write multiple records
+ for (int i = 0; i < 3; i++) {
+ InternalRow row = createRowWithMetaFields(i, "User" + i, 20L + i);
+ HoodieKey key = new HoodieKey("key" + i, "partition1");
+ writer.writeRowWithMetadata(key, row);
+ }
+ writer.close();
+
+ // Verify sequence IDs are unique and properly formatted
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator)) {
+
+ assertEquals(3, reader.numRows());
+
+ ArrowReader arrowReader = reader.readAll(null, null, Integer.MAX_VALUE);
+ arrowReader.loadNextBatch();
+ VectorSchemaRoot root = arrowReader.getVectorSchemaRoot();
+
+ VarCharVector seqNoVector = (VarCharVector) root.getVector(COMMIT_SEQNO_METADATA_FIELD.getFieldName());
Review Comment:
yes the seq no will be populated
<img width="724" height="902" alt="Screenshot 2025-12-21 at 8 25 44 AM" src="https://github.com/user-attachments/assets/8c0cb8b1-1fbc-442f-903a-5b4e1360afa6" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add HoodieBaseLanceFileWriter and implementation for SparkFileW… [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17629:
URL: https://github.com/apache/hudi/pull/17629#discussion_r2641907772
##########
hudi-hadoop-common/src/main/java/org/apache/hudi/io/lance/HoodieBaseLanceWriter.java:
##########
@@ -0,0 +1,185 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.io.lance;
+
+import com.lancedb.lance.file.LanceFileWriter;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.memory.RootAllocator;
+import org.apache.arrow.vector.VectorSchemaRoot;
+import org.apache.arrow.vector.types.pojo.Schema;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.storage.HoodieStorage;
+import org.apache.hudi.storage.StoragePath;
+
+import javax.annotation.concurrent.NotThreadSafe;
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * Base class for Hudi Lance file writers supporting different record types.
+ *
+ * This class handles common Lance file operations including:
+ * - LanceFileWriter lifecycle management
+ * - BufferAllocator management
+ * - Record buffering and batch flushing
+ * - File size checks
+ *
+ * Subclasses must implement type-specific conversion to Arrow format.
+ *
+ * @param <R> The record type (e.g., GenericRecord, InternalRow)
+ */
+@NotThreadSafe
+public abstract class HoodieBaseLanceWriter<R> implements Closeable {
+ protected static final int DEFAULT_BATCH_SIZE = 1000;
+ protected final HoodieStorage storage;
+ protected final StoragePath path;
+ protected final BufferAllocator allocator;
+ protected final List<R> bufferedRecords;
+ protected final int batchSize;
+ protected final long maxFileSize;
+ protected long writtenRecordCount = 0;
+ protected VectorSchemaRoot root;
+
+ private LanceFileWriter writer;
+
+ /**
+ * Constructor for base Lance writer.
+ *
+ * @param storage HoodieStorage instance
+ * @param path Path where Lance file will be written
+ * @param batchSize Number of records to buffer before flushing to Lance
+ * @param maxFileSize Maximum file size in bytes before rolling over to new file
Review Comment:
I think for now can remove, this might have been originally for the `canWrite()` which i can file a followup
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add HoodieBaseLanceFileWriter and implementation for SparkFileW… [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17629:
URL: https://github.com/apache/hudi/pull/17629#discussion_r2641907939
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/HoodieSparkLanceWriter.java:
##########
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.io.storage;
+
+import com.lancedb.lance.spark.arrow.LanceArrowWriter;
+import org.apache.arrow.vector.types.pojo.Schema;
+import org.apache.hudi.common.engine.TaskContextSupplier;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.io.lance.HoodieBaseLanceWriter;
+import org.apache.hudi.storage.HoodieStorage;
+import org.apache.hudi.storage.StoragePath;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.util.LanceArrowUtils;
+import org.apache.spark.unsafe.types.UTF8String;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.function.Function;
+
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.COMMIT_SEQNO_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.COMMIT_TIME_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.FILENAME_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.PARTITION_PATH_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.RECORD_KEY_METADATA_FIELD;
+
+/**
+ * Spark Lance file writer implementing {@link HoodieSparkFileWriter}.
+ *
+ * This writer integrates with Hudi's storage I/O layer and supports:
+ * - Hudi metadata field population
+ * - Record key tracking (for bloom filters - TODO https://github.com/apache/hudi/issues/17664)
+ * - Sequence ID generation
+ */
+public class HoodieSparkLanceWriter extends HoodieBaseLanceWriter<InternalRow> implements HoodieSparkFileWriter {
+
+ private static final long DEFAULT_MAX_FILE_SIZE = 120 * 1024 * 1024; // 120MB
Review Comment:
will make a task
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add HoodieBaseLanceFileWriter and implementation for SparkFileW… [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17629:
URL: https://github.com/apache/hudi/pull/17629#discussion_r2641907772
##########
hudi-hadoop-common/src/main/java/org/apache/hudi/io/lance/HoodieBaseLanceWriter.java:
##########
@@ -0,0 +1,185 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.io.lance;
+
+import com.lancedb.lance.file.LanceFileWriter;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.memory.RootAllocator;
+import org.apache.arrow.vector.VectorSchemaRoot;
+import org.apache.arrow.vector.types.pojo.Schema;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.storage.HoodieStorage;
+import org.apache.hudi.storage.StoragePath;
+
+import javax.annotation.concurrent.NotThreadSafe;
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * Base class for Hudi Lance file writers supporting different record types.
+ *
+ * This class handles common Lance file operations including:
+ * - LanceFileWriter lifecycle management
+ * - BufferAllocator management
+ * - Record buffering and batch flushing
+ * - File size checks
+ *
+ * Subclasses must implement type-specific conversion to Arrow format.
+ *
+ * @param <R> The record type (e.g., GenericRecord, InternalRow)
+ */
+@NotThreadSafe
+public abstract class HoodieBaseLanceWriter<R> implements Closeable {
+ protected static final int DEFAULT_BATCH_SIZE = 1000;
+ protected final HoodieStorage storage;
+ protected final StoragePath path;
+ protected final BufferAllocator allocator;
+ protected final List<R> bufferedRecords;
+ protected final int batchSize;
+ protected final long maxFileSize;
+ protected long writtenRecordCount = 0;
+ protected VectorSchemaRoot root;
+
+ private LanceFileWriter writer;
+
+ /**
+ * Constructor for base Lance writer.
+ *
+ * @param storage HoodieStorage instance
+ * @param path Path where Lance file will be written
+ * @param batchSize Number of records to buffer before flushing to Lance
+ * @param maxFileSize Maximum file size in bytes before rolling over to new file
Review Comment:
I think for now can remove, this might have been originally for the `canWrite()`. Will add a github issue link there
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add HoodieBaseLanceFileWriter and implementation for SparkFileW… [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on PR #17629:
URL: https://github.com/apache/hudi/pull/17629#issuecomment-3687189787
both java ci and azure ci flaking for unrelated issues. I can rerun the azure ci but will need a committer to rerun the java ci @the-other-tim-brown @yihua @voonhous
<img width="308" height="461" alt="Screenshot 2025-12-23 at 11 13 23 AM" src="https://github.com/user-attachments/assets/2b04a884-4c65-4a58-a9dd-6a8f33df469a" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org