You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hudi.apache.org by "rahil-c (via GitHub)" <gi...@apache.org> on 2025/10/22 14:09:46 UTC
[D] RFC-100: Lance File Format support in Hudi [hudi]
GitHub user rahil-c created a discussion: RFC-100: Lance File Format support in Hudi
## ✅ Lance File Format Integration Tasks
See the following feature for more context: https://github.com/apache/hudi/issues/14127
In regards to the following new feature for supporting unstructured data in Hudi via formats like Lance, here is the initial scope of what we are targeting(Note this list will continue to grow as we find get deeper within the integration):
- [ ] Add base `HoodieFileWriter` for Lance with a Spark implementation
- [ ] Add base `HoodieFileReader` for Lance with a Spark implementation
- [ ] Add basic Avro → Arrow schema conversion
- [ ] Add `SparkColumnarFileReader` implementation for Lance
- [ ] Implement append-only validation (bulk insert)
- [ ] Integrate Lance as a log file format
- [ ] Implement insert / upsert / delete validation
- [ ] Add predicate (filter) push-down
- [ ] Support `ColumnarBatch` vectorized reading
GitHub link: https://github.com/apache/hudi/discussions/14128
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] RFC-100: Lance File Format support in Hudi [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
GitHub user rahil-c closed a discussion: RFC-100: Lance File Format support in Hudi
## ✅ Lance File Format Integration Tasks
See the following feature for more context: https://github.com/apache/hudi/issues/14127
In regards to the following new feature for supporting unstructured data in Hudi via formats like Lance that are focused on AI/ML use cases. Here is the initial scope of what we are targeting(Note this list will continue to grow as we find get deeper within the integration, for now it aims to first support the Hudi Spark Client):
- [ ] Add base `HoodieFileWriter` for Lance with a Spark implementation, [PR](https://github.com/apache/hudi/pull/14131) (merged in feature branch)
- [ ] Add base `HoodieFileReader` for Lance with a Spark implementation, [PR](https://github.com/apache/hudi/pull/14132)
- [ ] Add `SparkColumnarFileReader` implementation for Spark Datasource Integration with Lance, [PR](https://github.com/apache/hudi/pull/14135)
- [ ] Implement bulk insert/ insert / upsert / delete validation for COW, [PR](https://github.com/apache/hudi/pull/14137)
- [ ] Implement bulk insert/ insert / upsert / delete validation for MOR(with lance base file and avro log file), [PR](https://github.com/apache/hudi/pull/14154)
- [ ] Ensure that basic compaction, and clustering works for Hudi Tables with Lance [PR](https://github.com/apache/hudi/pull/14156) (requires rebase from above)
- [ ] Ensure that schema evolution(add column) works for Hudi Tables with Lance [PR](https://github.com/apache/hudi/pull/14160)
- [ ] FileWriter improvements: Support canWrite() and bloom filter [comment1](https://github.com/apache/hudi/pull/14131#discussion_r2455261849), [comment2](https://github.com/apache/hudi/pull/14131#discussion_r2455244853)
- [ ] Validate that vector and blob content works [PR](https://github.com/apache/hudi/pull/14193)
- [ ] Integrate Lance as a log file format
- [ ] Add predicate (filter) push-down
- [ ] Support `ColumnarBatch` vectorized reading
Will be raising changes on the following open source feature branch: https://github.com/apache/hudi/tree/feature-branch-rfc100-unstructured-data
## Improving File Format Integration Points
Changes that are modifying existing hudi master (1.2.0) to allow for better integration with file formats in the future.
- [ ] Consolidate file writer side interfaces, [PR](https://github.com/apache/hudi/pull/14173)
GitHub link: https://github.com/apache/hudi/discussions/14128
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] RFC-100: Lance File Format support in Hudi [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
GitHub user rahil-c edited a discussion: RFC-100: Lance File Format support in Hudi
## ✅ Lance File Format Integration Tasks
See the following feature for more context: https://github.com/apache/hudi/issues/14127
In regards to the following new feature for supporting unstructured data in Hudi via formats like Lance that are focused on AI/ML use cases. Here is the initial scope of what we are targeting(Note this list will continue to grow as we find get deeper within the integration, for now it aims to first support the Hudi Spark Client):
- [ ] Add base `HoodieFileWriter` for Lance with a Spark implementation, [PR](https://github.com/apache/hudi/pull/14131)
- [ ] Add base `HoodieFileReader` for Lance with a Spark implementation, [PR](https://github.com/apache/hudi/pull/14132)
- [ ] Add basic Avro → Arrow schema conversion, [PR](https://github.com/apache/hudi/pull/14132)
- [ ] Add `SparkColumnarFileReader` implementation for Lance , WIP
- [ ] Implement append-only validation (bulk insert), WIP
- [ ] Integrate Lance as a log file format
- [ ] Implement insert / upsert / delete validation
- [ ] Add predicate (filter) push-down
- [ ] Support `ColumnarBatch` vectorized reading
Will be making on the following open source feature branch: https://github.com/apache/hudi/tree/feature-branch-rfc100-unstructured-data
GitHub link: https://github.com/apache/hudi/discussions/14128
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] RFC-100: Lance File Format support in Hudi [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
GitHub user rahil-c edited a discussion: RFC-100: Lance File Format support in Hudi
## ✅ Lance File Format Integration Tasks
See the following feature for more context: https://github.com/apache/hudi/issues/14127
In regards to the following new feature for supporting unstructured data in Hudi via formats like Lance that are focused on AI/ML use cases. Here is the initial scope of what we are targeting(Note this list will continue to grow as we find get deeper within the integration, for now it aims to first support the Hudi Spark Client):
- [ ] Add base `HoodieFileWriter` for Lance with a Spark implementation, [PR](https://github.com/apache/hudi/pull/14131)
- [ ] Add base `HoodieFileReader` for Lance with a Spark implementation, [PR](https://github.com/apache/hudi/pull/14132)
- [ ] Add `SparkColumnarFileReader` implementation for Spark Datasource Integration with Lance, [PR](https://github.com/apache/hudi/pull/14135)
- [ ] Implement bulk insert/ insert / upsert / delete validation for COW, [PR](https://github.com/apache/hudi/pull/14137)
- [ ] Implement bulk insert/ insert / upsert / delete validation for MOR(with lance base file and avro log file), [PR](https://github.com/apache/hudi/pull/14154)
- [ ] Ensure that basic compaction, and clustering works for Hudi Tables with Lance [PR](https://github.com/apache/hudi/pull/14156)
- [ ] Ensure that schema evolution works for Hudi Tables with Lance [PR](https://github.com/apache/hudi/pull/14160)
- [ ] Integrate Lance as a log file format
- [ ] Add predicate (filter) push-down
- [ ] Support `ColumnarBatch` vectorized reading
Will be raising changes on the following open source feature branch: https://github.com/apache/hudi/tree/feature-branch-rfc100-unstructured-data
GitHub link: https://github.com/apache/hudi/discussions/14128
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] RFC-100: Lance File Format support in Hudi [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
GitHub user rahil-c edited a discussion: RFC-100: Lance File Format support in Hudi
## ✅ Lance File Format Integration Tasks
See the following feature for more context: https://github.com/apache/hudi/issues/14127
In regards to the following new feature for supporting unstructured data in Hudi via formats like Lance that are focused on AI/ML use cases. Here is the initial scope of what we are targeting(Note this list will continue to grow as we find get deeper within the integration, for now it aims to first support the Hudi Spark Client):
- [ ] Add base `HoodieFileWriter` for Lance with a Spark implementation, [PR](https://github.com/apache/hudi/pull/14131) (merged in feature branch)
- [ ] Add base `HoodieFileReader` for Lance with a Spark implementation, [PR](https://github.com/apache/hudi/pull/14132)
- [ ] Add `SparkColumnarFileReader` implementation for Spark Datasource Integration with Lance, [PR](https://github.com/apache/hudi/pull/14135)
- [ ] Implement bulk insert/ insert / upsert / delete validation for COW, [PR](https://github.com/apache/hudi/pull/14137)
- [ ] Implement bulk insert/ insert / upsert / delete validation for MOR(with lance base file and avro log file), [PR](https://github.com/apache/hudi/pull/14154)
- [ ] Ensure that basic compaction, and clustering works for Hudi Tables with Lance [PR](https://github.com/apache/hudi/pull/14156) (requires rebase from above)
- [ ] Ensure that schema evolution(add column) works for Hudi Tables with Lance [PR](https://github.com/apache/hudi/pull/14160)
- [ ] FileWriter improvements: Support canWrite() and bloom filter [comment1](https://github.com/apache/hudi/pull/14131#discussion_r2455261849), [comment2](https://github.com/apache/hudi/pull/14131#discussion_r2455244853)
- [ ] Validate that vector and blob content works [PR](https://github.com/apache/hudi/pull/14193)
- [ ] Integrate Lance as a log file format
- [ ] Add predicate (filter) push-down
- [ ] Support `ColumnarBatch` vectorized reading
Will be raising changes on the following open source feature branch: https://github.com/apache/hudi/tree/feature-branch-rfc100-unstructured-data
## Improving File Format Integration Points
Changes that are modifying existing hudi master (1.2.0) to allow for better integration with file formats in the future.
- [ ] Consolidate file writer side interfaces, [PR](https://github.com/apache/hudi/pull/14173)
GitHub link: https://github.com/apache/hudi/discussions/14128
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] RFC-100: Lance File Format support in Hudi [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
GitHub user rahil-c edited a discussion: RFC-100: Lance File Format support in Hudi
## ✅ Lance File Format Integration Tasks
See the following feature for more context: https://github.com/apache/hudi/issues/14127
In regards to the following new feature for supporting unstructured data in Hudi via formats like Lance that are focused on AI/ML use cases. Here is the initial scope of what we are targeting(Note this list will continue to grow as we find get deeper within the integration, for now it aims to first support the Hudi Spark Client):
- [ ] Add base `HoodieFileWriter` for Lance with a Spark implementation, [PR](https://github.com/apache/hudi/pull/14131)
- [ ] Add base `HoodieFileReader` for Lance with a Spark implementation, [PR](https://github.com/apache/hudi/pull/14132)
- [ ] Add `SparkColumnarFileReader` implementation for Spark Datasource Integration with Lance, [PR](https://github.com/apache/hudi/pull/14135)
- [ ] Implement bulk insert/ insert / upsert / delete validation for COW, [PR](https://github.com/apache/hudi/pull/14137)
- [ ] Implement bulk insert/ insert / upsert / delete validation for MOR(with lance base file and avro log file), [PR](https://github.com/apache/hudi/pull/14154)
- [ ] Ensure that basic compaction, and clustering works for Hudi Tables with Lance [PR](https://github.com/apache/hudi/pull/14156)
- [ ] Integrate Lance as a log file format
- [ ] Add predicate (filter) push-down
- [ ] Support `ColumnarBatch` vectorized reading
Will be raising changes on the following open source feature branch: https://github.com/apache/hudi/tree/feature-branch-rfc100-unstructured-data
GitHub link: https://github.com/apache/hudi/discussions/14128
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] RFC-100: Lance File Format support in Hudi [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
GitHub user rahil-c edited a discussion: RFC-100: Lance File Format support in Hudi
## ✅ Lance File Format Integration Tasks
See the following feature for more context: https://github.com/apache/hudi/issues/14127
In regards to the following new feature for supporting unstructured data in Hudi via formats like Lance that are focused on AI/ML use cases. Here is the initial scope of what we are targeting(Note this list will continue to grow as we find get deeper within the integration, for now it aims to first support the Hudi Spark Client):
- [ ] Add base `HoodieFileWriter` for Lance with a Spark implementation, [PR](https://github.com/apache/hudi/pull/14131) (merged in feature branch)
- [ ] Add base `HoodieFileReader` for Lance with a Spark implementation, [PR](https://github.com/apache/hudi/pull/14132)
- [ ] Add `SparkColumnarFileReader` implementation for Spark Datasource Integration with Lance, [PR](https://github.com/apache/hudi/pull/14135)
- [ ] Implement bulk insert/ insert / upsert / delete validation for COW, [PR](https://github.com/apache/hudi/pull/14137)
- [ ] Implement bulk insert/ insert / upsert / delete validation for MOR(with lance base file and avro log file), [PR](https://github.com/apache/hudi/pull/14154)
- [ ] Ensure that basic compaction, and clustering works for Hudi Tables with Lance [PR](https://github.com/apache/hudi/pull/14156) (requires rebase from above)
- [ ] Ensure that schema evolution(add column) works for Hudi Tables with Lance [PR](https://github.com/apache/hudi/pull/14160)
- [ ] FileWriter improvements: Support canWrite() and bloom filter [comment1](https://github.com/apache/hudi/pull/14131#discussion_r2455261849), [comment2](https://github.com/apache/hudi/pull/14131#discussion_r2455244853)
- [ ] Integrate Lance as a log file format
- [ ] Add predicate (filter) push-down
- [ ] Support `ColumnarBatch` vectorized reading
Will be raising changes on the following open source feature branch: https://github.com/apache/hudi/tree/feature-branch-rfc100-unstructured-data
GitHub link: https://github.com/apache/hudi/discussions/14128
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] RFC-100: Lance File Format support in Hudi [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
GitHub user rahil-c edited a discussion: RFC-100: Lance File Format support in Hudi
## ✅ Lance File Format Integration Tasks
See the following feature for more context: https://github.com/apache/hudi/issues/14127
In regards to the following new feature for supporting unstructured data in Hudi via formats like Lance that are focused on AI/ML use cases. Here is the initial scope of what we are targeting(Note this list will continue to grow as we find get deeper within the integration, for now it aims to first support the Hudi Spark Client):
- [ ] Add base `HoodieFileWriter` for Lance with a Spark implementation, [PR](https://github.com/apache/hudi/pull/14131)
- [ ] Add base `HoodieFileReader` for Lance with a Spark implementation, [PR](https://github.com/apache/hudi/pull/14132)
- [ ] Add basic Avro → Arrow schema conversion
- [ ] Add `SparkColumnarFileReader` implementation for Lance
- [ ] Implement append-only validation (bulk insert)
- [ ] Integrate Lance as a log file format
- [ ] Implement insert / upsert / delete validation
- [ ] Add predicate (filter) push-down
- [ ] Support `ColumnarBatch` vectorized reading
Will be making on the following open source feature branch: https://github.com/apache/hudi/tree/feature-branch-rfc100-unstructured-data
GitHub link: https://github.com/apache/hudi/discussions/14128
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] RFC-100: Lance File Format support in Hudi [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
GitHub user rahil-c edited a discussion: RFC-100: Lance File Format support in Hudi
## ✅ Lance File Format Integration Tasks
See the following feature for more context: https://github.com/apache/hudi/issues/14127
In regards to the following new feature for supporting unstructured data in Hudi via formats like Lance that are focused on AI/ML use cases. Here is the initial scope of what we are targeting(Note this list will continue to grow as we find get deeper within the integration, for now it aims to first support the Hudi Spark Client):
- [ ] Add base `HoodieFileWriter` for Lance with a Spark implementation
- [ ] Add base `HoodieFileReader` for Lance with a Spark implementation
- [ ] Add basic Avro → Arrow schema conversion
- [ ] Add `SparkColumnarFileReader` implementation for Lance
- [ ] Implement append-only validation (bulk insert)
- [ ] Integrate Lance as a log file format
- [ ] Implement insert / upsert / delete validation
- [ ] Add predicate (filter) push-down
- [ ] Support `ColumnarBatch` vectorized reading
Will be making on the following open source feature branch: https://github.com/apache/hudi/tree/feature-branch-rfc100-unstructured-data
GitHub link: https://github.com/apache/hudi/discussions/14128
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] RFC-100: Lance File Format support in Hudi [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
GitHub user rahil-c edited a discussion: RFC-100: Lance File Format support in Hudi
## ✅ Lance File Format Integration Tasks
See the following feature for more context: https://github.com/apache/hudi/issues/14127
In regards to the following new feature for supporting unstructured data in Hudi via formats like Lance that are focused on AI/ML use cases. Here is the initial scope of what we are targeting(Note this list will continue to grow as we find get deeper within the integration, for now it aims to first support the Hudi Spark Client):
- [ ] Add base `HoodieFileWriter` for Lance with a Spark implementation, [PR](https://github.com/apache/hudi/pull/14131)
- [ ] Add base `HoodieFileReader` for Lance with a Spark implementation, [PR](https://github.com/apache/hudi/pull/14132)
- [ ] Add `SparkColumnarFileReader` implementation for Spark Datasource Integration with Lance, [PR](https://github.com/apache/hudi/pull/14135)
- [ ] Implement bulk insert/ insert / upsert / delete validation for COW, [PR](https://github.com/apache/hudi/pull/14137)
- [ ] Implement bulk insert/ insert / upsert / delete validation for MOR(with lance base file and avro log file), [PR](https://github.com/apache/hudi/pull/14154)
- [ ] Ensure that compaction, and clustering works for Hudi Tables with Lance [PR] https://github.com/apache/hudi/pull/14156
- [ ] Integrate Lance as a log file format
- [ ] Add predicate (filter) push-down
- [ ] Support `ColumnarBatch` vectorized reading
Will be raising changes on the following open source feature branch: https://github.com/apache/hudi/tree/feature-branch-rfc100-unstructured-data
GitHub link: https://github.com/apache/hudi/discussions/14128
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] RFC-100: Lance File Format support in Hudi [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
GitHub user rahil-c edited a discussion: RFC-100: Lance File Format support in Hudi
## ✅ Lance File Format Integration Tasks
See the following feature for more context: https://github.com/apache/hudi/issues/14127
In regards to the following new feature for supporting unstructured data in Hudi via formats like Lance that are focused on AI/ML use cases. Here is the initial scope of what we are targeting(Note this list will continue to grow as we find get deeper within the integration, for now it aims to first support the Hudi Spark Client):
- [ ] Add base `HoodieFileWriter` for Lance with a Spark implementation, [PR](https://github.com/apache/hudi/pull/14131) (merged in feature branch)
- [ ] Add base `HoodieFileReader` for Lance with a Spark implementation, [PR](https://github.com/apache/hudi/pull/14132)
- [ ] Add `SparkColumnarFileReader` implementation for Spark Datasource Integration with Lance, [PR](https://github.com/apache/hudi/pull/14135)
- [ ] Implement bulk insert/ insert / upsert / delete validation for COW, [PR](https://github.com/apache/hudi/pull/14137)
- [ ] Implement bulk insert/ insert / upsert / delete validation for MOR(with lance base file and avro log file), [PR](https://github.com/apache/hudi/pull/14154)
- [ ] Ensure that basic compaction, and clustering works for Hudi Tables with Lance [PR](https://github.com/apache/hudi/pull/14156) (requires rebase from above)
- [ ] Ensure that schema evolution(add column) works for Hudi Tables with Lance [PR](https://github.com/apache/hudi/pull/14160)
- [ ] FileWriter improvements: Support canWrite() and bloom filter [comment1](https://github.com/apache/hudi/pull/14131#discussion_r2455261849), [comment2](https://github.com/apache/hudi/pull/14131#discussion_r2455244853)
- [ ] Integrate Lance as a log file format
- [ ] Add predicate (filter) push-down
- [ ] Support `ColumnarBatch` vectorized reading
Will be raising changes on the following open source feature branch: https://github.com/apache/hudi/tree/feature-branch-rfc100-unstructured-data
## Improving File Format Integration Points
Changes that are modifying existing hudi master (1.2.0) to allow for better integration with file formats in the future.
- [ ] Consolidate file writer side interfaces, [PR](https://github.com/apache/hudi/pull/14173)
GitHub link: https://github.com/apache/hudi/discussions/14128
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] RFC-100: Lance File Format support in Hudi [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
GitHub user rahil-c edited a discussion: RFC-100: Lance File Format support in Hudi
## ✅ Lance File Format Integration Tasks
See the following feature for more context: https://github.com/apache/hudi/issues/14127
In regards to the following new feature for supporting unstructured data in Hudi via formats like Lance that are focused on AI/ML use cases. Here is the initial scope of what we are targeting(Note this list will continue to grow as we find get deeper within the integration, for now it aims to first support the Hudi Spark Client):
- [ ] Add base `HoodieFileWriter` for Lance with a Spark implementation, [PR](https://github.com/apache/hudi/pull/14131)
- [ ] Add base `HoodieFileReader` for Lance with a Spark implementation, [PR](https://github.com/apache/hudi/pull/14132)
- [ ] Add `SparkColumnarFileReader` implementation for Lance, [PR](https://github.com/apache/hudi/pull/14135)
- [ ] Implement bulk insert/ insert / upsert / delete validation for COW, [PR](https://github.com/apache/hudi/pull/14137)
- [ ] Integrate Lance as a log file format
- [ ] Implement bulk insert/ insert / upsert / delete validation for MOR
- [ ] Add predicate (filter) push-down
- [ ] Support `ColumnarBatch` vectorized reading
Will be raising changes on the following open source feature branch: https://github.com/apache/hudi/tree/feature-branch-rfc100-unstructured-data
GitHub link: https://github.com/apache/hudi/discussions/14128
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] RFC-100: Lance File Format support in Hudi [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
GitHub user the-other-tim-brown added a comment to the discussion: RFC-100: Lance File Format support in Hudi
Adding some notes from our discussion tonight:
- we should make sure that the query planning can skip based on partition filters and column stats
- do the col/partition stats get populated?
GitHub link: https://github.com/apache/hudi/discussions/14128#discussioncomment-15226571
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] RFC-100: Lance File Format support in Hudi [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
GitHub user rahil-c edited a discussion: RFC-100: Lance File Format support in Hudi
## ✅ Lance File Format Integration Tasks
See the following feature for more context: https://github.com/apache/hudi/issues/14127
In regards to the following new feature for supporting unstructured data in Hudi via formats like Lance that are focused on AI/ML use cases. Here is the initial scope of what we are targeting(Note this list will continue to grow as we find get deeper within the integration, for now it aims to first support the Hudi Spark Client):
- [ ] Add base `HoodieFileWriter` for Lance with a Spark implementation, [PR](https://github.com/apache/hudi/pull/14131)
- [ ] Add base `HoodieFileReader` for Lance with a Spark implementation, [PR](https://github.com/apache/hudi/pull/14132)
- [ ] Add `SparkColumnarFileReader` implementation for Spark Datasource Integration with Lance, [PR](https://github.com/apache/hudi/pull/14135)
- [ ] Implement bulk insert/ insert / upsert / delete validation for COW, [PR](https://github.com/apache/hudi/pull/14137)
- [ ] Implement bulk insert/ insert / upsert / delete validation for MOR(with lance base file and avro log file), [PR](https://github.com/apache/hudi/pull/14154)
- [ ] Ensure that basic compaction, and clustering works for Hudi Tables with Lance [PR](https://github.com/apache/hudi/pull/14156)
- [ ] Ensure that schema evolution(add column) works for Hudi Tables with Lance [PR](https://github.com/apache/hudi/pull/14160)
- [ ] Writer improvements: Support bloom filter, TODO [comment](https://github.com/apache/hudi/pull/14131#discussion_r2455244853)
- [ ] Integrate Lance as a log file format
- [ ] Add predicate (filter) push-down
- [ ] Support `ColumnarBatch` vectorized reading
Will be raising changes on the following open source feature branch: https://github.com/apache/hudi/tree/feature-branch-rfc100-unstructured-data
GitHub link: https://github.com/apache/hudi/discussions/14128
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] RFC-100: Lance File Format support in Hudi [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
GitHub user rahil-c edited a discussion: RFC-100: Lance File Format support in Hudi
## ✅ Lance File Format Integration Tasks
See the following feature for more context: https://github.com/apache/hudi/issues/14127
In regards to the following new feature for supporting unstructured data in Hudi via formats like Lance that are focused on AI/ML use cases. Here is the initial scope of what we are targeting(Note this list will continue to grow as we find get deeper within the integration, for now it aims to first support the Hudi Spark Client):
- [ ] Add base `HoodieFileWriter` for Lance with a Spark implementation, [PR](https://github.com/apache/hudi/pull/14131)
- [ ] Add base `HoodieFileReader` for Lance with a Spark implementation, [PR](https://github.com/apache/hudi/pull/14132)
- [ ] Add `SparkColumnarFileReader` implementation for Spark Datasource Integration with Lance, [PR](https://github.com/apache/hudi/pull/14135)
- [ ] Implement bulk insert/ insert / upsert / delete validation for COW, [PR](https://github.com/apache/hudi/pull/14137)
- [ ] Implement bulk insert/ insert / upsert / delete validation for MOR(with lance base file and avro log file), [PR](https://github.com/apache/hudi/pull/14154)
- [ ] Ensure that compaction, and clustering works for MOR table
- [ ] Integrate Lance as a log file format
- [ ] Add predicate (filter) push-down
- [ ] Support `ColumnarBatch` vectorized reading
Will be raising changes on the following open source feature branch: https://github.com/apache/hudi/tree/feature-branch-rfc100-unstructured-data
GitHub link: https://github.com/apache/hudi/discussions/14128
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] RFC-100: Lance File Format support in Hudi [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
GitHub user rahil-c edited a discussion: RFC-100: Lance File Format support in Hudi
## ✅ Lance File Format Integration Tasks
See the following feature for more context: https://github.com/apache/hudi/issues/14127
In regards to the following new feature for supporting unstructured data in Hudi via formats like Lance that are focused on AI/ML use cases. Here is the initial scope of what we are targeting(Note this list will continue to grow as we find get deeper within the integration, for now it aims to first support the Hudi Spark Client):
- [ ] Add base `HoodieFileWriter` for Lance with a Spark implementation, [PR](https://github.com/apache/hudi/pull/14131)
- [ ] Add base `HoodieFileReader` for Lance with a Spark implementation, [PR](https://github.com/apache/hudi/pull/14132)
- [ ] Add `SparkColumnarFileReader` implementation for Spark Datasource Integration with Lance, [PR](https://github.com/apache/hudi/pull/14135)
- [ ] Implement bulk insert/ insert / upsert / delete validation for COW, [PR](https://github.com/apache/hudi/pull/14137)
- [ ] Implement bulk insert/ insert / upsert / delete validation for MOR(with lance base file and avro log file), [PR](https://github.com/apache/hudi/pull/14154)
- [ ] Ensure that basic compaction, and clustering works for Hudi Tables with Lance [PR](https://github.com/apache/hudi/pull/14156)
- [ ] Ensure that schema evolution(add column) works for Hudi Tables with Lance [PR](https://github.com/apache/hudi/pull/14160)
- [ ] Integrate Lance as a log file format
- [ ] Add predicate (filter) push-down
- [ ] Support `ColumnarBatch` vectorized reading
Will be raising changes on the following open source feature branch: https://github.com/apache/hudi/tree/feature-branch-rfc100-unstructured-data
GitHub link: https://github.com/apache/hudi/discussions/14128
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] RFC-100: Lance File Format support in Hudi [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
GitHub user rahil-c edited a discussion: RFC-100: Lance File Format support in Hudi
## ✅ Lance File Format Integration Tasks
See the following feature for more context: https://github.com/apache/hudi/issues/14127
In regards to the following new feature for supporting unstructured data in Hudi via formats like Lance that are focused on AI/ML use cases. Here is the initial scope of what we are targeting(Note this list will continue to grow as we find get deeper within the integration, for now it aims to first support the Hudi Spark Client):
- [ ] Add base `HoodieFileWriter` for Lance with a Spark implementation, [PR](https://github.com/apache/hudi/pull/14131)
- [ ] Add base `HoodieFileReader` for Lance with a Spark implementation, [PR](https://github.com/apache/hudi/pull/14132)
- [ ] Add `SparkColumnarFileReader` implementation for Spark Datasource Integration with Lance, [PR](https://github.com/apache/hudi/pull/14135)
- [ ] Implement bulk insert/ insert / upsert / delete validation for COW, [PR](https://github.com/apache/hudi/pull/14137)
- [ ] Implement bulk insert/ insert / upsert / delete validation for MOR(with lance base file and avro log file), [PR](https://github.com/apache/hudi/pull/14154)
- [ ] Ensure that basic compaction, and clustering works for Hudi Tables with Lance [PR](https://github.com/apache/hudi/pull/14156) (requires rebase from above)
- [ ] Ensure that schema evolution(add column) works for Hudi Tables with Lance [PR](https://github.com/apache/hudi/pull/14160)
- [ ] FileWriter improvements: Support canWrite() and bloom filter [comment1](https://github.com/apache/hudi/pull/14131#discussion_r2455261849), [comment2](https://github.com/apache/hudi/pull/14131#discussion_r2455244853)
- [ ] Integrate Lance as a log file format
- [ ] Add predicate (filter) push-down
- [ ] Support `ColumnarBatch` vectorized reading
Will be raising changes on the following open source feature branch: https://github.com/apache/hudi/tree/feature-branch-rfc100-unstructured-data
GitHub link: https://github.com/apache/hudi/discussions/14128
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] RFC-100: Lance File Format support in Hudi [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
GitHub user rahil-c edited a discussion: RFC-100: Lance File Format support in Hudi
## ✅ Lance File Format Integration Tasks
See the following feature for more context: https://github.com/apache/hudi/issues/14127
In regards to the following new feature for supporting unstructured data in Hudi via formats like Lance that are focused on AI/ML use cases. Here is the initial scope of what we are targeting(Note this list will continue to grow as we find get deeper within the integration, for now it aims to first support the Hudi Spark Client):
- [ ] Add base `HoodieFileWriter` for Lance with a Spark implementation, [PR](https://github.com/apache/hudi/pull/14131)
- [ ] Add base `HoodieFileReader` for Lance with a Spark implementation, [PR](https://github.com/apache/hudi/pull/14132)
- [ ] Add `SparkColumnarFileReader` implementation for Spark Datasource Integration with Lance, [PR](https://github.com/apache/hudi/pull/14135)
- [ ] Implement bulk insert/ insert / upsert / delete validation for COW, [PR](https://github.com/apache/hudi/pull/14137)
- [ ] Implement bulk insert/ insert / upsert / delete validation for MOR(with lance base file and avro log file), [PR](https://github.com/apache/hudi/pull/14154)
- [ ] Integrate Lance as a log file format
- [ ] Add predicate (filter) push-down
- [ ] Support `ColumnarBatch` vectorized reading
Will be raising changes on the following open source feature branch: https://github.com/apache/hudi/tree/feature-branch-rfc100-unstructured-data
GitHub link: https://github.com/apache/hudi/discussions/14128
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] RFC-100: Lance File Format support in Hudi [hudi]
Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
GitHub user vinothchandar added a comment to the discussion: RFC-100: Lance File Format support in Hudi
@rahil-c I believe we are further along here.. Do we need to move/consolidate this discussion with some GH issue..
GitHub link: https://github.com/apache/hudi/discussions/14128#discussioncomment-15152366
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] RFC-100: Lance File Format support in Hudi [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
GitHub user rahil-c edited a discussion: RFC-100: Lance File Format support in Hudi
## ✅ Lance File Format Integration Tasks
See the following feature for more context: https://github.com/apache/hudi/issues/14127
In regards to the following new feature for supporting unstructured data in Hudi via formats like Lance that are focused on AI/ML use cases. Here is the initial scope of what we are targeting(Note this list will continue to grow as we find get deeper within the integration, for now it aims to first support the Hudi Spark Client):
- [ ] Add base `HoodieFileWriter` for Lance with a Spark implementation, [PR](https://github.com/apache/hudi/pull/14131)
- [ ] Add base `HoodieFileReader` for Lance with a Spark implementation, [PR](https://github.com/apache/hudi/pull/14132)
- [ ] Add basic Avro → Arrow schema conversion, [PR](https://github.com/apache/hudi/pull/14132)
- [ ] Add `SparkColumnarFileReader` implementation for Lance
- [ ] Implement append-only validation (bulk insert)
- [ ] Integrate Lance as a log file format
- [ ] Implement insert / upsert / delete validation
- [ ] Add predicate (filter) push-down
- [ ] Support `ColumnarBatch` vectorized reading
Will be making on the following open source feature branch: https://github.com/apache/hudi/tree/feature-branch-rfc100-unstructured-data
GitHub link: https://github.com/apache/hudi/discussions/14128
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] RFC-100: Lance File Format support in Hudi [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
GitHub user rahil-c edited a discussion: RFC-100: Lance File Format support in Hudi
## ✅ Lance File Format Integration Tasks
See the following feature for more context: https://github.com/apache/hudi/issues/14127
In regards to the following new feature for supporting unstructured data in Hudi via formats like Lance, here is the initial scope of what we are targeting(Note this list will continue to grow as we find get deeper within the integration):
- [ ] Add base `HoodieFileWriter` for Lance with a Spark implementation
- [ ] Add base `HoodieFileReader` for Lance with a Spark implementation
- [ ] Add basic Avro → Arrow schema conversion
- [ ] Add `SparkColumnarFileReader` implementation for Lance
- [ ] Implement append-only validation (bulk insert)
- [ ] Integrate Lance as a log file format
- [ ] Implement insert / upsert / delete validation
- [ ] Add predicate (filter) push-down
- [ ] Support `ColumnarBatch` vectorized reading
Will be making on the following open source feature branch: https://github.com/apache/hudi/tree/feature-branch-rfc100-unstructured-data
GitHub link: https://github.com/apache/hudi/discussions/14128
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] RFC-100: Lance File Format support in Hudi [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
GitHub user rahil-c edited a discussion: RFC-100: Lance File Format support in Hudi
## ✅ Lance File Format Integration Tasks
See the following feature for more context: https://github.com/apache/hudi/issues/14127
In regards to the following new feature for supporting unstructured data in Hudi via formats like Lance that are focused on AI/ML use cases. Here is the initial scope of what we are targeting(Note this list will continue to grow as we find get deeper within the integration, for now it aims to first support the Hudi Spark Client):
- [ ] Add base `HoodieFileWriter` for Lance with a Spark implementation, [PR](https://github.com/apache/hudi/pull/14131) (merged in feature branch)
- [ ] Add base `HoodieFileReader` for Lance with a Spark implementation, [PR](https://github.com/apache/hudi/pull/14132)
- [ ] Add `SparkColumnarFileReader` implementation for Spark Datasource Integration with Lance, [PR](https://github.com/apache/hudi/pull/14135)
- [ ] Implement bulk insert/ insert / upsert / delete validation for COW, [PR](https://github.com/apache/hudi/pull/14137)
- [ ] Implement bulk insert/ insert / upsert / delete validation for MOR(with lance base file and avro log file), [PR](https://github.com/apache/hudi/pull/14154)
- [ ] Ensure that basic compaction, and clustering works for Hudi Tables with Lance [PR](https://github.com/apache/hudi/pull/14156) (requires rebase from above)
- [ ] Ensure that schema evolution(add column) works for Hudi Tables with Lance [PR](https://github.com/apache/hudi/pull/14160)
- [ ] FileWriter improvements: Support canWrite() and bloom filter [comment1](https://github.com/apache/hudi/pull/14131#discussion_r2455261849), [comment2](https://github.com/apache/hudi/pull/14131#discussion_r2455244853)
- [ ] Integrate Lance as a log file format
- [ ] Add predicate (filter) push-down
- [ ] Support `ColumnarBatch` vectorized reading
Will be raising changes on the following open source feature branch: https://github.com/apache/hudi/tree/feature-branch-rfc100-unstructured-data
## General Hudi File Format Abstraction Changes
Changes that are modifying existing hudi master (1.2.0) to allow for better integration with file formats in the future.
- [] Consolidate file writer side interfaces for new file format integrations, [PR](https://github.com/apache/hudi/pull/14173)
GitHub link: https://github.com/apache/hudi/discussions/14128
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] RFC-100: Lance File Format support in Hudi [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
GitHub user rahil-c edited a discussion: RFC-100: Lance File Format support in Hudi
## ✅ Lance File Format Integration Tasks
See the following feature for more context: https://github.com/apache/hudi/issues/14127
In regards to the following new feature for supporting unstructured data in Hudi via formats like Lance that are focused on AI/ML use cases. Here is the initial scope of what we are targeting(Note this list will continue to grow as we find get deeper within the integration, for now it aims to first support the Hudi Spark Client):
- [ ] Add base `HoodieFileWriter` for Lance with a Spark implementation, [PR](https://github.com/apache/hudi/pull/14131)
- [ ] Add base `HoodieFileReader` for Lance with a Spark implementation, [PR](https://github.com/apache/hudi/pull/14132)
- [ ] Add `SparkColumnarFileReader` implementation for Spark Datasource Integration with Lance, [PR](https://github.com/apache/hudi/pull/14135)
- [ ] Implement bulk insert/ insert / upsert / delete validation for COW, [PR](https://github.com/apache/hudi/pull/14137)
- [ ] Implement bulk insert/ insert / upsert / delete validation for MOR(with lance base file and avro log file), [PR](https://github.com/apache/hudi/pull/14154)
- [ ] Ensure that compaction, and clustering works for Hudi Tables with Lance [PR](https://github.com/apache/hudi/pull/14156)
- [ ] Integrate Lance as a log file format
- [ ] Add predicate (filter) push-down
- [ ] Support `ColumnarBatch` vectorized reading
Will be raising changes on the following open source feature branch: https://github.com/apache/hudi/tree/feature-branch-rfc100-unstructured-data
GitHub link: https://github.com/apache/hudi/discussions/14128
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] RFC-100: Lance File Format support in Hudi [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
GitHub user rahil-c edited a discussion: RFC-100: Lance File Format support in Hudi
## ✅ Lance File Format Integration Tasks
See the following feature for more context: https://github.com/apache/hudi/issues/14127
In regards to the following new feature for supporting unstructured data in Hudi via formats like Lance that are focused on AI/ML use cases. Here is the initial scope of what we are targeting(Note this list will continue to grow as we find get deeper within the integration, for now it aims to first support the Hudi Spark Client):
- [ ] Add base `HoodieFileWriter` for Lance with a Spark implementation, [PR](https://github.com/apache/hudi/pull/14131) (merged in feature branch)
- [ ] Add base `HoodieFileReader` for Lance with a Spark implementation, [PR](https://github.com/apache/hudi/pull/14132)
- [ ] Add `SparkColumnarFileReader` implementation for Spark Datasource Integration with Lance, [PR](https://github.com/apache/hudi/pull/14135)
- [ ] Implement bulk insert/ insert / upsert / delete validation for COW, [PR](https://github.com/apache/hudi/pull/14137)
- [ ] Implement bulk insert/ insert / upsert / delete validation for MOR(with lance base file and avro log file), [PR](https://github.com/apache/hudi/pull/14154)
- [ ] Ensure that basic compaction, and clustering works for Hudi Tables with Lance [PR](https://github.com/apache/hudi/pull/14156) (requires rebase from above)
- [ ] Ensure that schema evolution(add column) works for Hudi Tables with Lance [PR](https://github.com/apache/hudi/pull/14160)
- [ ] FileWriter improvements: Support canWrite() and bloom filter [comment1](https://github.com/apache/hudi/pull/14131#discussion_r2455261849), [comment2](https://github.com/apache/hudi/pull/14131#discussion_r2455244853)
- [ ] Integrate Lance as a log file format
- [ ] Add predicate (filter) push-down
- [ ] Support `ColumnarBatch` vectorized reading
Will be raising changes on the following open source feature branch: https://github.com/apache/hudi/tree/feature-branch-rfc100-unstructured-data
## Improving File Format Integration Points
Changes that are modifying existing hudi master (1.2.0) to allow for better integration with file formats in the future.
- [ ] Consolidate file writer side interfaces for new file format integrations, [PR](https://github.com/apache/hudi/pull/14173)
GitHub link: https://github.com/apache/hudi/discussions/14128
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] RFC-100: Lance File Format support in Hudi [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
GitHub user rahil-c edited a discussion: RFC-100: Lance File Format support in Hudi
## ✅ Lance File Format Integration Tasks
See the following feature for more context: https://github.com/apache/hudi/issues/14127
In regards to the following new feature for supporting unstructured data in Hudi via formats like Lance that are focused on AI/ML use cases. Here is the initial scope of what we are targeting(Note this list will continue to grow as we find get deeper within the integration, for now it aims to first support the Hudi Spark Client):
- [ ] Add base `HoodieFileWriter` for Lance with a Spark implementation, [PR](https://github.com/apache/hudi/pull/14131)
- [ ] Add base `HoodieFileReader` for Lance with a Spark implementation, [PR](https://github.com/apache/hudi/pull/14132)
- [ ] Add `SparkColumnarFileReader` implementation for Spark Datasource Integration with Lance, [PR](https://github.com/apache/hudi/pull/14135)
- [ ] Implement bulk insert/ insert / upsert / delete validation for COW, [PR](https://github.com/apache/hudi/pull/14137)
- [ ] Implement bulk insert/ insert / upsert / delete validation for MOR(with lance base file and avro log file), [PR](https://github.com/apache/hudi/pull/14154)
- [ ] Ensure that basic compaction, and clustering works for Hudi Tables with Lance [PR](https://github.com/apache/hudi/pull/14156) requires rebase from above
- [ ] Ensure that schema evolution(add column) works for Hudi Tables with Lance [PR](https://github.com/apache/hudi/pull/14160)
- [ ] FileWriter improvements: Support canWrite() and bloom filter [comment1](https://github.com/apache/hudi/pull/14131#discussion_r2455261849), [comment2](https://github.com/apache/hudi/pull/14131#discussion_r2455244853)
- [ ] Integrate Lance as a log file format
- [ ] Add predicate (filter) push-down
- [ ] Support `ColumnarBatch` vectorized reading
Will be raising changes on the following open source feature branch: https://github.com/apache/hudi/tree/feature-branch-rfc100-unstructured-data
GitHub link: https://github.com/apache/hudi/discussions/14128
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] RFC-100: Lance File Format support in Hudi [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
GitHub user rahil-c edited a discussion: RFC-100: Lance File Format support in Hudi
## ✅ Lance File Format Integration Tasks
See the following feature for more context: https://github.com/apache/hudi/issues/14127
In regards to the following new feature for supporting unstructured data in Hudi via formats like Lance that are focused on AI/ML use cases. Here is the initial scope of what we are targeting(Note this list will continue to grow as we find get deeper within the integration, for now it aims to first support the Hudi Spark Client):
- [ ] Add base `HoodieFileWriter` for Lance with a Spark implementation, [PR](https://github.com/apache/hudi/pull/14131)
- [ ] Add base `HoodieFileReader` for Lance with a Spark implementation, [PR](https://github.com/apache/hudi/pull/14132)
- [ ] Add `SparkColumnarFileReader` implementation for Lance, [PR](https://github.com/apache/hudi/pull/14132)
- [ ] Implement append-only validation (bulk insert), WIP
- [ ] Implement insert / upsert / delete validation, WIP
- [ ] Integrate Lance as a log file format
- [ ] Add predicate (filter) push-down
- [ ] Support `ColumnarBatch` vectorized reading
Will be raising changes on the following open source feature branch: https://github.com/apache/hudi/tree/feature-branch-rfc100-unstructured-data
GitHub link: https://github.com/apache/hudi/discussions/14128
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] RFC-100: Lance File Format support in Hudi [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
GitHub user rahil-c added a comment to the discussion: RFC-100: Lance File Format support in Hudi
Will be porting issues and moving to the issue board here: https://github.com/apache/hudi/issues/14127. Feel free to add tasks to it
GitHub link: https://github.com/apache/hudi/discussions/14128#discussioncomment-15280514
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] RFC-100: Lance File Format support in Hudi [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
GitHub user rahil-c edited a discussion: RFC-100: Lance File Format support in Hudi
## ✅ Lance File Format Integration Tasks
See the following feature for more context: https://github.com/apache/hudi/issues/14127
In regards to the following new feature for supporting unstructured data in Hudi via formats like Lance that are focused on AI/ML use cases. Here is the initial scope of what we are targeting(Note this list will continue to grow as we find get deeper within the integration, for now it aims to first support the Hudi Spark Client):
- [ ] Add base `HoodieFileWriter` for Lance with a Spark implementation, [PR](https://github.com/apache/hudi/pull/14131)
- [ ] Add base `HoodieFileReader` for Lance with a Spark implementation, [PR](https://github.com/apache/hudi/pull/14132)
- [ ] Add `SparkColumnarFileReader` implementation for Spark Datasource Integration with Lance, [PR](https://github.com/apache/hudi/pull/14135)
- [ ] Implement bulk insert/ insert / upsert / delete validation for COW, [PR](https://github.com/apache/hudi/pull/14137)
- [ ] Integrate Lance as a log file format
- [ ] Implement bulk insert/ insert / upsert / delete validation for MOR
- [ ] Add predicate (filter) push-down
- [ ] Support `ColumnarBatch` vectorized reading
Will be raising changes on the following open source feature branch: https://github.com/apache/hudi/tree/feature-branch-rfc100-unstructured-data
GitHub link: https://github.com/apache/hudi/discussions/14128
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] RFC-100: Lance File Format support in Hudi [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
GitHub user rahil-c edited a discussion: RFC-100: Lance File Format support in Hudi
## ✅ Lance File Format Integration Tasks
See the following feature for more context: https://github.com/apache/hudi/issues/14127
In regards to the following new feature for supporting unstructured data in Hudi via formats like Lance that are focused on AI/ML use cases. Here is the initial scope of what we are targeting(Note this list will continue to grow as we find get deeper within the integration, for now it aims to first support the Hudi Spark Client):
- [ ] Add base `HoodieFileWriter` for Lance with a Spark implementation, [PR](https://github.com/apache/hudi/pull/14131)
- [ ] Add base `HoodieFileReader` for Lance with a Spark implementation
- [ ] Add basic Avro → Arrow schema conversion
- [ ] Add `SparkColumnarFileReader` implementation for Lance
- [ ] Implement append-only validation (bulk insert)
- [ ] Integrate Lance as a log file format
- [ ] Implement insert / upsert / delete validation
- [ ] Add predicate (filter) push-down
- [ ] Support `ColumnarBatch` vectorized reading
Will be making on the following open source feature branch: https://github.com/apache/hudi/tree/feature-branch-rfc100-unstructured-data
GitHub link: https://github.com/apache/hudi/discussions/14128
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] RFC-100: Lance File Format support in Hudi [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
GitHub user rahil-c edited a discussion: RFC-100: Lance File Format support in Hudi
## ✅ Lance File Format Integration Tasks
See the following feature for more context: https://github.com/apache/hudi/issues/14127
In regards to the following new feature for supporting unstructured data in Hudi via formats like Lance that are focused on AI/ML use cases. Here is the initial scope of what we are targeting(Note this list will continue to grow as we find get deeper within the integration, for now it aims to first support the Hudi Spark Client):
- [ ] Add base `HoodieFileWriter` for Lance with a Spark implementation, [PR](https://github.com/apache/hudi/pull/14131)
- [ ] Add base `HoodieFileReader` for Lance with a Spark implementation, [PR](https://github.com/apache/hudi/pull/14132)
- [ ] Add basic Avro → Arrow schema conversion, [PR](https://github.com/apache/hudi/pull/14132)
- [ ] Add `SparkColumnarFileReader` implementation for Lance , WIP
- [ ] Implement append-only validation (bulk insert), WIP
- [ ] Implement insert / upsert / delete validation, WIP
- [ ] Integrate Lance as a log file format
- [ ] Add predicate (filter) push-down
- [ ] Support `ColumnarBatch` vectorized reading
Will be making on the following open source feature branch: https://github.com/apache/hudi/tree/feature-branch-rfc100-unstructured-data
GitHub link: https://github.com/apache/hudi/discussions/14128
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] RFC-100: Lance File Format support in Hudi [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
GitHub user rahil-c edited a discussion: RFC-100: Lance File Format support in Hudi
## ✅ Lance File Format Integration Tasks
See the following feature for more context: https://github.com/apache/hudi/issues/14127
In regards to the following new feature for supporting unstructured data in Hudi via formats like Lance that are focused on AI/ML use cases. Here is the initial scope of what we are targeting(Note this list will continue to grow as we find get deeper within the integration, for now it aims to first support the Hudi Spark Client):
- [ ] Add base `HoodieFileWriter` for Lance with a Spark implementation, [PR](https://github.com/apache/hudi/pull/14131) (merged in feature branch)
- [ ] Add base `HoodieFileReader` for Lance with a Spark implementation, [PR](https://github.com/apache/hudi/pull/14132)
- [ ] Add `SparkColumnarFileReader` implementation for Spark Datasource Integration with Lance, [PR](https://github.com/apache/hudi/pull/14135)
- [ ] Implement bulk insert/ insert / upsert / delete validation for COW, [PR](https://github.com/apache/hudi/pull/14137)
- [ ] Implement bulk insert/ insert / upsert / delete validation for MOR(with lance base file and avro log file), [PR](https://github.com/apache/hudi/pull/14154)
- [ ] Ensure that basic compaction, and clustering works for Hudi Tables with Lance [PR](https://github.com/apache/hudi/pull/14156) (requires rebase from above)
- [ ] Ensure that schema evolution(add column) works for Hudi Tables with Lance [PR](https://github.com/apache/hudi/pull/14160)
- [ ] FileWriter improvements: Support canWrite() and bloom filter [comment1](https://github.com/apache/hudi/pull/14131#discussion_r2455261849), [comment2](https://github.com/apache/hudi/pull/14131#discussion_r2455244853)
- [ ] Integrate Lance as a log file format
- [ ] Add predicate (filter) push-down
- [ ] Support `ColumnarBatch` vectorized reading
Will be raising changes on the following open source feature branch: https://github.com/apache/hudi/tree/feature-branch-rfc100-unstructured-data
## General Hudi File Format Abstraction Changes
Changes that are modifying existing hudi master (1.2.0) to allow for better integration with file formats in the future.
-[] Consolidate file writer side interfaces for new file format integrations, [PR](https://github.com/apache/hudi/pull/14173)
GitHub link: https://github.com/apache/hudi/discussions/14128
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] RFC-100: Lance File Format support in Hudi [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
GitHub user rahil-c edited a discussion: RFC-100: Lance File Format support in Hudi
## ✅ Lance File Format Integration Tasks
See the following feature for more context: https://github.com/apache/hudi/issues/14127
In regards to the following new feature for supporting unstructured data in Hudi via formats like Lance that are focused on AI/ML use cases. Here is the initial scope of what we are targeting(Note this list will continue to grow as we find get deeper within the integration, for now it aims to first support the Hudi Spark Client):
- [ ] Add base `HoodieFileWriter` for Lance with a Spark implementation, [PR](https://github.com/apache/hudi/pull/14131)
- [ ] Add base `HoodieFileReader` for Lance with a Spark implementation, [PR](https://github.com/apache/hudi/pull/14132)
- [ ] Add `SparkColumnarFileReader` implementation for Spark Datasource Integration with Lance, [PR](https://github.com/apache/hudi/pull/14135)
- [ ] Implement bulk insert/ insert / upsert / delete validation for COW, [PR](https://github.com/apache/hudi/pull/14137)
- [ ] Implement bulk insert/ insert / upsert / delete validation for MOR(with lance base file and avro log file), [PR](https://github.com/apache/hudi/pull/14154)
- [ ] Ensure that basic compaction, and clustering works for Hudi Tables with Lance [PR](https://github.com/apache/hudi/pull/14156)
- [ ] Ensure that schema evolution works for Hudi Tables with Lance
- [ ] Integrate Lance as a log file format
- [ ] Add predicate (filter) push-down
- [ ] Support `ColumnarBatch` vectorized reading
Will be raising changes on the following open source feature branch: https://github.com/apache/hudi/tree/feature-branch-rfc100-unstructured-data
GitHub link: https://github.com/apache/hudi/discussions/14128
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] RFC-100: Lance File Format support in Hudi [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
GitHub user rahil-c edited a discussion: RFC-100: Lance File Format support in Hudi
## ✅ Lance File Format Integration Tasks
See the following feature for more context: https://github.com/apache/hudi/issues/14127
In regards to the following new feature for supporting unstructured data in Hudi via formats like Lance that are focused on AI/ML use cases. Here is the initial scope of what we are targeting(Note this list will continue to grow as we find get deeper within the integration, for now it aims to first support the Hudi Spark Client):
- [ ] Add base `HoodieFileWriter` for Lance with a Spark implementation, [PR](https://github.com/apache/hudi/pull/14131)
- [ ] Add base `HoodieFileReader` for Lance with a Spark implementation, [PR](https://github.com/apache/hudi/pull/14132)
- [ ] Add `SparkColumnarFileReader` implementation for Spark Datasource Integration with Lance, [PR](https://github.com/apache/hudi/pull/14135)
- [ ] Implement bulk insert/ insert / upsert / delete validation for COW, [PR](https://github.com/apache/hudi/pull/14137)
- [ ] Implement bulk insert/ insert / upsert / delete validation for MOR(with lance base file and avro log file), [PR](https://github.com/apache/hudi/pull/14154)
- [ ] Ensure that basic compaction, and clustering works for Hudi Tables with Lance [PR](https://github.com/apache/hudi/pull/14156)
- [ ] Ensure that schema evolution(add column) works for Hudi Tables with Lance [PR](https://github.com/apache/hudi/pull/14160)
- [ ] FileWriter improvements: Support canWrite() and bloom filter [comment1](https://github.com/apache/hudi/pull/14131#discussion_r2455261849), [comment2](https://github.com/apache/hudi/pull/14131#discussion_r2455244853)
- [ ] Integrate Lance as a log file format
- [ ] Add predicate (filter) push-down
- [ ] Support `ColumnarBatch` vectorized reading
Will be raising changes on the following open source feature branch: https://github.com/apache/hudi/tree/feature-branch-rfc100-unstructured-data
GitHub link: https://github.com/apache/hudi/discussions/14128
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org