You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "rahil-c (via GitHub)" <gi...@apache.org> on 2025/10/22 14:35:00 UTC
[PR] Add HoodieBaseLanceFileWriter and implementation for SparkFileWriter … [hudi]
rahil-c opened a new pull request, #14131:
URL: https://github.com/apache/hudi/pull/14131
…(#1614)
### Describe the issue this Pull Request addresses
<!-- Either describe the issue inline here with motivation behind the changes
(or) link to an issue by including `Closes #<issue-number>` for context.
If this PR includes changes to the storage format, public APIs,
or has breaking changes, use `!` (e.g., feat!: ...) -->
### Summary and Changelog
<!-- Short, plain-English summary of what users gain or what changed in behavior.
Followed by a detailed log of all the changes. Highlight if any code was copied. -->
### Impact
<!-- Describe any public API or user-facing feature change or any performance impact. -->
### Risk Level
<!-- Accepted values: none, low, medium or high. Other than `none`, explain the risk.
If medium or high, explain what verification was done to mitigate the risks. -->
### Documentation Update
<!-- Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none".
- The config description must be updated if new configs are added or the default value of the configs are changed.
- Any new feature or user-facing change requires updating the Hudi website. Please follow the
[instruction](https://hudi.apache.org/contribute/developer-setup#website) to make changes to the website. -->
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Enough context is provided in the sections above
- [ ] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add HoodieBaseLanceFileWriter and implementation for SparkFileWriter … [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #14131:
URL: https://github.com/apache/hudi/pull/14131#discussion_r2470700701
##########
.github/workflows/bot.yml:
##########
@@ -958,88 +650,6 @@ jobs:
HUDI_VERSION=$(mvn help:evaluate -Dexpression=project.version -q -DforceStdout)
./packaging/bundle-validation/run_docker_java17.sh
- validate-bundles:
Review Comment:
Keep this bundle validation and remove Java 8 parts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add HoodieBaseLanceFileWriter and implementation for SparkFileWriter … [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #14131:
URL: https://github.com/apache/hudi/pull/14131#discussion_r2462117989
##########
pom.xml:
##########
@@ -897,7 +900,29 @@
<version>${orc.spark.version}</version>
<scope>compile</scope>
</dependency>
-
+ <!-- Lance Core SDK -->
+ <dependency>
+ <groupId>com.lancedb</groupId>
+ <artifactId>lance-core</artifactId>
+ <version>${lance.version}</version>
+ </dependency>
+ <!-- Lance Spark for Arrow conversion -->
+ <dependency>
+ <groupId>com.lancedb</groupId>
+ <artifactId>lance-spark-base_${scala.binary.version}</artifactId>
Review Comment:
Can we try using their bundled jar? https://mvnrepository.com/artifact/com.lancedb/lance-spark-bundle-3.5 - wondering if this will help with the dependency issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add HoodieBaseLanceFileWriter and implementation for SparkFileWriter … [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #14131:
URL: https://github.com/apache/hudi/pull/14131#discussion_r2470662132
##########
.github/workflows/bot.yml:
##########
@@ -59,281 +61,6 @@ jobs:
- name: RAT check
run: ./scripts/release/validate_source_rat.sh
- test-spark-java-tests-part1:
- runs-on: ubuntu-latest
- strategy:
- matrix:
- include:
- - scalaProfile: "scala-2.12"
- sparkProfile: "spark3.3"
- sparkModules: "hudi-spark-datasource/hudi-spark3.3.x"
-
- - scalaProfile: "scala-2.12"
- sparkProfile: "spark3.4"
- sparkModules: "hudi-spark-datasource/hudi-spark3.4.x"
-
- - scalaProfile: "scala-2.12"
- sparkProfile: "spark3.5"
- sparkModules: "hudi-spark-datasource/hudi-spark3.5.x"
-
- - scalaProfile: "scala-2.13"
- sparkProfile: "spark3.5"
- sparkModules: "hudi-spark-datasource/hudi-spark3.5.x"
-
- steps:
- - uses: actions/checkout@v3
- - name: Set up JDK 8
- uses: actions/setup-java@v3
- with:
- java-version: '8'
Review Comment:
So the removal of these actions is because Java 8 is not supported for Lance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add HoodieBaseLanceFileWriter and implementation for SparkFileWriter … [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14131:
URL: https://github.com/apache/hudi/pull/14131#issuecomment-3458477997
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f1b8ff586652da6f82dd996fd0a6edcdb0f72ab4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9265",
"triggerID" : "f1b8ff586652da6f82dd996fd0a6edcdb0f72ab4",
"triggerType" : "PUSH"
}, {
"hash" : "7d356d05bf73a7263f448f4fad46ac0332ab8522",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9295",
"triggerID" : "7d356d05bf73a7263f448f4fad46ac0332ab8522",
"triggerType" : "PUSH"
}, {
"hash" : "41a8b4a7127307fa0263d701f5fc567c80857ea3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9360",
"triggerID" : "41a8b4a7127307fa0263d701f5fc567c80857ea3",
"triggerType" : "PUSH"
}, {
"hash" : "adc1713ee5af363f98d0a0ccfb2324b5c249a4fb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9362",
"triggerID" : "adc1713ee5af363f98d0a0ccfb2324b5c249a4fb",
"triggerType" : "PUSH"
}, {
"hash" : "0bbead9b3a4efb779b472a343731969ea0f4d7f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9364",
"triggerID" : "0bbead9b3a4efb779b472a343731969ea0f4d7f2",
"triggerType" : "PUSH"
}, {
"hash" : "5c1b891ad88030b8e286602b449eee96301dcb84",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9366",
"triggerID" : "5c1b891ad88030b8e286602b449eee96301dcb84",
"triggerType" : "PUSH"
}, {
"hash" : "9943cbbadb0ce515bb16ebbb7b39919711e8a362",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9369",
"triggerID" : "9943cbbadb0ce515bb16ebbb7b39919711e8a362",
"triggerType" : "PUSH"
}, {
"hash" : "ef0c4ca43b81d7b0c27a4e61ad93dea0c7df91a3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ef0c4ca43b81d7b0c27a4e61ad93dea0c7df91a3",
"triggerType" : "PUSH"
}, {
"hash" : "bc7d984512e3bdf695f35442a76581e815086e91",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9388",
"triggerID" : "bc7d984512e3bdf695f35442a76581e815086e91",
"triggerType" : "PUSH"
}, {
"hash" : "0f73e2d085ebacd3211a81e79c27b8a41f1b1b5f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9394",
"triggerID" : "0f73e2d085ebacd3211a81e79c27b8a41f1b1b5f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ef0c4ca43b81d7b0c27a4e61ad93dea0c7df91a3 UNKNOWN
* 0f73e2d085ebacd3211a81e79c27b8a41f1b1b5f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9394)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add HoodieBaseLanceFileWriter and implementation for SparkFileWriter … [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14131:
URL: https://github.com/apache/hudi/pull/14131#issuecomment-3453732577
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f1b8ff586652da6f82dd996fd0a6edcdb0f72ab4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9265",
"triggerID" : "f1b8ff586652da6f82dd996fd0a6edcdb0f72ab4",
"triggerType" : "PUSH"
}, {
"hash" : "7d356d05bf73a7263f448f4fad46ac0332ab8522",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9295",
"triggerID" : "7d356d05bf73a7263f448f4fad46ac0332ab8522",
"triggerType" : "PUSH"
}, {
"hash" : "41a8b4a7127307fa0263d701f5fc567c80857ea3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9360",
"triggerID" : "41a8b4a7127307fa0263d701f5fc567c80857ea3",
"triggerType" : "PUSH"
}, {
"hash" : "adc1713ee5af363f98d0a0ccfb2324b5c249a4fb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9362",
"triggerID" : "adc1713ee5af363f98d0a0ccfb2324b5c249a4fb",
"triggerType" : "PUSH"
}, {
"hash" : "0bbead9b3a4efb779b472a343731969ea0f4d7f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9364",
"triggerID" : "0bbead9b3a4efb779b472a343731969ea0f4d7f2",
"triggerType" : "PUSH"
}, {
"hash" : "5c1b891ad88030b8e286602b449eee96301dcb84",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9366",
"triggerID" : "5c1b891ad88030b8e286602b449eee96301dcb84",
"triggerType" : "PUSH"
}, {
"hash" : "9943cbbadb0ce515bb16ebbb7b39919711e8a362",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9369",
"triggerID" : "9943cbbadb0ce515bb16ebbb7b39919711e8a362",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5c1b891ad88030b8e286602b449eee96301dcb84 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9366)
* 9943cbbadb0ce515bb16ebbb7b39919711e8a362 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9369)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add HoodieBaseLanceFileWriter and implementation for SparkFileWriter … [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14131:
URL: https://github.com/apache/hudi/pull/14131#issuecomment-3453230564
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f1b8ff586652da6f82dd996fd0a6edcdb0f72ab4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9265",
"triggerID" : "f1b8ff586652da6f82dd996fd0a6edcdb0f72ab4",
"triggerType" : "PUSH"
}, {
"hash" : "7d356d05bf73a7263f448f4fad46ac0332ab8522",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9295",
"triggerID" : "7d356d05bf73a7263f448f4fad46ac0332ab8522",
"triggerType" : "PUSH"
}, {
"hash" : "41a8b4a7127307fa0263d701f5fc567c80857ea3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9360",
"triggerID" : "41a8b4a7127307fa0263d701f5fc567c80857ea3",
"triggerType" : "PUSH"
}, {
"hash" : "adc1713ee5af363f98d0a0ccfb2324b5c249a4fb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9362",
"triggerID" : "adc1713ee5af363f98d0a0ccfb2324b5c249a4fb",
"triggerType" : "PUSH"
}, {
"hash" : "0bbead9b3a4efb779b472a343731969ea0f4d7f2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9364",
"triggerID" : "0bbead9b3a4efb779b472a343731969ea0f4d7f2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0bbead9b3a4efb779b472a343731969ea0f4d7f2 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9364)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add HoodieBaseLanceFileWriter and implementation for SparkFileWriter … [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #14131:
URL: https://github.com/apache/hudi/pull/14131#discussion_r2466495709
##########
pom.xml:
##########
@@ -897,7 +900,29 @@
<version>${orc.spark.version}</version>
<scope>compile</scope>
</dependency>
-
+ <!-- Lance Core SDK -->
+ <dependency>
+ <groupId>com.lancedb</groupId>
+ <artifactId>lance-core</artifactId>
+ <version>${lance.version}</version>
+ </dependency>
+ <!-- Lance Spark for Arrow conversion -->
+ <dependency>
+ <groupId>com.lancedb</groupId>
+ <artifactId>lance-spark-base_${scala.binary.version}</artifactId>
Review Comment:
I think its more around the fact that our ci is currently running for spark 3.3 and spark 3.4, so we either need to restrict those profiles or have a way of passing the correct lance spark version for each spark profile we have in root pom. https://lancedb.github.io/lance/integrations/spark/install/#scala
Ill try using the specific spark version artfiact and if that does not work then will try bundle idea.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add HoodieBaseLanceFileWriter and implementation for SparkFileWriter … [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #14131:
URL: https://github.com/apache/hudi/pull/14131#discussion_r2470675817
##########
.github/workflows/bot.yml:
##########
@@ -958,88 +650,6 @@ jobs:
HUDI_VERSION=$(mvn help:evaluate -Dexpression=project.version -q -DforceStdout)
./packaging/bundle-validation/run_docker_java17.sh
- validate-bundles:
- runs-on: ubuntu-latest
- strategy:
- matrix:
- include:
- - scalaProfile: 'scala-2.13'
- flinkProfile: 'flink1.20'
- flinkAvroVersion: '1.11.4'
- flinkParquetVersion: '1.13.1'
- sparkProfile: 'spark3.5'
- sparkRuntime: 'spark3.5.1'
- - scalaProfile: 'scala-2.13'
- flinkProfile: 'flink1.19'
- flinkAvroVersion: '1.11.4'
- flinkParquetVersion: '1.13.1'
- sparkProfile: 'spark3.5'
- sparkRuntime: 'spark3.5.1'
- - scalaProfile: 'scala-2.12'
- flinkProfile: 'flink1.18'
- flinkAvroVersion: '1.11.4'
- flinkParquetVersion: '1.13.1'
- sparkProfile: 'spark3.4'
- sparkRuntime: 'spark3.4.3'
- - scalaProfile: 'scala-2.12'
- flinkProfile: 'flink1.17'
- flinkAvroVersion: '1.11.4'
- flinkParquetVersion: '1.12.3'
- sparkProfile: 'spark3.3'
- sparkRuntime: 'spark3.3.4'
-
- steps:
- - uses: actions/checkout@v3
- - name: Set up JDK 8
- uses: actions/setup-java@v3
- with:
- java-version: '8'
Review Comment:
will do so
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add HoodieBaseLanceFileWriter and implementation for SparkFileWriter … [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua merged PR #14131:
URL: https://github.com/apache/hudi/pull/14131
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add HoodieBaseLanceFileWriter and implementation for SparkFileWriter … [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14131:
URL: https://github.com/apache/hudi/pull/14131#issuecomment-3458297041
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f1b8ff586652da6f82dd996fd0a6edcdb0f72ab4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9265",
"triggerID" : "f1b8ff586652da6f82dd996fd0a6edcdb0f72ab4",
"triggerType" : "PUSH"
}, {
"hash" : "7d356d05bf73a7263f448f4fad46ac0332ab8522",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9295",
"triggerID" : "7d356d05bf73a7263f448f4fad46ac0332ab8522",
"triggerType" : "PUSH"
}, {
"hash" : "41a8b4a7127307fa0263d701f5fc567c80857ea3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9360",
"triggerID" : "41a8b4a7127307fa0263d701f5fc567c80857ea3",
"triggerType" : "PUSH"
}, {
"hash" : "adc1713ee5af363f98d0a0ccfb2324b5c249a4fb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9362",
"triggerID" : "adc1713ee5af363f98d0a0ccfb2324b5c249a4fb",
"triggerType" : "PUSH"
}, {
"hash" : "0bbead9b3a4efb779b472a343731969ea0f4d7f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9364",
"triggerID" : "0bbead9b3a4efb779b472a343731969ea0f4d7f2",
"triggerType" : "PUSH"
}, {
"hash" : "5c1b891ad88030b8e286602b449eee96301dcb84",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9366",
"triggerID" : "5c1b891ad88030b8e286602b449eee96301dcb84",
"triggerType" : "PUSH"
}, {
"hash" : "9943cbbadb0ce515bb16ebbb7b39919711e8a362",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9369",
"triggerID" : "9943cbbadb0ce515bb16ebbb7b39919711e8a362",
"triggerType" : "PUSH"
}, {
"hash" : "ef0c4ca43b81d7b0c27a4e61ad93dea0c7df91a3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ef0c4ca43b81d7b0c27a4e61ad93dea0c7df91a3",
"triggerType" : "PUSH"
}, {
"hash" : "bc7d984512e3bdf695f35442a76581e815086e91",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9388",
"triggerID" : "bc7d984512e3bdf695f35442a76581e815086e91",
"triggerType" : "PUSH"
}, {
"hash" : "0f73e2d085ebacd3211a81e79c27b8a41f1b1b5f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0f73e2d085ebacd3211a81e79c27b8a41f1b1b5f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ef0c4ca43b81d7b0c27a4e61ad93dea0c7df91a3 UNKNOWN
* bc7d984512e3bdf695f35442a76581e815086e91 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9388)
* 0f73e2d085ebacd3211a81e79c27b8a41f1b1b5f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add HoodieBaseLanceFileWriter and implementation for SparkFileWriter … [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14131:
URL: https://github.com/apache/hudi/pull/14131#issuecomment-3456957917
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f1b8ff586652da6f82dd996fd0a6edcdb0f72ab4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9265",
"triggerID" : "f1b8ff586652da6f82dd996fd0a6edcdb0f72ab4",
"triggerType" : "PUSH"
}, {
"hash" : "7d356d05bf73a7263f448f4fad46ac0332ab8522",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9295",
"triggerID" : "7d356d05bf73a7263f448f4fad46ac0332ab8522",
"triggerType" : "PUSH"
}, {
"hash" : "41a8b4a7127307fa0263d701f5fc567c80857ea3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9360",
"triggerID" : "41a8b4a7127307fa0263d701f5fc567c80857ea3",
"triggerType" : "PUSH"
}, {
"hash" : "adc1713ee5af363f98d0a0ccfb2324b5c249a4fb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9362",
"triggerID" : "adc1713ee5af363f98d0a0ccfb2324b5c249a4fb",
"triggerType" : "PUSH"
}, {
"hash" : "0bbead9b3a4efb779b472a343731969ea0f4d7f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9364",
"triggerID" : "0bbead9b3a4efb779b472a343731969ea0f4d7f2",
"triggerType" : "PUSH"
}, {
"hash" : "5c1b891ad88030b8e286602b449eee96301dcb84",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9366",
"triggerID" : "5c1b891ad88030b8e286602b449eee96301dcb84",
"triggerType" : "PUSH"
}, {
"hash" : "9943cbbadb0ce515bb16ebbb7b39919711e8a362",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9369",
"triggerID" : "9943cbbadb0ce515bb16ebbb7b39919711e8a362",
"triggerType" : "PUSH"
}, {
"hash" : "ef0c4ca43b81d7b0c27a4e61ad93dea0c7df91a3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ef0c4ca43b81d7b0c27a4e61ad93dea0c7df91a3",
"triggerType" : "PUSH"
}, {
"hash" : "bc7d984512e3bdf695f35442a76581e815086e91",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9388",
"triggerID" : "bc7d984512e3bdf695f35442a76581e815086e91",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ef0c4ca43b81d7b0c27a4e61ad93dea0c7df91a3 UNKNOWN
* bc7d984512e3bdf695f35442a76581e815086e91 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9388)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add HoodieBaseLanceFileWriter and implementation for SparkFileWriter … [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on PR #14131:
URL: https://github.com/apache/hudi/pull/14131#issuecomment-3443926660
After making the profiles changes to run java 11 and 17 still seeing some of the module fail for some java issue and dependency issues.
1. java version issue (need to check if remaining module still run on java8)
```
##[error] /home/vsts/work/1/s/hudi-hadoop-common/src/main/java/org/apache/hudi/io/lance/HoodieBaseLanceWriter.java(23,31): error : /home/vsts/work/1/s/hudi-hadoop-common/src/main/java/org/apache/hudi/io/lance/HoodieBaseLanceWriter.java:[23,31] cannot access org.apache.arrow.memory.BufferAllocator
[ERROR] /home/vsts/work/1/s/hudi-hadoop-common/src/main/java/org/apache/hudi/io/lance/HoodieBaseLanceWriter.java:[23,31] cannot access org.apache.arrow.memory.BufferAllocator
bad class file: /home/vsts/.m2/repository/org/apache/arrow/arrow-memory-core/18.3.0/arrow-memory-core-18.3.0.jar(org/apache/arrow/memory/BufferAllocator.class)
class file has wrong version 55.0, should be 52.0
Please remove or make sure it appears in the correct subdirectory of the classpath.
```
2. TestLanceSparkWriter failing in ci for dependency issues (locally this test class runs fine)
<img width="979" height="331" alt="Screenshot 2025-10-24 at 9 07 53 AM" src="https://github.com/user-attachments/assets/d2c74b33-4d82-43ae-849f-9cc5c50e6b6e" />
```
Error: testWriteStructType Time elapsed: 0 s <<< ERROR!
java.lang.NoSuchMethodError: 'org.json4s.JsonAST$JValue org.apache.spark.sql.types.Metadata.jsonValue()'
at org.apache.spark.sql.util.LanceArrowUtils$.toArrowField(LanceArrowUtils.scala:133)
at org.apache.spark.sql.util.LanceArrowUtils$.$anonfun$toArrowSchema$1(LanceArrowUtils.scala:112)
at scala.collection.Iterator$$anon$9.next(Iterator.scala:584)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add HoodieBaseLanceFileWriter and implementation for SparkFileWriter … [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on PR #14131:
URL: https://github.com/apache/hudi/pull/14131#issuecomment-3453967760
One flaky test failing in the test spark java 17, the only thing left is the azure ci which i configured to run for >= java 11. Need to check why some modules are still running a lower java.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add HoodieBaseLanceFileWriter and implementation for SparkFileWriter … [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14131:
URL: https://github.com/apache/hudi/pull/14131#issuecomment-3453569087
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f1b8ff586652da6f82dd996fd0a6edcdb0f72ab4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9265",
"triggerID" : "f1b8ff586652da6f82dd996fd0a6edcdb0f72ab4",
"triggerType" : "PUSH"
}, {
"hash" : "7d356d05bf73a7263f448f4fad46ac0332ab8522",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9295",
"triggerID" : "7d356d05bf73a7263f448f4fad46ac0332ab8522",
"triggerType" : "PUSH"
}, {
"hash" : "41a8b4a7127307fa0263d701f5fc567c80857ea3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9360",
"triggerID" : "41a8b4a7127307fa0263d701f5fc567c80857ea3",
"triggerType" : "PUSH"
}, {
"hash" : "adc1713ee5af363f98d0a0ccfb2324b5c249a4fb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9362",
"triggerID" : "adc1713ee5af363f98d0a0ccfb2324b5c249a4fb",
"triggerType" : "PUSH"
}, {
"hash" : "0bbead9b3a4efb779b472a343731969ea0f4d7f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9364",
"triggerID" : "0bbead9b3a4efb779b472a343731969ea0f4d7f2",
"triggerType" : "PUSH"
}, {
"hash" : "5c1b891ad88030b8e286602b449eee96301dcb84",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9366",
"triggerID" : "5c1b891ad88030b8e286602b449eee96301dcb84",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5c1b891ad88030b8e286602b449eee96301dcb84 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9366)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add HoodieBaseLanceFileWriter and implementation for SparkFileWriter … [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on PR #14131:
URL: https://github.com/apache/hudi/pull/14131#issuecomment-3434774731
Change yaml to use correct java version, change branch in yaml, limit to java 17 spark 3.5, 4.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add HoodieBaseLanceFileWriter and implementation for SparkFileWriter … [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14131:
URL: https://github.com/apache/hudi/pull/14131#issuecomment-3453001635
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f1b8ff586652da6f82dd996fd0a6edcdb0f72ab4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9265",
"triggerID" : "f1b8ff586652da6f82dd996fd0a6edcdb0f72ab4",
"triggerType" : "PUSH"
}, {
"hash" : "7d356d05bf73a7263f448f4fad46ac0332ab8522",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9295",
"triggerID" : "7d356d05bf73a7263f448f4fad46ac0332ab8522",
"triggerType" : "PUSH"
}, {
"hash" : "41a8b4a7127307fa0263d701f5fc567c80857ea3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9360",
"triggerID" : "41a8b4a7127307fa0263d701f5fc567c80857ea3",
"triggerType" : "PUSH"
}, {
"hash" : "adc1713ee5af363f98d0a0ccfb2324b5c249a4fb",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9362",
"triggerID" : "adc1713ee5af363f98d0a0ccfb2324b5c249a4fb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 41a8b4a7127307fa0263d701f5fc567c80857ea3 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9360)
* adc1713ee5af363f98d0a0ccfb2324b5c249a4fb Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9362)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add HoodieBaseLanceFileWriter and implementation for SparkFileWriter … [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14131:
URL: https://github.com/apache/hudi/pull/14131#issuecomment-3452986055
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f1b8ff586652da6f82dd996fd0a6edcdb0f72ab4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9265",
"triggerID" : "f1b8ff586652da6f82dd996fd0a6edcdb0f72ab4",
"triggerType" : "PUSH"
}, {
"hash" : "7d356d05bf73a7263f448f4fad46ac0332ab8522",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9295",
"triggerID" : "7d356d05bf73a7263f448f4fad46ac0332ab8522",
"triggerType" : "PUSH"
}, {
"hash" : "41a8b4a7127307fa0263d701f5fc567c80857ea3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9360",
"triggerID" : "41a8b4a7127307fa0263d701f5fc567c80857ea3",
"triggerType" : "PUSH"
}, {
"hash" : "adc1713ee5af363f98d0a0ccfb2324b5c249a4fb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "adc1713ee5af363f98d0a0ccfb2324b5c249a4fb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 41a8b4a7127307fa0263d701f5fc567c80857ea3 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9360)
* adc1713ee5af363f98d0a0ccfb2324b5c249a4fb UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add HoodieBaseLanceFileWriter and implementation for SparkFileWriter … [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14131:
URL: https://github.com/apache/hudi/pull/14131#issuecomment-3453024254
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f1b8ff586652da6f82dd996fd0a6edcdb0f72ab4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9265",
"triggerID" : "f1b8ff586652da6f82dd996fd0a6edcdb0f72ab4",
"triggerType" : "PUSH"
}, {
"hash" : "7d356d05bf73a7263f448f4fad46ac0332ab8522",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9295",
"triggerID" : "7d356d05bf73a7263f448f4fad46ac0332ab8522",
"triggerType" : "PUSH"
}, {
"hash" : "41a8b4a7127307fa0263d701f5fc567c80857ea3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9360",
"triggerID" : "41a8b4a7127307fa0263d701f5fc567c80857ea3",
"triggerType" : "PUSH"
}, {
"hash" : "adc1713ee5af363f98d0a0ccfb2324b5c249a4fb",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9362",
"triggerID" : "adc1713ee5af363f98d0a0ccfb2324b5c249a4fb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* adc1713ee5af363f98d0a0ccfb2324b5c249a4fb Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9362)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add HoodieBaseLanceFileWriter and implementation for SparkFileWriter … [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #14131:
URL: https://github.com/apache/hudi/pull/14131#discussion_r2466495709
##########
pom.xml:
##########
@@ -897,7 +900,29 @@
<version>${orc.spark.version}</version>
<scope>compile</scope>
</dependency>
-
+ <!-- Lance Core SDK -->
+ <dependency>
+ <groupId>com.lancedb</groupId>
+ <artifactId>lance-core</artifactId>
+ <version>${lance.version}</version>
+ </dependency>
+ <!-- Lance Spark for Arrow conversion -->
+ <dependency>
+ <groupId>com.lancedb</groupId>
+ <artifactId>lance-spark-base_${scala.binary.version}</artifactId>
Review Comment:
I think its more around the fact that our ci is currently running for spark 3.3 and spark 3.4, so we either need to restrict those profiles or have a way of passing the correct lance spark version for each spark profile we have in root pom. https://lancedb.github.io/lance/integrations/spark/install/#scala
Ill try using the specific spark version artfiact (lean jar) mentioned here https://lancedb.github.io/lance/integrations/spark/install/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add HoodieBaseLanceFileWriter and implementation for SparkFileWriter … [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14131:
URL: https://github.com/apache/hudi/pull/14131#issuecomment-3437251480
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f1b8ff586652da6f82dd996fd0a6edcdb0f72ab4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9265",
"triggerID" : "f1b8ff586652da6f82dd996fd0a6edcdb0f72ab4",
"triggerType" : "PUSH"
}, {
"hash" : "7d356d05bf73a7263f448f4fad46ac0332ab8522",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9295",
"triggerID" : "7d356d05bf73a7263f448f4fad46ac0332ab8522",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7d356d05bf73a7263f448f4fad46ac0332ab8522 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9295)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add HoodieBaseLanceFileWriter and implementation for SparkFileWriter … [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14131:
URL: https://github.com/apache/hudi/pull/14131#issuecomment-3453214216
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f1b8ff586652da6f82dd996fd0a6edcdb0f72ab4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9265",
"triggerID" : "f1b8ff586652da6f82dd996fd0a6edcdb0f72ab4",
"triggerType" : "PUSH"
}, {
"hash" : "7d356d05bf73a7263f448f4fad46ac0332ab8522",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9295",
"triggerID" : "7d356d05bf73a7263f448f4fad46ac0332ab8522",
"triggerType" : "PUSH"
}, {
"hash" : "41a8b4a7127307fa0263d701f5fc567c80857ea3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9360",
"triggerID" : "41a8b4a7127307fa0263d701f5fc567c80857ea3",
"triggerType" : "PUSH"
}, {
"hash" : "adc1713ee5af363f98d0a0ccfb2324b5c249a4fb",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9362",
"triggerID" : "adc1713ee5af363f98d0a0ccfb2324b5c249a4fb",
"triggerType" : "PUSH"
}, {
"hash" : "0bbead9b3a4efb779b472a343731969ea0f4d7f2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9364",
"triggerID" : "0bbead9b3a4efb779b472a343731969ea0f4d7f2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* adc1713ee5af363f98d0a0ccfb2324b5c249a4fb Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9362)
* 0bbead9b3a4efb779b472a343731969ea0f4d7f2 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9364)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add HoodieBaseLanceFileWriter and implementation for SparkFileWriter … [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14131:
URL: https://github.com/apache/hudi/pull/14131#issuecomment-3453552712
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f1b8ff586652da6f82dd996fd0a6edcdb0f72ab4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9265",
"triggerID" : "f1b8ff586652da6f82dd996fd0a6edcdb0f72ab4",
"triggerType" : "PUSH"
}, {
"hash" : "7d356d05bf73a7263f448f4fad46ac0332ab8522",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9295",
"triggerID" : "7d356d05bf73a7263f448f4fad46ac0332ab8522",
"triggerType" : "PUSH"
}, {
"hash" : "41a8b4a7127307fa0263d701f5fc567c80857ea3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9360",
"triggerID" : "41a8b4a7127307fa0263d701f5fc567c80857ea3",
"triggerType" : "PUSH"
}, {
"hash" : "adc1713ee5af363f98d0a0ccfb2324b5c249a4fb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9362",
"triggerID" : "adc1713ee5af363f98d0a0ccfb2324b5c249a4fb",
"triggerType" : "PUSH"
}, {
"hash" : "0bbead9b3a4efb779b472a343731969ea0f4d7f2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9364",
"triggerID" : "0bbead9b3a4efb779b472a343731969ea0f4d7f2",
"triggerType" : "PUSH"
}, {
"hash" : "5c1b891ad88030b8e286602b449eee96301dcb84",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5c1b891ad88030b8e286602b449eee96301dcb84",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0bbead9b3a4efb779b472a343731969ea0f4d7f2 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9364)
* 5c1b891ad88030b8e286602b449eee96301dcb84 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add HoodieBaseLanceFileWriter and implementation for SparkFileWriter … [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14131:
URL: https://github.com/apache/hudi/pull/14131#issuecomment-3453630720
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f1b8ff586652da6f82dd996fd0a6edcdb0f72ab4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9265",
"triggerID" : "f1b8ff586652da6f82dd996fd0a6edcdb0f72ab4",
"triggerType" : "PUSH"
}, {
"hash" : "7d356d05bf73a7263f448f4fad46ac0332ab8522",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9295",
"triggerID" : "7d356d05bf73a7263f448f4fad46ac0332ab8522",
"triggerType" : "PUSH"
}, {
"hash" : "41a8b4a7127307fa0263d701f5fc567c80857ea3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9360",
"triggerID" : "41a8b4a7127307fa0263d701f5fc567c80857ea3",
"triggerType" : "PUSH"
}, {
"hash" : "adc1713ee5af363f98d0a0ccfb2324b5c249a4fb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9362",
"triggerID" : "adc1713ee5af363f98d0a0ccfb2324b5c249a4fb",
"triggerType" : "PUSH"
}, {
"hash" : "0bbead9b3a4efb779b472a343731969ea0f4d7f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9364",
"triggerID" : "0bbead9b3a4efb779b472a343731969ea0f4d7f2",
"triggerType" : "PUSH"
}, {
"hash" : "5c1b891ad88030b8e286602b449eee96301dcb84",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9366",
"triggerID" : "5c1b891ad88030b8e286602b449eee96301dcb84",
"triggerType" : "PUSH"
}, {
"hash" : "9943cbbadb0ce515bb16ebbb7b39919711e8a362",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9943cbbadb0ce515bb16ebbb7b39919711e8a362",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5c1b891ad88030b8e286602b449eee96301dcb84 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9366)
* 9943cbbadb0ce515bb16ebbb7b39919711e8a362 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add HoodieBaseLanceFileWriter and implementation for SparkFileWriter … [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #14131:
URL: https://github.com/apache/hudi/pull/14131#discussion_r2466284250
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/HoodieSparkLanceWriter.java:
##########
@@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.io.storage;
+
+import com.lancedb.lance.spark.arrow.LanceArrowWriter;
+import org.apache.arrow.vector.types.pojo.Schema;
+import org.apache.hudi.common.engine.TaskContextSupplier;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.io.lance.HoodieBaseLanceWriter;
+import org.apache.hudi.storage.HoodieStorage;
+import org.apache.hudi.storage.StoragePath;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.util.LanceArrowUtils;
+import org.apache.spark.unsafe.types.UTF8String;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.function.Function;
+
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.COMMIT_SEQNO_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.COMMIT_TIME_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.FILENAME_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.PARTITION_PATH_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.RECORD_KEY_METADATA_FIELD;
+
+/**
+ * Spark Lance file writer implementing {@link HoodieSparkFileWriter}.
+ *
+ * This writer integrates with Hudi's storage I/O layer and supports:
+ * - Hudi metadata field population
+ * - Record key tracking (for bloom filters - TODO)
Review Comment:
will add to github discussion board https://github.com/apache/hudi/discussions/14128
##########
hudi-hadoop-common/src/main/java/org/apache/hudi/io/lance/HoodieBaseLanceWriter.java:
##########
@@ -0,0 +1,196 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.io.lance;
+
+import com.lancedb.lance.file.LanceFileWriter;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.memory.RootAllocator;
+import org.apache.arrow.vector.VectorSchemaRoot;
+import org.apache.arrow.vector.types.pojo.Schema;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.storage.HoodieStorage;
+import org.apache.hudi.storage.StoragePath;
+
+import javax.annotation.concurrent.NotThreadSafe;
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * Base class for Hudi Lance file writers supporting different record types.
+ *
+ * This class handles common Lance file operations including:
+ * - LanceFileWriter lifecycle management
+ * - BufferAllocator management
+ * - Record buffering and batch flushing
+ * - File size checks
+ *
+ * Subclasses must implement type-specific conversion to Arrow format.
+ *
+ * @param <R> The record type (e.g., GenericRecord, InternalRow)
+ */
+@NotThreadSafe
+public abstract class HoodieBaseLanceWriter<R> implements Closeable {
+ protected static final int DEFAULT_BATCH_SIZE = 1000;
+ protected final HoodieStorage storage;
+ protected final StoragePath path;
+ protected final BufferAllocator allocator;
+ protected final List<R> bufferedRecords;
+ protected final int batchSize;
+ protected final long maxFileSize;
+ protected long writtenRecordCount = 0;
+ protected VectorSchemaRoot root;
+
+ private LanceFileWriter writer;
+
+ /**
+ * Constructor for base Lance writer.
+ *
+ * @param storage HoodieStorage instance
+ * @param path Path where Lance file will be written
+ * @param batchSize Number of records to buffer before flushing to Lance
+ * @param maxFileSize Maximum file size in bytes before rolling over to new file
+ */
+ protected HoodieBaseLanceWriter(HoodieStorage storage, StoragePath path, int batchSize, long maxFileSize) {
+ this.storage = storage;
+ this.path = path;
+ this.allocator = new RootAllocator(Long.MAX_VALUE);
+ this.bufferedRecords = new ArrayList<>(batchSize);
+ this.batchSize = batchSize;
+ this.maxFileSize = maxFileSize;
+ }
+
+ /**
+ * Populate the VectorSchemaRoot with buffered records.
+ * Subclasses must implement type-specific conversion logic.
+ * The VectorSchemaRoot field is reused across batches and managed by this base class.
+ *
+ * @param records List of records to convert
+ */
+ protected abstract void populateVectorSchemaRoot(List<R> records);
+
+ /**
+ * Get the Arrow schema for this writer.
+ * Subclasses must provide the Arrow schema corresponding to their record type.
+ *
+ * @return Arrow schema
+ */
+ protected abstract Schema getArrowSchema();
+
+ /**
+ * Write a single record. Records are buffered and flushed in batches.
+ *
+ * @param record Record to write
+ * @throws IOException if write fails
+ */
+ public void write(R record) throws IOException {
+ bufferedRecords.add(record);
+ writtenRecordCount++;
+
+ if (bufferedRecords.size() >= batchSize) {
+ flushBatch();
+ }
+ }
+
+ /**
+ * Check if writer can accept more records based on file size.
+ * Uses filesystem-based size checking (similar to ORC/HFile approach).
+ *
+ * @return true if writer can accept more records, false if file size limit reached
+ */
+ public boolean canWrite() {
+ //TODO will need to implement proper way to compute this
Review Comment:
will add to github discussion board https://github.com/apache/hudi/discussions/14128
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add HoodieBaseLanceFileWriter and implementation for SparkFileWriter … [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14131:
URL: https://github.com/apache/hudi/pull/14131#issuecomment-3452589661
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f1b8ff586652da6f82dd996fd0a6edcdb0f72ab4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9265",
"triggerID" : "f1b8ff586652da6f82dd996fd0a6edcdb0f72ab4",
"triggerType" : "PUSH"
}, {
"hash" : "7d356d05bf73a7263f448f4fad46ac0332ab8522",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9295",
"triggerID" : "7d356d05bf73a7263f448f4fad46ac0332ab8522",
"triggerType" : "PUSH"
}, {
"hash" : "41a8b4a7127307fa0263d701f5fc567c80857ea3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "41a8b4a7127307fa0263d701f5fc567c80857ea3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7d356d05bf73a7263f448f4fad46ac0332ab8522 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9295)
* 41a8b4a7127307fa0263d701f5fc567c80857ea3 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add HoodieBaseLanceFileWriter and implementation for SparkFileWriter … [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14131:
URL: https://github.com/apache/hudi/pull/14131#issuecomment-3456605921
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f1b8ff586652da6f82dd996fd0a6edcdb0f72ab4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9265",
"triggerID" : "f1b8ff586652da6f82dd996fd0a6edcdb0f72ab4",
"triggerType" : "PUSH"
}, {
"hash" : "7d356d05bf73a7263f448f4fad46ac0332ab8522",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9295",
"triggerID" : "7d356d05bf73a7263f448f4fad46ac0332ab8522",
"triggerType" : "PUSH"
}, {
"hash" : "41a8b4a7127307fa0263d701f5fc567c80857ea3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9360",
"triggerID" : "41a8b4a7127307fa0263d701f5fc567c80857ea3",
"triggerType" : "PUSH"
}, {
"hash" : "adc1713ee5af363f98d0a0ccfb2324b5c249a4fb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9362",
"triggerID" : "adc1713ee5af363f98d0a0ccfb2324b5c249a4fb",
"triggerType" : "PUSH"
}, {
"hash" : "0bbead9b3a4efb779b472a343731969ea0f4d7f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9364",
"triggerID" : "0bbead9b3a4efb779b472a343731969ea0f4d7f2",
"triggerType" : "PUSH"
}, {
"hash" : "5c1b891ad88030b8e286602b449eee96301dcb84",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9366",
"triggerID" : "5c1b891ad88030b8e286602b449eee96301dcb84",
"triggerType" : "PUSH"
}, {
"hash" : "9943cbbadb0ce515bb16ebbb7b39919711e8a362",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9369",
"triggerID" : "9943cbbadb0ce515bb16ebbb7b39919711e8a362",
"triggerType" : "PUSH"
}, {
"hash" : "ef0c4ca43b81d7b0c27a4e61ad93dea0c7df91a3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ef0c4ca43b81d7b0c27a4e61ad93dea0c7df91a3",
"triggerType" : "PUSH"
}, {
"hash" : "bc7d984512e3bdf695f35442a76581e815086e91",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bc7d984512e3bdf695f35442a76581e815086e91",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9943cbbadb0ce515bb16ebbb7b39919711e8a362 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9369)
* ef0c4ca43b81d7b0c27a4e61ad93dea0c7df91a3 UNKNOWN
* bc7d984512e3bdf695f35442a76581e815086e91 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add HoodieBaseLanceFileWriter and implementation for SparkFileWriter … [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14131:
URL: https://github.com/apache/hudi/pull/14131#issuecomment-3456599809
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f1b8ff586652da6f82dd996fd0a6edcdb0f72ab4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9265",
"triggerID" : "f1b8ff586652da6f82dd996fd0a6edcdb0f72ab4",
"triggerType" : "PUSH"
}, {
"hash" : "7d356d05bf73a7263f448f4fad46ac0332ab8522",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9295",
"triggerID" : "7d356d05bf73a7263f448f4fad46ac0332ab8522",
"triggerType" : "PUSH"
}, {
"hash" : "41a8b4a7127307fa0263d701f5fc567c80857ea3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9360",
"triggerID" : "41a8b4a7127307fa0263d701f5fc567c80857ea3",
"triggerType" : "PUSH"
}, {
"hash" : "adc1713ee5af363f98d0a0ccfb2324b5c249a4fb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9362",
"triggerID" : "adc1713ee5af363f98d0a0ccfb2324b5c249a4fb",
"triggerType" : "PUSH"
}, {
"hash" : "0bbead9b3a4efb779b472a343731969ea0f4d7f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9364",
"triggerID" : "0bbead9b3a4efb779b472a343731969ea0f4d7f2",
"triggerType" : "PUSH"
}, {
"hash" : "5c1b891ad88030b8e286602b449eee96301dcb84",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9366",
"triggerID" : "5c1b891ad88030b8e286602b449eee96301dcb84",
"triggerType" : "PUSH"
}, {
"hash" : "9943cbbadb0ce515bb16ebbb7b39919711e8a362",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9369",
"triggerID" : "9943cbbadb0ce515bb16ebbb7b39919711e8a362",
"triggerType" : "PUSH"
}, {
"hash" : "ef0c4ca43b81d7b0c27a4e61ad93dea0c7df91a3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ef0c4ca43b81d7b0c27a4e61ad93dea0c7df91a3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9943cbbadb0ce515bb16ebbb7b39919711e8a362 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9369)
* ef0c4ca43b81d7b0c27a4e61ad93dea0c7df91a3 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add HoodieBaseLanceFileWriter and implementation for SparkFileWriter … [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #14131:
URL: https://github.com/apache/hudi/pull/14131#discussion_r2466284250
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/HoodieSparkLanceWriter.java:
##########
@@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.io.storage;
+
+import com.lancedb.lance.spark.arrow.LanceArrowWriter;
+import org.apache.arrow.vector.types.pojo.Schema;
+import org.apache.hudi.common.engine.TaskContextSupplier;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.io.lance.HoodieBaseLanceWriter;
+import org.apache.hudi.storage.HoodieStorage;
+import org.apache.hudi.storage.StoragePath;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.util.LanceArrowUtils;
+import org.apache.spark.unsafe.types.UTF8String;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.function.Function;
+
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.COMMIT_SEQNO_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.COMMIT_TIME_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.FILENAME_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.PARTITION_PATH_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.RECORD_KEY_METADATA_FIELD;
+
+/**
+ * Spark Lance file writer implementing {@link HoodieSparkFileWriter}.
+ *
+ * This writer integrates with Hudi's storage I/O layer and supports:
+ * - Hudi metadata field population
+ * - Record key tracking (for bloom filters - TODO)
Review Comment:
will do
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add HoodieBaseLanceFileWriter and implementation for SparkFileWriter … [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #14131:
URL: https://github.com/apache/hudi/pull/14131#discussion_r2466284250
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/HoodieSparkLanceWriter.java:
##########
@@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.io.storage;
+
+import com.lancedb.lance.spark.arrow.LanceArrowWriter;
+import org.apache.arrow.vector.types.pojo.Schema;
+import org.apache.hudi.common.engine.TaskContextSupplier;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.io.lance.HoodieBaseLanceWriter;
+import org.apache.hudi.storage.HoodieStorage;
+import org.apache.hudi.storage.StoragePath;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.util.LanceArrowUtils;
+import org.apache.spark.unsafe.types.UTF8String;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.function.Function;
+
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.COMMIT_SEQNO_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.COMMIT_TIME_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.FILENAME_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.PARTITION_PATH_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.RECORD_KEY_METADATA_FIELD;
+
+/**
+ * Spark Lance file writer implementing {@link HoodieSparkFileWriter}.
+ *
+ * This writer integrates with Hudi's storage I/O layer and supports:
+ * - Hudi metadata field population
+ * - Record key tracking (for bloom filters - TODO)
Review Comment:
will add to github discussion board
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add HoodieBaseLanceFileWriter and implementation for SparkFileWriter … [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14131:
URL: https://github.com/apache/hudi/pull/14131#issuecomment-3458462849
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f1b8ff586652da6f82dd996fd0a6edcdb0f72ab4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9265",
"triggerID" : "f1b8ff586652da6f82dd996fd0a6edcdb0f72ab4",
"triggerType" : "PUSH"
}, {
"hash" : "7d356d05bf73a7263f448f4fad46ac0332ab8522",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9295",
"triggerID" : "7d356d05bf73a7263f448f4fad46ac0332ab8522",
"triggerType" : "PUSH"
}, {
"hash" : "41a8b4a7127307fa0263d701f5fc567c80857ea3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9360",
"triggerID" : "41a8b4a7127307fa0263d701f5fc567c80857ea3",
"triggerType" : "PUSH"
}, {
"hash" : "adc1713ee5af363f98d0a0ccfb2324b5c249a4fb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9362",
"triggerID" : "adc1713ee5af363f98d0a0ccfb2324b5c249a4fb",
"triggerType" : "PUSH"
}, {
"hash" : "0bbead9b3a4efb779b472a343731969ea0f4d7f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9364",
"triggerID" : "0bbead9b3a4efb779b472a343731969ea0f4d7f2",
"triggerType" : "PUSH"
}, {
"hash" : "5c1b891ad88030b8e286602b449eee96301dcb84",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9366",
"triggerID" : "5c1b891ad88030b8e286602b449eee96301dcb84",
"triggerType" : "PUSH"
}, {
"hash" : "9943cbbadb0ce515bb16ebbb7b39919711e8a362",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9369",
"triggerID" : "9943cbbadb0ce515bb16ebbb7b39919711e8a362",
"triggerType" : "PUSH"
}, {
"hash" : "ef0c4ca43b81d7b0c27a4e61ad93dea0c7df91a3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ef0c4ca43b81d7b0c27a4e61ad93dea0c7df91a3",
"triggerType" : "PUSH"
}, {
"hash" : "bc7d984512e3bdf695f35442a76581e815086e91",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9388",
"triggerID" : "bc7d984512e3bdf695f35442a76581e815086e91",
"triggerType" : "PUSH"
}, {
"hash" : "0f73e2d085ebacd3211a81e79c27b8a41f1b1b5f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9394",
"triggerID" : "0f73e2d085ebacd3211a81e79c27b8a41f1b1b5f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ef0c4ca43b81d7b0c27a4e61ad93dea0c7df91a3 UNKNOWN
* bc7d984512e3bdf695f35442a76581e815086e91 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9388)
* 0f73e2d085ebacd3211a81e79c27b8a41f1b1b5f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9394)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add HoodieBaseLanceFileWriter and implementation for SparkFileWriter … [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14131:
URL: https://github.com/apache/hudi/pull/14131#issuecomment-3452594940
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f1b8ff586652da6f82dd996fd0a6edcdb0f72ab4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9265",
"triggerID" : "f1b8ff586652da6f82dd996fd0a6edcdb0f72ab4",
"triggerType" : "PUSH"
}, {
"hash" : "7d356d05bf73a7263f448f4fad46ac0332ab8522",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9295",
"triggerID" : "7d356d05bf73a7263f448f4fad46ac0332ab8522",
"triggerType" : "PUSH"
}, {
"hash" : "41a8b4a7127307fa0263d701f5fc567c80857ea3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9360",
"triggerID" : "41a8b4a7127307fa0263d701f5fc567c80857ea3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7d356d05bf73a7263f448f4fad46ac0332ab8522 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9295)
* 41a8b4a7127307fa0263d701f5fc567c80857ea3 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9360)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add HoodieBaseLanceFileWriter and implementation for SparkFileWriter … [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14131:
URL: https://github.com/apache/hudi/pull/14131#issuecomment-3453748742
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f1b8ff586652da6f82dd996fd0a6edcdb0f72ab4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9265",
"triggerID" : "f1b8ff586652da6f82dd996fd0a6edcdb0f72ab4",
"triggerType" : "PUSH"
}, {
"hash" : "7d356d05bf73a7263f448f4fad46ac0332ab8522",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9295",
"triggerID" : "7d356d05bf73a7263f448f4fad46ac0332ab8522",
"triggerType" : "PUSH"
}, {
"hash" : "41a8b4a7127307fa0263d701f5fc567c80857ea3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9360",
"triggerID" : "41a8b4a7127307fa0263d701f5fc567c80857ea3",
"triggerType" : "PUSH"
}, {
"hash" : "adc1713ee5af363f98d0a0ccfb2324b5c249a4fb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9362",
"triggerID" : "adc1713ee5af363f98d0a0ccfb2324b5c249a4fb",
"triggerType" : "PUSH"
}, {
"hash" : "0bbead9b3a4efb779b472a343731969ea0f4d7f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9364",
"triggerID" : "0bbead9b3a4efb779b472a343731969ea0f4d7f2",
"triggerType" : "PUSH"
}, {
"hash" : "5c1b891ad88030b8e286602b449eee96301dcb84",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9366",
"triggerID" : "5c1b891ad88030b8e286602b449eee96301dcb84",
"triggerType" : "PUSH"
}, {
"hash" : "9943cbbadb0ce515bb16ebbb7b39919711e8a362",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9369",
"triggerID" : "9943cbbadb0ce515bb16ebbb7b39919711e8a362",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9943cbbadb0ce515bb16ebbb7b39919711e8a362 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9369)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add HoodieBaseLanceFileWriter and implementation for SparkFileWriter … [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #14131:
URL: https://github.com/apache/hudi/pull/14131#discussion_r2470683559
##########
hudi-hadoop-common/src/main/java/org/apache/hudi/io/lance/HoodieBaseLanceWriter.java:
##########
@@ -0,0 +1,196 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.io.lance;
+
+import com.lancedb.lance.file.LanceFileWriter;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.memory.RootAllocator;
+import org.apache.arrow.vector.VectorSchemaRoot;
+import org.apache.arrow.vector.types.pojo.Schema;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.storage.HoodieStorage;
+import org.apache.hudi.storage.StoragePath;
+
+import javax.annotation.concurrent.NotThreadSafe;
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * Base class for Hudi Lance file writers supporting different record types.
+ *
+ * This class handles common Lance file operations including:
+ * - LanceFileWriter lifecycle management
+ * - BufferAllocator management
+ * - Record buffering and batch flushing
+ * - File size checks
+ *
+ * Subclasses must implement type-specific conversion to Arrow format.
+ *
+ * @param <R> The record type (e.g., GenericRecord, InternalRow)
+ */
+@NotThreadSafe
+public abstract class HoodieBaseLanceWriter<R> implements Closeable {
+ protected static final int DEFAULT_BATCH_SIZE = 1000;
+ protected final HoodieStorage storage;
+ protected final StoragePath path;
+ protected final BufferAllocator allocator;
+ protected final List<R> bufferedRecords;
+ protected final int batchSize;
+ protected final long maxFileSize;
+ protected long writtenRecordCount = 0;
+ protected VectorSchemaRoot root;
+
+ private LanceFileWriter writer;
+
+ /**
+ * Constructor for base Lance writer.
+ *
+ * @param storage HoodieStorage instance
+ * @param path Path where Lance file will be written
+ * @param batchSize Number of records to buffer before flushing to Lance
+ * @param maxFileSize Maximum file size in bytes before rolling over to new file
+ */
+ protected HoodieBaseLanceWriter(HoodieStorage storage, StoragePath path, int batchSize, long maxFileSize) {
+ this.storage = storage;
+ this.path = path;
+ this.allocator = new RootAllocator(Long.MAX_VALUE);
Review Comment:
nit: does `Long.MAX_VALUE` control the memory usage?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add HoodieBaseLanceFileWriter and implementation for SparkFileWriter … [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14131:
URL: https://github.com/apache/hudi/pull/14131#issuecomment-3437234574
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f1b8ff586652da6f82dd996fd0a6edcdb0f72ab4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9265",
"triggerID" : "f1b8ff586652da6f82dd996fd0a6edcdb0f72ab4",
"triggerType" : "PUSH"
}, {
"hash" : "7d356d05bf73a7263f448f4fad46ac0332ab8522",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7d356d05bf73a7263f448f4fad46ac0332ab8522",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f1b8ff586652da6f82dd996fd0a6edcdb0f72ab4 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9265)
* 7d356d05bf73a7263f448f4fad46ac0332ab8522 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add HoodieBaseLanceFileWriter and implementation for SparkFileWriter … [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #14131:
URL: https://github.com/apache/hudi/pull/14131#discussion_r2470666966
##########
.github/workflows/bot.yml:
##########
@@ -882,41 +609,6 @@ jobs:
mvn clean install -T 2 -Pintegration-tests -D"$SCALA_PROFILE" -D"$FLINK_PROFILE" -pl hudi-flink-datasource/hudi-flink -am -Davro.version="$FLINK_AVRO_VERSION" -Dparquet.version="$FLINK_PARQUET_VERSION" -DskipTests=true $MVN_ARGS
mvn verify -Pintegration-tests -D"$SCALA_PROFILE" -D"$FLINK_PROFILE" -Davro.version="$FLINK_AVRO_VERSION" -Dparquet.version="$FLINK_PARQUET_VERSION" $FLINK_IT_FILTER1 -pl hudi-flink-datasource/hudi-flink $MVN_ARGS
- test-flink-2:
- runs-on: ubuntu-latest
- strategy:
- matrix:
- include:
- - flinkProfile: "flink1.20"
- flinkAvroVersion: "1.11.4"
- flinkParquetVersion: '1.13.1'
- steps:
- - uses: actions/checkout@v3
- - name: Set up JDK 8
- uses: actions/setup-java@v3
- with:
- java-version: '8'
- distribution: 'temurin'
- architecture: x64
- - name: Build Project
- env:
- SCALA_PROFILE: 'scala-2.12'
- FLINK_PROFILE: ${{ matrix.flinkProfile }}
- FLINK_AVRO_VERSION: ${{ matrix.flinkAvroVersion }}
- FLINK_PARQUET_VERSION: ${{ matrix.flinkParquetVersion }}
- run:
- mvn clean install -T 2 -D"$SCALA_PROFILE" -D"$FLINK_PROFILE" -pl hudi-examples/hudi-examples-flink -am -Davro.version="$FLINK_AVRO_VERSION" -Dparquet.version="$FLINK_PARQUET_VERSION" -DskipTests=true $MVN_ARGS
- - name: Integration Test 2
- env:
- SCALA_PROFILE: 'scala-2.12'
- FLINK_PROFILE: ${{ matrix.flinkProfile }}
- FLINK_AVRO_VERSION: ${{ matrix.flinkAvroVersion }}
- FLINK_PARQUET_VERSION: ${{ matrix.flinkParquetVersion }}
- if: ${{ endsWith(env.FLINK_PROFILE, '1.20') }}
- run: |
- mvn clean install -T 2 -Pintegration-tests -D"$SCALA_PROFILE" -D"$FLINK_PROFILE" -pl hudi-flink-datasource/hudi-flink -am -Davro.version="$FLINK_AVRO_VERSION" -Dparquet.version="$FLINK_PARQUET_VERSION" -DskipTests=true $MVN_ARGS
- mvn verify -Pintegration-tests -D"$SCALA_PROFILE" -D"$FLINK_PROFILE" -Davro.version="$FLINK_AVRO_VERSION" -Dparquet.version="$FLINK_PARQUET_VERSION" $FLINK_IT_FILTER2 -pl hudi-flink-datasource/hudi-flink $MVN_ARGS
Review Comment:
Upgrade this to Java 11 so we have test coverage on flink-2?
##########
.github/workflows/bot.yml:
##########
@@ -958,88 +650,6 @@ jobs:
HUDI_VERSION=$(mvn help:evaluate -Dexpression=project.version -q -DforceStdout)
./packaging/bundle-validation/run_docker_java17.sh
- validate-bundles:
- runs-on: ubuntu-latest
- strategy:
- matrix:
- include:
- - scalaProfile: 'scala-2.13'
- flinkProfile: 'flink1.20'
- flinkAvroVersion: '1.11.4'
- flinkParquetVersion: '1.13.1'
- sparkProfile: 'spark3.5'
- sparkRuntime: 'spark3.5.1'
- - scalaProfile: 'scala-2.13'
- flinkProfile: 'flink1.19'
- flinkAvroVersion: '1.11.4'
- flinkParquetVersion: '1.13.1'
- sparkProfile: 'spark3.5'
- sparkRuntime: 'spark3.5.1'
- - scalaProfile: 'scala-2.12'
- flinkProfile: 'flink1.18'
- flinkAvroVersion: '1.11.4'
- flinkParquetVersion: '1.13.1'
- sparkProfile: 'spark3.4'
- sparkRuntime: 'spark3.4.3'
- - scalaProfile: 'scala-2.12'
- flinkProfile: 'flink1.17'
- flinkAvroVersion: '1.11.4'
- flinkParquetVersion: '1.12.3'
- sparkProfile: 'spark3.3'
- sparkRuntime: 'spark3.3.4'
-
- steps:
- - uses: actions/checkout@v3
- - name: Set up JDK 8
- uses: actions/setup-java@v3
- with:
- java-version: '8'
Review Comment:
Use Java 11?
##########
.github/workflows/bot.yml:
##########
@@ -59,281 +61,6 @@ jobs:
- name: RAT check
run: ./scripts/release/validate_source_rat.sh
- test-spark-java-tests-part1:
- runs-on: ubuntu-latest
- strategy:
- matrix:
- include:
- - scalaProfile: "scala-2.12"
- sparkProfile: "spark3.3"
- sparkModules: "hudi-spark-datasource/hudi-spark3.3.x"
-
- - scalaProfile: "scala-2.12"
- sparkProfile: "spark3.4"
- sparkModules: "hudi-spark-datasource/hudi-spark3.4.x"
-
- - scalaProfile: "scala-2.12"
- sparkProfile: "spark3.5"
- sparkModules: "hudi-spark-datasource/hudi-spark3.5.x"
-
- - scalaProfile: "scala-2.13"
- sparkProfile: "spark3.5"
- sparkModules: "hudi-spark-datasource/hudi-spark3.5.x"
-
- steps:
- - uses: actions/checkout@v3
- - name: Set up JDK 8
- uses: actions/setup-java@v3
- with:
- java-version: '8'
Review Comment:
So the removal of these actions is because Java 8 is not support for Lance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add HoodieBaseLanceFileWriter and implementation for SparkFileWriter … [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #14131:
URL: https://github.com/apache/hudi/pull/14131#discussion_r2470675272
##########
.github/workflows/bot.yml:
##########
@@ -59,281 +61,6 @@ jobs:
- name: RAT check
run: ./scripts/release/validate_source_rat.sh
- test-spark-java-tests-part1:
- runs-on: ubuntu-latest
- strategy:
- matrix:
- include:
- - scalaProfile: "scala-2.12"
- sparkProfile: "spark3.3"
- sparkModules: "hudi-spark-datasource/hudi-spark3.3.x"
-
- - scalaProfile: "scala-2.12"
- sparkProfile: "spark3.4"
- sparkModules: "hudi-spark-datasource/hudi-spark3.4.x"
-
- - scalaProfile: "scala-2.12"
- sparkProfile: "spark3.5"
- sparkModules: "hudi-spark-datasource/hudi-spark3.5.x"
-
- - scalaProfile: "scala-2.13"
- sparkProfile: "spark3.5"
- sparkModules: "hudi-spark-datasource/hudi-spark3.5.x"
-
- steps:
- - uses: actions/checkout@v3
- - name: Set up JDK 8
- uses: actions/setup-java@v3
- with:
- java-version: '8'
Review Comment:
More specifically arrow which is what lance depends on only supports java 11 or higher https://arrow.apache.org/docs/developers/java/building.html#system-setup
##########
.github/workflows/bot.yml:
##########
@@ -882,41 +609,6 @@ jobs:
mvn clean install -T 2 -Pintegration-tests -D"$SCALA_PROFILE" -D"$FLINK_PROFILE" -pl hudi-flink-datasource/hudi-flink -am -Davro.version="$FLINK_AVRO_VERSION" -Dparquet.version="$FLINK_PARQUET_VERSION" -DskipTests=true $MVN_ARGS
mvn verify -Pintegration-tests -D"$SCALA_PROFILE" -D"$FLINK_PROFILE" -Davro.version="$FLINK_AVRO_VERSION" -Dparquet.version="$FLINK_PARQUET_VERSION" $FLINK_IT_FILTER1 -pl hudi-flink-datasource/hudi-flink $MVN_ARGS
- test-flink-2:
- runs-on: ubuntu-latest
- strategy:
- matrix:
- include:
- - flinkProfile: "flink1.20"
- flinkAvroVersion: "1.11.4"
- flinkParquetVersion: '1.13.1'
- steps:
- - uses: actions/checkout@v3
- - name: Set up JDK 8
- uses: actions/setup-java@v3
- with:
- java-version: '8'
- distribution: 'temurin'
- architecture: x64
- - name: Build Project
- env:
- SCALA_PROFILE: 'scala-2.12'
- FLINK_PROFILE: ${{ matrix.flinkProfile }}
- FLINK_AVRO_VERSION: ${{ matrix.flinkAvroVersion }}
- FLINK_PARQUET_VERSION: ${{ matrix.flinkParquetVersion }}
- run:
- mvn clean install -T 2 -D"$SCALA_PROFILE" -D"$FLINK_PROFILE" -pl hudi-examples/hudi-examples-flink -am -Davro.version="$FLINK_AVRO_VERSION" -Dparquet.version="$FLINK_PARQUET_VERSION" -DskipTests=true $MVN_ARGS
- - name: Integration Test 2
- env:
- SCALA_PROFILE: 'scala-2.12'
- FLINK_PROFILE: ${{ matrix.flinkProfile }}
- FLINK_AVRO_VERSION: ${{ matrix.flinkAvroVersion }}
- FLINK_PARQUET_VERSION: ${{ matrix.flinkParquetVersion }}
- if: ${{ endsWith(env.FLINK_PROFILE, '1.20') }}
- run: |
- mvn clean install -T 2 -Pintegration-tests -D"$SCALA_PROFILE" -D"$FLINK_PROFILE" -pl hudi-flink-datasource/hudi-flink -am -Davro.version="$FLINK_AVRO_VERSION" -Dparquet.version="$FLINK_PARQUET_VERSION" -DskipTests=true $MVN_ARGS
- mvn verify -Pintegration-tests -D"$SCALA_PROFILE" -D"$FLINK_PROFILE" -Davro.version="$FLINK_AVRO_VERSION" -Dparquet.version="$FLINK_PARQUET_VERSION" $FLINK_IT_FILTER2 -pl hudi-flink-datasource/hudi-flink $MVN_ARGS
Review Comment:
yes will do so
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add HoodieBaseLanceFileWriter and implementation for SparkFileWriter … [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #14131:
URL: https://github.com/apache/hudi/pull/14131#discussion_r2470891231
##########
hudi-hadoop-common/src/main/java/org/apache/hudi/io/lance/HoodieBaseLanceWriter.java:
##########
@@ -0,0 +1,196 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.io.lance;
+
+import com.lancedb.lance.file.LanceFileWriter;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.memory.RootAllocator;
+import org.apache.arrow.vector.VectorSchemaRoot;
+import org.apache.arrow.vector.types.pojo.Schema;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.storage.HoodieStorage;
+import org.apache.hudi.storage.StoragePath;
+
+import javax.annotation.concurrent.NotThreadSafe;
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * Base class for Hudi Lance file writers supporting different record types.
+ *
+ * This class handles common Lance file operations including:
+ * - LanceFileWriter lifecycle management
+ * - BufferAllocator management
+ * - Record buffering and batch flushing
+ * - File size checks
+ *
+ * Subclasses must implement type-specific conversion to Arrow format.
+ *
+ * @param <R> The record type (e.g., GenericRecord, InternalRow)
+ */
+@NotThreadSafe
+public abstract class HoodieBaseLanceWriter<R> implements Closeable {
+ protected static final int DEFAULT_BATCH_SIZE = 1000;
+ protected final HoodieStorage storage;
+ protected final StoragePath path;
+ protected final BufferAllocator allocator;
+ protected final List<R> bufferedRecords;
+ protected final int batchSize;
+ protected final long maxFileSize;
+ protected long writtenRecordCount = 0;
+ protected VectorSchemaRoot root;
+
+ private LanceFileWriter writer;
+
+ /**
+ * Constructor for base Lance writer.
+ *
+ * @param storage HoodieStorage instance
+ * @param path Path where Lance file will be written
+ * @param batchSize Number of records to buffer before flushing to Lance
+ * @param maxFileSize Maximum file size in bytes before rolling over to new file
+ */
+ protected HoodieBaseLanceWriter(HoodieStorage storage, StoragePath path, int batchSize, long maxFileSize) {
+ this.storage = storage;
+ this.path = path;
+ this.allocator = new RootAllocator(Long.MAX_VALUE);
Review Comment:
yes https://arrow.apache.org/docs/java/vector.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add HoodieBaseLanceFileWriter and implementation for SparkFileWriter … [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14131:
URL: https://github.com/apache/hudi/pull/14131#issuecomment-3453147272
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f1b8ff586652da6f82dd996fd0a6edcdb0f72ab4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9265",
"triggerID" : "f1b8ff586652da6f82dd996fd0a6edcdb0f72ab4",
"triggerType" : "PUSH"
}, {
"hash" : "7d356d05bf73a7263f448f4fad46ac0332ab8522",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9295",
"triggerID" : "7d356d05bf73a7263f448f4fad46ac0332ab8522",
"triggerType" : "PUSH"
}, {
"hash" : "41a8b4a7127307fa0263d701f5fc567c80857ea3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9360",
"triggerID" : "41a8b4a7127307fa0263d701f5fc567c80857ea3",
"triggerType" : "PUSH"
}, {
"hash" : "adc1713ee5af363f98d0a0ccfb2324b5c249a4fb",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9362",
"triggerID" : "adc1713ee5af363f98d0a0ccfb2324b5c249a4fb",
"triggerType" : "PUSH"
}, {
"hash" : "0bbead9b3a4efb779b472a343731969ea0f4d7f2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0bbead9b3a4efb779b472a343731969ea0f4d7f2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* adc1713ee5af363f98d0a0ccfb2324b5c249a4fb Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9362)
* 0bbead9b3a4efb779b472a343731969ea0f4d7f2 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add HoodieBaseLanceFileWriter and implementation for SparkFileWriter … [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14131:
URL: https://github.com/apache/hudi/pull/14131#issuecomment-3456921798
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f1b8ff586652da6f82dd996fd0a6edcdb0f72ab4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9265",
"triggerID" : "f1b8ff586652da6f82dd996fd0a6edcdb0f72ab4",
"triggerType" : "PUSH"
}, {
"hash" : "7d356d05bf73a7263f448f4fad46ac0332ab8522",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9295",
"triggerID" : "7d356d05bf73a7263f448f4fad46ac0332ab8522",
"triggerType" : "PUSH"
}, {
"hash" : "41a8b4a7127307fa0263d701f5fc567c80857ea3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9360",
"triggerID" : "41a8b4a7127307fa0263d701f5fc567c80857ea3",
"triggerType" : "PUSH"
}, {
"hash" : "adc1713ee5af363f98d0a0ccfb2324b5c249a4fb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9362",
"triggerID" : "adc1713ee5af363f98d0a0ccfb2324b5c249a4fb",
"triggerType" : "PUSH"
}, {
"hash" : "0bbead9b3a4efb779b472a343731969ea0f4d7f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9364",
"triggerID" : "0bbead9b3a4efb779b472a343731969ea0f4d7f2",
"triggerType" : "PUSH"
}, {
"hash" : "5c1b891ad88030b8e286602b449eee96301dcb84",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9366",
"triggerID" : "5c1b891ad88030b8e286602b449eee96301dcb84",
"triggerType" : "PUSH"
}, {
"hash" : "9943cbbadb0ce515bb16ebbb7b39919711e8a362",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9369",
"triggerID" : "9943cbbadb0ce515bb16ebbb7b39919711e8a362",
"triggerType" : "PUSH"
}, {
"hash" : "ef0c4ca43b81d7b0c27a4e61ad93dea0c7df91a3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ef0c4ca43b81d7b0c27a4e61ad93dea0c7df91a3",
"triggerType" : "PUSH"
}, {
"hash" : "bc7d984512e3bdf695f35442a76581e815086e91",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9388",
"triggerID" : "bc7d984512e3bdf695f35442a76581e815086e91",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9943cbbadb0ce515bb16ebbb7b39919711e8a362 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9369)
* ef0c4ca43b81d7b0c27a4e61ad93dea0c7df91a3 UNKNOWN
* bc7d984512e3bdf695f35442a76581e815086e91 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9388)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add HoodieBaseLanceFileWriter and implementation for SparkFileWriter … [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14131:
URL: https://github.com/apache/hudi/pull/14131#issuecomment-3452600269
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f1b8ff586652da6f82dd996fd0a6edcdb0f72ab4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9265",
"triggerID" : "f1b8ff586652da6f82dd996fd0a6edcdb0f72ab4",
"triggerType" : "PUSH"
}, {
"hash" : "7d356d05bf73a7263f448f4fad46ac0332ab8522",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9295",
"triggerID" : "7d356d05bf73a7263f448f4fad46ac0332ab8522",
"triggerType" : "PUSH"
}, {
"hash" : "41a8b4a7127307fa0263d701f5fc567c80857ea3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9360",
"triggerID" : "41a8b4a7127307fa0263d701f5fc567c80857ea3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 41a8b4a7127307fa0263d701f5fc567c80857ea3 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9360)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add HoodieBaseLanceFileWriter and implementation for SparkFileWriter … [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14131:
URL: https://github.com/apache/hudi/pull/14131#issuecomment-3437242839
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f1b8ff586652da6f82dd996fd0a6edcdb0f72ab4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9265",
"triggerID" : "f1b8ff586652da6f82dd996fd0a6edcdb0f72ab4",
"triggerType" : "PUSH"
}, {
"hash" : "7d356d05bf73a7263f448f4fad46ac0332ab8522",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9295",
"triggerID" : "7d356d05bf73a7263f448f4fad46ac0332ab8522",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f1b8ff586652da6f82dd996fd0a6edcdb0f72ab4 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9265)
* 7d356d05bf73a7263f448f4fad46ac0332ab8522 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9295)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add HoodieBaseLanceFileWriter and implementation for SparkFileWriter … [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on PR #14131:
URL: https://github.com/apache/hudi/pull/14131#issuecomment-3432755931
cc @the-other-tim-brown @vinothchandar @nsivabalan @yihua
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add HoodieBaseLanceFileWriter and implementation for SparkFileWriter … [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14131:
URL: https://github.com/apache/hudi/pull/14131#issuecomment-3432709207
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f1b8ff586652da6f82dd996fd0a6edcdb0f72ab4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9265",
"triggerID" : "f1b8ff586652da6f82dd996fd0a6edcdb0f72ab4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f1b8ff586652da6f82dd996fd0a6edcdb0f72ab4 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9265)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add HoodieBaseLanceFileWriter and implementation for SparkFileWriter … [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14131:
URL: https://github.com/apache/hudi/pull/14131#issuecomment-3432701209
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f1b8ff586652da6f82dd996fd0a6edcdb0f72ab4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f1b8ff586652da6f82dd996fd0a6edcdb0f72ab4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f1b8ff586652da6f82dd996fd0a6edcdb0f72ab4 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add HoodieBaseLanceFileWriter and implementation for SparkFileWriter … [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #14131:
URL: https://github.com/apache/hudi/pull/14131#discussion_r2455261849
##########
hudi-hadoop-common/src/main/java/org/apache/hudi/io/lance/HoodieBaseLanceWriter.java:
##########
@@ -0,0 +1,196 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.io.lance;
+
+import com.lancedb.lance.file.LanceFileWriter;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.memory.RootAllocator;
+import org.apache.arrow.vector.VectorSchemaRoot;
+import org.apache.arrow.vector.types.pojo.Schema;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.storage.HoodieStorage;
+import org.apache.hudi.storage.StoragePath;
+
+import javax.annotation.concurrent.NotThreadSafe;
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * Base class for Hudi Lance file writers supporting different record types.
+ *
+ * This class handles common Lance file operations including:
+ * - LanceFileWriter lifecycle management
+ * - BufferAllocator management
+ * - Record buffering and batch flushing
+ * - File size checks
+ *
+ * Subclasses must implement type-specific conversion to Arrow format.
+ *
+ * @param <R> The record type (e.g., GenericRecord, InternalRow)
+ */
+@NotThreadSafe
+public abstract class HoodieBaseLanceWriter<R> implements Closeable {
+ protected static final int DEFAULT_BATCH_SIZE = 1000;
+ protected final HoodieStorage storage;
+ protected final StoragePath path;
+ protected final BufferAllocator allocator;
+ protected final List<R> bufferedRecords;
+ protected final int batchSize;
+ protected final long maxFileSize;
+ protected long writtenRecordCount = 0;
+ protected VectorSchemaRoot root;
+
+ private LanceFileWriter writer;
+
+ /**
+ * Constructor for base Lance writer.
+ *
+ * @param storage HoodieStorage instance
+ * @param path Path where Lance file will be written
+ * @param batchSize Number of records to buffer before flushing to Lance
+ * @param maxFileSize Maximum file size in bytes before rolling over to new file
+ */
+ protected HoodieBaseLanceWriter(HoodieStorage storage, StoragePath path, int batchSize, long maxFileSize) {
+ this.storage = storage;
+ this.path = path;
+ this.allocator = new RootAllocator(Long.MAX_VALUE);
+ this.bufferedRecords = new ArrayList<>(batchSize);
+ this.batchSize = batchSize;
+ this.maxFileSize = maxFileSize;
+ }
+
+ /**
+ * Populate the VectorSchemaRoot with buffered records.
+ * Subclasses must implement type-specific conversion logic.
+ * The VectorSchemaRoot field is reused across batches and managed by this base class.
+ *
+ * @param records List of records to convert
+ */
+ protected abstract void populateVectorSchemaRoot(List<R> records);
+
+ /**
+ * Get the Arrow schema for this writer.
+ * Subclasses must provide the Arrow schema corresponding to their record type.
+ *
+ * @return Arrow schema
+ */
+ protected abstract Schema getArrowSchema();
+
+ /**
+ * Write a single record. Records are buffered and flushed in batches.
+ *
+ * @param record Record to write
+ * @throws IOException if write fails
+ */
+ public void write(R record) throws IOException {
+ bufferedRecords.add(record);
+ writtenRecordCount++;
+
+ if (bufferedRecords.size() >= batchSize) {
+ flushBatch();
+ }
+ }
+
+ /**
+ * Check if writer can accept more records based on file size.
+ * Uses filesystem-based size checking (similar to ORC/HFile approach).
+ *
+ * @return true if writer can accept more records, false if file size limit reached
+ */
+ public boolean canWrite() {
+ //TODO will need to implement proper way to compute this
Review Comment:
Similarly here, let's make sure this is tracked
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/HoodieSparkLanceWriter.java:
##########
@@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.io.storage;
+
+import com.lancedb.lance.spark.arrow.LanceArrowWriter;
+import org.apache.arrow.vector.types.pojo.Schema;
+import org.apache.hudi.common.engine.TaskContextSupplier;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.io.lance.HoodieBaseLanceWriter;
+import org.apache.hudi.storage.HoodieStorage;
+import org.apache.hudi.storage.StoragePath;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.util.LanceArrowUtils;
+import org.apache.spark.unsafe.types.UTF8String;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.function.Function;
+
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.COMMIT_SEQNO_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.COMMIT_TIME_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.FILENAME_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.PARTITION_PATH_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.RECORD_KEY_METADATA_FIELD;
+
+/**
+ * Spark Lance file writer implementing {@link HoodieSparkFileWriter}.
+ *
+ * This writer integrates with Hudi's storage I/O layer and supports:
+ * - Hudi metadata field population
+ * - Record key tracking (for bloom filters - TODO)
Review Comment:
Can you make a GH subtask with Writer Improvements and we can track this line item there? Then let's link the task here.
##########
hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/io/storage/TestHoodieSparkLanceWriter.java:
##########
@@ -0,0 +1,501 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.io.storage;
+
+import com.lancedb.lance.file.LanceFileReader;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.memory.RootAllocator;
+import org.apache.arrow.vector.BigIntVector;
+import org.apache.arrow.vector.BitVector;
+import org.apache.arrow.vector.Float4Vector;
+import org.apache.arrow.vector.Float8Vector;
+import org.apache.arrow.vector.IntVector;
+import org.apache.arrow.vector.VarBinaryVector;
+import org.apache.arrow.vector.VarCharVector;
+import org.apache.arrow.vector.VectorSchemaRoot;
+import org.apache.arrow.vector.ipc.ArrowReader;
+import org.apache.hudi.client.SparkTaskContextSupplier;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.testutils.HoodieTestUtils;
+import org.apache.hudi.storage.HoodieStorage;
+import org.apache.hudi.storage.StoragePath;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.catalyst.expressions.GenericInternalRow;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.unsafe.types.UTF8String;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.COMMIT_SEQNO_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.COMMIT_TIME_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.FILENAME_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.PARTITION_PATH_METADATA_FIELD;
+import static org.apache.hudi.common.model.HoodieRecord.HoodieMetadataField.RECORD_KEY_METADATA_FIELD;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * Tests for {@link HoodieSparkLanceWriter}.
+ */
+public class TestHoodieSparkLanceWriter {
+
+ @TempDir
+ File tempDir;
+
+ private HoodieStorage storage;
+ private SparkTaskContextSupplier taskContextSupplier;
+ private String instantTime;
+
+ @BeforeEach
+ public void setUp() throws IOException {
+ storage = HoodieTestUtils.getStorage(tempDir.getAbsolutePath());
+ taskContextSupplier = new SparkTaskContextSupplier();
+ instantTime = "20251201120000000";
+ }
+
+ @AfterEach
+ public void tearDown() throws Exception {
+ if (storage != null) {
+ storage.close();
+ }
+ }
+
+ @Test
+ public void testWriteRowWithMetadataPopulation() throws Exception {
+ // Schema WITH meta fields (writer expects this when populateMetaFields=true)
+ StructType schema = createSchemaWithMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_with_metadata.lance");
+ HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, true);
+
+ // Create row with PLACEHOLDER meta fields + user data
+ InternalRow row = createRowWithMetaFields(1, "Alice", 30L);
+ HoodieKey key = new HoodieKey("key1", "partition1");
+
+ writer.writeRowWithMetadata(key, row);
+ writer.close();
+
+ // Verify using LanceFileReader
+ assertTrue(storage.exists(path), "Lance file should exist");
+
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator)) {
+
+ assertEquals(1, reader.numRows(), "Should have 1 record");
+
+ // Schema should have 5 meta fields + 3 user fields = 8 fields
+ assertEquals(8, reader.schema().getFields().size(), "Should have 8 fields");
+
+ // Read and verify data
+ ArrowReader arrowReader = reader.readAll(null, null, Integer.MAX_VALUE);
+ assertTrue(arrowReader.loadNextBatch(), "Should load batch");
+ VectorSchemaRoot root = arrowReader.getVectorSchemaRoot();
+
+ // Verify meta fields were populated
+ VarCharVector commitTimeVector = (VarCharVector) root.getVector(COMMIT_TIME_METADATA_FIELD.getFieldName());
+ assertNotNull(commitTimeVector);
+ assertEquals(instantTime, new String(commitTimeVector.get(0)), "Commit time should match");
+
+ VarCharVector recordKeyVector = (VarCharVector) root.getVector(RECORD_KEY_METADATA_FIELD.getFieldName());
+ assertEquals("key1", new String(recordKeyVector.get(0)), "Record key should match");
+
+ VarCharVector partitionPathVector = (VarCharVector) root.getVector(PARTITION_PATH_METADATA_FIELD.getFieldName());
+ assertEquals("partition1", new String(partitionPathVector.get(0)), "Partition path should match");
+
+ VarCharVector fileNameVector = (VarCharVector) root.getVector(FILENAME_METADATA_FIELD.getFieldName());
+ assertEquals(path.getName(), new String(fileNameVector.get(0)), "File name should match");
+
+ VarCharVector seqNoVector = (VarCharVector) root.getVector(COMMIT_SEQNO_METADATA_FIELD.getFieldName());
+ String seqNo = new String(seqNoVector.get(0));
+ assertTrue(seqNo.startsWith(instantTime), "Sequence number should start with instant time");
+
+ // Verify user data fields
+ IntVector idVector = (IntVector) root.getVector("id");
+ assertEquals(1, idVector.get(0), "ID should match");
+
+ VarCharVector nameVector = (VarCharVector) root.getVector("name");
+ assertEquals("Alice", new String(nameVector.get(0)), "Name should match");
+
+ BigIntVector ageVector = (BigIntVector) root.getVector("age");
+ assertEquals(30L, ageVector.get(0), "Age should match");
+
+ arrowReader.close();
+ }
+ }
+
+ @Test
+ public void testWriteRowWithoutMetadataPopulation() throws Exception {
+ // Schema WITHOUT meta fields
+ StructType schema = createSchemaWithoutMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_without_metadata.lance");
+ HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false);
+
+ // Create row with just user data (no meta fields)
+ InternalRow row = createRow(1, "Bob", 25L);
+ HoodieKey key = new HoodieKey("key2", "partition2");
+
+ writer.writeRowWithMetadata(key, row);
+ writer.close();
+
+ // Verify using LanceFileReader
+ assertTrue(storage.exists(path));
+
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator)) {
+
+ assertEquals(1, reader.numRows());
+
+ // Schema should have ONLY 3 user fields (no meta fields)
+ assertEquals(3, reader.schema().getFields().size(), "Should have only user fields");
+
+ // Read and verify data
+ ArrowReader arrowReader = reader.readAll(null, null, Integer.MAX_VALUE);
+ assertTrue(arrowReader.loadNextBatch());
+ VectorSchemaRoot root = arrowReader.getVectorSchemaRoot();
+
+ // Verify NO meta fields exist
+ assertFalse(hasField(root, COMMIT_TIME_METADATA_FIELD.getFieldName()), "Should not have commit time field");
+ assertFalse(hasField(root, RECORD_KEY_METADATA_FIELD.getFieldName()), "Should not have record key field");
+
+ // Verify user data
+ IntVector idVector = (IntVector) root.getVector("id");
+ assertEquals(1, idVector.get(0));
+
+ VarCharVector nameVector = (VarCharVector) root.getVector("name");
+ assertEquals("Bob", new String(nameVector.get(0)));
+
+ arrowReader.close();
+ }
+ }
+
+ @Test
+ public void testWriteRowSimple() throws Exception {
+ StructType schema = createSchemaWithoutMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_simple_write.lance");
+ HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false);
+
+ InternalRow row = createRow(1, "Charlie", 35L);
+ writer.writeRow("key3", row);
+ writer.close();
+
+ // Verify file exists and has correct record count
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator)) {
+ assertEquals(1, reader.numRows());
+ }
+ }
+
+ @Test
+ public void testSequenceIdGeneration() throws Exception {
+ StructType schema = createSchemaWithMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_sequence_id.lance");
+ HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, true);
+
+ // Write multiple records
+ for (int i = 0; i < 3; i++) {
+ InternalRow row = createRowWithMetaFields(i, "User" + i, 20L + i);
+ HoodieKey key = new HoodieKey("key" + i, "partition1");
+ writer.writeRowWithMetadata(key, row);
+ }
+ writer.close();
+
+ // Verify sequence IDs are unique and properly formatted
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator)) {
+
+ assertEquals(3, reader.numRows());
+
+ ArrowReader arrowReader = reader.readAll(null, null, Integer.MAX_VALUE);
+ arrowReader.loadNextBatch();
+ VectorSchemaRoot root = arrowReader.getVectorSchemaRoot();
+
+ VarCharVector seqNoVector = (VarCharVector) root.getVector(COMMIT_SEQNO_METADATA_FIELD.getFieldName());
+
+ List<String> seqNos = new ArrayList<>();
+ for (int i = 0; i < 3; i++) {
+ String seqNo = new String(seqNoVector.get(i));
+ assertTrue(seqNo.startsWith(instantTime + "_"), "Sequence ID should start with instantTime");
+ seqNos.add(seqNo);
+ }
+
+ // All sequence IDs should be unique
+ assertEquals(3, seqNos.stream().distinct().count(), "All sequence IDs should be unique");
+
+ arrowReader.close();
+ }
+ }
+
+ @Test
+ public void testBatchFlushing() throws Exception {
+ StructType schema = createSchemaWithoutMetaFields();
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_batch_flush.lance");
+ HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false);
+
+ // Write more than DEFAULT_BATCH_SIZE (1000) records
+ int recordCount = 2500;
+ for (int i = 0; i < recordCount; i++) {
+ InternalRow row = createRow(i, "User" + i, 20L + i);
+ writer.writeRow("key" + i, row);
+ }
+ writer.close();
+
+ // Verify all records were written
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator)) {
+ assertEquals(recordCount, reader.numRows(), "All records should be written");
+ }
+ }
+
+ @Test
+ public void testPrimitiveTypes() throws Exception {
+ StructType schema = new StructType()
+ .add("int_field", DataTypes.IntegerType, false)
+ .add("long_field", DataTypes.LongType, false)
+ .add("float_field", DataTypes.FloatType, false)
+ .add("double_field", DataTypes.DoubleType, false)
+ .add("bool_field", DataTypes.BooleanType, false)
+ .add("string_field", DataTypes.StringType, false)
+ .add("binary_field", DataTypes.BinaryType, false);
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_primitives.lance");
+ HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false);
+
+ GenericInternalRow row = new GenericInternalRow(new Object[]{
+ 42, // int
+ 123456789L, // long
+ 3.14f, // float
+ 2.71828, // double
+ true, // boolean
+ UTF8String.fromString("test"), // string
+ new byte[]{1, 2, 3, 4} // binary
+ });
+
+ writer.writeRow("key1", row);
+ writer.close();
+
+ // Verify all types were written correctly
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator)) {
+
+ assertEquals(1, reader.numRows());
+ assertEquals(7, reader.schema().getFields().size());
+
+ ArrowReader arrowReader = reader.readAll(null, null, Integer.MAX_VALUE);
+ arrowReader.loadNextBatch();
+ VectorSchemaRoot root = arrowReader.getVectorSchemaRoot();
+
+ assertEquals(42, ((IntVector) root.getVector("int_field")).get(0));
+ assertEquals(123456789L, ((BigIntVector) root.getVector("long_field")).get(0));
+ assertEquals(3.14f, ((Float4Vector) root.getVector("float_field")).get(0), 0.001);
+ assertEquals(2.71828, ((Float8Vector) root.getVector("double_field")).get(0), 0.00001);
+ assertEquals(1, ((BitVector) root.getVector("bool_field")).get(0));
+ assertEquals("test", new String(((VarCharVector) root.getVector("string_field")).get(0)));
+
+ byte[] binary = ((VarBinaryVector) root.getVector("binary_field")).get(0);
+ assertEquals(4, binary.length);
+
+ arrowReader.close();
+ }
+ }
+
+ @Test
+ public void testNullValues() throws Exception {
+ StructType schema = new StructType()
+ .add("id", DataTypes.IntegerType, false)
+ .add("name", DataTypes.StringType, true)
+ .add("age", DataTypes.LongType, true);
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_nulls.lance");
+ HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false);
+
+ // Write rows with null values
+ writer.writeRow("key1", createRow(1, "Alice", 30L));
+ writer.writeRow("key2", createRow(2, null, 25L)); // null name
+ writer.writeRow("key3", createRow(3, "Charlie", null)); // null age
+ writer.close();
+
+ // Verify nulls are preserved
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator)) {
+
+ assertEquals(3, reader.numRows());
+
+ ArrowReader arrowReader = reader.readAll(null, null, Integer.MAX_VALUE);
+ arrowReader.loadNextBatch();
+ VectorSchemaRoot root = arrowReader.getVectorSchemaRoot();
+
+ VarCharVector nameVector = (VarCharVector) root.getVector("name");
+ assertFalse(nameVector.isNull(0), "First name should not be null");
+ assertTrue(nameVector.isNull(1), "Second name should be null");
+ assertFalse(nameVector.isNull(2), "Third name should not be null");
+
+ BigIntVector ageVector = (BigIntVector) root.getVector("age");
+ assertFalse(ageVector.isNull(0), "First age should not be null");
+ assertFalse(ageVector.isNull(1), "Second age should not be null");
+ assertTrue(ageVector.isNull(2), "Third age should be null");
+
+ arrowReader.close();
+ }
+ }
+
+ @Test
+ public void testWriteEmptyDataset() throws Exception {
+ StructType schema = new StructType()
+ .add("id", DataTypes.IntegerType, false);
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_empty.lance");
+ HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false);
+ writer.close(); // Close without writing any rows
+
+ // Lance doesn't create a file if no data is written
+ assertFalse(storage.exists(path), "Lance file should not exist when no data is written");
+ }
+
+ @Test
+ public void testWriteStructType() throws Exception {
+ // Create schema with nested struct
+ StructType addressSchema = new StructType()
+ .add("street", DataTypes.StringType, true)
+ .add("city", DataTypes.StringType, true)
+ .add("zipcode", DataTypes.IntegerType, true);
+
+ StructType schema = new StructType()
+ .add("id", DataTypes.IntegerType, false)
+ .add("name", DataTypes.StringType, true)
+ .add("address", addressSchema, true);
+
+ // Create test data with nested struct
+ GenericInternalRow address1 = new GenericInternalRow(new Object[]{
+ UTF8String.fromString("123 Main St"),
+ UTF8String.fromString("New York"),
+ 10001
+ });
+
+ GenericInternalRow address2 = new GenericInternalRow(new Object[]{
+ UTF8String.fromString("456 Oak Ave"),
+ UTF8String.fromString("Los Angeles"),
+ 90001
+ });
+
+ List<InternalRow> rows = new ArrayList<>();
+ rows.add(new GenericInternalRow(new Object[]{1, UTF8String.fromString("Alice"), address1}));
+ rows.add(new GenericInternalRow(new Object[]{2, UTF8String.fromString("Bob"), address2}));
+
+ StoragePath path = new StoragePath(tempDir.getAbsolutePath() + "/test_struct.lance");
+ HoodieSparkLanceWriter writer = new HoodieSparkLanceWriter(
+ path, schema, instantTime, taskContextSupplier, storage, false);
+
+ for (InternalRow row : rows) {
+ writer.writeRow("key" + rows.indexOf(row), row);
+ }
+ writer.close();
+
+ assertTrue(storage.exists(path), "Lance file with struct type should exist");
+ try (BufferAllocator allocator = new RootAllocator();
+ LanceFileReader reader = LanceFileReader.open(path.toString(), allocator)) {
+ assertEquals(rows.size(), reader.numRows(), "Row count should match");
+ assertEquals(3, reader.schema().getFields().size(), "Should have 3 top-level fields (id, name, address)");
+ }
+ }
+
+ // Helper methods
+
+ private StructType createSchemaWithMetaFields() {
Review Comment:
You can also use `HoodieAvroUtils#addMetadataFields` to do this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add HoodieBaseLanceFileWriter and implementation for SparkFileWriter … [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #14131:
URL: https://github.com/apache/hudi/pull/14131#discussion_r2470700701
##########
.github/workflows/bot.yml:
##########
@@ -958,88 +650,6 @@ jobs:
HUDI_VERSION=$(mvn help:evaluate -Dexpression=project.version -q -DforceStdout)
./packaging/bundle-validation/run_docker_java17.sh
- validate-bundles:
Review Comment:
Keep this bundle validation and remove Java 8 parts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add HoodieBaseLanceFileWriter and implementation for SparkFileWriter … [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14131:
URL: https://github.com/apache/hudi/pull/14131#issuecomment-3453556654
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f1b8ff586652da6f82dd996fd0a6edcdb0f72ab4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9265",
"triggerID" : "f1b8ff586652da6f82dd996fd0a6edcdb0f72ab4",
"triggerType" : "PUSH"
}, {
"hash" : "7d356d05bf73a7263f448f4fad46ac0332ab8522",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9295",
"triggerID" : "7d356d05bf73a7263f448f4fad46ac0332ab8522",
"triggerType" : "PUSH"
}, {
"hash" : "41a8b4a7127307fa0263d701f5fc567c80857ea3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9360",
"triggerID" : "41a8b4a7127307fa0263d701f5fc567c80857ea3",
"triggerType" : "PUSH"
}, {
"hash" : "adc1713ee5af363f98d0a0ccfb2324b5c249a4fb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9362",
"triggerID" : "adc1713ee5af363f98d0a0ccfb2324b5c249a4fb",
"triggerType" : "PUSH"
}, {
"hash" : "0bbead9b3a4efb779b472a343731969ea0f4d7f2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9364",
"triggerID" : "0bbead9b3a4efb779b472a343731969ea0f4d7f2",
"triggerType" : "PUSH"
}, {
"hash" : "5c1b891ad88030b8e286602b449eee96301dcb84",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9366",
"triggerID" : "5c1b891ad88030b8e286602b449eee96301dcb84",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0bbead9b3a4efb779b472a343731969ea0f4d7f2 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9364)
* 5c1b891ad88030b8e286602b449eee96301dcb84 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9366)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add HoodieBaseLanceFileWriter and implementation for SparkFileWriter … [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #14131:
URL: https://github.com/apache/hudi/pull/14131#discussion_r2466495709
##########
pom.xml:
##########
@@ -897,7 +900,29 @@
<version>${orc.spark.version}</version>
<scope>compile</scope>
</dependency>
-
+ <!-- Lance Core SDK -->
+ <dependency>
+ <groupId>com.lancedb</groupId>
+ <artifactId>lance-core</artifactId>
+ <version>${lance.version}</version>
+ </dependency>
+ <!-- Lance Spark for Arrow conversion -->
+ <dependency>
+ <groupId>com.lancedb</groupId>
+ <artifactId>lance-spark-base_${scala.binary.version}</artifactId>
Review Comment:
I think its more around the fact that our ci is currently running for spark 3.3 and spark 3.4, so we either need to restrict those profiles or have a way of passing the correct lance spark version for each spark profile we have in root pom. https://lancedb.github.io/lance/integrations/spark/install/#scala
Ill try using the specific spark version artfiact (lean jar) mentioned above
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add HoodieBaseLanceFileWriter and implementation for SparkFileWriter … [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #14131:
URL: https://github.com/apache/hudi/pull/14131#discussion_r2470895051
##########
.github/workflows/bot.yml:
##########
@@ -958,88 +650,6 @@ jobs:
HUDI_VERSION=$(mvn help:evaluate -Dexpression=project.version -q -DforceStdout)
./packaging/bundle-validation/run_docker_java17.sh
- validate-bundles:
- runs-on: ubuntu-latest
- strategy:
- matrix:
- include:
- - scalaProfile: 'scala-2.13'
- flinkProfile: 'flink1.20'
- flinkAvroVersion: '1.11.4'
- flinkParquetVersion: '1.13.1'
- sparkProfile: 'spark3.5'
- sparkRuntime: 'spark3.5.1'
- - scalaProfile: 'scala-2.13'
- flinkProfile: 'flink1.19'
- flinkAvroVersion: '1.11.4'
- flinkParquetVersion: '1.13.1'
- sparkProfile: 'spark3.5'
- sparkRuntime: 'spark3.5.1'
- - scalaProfile: 'scala-2.12'
- flinkProfile: 'flink1.18'
- flinkAvroVersion: '1.11.4'
- flinkParquetVersion: '1.13.1'
- sparkProfile: 'spark3.4'
- sparkRuntime: 'spark3.4.3'
- - scalaProfile: 'scala-2.12'
- flinkProfile: 'flink1.17'
- flinkAvroVersion: '1.11.4'
- flinkParquetVersion: '1.12.3'
- sparkProfile: 'spark3.3'
- sparkRuntime: 'spark3.3.4'
-
- steps:
- - uses: actions/checkout@v3
- - name: Set up JDK 8
- uses: actions/setup-java@v3
- with:
- java-version: '8'
Review Comment:
Actually i think there is bundle validation for java 11 and 17 so dont think we need this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add HoodieBaseLanceFileWriter and implementation for SparkFileWriter … [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #14131:
URL: https://github.com/apache/hudi/pull/14131#discussion_r2466495709
##########
pom.xml:
##########
@@ -897,7 +900,29 @@
<version>${orc.spark.version}</version>
<scope>compile</scope>
</dependency>
-
+ <!-- Lance Core SDK -->
+ <dependency>
+ <groupId>com.lancedb</groupId>
+ <artifactId>lance-core</artifactId>
+ <version>${lance.version}</version>
+ </dependency>
+ <!-- Lance Spark for Arrow conversion -->
+ <dependency>
+ <groupId>com.lancedb</groupId>
+ <artifactId>lance-spark-base_${scala.binary.version}</artifactId>
Review Comment:
I think its more around the fact that our ci is currently running for spark 3.3 and spark 3.4, so we either need to restrict those profiles or have a way of passing the correct lance spark version for each spark profile we have in root pom.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add HoodieBaseLanceFileWriter and implementation for SparkFileWriter … [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #14131:
URL: https://github.com/apache/hudi/pull/14131#discussion_r2466495709
##########
pom.xml:
##########
@@ -897,7 +900,29 @@
<version>${orc.spark.version}</version>
<scope>compile</scope>
</dependency>
-
+ <!-- Lance Core SDK -->
+ <dependency>
+ <groupId>com.lancedb</groupId>
+ <artifactId>lance-core</artifactId>
+ <version>${lance.version}</version>
+ </dependency>
+ <!-- Lance Spark for Arrow conversion -->
+ <dependency>
+ <groupId>com.lancedb</groupId>
+ <artifactId>lance-spark-base_${scala.binary.version}</artifactId>
Review Comment:
I think its more around the fact that our ci is currently running for spark 3.3 and spark 3.4, so we either need to restrict those profiles or have a way of passing the correct lance spark version for each spark profile we have in root pom. https://lancedb.github.io/lance/integrations/spark/install/#scala
Ill try using the specific spark version artfiact (lean jar) mentioned above, as the bundle currently does not seem to be containing any special shading logic, or including anything https://github.com/lancedb/lance-spark/blob/main/lance-spark-bundle-3.5_2.12/pom.xml
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org