You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "vinishjail97 (via GitHub)" <gi...@apache.org> on 2023/08/01 14:27:18 UTC
[GitHub] [hudi] vinishjail97 opened a new pull request, #9335: [HUDI-6627] Fix NPE when spark client writer schema is null
vinishjail97 opened a new pull request, #9335:
URL: https://github.com/apache/hudi/pull/9335
### Change Logs
When source returns an empty option, the writer schema is null. This causes an NPE with the table schema validation in spark write client causing the below exception. Skipping the validation when writer schema is null.
```
org.apache.hudi.exception.HoodieInsertException: Failed insert schema compability check.
at org.apache.hudi.table.HoodieTable.validateInsertSchema(HoodieTable.java:851)
at org.apache.hudi.client.SparkRDDWriteClient.insert(SparkRDDWriteClient.java:185)
at org.apache.hudi.utilities.deltastreamer.DeltaSync.writeToSink(DeltaSync.java:690)
at org.apache.hudi.utilities.deltastreamer.DeltaSync.syncOnce(DeltaSync.java:396)
at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer$DeltaSyncService.ingestOnce(HoodieDeltaStreamer.java:876)
at org.apache.hudi.common.util.Option.ifPresent(Option.java:97)
at com.onehouse.hudi.OnehouseDeltaStreamer$MultiTableSyncService.lambda$null$1(OnehouseDeltaStreamer.java:319)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.hudi.exception.HoodieException: Failed to read schema/check compatibility for base path s3a://onehouse-customer-bucket-2451e78f/data-lake/chandra_data_lake_default/xml_flatten_struct_test
at org.apache.hudi.table.HoodieTable.validateSchema(HoodieTable.java:830)
at org.apache.hudi.table.HoodieTable.validateInsertSchema(HoodieTable.java:849)
... 10 more
Caused by: java.lang.NullPointerException
at com.fasterxml.jackson.core.JsonFactory.createParser(JsonFactory.java:1158)
at org.apache.avro.Schema$Parser.parse(Schema.java:1418)
at org.apache.hudi.avro.HoodieAvroUtils.createHoodieWriteSchema(HoodieAvroUtils.java:302)
at org.apache.hudi.table.HoodieTable.validateSchema(HoodieTable.java:826)
... 11 more
```
### Impact
Deltastreamer sources returning an empty rdd with no schema won't cause NPE in the deltastreamer spark job.
### Risk level (write none, low medium or high below)
Low.
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change_
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Change Logs and Impact were stated clearly
- [x] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9335: [HUDI-6627] Fix NPE when spark client writer schema is null
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9335:
URL: https://github.com/apache/hudi/pull/9335#issuecomment-1660469305
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b1bc2ab4ef91a572d21ded2468b9cf6711b55a85",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b1bc2ab4ef91a572d21ded2468b9cf6711b55a85",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b1bc2ab4ef91a572d21ded2468b9cf6711b55a85 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9335: [HUDI-6627] Fix NPE when spark client writer schema is null
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9335:
URL: https://github.com/apache/hudi/pull/9335#issuecomment-1661982421
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b1bc2ab4ef91a572d21ded2468b9cf6711b55a85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18963",
"triggerID" : "b1bc2ab4ef91a572d21ded2468b9cf6711b55a85",
"triggerType" : "PUSH"
}, {
"hash" : "1e5cc41028b53a99efacd59d15f610aa96d5b5e2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18966",
"triggerID" : "1e5cc41028b53a99efacd59d15f610aa96d5b5e2",
"triggerType" : "PUSH"
}, {
"hash" : "f5f782eb73f43a49310e547209415b695dafa108",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18998",
"triggerID" : "f5f782eb73f43a49310e547209415b695dafa108",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1e5cc41028b53a99efacd59d15f610aa96d5b5e2 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18966)
* f5f782eb73f43a49310e547209415b695dafa108 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18998)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9335: [HUDI-6627] Fix NPE when spark client writer schema is null
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9335:
URL: https://github.com/apache/hudi/pull/9335#issuecomment-1660872183
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b1bc2ab4ef91a572d21ded2468b9cf6711b55a85",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18963",
"triggerID" : "b1bc2ab4ef91a572d21ded2468b9cf6711b55a85",
"triggerType" : "PUSH"
}, {
"hash" : "1e5cc41028b53a99efacd59d15f610aa96d5b5e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1e5cc41028b53a99efacd59d15f610aa96d5b5e2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b1bc2ab4ef91a572d21ded2468b9cf6711b55a85 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18963)
* 1e5cc41028b53a99efacd59d15f610aa96d5b5e2 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9335: [HUDI-6627] Fix NPE when spark client writer schema is null
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9335:
URL: https://github.com/apache/hudi/pull/9335#issuecomment-1661242500
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b1bc2ab4ef91a572d21ded2468b9cf6711b55a85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18963",
"triggerID" : "b1bc2ab4ef91a572d21ded2468b9cf6711b55a85",
"triggerType" : "PUSH"
}, {
"hash" : "1e5cc41028b53a99efacd59d15f610aa96d5b5e2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18966",
"triggerID" : "1e5cc41028b53a99efacd59d15f610aa96d5b5e2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1e5cc41028b53a99efacd59d15f610aa96d5b5e2 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18966)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a diff in pull request #9335: [HUDI-6627] Fix NPE when spark client writer schema is null
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #9335:
URL: https://github.com/apache/hudi/pull/9335#discussion_r1280764110
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/testutils/HoodieClientTestBase.java:
##########
@@ -172,6 +172,10 @@ public HoodieWriteConfig.Builder getConfigBuilder(String schemaStr, IndexType in
.withEnableBackupForRemoteFileSystemView(false) // Fail test if problem connecting to timeline-server
.withRemoteServerPort(timelineServicePort)
.withStorageType(FileSystemViewStorageType.EMBEDDED_KV_STORE).build());
+ if (schemaStr != null) {
Review Comment:
Use StringUtils.nonEmpty?
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/functional/TestHoodieClientOnCopyOnWriteStorage.java:
##########
@@ -404,6 +406,38 @@ public void testPreCommitValidationWithMultipleInflights() throws Exception {
assertTrue(testTable.commitExists(instant2));
}
+ @Test
+ public void testInertsWithEmptyCommitsHavingWriterSchemaAsNull() throws Exception {
+ // Set autoCommit false
+ HoodieWriteConfig.Builder cfgBuilder = getConfigBuilder().withAutoCommit(false);
+ addConfigsForPopulateMetaFields(cfgBuilder, true);
+ SparkRDDWriteClient client = getHoodieWriteClient(cfgBuilder.build());
Review Comment:
Get the client in try-with-resource block, or close the client in finally block.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9335: [HUDI-6627] Fix NPE when spark client writer schema is null
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9335:
URL: https://github.com/apache/hudi/pull/9335#issuecomment-1661969373
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b1bc2ab4ef91a572d21ded2468b9cf6711b55a85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18963",
"triggerID" : "b1bc2ab4ef91a572d21ded2468b9cf6711b55a85",
"triggerType" : "PUSH"
}, {
"hash" : "1e5cc41028b53a99efacd59d15f610aa96d5b5e2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18966",
"triggerID" : "1e5cc41028b53a99efacd59d15f610aa96d5b5e2",
"triggerType" : "PUSH"
}, {
"hash" : "f5f782eb73f43a49310e547209415b695dafa108",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f5f782eb73f43a49310e547209415b695dafa108",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1e5cc41028b53a99efacd59d15f610aa96d5b5e2 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18966)
* f5f782eb73f43a49310e547209415b695dafa108 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9335: [HUDI-6627] Fix NPE when spark client writer schema is null
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9335:
URL: https://github.com/apache/hudi/pull/9335#issuecomment-1662137343
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b1bc2ab4ef91a572d21ded2468b9cf6711b55a85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18963",
"triggerID" : "b1bc2ab4ef91a572d21ded2468b9cf6711b55a85",
"triggerType" : "PUSH"
}, {
"hash" : "1e5cc41028b53a99efacd59d15f610aa96d5b5e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18966",
"triggerID" : "1e5cc41028b53a99efacd59d15f610aa96d5b5e2",
"triggerType" : "PUSH"
}, {
"hash" : "f5f782eb73f43a49310e547209415b695dafa108",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18998",
"triggerID" : "f5f782eb73f43a49310e547209415b695dafa108",
"triggerType" : "PUSH"
}, {
"hash" : "b1091bdeaf25dcd95f567a8e50c2c6d4dc80fb79",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b1091bdeaf25dcd95f567a8e50c2c6d4dc80fb79",
"triggerType" : "PUSH"
}, {
"hash" : "c37dc656ad139cebecd3d56a3d4bb7eca9477498",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19000",
"triggerID" : "c37dc656ad139cebecd3d56a3d4bb7eca9477498",
"triggerType" : "PUSH"
}, {
"hash" : "6386813364fd15848d9e63f4b77ce31c63e8a815",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19001",
"triggerID" : "6386813364fd15848d9e63f4b77ce31c63e8a815",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b1091bdeaf25dcd95f567a8e50c2c6d4dc80fb79 UNKNOWN
* c37dc656ad139cebecd3d56a3d4bb7eca9477498 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19000)
* 6386813364fd15848d9e63f4b77ce31c63e8a815 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19001)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 merged pull request #9335: [HUDI-6627] Fix NPE when spark client writer schema is null
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 merged PR #9335:
URL: https://github.com/apache/hudi/pull/9335
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9335: [HUDI-6627] Fix NPE when spark client writer schema is null
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9335:
URL: https://github.com/apache/hudi/pull/9335#issuecomment-1660884163
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b1bc2ab4ef91a572d21ded2468b9cf6711b55a85",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18963",
"triggerID" : "b1bc2ab4ef91a572d21ded2468b9cf6711b55a85",
"triggerType" : "PUSH"
}, {
"hash" : "1e5cc41028b53a99efacd59d15f610aa96d5b5e2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18966",
"triggerID" : "1e5cc41028b53a99efacd59d15f610aa96d5b5e2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b1bc2ab4ef91a572d21ded2468b9cf6711b55a85 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18963)
* 1e5cc41028b53a99efacd59d15f610aa96d5b5e2 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18966)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9335: [HUDI-6627] Fix NPE when spark client writer schema is null
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9335:
URL: https://github.com/apache/hudi/pull/9335#issuecomment-1662061693
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b1bc2ab4ef91a572d21ded2468b9cf6711b55a85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18963",
"triggerID" : "b1bc2ab4ef91a572d21ded2468b9cf6711b55a85",
"triggerType" : "PUSH"
}, {
"hash" : "1e5cc41028b53a99efacd59d15f610aa96d5b5e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18966",
"triggerID" : "1e5cc41028b53a99efacd59d15f610aa96d5b5e2",
"triggerType" : "PUSH"
}, {
"hash" : "f5f782eb73f43a49310e547209415b695dafa108",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18998",
"triggerID" : "f5f782eb73f43a49310e547209415b695dafa108",
"triggerType" : "PUSH"
}, {
"hash" : "b1091bdeaf25dcd95f567a8e50c2c6d4dc80fb79",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b1091bdeaf25dcd95f567a8e50c2c6d4dc80fb79",
"triggerType" : "PUSH"
}, {
"hash" : "c37dc656ad139cebecd3d56a3d4bb7eca9477498",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c37dc656ad139cebecd3d56a3d4bb7eca9477498",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f5f782eb73f43a49310e547209415b695dafa108 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18998)
* b1091bdeaf25dcd95f567a8e50c2c6d4dc80fb79 UNKNOWN
* c37dc656ad139cebecd3d56a3d4bb7eca9477498 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinishjail97 commented on a diff in pull request #9335: [HUDI-6627] Fix NPE when spark client writer schema is null
Posted by "vinishjail97 (via GitHub)" <gi...@apache.org>.
vinishjail97 commented on code in PR #9335:
URL: https://github.com/apache/hudi/pull/9335#discussion_r1280993432
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/functional/TestHoodieClientOnCopyOnWriteStorage.java:
##########
@@ -404,6 +406,38 @@ public void testPreCommitValidationWithMultipleInflights() throws Exception {
assertTrue(testTable.commitExists(instant2));
}
+ @Test
+ public void testInertsWithEmptyCommitsHavingWriterSchemaAsNull() throws Exception {
+ // Set autoCommit false
+ HoodieWriteConfig.Builder cfgBuilder = getConfigBuilder().withAutoCommit(false);
+ addConfigsForPopulateMetaFields(cfgBuilder, true);
+ SparkRDDWriteClient client = getHoodieWriteClient(cfgBuilder.build());
Review Comment:
> Get the client in try-with-resource block
This makes the client a final variable and re-init was not possible. Added the close in a finally block.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9335: [HUDI-6627] Fix NPE when spark client writer schema is null
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9335:
URL: https://github.com/apache/hudi/pull/9335#issuecomment-1662984338
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b1bc2ab4ef91a572d21ded2468b9cf6711b55a85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18963",
"triggerID" : "b1bc2ab4ef91a572d21ded2468b9cf6711b55a85",
"triggerType" : "PUSH"
}, {
"hash" : "1e5cc41028b53a99efacd59d15f610aa96d5b5e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18966",
"triggerID" : "1e5cc41028b53a99efacd59d15f610aa96d5b5e2",
"triggerType" : "PUSH"
}, {
"hash" : "f5f782eb73f43a49310e547209415b695dafa108",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18998",
"triggerID" : "f5f782eb73f43a49310e547209415b695dafa108",
"triggerType" : "PUSH"
}, {
"hash" : "b1091bdeaf25dcd95f567a8e50c2c6d4dc80fb79",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b1091bdeaf25dcd95f567a8e50c2c6d4dc80fb79",
"triggerType" : "PUSH"
}, {
"hash" : "c37dc656ad139cebecd3d56a3d4bb7eca9477498",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19000",
"triggerID" : "c37dc656ad139cebecd3d56a3d4bb7eca9477498",
"triggerType" : "PUSH"
}, {
"hash" : "6386813364fd15848d9e63f4b77ce31c63e8a815",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19001",
"triggerID" : "6386813364fd15848d9e63f4b77ce31c63e8a815",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b1091bdeaf25dcd95f567a8e50c2c6d4dc80fb79 UNKNOWN
* 6386813364fd15848d9e63f4b77ce31c63e8a815 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19001)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9335: [HUDI-6627] Fix NPE when spark client writer schema is null
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9335:
URL: https://github.com/apache/hudi/pull/9335#issuecomment-1660555453
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b1bc2ab4ef91a572d21ded2468b9cf6711b55a85",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18963",
"triggerID" : "b1bc2ab4ef91a572d21ded2468b9cf6711b55a85",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b1bc2ab4ef91a572d21ded2468b9cf6711b55a85 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18963)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9335: [HUDI-6627] Fix NPE when spark client writer schema is null
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9335:
URL: https://github.com/apache/hudi/pull/9335#issuecomment-1662049721
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b1bc2ab4ef91a572d21ded2468b9cf6711b55a85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18963",
"triggerID" : "b1bc2ab4ef91a572d21ded2468b9cf6711b55a85",
"triggerType" : "PUSH"
}, {
"hash" : "1e5cc41028b53a99efacd59d15f610aa96d5b5e2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18966",
"triggerID" : "1e5cc41028b53a99efacd59d15f610aa96d5b5e2",
"triggerType" : "PUSH"
}, {
"hash" : "f5f782eb73f43a49310e547209415b695dafa108",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18998",
"triggerID" : "f5f782eb73f43a49310e547209415b695dafa108",
"triggerType" : "PUSH"
}, {
"hash" : "b1091bdeaf25dcd95f567a8e50c2c6d4dc80fb79",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b1091bdeaf25dcd95f567a8e50c2c6d4dc80fb79",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1e5cc41028b53a99efacd59d15f610aa96d5b5e2 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18966)
* f5f782eb73f43a49310e547209415b695dafa108 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18998)
* b1091bdeaf25dcd95f567a8e50c2c6d4dc80fb79 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9335: [HUDI-6627] Fix NPE when spark client writer schema is null
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9335:
URL: https://github.com/apache/hudi/pull/9335#issuecomment-1662124730
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b1bc2ab4ef91a572d21ded2468b9cf6711b55a85",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18963",
"triggerID" : "b1bc2ab4ef91a572d21ded2468b9cf6711b55a85",
"triggerType" : "PUSH"
}, {
"hash" : "1e5cc41028b53a99efacd59d15f610aa96d5b5e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18966",
"triggerID" : "1e5cc41028b53a99efacd59d15f610aa96d5b5e2",
"triggerType" : "PUSH"
}, {
"hash" : "f5f782eb73f43a49310e547209415b695dafa108",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18998",
"triggerID" : "f5f782eb73f43a49310e547209415b695dafa108",
"triggerType" : "PUSH"
}, {
"hash" : "b1091bdeaf25dcd95f567a8e50c2c6d4dc80fb79",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b1091bdeaf25dcd95f567a8e50c2c6d4dc80fb79",
"triggerType" : "PUSH"
}, {
"hash" : "c37dc656ad139cebecd3d56a3d4bb7eca9477498",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19000",
"triggerID" : "c37dc656ad139cebecd3d56a3d4bb7eca9477498",
"triggerType" : "PUSH"
}, {
"hash" : "6386813364fd15848d9e63f4b77ce31c63e8a815",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6386813364fd15848d9e63f4b77ce31c63e8a815",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f5f782eb73f43a49310e547209415b695dafa108 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18998)
* b1091bdeaf25dcd95f567a8e50c2c6d4dc80fb79 UNKNOWN
* c37dc656ad139cebecd3d56a3d4bb7eca9477498 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19000)
* 6386813364fd15848d9e63f4b77ce31c63e8a815 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org