You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2023/01/10 11:43:49 UTC
[GitHub] [hudi] trushev opened a new pull request, #7637: [MINOR] Fix ContinuousFileSource and ITTestDataStreamWrite for flink 1.16 support
trushev opened a new pull request, #7637:
URL: https://github.com/apache/hudi/pull/7637
As mentioned here https://github.com/apache/hudi/pull/7584#issuecomment-1370531069, `JsonRowDataDeserializationSchema` should be opened in flink 1.16. This PR do open the deserializer in all places of use
### Change Logs
1. Moved `JsonDeserializationFunction` from test utils to main utils. It is needed to do the fixes below
1. Synchronized quickstart `ContinuousFileSource` with main utils `ContinuousFileSource`
1. Fixed several tests freezing with NPE in `ITTestDataStreamWrite`
```
[Map -> row_data_to_hoodie_record (4/4)#3] WARN org.apache.flink.runtime.taskmanager.Task [] - Map -> row_data_to_hoodie_record (4/4)#3 (c24fd03f94105404d61b4e42c1d737cb_20ba6b65f97481d5570070de90e4e791_3_3)
switched from RUNNING to FAILED with failure cause: java.lang.NullPointerException
at org.apache.hudi.avro.HoodieAvroUtils.getFieldVal(HoodieAvroUtils.java:515)
at org.apache.hudi.avro.HoodieAvroUtils.getNestedFieldVal(HoodieAvroUtils.java:548)
at org.apache.hudi.avro.HoodieAvroUtils.getNestedFieldValAsString(HoodieAvroUtils.java:533)
at org.apache.hudi.keygen.KeyGenUtils.getRecordKey(KeyGenUtils.java:144)
```
### Impact
Fixed NPE in `ContinuousFileSource` and in `ITTestDataStreamWrite`
### Risk level (write none, low medium or high below)
Low
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change_
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7637: [HUDI-5538] Fix ContinuousFileSource and ITTestDataStreamWrite for flink 1.16 support
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7637:
URL: https://github.com/apache/hudi/pull/7637#issuecomment-1381019092
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a3eeff3a77246c3a8a53833ca1bae8ee504aba58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14223",
"triggerID" : "a3eeff3a77246c3a8a53833ca1bae8ee504aba58",
"triggerType" : "PUSH"
}, {
"hash" : "5059462b6b9927325d75fd6b36290032a181dca0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14237",
"triggerID" : "5059462b6b9927325d75fd6b36290032a181dca0",
"triggerType" : "PUSH"
}, {
"hash" : "86b5e390741c3ffedffc0e32d5e73fb4413ef877",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14266",
"triggerID" : "86b5e390741c3ffedffc0e32d5e73fb4413ef877",
"triggerType" : "PUSH"
}, {
"hash" : "a72eafc88db51cf48b38da47719450a1d57ef3b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14276",
"triggerID" : "a72eafc88db51cf48b38da47719450a1d57ef3b0",
"triggerType" : "PUSH"
}, {
"hash" : "5a3df6098e7b2867080d2cacde328380c977b144",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14283",
"triggerID" : "5a3df6098e7b2867080d2cacde328380c977b144",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5a3df6098e7b2867080d2cacde328380c977b144 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14283)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] trushev commented on a diff in pull request #7637: [MINOR] Fix ContinuousFileSource and ITTestDataStreamWrite for flink 1.16 support
Posted by GitBox <gi...@apache.org>.
trushev commented on code in PR #7637:
URL: https://github.com/apache/hudi/pull/7637#discussion_r1067713867
##########
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/sink/ITTestDataStreamWrite.java:
##########
@@ -554,4 +518,20 @@ public void testHoodiePipelineBuilderSinkWithSchemaSet() throws Exception {
execute(execEnv, false, "Api_Sink_Test");
TestData.checkWrittenDataCOW(tempFile, EXPECTED);
}
+
+ private JsonDeserializationFunction getDeserializationFunc(Configuration conf) {
+ // Read from file source
Review Comment:
- Created factory method
- Updated azure for flink 1.16
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7637: [MINOR] Fix ContinuousFileSource and ITTestDataStreamWrite for flink 1.16 support
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7637:
URL: https://github.com/apache/hudi/pull/7637#issuecomment-1378275241
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a3eeff3a77246c3a8a53833ca1bae8ee504aba58",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14223",
"triggerID" : "a3eeff3a77246c3a8a53833ca1bae8ee504aba58",
"triggerType" : "PUSH"
}, {
"hash" : "5059462b6b9927325d75fd6b36290032a181dca0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14237",
"triggerID" : "5059462b6b9927325d75fd6b36290032a181dca0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a3eeff3a77246c3a8a53833ca1bae8ee504aba58 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14223)
* 5059462b6b9927325d75fd6b36290032a181dca0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14237)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7637: [MINOR] Fix ContinuousFileSource and ITTestDataStreamWrite for flink 1.16 support
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7637:
URL: https://github.com/apache/hudi/pull/7637#issuecomment-1378613355
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a3eeff3a77246c3a8a53833ca1bae8ee504aba58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14223",
"triggerID" : "a3eeff3a77246c3a8a53833ca1bae8ee504aba58",
"triggerType" : "PUSH"
}, {
"hash" : "5059462b6b9927325d75fd6b36290032a181dca0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14237",
"triggerID" : "5059462b6b9927325d75fd6b36290032a181dca0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5059462b6b9927325d75fd6b36290032a181dca0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14237)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7637: [MINOR] Fix ContinuousFileSource and ITTestDataStreamWrite for flink 1.16 support
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7637:
URL: https://github.com/apache/hudi/pull/7637#issuecomment-1378271290
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a3eeff3a77246c3a8a53833ca1bae8ee504aba58",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14223",
"triggerID" : "a3eeff3a77246c3a8a53833ca1bae8ee504aba58",
"triggerType" : "PUSH"
}, {
"hash" : "5059462b6b9927325d75fd6b36290032a181dca0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5059462b6b9927325d75fd6b36290032a181dca0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a3eeff3a77246c3a8a53833ca1bae8ee504aba58 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14223)
* 5059462b6b9927325d75fd6b36290032a181dca0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7637: [MINOR] Fix ContinuousFileSource and ITTestDataStreamWrite for flink 1.16 support
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7637:
URL: https://github.com/apache/hudi/pull/7637#issuecomment-1379824885
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a3eeff3a77246c3a8a53833ca1bae8ee504aba58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14223",
"triggerID" : "a3eeff3a77246c3a8a53833ca1bae8ee504aba58",
"triggerType" : "PUSH"
}, {
"hash" : "5059462b6b9927325d75fd6b36290032a181dca0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14237",
"triggerID" : "5059462b6b9927325d75fd6b36290032a181dca0",
"triggerType" : "PUSH"
}, {
"hash" : "86b5e390741c3ffedffc0e32d5e73fb4413ef877",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "86b5e390741c3ffedffc0e32d5e73fb4413ef877",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5059462b6b9927325d75fd6b36290032a181dca0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14237)
* 86b5e390741c3ffedffc0e32d5e73fb4413ef877 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] trushev commented on a diff in pull request #7637: [MINOR] Fix ContinuousFileSource and ITTestDataStreamWrite for flink 1.16 support
Posted by GitBox <gi...@apache.org>.
trushev commented on code in PR #7637:
URL: https://github.com/apache/hudi/pull/7637#discussion_r1066658597
##########
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/sink/ITTestDataStreamWrite.java:
##########
@@ -554,4 +518,20 @@ public void testHoodiePipelineBuilderSinkWithSchemaSet() throws Exception {
execute(execEnv, false, "Api_Sink_Test");
TestData.checkWrittenDataCOW(tempFile, EXPECTED);
}
+
+ private JsonDeserializationFunction getDeserializationFunc(Configuration conf) {
+ // Read from file source
Review Comment:
> Why CI does not fail
As I understand it happens because azure still runs tests with flink 1.14
azure-pipelines.yml:
```
BUILD_PROFILES: '-Dscala-2.11 -Dspark2.4 -Dflink1.14'
```
I was thinking 1.14 version is purposely left as main for azure but it looks like a bug now :)
> we can have a static factory method
thanks, I'll fix it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #7637: [MINOR] Fix ContinuousFileSource and ITTestDataStreamWrite for flink 1.16 support
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #7637:
URL: https://github.com/apache/hudi/pull/7637#discussion_r1066634939
##########
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/sink/ITTestDataStreamWrite.java:
##########
@@ -554,4 +518,20 @@ public void testHoodiePipelineBuilderSinkWithSchemaSet() throws Exception {
execute(execEnv, false, "Api_Sink_Test");
TestData.checkWrittenDataCOW(tempFile, EXPECTED);
}
+
+ private JsonDeserializationFunction getDeserializationFunc(Configuration conf) {
+ // Read from file source
Review Comment:
Nice catch, curious why the CI does not fail for the PR previously, can we move this `#getDeserializationFunc` into `JsonDeserializationFunction` and we can have a static factory method like `JsonDeserializationFunction#getInstance`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #7637: [MINOR] Fix ContinuousFileSource and ITTestDataStreamWrite for flink 1.16 support
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #7637:
URL: https://github.com/apache/hudi/pull/7637#discussion_r1066663498
##########
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/sink/ITTestDataStreamWrite.java:
##########
@@ -554,4 +518,20 @@ public void testHoodiePipelineBuilderSinkWithSchemaSet() throws Exception {
execute(execEnv, false, "Api_Sink_Test");
TestData.checkWrittenDataCOW(tempFile, EXPECTED);
}
+
+ private JsonDeserializationFunction getDeserializationFunc(Configuration conf) {
+ // Read from file source
Review Comment:
Yeah, we actually should use the latest flink version for Azure tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7637: [MINOR] Fix ContinuousFileSource and ITTestDataStreamWrite for flink 1.16 support
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7637:
URL: https://github.com/apache/hudi/pull/7637#issuecomment-1377188313
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a3eeff3a77246c3a8a53833ca1bae8ee504aba58",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14223",
"triggerID" : "a3eeff3a77246c3a8a53833ca1bae8ee504aba58",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a3eeff3a77246c3a8a53833ca1bae8ee504aba58 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14223)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] trushev commented on a diff in pull request #7637: [HUDI-5538] Fix ContinuousFileSource and ITTestDataStreamWrite for flink 1.16 support
Posted by GitBox <gi...@apache.org>.
trushev commented on code in PR #7637:
URL: https://github.com/apache/hudi/pull/7637#discussion_r1067840571
##########
azure-pipelines.yml:
##########
@@ -80,7 +82,7 @@ parameters:
- '!hudi-flink-datasource/hudi-flink1.15.x'
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7637: [MINOR] Fix ContinuousFileSource and ITTestDataStreamWrite for flink 1.16 support
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7637:
URL: https://github.com/apache/hudi/pull/7637#issuecomment-1377664114
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a3eeff3a77246c3a8a53833ca1bae8ee504aba58",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14223",
"triggerID" : "a3eeff3a77246c3a8a53833ca1bae8ee504aba58",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a3eeff3a77246c3a8a53833ca1bae8ee504aba58 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14223)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7637: [MINOR] Fix ContinuousFileSource and ITTestDataStreamWrite for flink 1.16 support
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7637:
URL: https://github.com/apache/hudi/pull/7637#issuecomment-1379829086
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a3eeff3a77246c3a8a53833ca1bae8ee504aba58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14223",
"triggerID" : "a3eeff3a77246c3a8a53833ca1bae8ee504aba58",
"triggerType" : "PUSH"
}, {
"hash" : "5059462b6b9927325d75fd6b36290032a181dca0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14237",
"triggerID" : "5059462b6b9927325d75fd6b36290032a181dca0",
"triggerType" : "PUSH"
}, {
"hash" : "86b5e390741c3ffedffc0e32d5e73fb4413ef877",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14266",
"triggerID" : "86b5e390741c3ffedffc0e32d5e73fb4413ef877",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5059462b6b9927325d75fd6b36290032a181dca0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14237)
* 86b5e390741c3ffedffc0e32d5e73fb4413ef877 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14266)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7637: [HUDI-5538] Fix ContinuousFileSource and ITTestDataStreamWrite for flink 1.16 support
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7637:
URL: https://github.com/apache/hudi/pull/7637#issuecomment-1380071275
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a3eeff3a77246c3a8a53833ca1bae8ee504aba58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14223",
"triggerID" : "a3eeff3a77246c3a8a53833ca1bae8ee504aba58",
"triggerType" : "PUSH"
}, {
"hash" : "5059462b6b9927325d75fd6b36290032a181dca0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14237",
"triggerID" : "5059462b6b9927325d75fd6b36290032a181dca0",
"triggerType" : "PUSH"
}, {
"hash" : "86b5e390741c3ffedffc0e32d5e73fb4413ef877",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14266",
"triggerID" : "86b5e390741c3ffedffc0e32d5e73fb4413ef877",
"triggerType" : "PUSH"
}, {
"hash" : "a72eafc88db51cf48b38da47719450a1d57ef3b0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14276",
"triggerID" : "a72eafc88db51cf48b38da47719450a1d57ef3b0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 86b5e390741c3ffedffc0e32d5e73fb4413ef877 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14266)
* a72eafc88db51cf48b38da47719450a1d57ef3b0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14276)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7637: [HUDI-5538] Fix ContinuousFileSource and ITTestDataStreamWrite for flink 1.16 support
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7637:
URL: https://github.com/apache/hudi/pull/7637#issuecomment-1380554774
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a3eeff3a77246c3a8a53833ca1bae8ee504aba58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14223",
"triggerID" : "a3eeff3a77246c3a8a53833ca1bae8ee504aba58",
"triggerType" : "PUSH"
}, {
"hash" : "5059462b6b9927325d75fd6b36290032a181dca0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14237",
"triggerID" : "5059462b6b9927325d75fd6b36290032a181dca0",
"triggerType" : "PUSH"
}, {
"hash" : "86b5e390741c3ffedffc0e32d5e73fb4413ef877",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14266",
"triggerID" : "86b5e390741c3ffedffc0e32d5e73fb4413ef877",
"triggerType" : "PUSH"
}, {
"hash" : "a72eafc88db51cf48b38da47719450a1d57ef3b0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14276",
"triggerID" : "a72eafc88db51cf48b38da47719450a1d57ef3b0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a72eafc88db51cf48b38da47719450a1d57ef3b0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14276)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 merged pull request #7637: [HUDI-5538] Fix ContinuousFileSource and ITTestDataStreamWrite for flink 1.16 support
Posted by GitBox <gi...@apache.org>.
danny0405 merged PR #7637:
URL: https://github.com/apache/hudi/pull/7637
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7637: [HUDI-5538] Fix ContinuousFileSource and ITTestDataStreamWrite for flink 1.16 support
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7637:
URL: https://github.com/apache/hudi/pull/7637#issuecomment-1380713647
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a3eeff3a77246c3a8a53833ca1bae8ee504aba58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14223",
"triggerID" : "a3eeff3a77246c3a8a53833ca1bae8ee504aba58",
"triggerType" : "PUSH"
}, {
"hash" : "5059462b6b9927325d75fd6b36290032a181dca0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14237",
"triggerID" : "5059462b6b9927325d75fd6b36290032a181dca0",
"triggerType" : "PUSH"
}, {
"hash" : "86b5e390741c3ffedffc0e32d5e73fb4413ef877",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14266",
"triggerID" : "86b5e390741c3ffedffc0e32d5e73fb4413ef877",
"triggerType" : "PUSH"
}, {
"hash" : "a72eafc88db51cf48b38da47719450a1d57ef3b0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14276",
"triggerID" : "a72eafc88db51cf48b38da47719450a1d57ef3b0",
"triggerType" : "PUSH"
}, {
"hash" : "5a3df6098e7b2867080d2cacde328380c977b144",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14283",
"triggerID" : "5a3df6098e7b2867080d2cacde328380c977b144",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a72eafc88db51cf48b38da47719450a1d57ef3b0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14276)
* 5a3df6098e7b2867080d2cacde328380c977b144 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14283)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] trushev commented on a diff in pull request #7637: [MINOR] Fix ContinuousFileSource and ITTestDataStreamWrite for flink 1.16 support
Posted by GitBox <gi...@apache.org>.
trushev commented on code in PR #7637:
URL: https://github.com/apache/hudi/pull/7637#discussion_r1067767154
##########
azure-pipelines.yml:
##########
@@ -80,7 +82,7 @@ parameters:
- '!hudi-flink-datasource/hudi-flink1.15.x'
Review Comment:
forgotten `hudi-flink1.16.x` here, needed to add after CI finishing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] trushev commented on pull request #7637: [HUDI-5538] Fix ContinuousFileSource and ITTestDataStreamWrite for flink 1.16 support
Posted by GitBox <gi...@apache.org>.
trushev commented on PR #7637:
URL: https://github.com/apache/hudi/pull/7637#issuecomment-1379983840
rebased on the latest master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7637: [MINOR] Fix ContinuousFileSource and ITTestDataStreamWrite for flink 1.16 support
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7637:
URL: https://github.com/apache/hudi/pull/7637#issuecomment-1377181302
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a3eeff3a77246c3a8a53833ca1bae8ee504aba58",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a3eeff3a77246c3a8a53833ca1bae8ee504aba58",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a3eeff3a77246c3a8a53833ca1bae8ee504aba58 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7637: [HUDI-5538] Fix ContinuousFileSource and ITTestDataStreamWrite for flink 1.16 support
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7637:
URL: https://github.com/apache/hudi/pull/7637#issuecomment-1380056914
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a3eeff3a77246c3a8a53833ca1bae8ee504aba58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14223",
"triggerID" : "a3eeff3a77246c3a8a53833ca1bae8ee504aba58",
"triggerType" : "PUSH"
}, {
"hash" : "5059462b6b9927325d75fd6b36290032a181dca0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14237",
"triggerID" : "5059462b6b9927325d75fd6b36290032a181dca0",
"triggerType" : "PUSH"
}, {
"hash" : "86b5e390741c3ffedffc0e32d5e73fb4413ef877",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14266",
"triggerID" : "86b5e390741c3ffedffc0e32d5e73fb4413ef877",
"triggerType" : "PUSH"
}, {
"hash" : "a72eafc88db51cf48b38da47719450a1d57ef3b0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a72eafc88db51cf48b38da47719450a1d57ef3b0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5059462b6b9927325d75fd6b36290032a181dca0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14237)
* 86b5e390741c3ffedffc0e32d5e73fb4413ef877 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14266)
* a72eafc88db51cf48b38da47719450a1d57ef3b0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] trushev commented on a diff in pull request #7637: [MINOR] Fix ContinuousFileSource and ITTestDataStreamWrite for flink 1.16 support
Posted by GitBox <gi...@apache.org>.
trushev commented on code in PR #7637:
URL: https://github.com/apache/hudi/pull/7637#discussion_r1066658597
##########
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/sink/ITTestDataStreamWrite.java:
##########
@@ -554,4 +518,20 @@ public void testHoodiePipelineBuilderSinkWithSchemaSet() throws Exception {
execute(execEnv, false, "Api_Sink_Test");
TestData.checkWrittenDataCOW(tempFile, EXPECTED);
}
+
+ private JsonDeserializationFunction getDeserializationFunc(Configuration conf) {
+ // Read from file source
Review Comment:
> Why CI does not fail
As I understand it happens because azure still runs test with flink 1.14
azure-pipelines.yml:
```
BUILD_PROFILES: '-Dscala-2.11 -Dspark2.4 -Dflink1.14'
```
I was thinking 1.14 version is purposely left as main for azure but it looks like a bug now :)
> we can have a static factory method
thanks, I'll fix it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7637: [HUDI-5538] Fix ContinuousFileSource and ITTestDataStreamWrite for flink 1.16 support
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7637:
URL: https://github.com/apache/hudi/pull/7637#issuecomment-1380590462
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a3eeff3a77246c3a8a53833ca1bae8ee504aba58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14223",
"triggerID" : "a3eeff3a77246c3a8a53833ca1bae8ee504aba58",
"triggerType" : "PUSH"
}, {
"hash" : "5059462b6b9927325d75fd6b36290032a181dca0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14237",
"triggerID" : "5059462b6b9927325d75fd6b36290032a181dca0",
"triggerType" : "PUSH"
}, {
"hash" : "86b5e390741c3ffedffc0e32d5e73fb4413ef877",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14266",
"triggerID" : "86b5e390741c3ffedffc0e32d5e73fb4413ef877",
"triggerType" : "PUSH"
}, {
"hash" : "a72eafc88db51cf48b38da47719450a1d57ef3b0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14276",
"triggerID" : "a72eafc88db51cf48b38da47719450a1d57ef3b0",
"triggerType" : "PUSH"
}, {
"hash" : "5a3df6098e7b2867080d2cacde328380c977b144",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5a3df6098e7b2867080d2cacde328380c977b144",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a72eafc88db51cf48b38da47719450a1d57ef3b0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14276)
* 5a3df6098e7b2867080d2cacde328380c977b144 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org