You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/09/25 12:33:23 UTC
[GitHub] [hudi] qianchutao opened a new pull request #3715: [HUDI-2487] fix JsonKafkaSource cannot filter empty messages from kafka
qianchutao opened a new pull request #3715:
URL: https://github.com/apache/hudi/pull/3715
## what is the purpose of the pull request
**This pull request solved the following problem:**
**(1)** When using DeltaStreamer in upsert mode, consume JSON data from Kafka to hudi and update hive tables, if some messages in Kafka are null, the task will throw an exception
## Brief change log
- *Filter empty messages from Kafka in org.apache.hudi.utilities.sources.JsonKafkaSource#fetchNewData*
## Verify this pull request
This pull request is already covered by existing tests, such as *org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer#testDistributedTestDataSource*.
[https://issues.apache.org/jira/browse/HUDI-2487](url)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3715: [HUDI-2487] fix JsonKafkaSource cannot filter empty messages from kafka
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3715:
URL: https://github.com/apache/hudi/pull/3715#issuecomment-927115196
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ea7a6f04b3c60354589c24bdb70eb45883a56e14",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2383",
"triggerID" : "ea7a6f04b3c60354589c24bdb70eb45883a56e14",
"triggerType" : "PUSH"
}, {
"hash" : "ea7a6f04b3c60354589c24bdb70eb45883a56e14",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2385",
"triggerID" : "927148664",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* ea7a6f04b3c60354589c24bdb70eb45883a56e14 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2383) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2385)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] qianchutao commented on pull request #3715: [HUDI-2487] fix JsonKafkaSource cannot filter empty messages from kafka
Posted by GitBox <gi...@apache.org>.
qianchutao commented on pull request #3715:
URL: https://github.com/apache/hudi/pull/3715#issuecomment-927199444
@yihua
An error occurred during azure deployment in the test case. Please help to check the cause. The following is the error message:
---------------------------------------------------------------------------------------------------------------------------------------------
Formatting using clusterid: testClusterID
3399 [main] WARN org.apache.hadoop.metrics2.impl.MetricsConfig - Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
5918 [DataStreamer for file /user/vsts/t3.props] WARN org.apache.hadoop.hdfs.DFSClient - Caught exception
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.Thread.join(Thread.java:1252)
at java.lang.Thread.join(Thread.java:1326)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.closeResponder(DFSOutputStream.java:609)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.endBlock(DFSOutputStream.java:370)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:546)
6398 [main] WARN org.apache.hadoop.hdfs.server.datanode.DirectoryScanner - DirectoryScanner: shutdown has been called
---------------------------------------------------------------------------------------------------------------------------------------------
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3715: [HUDI-2487] fix JsonKafkaSource cannot filter empty messages from kafka
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3715:
URL: https://github.com/apache/hudi/pull/3715#issuecomment-927115196
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ea7a6f04b3c60354589c24bdb70eb45883a56e14",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2383",
"triggerID" : "ea7a6f04b3c60354589c24bdb70eb45883a56e14",
"triggerType" : "PUSH"
}, {
"hash" : "ea7a6f04b3c60354589c24bdb70eb45883a56e14",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2385",
"triggerID" : "927148664",
"triggerType" : "MANUAL"
}, {
"hash" : "ea7a6f04b3c60354589c24bdb70eb45883a56e14",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2395",
"triggerID" : "927215399",
"triggerType" : "MANUAL"
}, {
"hash" : "deca323a98fa197fdb70e7451c958200308703ce",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2435",
"triggerID" : "deca323a98fa197fdb70e7451c958200308703ce",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* deca323a98fa197fdb70e7451c958200308703ce Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2435)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on pull request #3715: [HUDI-2487] fix JsonKafkaSource cannot filter empty messages from kafka
Posted by GitBox <gi...@apache.org>.
yanghua commented on pull request #3715:
URL: https://github.com/apache/hudi/pull/3715#issuecomment-927656921
> > @qianchutao : can you add a unit test please.
>
> Thanks for your reply, this method is covered by unit tests in the project, the unit test code is:
> org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer#testDistributedTestDataSource.
@qianchutao , @nsivabalan means that you can add a test to see if the null record can be filtered or not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua merged pull request #3715: [HUDI-2487] Fix JsonKafkaSource cannot filter empty messages from kafka
Posted by GitBox <gi...@apache.org>.
yanghua merged pull request #3715:
URL: https://github.com/apache/hudi/pull/3715
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3715: [HUDI-2487] fix JsonKafkaSource cannot filter empty messages from kafka
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3715:
URL: https://github.com/apache/hudi/pull/3715#issuecomment-927115196
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ea7a6f04b3c60354589c24bdb70eb45883a56e14",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2383",
"triggerID" : "ea7a6f04b3c60354589c24bdb70eb45883a56e14",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ea7a6f04b3c60354589c24bdb70eb45883a56e14 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2383)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan edited a comment on pull request #3715: [HUDI-2487] fix JsonKafkaSource cannot filter empty messages from kafka
Posted by GitBox <gi...@apache.org>.
nsivabalan edited a comment on pull request #3715:
URL: https://github.com/apache/hudi/pull/3715#issuecomment-928632193
LGTM. will let @yanghua land this in.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3715: [HUDI-2487] fix JsonKafkaSource cannot filter empty messages from kafka
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3715:
URL: https://github.com/apache/hudi/pull/3715#issuecomment-927115196
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ea7a6f04b3c60354589c24bdb70eb45883a56e14",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2383",
"triggerID" : "ea7a6f04b3c60354589c24bdb70eb45883a56e14",
"triggerType" : "PUSH"
}, {
"hash" : "ea7a6f04b3c60354589c24bdb70eb45883a56e14",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2385",
"triggerID" : "927148664",
"triggerType" : "MANUAL"
}, {
"hash" : "ea7a6f04b3c60354589c24bdb70eb45883a56e14",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2395",
"triggerID" : "927215399",
"triggerType" : "MANUAL"
}, {
"hash" : "deca323a98fa197fdb70e7451c958200308703ce",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "deca323a98fa197fdb70e7451c958200308703ce",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ea7a6f04b3c60354589c24bdb70eb45883a56e14 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2383) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2385) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2395)
* deca323a98fa197fdb70e7451c958200308703ce UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3715: [HUDI-2487] fix JsonKafkaSource cannot filter empty messages from kafka
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3715:
URL: https://github.com/apache/hudi/pull/3715#issuecomment-927115196
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ea7a6f04b3c60354589c24bdb70eb45883a56e14",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ea7a6f04b3c60354589c24bdb70eb45883a56e14",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ea7a6f04b3c60354589c24bdb70eb45883a56e14 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #3715: [HUDI-2487] fix JsonKafkaSource cannot filter empty messages from kafka
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3715:
URL: https://github.com/apache/hudi/pull/3715#issuecomment-928632193
LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3715: [HUDI-2487] fix JsonKafkaSource cannot filter empty messages from kafka
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3715:
URL: https://github.com/apache/hudi/pull/3715#issuecomment-927115196
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ea7a6f04b3c60354589c24bdb70eb45883a56e14",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2383",
"triggerID" : "ea7a6f04b3c60354589c24bdb70eb45883a56e14",
"triggerType" : "PUSH"
}, {
"hash" : "ea7a6f04b3c60354589c24bdb70eb45883a56e14",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2385",
"triggerID" : "927148664",
"triggerType" : "MANUAL"
}, {
"hash" : "ea7a6f04b3c60354589c24bdb70eb45883a56e14",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2395",
"triggerID" : "927215399",
"triggerType" : "MANUAL"
}, {
"hash" : "deca323a98fa197fdb70e7451c958200308703ce",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2435",
"triggerID" : "deca323a98fa197fdb70e7451c958200308703ce",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ea7a6f04b3c60354589c24bdb70eb45883a56e14 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2383) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2385) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2395)
* deca323a98fa197fdb70e7451c958200308703ce Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2435)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] qianchutao commented on pull request #3715: [HUDI-2487] fix JsonKafkaSource cannot filter empty messages from kafka
Posted by GitBox <gi...@apache.org>.
qianchutao commented on pull request #3715:
URL: https://github.com/apache/hudi/pull/3715#issuecomment-927215399
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] qianchutao commented on pull request #3715: [HUDI-2487] fix JsonKafkaSource cannot filter empty messages from kafka
Posted by GitBox <gi...@apache.org>.
qianchutao commented on pull request #3715:
URL: https://github.com/apache/hudi/pull/3715#issuecomment-927122916
In upsert mode, deltaStreamer is used to pull messages from Kafka and synchronize them to hive tables. If null messages are not filtered, the entire production task throws an exception in code: <org.apache.hudi.avro.HoodieAvroUtils> line:484
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on a change in pull request #3715: [HUDI-2487] fix JsonKafkaSource cannot filter empty messages from kafka
Posted by GitBox <gi...@apache.org>.
yanghua commented on a change in pull request #3715:
URL: https://github.com/apache/hudi/pull/3715#discussion_r716480508
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/JsonKafkaSource.java
##########
@@ -69,7 +69,14 @@ public JsonKafkaSource(TypedProperties properties, JavaSparkContext sparkContext
private JavaRDD<String> toRDD(OffsetRange[] offsetRanges) {
return KafkaUtils.createRDD(sparkContext, offsetGen.getKafkaParams(), offsetRanges,
- LocationStrategies.PreferConsistent()).map(x -> (String) x.value());
+ LocationStrategies.PreferConsistent()).filter(x -> {
+ String msgValue = (String) x.value();
+ //Filter null messages from Kafka to prevent Exceptions
+ if (msgValue == null) {
Review comment:
`return msgValue != null;`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] qianchutao commented on pull request #3715: [HUDI-2487] fix JsonKafkaSource cannot filter empty messages from kafka
Posted by GitBox <gi...@apache.org>.
qianchutao commented on pull request #3715:
URL: https://github.com/apache/hudi/pull/3715#issuecomment-928605496
@yanghua Mr. Yang, I have made corresponding modifications and added the unit test part. Please help me review it again ，thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua merged pull request #3715: [HUDI-2487] Fix JsonKafkaSource cannot filter empty messages from kafka
Posted by GitBox <gi...@apache.org>.
yanghua merged pull request #3715:
URL: https://github.com/apache/hudi/pull/3715
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #3715: [HUDI-2487] fix JsonKafkaSource cannot filter empty messages from kafka
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3715:
URL: https://github.com/apache/hudi/pull/3715#issuecomment-927175557
@qianchutao : can you add a unit test please.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] qianchutao commented on pull request #3715: [HUDI-2487] fix JsonKafkaSource cannot filter empty messages from kafka
Posted by GitBox <gi...@apache.org>.
qianchutao commented on pull request #3715:
URL: https://github.com/apache/hudi/pull/3715#issuecomment-927198979
> @qianchutao : can you add a unit test please.
Thanks for your reply, this method is covered by unit tests in the project, the unit test code is:
org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer#testDistributedTestDataSource.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan edited a comment on pull request #3715: [HUDI-2487] fix JsonKafkaSource cannot filter empty messages from kafka
Posted by GitBox <gi...@apache.org>.
nsivabalan edited a comment on pull request #3715:
URL: https://github.com/apache/hudi/pull/3715#issuecomment-928632193
LGTM. will let @yanghua land this in.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] qianchutao commented on pull request #3715: [HUDI-2487] fix JsonKafkaSource cannot filter empty messages from kafka
Posted by GitBox <gi...@apache.org>.
qianchutao commented on pull request #3715:
URL: https://github.com/apache/hudi/pull/3715#issuecomment-927751655
> > > @qianchutao : can you add a unit test please.
> >
> >
> > Thanks for your reply, this method is covered by unit tests in the project, the unit test code is:
> > org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer#testDistributedTestDataSource.
>
> @qianchutao , @nsivabalan means that you can add a test to see if the null record can be filtered or not.
ok,thanks.I will add unit tests to verify, give me some time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3715: [HUDI-2487] fix JsonKafkaSource cannot filter empty messages from kafka
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3715:
URL: https://github.com/apache/hudi/pull/3715#issuecomment-927115196
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ea7a6f04b3c60354589c24bdb70eb45883a56e14",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2383",
"triggerID" : "ea7a6f04b3c60354589c24bdb70eb45883a56e14",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ea7a6f04b3c60354589c24bdb70eb45883a56e14 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2383)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #3715: [HUDI-2487] fix JsonKafkaSource cannot filter empty messages from kafka
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3715:
URL: https://github.com/apache/hudi/pull/3715#issuecomment-927175448
@yihua : Can you take up reviewing this patch please.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] qianchutao commented on pull request #3715: [HUDI-2487] fix JsonKafkaSource cannot filter empty messages from kafka
Posted by GitBox <gi...@apache.org>.
qianchutao commented on pull request #3715:
URL: https://github.com/apache/hudi/pull/3715#issuecomment-927115517
[https://issues.apache.org/jira/browse/HUDI-2487](url)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] qianchutao commented on a change in pull request #3715: [HUDI-2487] fix JsonKafkaSource cannot filter empty messages from kafka
Posted by GitBox <gi...@apache.org>.
qianchutao commented on a change in pull request #3715:
URL: https://github.com/apache/hudi/pull/3715#discussion_r716579425
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/JsonKafkaSource.java
##########
@@ -69,7 +69,14 @@ public JsonKafkaSource(TypedProperties properties, JavaSparkContext sparkContext
private JavaRDD<String> toRDD(OffsetRange[] offsetRanges) {
return KafkaUtils.createRDD(sparkContext, offsetGen.getKafkaParams(), offsetRanges,
- LocationStrategies.PreferConsistent()).map(x -> (String) x.value());
+ LocationStrategies.PreferConsistent()).filter(x -> {
+ String msgValue = (String) x.value();
+ //Filter null messages from Kafka to prevent Exceptions
+ if (msgValue == null) {
Review comment:
> `return msgValue != null;`?
yes,this looks more concise, I will modify the code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3715: [HUDI-2487] fix JsonKafkaSource cannot filter empty messages from kafka
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3715:
URL: https://github.com/apache/hudi/pull/3715#issuecomment-927115196
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ea7a6f04b3c60354589c24bdb70eb45883a56e14",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2383",
"triggerID" : "ea7a6f04b3c60354589c24bdb70eb45883a56e14",
"triggerType" : "PUSH"
}, {
"hash" : "ea7a6f04b3c60354589c24bdb70eb45883a56e14",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2385",
"triggerID" : "927148664",
"triggerType" : "MANUAL"
}, {
"hash" : "ea7a6f04b3c60354589c24bdb70eb45883a56e14",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2395",
"triggerID" : "927215399",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* ea7a6f04b3c60354589c24bdb70eb45883a56e14 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2383) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2385) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2395)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] qianchutao commented on pull request #3715: [HUDI-2487] fix JsonKafkaSource cannot filter empty messages from kafka
Posted by GitBox <gi...@apache.org>.
qianchutao commented on pull request #3715:
URL: https://github.com/apache/hudi/pull/3715#issuecomment-927148664
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #3715: [HUDI-2487] fix JsonKafkaSource cannot filter empty messages from kafka
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3715:
URL: https://github.com/apache/hudi/pull/3715#issuecomment-928632193
LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3715: [HUDI-2487] fix JsonKafkaSource cannot filter empty messages from kafka
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3715:
URL: https://github.com/apache/hudi/pull/3715#issuecomment-927115196
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ea7a6f04b3c60354589c24bdb70eb45883a56e14",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2383",
"triggerID" : "ea7a6f04b3c60354589c24bdb70eb45883a56e14",
"triggerType" : "PUSH"
}, {
"hash" : "ea7a6f04b3c60354589c24bdb70eb45883a56e14",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2385",
"triggerID" : "927148664",
"triggerType" : "MANUAL"
}, {
"hash" : "ea7a6f04b3c60354589c24bdb70eb45883a56e14",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2395",
"triggerID" : "927215399",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* ea7a6f04b3c60354589c24bdb70eb45883a56e14 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2383) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2385) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2395)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] qianchutao commented on pull request #3715: [HUDI-2487] fix JsonKafkaSource cannot filter empty messages from kafka
Posted by GitBox <gi...@apache.org>.
qianchutao commented on pull request #3715:
URL: https://github.com/apache/hudi/pull/3715#issuecomment-928605496
@yanghua Mr. Yang, I have made corresponding modifications and added the unit test part. Please help me review it again ，thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3715: [HUDI-2487] fix JsonKafkaSource cannot filter empty messages from kafka
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3715:
URL: https://github.com/apache/hudi/pull/3715#issuecomment-927115196
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ea7a6f04b3c60354589c24bdb70eb45883a56e14",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2383",
"triggerID" : "ea7a6f04b3c60354589c24bdb70eb45883a56e14",
"triggerType" : "PUSH"
}, {
"hash" : "ea7a6f04b3c60354589c24bdb70eb45883a56e14",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2385",
"triggerID" : "927148664",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* ea7a6f04b3c60354589c24bdb70eb45883a56e14 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2383) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2385)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org