You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "vinishjail97 (via GitHub)" <gi...@apache.org> on 2024/03/15 19:43:07 UTC
[PR] [HUDI-7508] Avoid collecting records in HoodieStreamerUtils.createHoodieRecords and JsonKafkaSource mapPartitions [hudi]
vinishjail97 opened a new pull request, #10872:
URL: https://github.com/apache/hudi/pull/10872
### Change Logs
This block of code is problematic and can lead to OOM when we are we converting the iterator into a list and then returning the iterator back. This just holds up memory in the heap when the executor is running this block of code.
https://github.com/apache/hudi/blob/master/hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/HoodieStreamerUtils.java#L86
```
records = avroRDD.mapPartitions(
(FlatMapFunction<Iterator<GenericRecord>, Either<HoodieRecord,String>>) genericRecordIterator -> {
if (autoGenerateRecordKeys) {
props.setProperty(KeyGenUtils.RECORD_KEY_GEN_PARTITION_ID_CONFIG, String.valueOf(TaskContext.getPartitionId()));
props.setProperty(KeyGenUtils.RECORD_KEY_GEN_INSTANT_TIME_CONFIG, instantTime);
}
BuiltinKeyGenerator builtinKeyGenerator = (BuiltinKeyGenerator) HoodieSparkKeyGeneratorFactory.createKeyGenerator(props);
List<Either<HoodieRecord,String>> avroRecords = new ArrayList<>();
while (genericRecordIterator.hasNext()) {
GenericRecord genRec = genericRecordIterator.next();
try {
HoodieKey hoodieKey = new HoodieKey(builtinKeyGenerator.getRecordKey(genRec), builtinKeyGenerator.getPartitionPath(genRec));
GenericRecord gr = isDropPartitionColumns(props) ? HoodieAvroUtils.removeFields(genRec, partitionColumns) : genRec;
HoodieRecordPayload payload = shouldCombine ? DataSourceUtils.createPayload(cfg.payloadClassName, gr,
(Comparable) HoodieAvroUtils.getNestedFieldVal(gr, cfg.sourceOrderingField, false, props.getBoolean(
KeyGeneratorOptions.KEYGENERATOR_CONSISTENT_LOGICAL_TIMESTAMP_ENABLED.key(),
Boolean.parseBoolean(KeyGeneratorOptions.KEYGENERATOR_CONSISTENT_LOGICAL_TIMESTAMP_ENABLED.defaultValue()))))
: DataSourceUtils.createPayload(cfg.payloadClassName, gr);
avroRecords.add(Either.left(new HoodieAvroRecord<>(hoodieKey, payload)));
} catch (Exception e) {
if (!shouldErrorTable) {
throw e;
}
avroRecords.add(generateErrorRecord(genRec));
}
}
return avroRecords.iterator();
});
```
### Impact
No change in public API, this is making mapPartitions part of the code memory efficient instead of collecting the results in the executor.
### Risk level (write none, low medium or high below)
Medium
### Documentation Update
None.
### Contributor's checklist
- [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Change Logs and Impact were stated clearly
- [x] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7508] Avoid collecting records in HoodieStreamerUtils.createHoodieRecords and JsonKafkaSource mapPartitions [hudi]
Posted by "CTTY (via GitHub)" <gi...@apache.org>.
CTTY commented on code in PR #10872:
URL: https://github.com/apache/hudi/pull/10872#discussion_r1529666064
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/JsonKafkaSource.java:
##########
@@ -57,6 +57,8 @@
*/
public class JsonKafkaSource extends KafkaSource<String> {
+ private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper();
Review Comment:
Just being curious, do we have to use static object mapper here or it's fine to wrap it in the iterator as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7508] Avoid collecting records in HoodieStreamerUtils.createHoodieRecords and JsonKafkaSource mapPartitions [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #10872:
URL: https://github.com/apache/hudi/pull/10872#issuecomment-2103497489
hey @vinishjail97 : can you attach the memory profileing you did before and after this patch. and rebase w/ master. we are good to go
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7508] Avoid collecting records in HoodieStreamerUtils.createHoodieRecords and JsonKafkaSource mapPartitions [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10872:
URL: https://github.com/apache/hudi/pull/10872#issuecomment-2005124255
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "629e91bc0267c0728b98326eb84072965c600205",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22928",
"triggerID" : "629e91bc0267c0728b98326eb84072965c600205",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 629e91bc0267c0728b98326eb84072965c600205 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22928)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7508] Avoid collecting records in HoodieStreamerUtils.createHoodieRecords and JsonKafkaSource mapPartitions [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10872:
URL: https://github.com/apache/hudi/pull/10872#issuecomment-2004984687
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "629e91bc0267c0728b98326eb84072965c600205",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "629e91bc0267c0728b98326eb84072965c600205",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 629e91bc0267c0728b98326eb84072965c600205 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7508] Avoid collecting records in HoodieStreamerUtils.createHoodieRecords and JsonKafkaSource mapPartitions [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10872:
URL: https://github.com/apache/hudi/pull/10872#issuecomment-2105462852
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "629e91bc0267c0728b98326eb84072965c600205",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22928",
"triggerID" : "629e91bc0267c0728b98326eb84072965c600205",
"triggerType" : "PUSH"
}, {
"hash" : "ac7713c64afa1d2406463c8563a065362c95ecda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23640",
"triggerID" : "ac7713c64afa1d2406463c8563a065362c95ecda",
"triggerType" : "PUSH"
}, {
"hash" : "acbabdc64da321e77aaabd03bcd9d5f3c322c0ec",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23841",
"triggerID" : "acbabdc64da321e77aaabd03bcd9d5f3c322c0ec",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* acbabdc64da321e77aaabd03bcd9d5f3c322c0ec Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23841)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7508] Avoid collecting records in HoodieStreamerUtils.createHoodieRecords and JsonKafkaSource mapPartitions [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10872:
URL: https://github.com/apache/hudi/pull/10872#issuecomment-2092052549
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "629e91bc0267c0728b98326eb84072965c600205",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22928",
"triggerID" : "629e91bc0267c0728b98326eb84072965c600205",
"triggerType" : "PUSH"
}, {
"hash" : "ac7713c64afa1d2406463c8563a065362c95ecda",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ac7713c64afa1d2406463c8563a065362c95ecda",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 629e91bc0267c0728b98326eb84072965c600205 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22928)
* ac7713c64afa1d2406463c8563a065362c95ecda UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7508] Avoid collecting records in HoodieStreamerUtils.createHoodieRecords and JsonKafkaSource mapPartitions [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10872:
URL: https://github.com/apache/hudi/pull/10872#issuecomment-2105411384
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "629e91bc0267c0728b98326eb84072965c600205",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22928",
"triggerID" : "629e91bc0267c0728b98326eb84072965c600205",
"triggerType" : "PUSH"
}, {
"hash" : "ac7713c64afa1d2406463c8563a065362c95ecda",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23640",
"triggerID" : "ac7713c64afa1d2406463c8563a065362c95ecda",
"triggerType" : "PUSH"
}, {
"hash" : "acbabdc64da321e77aaabd03bcd9d5f3c322c0ec",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "acbabdc64da321e77aaabd03bcd9d5f3c322c0ec",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ac7713c64afa1d2406463c8563a065362c95ecda Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23640)
* acbabdc64da321e77aaabd03bcd9d5f3c322c0ec UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7508] Avoid collecting records in HoodieStreamerUtils.createHoodieRecords and JsonKafkaSource mapPartitions [hudi]
Posted by "vinishjail97 (via GitHub)" <gi...@apache.org>.
vinishjail97 commented on PR #10872:
URL: https://github.com/apache/hudi/pull/10872#issuecomment-2105950888
> hey @vinishjail97 : can you attach the memory profileing you did before and after this patch. and rebase w/ master. we are good to go
15th March: Basic OOM Test (Consume 2M events, each payload is approximately 1KB with 2 maxExecutors and 1GB memory) and dynamic allocation ratio was 0.002 so essentially only 1 executor will be used as tasks spawned are not enough.
driver:
coreLimit: 2000m
coreRequest: 1800m
cores: 2
labels:
orgId: 0c043996-9e42-4904-95b9-f98918ebeda4
version: 3.1.1
memory: 2g
serviceAccount: staging-spark
dynamicAllocation:
enabled: true
initialExecutors: 0
maxExecutors: 2
minExecutors: 0
executor:
coreLimit: 1000m
coreRequest: 750m
cores: 1
labels:
orgId: 0c043996-9e42-4904-95b9-f98918ebeda4
version: 3.1.1
memory: 1g
Without the fix, the stage was failing with executor OOM after 20min.

After using this fix, the same stage completed in 17min with one executor.

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7508] Avoid collecting records in HoodieStreamerUtils.createHoodieRecords and JsonKafkaSource mapPartitions [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #10872:
URL: https://github.com/apache/hudi/pull/10872#discussion_r1597306796
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/JsonKafkaSource.java:
##########
@@ -57,6 +57,8 @@
*/
public class JsonKafkaSource extends KafkaSource<String> {
+ private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper();
Review Comment:
Good point, I think we should revert this change, because anyway the `ObjectMapper` is serde from Spark driver to executor if using the static object, so there's not much gain.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7508] Avoid collecting records in HoodieStreamerUtils.createHoodieRecords and JsonKafkaSource mapPartitions [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10872:
URL: https://github.com/apache/hudi/pull/10872#issuecomment-2000362084
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "629e91bc0267c0728b98326eb84072965c600205",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "629e91bc0267c0728b98326eb84072965c600205",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 629e91bc0267c0728b98326eb84072965c600205 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7508] Avoid collecting records in HoodieStreamerUtils.createHoodieRecords and JsonKafkaSource mapPartitions [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10872:
URL: https://github.com/apache/hudi/pull/10872#issuecomment-2105414177
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "629e91bc0267c0728b98326eb84072965c600205",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22928",
"triggerID" : "629e91bc0267c0728b98326eb84072965c600205",
"triggerType" : "PUSH"
}, {
"hash" : "ac7713c64afa1d2406463c8563a065362c95ecda",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23640",
"triggerID" : "ac7713c64afa1d2406463c8563a065362c95ecda",
"triggerType" : "PUSH"
}, {
"hash" : "acbabdc64da321e77aaabd03bcd9d5f3c322c0ec",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23841",
"triggerID" : "acbabdc64da321e77aaabd03bcd9d5f3c322c0ec",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ac7713c64afa1d2406463c8563a065362c95ecda Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23640)
* acbabdc64da321e77aaabd03bcd9d5f3c322c0ec Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23841)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7508] Avoid collecting records in HoodieStreamerUtils.createHoodieRecords and JsonKafkaSource mapPartitions [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10872:
URL: https://github.com/apache/hudi/pull/10872#issuecomment-2092056890
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "629e91bc0267c0728b98326eb84072965c600205",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22928",
"triggerID" : "629e91bc0267c0728b98326eb84072965c600205",
"triggerType" : "PUSH"
}, {
"hash" : "ac7713c64afa1d2406463c8563a065362c95ecda",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23640",
"triggerID" : "ac7713c64afa1d2406463c8563a065362c95ecda",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 629e91bc0267c0728b98326eb84072965c600205 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22928)
* ac7713c64afa1d2406463c8563a065362c95ecda Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23640)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7508] Avoid collecting records in HoodieStreamerUtils.createHoodieRecords and JsonKafkaSource mapPartitions [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10872:
URL: https://github.com/apache/hudi/pull/10872#issuecomment-2004996708
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "629e91bc0267c0728b98326eb84072965c600205",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22928",
"triggerID" : "629e91bc0267c0728b98326eb84072965c600205",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 629e91bc0267c0728b98326eb84072965c600205 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22928)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7508] Avoid collecting records in HoodieStreamerUtils.createHoodieRecords and JsonKafkaSource mapPartitions [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #10872:
URL: https://github.com/apache/hudi/pull/10872#discussion_r1527412766
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/JsonKafkaSource.java:
##########
@@ -124,4 +123,4 @@ private JavaRDD<String> postProcess(JavaRDD<String> jsonStringRDD) {
return processor.process(jsonStringRDD);
}
-}
+}
Review Comment:
nit: new line in the end
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/HoodieStreamerUtils.java:
##########
@@ -186,4 +182,4 @@ static Set<String> getPartitionColumns(TypedProperties props) {
return Arrays.stream(partitionColumns.split(",")).collect(Collectors.toSet());
}
-}
+}
Review Comment:
nit: new line at the end
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/HoodieStreamerUtils.java:
##########
@@ -91,26 +89,24 @@ public static Option<JavaRDD<HoodieRecord>> createHoodieRecords(HoodieStreamer.C
props.setProperty(KeyGenUtils.RECORD_KEY_GEN_INSTANT_TIME_CONFIG, instantTime);
}
BuiltinKeyGenerator builtinKeyGenerator = (BuiltinKeyGenerator) HoodieSparkKeyGeneratorFactory.createKeyGenerator(props);
- List<Either<HoodieRecord,String>> avroRecords = new ArrayList<>();
- while (genericRecordIterator.hasNext()) {
- GenericRecord genRec = genericRecordIterator.next();
+ boolean useConsistentLogicalTimestamp = props.getBoolean(
+ KeyGeneratorOptions.KEYGENERATOR_CONSISTENT_LOGICAL_TIMESTAMP_ENABLED.key(),
+ Boolean.parseBoolean(KeyGeneratorOptions.KEYGENERATOR_CONSISTENT_LOGICAL_TIMESTAMP_ENABLED.defaultValue()));
Review Comment:
We can even extract this out of `mapPartitions`. This keygen config is not going to change across partitions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7508] Avoid collecting records in HoodieStreamerUtils.createHoodieRecords and JsonKafkaSource mapPartitions [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua merged PR #10872:
URL: https://github.com/apache/hudi/pull/10872
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7508] Avoid collecting records in HoodieStreamerUtils.createHoodieRecords and JsonKafkaSource mapPartitions [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10872:
URL: https://github.com/apache/hudi/pull/10872#issuecomment-2092180728
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "629e91bc0267c0728b98326eb84072965c600205",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22928",
"triggerID" : "629e91bc0267c0728b98326eb84072965c600205",
"triggerType" : "PUSH"
}, {
"hash" : "ac7713c64afa1d2406463c8563a065362c95ecda",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23640",
"triggerID" : "ac7713c64afa1d2406463c8563a065362c95ecda",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ac7713c64afa1d2406463c8563a065362c95ecda Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23640)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7508] Avoid collecting records in HoodieStreamerUtils.createHoodieRecords and JsonKafkaSource mapPartitions [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10872:
URL: https://github.com/apache/hudi/pull/10872#issuecomment-2000368965
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "629e91bc0267c0728b98326eb84072965c600205",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22928",
"triggerID" : "629e91bc0267c0728b98326eb84072965c600205",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 629e91bc0267c0728b98326eb84072965c600205 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22928)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org