You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "XuQianJin-Stars (via GitHub)" <gi...@apache.org> on 2023/02/27 02:00:11 UTC
[GitHub] [hudi] XuQianJin-Stars opened a new pull request, #8050: [MINOR] Fix oom caused by addResource in flink jm building HiveSyncCo…
XuQianJin-Stars opened a new pull request, #8050:
URL: https://github.com/apache/hudi/pull/8050
[MINOR] Fix oom caused by addResource in flink jm building HiveSyncConfig multiple times
<img width="933" alt="image" src="https://user-images.githubusercontent.com/10494131/221454471-d4fc7a27-d130-49bf-96c3-674a9ca43aaa.png">
### Change Logs
`HiveSyncContext` and `StreamWriteOperatorCoordinator`
### Impact
NA
### Risk level (write none, low medium or high below)
low
### Documentation Update
NA
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8050: [HUDI-5855] Release resource actively for Flink hive meta sync
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8050:
URL: https://github.com/apache/hudi/pull/8050#discussion_r1118315851
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HiveSyncConfig.java:
##########
@@ -98,8 +98,6 @@ public HiveSyncConfig(Properties props, Configuration hadoopConf) {
super(props, hadoopConf);
HiveConf hiveConf = hadoopConf instanceof HiveConf
? (HiveConf) hadoopConf : new HiveConf(hadoopConf, HiveConf.class);
- // HiveConf needs to load fs conf to allow instantiation via AWSGlueClientFactory
- hiveConf.addResource(getHadoopFileSystem().getConf());
Review Comment:
cc @xushiyan for visibility.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8050: [MINOR] Fix oom caused by addResource in flink jm building HiveSyncCo…
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8050:
URL: https://github.com/apache/hudi/pull/8050#issuecomment-1445647605
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3d2cd7e4760e2f52273ad1395dae06b73f316108",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3d2cd7e4760e2f52273ad1395dae06b73f316108",
"triggerType" : "PUSH"
}, {
"hash" : "da097d950bc412a7db7c5599788abcff5f7fb6b1",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15414",
"triggerID" : "da097d950bc412a7db7c5599788abcff5f7fb6b1",
"triggerType" : "PUSH"
}, {
"hash" : "0a67df5bd405255a912fd1f556efa29475ba0e6a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15418",
"triggerID" : "0a67df5bd405255a912fd1f556efa29475ba0e6a",
"triggerType" : "PUSH"
}, {
"hash" : "d37c672dcedcb5383cb30d01f0b8f679694dbf2e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d37c672dcedcb5383cb30d01f0b8f679694dbf2e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3d2cd7e4760e2f52273ad1395dae06b73f316108 UNKNOWN
* da097d950bc412a7db7c5599788abcff5f7fb6b1 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15414)
* 0a67df5bd405255a912fd1f556efa29475ba0e6a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15418)
* d37c672dcedcb5383cb30d01f0b8f679694dbf2e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8050: [MINOR] Fix oom caused by addResource in flink jm building HiveSyncCo…
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8050:
URL: https://github.com/apache/hudi/pull/8050#issuecomment-1445637520
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3d2cd7e4760e2f52273ad1395dae06b73f316108",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3d2cd7e4760e2f52273ad1395dae06b73f316108",
"triggerType" : "PUSH"
}, {
"hash" : "da097d950bc412a7db7c5599788abcff5f7fb6b1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15414",
"triggerID" : "da097d950bc412a7db7c5599788abcff5f7fb6b1",
"triggerType" : "PUSH"
}, {
"hash" : "0a67df5bd405255a912fd1f556efa29475ba0e6a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0a67df5bd405255a912fd1f556efa29475ba0e6a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3d2cd7e4760e2f52273ad1395dae06b73f316108 UNKNOWN
* da097d950bc412a7db7c5599788abcff5f7fb6b1 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15414)
* 0a67df5bd405255a912fd1f556efa29475ba0e6a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8050: [MINOR] Fix oom caused by addResource in flink jm building HiveSyncCo…
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8050:
URL: https://github.com/apache/hudi/pull/8050#issuecomment-1445597748
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3d2cd7e4760e2f52273ad1395dae06b73f316108",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3d2cd7e4760e2f52273ad1395dae06b73f316108",
"triggerType" : "PUSH"
}, {
"hash" : "da097d950bc412a7db7c5599788abcff5f7fb6b1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15414",
"triggerID" : "da097d950bc412a7db7c5599788abcff5f7fb6b1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3d2cd7e4760e2f52273ad1395dae06b73f316108 UNKNOWN
* da097d950bc412a7db7c5599788abcff5f7fb6b1 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15414)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8050: [MINOR] Release resource actively for Flink hive meta sync
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8050:
URL: https://github.com/apache/hudi/pull/8050#issuecomment-1445686481
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3d2cd7e4760e2f52273ad1395dae06b73f316108",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3d2cd7e4760e2f52273ad1395dae06b73f316108",
"triggerType" : "PUSH"
}, {
"hash" : "da097d950bc412a7db7c5599788abcff5f7fb6b1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15414",
"triggerID" : "da097d950bc412a7db7c5599788abcff5f7fb6b1",
"triggerType" : "PUSH"
}, {
"hash" : "0a67df5bd405255a912fd1f556efa29475ba0e6a",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15418",
"triggerID" : "0a67df5bd405255a912fd1f556efa29475ba0e6a",
"triggerType" : "PUSH"
}, {
"hash" : "d37c672dcedcb5383cb30d01f0b8f679694dbf2e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15420",
"triggerID" : "d37c672dcedcb5383cb30d01f0b8f679694dbf2e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3d2cd7e4760e2f52273ad1395dae06b73f316108 UNKNOWN
* 0a67df5bd405255a912fd1f556efa29475ba0e6a Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15418)
* d37c672dcedcb5383cb30d01f0b8f679694dbf2e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15420)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 merged pull request #8050: [HUDI-5855] Release resource actively for Flink hive meta sync
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 merged PR #8050:
URL: https://github.com/apache/hudi/pull/8050
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8050: [MINOR] Release resource actively for Flink hive meta sync
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8050:
URL: https://github.com/apache/hudi/pull/8050#issuecomment-1445691072
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3d2cd7e4760e2f52273ad1395dae06b73f316108",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3d2cd7e4760e2f52273ad1395dae06b73f316108",
"triggerType" : "PUSH"
}, {
"hash" : "da097d950bc412a7db7c5599788abcff5f7fb6b1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15414",
"triggerID" : "da097d950bc412a7db7c5599788abcff5f7fb6b1",
"triggerType" : "PUSH"
}, {
"hash" : "0a67df5bd405255a912fd1f556efa29475ba0e6a",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15418",
"triggerID" : "0a67df5bd405255a912fd1f556efa29475ba0e6a",
"triggerType" : "PUSH"
}, {
"hash" : "d37c672dcedcb5383cb30d01f0b8f679694dbf2e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15420",
"triggerID" : "d37c672dcedcb5383cb30d01f0b8f679694dbf2e",
"triggerType" : "PUSH"
}, {
"hash" : "2b51bbe98f26784ef8f4eee872a8e2a3b2429ee0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2b51bbe98f26784ef8f4eee872a8e2a3b2429ee0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3d2cd7e4760e2f52273ad1395dae06b73f316108 UNKNOWN
* 0a67df5bd405255a912fd1f556efa29475ba0e6a Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15418)
* d37c672dcedcb5383cb30d01f0b8f679694dbf2e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15420)
* 2b51bbe98f26784ef8f4eee872a8e2a3b2429ee0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8050: [MINOR] Fix oom caused by addResource in flink jm building HiveSyncCo…
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8050:
URL: https://github.com/apache/hudi/pull/8050#discussion_r1118251389
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/utils/HiveSyncContext.java:
##########
@@ -90,7 +97,27 @@ public static HiveSyncContext create(Configuration conf, SerializableConfigurati
hadoopConf.set(HiveConf.ConfVars.METASTOREURIS.varname, conf.getString(FlinkOptions.HIVE_SYNC_METASTORE_URIS));
}
hiveConf.addResource(hadoopConf);
- return new HiveSyncContext(props, hiveConf);
+
+ HiveSyncTool hiveSyncTool = buildHiveSyncTool(props, hiveConf);
+ return new HiveSyncContext(props, hiveConf, hiveSyncTool);
+ }
+
+ private static HiveSyncTool buildHiveSyncTool(Properties props, HiveConf hiveConf) {
+ HiveSyncTool hiveSyncTool;
+ HiveSyncMode syncMode = HiveSyncMode.of(props.getProperty(HIVE_SYNC_MODE.key()));
+ if (syncMode == HiveSyncMode.GLUE) {
Review Comment:
Revert this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8050: [HUDI-5855] Release resource actively for Flink hive meta sync
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8050:
URL: https://github.com/apache/hudi/pull/8050#issuecomment-1445785279
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3d2cd7e4760e2f52273ad1395dae06b73f316108",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3d2cd7e4760e2f52273ad1395dae06b73f316108",
"triggerType" : "PUSH"
}, {
"hash" : "da097d950bc412a7db7c5599788abcff5f7fb6b1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15414",
"triggerID" : "da097d950bc412a7db7c5599788abcff5f7fb6b1",
"triggerType" : "PUSH"
}, {
"hash" : "0a67df5bd405255a912fd1f556efa29475ba0e6a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15418",
"triggerID" : "0a67df5bd405255a912fd1f556efa29475ba0e6a",
"triggerType" : "PUSH"
}, {
"hash" : "d37c672dcedcb5383cb30d01f0b8f679694dbf2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15420",
"triggerID" : "d37c672dcedcb5383cb30d01f0b8f679694dbf2e",
"triggerType" : "PUSH"
}, {
"hash" : "2b51bbe98f26784ef8f4eee872a8e2a3b2429ee0",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15423",
"triggerID" : "2b51bbe98f26784ef8f4eee872a8e2a3b2429ee0",
"triggerType" : "PUSH"
}, {
"hash" : "509d43545f018d975e2a179e7e475513de5163e9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15426",
"triggerID" : "509d43545f018d975e2a179e7e475513de5163e9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3d2cd7e4760e2f52273ad1395dae06b73f316108 UNKNOWN
* 2b51bbe98f26784ef8f4eee872a8e2a3b2429ee0 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15423)
* 509d43545f018d975e2a179e7e475513de5163e9 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15426)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8050: [MINOR] Fix oom caused by addResource in flink jm building HiveSyncCo…
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8050:
URL: https://github.com/apache/hudi/pull/8050#issuecomment-1445577222
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3d2cd7e4760e2f52273ad1395dae06b73f316108",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3d2cd7e4760e2f52273ad1395dae06b73f316108",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3d2cd7e4760e2f52273ad1395dae06b73f316108 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8050: [MINOR] Fix oom caused by addResource in flink jm building HiveSyncCo…
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8050:
URL: https://github.com/apache/hudi/pull/8050#issuecomment-1445587117
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3d2cd7e4760e2f52273ad1395dae06b73f316108",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3d2cd7e4760e2f52273ad1395dae06b73f316108",
"triggerType" : "PUSH"
}, {
"hash" : "da097d950bc412a7db7c5599788abcff5f7fb6b1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "da097d950bc412a7db7c5599788abcff5f7fb6b1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3d2cd7e4760e2f52273ad1395dae06b73f316108 UNKNOWN
* da097d950bc412a7db7c5599788abcff5f7fb6b1 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8050: [MINOR] Fix oom caused by addResource in flink jm building HiveSyncCo…
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8050:
URL: https://github.com/apache/hudi/pull/8050#issuecomment-1445643684
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3d2cd7e4760e2f52273ad1395dae06b73f316108",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3d2cd7e4760e2f52273ad1395dae06b73f316108",
"triggerType" : "PUSH"
}, {
"hash" : "da097d950bc412a7db7c5599788abcff5f7fb6b1",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15414",
"triggerID" : "da097d950bc412a7db7c5599788abcff5f7fb6b1",
"triggerType" : "PUSH"
}, {
"hash" : "0a67df5bd405255a912fd1f556efa29475ba0e6a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15418",
"triggerID" : "0a67df5bd405255a912fd1f556efa29475ba0e6a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3d2cd7e4760e2f52273ad1395dae06b73f316108 UNKNOWN
* da097d950bc412a7db7c5599788abcff5f7fb6b1 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15414)
* 0a67df5bd405255a912fd1f556efa29475ba0e6a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15418)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8050: [MINOR] Release resource actively for Flink hive meta sync
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8050:
URL: https://github.com/apache/hudi/pull/8050#discussion_r1118314626
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HiveSyncConfig.java:
##########
@@ -98,8 +98,6 @@ public HiveSyncConfig(Properties props, Configuration hadoopConf) {
super(props, hadoopConf);
HiveConf hiveConf = hadoopConf instanceof HiveConf
? (HiveConf) hadoopConf : new HiveConf(hadoopConf, HiveConf.class);
- // HiveConf needs to load fs conf to allow instantiation via AWSGlueClientFactory
- hiveConf.addResource(getHadoopFileSystem().getConf());
Review Comment:
This line causes the object reference leak, we should never modify passed in param directly, the param should be immutable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8050: [HUDI-5855] Release resource actively for Flink hive meta sync
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8050:
URL: https://github.com/apache/hudi/pull/8050#issuecomment-1445779230
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3d2cd7e4760e2f52273ad1395dae06b73f316108",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3d2cd7e4760e2f52273ad1395dae06b73f316108",
"triggerType" : "PUSH"
}, {
"hash" : "da097d950bc412a7db7c5599788abcff5f7fb6b1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15414",
"triggerID" : "da097d950bc412a7db7c5599788abcff5f7fb6b1",
"triggerType" : "PUSH"
}, {
"hash" : "0a67df5bd405255a912fd1f556efa29475ba0e6a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15418",
"triggerID" : "0a67df5bd405255a912fd1f556efa29475ba0e6a",
"triggerType" : "PUSH"
}, {
"hash" : "d37c672dcedcb5383cb30d01f0b8f679694dbf2e",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15420",
"triggerID" : "d37c672dcedcb5383cb30d01f0b8f679694dbf2e",
"triggerType" : "PUSH"
}, {
"hash" : "2b51bbe98f26784ef8f4eee872a8e2a3b2429ee0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15423",
"triggerID" : "2b51bbe98f26784ef8f4eee872a8e2a3b2429ee0",
"triggerType" : "PUSH"
}, {
"hash" : "509d43545f018d975e2a179e7e475513de5163e9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "509d43545f018d975e2a179e7e475513de5163e9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3d2cd7e4760e2f52273ad1395dae06b73f316108 UNKNOWN
* d37c672dcedcb5383cb30d01f0b8f679694dbf2e Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15420)
* 2b51bbe98f26784ef8f4eee872a8e2a3b2429ee0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15423)
* 509d43545f018d975e2a179e7e475513de5163e9 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8050: [MINOR] Release resource actively for Flink hive meta sync
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8050:
URL: https://github.com/apache/hudi/pull/8050#issuecomment-1445695003
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3d2cd7e4760e2f52273ad1395dae06b73f316108",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3d2cd7e4760e2f52273ad1395dae06b73f316108",
"triggerType" : "PUSH"
}, {
"hash" : "da097d950bc412a7db7c5599788abcff5f7fb6b1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15414",
"triggerID" : "da097d950bc412a7db7c5599788abcff5f7fb6b1",
"triggerType" : "PUSH"
}, {
"hash" : "0a67df5bd405255a912fd1f556efa29475ba0e6a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15418",
"triggerID" : "0a67df5bd405255a912fd1f556efa29475ba0e6a",
"triggerType" : "PUSH"
}, {
"hash" : "d37c672dcedcb5383cb30d01f0b8f679694dbf2e",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15420",
"triggerID" : "d37c672dcedcb5383cb30d01f0b8f679694dbf2e",
"triggerType" : "PUSH"
}, {
"hash" : "2b51bbe98f26784ef8f4eee872a8e2a3b2429ee0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15423",
"triggerID" : "2b51bbe98f26784ef8f4eee872a8e2a3b2429ee0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3d2cd7e4760e2f52273ad1395dae06b73f316108 UNKNOWN
* d37c672dcedcb5383cb30d01f0b8f679694dbf2e Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15420)
* 2b51bbe98f26784ef8f4eee872a8e2a3b2429ee0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15423)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8050: [HUDI-5855] Release resource actively for Flink hive meta sync
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8050:
URL: https://github.com/apache/hudi/pull/8050#issuecomment-1446160365
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3d2cd7e4760e2f52273ad1395dae06b73f316108",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3d2cd7e4760e2f52273ad1395dae06b73f316108",
"triggerType" : "PUSH"
}, {
"hash" : "da097d950bc412a7db7c5599788abcff5f7fb6b1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15414",
"triggerID" : "da097d950bc412a7db7c5599788abcff5f7fb6b1",
"triggerType" : "PUSH"
}, {
"hash" : "0a67df5bd405255a912fd1f556efa29475ba0e6a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15418",
"triggerID" : "0a67df5bd405255a912fd1f556efa29475ba0e6a",
"triggerType" : "PUSH"
}, {
"hash" : "d37c672dcedcb5383cb30d01f0b8f679694dbf2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15420",
"triggerID" : "d37c672dcedcb5383cb30d01f0b8f679694dbf2e",
"triggerType" : "PUSH"
}, {
"hash" : "2b51bbe98f26784ef8f4eee872a8e2a3b2429ee0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15423",
"triggerID" : "2b51bbe98f26784ef8f4eee872a8e2a3b2429ee0",
"triggerType" : "PUSH"
}, {
"hash" : "509d43545f018d975e2a179e7e475513de5163e9",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15426",
"triggerID" : "509d43545f018d975e2a179e7e475513de5163e9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3d2cd7e4760e2f52273ad1395dae06b73f316108 UNKNOWN
* 509d43545f018d975e2a179e7e475513de5163e9 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15426)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org