You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/05/25 02:31:52 UTC
[GitHub] [hudi] cuibo01 opened a new pull request, #5675: [HUDI-4151] flink split_reader supports rocksdb
cuibo01 opened a new pull request, #5675:
URL: https://github.com/apache/hudi/pull/5675
## *Tips*
- *Thank you very much for contributing to Apache Hudi.*
- *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.*
## What is the purpose of the pull request
*(For example: This pull request adds quick-start document.)*
## Brief change log
*(for example:)*
- *Modify AnnotationLocation checkstyle rule in checkstyle.xml*
## Verify this pull request
*(Please pick either of the following options)*
This pull request is a trivial rework / code cleanup without any test coverage.
*(or)*
This pull request is already covered by existing tests, such as *(please describe tests)*.
(or)
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end.*
- *Added HoodieClientWriteTest to verify the change.*
- *Manually verified the change by running a job locally.*
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5675: [HUDI-4151] flink split_reader supports rocksdb
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5675:
URL: https://github.com/apache/hudi/pull/5675#issuecomment-1139423879
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e8a278ecd4bbd33e2cc5fb3582a41c3edd90377f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8908",
"triggerID" : "e8a278ecd4bbd33e2cc5fb3582a41c3edd90377f",
"triggerType" : "PUSH"
}, {
"hash" : "7f38c09425e02aab5da82a122a955272b1792f71",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8919",
"triggerID" : "7f38c09425e02aab5da82a122a955272b1792f71",
"triggerType" : "PUSH"
}, {
"hash" : "aa7df7c99896831a050c515a793f55cf5c716789",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8923",
"triggerID" : "aa7df7c99896831a050c515a793f55cf5c716789",
"triggerType" : "PUSH"
}, {
"hash" : "36161bf2dea866c5aecc745af9109e66b2af0726",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8926",
"triggerID" : "36161bf2dea866c5aecc745af9109e66b2af0726",
"triggerType" : "PUSH"
}, {
"hash" : "d84585bf3018a6cb9795059cfa24a8f61affa023",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8933",
"triggerID" : "d84585bf3018a6cb9795059cfa24a8f61affa023",
"triggerType" : "PUSH"
}, {
"hash" : "498df0d128c789e746241d7d2e0fce930f4b0dc4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8967",
"triggerID" : "498df0d128c789e746241d7d2e0fce930f4b0dc4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d84585bf3018a6cb9795059cfa24a8f61affa023 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8933)
* 498df0d128c789e746241d7d2e0fce930f4b0dc4 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8967)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #5675: [HUDI-4151] flink split_reader supports rocksdb
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #5675:
URL: https://github.com/apache/hudi/pull/5675#discussion_r881404871
##########
hudi-common/src/main/java/org/apache/hudi/common/util/collection/RocksDbDiskMap.java:
##########
@@ -41,21 +42,25 @@
* All of the data is stored using the RocksDB implementation.
*/
public final class RocksDbDiskMap<T extends Serializable, R extends Serializable> extends DiskMap<T, R> {
+ private static final Logger LOG = LogManager.getLogger(RocksDbDiskMap.class);
+
// ColumnFamily allows partitioning data within RockDB, which allows
// independent configuration and faster deletes across partitions
// https://github.com/facebook/rocksdb/wiki/Column-Families
// For this use case, we use a single static column family/ partition
//
- private static final String ROCKSDB_COL_FAMILY = "rocksdb-diskmap";
+ private static final String ROCKSDB_BASE_PATH = "rocksdb-diskmap";
- private static final Logger LOG = LogManager.getLogger(RocksDbDiskMap.class);
// Stores the key and corresponding value's latest metadata spilled to disk
private final Set<T> keySet;
- private RocksDBDAO rocksDb;
+ private static RocksDBDAO rocksDb = null;
+ private String rocksdbColFamily;
public RocksDbDiskMap(String rocksDbStoragePath) throws IOException {
super(rocksDbStoragePath, ExternalSpillableMap.DiskMapType.ROCKS_DB.name());
this.keySet = new HashSet<>();
Review Comment:
Do we share the `rocksDbStoragePath` ? Why we need multiple column family here ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #5675: [HUDI-4151] flink split_reader supports rocksdb
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #5675:
URL: https://github.com/apache/hudi/pull/5675#discussion_r883237773
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/format/FormatUtils.java:
##########
@@ -120,9 +123,12 @@ private static Object getVal(IndexedRecord record, int pos) {
public static HoodieMergedLogRecordScanner logScanner(
MergeOnReadInputSplit split,
Schema logSchema,
- Configuration config,
- boolean withOperationField) {
Review Comment:
Oops, sorry, i checked the code and find it is better to construct the `writeConfig` in `#logScanner`, you original code that passes in Flink Configuration is better :), sorry for the mistake. You can just construct the writeConfig inside `#logScanner` is only this method uses the write config.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5675: [HUDI-4151] flink split_reader supports rocksdb
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5675:
URL: https://github.com/apache/hudi/pull/5675#issuecomment-1139421011
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e8a278ecd4bbd33e2cc5fb3582a41c3edd90377f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8908",
"triggerID" : "e8a278ecd4bbd33e2cc5fb3582a41c3edd90377f",
"triggerType" : "PUSH"
}, {
"hash" : "7f38c09425e02aab5da82a122a955272b1792f71",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8919",
"triggerID" : "7f38c09425e02aab5da82a122a955272b1792f71",
"triggerType" : "PUSH"
}, {
"hash" : "aa7df7c99896831a050c515a793f55cf5c716789",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8923",
"triggerID" : "aa7df7c99896831a050c515a793f55cf5c716789",
"triggerType" : "PUSH"
}, {
"hash" : "36161bf2dea866c5aecc745af9109e66b2af0726",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8926",
"triggerID" : "36161bf2dea866c5aecc745af9109e66b2af0726",
"triggerType" : "PUSH"
}, {
"hash" : "d84585bf3018a6cb9795059cfa24a8f61affa023",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8933",
"triggerID" : "d84585bf3018a6cb9795059cfa24a8f61affa023",
"triggerType" : "PUSH"
}, {
"hash" : "498df0d128c789e746241d7d2e0fce930f4b0dc4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "498df0d128c789e746241d7d2e0fce930f4b0dc4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d84585bf3018a6cb9795059cfa24a8f61affa023 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8933)
* 498df0d128c789e746241d7d2e0fce930f4b0dc4 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5675: [HUDI-4151] flink split_reader supports rocksdb
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5675:
URL: https://github.com/apache/hudi/pull/5675#issuecomment-1139470927
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e8a278ecd4bbd33e2cc5fb3582a41c3edd90377f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8908",
"triggerID" : "e8a278ecd4bbd33e2cc5fb3582a41c3edd90377f",
"triggerType" : "PUSH"
}, {
"hash" : "7f38c09425e02aab5da82a122a955272b1792f71",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8919",
"triggerID" : "7f38c09425e02aab5da82a122a955272b1792f71",
"triggerType" : "PUSH"
}, {
"hash" : "aa7df7c99896831a050c515a793f55cf5c716789",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8923",
"triggerID" : "aa7df7c99896831a050c515a793f55cf5c716789",
"triggerType" : "PUSH"
}, {
"hash" : "36161bf2dea866c5aecc745af9109e66b2af0726",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8926",
"triggerID" : "36161bf2dea866c5aecc745af9109e66b2af0726",
"triggerType" : "PUSH"
}, {
"hash" : "d84585bf3018a6cb9795059cfa24a8f61affa023",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8933",
"triggerID" : "d84585bf3018a6cb9795059cfa24a8f61affa023",
"triggerType" : "PUSH"
}, {
"hash" : "498df0d128c789e746241d7d2e0fce930f4b0dc4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8967",
"triggerID" : "498df0d128c789e746241d7d2e0fce930f4b0dc4",
"triggerType" : "PUSH"
}, {
"hash" : "89ce5aae7139def8393ccd20dac2c07ea868d265",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8971",
"triggerID" : "89ce5aae7139def8393ccd20dac2c07ea868d265",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 498df0d128c789e746241d7d2e0fce930f4b0dc4 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8967)
* 89ce5aae7139def8393ccd20dac2c07ea868d265 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8971)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #5675: [HUDI-4151] flink split_reader supports rocksdb
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #5675:
URL: https://github.com/apache/hudi/pull/5675#discussion_r881420228
##########
hudi-common/src/main/java/org/apache/hudi/common/util/collection/RocksDbDiskMap.java:
##########
@@ -41,21 +42,25 @@
* All of the data is stored using the RocksDB implementation.
*/
public final class RocksDbDiskMap<T extends Serializable, R extends Serializable> extends DiskMap<T, R> {
+ private static final Logger LOG = LogManager.getLogger(RocksDbDiskMap.class);
+
// ColumnFamily allows partitioning data within RockDB, which allows
// independent configuration and faster deletes across partitions
// https://github.com/facebook/rocksdb/wiki/Column-Families
// For this use case, we use a single static column family/ partition
//
- private static final String ROCKSDB_COL_FAMILY = "rocksdb-diskmap";
+ private static final String ROCKSDB_BASE_PATH = "rocksdb-diskmap";
- private static final Logger LOG = LogManager.getLogger(RocksDbDiskMap.class);
// Stores the key and corresponding value's latest metadata spilled to disk
private final Set<T> keySet;
- private RocksDBDAO rocksDb;
+ private static RocksDBDAO rocksDb = null;
+ private String rocksdbColFamily;
public RocksDbDiskMap(String rocksDbStoragePath) throws IOException {
super(rocksDbStoragePath, ExternalSpillableMap.DiskMapType.ROCKS_DB.name());
this.keySet = new HashSet<>();
Review Comment:
Does multiple rocks db has some problem here ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5675: [HUDI-4151] flink split_reader supports rocksdb
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5675:
URL: https://github.com/apache/hudi/pull/5675#issuecomment-1139593176
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e8a278ecd4bbd33e2cc5fb3582a41c3edd90377f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8908",
"triggerID" : "e8a278ecd4bbd33e2cc5fb3582a41c3edd90377f",
"triggerType" : "PUSH"
}, {
"hash" : "7f38c09425e02aab5da82a122a955272b1792f71",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8919",
"triggerID" : "7f38c09425e02aab5da82a122a955272b1792f71",
"triggerType" : "PUSH"
}, {
"hash" : "aa7df7c99896831a050c515a793f55cf5c716789",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8923",
"triggerID" : "aa7df7c99896831a050c515a793f55cf5c716789",
"triggerType" : "PUSH"
}, {
"hash" : "36161bf2dea866c5aecc745af9109e66b2af0726",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8926",
"triggerID" : "36161bf2dea866c5aecc745af9109e66b2af0726",
"triggerType" : "PUSH"
}, {
"hash" : "d84585bf3018a6cb9795059cfa24a8f61affa023",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8933",
"triggerID" : "d84585bf3018a6cb9795059cfa24a8f61affa023",
"triggerType" : "PUSH"
}, {
"hash" : "498df0d128c789e746241d7d2e0fce930f4b0dc4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8967",
"triggerID" : "498df0d128c789e746241d7d2e0fce930f4b0dc4",
"triggerType" : "PUSH"
}, {
"hash" : "89ce5aae7139def8393ccd20dac2c07ea868d265",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8971",
"triggerID" : "89ce5aae7139def8393ccd20dac2c07ea868d265",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 89ce5aae7139def8393ccd20dac2c07ea868d265 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8971)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5675: [HUDI-4151] flink split_reader supports rocksdb
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5675:
URL: https://github.com/apache/hudi/pull/5675#issuecomment-1136693485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e8a278ecd4bbd33e2cc5fb3582a41c3edd90377f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8908",
"triggerID" : "e8a278ecd4bbd33e2cc5fb3582a41c3edd90377f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e8a278ecd4bbd33e2cc5fb3582a41c3edd90377f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8908)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] cuibo01 commented on a diff in pull request #5675: [HUDI-4151] flink split_reader supports rocksdb
Posted by GitBox <gi...@apache.org>.
cuibo01 commented on code in PR #5675:
URL: https://github.com/apache/hudi/pull/5675#discussion_r883448428
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/format/FormatUtils.java:
##########
@@ -120,9 +123,12 @@ private static Object getVal(IndexedRecord record, int pos) {
public static HoodieMergedLogRecordScanner logScanner(
MergeOnReadInputSplit split,
Schema logSchema,
- Configuration config,
- boolean withOperationField) {
Review Comment:
yes u are right 👍
hadoopConf does not support serialization
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5675: [HUDI-4151] flink split_reader supports rocksdb
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5675:
URL: https://github.com/apache/hudi/pull/5675#issuecomment-1136667885
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e8a278ecd4bbd33e2cc5fb3582a41c3edd90377f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8908",
"triggerID" : "e8a278ecd4bbd33e2cc5fb3582a41c3edd90377f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e8a278ecd4bbd33e2cc5fb3582a41c3edd90377f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8908)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5675: [HUDI-4151] flink split_reader supports rocksdb
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5675:
URL: https://github.com/apache/hudi/pull/5675#issuecomment-1138033419
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e8a278ecd4bbd33e2cc5fb3582a41c3edd90377f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8908",
"triggerID" : "e8a278ecd4bbd33e2cc5fb3582a41c3edd90377f",
"triggerType" : "PUSH"
}, {
"hash" : "7f38c09425e02aab5da82a122a955272b1792f71",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8919",
"triggerID" : "7f38c09425e02aab5da82a122a955272b1792f71",
"triggerType" : "PUSH"
}, {
"hash" : "aa7df7c99896831a050c515a793f55cf5c716789",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8923",
"triggerID" : "aa7df7c99896831a050c515a793f55cf5c716789",
"triggerType" : "PUSH"
}, {
"hash" : "36161bf2dea866c5aecc745af9109e66b2af0726",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8926",
"triggerID" : "36161bf2dea866c5aecc745af9109e66b2af0726",
"triggerType" : "PUSH"
}, {
"hash" : "d84585bf3018a6cb9795059cfa24a8f61affa023",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d84585bf3018a6cb9795059cfa24a8f61affa023",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 36161bf2dea866c5aecc745af9109e66b2af0726 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8926)
* d84585bf3018a6cb9795059cfa24a8f61affa023 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5675: [HUDI-4151] flink split_reader supports rocksdb
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5675:
URL: https://github.com/apache/hudi/pull/5675#issuecomment-1137195688
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e8a278ecd4bbd33e2cc5fb3582a41c3edd90377f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8908",
"triggerID" : "e8a278ecd4bbd33e2cc5fb3582a41c3edd90377f",
"triggerType" : "PUSH"
}, {
"hash" : "7f38c09425e02aab5da82a122a955272b1792f71",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8919",
"triggerID" : "7f38c09425e02aab5da82a122a955272b1792f71",
"triggerType" : "PUSH"
}, {
"hash" : "aa7df7c99896831a050c515a793f55cf5c716789",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8923",
"triggerID" : "aa7df7c99896831a050c515a793f55cf5c716789",
"triggerType" : "PUSH"
}, {
"hash" : "36161bf2dea866c5aecc745af9109e66b2af0726",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "36161bf2dea866c5aecc745af9109e66b2af0726",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aa7df7c99896831a050c515a793f55cf5c716789 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8923)
* 36161bf2dea866c5aecc745af9109e66b2af0726 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5675: [HUDI-4151] flink split_reader supports rocksdb
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5675:
URL: https://github.com/apache/hudi/pull/5675#issuecomment-1137090443
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e8a278ecd4bbd33e2cc5fb3582a41c3edd90377f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8908",
"triggerID" : "e8a278ecd4bbd33e2cc5fb3582a41c3edd90377f",
"triggerType" : "PUSH"
}, {
"hash" : "7f38c09425e02aab5da82a122a955272b1792f71",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8919",
"triggerID" : "7f38c09425e02aab5da82a122a955272b1792f71",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7f38c09425e02aab5da82a122a955272b1792f71 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8919)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5675: [HUDI-4151] flink split_reader supports rocksdb
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5675:
URL: https://github.com/apache/hudi/pull/5675#issuecomment-1139468160
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e8a278ecd4bbd33e2cc5fb3582a41c3edd90377f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8908",
"triggerID" : "e8a278ecd4bbd33e2cc5fb3582a41c3edd90377f",
"triggerType" : "PUSH"
}, {
"hash" : "7f38c09425e02aab5da82a122a955272b1792f71",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8919",
"triggerID" : "7f38c09425e02aab5da82a122a955272b1792f71",
"triggerType" : "PUSH"
}, {
"hash" : "aa7df7c99896831a050c515a793f55cf5c716789",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8923",
"triggerID" : "aa7df7c99896831a050c515a793f55cf5c716789",
"triggerType" : "PUSH"
}, {
"hash" : "36161bf2dea866c5aecc745af9109e66b2af0726",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8926",
"triggerID" : "36161bf2dea866c5aecc745af9109e66b2af0726",
"triggerType" : "PUSH"
}, {
"hash" : "d84585bf3018a6cb9795059cfa24a8f61affa023",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8933",
"triggerID" : "d84585bf3018a6cb9795059cfa24a8f61affa023",
"triggerType" : "PUSH"
}, {
"hash" : "498df0d128c789e746241d7d2e0fce930f4b0dc4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8967",
"triggerID" : "498df0d128c789e746241d7d2e0fce930f4b0dc4",
"triggerType" : "PUSH"
}, {
"hash" : "89ce5aae7139def8393ccd20dac2c07ea868d265",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "89ce5aae7139def8393ccd20dac2c07ea868d265",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 498df0d128c789e746241d7d2e0fce930f4b0dc4 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8967)
* 89ce5aae7139def8393ccd20dac2c07ea868d265 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5675: [HUDI-4151] flink split_reader supports rocksdb
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5675:
URL: https://github.com/apache/hudi/pull/5675#issuecomment-1137413084
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e8a278ecd4bbd33e2cc5fb3582a41c3edd90377f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8908",
"triggerID" : "e8a278ecd4bbd33e2cc5fb3582a41c3edd90377f",
"triggerType" : "PUSH"
}, {
"hash" : "7f38c09425e02aab5da82a122a955272b1792f71",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8919",
"triggerID" : "7f38c09425e02aab5da82a122a955272b1792f71",
"triggerType" : "PUSH"
}, {
"hash" : "aa7df7c99896831a050c515a793f55cf5c716789",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8923",
"triggerID" : "aa7df7c99896831a050c515a793f55cf5c716789",
"triggerType" : "PUSH"
}, {
"hash" : "36161bf2dea866c5aecc745af9109e66b2af0726",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8926",
"triggerID" : "36161bf2dea866c5aecc745af9109e66b2af0726",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 36161bf2dea866c5aecc745af9109e66b2af0726 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8926)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5675: [HUDI-4151] flink split_reader supports rocksdb
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5675:
URL: https://github.com/apache/hudi/pull/5675#issuecomment-1137144711
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e8a278ecd4bbd33e2cc5fb3582a41c3edd90377f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8908",
"triggerID" : "e8a278ecd4bbd33e2cc5fb3582a41c3edd90377f",
"triggerType" : "PUSH"
}, {
"hash" : "7f38c09425e02aab5da82a122a955272b1792f71",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8919",
"triggerID" : "7f38c09425e02aab5da82a122a955272b1792f71",
"triggerType" : "PUSH"
}, {
"hash" : "aa7df7c99896831a050c515a793f55cf5c716789",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aa7df7c99896831a050c515a793f55cf5c716789",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7f38c09425e02aab5da82a122a955272b1792f71 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8919)
* aa7df7c99896831a050c515a793f55cf5c716789 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5675: [HUDI-4151] flink split_reader supports rocksdb
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5675:
URL: https://github.com/apache/hudi/pull/5675#issuecomment-1136908195
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e8a278ecd4bbd33e2cc5fb3582a41c3edd90377f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8908",
"triggerID" : "e8a278ecd4bbd33e2cc5fb3582a41c3edd90377f",
"triggerType" : "PUSH"
}, {
"hash" : "7f38c09425e02aab5da82a122a955272b1792f71",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7f38c09425e02aab5da82a122a955272b1792f71",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e8a278ecd4bbd33e2cc5fb3582a41c3edd90377f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8908)
* 7f38c09425e02aab5da82a122a955272b1792f71 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #5675: [HUDI-4151] flink split_reader supports rocksdb
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #5675:
URL: https://github.com/apache/hudi/pull/5675#discussion_r881433551
##########
hudi-common/src/main/java/org/apache/hudi/common/util/collection/RocksDbDiskMap.java:
##########
@@ -41,21 +42,25 @@
* All of the data is stored using the RocksDB implementation.
*/
public final class RocksDbDiskMap<T extends Serializable, R extends Serializable> extends DiskMap<T, R> {
+ private static final Logger LOG = LogManager.getLogger(RocksDbDiskMap.class);
+
// ColumnFamily allows partitioning data within RockDB, which allows
// independent configuration and faster deletes across partitions
// https://github.com/facebook/rocksdb/wiki/Column-Families
// For this use case, we use a single static column family/ partition
//
- private static final String ROCKSDB_COL_FAMILY = "rocksdb-diskmap";
+ private static final String ROCKSDB_BASE_PATH = "rocksdb-diskmap";
- private static final Logger LOG = LogManager.getLogger(RocksDbDiskMap.class);
// Stores the key and corresponding value's latest metadata spilled to disk
private final Set<T> keySet;
- private RocksDBDAO rocksDb;
+ private static RocksDBDAO rocksDb = null;
+ private String rocksdbColFamily;
public RocksDbDiskMap(String rocksDbStoragePath) throws IOException {
super(rocksDbStoragePath, ExternalSpillableMap.DiskMapType.ROCKS_DB.name());
this.keySet = new HashSet<>();
Review Comment:
The `RocksDbDiskMap` was designed to be in the thread scope, it is like the `BitCaskMap` each reader iterator holds a map instance with a random base path, see `DiskMap.diskMapPath`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] cuibo01 commented on a diff in pull request #5675: [HUDI-4151] flink split_reader supports rocksdb
Posted by GitBox <gi...@apache.org>.
cuibo01 commented on code in PR #5675:
URL: https://github.com/apache/hudi/pull/5675#discussion_r881434882
##########
hudi-common/src/main/java/org/apache/hudi/common/util/collection/RocksDbDiskMap.java:
##########
@@ -41,21 +42,25 @@
* All of the data is stored using the RocksDB implementation.
*/
public final class RocksDbDiskMap<T extends Serializable, R extends Serializable> extends DiskMap<T, R> {
+ private static final Logger LOG = LogManager.getLogger(RocksDbDiskMap.class);
+
// ColumnFamily allows partitioning data within RockDB, which allows
// independent configuration and faster deletes across partitions
// https://github.com/facebook/rocksdb/wiki/Column-Families
// For this use case, we use a single static column family/ partition
//
- private static final String ROCKSDB_COL_FAMILY = "rocksdb-diskmap";
+ private static final String ROCKSDB_BASE_PATH = "rocksdb-diskmap";
- private static final Logger LOG = LogManager.getLogger(RocksDbDiskMap.class);
// Stores the key and corresponding value's latest metadata spilled to disk
private final Set<T> keySet;
- private RocksDBDAO rocksDb;
+ private static RocksDBDAO rocksDb = null;
+ private String rocksdbColFamily;
public RocksDbDiskMap(String rocksDbStoragePath) throws IOException {
super(rocksDbStoragePath, ExternalSpillableMap.DiskMapType.ROCKS_DB.name());
this.keySet = new HashSet<>();
Review Comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #5675: [HUDI-4151] flink split_reader supports rocksdb
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #5675:
URL: https://github.com/apache/hudi/pull/5675#discussion_r881425275
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/format/FormatUtils.java:
##########
@@ -120,9 +123,12 @@ private static Object getVal(IndexedRecord record, int pos) {
public static HoodieMergedLogRecordScanner logScanner(
MergeOnReadInputSplit split,
Schema logSchema,
- Configuration config,
- boolean withOperationField) {
Review Comment:
We can pass in `writeConfig` instead of flink configuration instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] cuibo01 commented on a diff in pull request #5675: [HUDI-4151] flink split_reader supports rocksdb
Posted by GitBox <gi...@apache.org>.
cuibo01 commented on code in PR #5675:
URL: https://github.com/apache/hudi/pull/5675#discussion_r883366643
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/format/FormatUtils.java:
##########
@@ -120,9 +123,12 @@ private static Object getVal(IndexedRecord record, int pos) {
public static HoodieMergedLogRecordScanner logScanner(
MergeOnReadInputSplit split,
Schema logSchema,
- Configuration config,
- boolean withOperationField) {
Review Comment:
But I think it's better to pass `writeConfig` on, because we can reuse MergeOnReadInputFormat in the class, avoiding repeated creations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] cuibo01 commented on a diff in pull request #5675: [HUDI-4151] flink split_reader supports rocksdb
Posted by GitBox <gi...@apache.org>.
cuibo01 commented on code in PR #5675:
URL: https://github.com/apache/hudi/pull/5675#discussion_r883371699
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/format/FormatUtils.java:
##########
@@ -120,9 +123,12 @@ private static Object getVal(IndexedRecord record, int pos) {
public static HoodieMergedLogRecordScanner logScanner(
MergeOnReadInputSplit split,
Schema logSchema,
- Configuration config,
- boolean withOperationField) {
Review Comment:
we can move `this.hadoopConf = HadoopConfigurations.getHadoopConf(this.conf); this.writeConfig = StreamerUtil.getHoodieClientConfig(this.conf);` from MergeOnReadInputFormat#open to MergeOnReadInputFormat#construction
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 merged pull request #5675: [HUDI-4151] flink split_reader supports rocksdb
Posted by GitBox <gi...@apache.org>.
danny0405 merged PR #5675:
URL: https://github.com/apache/hudi/pull/5675
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] cuibo01 commented on pull request #5675: [HUDI-4151] flink split_reader supports rocksdb
Posted by GitBox <gi...@apache.org>.
cuibo01 commented on PR #5675:
URL: https://github.com/apache/hudi/pull/5675#issuecomment-1136818900
BitCaskDiskMap require a lot of memory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] cuibo01 commented on a diff in pull request #5675: [HUDI-4151] flink split_reader supports rocksdb
Posted by GitBox <gi...@apache.org>.
cuibo01 commented on code in PR #5675:
URL: https://github.com/apache/hudi/pull/5675#discussion_r881408190
##########
hudi-common/src/main/java/org/apache/hudi/common/util/collection/RocksDbDiskMap.java:
##########
@@ -41,21 +42,25 @@
* All of the data is stored using the RocksDB implementation.
*/
public final class RocksDbDiskMap<T extends Serializable, R extends Serializable> extends DiskMap<T, R> {
+ private static final Logger LOG = LogManager.getLogger(RocksDbDiskMap.class);
+
// ColumnFamily allows partitioning data within RockDB, which allows
// independent configuration and faster deletes across partitions
// https://github.com/facebook/rocksdb/wiki/Column-Families
// For this use case, we use a single static column family/ partition
//
- private static final String ROCKSDB_COL_FAMILY = "rocksdb-diskmap";
+ private static final String ROCKSDB_BASE_PATH = "rocksdb-diskmap";
- private static final Logger LOG = LogManager.getLogger(RocksDbDiskMap.class);
// Stores the key and corresponding value's latest metadata spilled to disk
private final Set<T> keySet;
- private RocksDBDAO rocksDb;
+ private static RocksDBDAO rocksDb = null;
+ private String rocksdbColFamily;
public RocksDbDiskMap(String rocksDbStoragePath) throws IOException {
super(rocksDbStoragePath, ExternalSpillableMap.DiskMapType.ROCKS_DB.name());
this.keySet = new HashSet<>();
Review Comment:
A TM has only one rocksdb, and all logscanner use the same rocksdb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] cuibo01 commented on a diff in pull request #5675: [HUDI-4151] flink split_reader supports rocksdb
Posted by GitBox <gi...@apache.org>.
cuibo01 commented on code in PR #5675:
URL: https://github.com/apache/hudi/pull/5675#discussion_r881406121
##########
hudi-common/src/main/java/org/apache/hudi/common/util/collection/RocksDbDiskMap.java:
##########
@@ -41,21 +42,25 @@
* All of the data is stored using the RocksDB implementation.
*/
public final class RocksDbDiskMap<T extends Serializable, R extends Serializable> extends DiskMap<T, R> {
+ private static final Logger LOG = LogManager.getLogger(RocksDbDiskMap.class);
+
// ColumnFamily allows partitioning data within RockDB, which allows
// independent configuration and faster deletes across partitions
// https://github.com/facebook/rocksdb/wiki/Column-Families
// For this use case, we use a single static column family/ partition
//
- private static final String ROCKSDB_COL_FAMILY = "rocksdb-diskmap";
+ private static final String ROCKSDB_BASE_PATH = "rocksdb-diskmap";
- private static final Logger LOG = LogManager.getLogger(RocksDbDiskMap.class);
// Stores the key and corresponding value's latest metadata spilled to disk
private final Set<T> keySet;
- private RocksDBDAO rocksDb;
+ private static RocksDBDAO rocksDb = null;
+ private String rocksdbColFamily;
public RocksDbDiskMap(String rocksDbStoragePath) throws IOException {
super(rocksDbStoragePath, ExternalSpillableMap.DiskMapType.ROCKS_DB.name());
this.keySet = new HashSet<>();
Review Comment:
one TM one Rocksdb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] cuibo01 commented on a diff in pull request #5675: [HUDI-4151] flink split_reader supports rocksdb
Posted by GitBox <gi...@apache.org>.
cuibo01 commented on code in PR #5675:
URL: https://github.com/apache/hudi/pull/5675#discussion_r881404844
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/format/FormatUtils.java:
##########
@@ -241,9 +250,9 @@ public static HoodieMergedLogRecordScanner logScanner(
.withReadBlocksLazily(writeConfig.getCompactionLazyBlockReadEnabled())
.withReverseReader(false)
.withBufferSize(writeConfig.getMaxDFSStreamBufferSize())
- .withMaxMemorySizeInBytes(writeConfig.getMaxMemoryPerPartitionMerge())
+ .withMaxMemorySizeInBytes(diskMapType == ExternalSpillableMap.DiskMapType.ROCKS_DB ? 0 : writeConfig.getMaxMemoryPerPartitionMerge())
.withSpillableMapBasePath(writeConfig.getSpillableMapBasePath())
Review Comment:
RocksDB has its own memory(default 64MB)？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] cuibo01 commented on a diff in pull request #5675: [HUDI-4151] flink split_reader supports rocksdb
Posted by GitBox <gi...@apache.org>.
cuibo01 commented on code in PR #5675:
URL: https://github.com/apache/hudi/pull/5675#discussion_r881425634
##########
hudi-common/src/main/java/org/apache/hudi/common/util/collection/RocksDbDiskMap.java:
##########
@@ -41,21 +42,25 @@
* All of the data is stored using the RocksDB implementation.
*/
public final class RocksDbDiskMap<T extends Serializable, R extends Serializable> extends DiskMap<T, R> {
+ private static final Logger LOG = LogManager.getLogger(RocksDbDiskMap.class);
+
// ColumnFamily allows partitioning data within RockDB, which allows
// independent configuration and faster deletes across partitions
// https://github.com/facebook/rocksdb/wiki/Column-Families
// For this use case, we use a single static column family/ partition
//
- private static final String ROCKSDB_COL_FAMILY = "rocksdb-diskmap";
+ private static final String ROCKSDB_BASE_PATH = "rocksdb-diskmap";
- private static final Logger LOG = LogManager.getLogger(RocksDbDiskMap.class);
// Stores the key and corresponding value's latest metadata spilled to disk
private final Set<T> keySet;
- private RocksDBDAO rocksDb;
+ private static RocksDBDAO rocksDb = null;
+ private String rocksdbColFamily;
public RocksDbDiskMap(String rocksDbStoragePath) throws IOException {
super(rocksDbStoragePath, ExternalSpillableMap.DiskMapType.ROCKS_DB.name());
this.keySet = new HashSet<>();
Review Comment:
no, avoid creating rocksdb repeatedly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5675: [HUDI-4151] flink split_reader supports rocksdb
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5675:
URL: https://github.com/apache/hudi/pull/5675#issuecomment-1139465361
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e8a278ecd4bbd33e2cc5fb3582a41c3edd90377f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8908",
"triggerID" : "e8a278ecd4bbd33e2cc5fb3582a41c3edd90377f",
"triggerType" : "PUSH"
}, {
"hash" : "7f38c09425e02aab5da82a122a955272b1792f71",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8919",
"triggerID" : "7f38c09425e02aab5da82a122a955272b1792f71",
"triggerType" : "PUSH"
}, {
"hash" : "aa7df7c99896831a050c515a793f55cf5c716789",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8923",
"triggerID" : "aa7df7c99896831a050c515a793f55cf5c716789",
"triggerType" : "PUSH"
}, {
"hash" : "36161bf2dea866c5aecc745af9109e66b2af0726",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8926",
"triggerID" : "36161bf2dea866c5aecc745af9109e66b2af0726",
"triggerType" : "PUSH"
}, {
"hash" : "d84585bf3018a6cb9795059cfa24a8f61affa023",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8933",
"triggerID" : "d84585bf3018a6cb9795059cfa24a8f61affa023",
"triggerType" : "PUSH"
}, {
"hash" : "498df0d128c789e746241d7d2e0fce930f4b0dc4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8967",
"triggerID" : "498df0d128c789e746241d7d2e0fce930f4b0dc4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 498df0d128c789e746241d7d2e0fce930f4b0dc4 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8967)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5675: [HUDI-4151] flink split_reader supports rocksdb
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5675:
URL: https://github.com/apache/hudi/pull/5675#issuecomment-1136665826
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e8a278ecd4bbd33e2cc5fb3582a41c3edd90377f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e8a278ecd4bbd33e2cc5fb3582a41c3edd90377f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e8a278ecd4bbd33e2cc5fb3582a41c3edd90377f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5675: [HUDI-4151] flink split_reader supports rocksdb
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5675:
URL: https://github.com/apache/hudi/pull/5675#issuecomment-1136911682
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e8a278ecd4bbd33e2cc5fb3582a41c3edd90377f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8908",
"triggerID" : "e8a278ecd4bbd33e2cc5fb3582a41c3edd90377f",
"triggerType" : "PUSH"
}, {
"hash" : "7f38c09425e02aab5da82a122a955272b1792f71",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8919",
"triggerID" : "7f38c09425e02aab5da82a122a955272b1792f71",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e8a278ecd4bbd33e2cc5fb3582a41c3edd90377f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8908)
* 7f38c09425e02aab5da82a122a955272b1792f71 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8919)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #5675: [HUDI-4151] flink split_reader supports rocksdb
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #5675:
URL: https://github.com/apache/hudi/pull/5675#discussion_r881401621
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/format/FormatUtils.java:
##########
@@ -241,9 +250,9 @@ public static HoodieMergedLogRecordScanner logScanner(
.withReadBlocksLazily(writeConfig.getCompactionLazyBlockReadEnabled())
.withReverseReader(false)
.withBufferSize(writeConfig.getMaxDFSStreamBufferSize())
- .withMaxMemorySizeInBytes(writeConfig.getMaxMemoryPerPartitionMerge())
+ .withMaxMemorySizeInBytes(diskMapType == ExternalSpillableMap.DiskMapType.ROCKS_DB ? 0 : writeConfig.getMaxMemoryPerPartitionMerge())
.withSpillableMapBasePath(writeConfig.getSpillableMapBasePath())
Review Comment:
RocksDB also needs memory as cache.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5675: [HUDI-4151] flink split_reader supports rocksdb
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5675:
URL: https://github.com/apache/hudi/pull/5675#issuecomment-1137147846
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e8a278ecd4bbd33e2cc5fb3582a41c3edd90377f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8908",
"triggerID" : "e8a278ecd4bbd33e2cc5fb3582a41c3edd90377f",
"triggerType" : "PUSH"
}, {
"hash" : "7f38c09425e02aab5da82a122a955272b1792f71",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8919",
"triggerID" : "7f38c09425e02aab5da82a122a955272b1792f71",
"triggerType" : "PUSH"
}, {
"hash" : "aa7df7c99896831a050c515a793f55cf5c716789",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8923",
"triggerID" : "aa7df7c99896831a050c515a793f55cf5c716789",
"triggerType" : "PUSH"
}, {
"hash" : "36161bf2dea866c5aecc745af9109e66b2af0726",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "36161bf2dea866c5aecc745af9109e66b2af0726",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7f38c09425e02aab5da82a122a955272b1792f71 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8919)
* aa7df7c99896831a050c515a793f55cf5c716789 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8923)
* 36161bf2dea866c5aecc745af9109e66b2af0726 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5675: [HUDI-4151] flink split_reader supports rocksdb
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5675:
URL: https://github.com/apache/hudi/pull/5675#issuecomment-1137206129
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e8a278ecd4bbd33e2cc5fb3582a41c3edd90377f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8908",
"triggerID" : "e8a278ecd4bbd33e2cc5fb3582a41c3edd90377f",
"triggerType" : "PUSH"
}, {
"hash" : "7f38c09425e02aab5da82a122a955272b1792f71",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8919",
"triggerID" : "7f38c09425e02aab5da82a122a955272b1792f71",
"triggerType" : "PUSH"
}, {
"hash" : "aa7df7c99896831a050c515a793f55cf5c716789",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8923",
"triggerID" : "aa7df7c99896831a050c515a793f55cf5c716789",
"triggerType" : "PUSH"
}, {
"hash" : "36161bf2dea866c5aecc745af9109e66b2af0726",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8926",
"triggerID" : "36161bf2dea866c5aecc745af9109e66b2af0726",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aa7df7c99896831a050c515a793f55cf5c716789 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8923)
* 36161bf2dea866c5aecc745af9109e66b2af0726 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8926)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] cuibo01 commented on a diff in pull request #5675: [HUDI-4151] flink split_reader supports rocksdb
Posted by GitBox <gi...@apache.org>.
cuibo01 commented on code in PR #5675:
URL: https://github.com/apache/hudi/pull/5675#discussion_r881654513
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/format/FormatUtils.java:
##########
@@ -241,9 +250,9 @@ public static HoodieMergedLogRecordScanner logScanner(
.withReadBlocksLazily(writeConfig.getCompactionLazyBlockReadEnabled())
.withReverseReader(false)
.withBufferSize(writeConfig.getMaxDFSStreamBufferSize())
- .withMaxMemorySizeInBytes(writeConfig.getMaxMemoryPerPartitionMerge())
+ .withMaxMemorySizeInBytes(diskMapType == ExternalSpillableMap.DiskMapType.ROCKS_DB ? 0 : writeConfig.getMaxMemoryPerPartitionMerge())
.withSpillableMapBasePath(writeConfig.getSpillableMapBasePath())
Review Comment:
WDYT? in our production env, memory cache has not been disabled yet, but the rocksdb has its own memory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5675: [HUDI-4151] flink split_reader supports rocksdb
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5675:
URL: https://github.com/apache/hudi/pull/5675#issuecomment-1138056176
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e8a278ecd4bbd33e2cc5fb3582a41c3edd90377f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8908",
"triggerID" : "e8a278ecd4bbd33e2cc5fb3582a41c3edd90377f",
"triggerType" : "PUSH"
}, {
"hash" : "7f38c09425e02aab5da82a122a955272b1792f71",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8919",
"triggerID" : "7f38c09425e02aab5da82a122a955272b1792f71",
"triggerType" : "PUSH"
}, {
"hash" : "aa7df7c99896831a050c515a793f55cf5c716789",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8923",
"triggerID" : "aa7df7c99896831a050c515a793f55cf5c716789",
"triggerType" : "PUSH"
}, {
"hash" : "36161bf2dea866c5aecc745af9109e66b2af0726",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8926",
"triggerID" : "36161bf2dea866c5aecc745af9109e66b2af0726",
"triggerType" : "PUSH"
}, {
"hash" : "d84585bf3018a6cb9795059cfa24a8f61affa023",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8933",
"triggerID" : "d84585bf3018a6cb9795059cfa24a8f61affa023",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 36161bf2dea866c5aecc745af9109e66b2af0726 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8926)
* d84585bf3018a6cb9795059cfa24a8f61affa023 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8933)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5675: [HUDI-4151] flink split_reader supports rocksdb
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5675:
URL: https://github.com/apache/hudi/pull/5675#issuecomment-1138089408
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e8a278ecd4bbd33e2cc5fb3582a41c3edd90377f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8908",
"triggerID" : "e8a278ecd4bbd33e2cc5fb3582a41c3edd90377f",
"triggerType" : "PUSH"
}, {
"hash" : "7f38c09425e02aab5da82a122a955272b1792f71",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8919",
"triggerID" : "7f38c09425e02aab5da82a122a955272b1792f71",
"triggerType" : "PUSH"
}, {
"hash" : "aa7df7c99896831a050c515a793f55cf5c716789",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8923",
"triggerID" : "aa7df7c99896831a050c515a793f55cf5c716789",
"triggerType" : "PUSH"
}, {
"hash" : "36161bf2dea866c5aecc745af9109e66b2af0726",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8926",
"triggerID" : "36161bf2dea866c5aecc745af9109e66b2af0726",
"triggerType" : "PUSH"
}, {
"hash" : "d84585bf3018a6cb9795059cfa24a8f61affa023",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8933",
"triggerID" : "d84585bf3018a6cb9795059cfa24a8f61affa023",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d84585bf3018a6cb9795059cfa24a8f61affa023 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8933)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org