You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "TheR1sing3un (via GitHub)" <gi...@apache.org> on 2024/12/20 10:12:31 UTC
[PR] [HUDI-8787] Improve compaction performance by reducing unnecessary disk access [hudi]
TheR1sing3un opened a new pull request, #12531:
URL: https://github.com/apache/hudi/pull/12531
`HoodieMergeHandle::writeIncomingRecords` will always read all the log records from disk even thought hese log records we have previously confirmed that we do not need to insert.
<img width="807" alt="image" src="https://github.com/user-attachments/assets/08e8c5f9-8f12-4f9b-8ab8-3d0295855dfd" />
So we should push down this predicate to `ExternalSpillableMap::iterator` to avoid unnecessary disk access.
### Change Logs
1. improve compaction performance by avoid unnecessary disk visiting
2. support push down predicate to `ExternalSpillableMap`
_Describe context and summary for this change. Highlight if any code was copied._
### Impact
none
_Describe any public API or user-facing feature change or any performance impact._
### Risk level (write none, low medium or high below)
none
_If medium or high, explain what verification was done to mitigate the risks._
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none"._
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Change Logs and Impact were stated clearly
- [x] Adequate tests were added if applicable
- [x] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8787] Improve compaction performance by reducing unnecessary disk access [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12531:
URL: https://github.com/apache/hudi/pull/12531#discussion_r1900527124
##########
hudi-common/src/main/java/org/apache/hudi/common/util/collection/FilterIterator.java:
##########
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.collection;
+
+import org.apache.hudi.common.util.ValidationUtils;
+
+import java.util.Iterator;
+import java.util.function.Predicate;
+
+public class FilterIterator<R> implements Iterator<R> {
+
+ private final Iterator<R> source;
+
+ private final Predicate<R> filter;
+
+ private R current;
+
+ private boolean init;
+
+ public FilterIterator(Iterator<R> source, Predicate<R> filter) {
+ this.source = source;
+ this.filter = filter;
+ }
+
+ private void init() {
+ if (!init) {
+ while (source.hasNext()) {
+ R next = source.next();
+ if (filter.test(next)) {
+ current = next;
+ break;
+ }
+ }
+ init = true;
+ }
+ }
+
+ @Override
+ public boolean hasNext() {
+ init();
Review Comment:
Looks like the `init()` can be elimimated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8787] Improve compaction performance by reducing unnecessary disk access [hudi]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on code in PR #12531:
URL: https://github.com/apache/hudi/pull/12531#discussion_r1896382467
##########
hudi-common/src/main/java/org/apache/hudi/common/util/collection/RocksDBDAO.java:
##########
@@ -378,9 +416,10 @@ public <T extends Serializable> Stream<Pair<String, T>> prefixSearch(String colu
* Return Iterator of key-value pairs from RocksIterator.
*
* @param columnFamilyName Column Family Name
- * @param <T> Type of value stored
+ * @param <T> Type of key stored
+ * @param <R> Type of value stored
*/
- public <T extends Serializable> Iterator<T> iterator(String columnFamilyName) {
+ public <T extends Serializable, R extends Serializable> Iterator<Pair<T, R>> iterator(String columnFamilyName) {
Review Comment:
Why I refactor this?
1. This is an mistake because the method comment says to return all Key-value pairs, but only value is returned. It makes more sense to return key-value pairs here.
2. If I do not return key-value pairs, I cannot filter the keys after `RocksDbDiskMap::iterator(filter)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8787] Improve compaction performance by reducing unnecessary disk access [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12531:
URL: https://github.com/apache/hudi/pull/12531#discussion_r1900527218
##########
hudi-common/src/main/java/org/apache/hudi/common/util/collection/FilterIterator.java:
##########
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.collection;
+
+import org.apache.hudi.common.util.ValidationUtils;
+
+import java.util.Iterator;
+import java.util.function.Predicate;
+
+public class FilterIterator<R> implements Iterator<R> {
Review Comment:
Can we add some doc for it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8787] Improve compaction performance by reducing unnecessary disk access [hudi]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on code in PR #12531:
URL: https://github.com/apache/hudi/pull/12531#discussion_r1900529453
##########
hudi-common/src/main/java/org/apache/hudi/common/util/collection/RocksDBDAO.java:
##########
@@ -337,15 +332,58 @@ public <T extends Serializable> T get(String columnFamilyName, String key) {
* @param <T> Type of object stored.
*/
public <K extends Serializable, T extends Serializable> T get(String columnFamilyName, K key) {
+ return get(columnFamilyName, getKeyBytes(key));
+ }
+
+ /**
+ * Retrieve a value for a given key in a column family.
+ *
+ * @param columnFamilyName Column Family Name
+ * @param key Key to be retrieved
+ * @param <T> Type of object stored.
+ */
+ public <K extends Serializable, T extends Serializable> T get(String columnFamilyName, byte[] key) {
ValidationUtils.checkArgument(!closed);
try {
- byte[] val = getRocksDB().get(managedHandlesMap.get(columnFamilyName), SerializationUtils.serialize(key));
+ byte[] val = getRocksDB().get(managedHandlesMap.get(columnFamilyName), key);
return val == null ? null : SerializationUtils.deserialize(val);
} catch (Exception e) {
throw new HoodieException(e);
}
}
+ /**
+ * Retrieve values for the given keys in a column family.
+ *
+ * @param columnFamilyName Column Family Name
+ * @param keys Keys to be retrieved
+ * @param <T> Type of object stored.
+ */
+ public <K extends Serializable, T extends Serializable> List<T> multiGetAsList(String columnFamilyName, List<K> keys) {
Review Comment:
> Remove the unused method.
done~
##########
hudi-common/src/main/java/org/apache/hudi/common/util/collection/RocksDBDAO.java:
##########
@@ -337,15 +332,58 @@ public <T extends Serializable> T get(String columnFamilyName, String key) {
* @param <T> Type of object stored.
*/
public <K extends Serializable, T extends Serializable> T get(String columnFamilyName, K key) {
+ return get(columnFamilyName, getKeyBytes(key));
+ }
+
+ /**
+ * Retrieve a value for a given key in a column family.
+ *
+ * @param columnFamilyName Column Family Name
+ * @param key Key to be retrieved
+ * @param <T> Type of object stored.
+ */
+ public <K extends Serializable, T extends Serializable> T get(String columnFamilyName, byte[] key) {
ValidationUtils.checkArgument(!closed);
try {
- byte[] val = getRocksDB().get(managedHandlesMap.get(columnFamilyName), SerializationUtils.serialize(key));
+ byte[] val = getRocksDB().get(managedHandlesMap.get(columnFamilyName), key);
return val == null ? null : SerializationUtils.deserialize(val);
} catch (Exception e) {
throw new HoodieException(e);
}
}
+ /**
+ * Retrieve values for the given keys in a column family.
+ *
+ * @param columnFamilyName Column Family Name
+ * @param keys Keys to be retrieved
+ * @param <T> Type of object stored.
+ */
+ public <K extends Serializable, T extends Serializable> List<T> multiGetAsList(String columnFamilyName, List<K> keys) {
+ List<byte[]> byteKeys = keys.stream().map(this::getKeyBytes).collect(Collectors.toList());
+ ColumnFamilyHandle handle = managedHandlesMap.get(columnFamilyName);
+ ValidationUtils.checkArgument(handle != null, "Column Family not found :" + columnFamilyName);
+ List<ColumnFamilyHandle> columnFamilyHandles = byteKeys.stream().map(key -> handle).collect(Collectors.toList());
+ return multiGetAsList(columnFamilyHandles, byteKeys);
+ }
+
+ /**
+ * Retrieve values for the given keys in the given column families.
+ *
+ * @param columnFamilyHandles Column Family Handles
+ * @param keys Keys to be retrieved
+ * @param <T> Type of object stored.
+ */
+ private <T extends Serializable> List<T> multiGetAsList(List<ColumnFamilyHandle> columnFamilyHandles, List<byte[]> keys) {
Review Comment:
> Remove the unused method.
done~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8787] Improve compaction performance by reducing unnecessary disk access [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 merged PR #12531:
URL: https://github.com/apache/hudi/pull/12531
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8787] Improve compaction performance by reducing unnecessary disk access [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12531:
URL: https://github.com/apache/hudi/pull/12531#issuecomment-2556692214
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "14116b543e96e41e301b4c52c896797ad0a811b2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2540",
"triggerID" : "14116b543e96e41e301b4c52c896797ad0a811b2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 14116b543e96e41e301b4c52c896797ad0a811b2 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2540)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8787] Improve compaction performance by reducing unnecessary disk access [hudi]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on PR #12531:
URL: https://github.com/apache/hudi/pull/12531#issuecomment-2559000957
> @TheR1sing3un Hi, can we do some micro-benchmark for the RocksDB `multiGet` & full get with in-memory filtering.
Benchmark code:
<img width="939" alt="image" src="https://github.com/user-attachments/assets/cab00dfb-84a2-4c5a-9737-b7ec6388df19" />
For rocksdb:
For bitcask:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8787] Improve compaction performance by reducing unnecessary disk access [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12531:
URL: https://github.com/apache/hudi/pull/12531#issuecomment-2558960519
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "14116b543e96e41e301b4c52c896797ad0a811b2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2540",
"triggerID" : "14116b543e96e41e301b4c52c896797ad0a811b2",
"triggerType" : "PUSH"
}, {
"hash" : "8df408e4a71dd533ac9140b4015fc66b05b26183",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2554",
"triggerID" : "8df408e4a71dd533ac9140b4015fc66b05b26183",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8df408e4a71dd533ac9140b4015fc66b05b26183 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2554)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8787] Improve compaction performance by reducing unnecessary disk access [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12531:
URL: https://github.com/apache/hudi/pull/12531#issuecomment-2558896325
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "14116b543e96e41e301b4c52c896797ad0a811b2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2540",
"triggerID" : "14116b543e96e41e301b4c52c896797ad0a811b2",
"triggerType" : "PUSH"
}, {
"hash" : "8df408e4a71dd533ac9140b4015fc66b05b26183",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2554",
"triggerID" : "8df408e4a71dd533ac9140b4015fc66b05b26183",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 14116b543e96e41e301b4c52c896797ad0a811b2 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2540)
* 8df408e4a71dd533ac9140b4015fc66b05b26183 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2554)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8787] Improve compaction performance by reducing unnecessary disk access [hudi]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on PR #12531:
URL: https://github.com/apache/hudi/pull/12531#issuecomment-2559021636
@danny0405 For rocksdb in our benchmark, there will be a performance regression when the filtering ratio is higher than 50%, which is related to rocksdb's own optimization of iterator, and for bitcask, there will be no performance regresstion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8787] Improve compaction performance by reducing unnecessary disk access [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12531:
URL: https://github.com/apache/hudi/pull/12531#discussion_r1900526833
##########
hudi-common/src/main/java/org/apache/hudi/common/util/collection/RocksDBDAO.java:
##########
@@ -337,15 +332,58 @@ public <T extends Serializable> T get(String columnFamilyName, String key) {
* @param <T> Type of object stored.
*/
public <K extends Serializable, T extends Serializable> T get(String columnFamilyName, K key) {
+ return get(columnFamilyName, getKeyBytes(key));
+ }
+
+ /**
+ * Retrieve a value for a given key in a column family.
+ *
+ * @param columnFamilyName Column Family Name
+ * @param key Key to be retrieved
+ * @param <T> Type of object stored.
+ */
+ public <K extends Serializable, T extends Serializable> T get(String columnFamilyName, byte[] key) {
ValidationUtils.checkArgument(!closed);
try {
- byte[] val = getRocksDB().get(managedHandlesMap.get(columnFamilyName), SerializationUtils.serialize(key));
+ byte[] val = getRocksDB().get(managedHandlesMap.get(columnFamilyName), key);
return val == null ? null : SerializationUtils.deserialize(val);
} catch (Exception e) {
throw new HoodieException(e);
}
}
+ /**
+ * Retrieve values for the given keys in a column family.
+ *
+ * @param columnFamilyName Column Family Name
+ * @param keys Keys to be retrieved
+ * @param <T> Type of object stored.
+ */
+ public <K extends Serializable, T extends Serializable> List<T> multiGetAsList(String columnFamilyName, List<K> keys) {
Review Comment:
Remove the unused method.
##########
hudi-common/src/main/java/org/apache/hudi/common/util/collection/RocksDBDAO.java:
##########
@@ -337,15 +332,58 @@ public <T extends Serializable> T get(String columnFamilyName, String key) {
* @param <T> Type of object stored.
*/
public <K extends Serializable, T extends Serializable> T get(String columnFamilyName, K key) {
+ return get(columnFamilyName, getKeyBytes(key));
+ }
+
+ /**
+ * Retrieve a value for a given key in a column family.
+ *
+ * @param columnFamilyName Column Family Name
+ * @param key Key to be retrieved
+ * @param <T> Type of object stored.
+ */
+ public <K extends Serializable, T extends Serializable> T get(String columnFamilyName, byte[] key) {
ValidationUtils.checkArgument(!closed);
try {
- byte[] val = getRocksDB().get(managedHandlesMap.get(columnFamilyName), SerializationUtils.serialize(key));
+ byte[] val = getRocksDB().get(managedHandlesMap.get(columnFamilyName), key);
return val == null ? null : SerializationUtils.deserialize(val);
} catch (Exception e) {
throw new HoodieException(e);
}
}
+ /**
+ * Retrieve values for the given keys in a column family.
+ *
+ * @param columnFamilyName Column Family Name
+ * @param keys Keys to be retrieved
+ * @param <T> Type of object stored.
+ */
+ public <K extends Serializable, T extends Serializable> List<T> multiGetAsList(String columnFamilyName, List<K> keys) {
+ List<byte[]> byteKeys = keys.stream().map(this::getKeyBytes).collect(Collectors.toList());
+ ColumnFamilyHandle handle = managedHandlesMap.get(columnFamilyName);
+ ValidationUtils.checkArgument(handle != null, "Column Family not found :" + columnFamilyName);
+ List<ColumnFamilyHandle> columnFamilyHandles = byteKeys.stream().map(key -> handle).collect(Collectors.toList());
+ return multiGetAsList(columnFamilyHandles, byteKeys);
+ }
+
+ /**
+ * Retrieve values for the given keys in the given column families.
+ *
+ * @param columnFamilyHandles Column Family Handles
+ * @param keys Keys to be retrieved
+ * @param <T> Type of object stored.
+ */
+ private <T extends Serializable> List<T> multiGetAsList(List<ColumnFamilyHandle> columnFamilyHandles, List<byte[]> keys) {
Review Comment:
Remove the unused method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8787] Improve compaction performance by reducing unnecessary disk access [hudi]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on code in PR #12531:
URL: https://github.com/apache/hudi/pull/12531#discussion_r1896382467
##########
hudi-common/src/main/java/org/apache/hudi/common/util/collection/RocksDBDAO.java:
##########
@@ -378,9 +416,10 @@ public <T extends Serializable> Stream<Pair<String, T>> prefixSearch(String colu
* Return Iterator of key-value pairs from RocksIterator.
*
* @param columnFamilyName Column Family Name
- * @param <T> Type of value stored
+ * @param <T> Type of key stored
+ * @param <R> Type of value stored
*/
- public <T extends Serializable> Iterator<T> iterator(String columnFamilyName) {
+ public <T extends Serializable, R extends Serializable> Iterator<Pair<T, R>> iterator(String columnFamilyName) {
Review Comment:
Why I refactor this?
1. This looks like a mistake because the method comment says to return all Key-value pairs, but only value is returned. It makes more sense to return key-value pairs here.
2. If I do not return key-value pairs, I cannot filter the keys after `RocksDbDiskMap::iterator(filter)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8787] Improve compaction performance by reducing unnecessary disk access [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #12531:
URL: https://github.com/apache/hudi/pull/12531#issuecomment-2560575480
> @danny0405 For rocksdb in our benchmark, there will be a performance regression when the filtering ratio is higher than 50%, which is related to rocksdb's own optimization of iterator, and for bitcask, there will be no performance regresstion.
If the RocksDB has uncertainty for performance, let's keep it as it is specifically.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8787] Improve compaction performance by reducing unnecessary disk access [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12531:
URL: https://github.com/apache/hudi/pull/12531#issuecomment-2560693490
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "14116b543e96e41e301b4c52c896797ad0a811b2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2540",
"triggerID" : "14116b543e96e41e301b4c52c896797ad0a811b2",
"triggerType" : "PUSH"
}, {
"hash" : "8df408e4a71dd533ac9140b4015fc66b05b26183",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2554",
"triggerID" : "8df408e4a71dd533ac9140b4015fc66b05b26183",
"triggerType" : "PUSH"
}, {
"hash" : "39f0eac55aeb28e34b45b1159c0615bcf0931d0f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2567",
"triggerID" : "39f0eac55aeb28e34b45b1159c0615bcf0931d0f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 39f0eac55aeb28e34b45b1159c0615bcf0931d0f Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2567)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8787] Improve compaction performance by reducing unnecessary disk access [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #12531:
URL: https://github.com/apache/hudi/pull/12531#issuecomment-2558894867
@TheR1sing3un Hi, can we do some micro-benchmark for the RocksDB `multiGet` & full get with in-memory filtering.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8787] Improve compaction performance by reducing unnecessary disk access [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12531:
URL: https://github.com/apache/hudi/pull/12531#issuecomment-2567273444
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "14116b543e96e41e301b4c52c896797ad0a811b2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2540",
"triggerID" : "14116b543e96e41e301b4c52c896797ad0a811b2",
"triggerType" : "PUSH"
}, {
"hash" : "8df408e4a71dd533ac9140b4015fc66b05b26183",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2554",
"triggerID" : "8df408e4a71dd533ac9140b4015fc66b05b26183",
"triggerType" : "PUSH"
}, {
"hash" : "39f0eac55aeb28e34b45b1159c0615bcf0931d0f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2567",
"triggerID" : "39f0eac55aeb28e34b45b1159c0615bcf0931d0f",
"triggerType" : "PUSH"
}, {
"hash" : "3d72a9aac361666493096ed32da24a3350c19686",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2616",
"triggerID" : "3d72a9aac361666493096ed32da24a3350c19686",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 39f0eac55aeb28e34b45b1159c0615bcf0931d0f Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2567)
* 3d72a9aac361666493096ed32da24a3350c19686 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2616)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8787] Improve compaction performance by reducing unnecessary disk access [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12531:
URL: https://github.com/apache/hudi/pull/12531#issuecomment-2560642609
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "14116b543e96e41e301b4c52c896797ad0a811b2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2540",
"triggerID" : "14116b543e96e41e301b4c52c896797ad0a811b2",
"triggerType" : "PUSH"
}, {
"hash" : "8df408e4a71dd533ac9140b4015fc66b05b26183",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2554",
"triggerID" : "8df408e4a71dd533ac9140b4015fc66b05b26183",
"triggerType" : "PUSH"
}, {
"hash" : "39f0eac55aeb28e34b45b1159c0615bcf0931d0f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "39f0eac55aeb28e34b45b1159c0615bcf0931d0f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8df408e4a71dd533ac9140b4015fc66b05b26183 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2554)
* 39f0eac55aeb28e34b45b1159c0615bcf0931d0f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8787] Improve compaction performance by reducing unnecessary disk access [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12531:
URL: https://github.com/apache/hudi/pull/12531#issuecomment-2567272847
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "14116b543e96e41e301b4c52c896797ad0a811b2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2540",
"triggerID" : "14116b543e96e41e301b4c52c896797ad0a811b2",
"triggerType" : "PUSH"
}, {
"hash" : "8df408e4a71dd533ac9140b4015fc66b05b26183",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2554",
"triggerID" : "8df408e4a71dd533ac9140b4015fc66b05b26183",
"triggerType" : "PUSH"
}, {
"hash" : "39f0eac55aeb28e34b45b1159c0615bcf0931d0f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2567",
"triggerID" : "39f0eac55aeb28e34b45b1159c0615bcf0931d0f",
"triggerType" : "PUSH"
}, {
"hash" : "3d72a9aac361666493096ed32da24a3350c19686",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3d72a9aac361666493096ed32da24a3350c19686",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 39f0eac55aeb28e34b45b1159c0615bcf0931d0f Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2567)
* 3d72a9aac361666493096ed32da24a3350c19686 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8787] Improve compaction performance by reducing unnecessary disk access [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12531:
URL: https://github.com/apache/hudi/pull/12531#issuecomment-2558895167
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "14116b543e96e41e301b4c52c896797ad0a811b2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2540",
"triggerID" : "14116b543e96e41e301b4c52c896797ad0a811b2",
"triggerType" : "PUSH"
}, {
"hash" : "8df408e4a71dd533ac9140b4015fc66b05b26183",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8df408e4a71dd533ac9140b4015fc66b05b26183",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 14116b543e96e41e301b4c52c896797ad0a811b2 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2540)
* 8df408e4a71dd533ac9140b4015fc66b05b26183 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8787] Improve compaction performance by reducing unnecessary disk access [hudi]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on PR #12531:
URL: https://github.com/apache/hudi/pull/12531#issuecomment-2560642746
> > @danny0405 For rocksdb in our benchmark, there will be a performance regression when the filtering ratio is higher than 50%, which is related to rocksdb's own optimization of iterator, and for bitcask, there will be no performance regresstion.
>
> If the RocksDB has uncertainty for performance, let's keep it as it is specifically.
Does that mean we'll only make predicate push-down on bitcask's implementation, but will the rocksdb implementation still have the same logic?
> > @danny0405 For rocksdb in our benchmark, there will be a performance regression when the filtering ratio is higher than 50%, which is related to rocksdb's own optimization of iterator, and for bitcask, there will be no performance regresstion.
>
> If the RocksDB has uncertainty for performance, let's keep it as it is specifically.
Nice suggestion! I still keep `RocksDbDiskMap::iterator(filter)` method but not push down predicate before iterating all key-values, just iterate all key-values and then filter by key.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8787] Improve compaction performance by reducing unnecessary disk access [hudi]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on PR #12531:
URL: https://github.com/apache/hudi/pull/12531#issuecomment-2564993089
@danny0405 Hi,danny! I have updated the code based on your suggestion, could u have a review again? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8787] Improve compaction performance by reducing unnecessary disk access [hudi]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on code in PR #12531:
URL: https://github.com/apache/hudi/pull/12531#discussion_r1900529546
##########
hudi-common/src/main/java/org/apache/hudi/common/util/collection/FilterIterator.java:
##########
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.collection;
+
+import org.apache.hudi.common.util.ValidationUtils;
+
+import java.util.Iterator;
+import java.util.function.Predicate;
+
+public class FilterIterator<R> implements Iterator<R> {
+
+ private final Iterator<R> source;
+
+ private final Predicate<R> filter;
+
+ private R current;
+
+ private boolean init;
+
+ public FilterIterator(Iterator<R> source, Predicate<R> filter) {
+ this.source = source;
+ this.filter = filter;
+ }
+
+ private void init() {
+ if (!init) {
+ while (source.hasNext()) {
+ R next = source.next();
+ if (filter.test(next)) {
+ current = next;
+ break;
+ }
+ }
+ init = true;
+ }
+ }
+
+ @Override
+ public boolean hasNext() {
+ init();
Review Comment:
> Looks like the `init()` can be elimimated.
Nice catch~ Changed.
##########
hudi-common/src/main/java/org/apache/hudi/common/util/collection/FilterIterator.java:
##########
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.collection;
+
+import org.apache.hudi.common.util.ValidationUtils;
+
+import java.util.Iterator;
+import java.util.function.Predicate;
+
+public class FilterIterator<R> implements Iterator<R> {
Review Comment:
> Can we add some doc for it.
Done~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8787] Improve compaction performance by reducing unnecessary disk access [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12531:
URL: https://github.com/apache/hudi/pull/12531#issuecomment-2567306208
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "14116b543e96e41e301b4c52c896797ad0a811b2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2540",
"triggerID" : "14116b543e96e41e301b4c52c896797ad0a811b2",
"triggerType" : "PUSH"
}, {
"hash" : "8df408e4a71dd533ac9140b4015fc66b05b26183",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2554",
"triggerID" : "8df408e4a71dd533ac9140b4015fc66b05b26183",
"triggerType" : "PUSH"
}, {
"hash" : "39f0eac55aeb28e34b45b1159c0615bcf0931d0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2567",
"triggerID" : "39f0eac55aeb28e34b45b1159c0615bcf0931d0f",
"triggerType" : "PUSH"
}, {
"hash" : "3d72a9aac361666493096ed32da24a3350c19686",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2616",
"triggerID" : "3d72a9aac361666493096ed32da24a3350c19686",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3d72a9aac361666493096ed32da24a3350c19686 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2616)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8787] Improve compaction performance by reducing unnecessary disk access [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12531:
URL: https://github.com/apache/hudi/pull/12531#issuecomment-2556689565
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "14116b543e96e41e301b4c52c896797ad0a811b2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "14116b543e96e41e301b4c52c896797ad0a811b2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 14116b543e96e41e301b4c52c896797ad0a811b2 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8787] Improve compaction performance by reducing unnecessary disk access [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12531:
URL: https://github.com/apache/hudi/pull/12531#issuecomment-2556829994
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "14116b543e96e41e301b4c52c896797ad0a811b2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2540",
"triggerID" : "14116b543e96e41e301b4c52c896797ad0a811b2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 14116b543e96e41e301b4c52c896797ad0a811b2 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2540)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8787] Improve compaction performance by reducing unnecessary disk access [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12531:
URL: https://github.com/apache/hudi/pull/12531#issuecomment-2560643455
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "14116b543e96e41e301b4c52c896797ad0a811b2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2540",
"triggerID" : "14116b543e96e41e301b4c52c896797ad0a811b2",
"triggerType" : "PUSH"
}, {
"hash" : "8df408e4a71dd533ac9140b4015fc66b05b26183",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2554",
"triggerID" : "8df408e4a71dd533ac9140b4015fc66b05b26183",
"triggerType" : "PUSH"
}, {
"hash" : "39f0eac55aeb28e34b45b1159c0615bcf0931d0f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2567",
"triggerID" : "39f0eac55aeb28e34b45b1159c0615bcf0931d0f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8df408e4a71dd533ac9140b4015fc66b05b26183 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2554)
* 39f0eac55aeb28e34b45b1159c0615bcf0931d0f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2567)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org