You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/07/01 09:39:34 UTC
[GitHub] [hudi] xiarixiaoyao opened a new pull request #3203: [HUDI-2086] redo the logical of mor_incremental_view for hive
xiarixiaoyao opened a new pull request #3203:
URL: https://github.com/apache/hudi/pull/3203
## *Tips*
- *Thank you very much for contributing to Apache Hudi.*
- *Please review https://hudi.apache.org/contributing.html before opening a pull request.*
## What is the purpose of the pull request
redo the logical of mor_incremental_view for hive to fix some bugs for mor_incremental_view for hive/sparksql
purpose of the pull request:
1) support read the lastest incremental datas which are stored by logs
2) support read incremental datas which before replacecommit
3) support read file groups which has only logs
4) keep the logical of mor_incremental_view as the same logicl as spark dataSource
## Brief change log
*(for example:)*
- *Modify AnnotationLocation checkstyle rule in checkstyle.xml*
## Verify this pull request
new UT added
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-940624808
@codope yes, it will be ok after compact.
but consider that case: read incremental datas which before replacecommit, origin read logical cannot resolve this problem.
this pr also solve some hidden bugs in our product env。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r725752324
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/RealtimeFileStatus.java
##########
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+public class RealtimeFileStatus extends FileStatus {
Review comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-939384798
@vinothchandar @bvaradar : Do you think one of you folks need to review this PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r725746415
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/RealtimeCompactedRecordReader.java
##########
@@ -102,97 +102,107 @@ private HoodieMergedLogRecordScanner getMergedLogRecordScanner() throws IOExcept
@Override
public boolean next(NullWritable aVoid, ArrayWritable arrayWritable) throws IOException {
+ // deal with DeltaOnlySplits
+ if (logReader.isPresent()) {
+ return logReader.get().next(aVoid, arrayWritable);
+ }
// Call the underlying parquetReader.next - which may replace the passed in ArrayWritable
// with a new block of values
- while (this.parquetReader.next(aVoid, arrayWritable)) {
- if (!deltaRecordMap.isEmpty()) {
- String key = arrayWritable.get()[recordKeyIndex].toString();
- if (deltaRecordMap.containsKey(key)) {
- // mark the key as handled
- this.deltaRecordKeys.remove(key);
- // TODO(NA): Invoke preCombine here by converting arrayWritable to Avro. This is required since the
- // deltaRecord may not be a full record and needs values of columns from the parquet
- Option<GenericRecord> rec = buildGenericRecordwithCustomPayload(deltaRecordMap.get(key));
- // If the record is not present, this is a delete record using an empty payload so skip this base record
- // and move to the next record
- if (!rec.isPresent()) {
- continue;
+ boolean result = this.parquetReader.next(aVoid, arrayWritable);
+ if (!result) {
+ // if the result is false, then there are no more records
+ return false;
+ }
+ if (!deltaRecordMap.isEmpty()) {
+ // TODO(VC): Right now, we assume all records in log, have a matching base record. (which
+ // would be true until we have a way to index logs too)
Review comment:
I need to clarify two points on this issue。
first： those code is produced by revert [HUDI-1969] Support reading logs for MOR Hive rt table (#3033) and those code are original hudi kernal code。
second： At present, we cannot distinguish between the newly added data and the updated data in a log file， so hbase index will not be the case。 HUDI-1969 try to solve this problem but introduce a new bug。 see https://github.com/apache/hudi/pull/3203#issuecomment-927820027
@nsivabalan what's your suggest about hbase index， maybe we cannot handle this case util we have a way to index logs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r736332305
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieRealtimeInputFormatUtils.java
##########
@@ -161,6 +162,46 @@
return rtSplits.toArray(new InputSplit[0]);
}
+ // pick all incremental files and add them to rtSplits then filter out those files.
+ private static Map<Path, List<FileSplit>> filterOutIncrementalSplits(
+ List<FileSplit> fileSplitList,
Review comment:
good, let me try it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-961745735
@danny0405 ok, i have already join in the dingding group. my name is mengtao thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
}, {
"hash" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325",
"triggerID" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"triggerType" : "PUSH"
}, {
"hash" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2343",
"triggerID" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"triggerType" : "PUSH"
}, {
"hash" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2424",
"triggerID" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2427",
"triggerID" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2429",
"triggerID" : "927749497",
"triggerType" : "MANUAL"
}, {
"hash" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2431",
"triggerID" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"triggerType" : "PUSH"
}, {
"hash" : "29e6acb65044597e35f9d499385dad940580d9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2432",
"triggerID" : "29e6acb65044597e35f9d499385dad940580d9df",
"triggerType" : "PUSH"
}, {
"hash" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2441",
"triggerID" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"triggerType" : "PUSH"
}, {
"hash" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2443",
"triggerID" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2447",
"triggerID" : "928890183",
"triggerType" : "MANUAL"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2470",
"triggerID" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"triggerType" : "PUSH"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2560",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562",
"triggerID" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"triggerType" : "PUSH"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
}, {
"hash" : "6fa031e1dcf6e21dfb24cf4296cb00d910b0a204",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2589",
"triggerID" : "6fa031e1dcf6e21dfb24cf4296cb00d910b0a204",
"triggerType" : "PUSH"
}, {
"hash" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2593",
"triggerID" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"triggerType" : "PUSH"
}, {
"hash" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2604",
"triggerID" : "941865271",
"triggerType" : "MANUAL"
}, {
"hash" : "6d67b68e19a43f8668e5773d27ca9c33a8de0a37",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2642",
"triggerID" : "6d67b68e19a43f8668e5773d27ca9c33a8de0a37",
"triggerType" : "PUSH"
}, {
"hash" : "1a80559bd98829552acffdaf20d3ea0384d1d936",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2645",
"triggerID" : "1a80559bd98829552acffdaf20d3ea0384d1d936",
"triggerType" : "PUSH"
}, {
"hash" : "c4af04dab3dab31ef05ba6007000738a2dfb81ce",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2649",
"triggerID" : "c4af04dab3dab31ef05ba6007000738a2dfb81ce",
"triggerType" : "PUSH"
}, {
"hash" : "369c5d6f1f4b6eed3c86f838fb6148d127cc6703",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2660",
"triggerID" : "369c5d6f1f4b6eed3c86f838fb6148d127cc6703",
"triggerType" : "PUSH"
}, {
"hash" : "369c5d6f1f4b6eed3c86f838fb6148d127cc6703",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2898",
"triggerID" : "952859446",
"triggerType" : "MANUAL"
}, {
"hash" : "196a40e3273e98f2317856a55de3bd1888e9f754",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2899",
"triggerID" : "196a40e3273e98f2317856a55de3bd1888e9f754",
"triggerType" : "PUSH"
}, {
"hash" : "196a40e3273e98f2317856a55de3bd1888e9f754",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2899",
"triggerID" : "952859446",
"triggerType" : "MANUAL"
}, {
"hash" : "633b7370155185dead726ddfafd773e284ed510c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2935",
"triggerID" : "633b7370155185dead726ddfafd773e284ed510c",
"triggerType" : "PUSH"
}, {
"hash" : "2182f542acd4ae3d1fd239e6441e9761eb2d33a1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3069",
"triggerID" : "2182f542acd4ae3d1fd239e6441e9761eb2d33a1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0fa6297ce58eb877fd5c4eba59fef20ad9335d26 UNKNOWN
* 633b7370155185dead726ddfafd773e284ed510c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2935)
* 2182f542acd4ae3d1fd239e6441e9761eb2d33a1 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3069)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
}, {
"hash" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325",
"triggerID" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"triggerType" : "PUSH"
}, {
"hash" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 42735e78f24bd3a5f622026627d6ff8b95708705 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325)
* c470b4c2c6a4d1d33a5facf729971b4331fe0771 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao removed a comment on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao removed a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-927622687
@danny0405 @nsivabalan could you pls help to review this pr，
@danny0405 we find another critical bug about hudi 0.9.0 : [HUDI-1969] Support reading logs for MOR Hive rt table (#3033). this pr introduce a critical bug. if hive/presto split a large base file into two splits, each splits contains a log file. the log file will read twice, which produce error result.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-927622687
@danny0405 @nsivabalan could you pls help to review this pr，
@danny0405 we find another critical bug about hudi 0.9.0 : [HUDI-1969] Support reading logs for MOR Hive rt table (#3033). this pr introduce a critical bug. if hive/presto split a large base file into two splits, each splits contains a log file. the log file will read twice, which produce error result.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-927749497
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
}, {
"hash" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325",
"triggerID" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"triggerType" : "PUSH"
}, {
"hash" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2343",
"triggerID" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"triggerType" : "PUSH"
}, {
"hash" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2424",
"triggerID" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2427",
"triggerID" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2429",
"triggerID" : "927749497",
"triggerType" : "MANUAL"
}, {
"hash" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 814b7eb1e147197c29cc2701a08b6bc77bf6c2f8 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2427) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2429)
* 40b81794cbd4d6e7b7356b42ece3290b0904e987 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r728890684
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieRealtimeInputFormatUtils.java
##########
@@ -161,6 +162,51 @@
return rtSplits.toArray(new InputSplit[0]);
}
+ // pick all incremental files and add them to rtSplits then filter out those files.
+ private static Map<Path, List<FileSplit>> filterOutIncrementalSplits(
+ List<FileSplit> fileSplitList,
+ List<InputSplit> rtSplits,
+ final Option<HoodieVirtualKeyInfo> finalHoodieVirtualKeyInfo) {
+ return fileSplitList.stream().filter(s -> {
+ // deal with incremental query.
+ try {
+ if (s instanceof BaseFileWithLogsSplit) {
+ BaseFileWithLogsSplit bs = (BaseFileWithLogsSplit)s;
+ if (bs.getBelongToIncrementalSplit()) {
+ rtSplits.add(new HoodieRealtimeFileSplit(bs, bs.getBasePath(), bs.getDeltaLogPaths(), bs.getMaxCommitTime(), finalHoodieVirtualKeyInfo));
+ }
+ } else if (s instanceof RealtimeBootstrapBaseFileSplit) {
+ rtSplits.add(s);
+ }
+ } catch (IOException e) {
+ throw new HoodieIOException("Error creating hoodie real time split ", e);
+ }
+ // filter the snapshot split.
+ if (s instanceof RealtimeBootstrapBaseFileSplit) {
+ return false;
+ } else if ((s instanceof BaseFileWithLogsSplit) && ((BaseFileWithLogsSplit) s).getBelongToIncrementalSplit()) {
Review comment:
i just want to split the logical of incremental query and snapshot query. ok i will change it ,thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
}, {
"hash" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325",
"triggerID" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"triggerType" : "PUSH"
}, {
"hash" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2343",
"triggerID" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"triggerType" : "PUSH"
}, {
"hash" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2424",
"triggerID" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2427",
"triggerID" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2429",
"triggerID" : "927749497",
"triggerType" : "MANUAL"
}, {
"hash" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2431",
"triggerID" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"triggerType" : "PUSH"
}, {
"hash" : "29e6acb65044597e35f9d499385dad940580d9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2432",
"triggerID" : "29e6acb65044597e35f9d499385dad940580d9df",
"triggerType" : "PUSH"
}, {
"hash" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2441",
"triggerID" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"triggerType" : "PUSH"
}, {
"hash" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2443",
"triggerID" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 158bfbfd6991b583b3b708922459fb2c0eaf4f62 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2441)
* 0fa6297ce58eb877fd5c4eba59fef20ad9335d26 UNKNOWN
* 423f047e00904b17735a5f0120959de87e4cb0a3 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2443)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r718189316
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieMergedLogReader.java
##########
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop.realtime;
+
+import java.io.IOException;
+import java.text.MessageFormat;
+import java.util.Iterator;
+
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.io.ArrayWritable;
+import org.apache.hadoop.io.NullWritable;
+import org.apache.hadoop.io.Writable;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.RecordReader;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.hadoop.utils.HoodieRealtimeRecordReaderUtils;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+/**
+ * Record Reader implementation to read avro data, to support inc queries.
+ */
+public class HoodieMergedLogReader extends AbstractRealtimeRecordReader
+ implements RecordReader<NullWritable, ArrayWritable> {
+ private static final Logger LOG = LogManager.getLogger(AbstractRealtimeRecordReader.class);
+ private final HoodieMergedLogRecordScanner logRecordScanner;
+ private final Iterator<HoodieRecord<? extends HoodieRecordPayload>> logRecordsKeyIterator;
+ private ArrayWritable valueObj;
+ private boolean supportTimeStamp = false;
+
+ private int end;
+ private int offset;
+
+ public HoodieMergedLogReader(RealtimeSplit split, JobConf job, HoodieMergedLogRecordScanner logRecordScanner) {
+ super(split, job);
+ this.logRecordScanner = logRecordScanner;
+ this.end = logRecordScanner.getRecords().size();
+ this.logRecordsKeyIterator = logRecordScanner.iterator();
+ this.valueObj = new ArrayWritable(Writable.class, new Writable[getHiveSchema().getFields().size()]);
+ }
+
+ private Option buildGenericRecordwithCustomPayload(HoodieRecord record) throws IOException {
+ if (usesCustomPayload) {
+ return record.getData().getInsertValue(getWriterSchema());
+ } else {
+ return record.getData().getInsertValue(getReaderSchema());
+ }
+ }
+
+ @Override
+ public boolean next(NullWritable key, ArrayWritable arrayWritable) throws IOException {
+ if (!logRecordsKeyIterator.hasNext()) {
+ return false;
+ }
+ Option<GenericRecord> rec;
+ HoodieRecord currentRecord = logRecordsKeyIterator.next();
+
+ rec = buildGenericRecordwithCustomPayload(currentRecord);
+ // try to skip delete record
+ while (!rec.isPresent() && logRecordsKeyIterator.hasNext()) {
+ offset++;
+ rec = buildGenericRecordwithCustomPayload(logRecordsKeyIterator.next());
+ }
+ if (!rec.isPresent()) {
+ return false;
+ }
+
+ GenericRecord recordToReturn = rec.get();
+ if (usesCustomPayload) {
+ // If using a custom payload, return only the projection fields. The readerSchema is a schema derived from
+ // the writerSchema with only the projection fields
+ recordToReturn = HoodieAvroUtils.rewriteRecord(rec.get(), getReaderSchema());
+ }
+ ArrayWritable curWritable = (ArrayWritable) HoodieRealtimeRecordReaderUtils.avroToArrayWritable(recordToReturn, getHiveSchema());
+
+ if (arrayWritable != curWritable) {
+ final Writable[] arrValue = arrayWritable.get();
+ final Writable[] arrCurrent = curWritable.get();
+ if (arrayWritable != null && arrValue.length == arrCurrent.length) {
+ System.arraycopy(arrCurrent, 0, arrValue, 0, arrCurrent.length);
+ } else {
+ if (arrValue.length != arrCurrent.length) {
+ throw new IOException(MessageFormat.format("HoodieMergeLogReader : size of object differs. Value size : {0}, Current Object size : {1}",
+ arrValue.length, arrCurrent.length));
+ } else {
+ throw new IOException("HoodieMergeLogReader can not support RecordReaders that don't return same key & value & value is null");
+ }
+ }
+ }
+
+ offset++;
+ return true;
+ }
+
+ @Override
+ public NullWritable createKey() {
+ return null;
+ }
+
+ @Override
+ public ArrayWritable createValue() {
+ return valueObj;
+ }
+
+ @Override
+ public long getPos() throws IOException {
+ return offset;
+ }
+
+ @Override
+ public void close() throws IOException {
+ if (logRecordScanner != null) {
+ logRecordScanner.close();
+ }
+ }
+
+ @Override
+ public float getProgress() throws IOException {
+ if (end == offset) {
+ return 0.0f;
Review comment:
oh， no。 i am sorry for it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a change in pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
danny0405 commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r731567300
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieRealtimeInputFormatUtils.java
##########
@@ -161,6 +162,46 @@
return rtSplits.toArray(new InputSplit[0]);
}
+ // pick all incremental files and add them to rtSplits then filter out those files.
+ private static Map<Path, List<FileSplit>> filterOutIncrementalSplits(
+ List<FileSplit> fileSplitList,
Review comment:
I still think there is no need to handle the incremental splits first, can we just merge the handling into the line 139 ~ line 148, and logic for `BaseFileWithLogsSplit` can be reused ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
danny0405 commented on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-922561725
> @danny0405 Thank you for your attention to this pr。 This pr has not been updated for a long time. Do we still need this now？
We need it definitely, for a complete Hive reading.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
danny0405 commented on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-922256804
Hi, can we push this PR forward, this is important because the global index table can not query through Hive, which is a critical bug.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r725741007
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieParquetRealtimeInputFormat.java
##########
@@ -66,6 +91,139 @@
return HoodieRealtimeInputFormatUtils.getRealtimeSplits(job, fileSplits);
}
+ /**
+ * keep the logical of mor_incr_view as same as spark datasource.
+ * to do: unify the incremental view code between hive/spark-sql and spark datasource
+ */
+ @Override
+ protected List<FileStatus> listStatusForIncrementalMode(
+ JobConf job, HoodieTableMetaClient tableMetaClient, List<Path> inputPaths) throws IOException {
+ List<FileStatus> result = new ArrayList<>();
+ String tableName = tableMetaClient.getTableConfig().getTableName();
+ Job jobContext = Job.getInstance(job);
+
+ Option<HoodieTimeline> timeline = HoodieInputFormatUtils.getFilteredCommitsTimeline(jobContext, tableMetaClient);
+ if (!timeline.isPresent()) {
+ return result;
+ }
+ String lastIncrementalTs = HoodieHiveUtils.readStartCommitTime(jobContext, tableName);
+ // Total number of commits to return in this batch. Set this to -1 to get all the commits.
+ Integer maxCommits = HoodieHiveUtils.readMaxCommits(jobContext, tableName);
+ HoodieTimeline commitsTimelineToReturn = timeline.get().findInstantsAfter(lastIncrementalTs, maxCommits);
+ Option<List<HoodieInstant>> commitsToCheck = Option.of(commitsTimelineToReturn.getInstants().collect(Collectors.toList()));
+ if (!commitsToCheck.isPresent()) {
+ return result;
+ }
+ Map<String, HashMap<String, FileStatus>> partitionsWithFileStatus = HoodieInputFormatUtils
Review comment:
all files base files(if exitsts) and log files (if exists).
we use those files to build HoodieTableFileSystemView.
and HoodieTableFileSystemView.getAllFileGroups will help us to deal with replacement commit.
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieParquetRealtimeInputFormat.java
##########
@@ -66,6 +91,139 @@
return HoodieRealtimeInputFormatUtils.getRealtimeSplits(job, fileSplits);
}
+ /**
+ * keep the logical of mor_incr_view as same as spark datasource.
+ * to do: unify the incremental view code between hive/spark-sql and spark datasource
+ */
+ @Override
+ protected List<FileStatus> listStatusForIncrementalMode(
+ JobConf job, HoodieTableMetaClient tableMetaClient, List<Path> inputPaths) throws IOException {
+ List<FileStatus> result = new ArrayList<>();
+ String tableName = tableMetaClient.getTableConfig().getTableName();
+ Job jobContext = Job.getInstance(job);
+
+ Option<HoodieTimeline> timeline = HoodieInputFormatUtils.getFilteredCommitsTimeline(jobContext, tableMetaClient);
+ if (!timeline.isPresent()) {
+ return result;
+ }
+ String lastIncrementalTs = HoodieHiveUtils.readStartCommitTime(jobContext, tableName);
+ // Total number of commits to return in this batch. Set this to -1 to get all the commits.
+ Integer maxCommits = HoodieHiveUtils.readMaxCommits(jobContext, tableName);
+ HoodieTimeline commitsTimelineToReturn = timeline.get().findInstantsAfter(lastIncrementalTs, maxCommits);
+ Option<List<HoodieInstant>> commitsToCheck = Option.of(commitsTimelineToReturn.getInstants().collect(Collectors.toList()));
+ if (!commitsToCheck.isPresent()) {
+ return result;
+ }
+ Map<String, HashMap<String, FileStatus>> partitionsWithFileStatus = HoodieInputFormatUtils
+ .listAffectedFilesForCommits(new Path(tableMetaClient.getBasePath()), commitsToCheck.get(), commitsTimelineToReturn);
+ // build fileGroup from fsView
+ List<FileStatus> affectedFileStatus = new ArrayList<>();
+ partitionsWithFileStatus.forEach((key, value) -> value.forEach((k, v) -> affectedFileStatus.add(v)));
+ HoodieTableFileSystemView fsView = new HoodieTableFileSystemView(tableMetaClient, commitsTimelineToReturn, affectedFileStatus.toArray(new FileStatus[0]));
+ // build fileGroup from fsView
+ String basePath = tableMetaClient.getBasePath();
Review comment:
agree
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r725492758
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieParquetRealtimeInputFormat.java
##########
@@ -66,6 +91,139 @@
return HoodieRealtimeInputFormatUtils.getRealtimeSplits(job, fileSplits);
}
+ /**
+ * keep the logical of mor_incr_view as same as spark datasource.
+ * to do: unify the incremental view code between hive/spark-sql and spark datasource
Review comment:
nit: TODO
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-939236773
@danny0405 addressed all comments, could you pls review again. @nsivabalan thanks for your attention to this pr。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
}, {
"hash" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325",
"triggerID" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"triggerType" : "PUSH"
}, {
"hash" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2343",
"triggerID" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"triggerType" : "PUSH"
}, {
"hash" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2424",
"triggerID" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2427",
"triggerID" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2429",
"triggerID" : "927749497",
"triggerType" : "MANUAL"
}, {
"hash" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2431",
"triggerID" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"triggerType" : "PUSH"
}, {
"hash" : "29e6acb65044597e35f9d499385dad940580d9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2432",
"triggerID" : "29e6acb65044597e35f9d499385dad940580d9df",
"triggerType" : "PUSH"
}, {
"hash" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2441",
"triggerID" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"triggerType" : "PUSH"
}, {
"hash" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2443",
"triggerID" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2447",
"triggerID" : "928890183",
"triggerType" : "MANUAL"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2470",
"triggerID" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"triggerType" : "PUSH"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0fa6297ce58eb877fd5c4eba59fef20ad9335d26 UNKNOWN
* c14bded2e903e9c55d86fddc77d40effccca5e01 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2470)
* 6eebb1a711e3655061d471c2a96b54e205dac630 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
}, {
"hash" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325",
"triggerID" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"triggerType" : "PUSH"
}, {
"hash" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2343",
"triggerID" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"triggerType" : "PUSH"
}, {
"hash" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2424",
"triggerID" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2427",
"triggerID" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2429",
"triggerID" : "927749497",
"triggerType" : "MANUAL"
}, {
"hash" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2431",
"triggerID" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"triggerType" : "PUSH"
}, {
"hash" : "29e6acb65044597e35f9d499385dad940580d9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2432",
"triggerID" : "29e6acb65044597e35f9d499385dad940580d9df",
"triggerType" : "PUSH"
}, {
"hash" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2441",
"triggerID" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"triggerType" : "PUSH"
}, {
"hash" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2443",
"triggerID" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2447",
"triggerID" : "928890183",
"triggerType" : "MANUAL"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2470",
"triggerID" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"triggerType" : "PUSH"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2560",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562",
"triggerID" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"triggerType" : "PUSH"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
}, {
"hash" : "6fa031e1dcf6e21dfb24cf4296cb00d910b0a204",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2589",
"triggerID" : "6fa031e1dcf6e21dfb24cf4296cb00d910b0a204",
"triggerType" : "PUSH"
}, {
"hash" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2593",
"triggerID" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"triggerType" : "PUSH"
}, {
"hash" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2604",
"triggerID" : "941865271",
"triggerType" : "MANUAL"
}, {
"hash" : "6d67b68e19a43f8668e5773d27ca9c33a8de0a37",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2642",
"triggerID" : "6d67b68e19a43f8668e5773d27ca9c33a8de0a37",
"triggerType" : "PUSH"
}, {
"hash" : "1a80559bd98829552acffdaf20d3ea0384d1d936",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2645",
"triggerID" : "1a80559bd98829552acffdaf20d3ea0384d1d936",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0fa6297ce58eb877fd5c4eba59fef20ad9335d26 UNKNOWN
* 1a80559bd98829552acffdaf20d3ea0384d1d936 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2645)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a change in pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
danny0405 commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r737998688
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieRealtimeInputFormatUtils.java
##########
@@ -161,6 +148,110 @@
return rtSplits.toArray(new InputSplit[0]);
}
+ // get IncrementalRealtimeSplits
+ public static InputSplit[] getIncrementalRealtimeSplits(Configuration conf, Stream<FileSplit> fileSplits) throws IOException {
+ List<InputSplit> rtSplits = new ArrayList<>();
+ List<FileSplit> fileSplitList = fileSplits.collect(Collectors.toList());
+ Set<Path> partitionSet = fileSplitList.stream().map(f -> f.getPath().getParent()).collect(Collectors.toSet());
+ Map<Path, HoodieTableMetaClient> partitionsToMetaClient = getTableMetaClientByPartitionPath(conf, partitionSet);
+ // Pre process tableConfig from first partition to fetch virtual key info
+ Option<HoodieVirtualKeyInfo> hoodieVirtualKeyInfo = Option.empty();
+ if (partitionSet.size() > 0) {
+ hoodieVirtualKeyInfo = getHoodieVirtualKeyInfo(partitionsToMetaClient.get(partitionSet.iterator().next()));
+ }
+ Option<HoodieVirtualKeyInfo> finalHoodieVirtualKeyInfo = hoodieVirtualKeyInfo;
+ fileSplitList.stream().forEach(s -> {
+ // deal with incremental query.
+ try {
+ if (s instanceof BaseFileWithLogsSplit) {
+ BaseFileWithLogsSplit bs = (BaseFileWithLogsSplit)s;
+ if (bs.getBelongToIncrementalSplit()) {
+ rtSplits.add(new HoodieRealtimeFileSplit(bs, bs.getBasePath(), bs.getDeltaLogPaths(), bs.getMaxCommitTime(), finalHoodieVirtualKeyInfo));
+ }
+ } else if (s instanceof RealtimeBootstrapBaseFileSplit) {
+ rtSplits.add(s);
+ }
+ } catch (IOException e) {
+ throw new HoodieIOException("Error creating hoodie real time split ", e);
+ }
+ });
+ LOG.info("Returning a total splits of " + rtSplits.size());
+ return rtSplits.toArray(new InputSplit[0]);
+ }
+
+ public static Option<HoodieVirtualKeyInfo> getHoodieVirtualKeyInfo(HoodieTableMetaClient metaClient) {
+ HoodieTableConfig tableConfig = metaClient.getTableConfig();
+ if (!tableConfig.populateMetaFields()) {
+ TableSchemaResolver tableSchemaResolver = new TableSchemaResolver(metaClient);
+ try {
+ MessageType parquetSchema = tableSchemaResolver.getTableParquetSchema();
+ return Option.of(new HoodieVirtualKeyInfo(tableConfig.getRecordKeyFieldProp(),
+ tableConfig.getPartitionFieldProp(), parquetSchema.getFieldIndex(tableConfig.getRecordKeyFieldProp()),
+ parquetSchema.getFieldIndex(tableConfig.getPartitionFieldProp())));
+ } catch (Exception exception) {
+ throw new HoodieException("Fetching table schema failed with exception ", exception);
+ }
+ }
+ return Option.empty();
+ }
+
+ public static boolean isIncrementalQuerySplits(List<FileSplit> fileSplits) {
+ if (fileSplits == null || fileSplits.size() == 0) {
+ return false;
+ }
+ return fileSplits.stream().anyMatch(s -> {
+ if (s instanceof BaseFileWithLogsSplit) {
Review comment:
Can be simplified as:
```java
return fileSplits.stream().anyMatch(s -> {
if (s instanceof BaseFileWithLogsSplit) {
BaseFileWithLogsSplit bs = (BaseFileWithLogsSplit)s;
return bs.getBelongToIncrementalSplit();
} else {
return s instanceof RealtimeBootstrapBaseFileSplit;
}
});
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r738013103
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieRealtimeInputFormatUtils.java
##########
@@ -161,6 +148,110 @@
return rtSplits.toArray(new InputSplit[0]);
}
+ // get IncrementalRealtimeSplits
+ public static InputSplit[] getIncrementalRealtimeSplits(Configuration conf, Stream<FileSplit> fileSplits) throws IOException {
+ List<InputSplit> rtSplits = new ArrayList<>();
+ List<FileSplit> fileSplitList = fileSplits.collect(Collectors.toList());
+ Set<Path> partitionSet = fileSplitList.stream().map(f -> f.getPath().getParent()).collect(Collectors.toSet());
+ Map<Path, HoodieTableMetaClient> partitionsToMetaClient = getTableMetaClientByPartitionPath(conf, partitionSet);
+ // Pre process tableConfig from first partition to fetch virtual key info
+ Option<HoodieVirtualKeyInfo> hoodieVirtualKeyInfo = Option.empty();
+ if (partitionSet.size() > 0) {
+ hoodieVirtualKeyInfo = getHoodieVirtualKeyInfo(partitionsToMetaClient.get(partitionSet.iterator().next()));
+ }
+ Option<HoodieVirtualKeyInfo> finalHoodieVirtualKeyInfo = hoodieVirtualKeyInfo;
+ fileSplitList.stream().forEach(s -> {
+ // deal with incremental query.
+ try {
+ if (s instanceof BaseFileWithLogsSplit) {
+ BaseFileWithLogsSplit bs = (BaseFileWithLogsSplit)s;
+ if (bs.getBelongToIncrementalSplit()) {
+ rtSplits.add(new HoodieRealtimeFileSplit(bs, bs.getBasePath(), bs.getDeltaLogPaths(), bs.getMaxCommitTime(), finalHoodieVirtualKeyInfo));
+ }
+ } else if (s instanceof RealtimeBootstrapBaseFileSplit) {
+ rtSplits.add(s);
+ }
+ } catch (IOException e) {
+ throw new HoodieIOException("Error creating hoodie real time split ", e);
+ }
+ });
+ LOG.info("Returning a total splits of " + rtSplits.size());
+ return rtSplits.toArray(new InputSplit[0]);
+ }
+
+ public static Option<HoodieVirtualKeyInfo> getHoodieVirtualKeyInfo(HoodieTableMetaClient metaClient) {
+ HoodieTableConfig tableConfig = metaClient.getTableConfig();
+ if (!tableConfig.populateMetaFields()) {
+ TableSchemaResolver tableSchemaResolver = new TableSchemaResolver(metaClient);
+ try {
+ MessageType parquetSchema = tableSchemaResolver.getTableParquetSchema();
+ return Option.of(new HoodieVirtualKeyInfo(tableConfig.getRecordKeyFieldProp(),
+ tableConfig.getPartitionFieldProp(), parquetSchema.getFieldIndex(tableConfig.getRecordKeyFieldProp()),
+ parquetSchema.getFieldIndex(tableConfig.getPartitionFieldProp())));
+ } catch (Exception exception) {
+ throw new HoodieException("Fetching table schema failed with exception ", exception);
+ }
+ }
+ return Option.empty();
+ }
+
+ public static boolean isIncrementalQuerySplits(List<FileSplit> fileSplits) {
+ if (fileSplits == null || fileSplits.size() == 0) {
+ return false;
+ }
+ return fileSplits.stream().anyMatch(s -> {
+ if (s instanceof BaseFileWithLogsSplit) {
+ BaseFileWithLogsSplit bs = (BaseFileWithLogsSplit)s;
+ if (bs.getBelongToIncrementalSplit()) {
+ return true;
+ }
+ } else if (s instanceof RealtimeBootstrapBaseFileSplit) {
+ return true;
+ }
+ return false;
+ });
+ }
+
+ // Pick all incremental files and add them to rtSplits then filter out those files.
+ private static Map<Path, List<FileSplit>> filterOutIncrementalSplits(
+ List<FileSplit> fileSplitList,
Review comment:
sorry， will update。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
}, {
"hash" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325",
"triggerID" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"triggerType" : "PUSH"
}, {
"hash" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2343",
"triggerID" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"triggerType" : "PUSH"
}, {
"hash" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2424",
"triggerID" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2427",
"triggerID" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2429",
"triggerID" : "927749497",
"triggerType" : "MANUAL"
}, {
"hash" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2431",
"triggerID" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"triggerType" : "PUSH"
}, {
"hash" : "29e6acb65044597e35f9d499385dad940580d9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2432",
"triggerID" : "29e6acb65044597e35f9d499385dad940580d9df",
"triggerType" : "PUSH"
}, {
"hash" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2441",
"triggerID" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"triggerType" : "PUSH"
}, {
"hash" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2443",
"triggerID" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2447",
"triggerID" : "928890183",
"triggerType" : "MANUAL"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2470",
"triggerID" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"triggerType" : "PUSH"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2560",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562",
"triggerID" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"triggerType" : "PUSH"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
}, {
"hash" : "6fa031e1dcf6e21dfb24cf4296cb00d910b0a204",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2589",
"triggerID" : "6fa031e1dcf6e21dfb24cf4296cb00d910b0a204",
"triggerType" : "PUSH"
}, {
"hash" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2593",
"triggerID" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"triggerType" : "PUSH"
}, {
"hash" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2604",
"triggerID" : "941865271",
"triggerType" : "MANUAL"
}, {
"hash" : "6d67b68e19a43f8668e5773d27ca9c33a8de0a37",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2642",
"triggerID" : "6d67b68e19a43f8668e5773d27ca9c33a8de0a37",
"triggerType" : "PUSH"
}, {
"hash" : "1a80559bd98829552acffdaf20d3ea0384d1d936",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2645",
"triggerID" : "1a80559bd98829552acffdaf20d3ea0384d1d936",
"triggerType" : "PUSH"
}, {
"hash" : "c4af04dab3dab31ef05ba6007000738a2dfb81ce",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2649",
"triggerID" : "c4af04dab3dab31ef05ba6007000738a2dfb81ce",
"triggerType" : "PUSH"
}, {
"hash" : "369c5d6f1f4b6eed3c86f838fb6148d127cc6703",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2660",
"triggerID" : "369c5d6f1f4b6eed3c86f838fb6148d127cc6703",
"triggerType" : "PUSH"
}, {
"hash" : "369c5d6f1f4b6eed3c86f838fb6148d127cc6703",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2898",
"triggerID" : "952859446",
"triggerType" : "MANUAL"
}, {
"hash" : "196a40e3273e98f2317856a55de3bd1888e9f754",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2899",
"triggerID" : "196a40e3273e98f2317856a55de3bd1888e9f754",
"triggerType" : "PUSH"
}, {
"hash" : "196a40e3273e98f2317856a55de3bd1888e9f754",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2899",
"triggerID" : "952859446",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 0fa6297ce58eb877fd5c4eba59fef20ad9335d26 UNKNOWN
* 369c5d6f1f4b6eed3c86f838fb6148d127cc6703 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2660) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2898)
* 196a40e3273e98f2317856a55de3bd1888e9f754 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2899)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r740695822
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieParquetRealtimeInputFormat.java
##########
@@ -61,9 +85,183 @@
@Override
public InputSplit[] getSplits(JobConf job, int numSplits) throws IOException {
- Stream<FileSplit> fileSplits = Arrays.stream(super.getSplits(job, numSplits)).map(is -> (FileSplit) is);
+ List<FileSplit> fileSplits = Arrays.stream(super.getSplits(job, numSplits)).map(is -> (FileSplit) is).collect(Collectors.toList());
+
+ boolean isIncrementalSplits = HoodieRealtimeInputFormatUtils.isIncrementalQuerySplits(fileSplits);
+
+ return isIncrementalSplits ? HoodieRealtimeInputFormatUtils.getIncrementalRealtimeSplits(job, fileSplits.stream()) : HoodieRealtimeInputFormatUtils.getRealtimeSplits(job, fileSplits.stream());
+ }
+
+ /**
+ * Keep the logic of mor_incr_view as same as spark datasource.
+ * Step1: Get list of commits to be fetched based on start commit and max commits(for snapshot max commits is -1).
+ * Step2: Get list of affected files status for these affected file status.
+ * Step3: Construct HoodieTableFileSystemView based on those affected file status.
+ * a. Filter affected partitions based on inputPaths.
+ * b. Get list of fileGroups based on affected partitions by fsView.getAllFileGroups.
+ * Step4: Set input paths based on filtered affected partition paths. changes that amony original input paths passed to
+ * this method. some partitions did not have commits as part of the trimmed down list of commits and hence we need this step.
+ * Step5: Find candidate fileStatus, since when we get baseFileStatus from HoodieTableFileSystemView,
+ * the BaseFileStatus will missing file size information.
+ * We should use candidate fileStatus to update the size information for BaseFileStatus.
+ * Step6: For every file group from step3(b)
+ * Get 1st available base file from all file slices. then we use candidate file status to update the baseFileStatus,
+ * and construct RealTimeFileStatus and add it to result along with log files.
+ * If file group just has log files, construct RealTimeFileStatus and add it to result.
+ * TODO: unify the incremental view code between hive/spark-sql and spark datasource
+ */
+ @Override
+ protected List<FileStatus> listStatusForIncrementalMode(
+ JobConf job, HoodieTableMetaClient tableMetaClient, List<Path> inputPaths) throws IOException {
+ List<FileStatus> result = new ArrayList<>();
+ String tableName = tableMetaClient.getTableConfig().getTableName();
+ Job jobContext = Job.getInstance(job);
+
+ // step1
+ Option<HoodieTimeline> timeline = HoodieInputFormatUtils.getFilteredCommitsTimeline(jobContext, tableMetaClient);
+ if (!timeline.isPresent()) {
+ return result;
+ }
+ String lastIncrementalTs = HoodieHiveUtils.readStartCommitTime(jobContext, tableName);
+ // Total number of commits to return in this batch. Set this to -1 to get all the commits.
+ Integer maxCommits = HoodieHiveUtils.readMaxCommits(jobContext, tableName);
Review comment:
good suggest， thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
danny0405 commented on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-961732455
@xiarixiaoyao can you please join the Dingtalk group please, there are so many hive query issues every day !!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r740695703
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieRealtimeInputFormatUtils.java
##########
@@ -161,6 +148,82 @@
return rtSplits.toArray(new InputSplit[0]);
}
+ // get IncrementalRealtimeSplits
+ public static InputSplit[] getIncrementalRealtimeSplits(Configuration conf, Stream<FileSplit> fileSplits) throws IOException {
+ List<InputSplit> rtSplits = new ArrayList<>();
+ List<FileSplit> fileSplitList = fileSplits.collect(Collectors.toList());
+ Set<Path> partitionSet = fileSplitList.stream().map(f -> f.getPath().getParent()).collect(Collectors.toSet());
+ Map<Path, HoodieTableMetaClient> partitionsToMetaClient = getTableMetaClientByPartitionPath(conf, partitionSet);
+ // Pre process tableConfig from first partition to fetch virtual key info
+ Option<HoodieVirtualKeyInfo> hoodieVirtualKeyInfo = Option.empty();
+ if (partitionSet.size() > 0) {
+ hoodieVirtualKeyInfo = getHoodieVirtualKeyInfo(partitionsToMetaClient.get(partitionSet.iterator().next()));
+ }
+ Option<HoodieVirtualKeyInfo> finalHoodieVirtualKeyInfo = hoodieVirtualKeyInfo;
+ fileSplitList.stream().forEach(s -> {
+ // deal with incremental query.
+ try {
+ if (s instanceof BaseFileWithLogsSplit) {
+ BaseFileWithLogsSplit bs = (BaseFileWithLogsSplit)s;
+ if (bs.getBelongToIncrementalSplit()) {
+ rtSplits.add(new HoodieRealtimeFileSplit(bs, bs.getBasePath(), bs.getDeltaLogPaths(), bs.getMaxCommitTime(), finalHoodieVirtualKeyInfo));
+ }
+ } else if (s instanceof RealtimeBootstrapBaseFileSplit) {
+ rtSplits.add(s);
+ }
+ } catch (IOException e) {
+ throw new HoodieIOException("Error creating hoodie real time split ", e);
+ }
+ });
+ LOG.info("Returning a total splits of " + rtSplits.size());
+ return rtSplits.toArray(new InputSplit[0]);
+ }
+
+ public static Option<HoodieVirtualKeyInfo> getHoodieVirtualKeyInfo(HoodieTableMetaClient metaClient) {
+ HoodieTableConfig tableConfig = metaClient.getTableConfig();
+ if (!tableConfig.populateMetaFields()) {
+ TableSchemaResolver tableSchemaResolver = new TableSchemaResolver(metaClient);
+ try {
+ MessageType parquetSchema = tableSchemaResolver.getTableParquetSchema();
+ return Option.of(new HoodieVirtualKeyInfo(tableConfig.getRecordKeyFieldProp(),
+ tableConfig.getPartitionFieldProp(), parquetSchema.getFieldIndex(tableConfig.getRecordKeyFieldProp()),
+ parquetSchema.getFieldIndex(tableConfig.getPartitionFieldProp())));
+ } catch (Exception exception) {
+ throw new HoodieException("Fetching table schema failed with exception ", exception);
+ }
+ }
+ return Option.empty();
+ }
+
+ public static boolean isIncrementalQuerySplits(List<FileSplit> fileSplits) {
+ if (fileSplits == null || fileSplits.size() == 0) {
+ return false;
+ }
+ return fileSplits.stream().anyMatch(s -> {
+ if (s instanceof BaseFileWithLogsSplit) {
+ BaseFileWithLogsSplit bs = (BaseFileWithLogsSplit)s;
+ return bs.getBelongToIncrementalSplit();
+ } else {
+ return s instanceof RealtimeBootstrapBaseFileSplit;
+ }
+ });
+ }
+
+ public static RealtimeBootstrapBaseFileSplit createRealimeBootstrapBaseFileSplit(
Review comment:
ok
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieParquetRealtimeInputFormat.java
##########
@@ -61,9 +85,183 @@
@Override
public InputSplit[] getSplits(JobConf job, int numSplits) throws IOException {
- Stream<FileSplit> fileSplits = Arrays.stream(super.getSplits(job, numSplits)).map(is -> (FileSplit) is);
+ List<FileSplit> fileSplits = Arrays.stream(super.getSplits(job, numSplits)).map(is -> (FileSplit) is).collect(Collectors.toList());
+
+ boolean isIncrementalSplits = HoodieRealtimeInputFormatUtils.isIncrementalQuerySplits(fileSplits);
+
+ return isIncrementalSplits ? HoodieRealtimeInputFormatUtils.getIncrementalRealtimeSplits(job, fileSplits.stream()) : HoodieRealtimeInputFormatUtils.getRealtimeSplits(job, fileSplits.stream());
+ }
+
+ /**
+ * Keep the logic of mor_incr_view as same as spark datasource.
+ * Step1: Get list of commits to be fetched based on start commit and max commits(for snapshot max commits is -1).
+ * Step2: Get list of affected files status for these affected file status.
+ * Step3: Construct HoodieTableFileSystemView based on those affected file status.
+ * a. Filter affected partitions based on inputPaths.
+ * b. Get list of fileGroups based on affected partitions by fsView.getAllFileGroups.
+ * Step4: Set input paths based on filtered affected partition paths. changes that amony original input paths passed to
+ * this method. some partitions did not have commits as part of the trimmed down list of commits and hence we need this step.
+ * Step5: Find candidate fileStatus, since when we get baseFileStatus from HoodieTableFileSystemView,
+ * the BaseFileStatus will missing file size information.
+ * We should use candidate fileStatus to update the size information for BaseFileStatus.
+ * Step6: For every file group from step3(b)
+ * Get 1st available base file from all file slices. then we use candidate file status to update the baseFileStatus,
+ * and construct RealTimeFileStatus and add it to result along with log files.
+ * If file group just has log files, construct RealTimeFileStatus and add it to result.
+ * TODO: unify the incremental view code between hive/spark-sql and spark datasource
+ */
+ @Override
+ protected List<FileStatus> listStatusForIncrementalMode(
+ JobConf job, HoodieTableMetaClient tableMetaClient, List<Path> inputPaths) throws IOException {
+ List<FileStatus> result = new ArrayList<>();
+ String tableName = tableMetaClient.getTableConfig().getTableName();
+ Job jobContext = Job.getInstance(job);
+
+ // step1
+ Option<HoodieTimeline> timeline = HoodieInputFormatUtils.getFilteredCommitsTimeline(jobContext, tableMetaClient);
+ if (!timeline.isPresent()) {
+ return result;
+ }
+ String lastIncrementalTs = HoodieHiveUtils.readStartCommitTime(jobContext, tableName);
+ // Total number of commits to return in this batch. Set this to -1 to get all the commits.
+ Integer maxCommits = HoodieHiveUtils.readMaxCommits(jobContext, tableName);
Review comment:
good suggest， thanks
##########
File path: hudi-hadoop-mr/src/test/java/org/apache/hudi/hadoop/realtime/TestHoodieRealtimeRecordReader.java
##########
@@ -535,4 +548,211 @@ public void testSchemaEvolutionAndRollbackBlockInLastLogFile(ExternalSpillableMa
arguments(ExternalSpillableMap.DiskMapType.ROCKS_DB, true, true)
);
}
+
+ @Test
+ public void testIncremetalWithOnlylog() throws Exception {
+ // initial commit
+ Schema schema = HoodieAvroUtils.addMetadataFields(SchemaTestUtil.getEvolvedSchema());
+ HoodieTestUtils.init(hadoopConf, basePath.toString(), HoodieTableType.MERGE_ON_READ);
+ String instantTime = "100";
+ final int numRecords = 1000;
+ File partitionDir = InputFormatTestUtil.prepareParquetTable(basePath, schema, 1, numRecords, instantTime,
+ HoodieTableType.MERGE_ON_READ);
+ //FileCreateUtils.createDeltaCommit(basePath.toString(), instantTime);
+ createDeltaCommitFile(basePath, instantTime,"2016/05/01", "2016/05/01/fileid0_1-0-1_100.parquet", "fileid0");
+ // Add the paths
+ FileInputFormat.setInputPaths(baseJobConf, partitionDir.getPath());
+
+ // insert new records to log file
+ try {
+ String newCommitTime = "102";
+ HoodieLogFormat.Writer writer =
+ InputFormatTestUtil.writeDataBlockToLogFile(partitionDir, fs, schema, "fileid0", instantTime, newCommitTime,
+ numRecords, numRecords, 0);
+ writer.close();
+ createDeltaCommitFile(basePath, newCommitTime,"2016/05/01", "2016/05/01/.fileid0_100.log.1_1-0-1", "fileid0");
+
+ InputFormatTestUtil.setupIncremental(baseJobConf, "101", 1);
+
+ HoodieParquetRealtimeInputFormat inputFormat = new HoodieParquetRealtimeInputFormat();
+ inputFormat.setConf(baseJobConf);
+ InputSplit[] splits = inputFormat.getSplits(baseJobConf, 1);
+ assertTrue(splits.length == 1);
+ JobConf newJobConf = new JobConf(baseJobConf);
+ List<Schema.Field> fields = schema.getFields();
+ setHiveColumnNameProps(fields, newJobConf, false);
+ RecordReader<NullWritable, ArrayWritable> reader = inputFormat.getRecordReader(splits[0], newJobConf, Reporter.NULL);
+ // use reader to read log file.
+ NullWritable key = reader.createKey();
+ ArrayWritable value = reader.createValue();
+ while (reader.next(key, value)) {
+ Writable[] values = value.get();
+ // check if the record written is with latest commit, here "101"
+ assertEquals(newCommitTime, values[0].toString());
+ key = reader.createKey();
+ value = reader.createValue();
+ }
+ reader.close();
+ } catch (IOException e) {
+ throw new HoodieException(e.getMessage(), e);
+ }
+ }
+
+ @Test
+ public void testIncremetalWithReplace() throws Exception {
Review comment:
yes will add test case for compaction，
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieParquetRealtimeInputFormat.java
##########
@@ -61,9 +85,183 @@
@Override
public InputSplit[] getSplits(JobConf job, int numSplits) throws IOException {
- Stream<FileSplit> fileSplits = Arrays.stream(super.getSplits(job, numSplits)).map(is -> (FileSplit) is);
+ List<FileSplit> fileSplits = Arrays.stream(super.getSplits(job, numSplits)).map(is -> (FileSplit) is).collect(Collectors.toList());
+
+ boolean isIncrementalSplits = HoodieRealtimeInputFormatUtils.isIncrementalQuerySplits(fileSplits);
+
+ return isIncrementalSplits ? HoodieRealtimeInputFormatUtils.getIncrementalRealtimeSplits(job, fileSplits.stream()) : HoodieRealtimeInputFormatUtils.getRealtimeSplits(job, fileSplits.stream());
+ }
+
+ /**
+ * Keep the logic of mor_incr_view as same as spark datasource.
+ * Step1: Get list of commits to be fetched based on start commit and max commits(for snapshot max commits is -1).
+ * Step2: Get list of affected files status for these affected file status.
+ * Step3: Construct HoodieTableFileSystemView based on those affected file status.
+ * a. Filter affected partitions based on inputPaths.
+ * b. Get list of fileGroups based on affected partitions by fsView.getAllFileGroups.
+ * Step4: Set input paths based on filtered affected partition paths. changes that amony original input paths passed to
+ * this method. some partitions did not have commits as part of the trimmed down list of commits and hence we need this step.
+ * Step5: Find candidate fileStatus, since when we get baseFileStatus from HoodieTableFileSystemView,
+ * the BaseFileStatus will missing file size information.
+ * We should use candidate fileStatus to update the size information for BaseFileStatus.
+ * Step6: For every file group from step3(b)
+ * Get 1st available base file from all file slices. then we use candidate file status to update the baseFileStatus,
+ * and construct RealTimeFileStatus and add it to result along with log files.
+ * If file group just has log files, construct RealTimeFileStatus and add it to result.
+ * TODO: unify the incremental view code between hive/spark-sql and spark datasource
+ */
+ @Override
+ protected List<FileStatus> listStatusForIncrementalMode(
+ JobConf job, HoodieTableMetaClient tableMetaClient, List<Path> inputPaths) throws IOException {
+ List<FileStatus> result = new ArrayList<>();
+ String tableName = tableMetaClient.getTableConfig().getTableName();
+ Job jobContext = Job.getInstance(job);
+
+ // step1
+ Option<HoodieTimeline> timeline = HoodieInputFormatUtils.getFilteredCommitsTimeline(jobContext, tableMetaClient);
+ if (!timeline.isPresent()) {
+ return result;
+ }
+ String lastIncrementalTs = HoodieHiveUtils.readStartCommitTime(jobContext, tableName);
+ // Total number of commits to return in this batch. Set this to -1 to get all the commits.
+ Integer maxCommits = HoodieHiveUtils.readMaxCommits(jobContext, tableName);
+ HoodieTimeline commitsTimelineToReturn = timeline.get().findInstantsAfter(lastIncrementalTs, maxCommits);
+ Option<List<HoodieInstant>> commitsToCheck = Option.of(commitsTimelineToReturn.getInstants().collect(Collectors.toList()));
+ if (!commitsToCheck.isPresent()) {
+ return result;
+ }
+ // step2
+ commitsToCheck.get().sort(HoodieInstant::compareTo);
+ List<HoodieCommitMetadata> metadataList = commitsToCheck
+ .get().stream().map(instant -> {
+ try {
+ return HoodieInputFormatUtils.getCommitMetadata(instant, commitsTimelineToReturn);
+ } catch (IOException e) {
+ throw new HoodieException(String.format("cannot get metadata for instant: %s", instant));
+ }
+ }).collect(Collectors.toList());
+
+ // build fileGroup from fsView
+ List<FileStatus> affectedFileStatus = Arrays.asList(HoodieInputFormatUtils
+ .listAffectedFilesForCommits(new Path(tableMetaClient.getBasePath()), metadataList));
+ // step3
+ HoodieTableFileSystemView fsView = new HoodieTableFileSystemView(tableMetaClient, commitsTimelineToReturn, affectedFileStatus.toArray(new FileStatus[0]));
+ // build fileGroup from fsView
+ Path basePath = new Path(tableMetaClient.getBasePath());
+ // filter affectedPartition by inputPaths
+ List<String> affectedPartition = HoodieInputFormatUtils.getWritePartitionPaths(metadataList).stream()
+ .filter(k -> k.isEmpty() ? inputPaths.contains(basePath) : inputPaths.contains(new Path(basePath, k))).collect(Collectors.toList());
+ if (affectedPartition.isEmpty()) {
+ return result;
+ }
+ List<HoodieFileGroup> fileGroups = affectedPartition.stream()
Review comment:
L148 has already added all files in HoodieTableFileSystemView， and partitionToFileGroupsMap will be filled and cached。
L157 take the read lock， but it can load fileGroups from partitionToFileGroupsMap directly no need fs rpc call， i thinks It won't cost much to read the lock。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
}, {
"hash" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325",
"triggerID" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"triggerType" : "PUSH"
}, {
"hash" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2343",
"triggerID" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"triggerType" : "PUSH"
}, {
"hash" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2424",
"triggerID" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2427",
"triggerID" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2429",
"triggerID" : "927749497",
"triggerType" : "MANUAL"
}, {
"hash" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2431",
"triggerID" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"triggerType" : "PUSH"
}, {
"hash" : "29e6acb65044597e35f9d499385dad940580d9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2432",
"triggerID" : "29e6acb65044597e35f9d499385dad940580d9df",
"triggerType" : "PUSH"
}, {
"hash" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2441",
"triggerID" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"triggerType" : "PUSH"
}, {
"hash" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2443",
"triggerID" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2447",
"triggerID" : "928890183",
"triggerType" : "MANUAL"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2470",
"triggerID" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"triggerType" : "PUSH"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2560",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562",
"triggerID" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"triggerType" : "PUSH"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
}, {
"hash" : "6fa031e1dcf6e21dfb24cf4296cb00d910b0a204",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2589",
"triggerID" : "6fa031e1dcf6e21dfb24cf4296cb00d910b0a204",
"triggerType" : "PUSH"
}, {
"hash" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2593",
"triggerID" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"triggerType" : "PUSH"
}, {
"hash" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2604",
"triggerID" : "941865271",
"triggerType" : "MANUAL"
}, {
"hash" : "6d67b68e19a43f8668e5773d27ca9c33a8de0a37",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2642",
"triggerID" : "6d67b68e19a43f8668e5773d27ca9c33a8de0a37",
"triggerType" : "PUSH"
}, {
"hash" : "1a80559bd98829552acffdaf20d3ea0384d1d936",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2645",
"triggerID" : "1a80559bd98829552acffdaf20d3ea0384d1d936",
"triggerType" : "PUSH"
}, {
"hash" : "c4af04dab3dab31ef05ba6007000738a2dfb81ce",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2649",
"triggerID" : "c4af04dab3dab31ef05ba6007000738a2dfb81ce",
"triggerType" : "PUSH"
}, {
"hash" : "369c5d6f1f4b6eed3c86f838fb6148d127cc6703",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2660",
"triggerID" : "369c5d6f1f4b6eed3c86f838fb6148d127cc6703",
"triggerType" : "PUSH"
}, {
"hash" : "369c5d6f1f4b6eed3c86f838fb6148d127cc6703",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2898",
"triggerID" : "952859446",
"triggerType" : "MANUAL"
}, {
"hash" : "196a40e3273e98f2317856a55de3bd1888e9f754",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2899",
"triggerID" : "196a40e3273e98f2317856a55de3bd1888e9f754",
"triggerType" : "PUSH"
}, {
"hash" : "196a40e3273e98f2317856a55de3bd1888e9f754",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2899",
"triggerID" : "952859446",
"triggerType" : "MANUAL"
}, {
"hash" : "633b7370155185dead726ddfafd773e284ed510c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2935",
"triggerID" : "633b7370155185dead726ddfafd773e284ed510c",
"triggerType" : "PUSH"
}, {
"hash" : "2182f542acd4ae3d1fd239e6441e9761eb2d33a1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2182f542acd4ae3d1fd239e6441e9761eb2d33a1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0fa6297ce58eb877fd5c4eba59fef20ad9335d26 UNKNOWN
* 633b7370155185dead726ddfafd773e284ed510c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2935)
* 2182f542acd4ae3d1fd239e6441e9761eb2d33a1 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
codope commented on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-962923456
> +1, would merge this if there are no more comments in 8 hours.
@danny0405 @xiarixiaoyao Please hold on for some time.
@nsivabalan faced an issue while running incremental query with this patch: https://gist.github.com/nsivabalan/950bdde0b3b64349bf0395b2e3e928d8
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-963041907
@nsivabalan @codope
for this test case https://gist.github.com/nsivabalan/950bdde0b3b64349bf0395b2e3e928d8,
when sync to hive , Why use "_ hoodie_ partition_ Path" as partition instead of "driver".
L102 in HoodieParquetRealtimeInputFormat will add "_ hoodie_ partition_ Path" in RequiedProjectionFields
the method HoodieRealtimeRecordReaderUtils.orderFields will remove "_ hoodie_ partition_ Path" from projectCols since partitionFiedls contains this col. then L264 will throw Exception.
do you think we need fixed that problem, If necessary, I will raise an additional JIRA to track this issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
danny0405 commented on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-960471632
Hi, @xiarixiaoyao , before we merge, can you checkout this issue: https://issues.apache.org/jira/browse/HUDI-2649, to make sure that all the problems are resolved. Thanks in advance ~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a change in pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
codope commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r726012801
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/BaseFileWithLogsSplit.java
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop;
+
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.apache.hadoop.mapred.FileSplit;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * we need to encode additional information in split to track matching log file and base files.
+ * Hence, this weird looking class which tracks an log/base file split
+ */
+public class BaseFileWithLogsSplit extends FileSplit {
+ // a flag to mark this split is produced by incremental query or not.
+ private boolean belongToIncrementalSplit = false;
Review comment:
Minor: Rename to `isIncrementalSplit`. Sorry for nitpicking on the name but I feel if the comment is clear enough then the name can be a bit less verbose.
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/BaseFileWithLogsSplit.java
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop;
+
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.apache.hadoop.mapred.FileSplit;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * we need to encode additional information in split to track matching log file and base files.
Review comment:
How is this different from `HoodieRealtimeFileSplit`? Why can't we encode additional info there?
Also, if we do need a separate class then should it not implement `RealtimeSplit` interface?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
}, {
"hash" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325",
"triggerID" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"triggerType" : "PUSH"
}, {
"hash" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2343",
"triggerID" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"triggerType" : "PUSH"
}, {
"hash" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2424",
"triggerID" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2427",
"triggerID" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2429",
"triggerID" : "927749497",
"triggerType" : "MANUAL"
}, {
"hash" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2431",
"triggerID" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"triggerType" : "PUSH"
}, {
"hash" : "29e6acb65044597e35f9d499385dad940580d9df",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "29e6acb65044597e35f9d499385dad940580d9df",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 40b81794cbd4d6e7b7356b42ece3290b0904e987 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2431)
* 29e6acb65044597e35f9d499385dad940580d9df UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
danny0405 commented on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-928786962
Thanks, i will take a look tomorrow :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
}, {
"hash" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325",
"triggerID" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"triggerType" : "PUSH"
}, {
"hash" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2343",
"triggerID" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"triggerType" : "PUSH"
}, {
"hash" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2424",
"triggerID" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2427",
"triggerID" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2429",
"triggerID" : "927749497",
"triggerType" : "MANUAL"
}, {
"hash" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2431",
"triggerID" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"triggerType" : "PUSH"
}, {
"hash" : "29e6acb65044597e35f9d499385dad940580d9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2432",
"triggerID" : "29e6acb65044597e35f9d499385dad940580d9df",
"triggerType" : "PUSH"
}, {
"hash" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2441",
"triggerID" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"triggerType" : "PUSH"
}, {
"hash" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2443",
"triggerID" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2447",
"triggerID" : "928890183",
"triggerType" : "MANUAL"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2470",
"triggerID" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"triggerType" : "PUSH"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2560",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562",
"triggerID" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"triggerType" : "PUSH"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
}, {
"hash" : "6fa031e1dcf6e21dfb24cf4296cb00d910b0a204",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2589",
"triggerID" : "6fa031e1dcf6e21dfb24cf4296cb00d910b0a204",
"triggerType" : "PUSH"
}, {
"hash" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2593",
"triggerID" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0fa6297ce58eb877fd5c4eba59fef20ad9335d26 UNKNOWN
* 6fa031e1dcf6e21dfb24cf4296cb00d910b0a204 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2589)
* b5649459e5e8a5ebf5a140418c16951294a29689 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2593)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-929873306
@danny0405 @leesf thanks for your review， i will update the code。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r716732432
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/HoodieParquetInputFormat.java
##########
@@ -73,6 +73,14 @@ protected HoodieDefaultTimeline filterInstantsTimeline(HoodieDefaultTimeline tim
return HoodieInputFormatUtils.filterInstantsTimeline(timeline);
}
+ protected FileStatus[] getStatus(JobConf job) throws IOException {
Review comment:
would we remove the method and call `listStatus` directly?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872264798
# [Codecov](https://codecov.io/gh/apache/hudi/pull/3203?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3203](https://codecov.io/gh/apache/hudi/pull/3203?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (792ae1d) into [master](https://codecov.io/gh/apache/hudi/commit/94f0f40fec44c7eecb4e886df62f52a28226974c?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (94f0f40) will **decrease** coverage by `30.70%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/hudi/pull/3203?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3203 +/- ##
=============================================
- Coverage 46.20% 15.49% -30.71%
+ Complexity 5389 479 -4910
=============================================
Files 921 280 -641
Lines 40053 11548 -28505
Branches 4297 945 -3352
=============================================
- Hits 18505 1789 -16716
+ Misses 19664 9601 -10063
+ Partials 1884 158 -1726
```
| Flag | Coverage Δ | |
|---|---|---|
| hudicli | `?` | |
| hudiclient | `0.00% <ø> (-30.46%)` | :arrow_down: |
| hudicommon | `?` | |
| hudiflink | `?` | |
| hudihadoopmr | `?` | |
| hudisparkdatasource | `?` | |
| hudisync | `5.38% <ø> (-48.67%)` | :arrow_down: |
| huditimelineservice | `?` | |
| hudiutilities | `58.05% <ø> (-0.54%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3203?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3203/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3203/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3203/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...a/org/apache/hudi/metrics/MetricsReporterType.java](https://codecov.io/gh/apache/hudi/pull/3203/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyVHlwZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...rg/apache/hudi/client/bootstrap/BootstrapMode.java](https://codecov.io/gh/apache/hudi/pull/3203/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9ib290c3RyYXAvQm9vdHN0cmFwTW9kZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...he/hudi/hive/HiveStylePartitionValueExtractor.java](https://codecov.io/gh/apache/hudi/pull/3203/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN0eWxlUGFydGl0aW9uVmFsdWVFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../apache/hudi/keygen/constant/KeyGeneratorType.java](https://codecov.io/gh/apache/hudi/pull/3203/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2tleWdlbi9jb25zdGFudC9LZXlHZW5lcmF0b3JUeXBlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...pache/hudi/client/utils/ConcatenatingIterator.java](https://codecov.io/gh/apache/hudi/pull/3203/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC91dGlscy9Db25jYXRlbmF0aW5nSXRlcmF0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../hudi/execution/bulkinsert/BulkInsertSortMode.java](https://codecov.io/gh/apache/hudi/pull/3203/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2V4ZWN1dGlvbi9idWxraW5zZXJ0L0J1bGtJbnNlcnRTb3J0TW9kZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...able/action/compact/CompactionTriggerStrategy.java](https://codecov.io/gh/apache/hudi/pull/3203/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL3RhYmxlL2FjdGlvbi9jb21wYWN0L0NvbXBhY3Rpb25UcmlnZ2VyU3RyYXRlZ3kuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [709 more](https://codecov.io/gh/apache/hudi/pull/3203/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/hudi/pull/3203?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/hudi/pull/3203?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [94f0f40...792ae1d](https://codecov.io/gh/apache/hudi/pull/3203?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
}, {
"hash" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 792ae1d9884eac5eb7004e4edb617ea6e86d79b7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631)
* 42735e78f24bd3a5f622026627d6ff8b95708705 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r717259013
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/HoodieParquetInputFormat.java
##########
@@ -73,6 +73,14 @@ protected HoodieDefaultTimeline filterInstantsTimeline(HoodieDefaultTimeline tim
return HoodieInputFormatUtils.filterInstantsTimeline(timeline);
}
+ protected FileStatus[] getStatus(JobConf job) throws IOException {
Review comment:
no, we need to call ParquetInputFormat.listStatus in HoodieParquetRealtimeInputFormat.java, but we cannot access that method directly, so we overwrite getStatus in ParquetRealtimeInputFormat.java call call listStatus in it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
}, {
"hash" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325",
"triggerID" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"triggerType" : "PUSH"
}, {
"hash" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2343",
"triggerID" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"triggerType" : "PUSH"
}, {
"hash" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2424",
"triggerID" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 679a0104259b01a85be2def1b5cd396ae4f0e858 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2424)
* 814b7eb1e147197c29cc2701a08b6bc77bf6c2f8 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-927820027
@danny0405 i revert this pr: [HUDI-1969] Support reading logs for MOR Hive rt table (#3033), since this pr introduce a critical bug which may cause the query result repeat。
give a example: now we have a base file(file1) which contains two record: key1, p1
key2, p2
then we update key2, the log file(logfile1) produced and contains record: key2, p2
when hive/presto query the table: file1 has been split into two file parts, one is file1_part1, which contains all the record; the other is file2_part2 wich contains no record. notice those two file parts will be bound to logfile1 and produce two realtimesplit。 then the log file will be read twice， repeat result produced。
The above is a very simple example， in fact query engine split one parquet file into small parts is very common。
This pr also covers the [HUDI-1969] 。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-939668486
@danny0405 @nsivabalan i think i find a way to solve the bug of HUDI-1969 and make it work together with this pr to solve the (base + log files file group) read of hbase and /flink mor table 。 i will update the code tomorrow， thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r725740308
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieParquetRealtimeInputFormat.java
##########
@@ -66,6 +91,139 @@
return HoodieRealtimeInputFormatUtils.getRealtimeSplits(job, fileSplits);
}
+ /**
+ * keep the logical of mor_incr_view as same as spark datasource.
+ * to do: unify the incremental view code between hive/spark-sql and spark datasource
+ */
+ @Override
+ protected List<FileStatus> listStatusForIncrementalMode(
+ JobConf job, HoodieTableMetaClient tableMetaClient, List<Path> inputPaths) throws IOException {
+ List<FileStatus> result = new ArrayList<>();
+ String tableName = tableMetaClient.getTableConfig().getTableName();
+ Job jobContext = Job.getInstance(job);
+
+ Option<HoodieTimeline> timeline = HoodieInputFormatUtils.getFilteredCommitsTimeline(jobContext, tableMetaClient);
+ if (!timeline.isPresent()) {
+ return result;
+ }
+ String lastIncrementalTs = HoodieHiveUtils.readStartCommitTime(jobContext, tableName);
+ // Total number of commits to return in this batch. Set this to -1 to get all the commits.
+ Integer maxCommits = HoodieHiveUtils.readMaxCommits(jobContext, tableName);
+ HoodieTimeline commitsTimelineToReturn = timeline.get().findInstantsAfter(lastIncrementalTs, maxCommits);
Review comment:
_can you help me understand, if there was a replace commit made, where exactly the filtering will happen? does findInstantsAfter will take care of that? or is the code is else where?_
for this question: findInstantsAfter will not take care of replace commit, but fsView.getAllFileGroups(partitionPath)).collect(Collectors.toList()) from line 131, will deal with replace commits.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a change in pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
danny0405 commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r725759641
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/BaseFileWithLogsSplit.java
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop;
+
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.apache.hadoop.mapred.FileSplit;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+public class BaseFileWithLogsSplit extends FileSplit {
Review comment:
All the comments should start with uppercase character: `we => We` then end with dot `.`.
Please fix all of the left comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5d49bd4e8c638ffb9ced102dc6771d6291c199e3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603)
* 792ae1d9884eac5eb7004e4edb617ea6e86d79b7 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
}, {
"hash" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325",
"triggerID" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"triggerType" : "PUSH"
}, {
"hash" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2343",
"triggerID" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"triggerType" : "PUSH"
}, {
"hash" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2424",
"triggerID" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2427",
"triggerID" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2429",
"triggerID" : "927749497",
"triggerType" : "MANUAL"
}, {
"hash" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2431",
"triggerID" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"triggerType" : "PUSH"
}, {
"hash" : "29e6acb65044597e35f9d499385dad940580d9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2432",
"triggerID" : "29e6acb65044597e35f9d499385dad940580d9df",
"triggerType" : "PUSH"
}, {
"hash" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2441",
"triggerID" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"triggerType" : "PUSH"
}, {
"hash" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2443",
"triggerID" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2447",
"triggerID" : "928890183",
"triggerType" : "MANUAL"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2470",
"triggerID" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"triggerType" : "PUSH"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2560",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562",
"triggerID" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"triggerType" : "PUSH"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
}, {
"hash" : "6fa031e1dcf6e21dfb24cf4296cb00d910b0a204",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2589",
"triggerID" : "6fa031e1dcf6e21dfb24cf4296cb00d910b0a204",
"triggerType" : "PUSH"
}, {
"hash" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2593",
"triggerID" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"triggerType" : "PUSH"
}, {
"hash" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2604",
"triggerID" : "941865271",
"triggerType" : "MANUAL"
}, {
"hash" : "6d67b68e19a43f8668e5773d27ca9c33a8de0a37",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2642",
"triggerID" : "6d67b68e19a43f8668e5773d27ca9c33a8de0a37",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0fa6297ce58eb877fd5c4eba59fef20ad9335d26 UNKNOWN
* b5649459e5e8a5ebf5a140418c16951294a29689 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2593) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2604)
* 6d67b68e19a43f8668e5773d27ca9c33a8de0a37 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2642)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
}, {
"hash" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325",
"triggerID" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"triggerType" : "PUSH"
}, {
"hash" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2343",
"triggerID" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"triggerType" : "PUSH"
}, {
"hash" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2424",
"triggerID" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2427",
"triggerID" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2429",
"triggerID" : "927749497",
"triggerType" : "MANUAL"
}, {
"hash" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2431",
"triggerID" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"triggerType" : "PUSH"
}, {
"hash" : "29e6acb65044597e35f9d499385dad940580d9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2432",
"triggerID" : "29e6acb65044597e35f9d499385dad940580d9df",
"triggerType" : "PUSH"
}, {
"hash" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2441",
"triggerID" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"triggerType" : "PUSH"
}, {
"hash" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2443",
"triggerID" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2447",
"triggerID" : "928890183",
"triggerType" : "MANUAL"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2470",
"triggerID" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"triggerType" : "PUSH"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2560",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562",
"triggerID" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"triggerType" : "PUSH"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
}, {
"hash" : "6fa031e1dcf6e21dfb24cf4296cb00d910b0a204",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2589",
"triggerID" : "6fa031e1dcf6e21dfb24cf4296cb00d910b0a204",
"triggerType" : "PUSH"
}, {
"hash" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2593",
"triggerID" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"triggerType" : "PUSH"
}, {
"hash" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2604",
"triggerID" : "941865271",
"triggerType" : "MANUAL"
}, {
"hash" : "6d67b68e19a43f8668e5773d27ca9c33a8de0a37",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2642",
"triggerID" : "6d67b68e19a43f8668e5773d27ca9c33a8de0a37",
"triggerType" : "PUSH"
}, {
"hash" : "1a80559bd98829552acffdaf20d3ea0384d1d936",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2645",
"triggerID" : "1a80559bd98829552acffdaf20d3ea0384d1d936",
"triggerType" : "PUSH"
}, {
"hash" : "c4af04dab3dab31ef05ba6007000738a2dfb81ce",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2649",
"triggerID" : "c4af04dab3dab31ef05ba6007000738a2dfb81ce",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0fa6297ce58eb877fd5c4eba59fef20ad9335d26 UNKNOWN
* 1a80559bd98829552acffdaf20d3ea0384d1d936 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2645)
* c4af04dab3dab31ef05ba6007000738a2dfb81ce Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2649)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
danny0405 commented on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-960614234
> @danny0405 This pr is try to solve the problem of incremental views in various scenarios. These problems in https://issues.apache.org/jira/browse/HUDI-2649 have nothing to do with this pr, right?
>
> first question：i test on our cluster it's ok. (hive3.1.1 flink1.12.x) second question：hudi-hive-sync is not compatible withhive-2.1.1 third question: is a Typical classNotFound problem, i donnot think it's a problem of hudi. i test on our cluster it's ok.(hive3.1.1 flink1.12.x) select count(*) from xx will trigger mr task, but select * from xx will run in local if the data is small enought
>
> other question: Are these problems from the dingding group? Can you pull me into the dingding group?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
}, {
"hash" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325",
"triggerID" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"triggerType" : "PUSH"
}, {
"hash" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2343",
"triggerID" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"triggerType" : "PUSH"
}, {
"hash" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2424",
"triggerID" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2427",
"triggerID" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2429",
"triggerID" : "927749497",
"triggerType" : "MANUAL"
}, {
"hash" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2431",
"triggerID" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"triggerType" : "PUSH"
}, {
"hash" : "29e6acb65044597e35f9d499385dad940580d9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2432",
"triggerID" : "29e6acb65044597e35f9d499385dad940580d9df",
"triggerType" : "PUSH"
}, {
"hash" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2441",
"triggerID" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 158bfbfd6991b583b3b708922459fb2c0eaf4f62 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2441)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
}, {
"hash" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325",
"triggerID" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"triggerType" : "PUSH"
}, {
"hash" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2343",
"triggerID" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"triggerType" : "PUSH"
}, {
"hash" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2424",
"triggerID" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2427",
"triggerID" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 679a0104259b01a85be2def1b5cd396ae4f0e858 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2424)
* 814b7eb1e147197c29cc2701a08b6bc77bf6c2f8 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2427)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
}, {
"hash" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325",
"triggerID" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"triggerType" : "PUSH"
}, {
"hash" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2343",
"triggerID" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"triggerType" : "PUSH"
}, {
"hash" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2424",
"triggerID" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2427",
"triggerID" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2429",
"triggerID" : "927749497",
"triggerType" : "MANUAL"
}, {
"hash" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2431",
"triggerID" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"triggerType" : "PUSH"
}, {
"hash" : "29e6acb65044597e35f9d499385dad940580d9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2432",
"triggerID" : "29e6acb65044597e35f9d499385dad940580d9df",
"triggerType" : "PUSH"
}, {
"hash" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2441",
"triggerID" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"triggerType" : "PUSH"
}, {
"hash" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2443",
"triggerID" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2447",
"triggerID" : "928890183",
"triggerType" : "MANUAL"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2470",
"triggerID" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"triggerType" : "PUSH"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2560",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562",
"triggerID" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"triggerType" : "PUSH"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
}, {
"hash" : "6fa031e1dcf6e21dfb24cf4296cb00d910b0a204",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2589",
"triggerID" : "6fa031e1dcf6e21dfb24cf4296cb00d910b0a204",
"triggerType" : "PUSH"
}, {
"hash" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2593",
"triggerID" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"triggerType" : "PUSH"
}, {
"hash" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2604",
"triggerID" : "941865271",
"triggerType" : "MANUAL"
}, {
"hash" : "6d67b68e19a43f8668e5773d27ca9c33a8de0a37",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2642",
"triggerID" : "6d67b68e19a43f8668e5773d27ca9c33a8de0a37",
"triggerType" : "PUSH"
}, {
"hash" : "1a80559bd98829552acffdaf20d3ea0384d1d936",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2645",
"triggerID" : "1a80559bd98829552acffdaf20d3ea0384d1d936",
"triggerType" : "PUSH"
}, {
"hash" : "c4af04dab3dab31ef05ba6007000738a2dfb81ce",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2649",
"triggerID" : "c4af04dab3dab31ef05ba6007000738a2dfb81ce",
"triggerType" : "PUSH"
}, {
"hash" : "369c5d6f1f4b6eed3c86f838fb6148d127cc6703",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2660",
"triggerID" : "369c5d6f1f4b6eed3c86f838fb6148d127cc6703",
"triggerType" : "PUSH"
}, {
"hash" : "196a40e3273e98f2317856a55de3bd1888e9f754",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "196a40e3273e98f2317856a55de3bd1888e9f754",
"triggerType" : "PUSH"
}, {
"hash" : "196a40e3273e98f2317856a55de3bd1888e9f754",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "952859446",
"triggerType" : "MANUAL"
}, {
"hash" : "369c5d6f1f4b6eed3c86f838fb6148d127cc6703",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2898",
"triggerID" : "952859446",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 0fa6297ce58eb877fd5c4eba59fef20ad9335d26 UNKNOWN
* 369c5d6f1f4b6eed3c86f838fb6148d127cc6703 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2660) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2898)
* 196a40e3273e98f2317856a55de3bd1888e9f754 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
}, {
"hash" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325",
"triggerID" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"triggerType" : "PUSH"
}, {
"hash" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2343",
"triggerID" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"triggerType" : "PUSH"
}, {
"hash" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2424",
"triggerID" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2427",
"triggerID" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2429",
"triggerID" : "927749497",
"triggerType" : "MANUAL"
}, {
"hash" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2431",
"triggerID" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"triggerType" : "PUSH"
}, {
"hash" : "29e6acb65044597e35f9d499385dad940580d9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2432",
"triggerID" : "29e6acb65044597e35f9d499385dad940580d9df",
"triggerType" : "PUSH"
}, {
"hash" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2441",
"triggerID" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"triggerType" : "PUSH"
}, {
"hash" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2443",
"triggerID" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2447",
"triggerID" : "928890183",
"triggerType" : "MANUAL"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2470",
"triggerID" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"triggerType" : "PUSH"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2560",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562",
"triggerID" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"triggerType" : "PUSH"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
}, {
"hash" : "6fa031e1dcf6e21dfb24cf4296cb00d910b0a204",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2589",
"triggerID" : "6fa031e1dcf6e21dfb24cf4296cb00d910b0a204",
"triggerType" : "PUSH"
}, {
"hash" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2593",
"triggerID" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"triggerType" : "PUSH"
}, {
"hash" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2604",
"triggerID" : "941865271",
"triggerType" : "MANUAL"
}, {
"hash" : "6d67b68e19a43f8668e5773d27ca9c33a8de0a37",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2642",
"triggerID" : "6d67b68e19a43f8668e5773d27ca9c33a8de0a37",
"triggerType" : "PUSH"
}, {
"hash" : "1a80559bd98829552acffdaf20d3ea0384d1d936",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2645",
"triggerID" : "1a80559bd98829552acffdaf20d3ea0384d1d936",
"triggerType" : "PUSH"
}, {
"hash" : "c4af04dab3dab31ef05ba6007000738a2dfb81ce",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2649",
"triggerID" : "c4af04dab3dab31ef05ba6007000738a2dfb81ce",
"triggerType" : "PUSH"
}, {
"hash" : "369c5d6f1f4b6eed3c86f838fb6148d127cc6703",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2660",
"triggerID" : "369c5d6f1f4b6eed3c86f838fb6148d127cc6703",
"triggerType" : "PUSH"
}, {
"hash" : "369c5d6f1f4b6eed3c86f838fb6148d127cc6703",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2898",
"triggerID" : "952859446",
"triggerType" : "MANUAL"
}, {
"hash" : "196a40e3273e98f2317856a55de3bd1888e9f754",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2899",
"triggerID" : "196a40e3273e98f2317856a55de3bd1888e9f754",
"triggerType" : "PUSH"
}, {
"hash" : "196a40e3273e98f2317856a55de3bd1888e9f754",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2899",
"triggerID" : "952859446",
"triggerType" : "MANUAL"
}, {
"hash" : "633b7370155185dead726ddfafd773e284ed510c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "633b7370155185dead726ddfafd773e284ed510c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0fa6297ce58eb877fd5c4eba59fef20ad9335d26 UNKNOWN
* 196a40e3273e98f2317856a55de3bd1888e9f754 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2899)
* 633b7370155185dead726ddfafd773e284ed510c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-962944389
@danny0405 @nsivabalan i am going to verify it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
}, {
"hash" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325",
"triggerID" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"triggerType" : "PUSH"
}, {
"hash" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2343",
"triggerID" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"triggerType" : "PUSH"
}, {
"hash" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2424",
"triggerID" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2427",
"triggerID" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2429",
"triggerID" : "927749497",
"triggerType" : "MANUAL"
}, {
"hash" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2431",
"triggerID" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"triggerType" : "PUSH"
}, {
"hash" : "29e6acb65044597e35f9d499385dad940580d9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2432",
"triggerID" : "29e6acb65044597e35f9d499385dad940580d9df",
"triggerType" : "PUSH"
}, {
"hash" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2441",
"triggerID" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"triggerType" : "PUSH"
}, {
"hash" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2443",
"triggerID" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2447",
"triggerID" : "928890183",
"triggerType" : "MANUAL"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2470",
"triggerID" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"triggerType" : "PUSH"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2560",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562",
"triggerID" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"triggerType" : "PUSH"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
}, {
"hash" : "6fa031e1dcf6e21dfb24cf4296cb00d910b0a204",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2589",
"triggerID" : "6fa031e1dcf6e21dfb24cf4296cb00d910b0a204",
"triggerType" : "PUSH"
}, {
"hash" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2593",
"triggerID" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"triggerType" : "PUSH"
}, {
"hash" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2604",
"triggerID" : "941865271",
"triggerType" : "MANUAL"
}, {
"hash" : "6d67b68e19a43f8668e5773d27ca9c33a8de0a37",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2642",
"triggerID" : "6d67b68e19a43f8668e5773d27ca9c33a8de0a37",
"triggerType" : "PUSH"
}, {
"hash" : "1a80559bd98829552acffdaf20d3ea0384d1d936",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2645",
"triggerID" : "1a80559bd98829552acffdaf20d3ea0384d1d936",
"triggerType" : "PUSH"
}, {
"hash" : "c4af04dab3dab31ef05ba6007000738a2dfb81ce",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2649",
"triggerID" : "c4af04dab3dab31ef05ba6007000738a2dfb81ce",
"triggerType" : "PUSH"
}, {
"hash" : "369c5d6f1f4b6eed3c86f838fb6148d127cc6703",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2660",
"triggerID" : "369c5d6f1f4b6eed3c86f838fb6148d127cc6703",
"triggerType" : "PUSH"
}, {
"hash" : "369c5d6f1f4b6eed3c86f838fb6148d127cc6703",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2898",
"triggerID" : "952859446",
"triggerType" : "MANUAL"
}, {
"hash" : "196a40e3273e98f2317856a55de3bd1888e9f754",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2899",
"triggerID" : "196a40e3273e98f2317856a55de3bd1888e9f754",
"triggerType" : "PUSH"
}, {
"hash" : "196a40e3273e98f2317856a55de3bd1888e9f754",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2899",
"triggerID" : "952859446",
"triggerType" : "MANUAL"
}, {
"hash" : "633b7370155185dead726ddfafd773e284ed510c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2935",
"triggerID" : "633b7370155185dead726ddfafd773e284ed510c",
"triggerType" : "PUSH"
}, {
"hash" : "2182f542acd4ae3d1fd239e6441e9761eb2d33a1",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3069",
"triggerID" : "2182f542acd4ae3d1fd239e6441e9761eb2d33a1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0fa6297ce58eb877fd5c4eba59fef20ad9335d26 UNKNOWN
* 2182f542acd4ae3d1fd239e6441e9761eb2d33a1 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3069)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 merged pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
danny0405 merged pull request #3203:
URL: https://github.com/apache/hudi/pull/3203
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
}, {
"hash" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325",
"triggerID" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"triggerType" : "PUSH"
}, {
"hash" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2343",
"triggerID" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"triggerType" : "PUSH"
}, {
"hash" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2424",
"triggerID" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2427",
"triggerID" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2429",
"triggerID" : "927749497",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 814b7eb1e147197c29cc2701a08b6bc77bf6c2f8 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2427) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2429)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-927820836
@danny0405 @nsivabalan @leesf could you pls help me reivew this pr. thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-929873306
@danny0405 @leesf thanks for your review， i will update the code。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
danny0405 commented on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-928786962
Thanks, i will take a look tomorrow :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872264798
# [Codecov](https://codecov.io/gh/apache/hudi/pull/3203?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3203](https://codecov.io/gh/apache/hudi/pull/3203?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (792ae1d) into [master](https://codecov.io/gh/apache/hudi/commit/94f0f40fec44c7eecb4e886df62f52a28226974c?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (94f0f40) will **decrease** coverage by `18.86%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/hudi/pull/3203?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3203 +/- ##
=============================================
- Coverage 46.20% 27.33% -18.87%
+ Complexity 5389 1267 -4122
=============================================
Files 921 376 -545
Lines 40053 15004 -25049
Branches 4297 1297 -3000
=============================================
- Hits 18505 4101 -14404
+ Misses 19664 10608 -9056
+ Partials 1884 295 -1589
```
| Flag | Coverage Δ | |
|---|---|---|
| hudicli | `?` | |
| hudiclient | `21.05% <ø> (-9.40%)` | :arrow_down: |
| hudicommon | `?` | |
| hudiflink | `?` | |
| hudihadoopmr | `?` | |
| hudisparkdatasource | `?` | |
| hudisync | `5.38% <ø> (-48.67%)` | :arrow_down: |
| huditimelineservice | `?` | |
| hudiutilities | `58.05% <ø> (-0.54%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3203?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3203/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3203/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3203/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...a/org/apache/hudi/metrics/MetricsReporterType.java](https://codecov.io/gh/apache/hudi/pull/3203/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyVHlwZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...rg/apache/hudi/client/bootstrap/BootstrapMode.java](https://codecov.io/gh/apache/hudi/pull/3203/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9ib290c3RyYXAvQm9vdHN0cmFwTW9kZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...he/hudi/hive/HiveStylePartitionValueExtractor.java](https://codecov.io/gh/apache/hudi/pull/3203/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN0eWxlUGFydGl0aW9uVmFsdWVFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../apache/hudi/keygen/constant/KeyGeneratorType.java](https://codecov.io/gh/apache/hudi/pull/3203/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2tleWdlbi9jb25zdGFudC9LZXlHZW5lcmF0b3JUeXBlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...pache/hudi/client/utils/ConcatenatingIterator.java](https://codecov.io/gh/apache/hudi/pull/3203/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC91dGlscy9Db25jYXRlbmF0aW5nSXRlcmF0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../hudi/execution/bulkinsert/BulkInsertSortMode.java](https://codecov.io/gh/apache/hudi/pull/3203/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2V4ZWN1dGlvbi9idWxraW5zZXJ0L0J1bGtJbnNlcnRTb3J0TW9kZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...able/action/compact/CompactionTriggerStrategy.java](https://codecov.io/gh/apache/hudi/pull/3203/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL3RhYmxlL2FjdGlvbi9jb21wYWN0L0NvbXBhY3Rpb25UcmlnZ2VyU3RyYXRlZ3kuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [615 more](https://codecov.io/gh/apache/hudi/pull/3203/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/hudi/pull/3203?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/hudi/pull/3203?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [94f0f40...792ae1d](https://codecov.io/gh/apache/hudi/pull/3203?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
}, {
"hash" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325",
"triggerID" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"triggerType" : "PUSH"
}, {
"hash" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2343",
"triggerID" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"triggerType" : "PUSH"
}, {
"hash" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2424",
"triggerID" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2427",
"triggerID" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2429",
"triggerID" : "927749497",
"triggerType" : "MANUAL"
}, {
"hash" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2431",
"triggerID" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"triggerType" : "PUSH"
}, {
"hash" : "29e6acb65044597e35f9d499385dad940580d9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2432",
"triggerID" : "29e6acb65044597e35f9d499385dad940580d9df",
"triggerType" : "PUSH"
}, {
"hash" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2441",
"triggerID" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"triggerType" : "PUSH"
}, {
"hash" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2443",
"triggerID" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2447",
"triggerID" : "928890183",
"triggerType" : "MANUAL"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2470",
"triggerID" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"triggerType" : "PUSH"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2560",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562",
"triggerID" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"triggerType" : "PUSH"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 0fa6297ce58eb877fd5c4eba59fef20ad9335d26 UNKNOWN
* c14bded2e903e9c55d86fddc77d40effccca5e01 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2470) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2560)
* 6eebb1a711e3655061d471c2a96b54e205dac630 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
}, {
"hash" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325",
"triggerID" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"triggerType" : "PUSH"
}, {
"hash" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2343",
"triggerID" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"triggerType" : "PUSH"
}, {
"hash" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2424",
"triggerID" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2427",
"triggerID" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2429",
"triggerID" : "927749497",
"triggerType" : "MANUAL"
}, {
"hash" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2431",
"triggerID" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"triggerType" : "PUSH"
}, {
"hash" : "29e6acb65044597e35f9d499385dad940580d9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2432",
"triggerID" : "29e6acb65044597e35f9d499385dad940580d9df",
"triggerType" : "PUSH"
}, {
"hash" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2441",
"triggerID" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"triggerType" : "PUSH"
}, {
"hash" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2443",
"triggerID" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2447",
"triggerID" : "928890183",
"triggerType" : "MANUAL"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2470",
"triggerID" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"triggerType" : "PUSH"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2560",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562",
"triggerID" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"triggerType" : "PUSH"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
}, {
"hash" : "6fa031e1dcf6e21dfb24cf4296cb00d910b0a204",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2589",
"triggerID" : "6fa031e1dcf6e21dfb24cf4296cb00d910b0a204",
"triggerType" : "PUSH"
}, {
"hash" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2593",
"triggerID" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"triggerType" : "PUSH"
}, {
"hash" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2604",
"triggerID" : "941865271",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 0fa6297ce58eb877fd5c4eba59fef20ad9335d26 UNKNOWN
* b5649459e5e8a5ebf5a140418c16951294a29689 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2593) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2604)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r726838452
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieRealtimeInputFormatUtils.java
##########
@@ -99,7 +101,32 @@
}
}
Option<HoodieVirtualKeyInfo> finalHoodieVirtualKeyInfo = hoodieVirtualKeyInfo;
- partitionsToParquetSplits.keySet().forEach(partitionPath -> {
+ // deal with incremental query
+ candidateFileSplits.stream().forEach(s -> {
+ try {
Review comment:
thanks， if we can do it in one iteration， the logical of deal with incremental query and the logical of deal with snapshot query are mixed together， are you sure we need to do that？
ok， i try to do it in one iteration。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-938298379
sorry for long delay. I will review this by this weekend.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r725753130
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieParquetRealtimeInputFormat.java
##########
@@ -66,6 +91,139 @@
return HoodieRealtimeInputFormatUtils.getRealtimeSplits(job, fileSplits);
}
+ /**
+ * keep the logical of mor_incr_view as same as spark datasource.
+ * to do: unify the incremental view code between hive/spark-sql and spark datasource
+ */
+ @Override
+ protected List<FileStatus> listStatusForIncrementalMode(
+ JobConf job, HoodieTableMetaClient tableMetaClient, List<Path> inputPaths) throws IOException {
+ List<FileStatus> result = new ArrayList<>();
+ String tableName = tableMetaClient.getTableConfig().getTableName();
+ Job jobContext = Job.getInstance(job);
+
+ Option<HoodieTimeline> timeline = HoodieInputFormatUtils.getFilteredCommitsTimeline(jobContext, tableMetaClient);
+ if (!timeline.isPresent()) {
+ return result;
+ }
+ String lastIncrementalTs = HoodieHiveUtils.readStartCommitTime(jobContext, tableName);
+ // Total number of commits to return in this batch. Set this to -1 to get all the commits.
+ Integer maxCommits = HoodieHiveUtils.readMaxCommits(jobContext, tableName);
+ HoodieTimeline commitsTimelineToReturn = timeline.get().findInstantsAfter(lastIncrementalTs, maxCommits);
+ Option<List<HoodieInstant>> commitsToCheck = Option.of(commitsTimelineToReturn.getInstants().collect(Collectors.toList()));
+ if (!commitsToCheck.isPresent()) {
+ return result;
+ }
+ Map<String, HashMap<String, FileStatus>> partitionsWithFileStatus = HoodieInputFormatUtils
+ .listAffectedFilesForCommits(new Path(tableMetaClient.getBasePath()), commitsToCheck.get(), commitsTimelineToReturn);
+ // build fileGroup from fsView
+ List<FileStatus> affectedFileStatus = new ArrayList<>();
+ partitionsWithFileStatus.forEach((key, value) -> value.forEach((k, v) -> affectedFileStatus.add(v)));
+ HoodieTableFileSystemView fsView = new HoodieTableFileSystemView(tableMetaClient, commitsTimelineToReturn, affectedFileStatus.toArray(new FileStatus[0]));
+ // build fileGroup from fsView
+ String basePath = tableMetaClient.getBasePath();
+ // filter affectedPartition by inputPaths
+ List<String> affectedPartition = partitionsWithFileStatus.keySet().stream()
+ .filter(k -> k.isEmpty() ? inputPaths.contains(new Path(basePath)) : inputPaths.contains(new Path(basePath, k))).collect(Collectors.toList());
+ if (affectedPartition.isEmpty()) {
+ return result;
+ }
+ List<HoodieFileGroup> fileGroups = affectedPartition.stream()
+ .flatMap(partitionPath -> fsView.getAllFileGroups(partitionPath)).collect(Collectors.toList());
+ setInputPaths(job, affectedPartition.stream()
+ .map(p -> p.isEmpty() ? basePath : new Path(basePath, p).toUri().toString()).collect(Collectors.joining(",")));
+
+ // find all file status in current partitionPath
+ FileStatus[] fileStatuses = getStatus(job);
+ Map<String, FileStatus> candidateFileStatus = new HashMap<>();
+ for (int i = 0; i < fileStatuses.length; i++) {
+ String key = fileStatuses[i].getPath().toString();
+ candidateFileStatus.put(key, fileStatuses[i]);
+ }
+
+ String maxCommitTime = fsView.getLastInstant().get().getTimestamp();
+ fileGroups.stream().forEach(f -> {
+ try {
+ List<FileSlice> baseFiles = f.getAllFileSlices().filter(slice -> slice.getBaseFile().isPresent()).collect(Collectors.toList());
+ if (!baseFiles.isEmpty()) {
+ FileStatus baseFileStatus = HoodieInputFormatUtils.getFileStatus(baseFiles.get(0).getBaseFile().get());
+ String baseFilePath = baseFileStatus.getPath().toUri().toString();
+ if (!candidateFileStatus.containsKey(baseFilePath)) {
+ throw new HoodieException("Error obtaining fileStatus for file: " + baseFilePath);
+ }
+ RealtimeFileStatus fileStatus = new RealtimeFileStatus(candidateFileStatus.get(baseFilePath));
+ fileStatus.setMaxCommitTime(maxCommitTime);
+ fileStatus.setBelongToIncrementalFileStatus(true);
+ fileStatus.setBasePath(basePath);
+ fileStatus.setBaseFilePath(baseFilePath);
+ fileStatus.setDeltaLogPaths(f.getLatestFileSlice().get().getLogFiles().map(l -> l.getPath().toString()).collect(Collectors.toList()));
+ // try to set bootstrapfileStatus
+ if (baseFileStatus instanceof LocatedFileStatusWithBootstrapBaseFile || baseFileStatus instanceof FileStatusWithBootstrapBaseFile) {
+ fileStatus.setBootStrapFileStatus(baseFileStatus);
+ }
+ result.add(fileStatus);
+ }
+ // add file group which has only logs.
+ if (f.getLatestFileSlice().isPresent() && baseFiles.isEmpty()) {
+ List<FileStatus> logFileStatus = f.getLatestFileSlice().get().getLogFiles().map(logFile -> logFile.getFileStatus()).collect(Collectors.toList());
+ if (logFileStatus.size() > 0) {
+ RealtimeFileStatus fileStatus = new RealtimeFileStatus(logFileStatus.get(0));
+ fileStatus.setBelongToIncrementalFileStatus(true);
+ fileStatus.setDeltaLogPaths(logFileStatus.stream().map(l -> l.getPath().toString()).collect(Collectors.toList()));
+ fileStatus.setMaxCommitTime(maxCommitTime);
+ fileStatus.setBasePath(basePath);
+ result.add(fileStatus);
+ }
+ }
+ } catch (IOException e) {
+ throw new HoodieException("Error obtaining data file/log file grouping ", e);
+ }
+ });
+ return result;
+ }
+
+ @Override
+ protected boolean includeLogFilesForSnapShotView() {
+ return true;
+ }
+
+ @Override
+ protected boolean isSplitable(FileSystem fs, Path filename) {
+ if (filename instanceof PathWithLogFilePath) {
+ return ((PathWithLogFilePath)filename).splitable();
+ }
+ return super.isSplitable(fs, filename);
+ }
+
+ @Override
+ protected FileSplit makeSplit(Path file, long start, long length, String[] hosts) {
+ if (file instanceof PathWithLogFilePath) {
+ return doMakeSplitForPathWithLogFilePath((PathWithLogFilePath) file, start, length, hosts, null);
+ }
+ return super.makeSplit(file, start, length, hosts);
+ }
+
+ @Override
+ protected FileSplit makeSplit(Path file, long start, long length, String[] hosts, String[] inMemoryHosts) {
+ if (file instanceof PathWithLogFilePath) {
+ return doMakeSplitForPathWithLogFilePath((PathWithLogFilePath) file, start, length, hosts, inMemoryHosts);
+ }
+ return super.makeSplit(file, start, length, hosts, inMemoryHosts);
+ }
+
+ private FileSplit doMakeSplitForPathWithLogFilePath(PathWithLogFilePath path, long start, long length, String[] hosts, String[] inMemoryHosts) {
+ if (path.getPathWithBootstrapFileStatus() == null) {
Review comment:
agree
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r725518350
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/BaseFileWithLogsSplit.java
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop;
+
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.apache.hadoop.mapred.FileSplit;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * we need to encode additional information in split to track matching log file and base files.
+ * Hence, this weird looking class which tracks an log/base file split
+ */
+public class BaseFileWithLogsSplit extends FileSplit {
+ // a flag to mark this split is produced by incremental query or not.
+ private boolean belongToIncrementalSplit = false;
+ // the log files of this split.
+ private List<String> deltaLogPaths = new ArrayList<>();
+ // max commit time of current split.
+ private String maxCommitTime = "";
+ // the basePath of current hoodie table.
+ private String basePath = "";
+ // the base file of this split.
+ private String baseFilePath = "";
Review comment:
we don't need bootstrap related variable in this class is it?
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieParquetRealtimeInputFormat.java
##########
@@ -66,6 +91,139 @@
return HoodieRealtimeInputFormatUtils.getRealtimeSplits(job, fileSplits);
}
+ /**
+ * keep the logical of mor_incr_view as same as spark datasource.
+ * to do: unify the incremental view code between hive/spark-sql and spark datasource
+ */
+ @Override
+ protected List<FileStatus> listStatusForIncrementalMode(
+ JobConf job, HoodieTableMetaClient tableMetaClient, List<Path> inputPaths) throws IOException {
+ List<FileStatus> result = new ArrayList<>();
+ String tableName = tableMetaClient.getTableConfig().getTableName();
+ Job jobContext = Job.getInstance(job);
+
+ Option<HoodieTimeline> timeline = HoodieInputFormatUtils.getFilteredCommitsTimeline(jobContext, tableMetaClient);
+ if (!timeline.isPresent()) {
+ return result;
+ }
+ String lastIncrementalTs = HoodieHiveUtils.readStartCommitTime(jobContext, tableName);
+ // Total number of commits to return in this batch. Set this to -1 to get all the commits.
+ Integer maxCommits = HoodieHiveUtils.readMaxCommits(jobContext, tableName);
+ HoodieTimeline commitsTimelineToReturn = timeline.get().findInstantsAfter(lastIncrementalTs, maxCommits);
+ Option<List<HoodieInstant>> commitsToCheck = Option.of(commitsTimelineToReturn.getInstants().collect(Collectors.toList()));
+ if (!commitsToCheck.isPresent()) {
+ return result;
+ }
+ Map<String, HashMap<String, FileStatus>> partitionsWithFileStatus = HoodieInputFormatUtils
Review comment:
will listAffectedFilesForCommits() also return log files or just base files ? I assume all files and so it could include log files too if touched by a commit. can you confirm please?
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieRealtimeInputFormatUtils.java
##########
@@ -161,6 +185,21 @@
return rtSplits.toArray(new InputSplit[0]);
}
+ public static RealtimeBootstrapBaseFileSplit createRealimeBootstrapBaseFileSplit(
+ BootstrapBaseFileSplit split, String basePath, List<String> deltaLogPaths, String maxInstantTime) {
+ try {
+ String[] hosts = split.getLocationInfo() != null ? Arrays.stream(split.getLocationInfo())
+ .filter(x -> !x.isInMemory()).toArray(String[]::new) : new String[0];
+ String[] inMemoryHosts = split.getLocationInfo() != null ? Arrays.stream(split.getLocationInfo())
+ .filter(SplitLocationInfo::isInMemory).toArray(String[]::new) : new String[0];
+ FileSplit baseSplit = new FileSplit(split.getPath(), split.getStart(), split.getLength(),
Review comment:
there are two constructor in Filesplit based on whether inMemoryHosts is null or not. don't we need call diff constructor here?
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieParquetRealtimeInputFormat.java
##########
@@ -66,6 +91,139 @@
return HoodieRealtimeInputFormatUtils.getRealtimeSplits(job, fileSplits);
}
+ /**
+ * keep the logical of mor_incr_view as same as spark datasource.
+ * to do: unify the incremental view code between hive/spark-sql and spark datasource
+ */
+ @Override
+ protected List<FileStatus> listStatusForIncrementalMode(
+ JobConf job, HoodieTableMetaClient tableMetaClient, List<Path> inputPaths) throws IOException {
+ List<FileStatus> result = new ArrayList<>();
+ String tableName = tableMetaClient.getTableConfig().getTableName();
+ Job jobContext = Job.getInstance(job);
+
+ Option<HoodieTimeline> timeline = HoodieInputFormatUtils.getFilteredCommitsTimeline(jobContext, tableMetaClient);
+ if (!timeline.isPresent()) {
+ return result;
+ }
+ String lastIncrementalTs = HoodieHiveUtils.readStartCommitTime(jobContext, tableName);
+ // Total number of commits to return in this batch. Set this to -1 to get all the commits.
+ Integer maxCommits = HoodieHiveUtils.readMaxCommits(jobContext, tableName);
+ HoodieTimeline commitsTimelineToReturn = timeline.get().findInstantsAfter(lastIncrementalTs, maxCommits);
+ Option<List<HoodieInstant>> commitsToCheck = Option.of(commitsTimelineToReturn.getInstants().collect(Collectors.toList()));
+ if (!commitsToCheck.isPresent()) {
+ return result;
+ }
+ Map<String, HashMap<String, FileStatus>> partitionsWithFileStatus = HoodieInputFormatUtils
+ .listAffectedFilesForCommits(new Path(tableMetaClient.getBasePath()), commitsToCheck.get(), commitsTimelineToReturn);
+ // build fileGroup from fsView
+ List<FileStatus> affectedFileStatus = new ArrayList<>();
+ partitionsWithFileStatus.forEach((key, value) -> value.forEach((k, v) -> affectedFileStatus.add(v)));
+ HoodieTableFileSystemView fsView = new HoodieTableFileSystemView(tableMetaClient, commitsTimelineToReturn, affectedFileStatus.toArray(new FileStatus[0]));
+ // build fileGroup from fsView
+ String basePath = tableMetaClient.getBasePath();
+ // filter affectedPartition by inputPaths
+ List<String> affectedPartition = partitionsWithFileStatus.keySet().stream()
+ .filter(k -> k.isEmpty() ? inputPaths.contains(new Path(basePath)) : inputPaths.contains(new Path(basePath, k))).collect(Collectors.toList());
+ if (affectedPartition.isEmpty()) {
+ return result;
+ }
+ List<HoodieFileGroup> fileGroups = affectedPartition.stream()
+ .flatMap(partitionPath -> fsView.getAllFileGroups(partitionPath)).collect(Collectors.toList());
+ setInputPaths(job, affectedPartition.stream()
+ .map(p -> p.isEmpty() ? basePath : new Path(basePath, p).toUri().toString()).collect(Collectors.joining(",")));
+
+ // find all file status in current partitionPath
Review comment:
why just current partition path. can't there be more than 1 partition paths of interest? can you help me understand whats going on here
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieParquetRealtimeInputFormat.java
##########
@@ -66,6 +91,139 @@
return HoodieRealtimeInputFormatUtils.getRealtimeSplits(job, fileSplits);
}
+ /**
+ * keep the logical of mor_incr_view as same as spark datasource.
+ * to do: unify the incremental view code between hive/spark-sql and spark datasource
+ */
+ @Override
+ protected List<FileStatus> listStatusForIncrementalMode(
+ JobConf job, HoodieTableMetaClient tableMetaClient, List<Path> inputPaths) throws IOException {
+ List<FileStatus> result = new ArrayList<>();
+ String tableName = tableMetaClient.getTableConfig().getTableName();
+ Job jobContext = Job.getInstance(job);
+
+ Option<HoodieTimeline> timeline = HoodieInputFormatUtils.getFilteredCommitsTimeline(jobContext, tableMetaClient);
+ if (!timeline.isPresent()) {
+ return result;
+ }
+ String lastIncrementalTs = HoodieHiveUtils.readStartCommitTime(jobContext, tableName);
+ // Total number of commits to return in this batch. Set this to -1 to get all the commits.
+ Integer maxCommits = HoodieHiveUtils.readMaxCommits(jobContext, tableName);
+ HoodieTimeline commitsTimelineToReturn = timeline.get().findInstantsAfter(lastIncrementalTs, maxCommits);
+ Option<List<HoodieInstant>> commitsToCheck = Option.of(commitsTimelineToReturn.getInstants().collect(Collectors.toList()));
+ if (!commitsToCheck.isPresent()) {
+ return result;
+ }
+ Map<String, HashMap<String, FileStatus>> partitionsWithFileStatus = HoodieInputFormatUtils
+ .listAffectedFilesForCommits(new Path(tableMetaClient.getBasePath()), commitsToCheck.get(), commitsTimelineToReturn);
+ // build fileGroup from fsView
+ List<FileStatus> affectedFileStatus = new ArrayList<>();
+ partitionsWithFileStatus.forEach((key, value) -> value.forEach((k, v) -> affectedFileStatus.add(v)));
+ HoodieTableFileSystemView fsView = new HoodieTableFileSystemView(tableMetaClient, commitsTimelineToReturn, affectedFileStatus.toArray(new FileStatus[0]));
+ // build fileGroup from fsView
+ String basePath = tableMetaClient.getBasePath();
Review comment:
can we create this as Path object and use toString where ever required rather than constructing new Path(basePath) everywhere.
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieParquetRealtimeInputFormat.java
##########
@@ -66,6 +91,139 @@
return HoodieRealtimeInputFormatUtils.getRealtimeSplits(job, fileSplits);
}
+ /**
+ * keep the logical of mor_incr_view as same as spark datasource.
+ * to do: unify the incremental view code between hive/spark-sql and spark datasource
+ */
+ @Override
+ protected List<FileStatus> listStatusForIncrementalMode(
+ JobConf job, HoodieTableMetaClient tableMetaClient, List<Path> inputPaths) throws IOException {
+ List<FileStatus> result = new ArrayList<>();
+ String tableName = tableMetaClient.getTableConfig().getTableName();
+ Job jobContext = Job.getInstance(job);
+
+ Option<HoodieTimeline> timeline = HoodieInputFormatUtils.getFilteredCommitsTimeline(jobContext, tableMetaClient);
+ if (!timeline.isPresent()) {
+ return result;
+ }
+ String lastIncrementalTs = HoodieHiveUtils.readStartCommitTime(jobContext, tableName);
+ // Total number of commits to return in this batch. Set this to -1 to get all the commits.
+ Integer maxCommits = HoodieHiveUtils.readMaxCommits(jobContext, tableName);
+ HoodieTimeline commitsTimelineToReturn = timeline.get().findInstantsAfter(lastIncrementalTs, maxCommits);
+ Option<List<HoodieInstant>> commitsToCheck = Option.of(commitsTimelineToReturn.getInstants().collect(Collectors.toList()));
+ if (!commitsToCheck.isPresent()) {
+ return result;
+ }
+ Map<String, HashMap<String, FileStatus>> partitionsWithFileStatus = HoodieInputFormatUtils
+ .listAffectedFilesForCommits(new Path(tableMetaClient.getBasePath()), commitsToCheck.get(), commitsTimelineToReturn);
+ // build fileGroup from fsView
+ List<FileStatus> affectedFileStatus = new ArrayList<>();
+ partitionsWithFileStatus.forEach((key, value) -> value.forEach((k, v) -> affectedFileStatus.add(v)));
+ HoodieTableFileSystemView fsView = new HoodieTableFileSystemView(tableMetaClient, commitsTimelineToReturn, affectedFileStatus.toArray(new FileStatus[0]));
+ // build fileGroup from fsView
+ String basePath = tableMetaClient.getBasePath();
+ // filter affectedPartition by inputPaths
+ List<String> affectedPartition = partitionsWithFileStatus.keySet().stream()
+ .filter(k -> k.isEmpty() ? inputPaths.contains(new Path(basePath)) : inputPaths.contains(new Path(basePath, k))).collect(Collectors.toList());
+ if (affectedPartition.isEmpty()) {
+ return result;
+ }
+ List<HoodieFileGroup> fileGroups = affectedPartition.stream()
+ .flatMap(partitionPath -> fsView.getAllFileGroups(partitionPath)).collect(Collectors.toList());
+ setInputPaths(job, affectedPartition.stream()
+ .map(p -> p.isEmpty() ? basePath : new Path(basePath, p).toUri().toString()).collect(Collectors.joining(",")));
+
+ // find all file status in current partitionPath
+ FileStatus[] fileStatuses = getStatus(job);
+ Map<String, FileStatus> candidateFileStatus = new HashMap<>();
+ for (int i = 0; i < fileStatuses.length; i++) {
+ String key = fileStatuses[i].getPath().toString();
+ candidateFileStatus.put(key, fileStatuses[i]);
+ }
+
+ String maxCommitTime = fsView.getLastInstant().get().getTimestamp();
+ fileGroups.stream().forEach(f -> {
+ try {
+ List<FileSlice> baseFiles = f.getAllFileSlices().filter(slice -> slice.getBaseFile().isPresent()).collect(Collectors.toList());
+ if (!baseFiles.isEmpty()) {
+ FileStatus baseFileStatus = HoodieInputFormatUtils.getFileStatus(baseFiles.get(0).getBaseFile().get());
+ String baseFilePath = baseFileStatus.getPath().toUri().toString();
+ if (!candidateFileStatus.containsKey(baseFilePath)) {
+ throw new HoodieException("Error obtaining fileStatus for file: " + baseFilePath);
+ }
+ RealtimeFileStatus fileStatus = new RealtimeFileStatus(candidateFileStatus.get(baseFilePath));
+ fileStatus.setMaxCommitTime(maxCommitTime);
+ fileStatus.setBelongToIncrementalFileStatus(true);
+ fileStatus.setBasePath(basePath);
+ fileStatus.setBaseFilePath(baseFilePath);
+ fileStatus.setDeltaLogPaths(f.getLatestFileSlice().get().getLogFiles().map(l -> l.getPath().toString()).collect(Collectors.toList()));
+ // try to set bootstrapfileStatus
+ if (baseFileStatus instanceof LocatedFileStatusWithBootstrapBaseFile || baseFileStatus instanceof FileStatusWithBootstrapBaseFile) {
+ fileStatus.setBootStrapFileStatus(baseFileStatus);
+ }
+ result.add(fileStatus);
+ }
+ // add file group which has only logs.
+ if (f.getLatestFileSlice().isPresent() && baseFiles.isEmpty()) {
+ List<FileStatus> logFileStatus = f.getLatestFileSlice().get().getLogFiles().map(logFile -> logFile.getFileStatus()).collect(Collectors.toList());
+ if (logFileStatus.size() > 0) {
+ RealtimeFileStatus fileStatus = new RealtimeFileStatus(logFileStatus.get(0));
+ fileStatus.setBelongToIncrementalFileStatus(true);
+ fileStatus.setDeltaLogPaths(logFileStatus.stream().map(l -> l.getPath().toString()).collect(Collectors.toList()));
+ fileStatus.setMaxCommitTime(maxCommitTime);
+ fileStatus.setBasePath(basePath);
+ result.add(fileStatus);
+ }
+ }
+ } catch (IOException e) {
+ throw new HoodieException("Error obtaining data file/log file grouping ", e);
+ }
+ });
+ return result;
+ }
+
+ @Override
+ protected boolean includeLogFilesForSnapShotView() {
+ return true;
+ }
+
+ @Override
+ protected boolean isSplitable(FileSystem fs, Path filename) {
+ if (filename instanceof PathWithLogFilePath) {
+ return ((PathWithLogFilePath)filename).splitable();
+ }
+ return super.isSplitable(fs, filename);
+ }
+
+ @Override
+ protected FileSplit makeSplit(Path file, long start, long length, String[] hosts) {
+ if (file instanceof PathWithLogFilePath) {
+ return doMakeSplitForPathWithLogFilePath((PathWithLogFilePath) file, start, length, hosts, null);
+ }
+ return super.makeSplit(file, start, length, hosts);
+ }
+
+ @Override
+ protected FileSplit makeSplit(Path file, long start, long length, String[] hosts, String[] inMemoryHosts) {
+ if (file instanceof PathWithLogFilePath) {
+ return doMakeSplitForPathWithLogFilePath((PathWithLogFilePath) file, start, length, hosts, inMemoryHosts);
+ }
+ return super.makeSplit(file, start, length, hosts, inMemoryHosts);
+ }
+
+ private FileSplit doMakeSplitForPathWithLogFilePath(PathWithLogFilePath path, long start, long length, String[] hosts, String[] inMemoryHosts) {
+ if (path.getPathWithBootstrapFileStatus() == null) {
Review comment:
we could expose a method like isBootstrapFilePath() or something in PathWithLogFilePath rather than checking for null here.
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/BaseFileWithLogsSplit.java
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop;
+
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.apache.hadoop.mapred.FileSplit;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+public class BaseFileWithLogsSplit extends FileSplit {
Review comment:
can we avoid the word "weird" in the documentation please.
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieParquetRealtimeInputFormat.java
##########
@@ -66,6 +91,139 @@
return HoodieRealtimeInputFormatUtils.getRealtimeSplits(job, fileSplits);
}
+ /**
+ * keep the logical of mor_incr_view as same as spark datasource.
+ * to do: unify the incremental view code between hive/spark-sql and spark datasource
+ */
+ @Override
+ protected List<FileStatus> listStatusForIncrementalMode(
+ JobConf job, HoodieTableMetaClient tableMetaClient, List<Path> inputPaths) throws IOException {
+ List<FileStatus> result = new ArrayList<>();
+ String tableName = tableMetaClient.getTableConfig().getTableName();
+ Job jobContext = Job.getInstance(job);
+
+ Option<HoodieTimeline> timeline = HoodieInputFormatUtils.getFilteredCommitsTimeline(jobContext, tableMetaClient);
+ if (!timeline.isPresent()) {
+ return result;
+ }
+ String lastIncrementalTs = HoodieHiveUtils.readStartCommitTime(jobContext, tableName);
+ // Total number of commits to return in this batch. Set this to -1 to get all the commits.
+ Integer maxCommits = HoodieHiveUtils.readMaxCommits(jobContext, tableName);
+ HoodieTimeline commitsTimelineToReturn = timeline.get().findInstantsAfter(lastIncrementalTs, maxCommits);
+ Option<List<HoodieInstant>> commitsToCheck = Option.of(commitsTimelineToReturn.getInstants().collect(Collectors.toList()));
+ if (!commitsToCheck.isPresent()) {
+ return result;
+ }
+ Map<String, HashMap<String, FileStatus>> partitionsWithFileStatus = HoodieInputFormatUtils
+ .listAffectedFilesForCommits(new Path(tableMetaClient.getBasePath()), commitsToCheck.get(), commitsTimelineToReturn);
+ // build fileGroup from fsView
+ List<FileStatus> affectedFileStatus = new ArrayList<>();
+ partitionsWithFileStatus.forEach((key, value) -> value.forEach((k, v) -> affectedFileStatus.add(v)));
+ HoodieTableFileSystemView fsView = new HoodieTableFileSystemView(tableMetaClient, commitsTimelineToReturn, affectedFileStatus.toArray(new FileStatus[0]));
+ // build fileGroup from fsView
+ String basePath = tableMetaClient.getBasePath();
+ // filter affectedPartition by inputPaths
+ List<String> affectedPartition = partitionsWithFileStatus.keySet().stream()
+ .filter(k -> k.isEmpty() ? inputPaths.contains(new Path(basePath)) : inputPaths.contains(new Path(basePath, k))).collect(Collectors.toList());
+ if (affectedPartition.isEmpty()) {
+ return result;
+ }
+ List<HoodieFileGroup> fileGroups = affectedPartition.stream()
+ .flatMap(partitionPath -> fsView.getAllFileGroups(partitionPath)).collect(Collectors.toList());
+ setInputPaths(job, affectedPartition.stream()
+ .map(p -> p.isEmpty() ? basePath : new Path(basePath, p).toUri().toString()).collect(Collectors.joining(",")));
+
+ // find all file status in current partitionPath
+ FileStatus[] fileStatuses = getStatus(job);
+ Map<String, FileStatus> candidateFileStatus = new HashMap<>();
+ for (int i = 0; i < fileStatuses.length; i++) {
+ String key = fileStatuses[i].getPath().toString();
+ candidateFileStatus.put(key, fileStatuses[i]);
+ }
+
+ String maxCommitTime = fsView.getLastInstant().get().getTimestamp();
Review comment:
this method is too long. Can we try to make it concise (may be slice into some smaller methods).
anyways, since we do lot of things here, can we document in java docs (line 95) as to what all we do in this method for maintanence.
Here is what I could infer. please correct it and add to java docs.
1. Get list of commits to be fetched based on start commit and max commits (for snapshot max commits is -1)
2. Get list of affected file status for these commits of interest
3. Construct HoodieTableFileSystemVeiw based on this affected file status.
a. Filter for affected partitions based in inputPaths
b. Get list of fileGroups based on affected partitions and fsView.getAllFileGroups
4. Set input paths based on filtered affected partition paths. chances that among original input paths passed to this method, some partitions did not have commits as part of the trimmed down list of commits and hence we need this step.
5. Find candidate fileStatus. // I don't understand what are we doing here. can you help me with it.
6. for every file group from 3(b)
Get 1st available base file from all file slices. If part of candidate file status, construct RealtimeFileStatus and add it to result along with log files.
If file group just has log files w/o base file and part of candidate file status, construct RealtimeFileStatus and add it to result
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/RealtimeCompactedRecordReader.java
##########
@@ -102,97 +102,107 @@ private HoodieMergedLogRecordScanner getMergedLogRecordScanner() throws IOExcept
@Override
public boolean next(NullWritable aVoid, ArrayWritable arrayWritable) throws IOException {
+ // deal with DeltaOnlySplits
+ if (logReader.isPresent()) {
+ return logReader.get().next(aVoid, arrayWritable);
+ }
// Call the underlying parquetReader.next - which may replace the passed in ArrayWritable
// with a new block of values
- while (this.parquetReader.next(aVoid, arrayWritable)) {
- if (!deltaRecordMap.isEmpty()) {
- String key = arrayWritable.get()[recordKeyIndex].toString();
- if (deltaRecordMap.containsKey(key)) {
- // mark the key as handled
- this.deltaRecordKeys.remove(key);
- // TODO(NA): Invoke preCombine here by converting arrayWritable to Avro. This is required since the
- // deltaRecord may not be a full record and needs values of columns from the parquet
- Option<GenericRecord> rec = buildGenericRecordwithCustomPayload(deltaRecordMap.get(key));
- // If the record is not present, this is a delete record using an empty payload so skip this base record
- // and move to the next record
- if (!rec.isPresent()) {
- continue;
+ boolean result = this.parquetReader.next(aVoid, arrayWritable);
+ if (!result) {
+ // if the result is false, then there are no more records
+ return false;
+ }
+ if (!deltaRecordMap.isEmpty()) {
+ // TODO(VC): Right now, we assume all records in log, have a matching base record. (which
+ // would be true until we have a way to index logs too)
Review comment:
guess if someone uses hbase index, it may not be the case.
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/RealtimeCompactedRecordReader.java
##########
@@ -102,97 +102,107 @@ private HoodieMergedLogRecordScanner getMergedLogRecordScanner() throws IOExcept
@Override
public boolean next(NullWritable aVoid, ArrayWritable arrayWritable) throws IOException {
+ // deal with DeltaOnlySplits
+ if (logReader.isPresent()) {
+ return logReader.get().next(aVoid, arrayWritable);
+ }
// Call the underlying parquetReader.next - which may replace the passed in ArrayWritable
// with a new block of values
- while (this.parquetReader.next(aVoid, arrayWritable)) {
- if (!deltaRecordMap.isEmpty()) {
- String key = arrayWritable.get()[recordKeyIndex].toString();
- if (deltaRecordMap.containsKey(key)) {
- // mark the key as handled
- this.deltaRecordKeys.remove(key);
- // TODO(NA): Invoke preCombine here by converting arrayWritable to Avro. This is required since the
- // deltaRecord may not be a full record and needs values of columns from the parquet
- Option<GenericRecord> rec = buildGenericRecordwithCustomPayload(deltaRecordMap.get(key));
- // If the record is not present, this is a delete record using an empty payload so skip this base record
- // and move to the next record
- if (!rec.isPresent()) {
- continue;
+ boolean result = this.parquetReader.next(aVoid, arrayWritable);
+ if (!result) {
+ // if the result is false, then there are no more records
+ return false;
Review comment:
if a Realtime split consists of a base file and few log files, we can't return here? or am I misunderstanding something
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/fs/FSUtils.java
##########
@@ -336,6 +336,11 @@ public static String getFileExtensionFromLog(Path logPath) {
return matcher.group(3);
}
+ public static String getLogFileExtension(String fullName) {
+ Matcher matcher = LOG_FILE_PATTERN.matcher(fullName);
Review comment:
can we reuse variable for ".log"
HoodieLogFile.DELTA_EXTENSION
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/RealtimeFileStatus.java
##########
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+public class RealtimeFileStatus extends FileStatus {
Review comment:
java docs.
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieParquetRealtimeInputFormat.java
##########
@@ -66,6 +91,139 @@
return HoodieRealtimeInputFormatUtils.getRealtimeSplits(job, fileSplits);
}
+ /**
+ * keep the logical of mor_incr_view as same as spark datasource.
+ * to do: unify the incremental view code between hive/spark-sql and spark datasource
+ */
+ @Override
+ protected List<FileStatus> listStatusForIncrementalMode(
+ JobConf job, HoodieTableMetaClient tableMetaClient, List<Path> inputPaths) throws IOException {
+ List<FileStatus> result = new ArrayList<>();
+ String tableName = tableMetaClient.getTableConfig().getTableName();
+ Job jobContext = Job.getInstance(job);
+
+ Option<HoodieTimeline> timeline = HoodieInputFormatUtils.getFilteredCommitsTimeline(jobContext, tableMetaClient);
+ if (!timeline.isPresent()) {
+ return result;
+ }
+ String lastIncrementalTs = HoodieHiveUtils.readStartCommitTime(jobContext, tableName);
+ // Total number of commits to return in this batch. Set this to -1 to get all the commits.
+ Integer maxCommits = HoodieHiveUtils.readMaxCommits(jobContext, tableName);
+ HoodieTimeline commitsTimelineToReturn = timeline.get().findInstantsAfter(lastIncrementalTs, maxCommits);
+ Option<List<HoodieInstant>> commitsToCheck = Option.of(commitsTimelineToReturn.getInstants().collect(Collectors.toList()));
+ if (!commitsToCheck.isPresent()) {
+ return result;
+ }
+ Map<String, HashMap<String, FileStatus>> partitionsWithFileStatus = HoodieInputFormatUtils
+ .listAffectedFilesForCommits(new Path(tableMetaClient.getBasePath()), commitsToCheck.get(), commitsTimelineToReturn);
+ // build fileGroup from fsView
+ List<FileStatus> affectedFileStatus = new ArrayList<>();
+ partitionsWithFileStatus.forEach((key, value) -> value.forEach((k, v) -> affectedFileStatus.add(v)));
+ HoodieTableFileSystemView fsView = new HoodieTableFileSystemView(tableMetaClient, commitsTimelineToReturn, affectedFileStatus.toArray(new FileStatus[0]));
+ // build fileGroup from fsView
+ String basePath = tableMetaClient.getBasePath();
+ // filter affectedPartition by inputPaths
+ List<String> affectedPartition = partitionsWithFileStatus.keySet().stream()
+ .filter(k -> k.isEmpty() ? inputPaths.contains(new Path(basePath)) : inputPaths.contains(new Path(basePath, k))).collect(Collectors.toList());
+ if (affectedPartition.isEmpty()) {
+ return result;
+ }
+ List<HoodieFileGroup> fileGroups = affectedPartition.stream()
+ .flatMap(partitionPath -> fsView.getAllFileGroups(partitionPath)).collect(Collectors.toList());
+ setInputPaths(job, affectedPartition.stream()
+ .map(p -> p.isEmpty() ? basePath : new Path(basePath, p).toUri().toString()).collect(Collectors.joining(",")));
+
+ // find all file status in current partitionPath
+ FileStatus[] fileStatuses = getStatus(job);
+ Map<String, FileStatus> candidateFileStatus = new HashMap<>();
+ for (int i = 0; i < fileStatuses.length; i++) {
+ String key = fileStatuses[i].getPath().toString();
+ candidateFileStatus.put(key, fileStatuses[i]);
+ }
+
+ String maxCommitTime = fsView.getLastInstant().get().getTimestamp();
+ fileGroups.stream().forEach(f -> {
+ try {
+ List<FileSlice> baseFiles = f.getAllFileSlices().filter(slice -> slice.getBaseFile().isPresent()).collect(Collectors.toList());
+ if (!baseFiles.isEmpty()) {
+ FileStatus baseFileStatus = HoodieInputFormatUtils.getFileStatus(baseFiles.get(0).getBaseFile().get());
+ String baseFilePath = baseFileStatus.getPath().toUri().toString();
+ if (!candidateFileStatus.containsKey(baseFilePath)) {
+ throw new HoodieException("Error obtaining fileStatus for file: " + baseFilePath);
+ }
+ RealtimeFileStatus fileStatus = new RealtimeFileStatus(candidateFileStatus.get(baseFilePath));
+ fileStatus.setMaxCommitTime(maxCommitTime);
+ fileStatus.setBelongToIncrementalFileStatus(true);
+ fileStatus.setBasePath(basePath);
+ fileStatus.setBaseFilePath(baseFilePath);
+ fileStatus.setDeltaLogPaths(f.getLatestFileSlice().get().getLogFiles().map(l -> l.getPath().toString()).collect(Collectors.toList()));
Review comment:
I see we call f.getAllFileSlices() at line 147, where as here we call f.getLatestFileSlice(). is that intentional?
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieParquetRealtimeInputFormat.java
##########
@@ -66,6 +91,139 @@
return HoodieRealtimeInputFormatUtils.getRealtimeSplits(job, fileSplits);
}
+ /**
+ * keep the logical of mor_incr_view as same as spark datasource.
+ * to do: unify the incremental view code between hive/spark-sql and spark datasource
+ */
+ @Override
+ protected List<FileStatus> listStatusForIncrementalMode(
+ JobConf job, HoodieTableMetaClient tableMetaClient, List<Path> inputPaths) throws IOException {
+ List<FileStatus> result = new ArrayList<>();
+ String tableName = tableMetaClient.getTableConfig().getTableName();
+ Job jobContext = Job.getInstance(job);
+
+ Option<HoodieTimeline> timeline = HoodieInputFormatUtils.getFilteredCommitsTimeline(jobContext, tableMetaClient);
+ if (!timeline.isPresent()) {
+ return result;
+ }
+ String lastIncrementalTs = HoodieHiveUtils.readStartCommitTime(jobContext, tableName);
+ // Total number of commits to return in this batch. Set this to -1 to get all the commits.
+ Integer maxCommits = HoodieHiveUtils.readMaxCommits(jobContext, tableName);
+ HoodieTimeline commitsTimelineToReturn = timeline.get().findInstantsAfter(lastIncrementalTs, maxCommits);
+ Option<List<HoodieInstant>> commitsToCheck = Option.of(commitsTimelineToReturn.getInstants().collect(Collectors.toList()));
+ if (!commitsToCheck.isPresent()) {
+ return result;
+ }
+ Map<String, HashMap<String, FileStatus>> partitionsWithFileStatus = HoodieInputFormatUtils
+ .listAffectedFilesForCommits(new Path(tableMetaClient.getBasePath()), commitsToCheck.get(), commitsTimelineToReturn);
+ // build fileGroup from fsView
+ List<FileStatus> affectedFileStatus = new ArrayList<>();
+ partitionsWithFileStatus.forEach((key, value) -> value.forEach((k, v) -> affectedFileStatus.add(v)));
+ HoodieTableFileSystemView fsView = new HoodieTableFileSystemView(tableMetaClient, commitsTimelineToReturn, affectedFileStatus.toArray(new FileStatus[0]));
+ // build fileGroup from fsView
+ String basePath = tableMetaClient.getBasePath();
+ // filter affectedPartition by inputPaths
+ List<String> affectedPartition = partitionsWithFileStatus.keySet().stream()
+ .filter(k -> k.isEmpty() ? inputPaths.contains(new Path(basePath)) : inputPaths.contains(new Path(basePath, k))).collect(Collectors.toList());
Review comment:
whats the scenario where k.isEmpty() will be true? is it non partitioned case?
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/RealTimeMergedRecordReader.java
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop.realtime;
+
+import java.io.IOException;
+import java.text.MessageFormat;
+import java.util.Iterator;
+
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.io.ArrayWritable;
+import org.apache.hadoop.io.NullWritable;
+import org.apache.hadoop.io.Writable;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.RecordReader;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.hadoop.utils.HoodieRealtimeRecordReaderUtils;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+/**
+ * Record Reader implementation to read avro data, to support inc queries.
+ */
+public class RealTimeMergedRecordReader extends AbstractRealtimeRecordReader
+ implements RecordReader<NullWritable, ArrayWritable> {
+ private static final Logger LOG = LogManager.getLogger(AbstractRealtimeRecordReader.class);
+ private final HoodieMergedLogRecordScanner logRecordScanner;
+ private final Iterator<HoodieRecord<? extends HoodieRecordPayload>> iterator;
+ private ArrayWritable valueObj;
+
+ private int end;
+ private int offset;
+
+ public RealTimeMergedRecordReader(RealtimeSplit split, JobConf job, HoodieMergedLogRecordScanner logRecordScanner) {
+ super(split, job);
+ this.logRecordScanner = logRecordScanner;
+ this.end = logRecordScanner.getRecords().size();
+ this.iterator = logRecordScanner.iterator();
+ this.valueObj = new ArrayWritable(Writable.class, new Writable[getHiveSchema().getFields().size()]);
+ }
+
+ private Option buildGenericRecord(HoodieRecord record) throws IOException {
+ if (usesCustomPayload) {
+ return record.getData().getInsertValue(getWriterSchema());
+ } else {
+ return record.getData().getInsertValue(getReaderSchema());
+ }
+ }
+
+ @Override
+ public boolean next(NullWritable key, ArrayWritable value) throws IOException {
Review comment:
have asked a clarification question in RealtimeCompactedRecordReader. will review this once I get those clarified.
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieParquetRealtimeInputFormat.java
##########
@@ -66,6 +91,139 @@
return HoodieRealtimeInputFormatUtils.getRealtimeSplits(job, fileSplits);
}
+ /**
+ * keep the logical of mor_incr_view as same as spark datasource.
+ * to do: unify the incremental view code between hive/spark-sql and spark datasource
+ */
+ @Override
+ protected List<FileStatus> listStatusForIncrementalMode(
+ JobConf job, HoodieTableMetaClient tableMetaClient, List<Path> inputPaths) throws IOException {
+ List<FileStatus> result = new ArrayList<>();
+ String tableName = tableMetaClient.getTableConfig().getTableName();
+ Job jobContext = Job.getInstance(job);
+
+ Option<HoodieTimeline> timeline = HoodieInputFormatUtils.getFilteredCommitsTimeline(jobContext, tableMetaClient);
+ if (!timeline.isPresent()) {
+ return result;
+ }
+ String lastIncrementalTs = HoodieHiveUtils.readStartCommitTime(jobContext, tableName);
+ // Total number of commits to return in this batch. Set this to -1 to get all the commits.
+ Integer maxCommits = HoodieHiveUtils.readMaxCommits(jobContext, tableName);
+ HoodieTimeline commitsTimelineToReturn = timeline.get().findInstantsAfter(lastIncrementalTs, maxCommits);
Review comment:
can you help me understand, if there was a replace commit made, where exactly the filtering will happen? does findInstantsAfter will take care of that? or is the code is else where?
let me illustrate.
lets say these are the commits.
c1, c2, replaceCommit3, c4, c5
If someone does incremental with start commit c2 and asks for 1 commit, we will return just c2.
if someone queries for incremental with start commit c2 and asks for 3 commits, we should return just replaceCommit3 and c4 right?
do I make sense or am I getting it wrong ?
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieParquetRealtimeInputFormat.java
##########
@@ -66,6 +91,139 @@
return HoodieRealtimeInputFormatUtils.getRealtimeSplits(job, fileSplits);
}
+ /**
+ * keep the logical of mor_incr_view as same as spark datasource.
+ * to do: unify the incremental view code between hive/spark-sql and spark datasource
+ */
+ @Override
+ protected List<FileStatus> listStatusForIncrementalMode(
+ JobConf job, HoodieTableMetaClient tableMetaClient, List<Path> inputPaths) throws IOException {
Review comment:
inputPaths can only contain partition paths right and not any log files? is my understanding right.
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/RealtimeCompactedRecordReader.java
##########
@@ -52,16 +49,19 @@
protected final RecordReader<NullWritable, ArrayWritable> parquetReader;
private final Map<String, HoodieRecord<? extends HoodieRecordPayload>> deltaRecordMap;
- private final Set<String> deltaRecordKeys;
private int recordKeyIndex = HoodieInputFormatUtils.HOODIE_RECORD_KEY_COL_POS;
- private Iterator<String> deltaItr;
+ private Option<RealTimeMergedRecordReader> logReader;
public RealtimeCompactedRecordReader(RealtimeSplit split, JobConf job,
RecordReader<NullWritable, ArrayWritable> realReader) throws IOException {
super(split, job);
this.parquetReader = realReader;
- this.deltaRecordMap = getMergedLogRecordScanner().getRecords();
- this.deltaRecordKeys = new HashSet<>(this.deltaRecordMap.keySet());
+ HoodieMergedLogRecordScanner hoodieMergedLogRecordScanner = getMergedLogRecordScanner();
+ this.deltaRecordMap = hoodieMergedLogRecordScanner.getRecords();
+ this.logReader = Option.empty();
+ if (FSUtils.isLogFile(split.getPath())) {
Review comment:
help me clarify something.
A RealtimeSplit can be any of the 3 right?
a. just base file.
b. base file and few log files.
c. just log files.
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/RealtimeCompactedRecordReader.java
##########
@@ -52,16 +49,19 @@
protected final RecordReader<NullWritable, ArrayWritable> parquetReader;
private final Map<String, HoodieRecord<? extends HoodieRecordPayload>> deltaRecordMap;
- private final Set<String> deltaRecordKeys;
private int recordKeyIndex = HoodieInputFormatUtils.HOODIE_RECORD_KEY_COL_POS;
- private Iterator<String> deltaItr;
+ private Option<RealTimeMergedRecordReader> logReader;
public RealtimeCompactedRecordReader(RealtimeSplit split, JobConf job,
RecordReader<NullWritable, ArrayWritable> realReader) throws IOException {
super(split, job);
this.parquetReader = realReader;
- this.deltaRecordMap = getMergedLogRecordScanner().getRecords();
- this.deltaRecordKeys = new HashSet<>(this.deltaRecordMap.keySet());
+ HoodieMergedLogRecordScanner hoodieMergedLogRecordScanner = getMergedLogRecordScanner();
+ this.deltaRecordMap = hoodieMergedLogRecordScanner.getRecords();
+ this.logReader = Option.empty();
+ if (FSUtils.isLogFile(split.getPath())) {
Review comment:
will review this class once I get clarification on above as its key for my review.
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/RealtimeCompactedRecordReader.java
##########
@@ -52,16 +49,19 @@
protected final RecordReader<NullWritable, ArrayWritable> parquetReader;
private final Map<String, HoodieRecord<? extends HoodieRecordPayload>> deltaRecordMap;
- private final Set<String> deltaRecordKeys;
private int recordKeyIndex = HoodieInputFormatUtils.HOODIE_RECORD_KEY_COL_POS;
- private Iterator<String> deltaItr;
+ private Option<RealTimeMergedRecordReader> logReader;
public RealtimeCompactedRecordReader(RealtimeSplit split, JobConf job,
RecordReader<NullWritable, ArrayWritable> realReader) throws IOException {
super(split, job);
this.parquetReader = realReader;
- this.deltaRecordMap = getMergedLogRecordScanner().getRecords();
- this.deltaRecordKeys = new HashSet<>(this.deltaRecordMap.keySet());
+ HoodieMergedLogRecordScanner hoodieMergedLogRecordScanner = getMergedLogRecordScanner();
+ this.deltaRecordMap = hoodieMergedLogRecordScanner.getRecords();
+ this.logReader = Option.empty();
+ if (FSUtils.isLogFile(split.getPath())) {
Review comment:
and only incase of (c), FSUtils.isLogFile(split.getPath()) will return true?
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieRealtimeInputFormatUtils.java
##########
@@ -99,7 +101,32 @@
}
}
Option<HoodieVirtualKeyInfo> finalHoodieVirtualKeyInfo = hoodieVirtualKeyInfo;
- partitionsToParquetSplits.keySet().forEach(partitionPath -> {
+ // deal with incremental query
+ candidateFileSplits.stream().forEach(s -> {
+ try {
Review comment:
guess what he meant is, we are iterating through candidateFileSplits twice, once to populate rtSplits and another time to populate filteredPartitionsToParquetSplits. can we see if we can do it in one iteration? @dannyhchen : correct me if you meant something else.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
}, {
"hash" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325",
"triggerID" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"triggerType" : "PUSH"
}, {
"hash" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2343",
"triggerID" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"triggerType" : "PUSH"
}, {
"hash" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2424",
"triggerID" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2427",
"triggerID" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2429",
"triggerID" : "927749497",
"triggerType" : "MANUAL"
}, {
"hash" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2431",
"triggerID" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"triggerType" : "PUSH"
}, {
"hash" : "29e6acb65044597e35f9d499385dad940580d9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2432",
"triggerID" : "29e6acb65044597e35f9d499385dad940580d9df",
"triggerType" : "PUSH"
}, {
"hash" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2441",
"triggerID" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"triggerType" : "PUSH"
}, {
"hash" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2443",
"triggerID" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2447",
"triggerID" : "928890183",
"triggerType" : "MANUAL"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0fa6297ce58eb877fd5c4eba59fef20ad9335d26 UNKNOWN
* 423f047e00904b17735a5f0120959de87e4cb0a3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2443) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2447)
* c14bded2e903e9c55d86fddc77d40effccca5e01 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-928890183
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r717259013
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/HoodieParquetInputFormat.java
##########
@@ -73,6 +73,14 @@ protected HoodieDefaultTimeline filterInstantsTimeline(HoodieDefaultTimeline tim
return HoodieInputFormatUtils.filterInstantsTimeline(timeline);
}
+ protected FileStatus[] getStatus(JobConf job) throws IOException {
Review comment:
no, we need to call ParquetInputFormat.listStatus in HoodieParquetRealtimeInputFormat.java, but we cannot access that method directly, so we overwrite getStatus in ParquetRealtimeInputFormat.java call call listStatus in it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a change in pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
danny0405 commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r737998861
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieRealtimeInputFormatUtils.java
##########
@@ -161,6 +148,110 @@
return rtSplits.toArray(new InputSplit[0]);
}
+ // get IncrementalRealtimeSplits
+ public static InputSplit[] getIncrementalRealtimeSplits(Configuration conf, Stream<FileSplit> fileSplits) throws IOException {
+ List<InputSplit> rtSplits = new ArrayList<>();
+ List<FileSplit> fileSplitList = fileSplits.collect(Collectors.toList());
+ Set<Path> partitionSet = fileSplitList.stream().map(f -> f.getPath().getParent()).collect(Collectors.toSet());
+ Map<Path, HoodieTableMetaClient> partitionsToMetaClient = getTableMetaClientByPartitionPath(conf, partitionSet);
+ // Pre process tableConfig from first partition to fetch virtual key info
+ Option<HoodieVirtualKeyInfo> hoodieVirtualKeyInfo = Option.empty();
+ if (partitionSet.size() > 0) {
+ hoodieVirtualKeyInfo = getHoodieVirtualKeyInfo(partitionsToMetaClient.get(partitionSet.iterator().next()));
+ }
+ Option<HoodieVirtualKeyInfo> finalHoodieVirtualKeyInfo = hoodieVirtualKeyInfo;
+ fileSplitList.stream().forEach(s -> {
+ // deal with incremental query.
+ try {
+ if (s instanceof BaseFileWithLogsSplit) {
+ BaseFileWithLogsSplit bs = (BaseFileWithLogsSplit)s;
+ if (bs.getBelongToIncrementalSplit()) {
+ rtSplits.add(new HoodieRealtimeFileSplit(bs, bs.getBasePath(), bs.getDeltaLogPaths(), bs.getMaxCommitTime(), finalHoodieVirtualKeyInfo));
+ }
+ } else if (s instanceof RealtimeBootstrapBaseFileSplit) {
+ rtSplits.add(s);
+ }
+ } catch (IOException e) {
+ throw new HoodieIOException("Error creating hoodie real time split ", e);
+ }
+ });
+ LOG.info("Returning a total splits of " + rtSplits.size());
+ return rtSplits.toArray(new InputSplit[0]);
+ }
+
+ public static Option<HoodieVirtualKeyInfo> getHoodieVirtualKeyInfo(HoodieTableMetaClient metaClient) {
+ HoodieTableConfig tableConfig = metaClient.getTableConfig();
+ if (!tableConfig.populateMetaFields()) {
+ TableSchemaResolver tableSchemaResolver = new TableSchemaResolver(metaClient);
+ try {
+ MessageType parquetSchema = tableSchemaResolver.getTableParquetSchema();
+ return Option.of(new HoodieVirtualKeyInfo(tableConfig.getRecordKeyFieldProp(),
+ tableConfig.getPartitionFieldProp(), parquetSchema.getFieldIndex(tableConfig.getRecordKeyFieldProp()),
+ parquetSchema.getFieldIndex(tableConfig.getPartitionFieldProp())));
+ } catch (Exception exception) {
+ throw new HoodieException("Fetching table schema failed with exception ", exception);
+ }
+ }
+ return Option.empty();
+ }
+
+ public static boolean isIncrementalQuerySplits(List<FileSplit> fileSplits) {
+ if (fileSplits == null || fileSplits.size() == 0) {
+ return false;
+ }
+ return fileSplits.stream().anyMatch(s -> {
+ if (s instanceof BaseFileWithLogsSplit) {
+ BaseFileWithLogsSplit bs = (BaseFileWithLogsSplit)s;
+ if (bs.getBelongToIncrementalSplit()) {
+ return true;
+ }
+ } else if (s instanceof RealtimeBootstrapBaseFileSplit) {
+ return true;
+ }
+ return false;
+ });
+ }
+
+ // Pick all incremental files and add them to rtSplits then filter out those files.
+ private static Map<Path, List<FileSplit>> filterOutIncrementalSplits(
+ List<FileSplit> fileSplitList,
Review comment:
The code is useless now: `#filterOutIncrementalSplits`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
}, {
"hash" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325",
"triggerID" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"triggerType" : "PUSH"
}, {
"hash" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2343",
"triggerID" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"triggerType" : "PUSH"
}, {
"hash" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2424",
"triggerID" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2427",
"triggerID" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2429",
"triggerID" : "927749497",
"triggerType" : "MANUAL"
}, {
"hash" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2431",
"triggerID" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"triggerType" : "PUSH"
}, {
"hash" : "29e6acb65044597e35f9d499385dad940580d9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2432",
"triggerID" : "29e6acb65044597e35f9d499385dad940580d9df",
"triggerType" : "PUSH"
}, {
"hash" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2441",
"triggerID" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"triggerType" : "PUSH"
}, {
"hash" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2443",
"triggerID" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2447",
"triggerID" : "928890183",
"triggerType" : "MANUAL"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2470",
"triggerID" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"triggerType" : "PUSH"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2560",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562",
"triggerID" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"triggerType" : "PUSH"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
}, {
"hash" : "6fa031e1dcf6e21dfb24cf4296cb00d910b0a204",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2589",
"triggerID" : "6fa031e1dcf6e21dfb24cf4296cb00d910b0a204",
"triggerType" : "PUSH"
}, {
"hash" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2593",
"triggerID" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"triggerType" : "PUSH"
}, {
"hash" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2604",
"triggerID" : "941865271",
"triggerType" : "MANUAL"
}, {
"hash" : "6d67b68e19a43f8668e5773d27ca9c33a8de0a37",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2642",
"triggerID" : "6d67b68e19a43f8668e5773d27ca9c33a8de0a37",
"triggerType" : "PUSH"
}, {
"hash" : "1a80559bd98829552acffdaf20d3ea0384d1d936",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2645",
"triggerID" : "1a80559bd98829552acffdaf20d3ea0384d1d936",
"triggerType" : "PUSH"
}, {
"hash" : "c4af04dab3dab31ef05ba6007000738a2dfb81ce",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2649",
"triggerID" : "c4af04dab3dab31ef05ba6007000738a2dfb81ce",
"triggerType" : "PUSH"
}, {
"hash" : "369c5d6f1f4b6eed3c86f838fb6148d127cc6703",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2660",
"triggerID" : "369c5d6f1f4b6eed3c86f838fb6148d127cc6703",
"triggerType" : "PUSH"
}, {
"hash" : "369c5d6f1f4b6eed3c86f838fb6148d127cc6703",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2898",
"triggerID" : "952859446",
"triggerType" : "MANUAL"
}, {
"hash" : "196a40e3273e98f2317856a55de3bd1888e9f754",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2899",
"triggerID" : "196a40e3273e98f2317856a55de3bd1888e9f754",
"triggerType" : "PUSH"
}, {
"hash" : "196a40e3273e98f2317856a55de3bd1888e9f754",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2899",
"triggerID" : "952859446",
"triggerType" : "MANUAL"
}, {
"hash" : "633b7370155185dead726ddfafd773e284ed510c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2935",
"triggerID" : "633b7370155185dead726ddfafd773e284ed510c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0fa6297ce58eb877fd5c4eba59fef20ad9335d26 UNKNOWN
* 196a40e3273e98f2317856a55de3bd1888e9f754 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2899)
* 633b7370155185dead726ddfafd773e284ed510c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2935)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-957530371
@codope thanks for your review.
Adressed all comments。
Add test case for compaction to verify that incremental reads do NOT show inserts after compaction timestamp， other situations are currently covered by hudi.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-961589519
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
}, {
"hash" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325",
"triggerID" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"triggerType" : "PUSH"
}, {
"hash" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2343",
"triggerID" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"triggerType" : "PUSH"
}, {
"hash" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2424",
"triggerID" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2427",
"triggerID" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2429",
"triggerID" : "927749497",
"triggerType" : "MANUAL"
}, {
"hash" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2431",
"triggerID" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"triggerType" : "PUSH"
}, {
"hash" : "29e6acb65044597e35f9d499385dad940580d9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2432",
"triggerID" : "29e6acb65044597e35f9d499385dad940580d9df",
"triggerType" : "PUSH"
}, {
"hash" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2441",
"triggerID" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"triggerType" : "PUSH"
}, {
"hash" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2443",
"triggerID" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2447",
"triggerID" : "928890183",
"triggerType" : "MANUAL"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2470",
"triggerID" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"triggerType" : "PUSH"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2560",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562",
"triggerID" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"triggerType" : "PUSH"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
}, {
"hash" : "6fa031e1dcf6e21dfb24cf4296cb00d910b0a204",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2589",
"triggerID" : "6fa031e1dcf6e21dfb24cf4296cb00d910b0a204",
"triggerType" : "PUSH"
}, {
"hash" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2593",
"triggerID" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"triggerType" : "PUSH"
}, {
"hash" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2604",
"triggerID" : "941865271",
"triggerType" : "MANUAL"
}, {
"hash" : "6d67b68e19a43f8668e5773d27ca9c33a8de0a37",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2642",
"triggerID" : "6d67b68e19a43f8668e5773d27ca9c33a8de0a37",
"triggerType" : "PUSH"
}, {
"hash" : "1a80559bd98829552acffdaf20d3ea0384d1d936",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2645",
"triggerID" : "1a80559bd98829552acffdaf20d3ea0384d1d936",
"triggerType" : "PUSH"
}, {
"hash" : "c4af04dab3dab31ef05ba6007000738a2dfb81ce",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2649",
"triggerID" : "c4af04dab3dab31ef05ba6007000738a2dfb81ce",
"triggerType" : "PUSH"
}, {
"hash" : "369c5d6f1f4b6eed3c86f838fb6148d127cc6703",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2660",
"triggerID" : "369c5d6f1f4b6eed3c86f838fb6148d127cc6703",
"triggerType" : "PUSH"
}, {
"hash" : "369c5d6f1f4b6eed3c86f838fb6148d127cc6703",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2898",
"triggerID" : "952859446",
"triggerType" : "MANUAL"
}, {
"hash" : "196a40e3273e98f2317856a55de3bd1888e9f754",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2899",
"triggerID" : "196a40e3273e98f2317856a55de3bd1888e9f754",
"triggerType" : "PUSH"
}, {
"hash" : "196a40e3273e98f2317856a55de3bd1888e9f754",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2899",
"triggerID" : "952859446",
"triggerType" : "MANUAL"
}, {
"hash" : "633b7370155185dead726ddfafd773e284ed510c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2935",
"triggerID" : "633b7370155185dead726ddfafd773e284ed510c",
"triggerType" : "PUSH"
}, {
"hash" : "2182f542acd4ae3d1fd239e6441e9761eb2d33a1",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3069",
"triggerID" : "2182f542acd4ae3d1fd239e6441e9761eb2d33a1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0fa6297ce58eb877fd5c4eba59fef20ad9335d26 UNKNOWN
* 2182f542acd4ae3d1fd239e6441e9761eb2d33a1 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3069)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
codope commented on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-963247562
> @nsivabalan @codope for this test case https://gist.github.com/nsivabalan/950bdde0b3b64349bf0395b2e3e928d8, when sync to hive , Why use "_ hoodie_ partition_ Path" as partition instead of "driver". L102 in HoodieParquetRealtimeInputFormat will add "_ hoodie_ partition_ Path" in RequiedProjectionFields the method HoodieRealtimeRecordReaderUtils.orderFields will remove "_ hoodie_ partition_ Path" from projectCols since partitionFiedls contains this col. then L264 will throw Exception. do you think we need fixed that problem, If necessary, I will raise an additional JIRA to track this issue
Good point @xiarixiaoyao
Will #3630 fix this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
}, {
"hash" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325",
"triggerID" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"triggerType" : "PUSH"
}, {
"hash" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2343",
"triggerID" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"triggerType" : "PUSH"
}, {
"hash" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2424",
"triggerID" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2427",
"triggerID" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2429",
"triggerID" : "927749497",
"triggerType" : "MANUAL"
}, {
"hash" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2431",
"triggerID" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"triggerType" : "PUSH"
}, {
"hash" : "29e6acb65044597e35f9d499385dad940580d9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2432",
"triggerID" : "29e6acb65044597e35f9d499385dad940580d9df",
"triggerType" : "PUSH"
}, {
"hash" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2441",
"triggerID" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"triggerType" : "PUSH"
}, {
"hash" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2443",
"triggerID" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2447",
"triggerID" : "928890183",
"triggerType" : "MANUAL"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2470",
"triggerID" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"triggerType" : "PUSH"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2560",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562",
"triggerID" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"triggerType" : "PUSH"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
}, {
"hash" : "6fa031e1dcf6e21dfb24cf4296cb00d910b0a204",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2589",
"triggerID" : "6fa031e1dcf6e21dfb24cf4296cb00d910b0a204",
"triggerType" : "PUSH"
}, {
"hash" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2593",
"triggerID" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"triggerType" : "PUSH"
}, {
"hash" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2604",
"triggerID" : "941865271",
"triggerType" : "MANUAL"
}, {
"hash" : "6d67b68e19a43f8668e5773d27ca9c33a8de0a37",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2642",
"triggerID" : "6d67b68e19a43f8668e5773d27ca9c33a8de0a37",
"triggerType" : "PUSH"
}, {
"hash" : "1a80559bd98829552acffdaf20d3ea0384d1d936",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2645",
"triggerID" : "1a80559bd98829552acffdaf20d3ea0384d1d936",
"triggerType" : "PUSH"
}, {
"hash" : "c4af04dab3dab31ef05ba6007000738a2dfb81ce",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2649",
"triggerID" : "c4af04dab3dab31ef05ba6007000738a2dfb81ce",
"triggerType" : "PUSH"
}, {
"hash" : "369c5d6f1f4b6eed3c86f838fb6148d127cc6703",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2660",
"triggerID" : "369c5d6f1f4b6eed3c86f838fb6148d127cc6703",
"triggerType" : "PUSH"
}, {
"hash" : "369c5d6f1f4b6eed3c86f838fb6148d127cc6703",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2898",
"triggerID" : "952859446",
"triggerType" : "MANUAL"
}, {
"hash" : "196a40e3273e98f2317856a55de3bd1888e9f754",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2899",
"triggerID" : "196a40e3273e98f2317856a55de3bd1888e9f754",
"triggerType" : "PUSH"
}, {
"hash" : "196a40e3273e98f2317856a55de3bd1888e9f754",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2899",
"triggerID" : "952859446",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 0fa6297ce58eb877fd5c4eba59fef20ad9335d26 UNKNOWN
* 369c5d6f1f4b6eed3c86f838fb6148d127cc6703 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2660) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2898)
* 196a40e3273e98f2317856a55de3bd1888e9f754 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2899)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5d49bd4e8c638ffb9ced102dc6771d6291c199e3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3203: [HUDI-2086] redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5d49bd4e8c638ffb9ced102dc6771d6291c199e3 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-commenter commented on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872264798
# [Codecov](https://codecov.io/gh/apache/hudi/pull/3203?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3203](https://codecov.io/gh/apache/hudi/pull/3203?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (5d49bd4) into [master](https://codecov.io/gh/apache/hudi/commit/94f0f40fec44c7eecb4e886df62f52a28226974c?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (94f0f40) will **decrease** coverage by `30.71%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/hudi/pull/3203?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3203 +/- ##
=============================================
- Coverage 46.20% 15.48% -30.72%
+ Complexity 5389 478 -4911
=============================================
Files 921 280 -641
Lines 40053 11548 -28505
Branches 4297 945 -3352
=============================================
- Hits 18505 1788 -16717
+ Misses 19664 9601 -10063
+ Partials 1884 159 -1725
```
| Flag | Coverage Δ | |
|---|---|---|
| hudicli | `?` | |
| hudiclient | `0.00% <ø> (-30.46%)` | :arrow_down: |
| hudicommon | `?` | |
| hudiflink | `?` | |
| hudihadoopmr | `?` | |
| hudisparkdatasource | `?` | |
| hudisync | `5.38% <ø> (-48.67%)` | :arrow_down: |
| huditimelineservice | `?` | |
| hudiutilities | `58.02% <ø> (-0.57%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3203?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3203/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3203/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3203/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...a/org/apache/hudi/metrics/MetricsReporterType.java](https://codecov.io/gh/apache/hudi/pull/3203/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyVHlwZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...rg/apache/hudi/client/bootstrap/BootstrapMode.java](https://codecov.io/gh/apache/hudi/pull/3203/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9ib290c3RyYXAvQm9vdHN0cmFwTW9kZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...he/hudi/hive/HiveStylePartitionValueExtractor.java](https://codecov.io/gh/apache/hudi/pull/3203/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN0eWxlUGFydGl0aW9uVmFsdWVFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../apache/hudi/keygen/constant/KeyGeneratorType.java](https://codecov.io/gh/apache/hudi/pull/3203/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2tleWdlbi9jb25zdGFudC9LZXlHZW5lcmF0b3JUeXBlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...pache/hudi/client/utils/ConcatenatingIterator.java](https://codecov.io/gh/apache/hudi/pull/3203/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC91dGlscy9Db25jYXRlbmF0aW5nSXRlcmF0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../hudi/execution/bulkinsert/BulkInsertSortMode.java](https://codecov.io/gh/apache/hudi/pull/3203/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2V4ZWN1dGlvbi9idWxraW5zZXJ0L0J1bGtJbnNlcnRTb3J0TW9kZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...able/action/compact/CompactionTriggerStrategy.java](https://codecov.io/gh/apache/hudi/pull/3203/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL3RhYmxlL2FjdGlvbi9jb21wYWN0L0NvbXBhY3Rpb25UcmlnZ2VyU3RyYXRlZ3kuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [709 more](https://codecov.io/gh/apache/hudi/pull/3203/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/hudi/pull/3203?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/hudi/pull/3203?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [94f0f40...5d49bd4](https://codecov.io/gh/apache/hudi/pull/3203?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao closed pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao closed pull request #3203:
URL: https://github.com/apache/hudi/pull/3203
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-952859446
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
}, {
"hash" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325",
"triggerID" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"triggerType" : "PUSH"
}, {
"hash" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2343",
"triggerID" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"triggerType" : "PUSH"
}, {
"hash" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2424",
"triggerID" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2427",
"triggerID" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2429",
"triggerID" : "927749497",
"triggerType" : "MANUAL"
}, {
"hash" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2431",
"triggerID" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"triggerType" : "PUSH"
}, {
"hash" : "29e6acb65044597e35f9d499385dad940580d9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2432",
"triggerID" : "29e6acb65044597e35f9d499385dad940580d9df",
"triggerType" : "PUSH"
}, {
"hash" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2441",
"triggerID" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"triggerType" : "PUSH"
}, {
"hash" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2443",
"triggerID" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2447",
"triggerID" : "928890183",
"triggerType" : "MANUAL"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2470",
"triggerID" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"triggerType" : "PUSH"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2560",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562",
"triggerID" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"triggerType" : "PUSH"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
}, {
"hash" : "6fa031e1dcf6e21dfb24cf4296cb00d910b0a204",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2589",
"triggerID" : "6fa031e1dcf6e21dfb24cf4296cb00d910b0a204",
"triggerType" : "PUSH"
}, {
"hash" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2593",
"triggerID" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"triggerType" : "PUSH"
}, {
"hash" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2604",
"triggerID" : "941865271",
"triggerType" : "MANUAL"
}, {
"hash" : "6d67b68e19a43f8668e5773d27ca9c33a8de0a37",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2642",
"triggerID" : "6d67b68e19a43f8668e5773d27ca9c33a8de0a37",
"triggerType" : "PUSH"
}, {
"hash" : "1a80559bd98829552acffdaf20d3ea0384d1d936",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2645",
"triggerID" : "1a80559bd98829552acffdaf20d3ea0384d1d936",
"triggerType" : "PUSH"
}, {
"hash" : "c4af04dab3dab31ef05ba6007000738a2dfb81ce",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2649",
"triggerID" : "c4af04dab3dab31ef05ba6007000738a2dfb81ce",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0fa6297ce58eb877fd5c4eba59fef20ad9335d26 UNKNOWN
* c4af04dab3dab31ef05ba6007000738a2dfb81ce Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2649)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r728743157
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieEmptyRecordReader.java
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop.realtime;
+
+import org.apache.hadoop.io.ArrayWritable;
+import org.apache.hadoop.io.NullWritable;
+import org.apache.hadoop.io.Writable;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.RecordReader;
+
+import java.io.IOException;
+
+/**
+ * dummy record for log only realtime split.
+ */
Review comment:
agree
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r728741908
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieParquetRealtimeInputFormat.java
##########
@@ -66,6 +90,170 @@
return HoodieRealtimeInputFormatUtils.getRealtimeSplits(job, fileSplits);
}
+ /**
+ * Keep the logical of mor_incr_view as same as spark datasource.
+ * Step1: Get list of commits to be fetched based on start commit and max commits(for snapshot max commits is -1).
Review comment:
agree
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r725422730
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/fs/FSUtils.java
##########
@@ -336,6 +336,11 @@ public static String getFileExtensionFromLog(Path logPath) {
return matcher.group(3);
}
+ public static String getLogFileExtension(String fullName) {
+ Matcher matcher = LOG_FILE_PATTERN.matcher(fullName);
Review comment:
ok, change it. and remove getLogFileExtension
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r725753064
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/BaseFileWithLogsSplit.java
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop;
+
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.apache.hadoop.mapred.FileSplit;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * we need to encode additional information in split to track matching log file and base files.
+ * Hence, this weird looking class which tracks an log/base file split
+ */
+public class BaseFileWithLogsSplit extends FileSplit {
+ // a flag to mark this split is produced by incremental query or not.
+ private boolean belongToIncrementalSplit = false;
+ // the log files of this split.
+ private List<String> deltaLogPaths = new ArrayList<>();
+ // max commit time of current split.
+ private String maxCommitTime = "";
+ // the basePath of current hoodie table.
+ private String basePath = "";
+ // the base file of this split.
+ private String baseFilePath = "";
Review comment:
yes, we will deal with bootstrap split in HoodieParquetRealtimeInputFormat.makeSplit. so we no need to add boostrap variable here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-939217992
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r725743002
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieParquetRealtimeInputFormat.java
##########
@@ -66,6 +91,139 @@
return HoodieRealtimeInputFormatUtils.getRealtimeSplits(job, fileSplits);
}
+ /**
+ * keep the logical of mor_incr_view as same as spark datasource.
+ * to do: unify the incremental view code between hive/spark-sql and spark datasource
+ */
+ @Override
+ protected List<FileStatus> listStatusForIncrementalMode(
+ JobConf job, HoodieTableMetaClient tableMetaClient, List<Path> inputPaths) throws IOException {
+ List<FileStatus> result = new ArrayList<>();
+ String tableName = tableMetaClient.getTableConfig().getTableName();
+ Job jobContext = Job.getInstance(job);
+
+ Option<HoodieTimeline> timeline = HoodieInputFormatUtils.getFilteredCommitsTimeline(jobContext, tableMetaClient);
+ if (!timeline.isPresent()) {
+ return result;
+ }
+ String lastIncrementalTs = HoodieHiveUtils.readStartCommitTime(jobContext, tableName);
+ // Total number of commits to return in this batch. Set this to -1 to get all the commits.
+ Integer maxCommits = HoodieHiveUtils.readMaxCommits(jobContext, tableName);
+ HoodieTimeline commitsTimelineToReturn = timeline.get().findInstantsAfter(lastIncrementalTs, maxCommits);
+ Option<List<HoodieInstant>> commitsToCheck = Option.of(commitsTimelineToReturn.getInstants().collect(Collectors.toList()));
+ if (!commitsToCheck.isPresent()) {
+ return result;
+ }
+ Map<String, HashMap<String, FileStatus>> partitionsWithFileStatus = HoodieInputFormatUtils
+ .listAffectedFilesForCommits(new Path(tableMetaClient.getBasePath()), commitsToCheck.get(), commitsTimelineToReturn);
+ // build fileGroup from fsView
+ List<FileStatus> affectedFileStatus = new ArrayList<>();
+ partitionsWithFileStatus.forEach((key, value) -> value.forEach((k, v) -> affectedFileStatus.add(v)));
+ HoodieTableFileSystemView fsView = new HoodieTableFileSystemView(tableMetaClient, commitsTimelineToReturn, affectedFileStatus.toArray(new FileStatus[0]));
+ // build fileGroup from fsView
+ String basePath = tableMetaClient.getBasePath();
+ // filter affectedPartition by inputPaths
+ List<String> affectedPartition = partitionsWithFileStatus.keySet().stream()
+ .filter(k -> k.isEmpty() ? inputPaths.contains(new Path(basePath)) : inputPaths.contains(new Path(basePath, k))).collect(Collectors.toList());
+ if (affectedPartition.isEmpty()) {
+ return result;
+ }
+ List<HoodieFileGroup> fileGroups = affectedPartition.stream()
+ .flatMap(partitionPath -> fsView.getAllFileGroups(partitionPath)).collect(Collectors.toList());
+ setInputPaths(job, affectedPartition.stream()
+ .map(p -> p.isEmpty() ? basePath : new Path(basePath, p).toUri().toString()).collect(Collectors.joining(",")));
+
+ // find all file status in current partitionPath
+ FileStatus[] fileStatuses = getStatus(job);
+ Map<String, FileStatus> candidateFileStatus = new HashMap<>();
+ for (int i = 0; i < fileStatuses.length; i++) {
+ String key = fileStatuses[i].getPath().toString();
+ candidateFileStatus.put(key, fileStatuses[i]);
+ }
+
+ String maxCommitTime = fsView.getLastInstant().get().getTimestamp();
+ fileGroups.stream().forEach(f -> {
+ try {
+ List<FileSlice> baseFiles = f.getAllFileSlices().filter(slice -> slice.getBaseFile().isPresent()).collect(Collectors.toList());
+ if (!baseFiles.isEmpty()) {
+ FileStatus baseFileStatus = HoodieInputFormatUtils.getFileStatus(baseFiles.get(0).getBaseFile().get());
+ String baseFilePath = baseFileStatus.getPath().toUri().toString();
+ if (!candidateFileStatus.containsKey(baseFilePath)) {
+ throw new HoodieException("Error obtaining fileStatus for file: " + baseFilePath);
+ }
+ RealtimeFileStatus fileStatus = new RealtimeFileStatus(candidateFileStatus.get(baseFilePath));
+ fileStatus.setMaxCommitTime(maxCommitTime);
+ fileStatus.setBelongToIncrementalFileStatus(true);
+ fileStatus.setBasePath(basePath);
+ fileStatus.setBaseFilePath(baseFilePath);
+ fileStatus.setDeltaLogPaths(f.getLatestFileSlice().get().getLogFiles().map(l -> l.getPath().toString()).collect(Collectors.toList()));
Review comment:
interntional. notice that , we add all incremental files(both logs and parquet files) to HoodieTableFileSystemView, so we should call getAllFileSlices() instead of getLastestFileSlice
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r725419790
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieParquetRealtimeInputFormat.java
##########
@@ -66,6 +91,139 @@
return HoodieRealtimeInputFormatUtils.getRealtimeSplits(job, fileSplits);
}
+ /**
+ * keep the logical of mor_incr_view as same as spark datasource.
+ * to do: unify the incremental view code between hive/spark-sql and spark datasource
+ */
+ @Override
+ protected List<FileStatus> listStatusForIncrementalMode(
+ JobConf job, HoodieTableMetaClient tableMetaClient, List<Path> inputPaths) throws IOException {
+ List<FileStatus> result = new ArrayList<>();
+ String tableName = tableMetaClient.getTableConfig().getTableName();
+ Job jobContext = Job.getInstance(job);
+
+ Option<HoodieTimeline> timeline = HoodieInputFormatUtils.getFilteredCommitsTimeline(jobContext, tableMetaClient);
+ if (!timeline.isPresent()) {
+ return result;
+ }
+ String lastIncrementalTs = HoodieHiveUtils.readStartCommitTime(jobContext, tableName);
+ // Total number of commits to return in this batch. Set this to -1 to get all the commits.
+ Integer maxCommits = HoodieHiveUtils.readMaxCommits(jobContext, tableName);
+ HoodieTimeline commitsTimelineToReturn = timeline.get().findInstantsAfter(lastIncrementalTs, maxCommits);
+ Option<List<HoodieInstant>> commitsToCheck = Option.of(commitsTimelineToReturn.getInstants().collect(Collectors.toList()));
+ if (!commitsToCheck.isPresent()) {
+ return result;
+ }
+ Map<String, HashMap<String, FileStatus>> partitionsWithFileStatus = HoodieInputFormatUtils
+ .listAffectedFilesForCommits(new Path(tableMetaClient.getBasePath()), commitsToCheck.get(), commitsTimelineToReturn);
+ // build fileGroup from fsView
+ List<FileStatus> affectedFileStatus = new ArrayList<>();
+ partitionsWithFileStatus.forEach((key, value) -> value.forEach((k, v) -> affectedFileStatus.add(v)));
+ HoodieTableFileSystemView fsView = new HoodieTableFileSystemView(tableMetaClient, commitsTimelineToReturn, affectedFileStatus.toArray(new FileStatus[0]));
+ // build fileGroup from fsView
+ String basePath = tableMetaClient.getBasePath();
+ // filter affectedPartition by inputPaths
+ List<String> affectedPartition = partitionsWithFileStatus.keySet().stream()
+ .filter(k -> k.isEmpty() ? inputPaths.contains(new Path(basePath)) : inputPaths.contains(new Path(basePath, k))).collect(Collectors.toList());
+ if (affectedPartition.isEmpty()) {
+ return result;
+ }
+ List<HoodieFileGroup> fileGroups = affectedPartition.stream()
+ .flatMap(partitionPath -> fsView.getAllFileGroups(partitionPath)).collect(Collectors.toList());
+ setInputPaths(job, affectedPartition.stream()
+ .map(p -> p.isEmpty() ? basePath : new Path(basePath, p).toUri().toString()).collect(Collectors.joining(",")));
+
+ // find all file status in current partitionPath
+ FileStatus[] fileStatuses = getStatus(job);
+ Map<String, FileStatus> candidateFileStatus = new HashMap<>();
+ for (int i = 0; i < fileStatuses.length; i++) {
+ String key = fileStatuses[i].getPath().toString();
+ candidateFileStatus.put(key, fileStatuses[i]);
+ }
+
+ String maxCommitTime = fsView.getLastInstant().get().getTimestamp();
+ fileGroups.stream().forEach(f -> {
+ try {
+ List<FileSlice> baseFiles = f.getAllFileSlices().filter(slice -> slice.getBaseFile().isPresent()).collect(Collectors.toList());
+ if (!baseFiles.isEmpty()) {
+ FileStatus baseFileStatus = HoodieInputFormatUtils.getFileStatus(baseFiles.get(0).getBaseFile().get());
+ String baseFilePath = baseFileStatus.getPath().toUri().toString();
+ if (!candidateFileStatus.containsKey(baseFilePath)) {
+ throw new HoodieException("Error obtaining fileStatus for file: " + baseFilePath);
+ }
+ RealtimeFileStatus fileStatus = new RealtimeFileStatus(candidateFileStatus.get(baseFilePath));
+ fileStatus.setMaxCommitTime(maxCommitTime);
+ fileStatus.setBelongToIncrementalFileStatus(true);
+ fileStatus.setBasePath(basePath);
+ fileStatus.setBaseFilePath(baseFilePath);
+ fileStatus.setDeltaLogPaths(f.getLatestFileSlice().get().getLogFiles().map(l -> l.getPath().toString()).collect(Collectors.toList()));
+ // try to set bootstrapfileStatus
+ if (baseFileStatus instanceof LocatedFileStatusWithBootstrapBaseFile || baseFileStatus instanceof FileStatusWithBootstrapBaseFile) {
+ fileStatus.setBootStrapFileStatus(baseFileStatus);
+ }
+ result.add(fileStatus);
+ }
+ // add file group which has only logs.
+ if (f.getLatestFileSlice().isPresent() && baseFiles.isEmpty()) {
+ List<FileStatus> logFileStatus = f.getLatestFileSlice().get().getLogFiles().map(logFile -> logFile.getFileStatus()).collect(Collectors.toList());
+ if (logFileStatus.size() > 0) {
+ RealtimeFileStatus fileStatus = new RealtimeFileStatus(logFileStatus.get(0));
+ fileStatus.setBelongToIncrementalFileStatus(true);
+ fileStatus.setDeltaLogPaths(logFileStatus.stream().map(l -> l.getPath().toString()).collect(Collectors.toList()));
+ fileStatus.setMaxCommitTime(maxCommitTime);
+ fileStatus.setBasePath(basePath);
+ result.add(fileStatus);
+ }
+ }
+ } catch (IOException e) {
+ throw new HoodieException("Error obtaining data file/log file grouping ", e);
+ }
+ });
+ return result;
+ }
+
+ @Override
+ protected boolean includeLogFilesForSnapShotView() {
+ return true;
+ }
+
+ @Override
+ protected boolean isSplitable(FileSystem fs, Path filename) {
+ if (filename instanceof PathWithLogFilePath) {
+ return ((PathWithLogFilePath)filename).splitable();
+ }
+ return super.isSplitable(fs, filename);
+ }
+
+ @Override
+ protected FileSplit makeSplit(Path file, long start, long length, String[] hosts) {
+ if (file instanceof PathWithLogFilePath) {
+ return doMakeSplitForPathWithLogFilePath((PathWithLogFilePath) file, start, length, hosts, null);
+ }
+ return super.makeSplit(file, start, length, hosts);
+ }
+
+ @Override
+ protected FileSplit makeSplit(Path file, long start, long length, String[] hosts, String[] inMemoryHosts) {
+ if (file instanceof PathWithLogFilePath) {
Review comment:
For boostrap tables.
1） When we query the incremental view, the read files may be boostrapfiles, we will wrap this boostrapfile into PathWithLogFilePath, this logical can help to fix the bug that hudi cannot support read incremental datas which before replacecommit.
2) Of course, if it is not for querying incremental views, this place is PathWithBootstrapFileStatus, no PathWithLogFilePath will produced.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
}, {
"hash" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325",
"triggerID" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"triggerType" : "PUSH"
}, {
"hash" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2343",
"triggerID" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c470b4c2c6a4d1d33a5facf729971b4331fe0771 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2343)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
}, {
"hash" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325",
"triggerID" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"triggerType" : "PUSH"
}, {
"hash" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2343",
"triggerID" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 42735e78f24bd3a5f622026627d6ff8b95708705 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325)
* c470b4c2c6a4d1d33a5facf729971b4331fe0771 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2343)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
}, {
"hash" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325",
"triggerID" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 792ae1d9884eac5eb7004e4edb617ea6e86d79b7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631)
* 42735e78f24bd3a5f622026627d6ff8b95708705 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r718189151
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieParquetRealtimeInputFormat.java
##########
@@ -66,6 +90,138 @@
return HoodieRealtimeInputFormatUtils.getRealtimeSplits(job, fileSplits);
}
+ /**
+ * keep the logical of mor_incr_view as same as spark datasource.
+ * to do: unify the incremental view code between hive/spark-sql and spark datasource
Review comment:
the logic has already be unified，now the logic of hive incremental and spark incremental is same。 but to do unify the code has a little difficult。 may be can do it in another pr。
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieMergedLogReader.java
##########
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop.realtime;
+
+import java.io.IOException;
+import java.text.MessageFormat;
+import java.util.Iterator;
+
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.io.ArrayWritable;
+import org.apache.hadoop.io.NullWritable;
+import org.apache.hadoop.io.Writable;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.RecordReader;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.hadoop.utils.HoodieRealtimeRecordReaderUtils;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+/**
+ * Record Reader implementation to read avro data, to support inc queries.
+ */
+public class HoodieMergedLogReader extends AbstractRealtimeRecordReader
+ implements RecordReader<NullWritable, ArrayWritable> {
+ private static final Logger LOG = LogManager.getLogger(AbstractRealtimeRecordReader.class);
+ private final HoodieMergedLogRecordScanner logRecordScanner;
+ private final Iterator<HoodieRecord<? extends HoodieRecordPayload>> logRecordsKeyIterator;
+ private ArrayWritable valueObj;
+ private boolean supportTimeStamp = false;
+
+ private int end;
+ private int offset;
+
+ public HoodieMergedLogReader(RealtimeSplit split, JobConf job, HoodieMergedLogRecordScanner logRecordScanner) {
+ super(split, job);
+ this.logRecordScanner = logRecordScanner;
+ this.end = logRecordScanner.getRecords().size();
+ this.logRecordsKeyIterator = logRecordScanner.iterator();
+ this.valueObj = new ArrayWritable(Writable.class, new Writable[getHiveSchema().getFields().size()]);
+ }
+
+ private Option buildGenericRecordwithCustomPayload(HoodieRecord record) throws IOException {
+ if (usesCustomPayload) {
+ return record.getData().getInsertValue(getWriterSchema());
+ } else {
+ return record.getData().getInsertValue(getReaderSchema());
+ }
+ }
+
+ @Override
+ public boolean next(NullWritable key, ArrayWritable arrayWritable) throws IOException {
+ if (!logRecordsKeyIterator.hasNext()) {
+ return false;
+ }
+ Option<GenericRecord> rec;
+ HoodieRecord currentRecord = logRecordsKeyIterator.next();
+
+ rec = buildGenericRecordwithCustomPayload(currentRecord);
+ // try to skip delete record
+ while (!rec.isPresent() && logRecordsKeyIterator.hasNext()) {
+ offset++;
+ rec = buildGenericRecordwithCustomPayload(logRecordsKeyIterator.next());
+ }
+ if (!rec.isPresent()) {
+ return false;
+ }
+
+ GenericRecord recordToReturn = rec.get();
+ if (usesCustomPayload) {
+ // If using a custom payload, return only the projection fields. The readerSchema is a schema derived from
+ // the writerSchema with only the projection fields
+ recordToReturn = HoodieAvroUtils.rewriteRecord(rec.get(), getReaderSchema());
+ }
+ ArrayWritable curWritable = (ArrayWritable) HoodieRealtimeRecordReaderUtils.avroToArrayWritable(recordToReturn, getHiveSchema());
+
+ if (arrayWritable != curWritable) {
+ final Writable[] arrValue = arrayWritable.get();
+ final Writable[] arrCurrent = curWritable.get();
+ if (arrayWritable != null && arrValue.length == arrCurrent.length) {
+ System.arraycopy(arrCurrent, 0, arrValue, 0, arrCurrent.length);
+ } else {
+ if (arrValue.length != arrCurrent.length) {
+ throw new IOException(MessageFormat.format("HoodieMergeLogReader : size of object differs. Value size : {0}, Current Object size : {1}",
+ arrValue.length, arrCurrent.length));
+ } else {
+ throw new IOException("HoodieMergeLogReader can not support RecordReaders that don't return same key & value & value is null");
+ }
+ }
+ }
+
+ offset++;
+ return true;
+ }
+
+ @Override
+ public NullWritable createKey() {
+ return null;
+ }
+
+ @Override
+ public ArrayWritable createValue() {
+ return valueObj;
+ }
+
+ @Override
+ public long getPos() throws IOException {
+ return offset;
+ }
+
+ @Override
+ public void close() throws IOException {
+ if (logRecordScanner != null) {
+ logRecordScanner.close();
+ }
+ }
+
+ @Override
+ public float getProgress() throws IOException {
+ if (end == offset) {
+ return 0.0f;
Review comment:
oh， no。 i am sorry for it
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieMergedLogReader.java
##########
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop.realtime;
+
+import java.io.IOException;
+import java.text.MessageFormat;
+import java.util.Iterator;
+
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.io.ArrayWritable;
+import org.apache.hadoop.io.NullWritable;
+import org.apache.hadoop.io.Writable;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.RecordReader;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.hadoop.utils.HoodieRealtimeRecordReaderUtils;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+/**
+ * Record Reader implementation to read avro data, to support inc queries.
+ */
+public class HoodieMergedLogReader extends AbstractRealtimeRecordReader
+ implements RecordReader<NullWritable, ArrayWritable> {
+ private static final Logger LOG = LogManager.getLogger(AbstractRealtimeRecordReader.class);
+ private final HoodieMergedLogRecordScanner logRecordScanner;
+ private final Iterator<HoodieRecord<? extends HoodieRecordPayload>> logRecordsKeyIterator;
+ private ArrayWritable valueObj;
+ private boolean supportTimeStamp = false;
+
+ private int end;
+ private int offset;
+
+ public HoodieMergedLogReader(RealtimeSplit split, JobConf job, HoodieMergedLogRecordScanner logRecordScanner) {
+ super(split, job);
+ this.logRecordScanner = logRecordScanner;
+ this.end = logRecordScanner.getRecords().size();
+ this.logRecordsKeyIterator = logRecordScanner.iterator();
+ this.valueObj = new ArrayWritable(Writable.class, new Writable[getHiveSchema().getFields().size()]);
+ }
+
+ private Option buildGenericRecordwithCustomPayload(HoodieRecord record) throws IOException {
+ if (usesCustomPayload) {
+ return record.getData().getInsertValue(getWriterSchema());
+ } else {
+ return record.getData().getInsertValue(getReaderSchema());
+ }
+ }
+
+ @Override
+ public boolean next(NullWritable key, ArrayWritable arrayWritable) throws IOException {
+ if (!logRecordsKeyIterator.hasNext()) {
+ return false;
+ }
+ Option<GenericRecord> rec;
+ HoodieRecord currentRecord = logRecordsKeyIterator.next();
+
+ rec = buildGenericRecordwithCustomPayload(currentRecord);
+ // try to skip delete record
+ while (!rec.isPresent() && logRecordsKeyIterator.hasNext()) {
+ offset++;
+ rec = buildGenericRecordwithCustomPayload(logRecordsKeyIterator.next());
+ }
+ if (!rec.isPresent()) {
+ return false;
+ }
+
+ GenericRecord recordToReturn = rec.get();
+ if (usesCustomPayload) {
+ // If using a custom payload, return only the projection fields. The readerSchema is a schema derived from
+ // the writerSchema with only the projection fields
+ recordToReturn = HoodieAvroUtils.rewriteRecord(rec.get(), getReaderSchema());
+ }
+ ArrayWritable curWritable = (ArrayWritable) HoodieRealtimeRecordReaderUtils.avroToArrayWritable(recordToReturn, getHiveSchema());
+
+ if (arrayWritable != curWritable) {
+ final Writable[] arrValue = arrayWritable.get();
+ final Writable[] arrCurrent = curWritable.get();
+ if (arrayWritable != null && arrValue.length == arrCurrent.length) {
+ System.arraycopy(arrCurrent, 0, arrValue, 0, arrCurrent.length);
Review comment:
yes， i will remove it。
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/fs/FSUtils.java
##########
@@ -336,6 +336,11 @@ public static String getFileExtensionFromLog(Path logPath) {
return matcher.group(3);
}
+ public static String getLogFileExtension(String fullName) {
+ Matcher matcher = LOG_FILE_PATTERN.matcher(fullName);
Review comment:
getFileExtensionFromLog will throw InvalidHoodiePathException is the file is not a log file。 When we check whether the current file is a log file, we don’t want to throw an exception
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieMergedLogReader.java
##########
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop.realtime;
+
+import java.io.IOException;
+import java.text.MessageFormat;
+import java.util.Iterator;
+
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.io.ArrayWritable;
+import org.apache.hadoop.io.NullWritable;
+import org.apache.hadoop.io.Writable;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.RecordReader;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.hadoop.utils.HoodieRealtimeRecordReaderUtils;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+/**
+ * Record Reader implementation to read avro data, to support inc queries.
+ */
+public class HoodieMergedLogReader extends AbstractRealtimeRecordReader
+ implements RecordReader<NullWritable, ArrayWritable> {
+ private static final Logger LOG = LogManager.getLogger(AbstractRealtimeRecordReader.class);
+ private final HoodieMergedLogRecordScanner logRecordScanner;
+ private final Iterator<HoodieRecord<? extends HoodieRecordPayload>> logRecordsKeyIterator;
+ private ArrayWritable valueObj;
+ private boolean supportTimeStamp = false;
+
+ private int end;
+ private int offset;
+
+ public HoodieMergedLogReader(RealtimeSplit split, JobConf job, HoodieMergedLogRecordScanner logRecordScanner) {
+ super(split, job);
+ this.logRecordScanner = logRecordScanner;
+ this.end = logRecordScanner.getRecords().size();
+ this.logRecordsKeyIterator = logRecordScanner.iterator();
+ this.valueObj = new ArrayWritable(Writable.class, new Writable[getHiveSchema().getFields().size()]);
+ }
+
+ private Option buildGenericRecordwithCustomPayload(HoodieRecord record) throws IOException {
+ if (usesCustomPayload) {
Review comment:
ok
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/PathWithLogFilePath.java
##########
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop;
+
+import org.apache.hadoop.fs.Path;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * we need to encode additional information in Path to track matching log file and base files.
+ * Hence, this weird looking class which tracks an log/base file status
+ */
+public class PathWithLogFilePath extends Path {
+
+ private boolean belongToIncrementalPath = false;
+ private List<String> deltaLogPaths = new ArrayList<>();
Review comment:
ok
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieParquetRealtimeInputFormat.java
##########
@@ -119,6 +275,11 @@ void addProjectionToJobConf(final RealtimeSplit realtimeSplit, final JobConf job
addProjectionToJobConf(realtimeSplit, jobConf);
LOG.info("Creating record reader with readCols :" + jobConf.get(ColumnProjectionUtils.READ_COLUMN_NAMES_CONF_STR)
+ ", Ids :" + jobConf.get(ColumnProjectionUtils.READ_COLUMN_IDS_CONF_STR));
+
+ // for log only split, we no need parquet reader, set it to empty
+ if (realtimeSplit.getPath().toString().contains(".log.")) {
+ return new HoodieRealtimeRecordReader(realtimeSplit, jobConf, new HoodieEmptyRecordReader());
+ }
Review comment:
agree
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/RealtimeCompactedRecordReader.java
##########
@@ -55,13 +55,20 @@
private final Set<String> deltaRecordKeys;
private int recordKeyIndex = HoodieInputFormatUtils.HOODIE_RECORD_KEY_COL_POS;
private Iterator<String> deltaItr;
+ private boolean logFileOnlySplit;
+ private HoodieMergedLogReader logReader;
public RealtimeCompactedRecordReader(RealtimeSplit split, JobConf job,
RecordReader<NullWritable, ArrayWritable> realReader) throws IOException {
super(split, job);
this.parquetReader = realReader;
- this.deltaRecordMap = getMergedLogRecordScanner().getRecords();
+ HoodieMergedLogRecordScanner hoodieMergedLogRecordScanner = getMergedLogRecordScanner();
+ this.deltaRecordMap = hoodieMergedLogRecordScanner.getRecords();
this.deltaRecordKeys = new HashSet<>(this.deltaRecordMap.keySet());
+ if (split.getPath().toString().contains(".log.")) {
+ this.logFileOnlySplit = true;
Review comment:
agree
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieInputFormatUtils.java
##########
@@ -445,10 +457,21 @@ public static HoodieMetadataConfig buildMetadataConfig(Configuration conf) {
HoodieTableFileSystemView fsView = fsViewCache.computeIfAbsent(metaClient, tableMetaClient ->
FileSystemViewManager.createInMemoryFileSystemViewWithTimeline(engineContext, tableMetaClient, buildMetadataConfig(job), timeline));
List<HoodieBaseFile> filteredBaseFiles = new ArrayList<>();
+ Map<HoodieLogFile, List<String>> filteredLogs = new HashMap<>();
for (Path p : entry.getValue()) {
Review comment:
will change it。
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieRealtimeInputFormatUtils.java
##########
@@ -120,28 +147,32 @@
.orElse(Stream.empty());
// subgroup splits again by file id & match with log files.
- Map<String, List<FileSplit>> groupedInputSplits = partitionsToParquetSplits.get(partitionPath).stream()
- .collect(Collectors.groupingBy(split -> FSUtils.getFileId(split.getPath().getName())));
+ Map<String, List<FileSplit>> groupedInputSplits = filteredPartitionsToParquetSplits.get(partitionPath).stream()
+ .collect(Collectors.groupingBy(split -> {
+ String fileName = split.getPath().getName();
+ // mor snapsht view may contains some raw log files, for example: if user use Hbase index, hudi will produce only log Files
+ if (split instanceof BaseFileWithLogsSplit && ((BaseFileWithLogsSplit) split).getBaseFilePath().isEmpty()) {
+ fileName = split.getPath().getName().substring(1, fileName.length() - 1);
+ }
Review comment:
agree
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/BaseFileWithLogsSplit.java
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop;
+
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.apache.hadoop.mapred.FileSplit;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+public class BaseFileWithLogsSplit extends FileSplit {
Review comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a change in pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
danny0405 commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r724660978
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/fs/FSUtils.java
##########
@@ -336,6 +336,11 @@ public static String getFileExtensionFromLog(Path logPath) {
return matcher.group(3);
}
+ public static String getLogFileExtension(String fullName) {
+ Matcher matcher = LOG_FILE_PATTERN.matcher(fullName);
Review comment:
I think using `FSUtils.isLogFile` directly is more suitable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-940777833
@danny0405 @nsivabalan @leesf
update the code. addressed all comments。
1. recover HUDI-1969, and remove RealTimeMergedRecordReader.java which is no needed.
2. add more docs on HoodieParquetRealtimeInputFormat.listStatusForIncrementalMode and split this function.
3. keep PathWithLogFileStatus unSplitable. just like spark MergeOnReadIncrementalRelation. now both hbase index and flink write table will be ok.
last things: now we have no way to distinguish between the newly added data and the updated data in a log file. we should make the split unSplitable(just like the logical MergeOnReadIncrementalRelation)
if we can find a way to distinguish added data and updated data, we can remove this limited.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
}, {
"hash" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325",
"triggerID" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"triggerType" : "PUSH"
}, {
"hash" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2343",
"triggerID" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"triggerType" : "PUSH"
}, {
"hash" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2424",
"triggerID" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2427",
"triggerID" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2429",
"triggerID" : "927749497",
"triggerType" : "MANUAL"
}, {
"hash" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2431",
"triggerID" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"triggerType" : "PUSH"
}, {
"hash" : "29e6acb65044597e35f9d499385dad940580d9df",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "29e6acb65044597e35f9d499385dad940580d9df",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 814b7eb1e147197c29cc2701a08b6bc77bf6c2f8 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2427) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2429)
* 40b81794cbd4d6e7b7356b42ece3290b0904e987 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2431)
* 29e6acb65044597e35f9d499385dad940580d9df UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
danny0405 commented on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-963803515
I'm going to merge it soon ~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-961560763
@nsivabalan @vinothchandar could you help me review this pr, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-961560763
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
}, {
"hash" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325",
"triggerID" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"triggerType" : "PUSH"
}, {
"hash" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2343",
"triggerID" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"triggerType" : "PUSH"
}, {
"hash" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2424",
"triggerID" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2427",
"triggerID" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2429",
"triggerID" : "927749497",
"triggerType" : "MANUAL"
}, {
"hash" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2431",
"triggerID" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"triggerType" : "PUSH"
}, {
"hash" : "29e6acb65044597e35f9d499385dad940580d9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2432",
"triggerID" : "29e6acb65044597e35f9d499385dad940580d9df",
"triggerType" : "PUSH"
}, {
"hash" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2441",
"triggerID" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"triggerType" : "PUSH"
}, {
"hash" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2443",
"triggerID" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2447",
"triggerID" : "928890183",
"triggerType" : "MANUAL"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2470",
"triggerID" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"triggerType" : "PUSH"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2560",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562",
"triggerID" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"triggerType" : "PUSH"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
}, {
"hash" : "6fa031e1dcf6e21dfb24cf4296cb00d910b0a204",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2589",
"triggerID" : "6fa031e1dcf6e21dfb24cf4296cb00d910b0a204",
"triggerType" : "PUSH"
}, {
"hash" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2593",
"triggerID" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"triggerType" : "PUSH"
}, {
"hash" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2604",
"triggerID" : "941865271",
"triggerType" : "MANUAL"
}, {
"hash" : "6d67b68e19a43f8668e5773d27ca9c33a8de0a37",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2642",
"triggerID" : "6d67b68e19a43f8668e5773d27ca9c33a8de0a37",
"triggerType" : "PUSH"
}, {
"hash" : "1a80559bd98829552acffdaf20d3ea0384d1d936",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2645",
"triggerID" : "1a80559bd98829552acffdaf20d3ea0384d1d936",
"triggerType" : "PUSH"
}, {
"hash" : "c4af04dab3dab31ef05ba6007000738a2dfb81ce",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2649",
"triggerID" : "c4af04dab3dab31ef05ba6007000738a2dfb81ce",
"triggerType" : "PUSH"
}, {
"hash" : "369c5d6f1f4b6eed3c86f838fb6148d127cc6703",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2660",
"triggerID" : "369c5d6f1f4b6eed3c86f838fb6148d127cc6703",
"triggerType" : "PUSH"
}, {
"hash" : "369c5d6f1f4b6eed3c86f838fb6148d127cc6703",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2898",
"triggerID" : "952859446",
"triggerType" : "MANUAL"
}, {
"hash" : "196a40e3273e98f2317856a55de3bd1888e9f754",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2899",
"triggerID" : "196a40e3273e98f2317856a55de3bd1888e9f754",
"triggerType" : "PUSH"
}, {
"hash" : "196a40e3273e98f2317856a55de3bd1888e9f754",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2899",
"triggerID" : "952859446",
"triggerType" : "MANUAL"
}, {
"hash" : "633b7370155185dead726ddfafd773e284ed510c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2935",
"triggerID" : "633b7370155185dead726ddfafd773e284ed510c",
"triggerType" : "PUSH"
}, {
"hash" : "2182f542acd4ae3d1fd239e6441e9761eb2d33a1",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3069",
"triggerID" : "2182f542acd4ae3d1fd239e6441e9761eb2d33a1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0fa6297ce58eb877fd5c4eba59fef20ad9335d26 UNKNOWN
* 2182f542acd4ae3d1fd239e6441e9761eb2d33a1 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3069)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
danny0405 commented on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-962937092
> > +1, would merge this if there are no more comments in 8 hours.
>
> @danny0405 @xiarixiaoyao Please hold on for some time. @nsivabalan faced an issue while running incremental query with this patch: https://gist.github.com/nsivabalan/950bdde0b3b64349bf0395b2e3e928d8
Sure, let's see if we can fix that in the patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r718193385
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/fs/FSUtils.java
##########
@@ -336,6 +336,11 @@ public static String getFileExtensionFromLog(Path logPath) {
return matcher.group(3);
}
+ public static String getLogFileExtension(String fullName) {
+ Matcher matcher = LOG_FILE_PATTERN.matcher(fullName);
Review comment:
getFileExtensionFromLog will throw InvalidHoodiePathException is the file is not a log file。 When we check whether the current file is a log file, we don’t want to throw an exception
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a change in pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
danny0405 commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r728895613
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/RealtimeFileStatus.java
##########
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+public class RealtimeFileStatus extends FileStatus {
Review comment:
an -> a
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieRealtimeInputFormatUtils.java
##########
@@ -161,6 +162,51 @@
return rtSplits.toArray(new InputSplit[0]);
}
+ // pick all incremental files and add them to rtSplits then filter out those files.
+ private static Map<Path, List<FileSplit>> filterOutIncrementalSplits(
+ List<FileSplit> fileSplitList,
+ List<InputSplit> rtSplits,
+ final Option<HoodieVirtualKeyInfo> finalHoodieVirtualKeyInfo) {
+ return fileSplitList.stream().filter(s -> {
+ // deal with incremental query.
+ try {
+ if (s instanceof BaseFileWithLogsSplit) {
+ BaseFileWithLogsSplit bs = (BaseFileWithLogsSplit)s;
+ if (bs.getBelongToIncrementalSplit()) {
+ rtSplits.add(new HoodieRealtimeFileSplit(bs, bs.getBasePath(), bs.getDeltaLogPaths(), bs.getMaxCommitTime(), finalHoodieVirtualKeyInfo));
+ }
+ } else if (s instanceof RealtimeBootstrapBaseFileSplit) {
+ rtSplits.add(s);
+ }
+ } catch (IOException e) {
+ throw new HoodieIOException("Error creating hoodie real time split ", e);
+ }
+ // filter the snapshot split.
+ if (s instanceof RealtimeBootstrapBaseFileSplit) {
+ return false;
+ } else if ((s instanceof BaseFileWithLogsSplit) && ((BaseFileWithLogsSplit) s).getBelongToIncrementalSplit()) {
Review comment:
Yeah, we can actually abstract the handling of all kinds of splits into a separate method i guess.
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/PathWithBootstrapFileStatus.java
##########
@@ -24,7 +24,7 @@
/**
* Hacky Workaround !!!
* With the base input format implementations in Hadoop/Hive,
- * we need to encode additional information in Path to track matching external file.
+ * Encode additional information in Path to track matching external file.
* Hence, this weird looking class which tracks an external file status
Review comment:
No need to change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-924818277
@danny0405 already rebase this pr. pls tell me what problems did you encounter when you query global index table through Hive, i will test that problem with this pr, thanks .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
}, {
"hash" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325",
"triggerID" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 42735e78f24bd3a5f622026627d6ff8b95708705 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r725419860
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieInputFormatUtils.java
##########
@@ -445,10 +457,21 @@ public static HoodieMetadataConfig buildMetadataConfig(Configuration conf) {
HoodieTableFileSystemView fsView = fsViewCache.computeIfAbsent(metaClient, tableMetaClient ->
FileSystemViewManager.createInMemoryFileSystemViewWithTimeline(engineContext, tableMetaClient, buildMetadataConfig(job), timeline));
List<HoodieBaseFile> filteredBaseFiles = new ArrayList<>();
+ Map<HoodieLogFile, List<String>> filteredLogs = new HashMap<>();
for (Path p : entry.getValue()) {
Review comment:
ok, will change it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a change in pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
danny0405 commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r725764091
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/PathWithLogFilePath.java
##########
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop;
+
+import org.apache.hadoop.fs.Path;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * we need to encode additional information in Path to track matching log file and base files.
+ * Hence, this weird looking class which tracks an log/base file status
+ */
+public class PathWithLogFilePath extends Path {
+ // a flag to mark this split is produced by incremental query or not.
+ private boolean belongToIncrementalPath = false;
+ // the log files belong this path.
+ private List<String> deltaLogPaths = new ArrayList<>();
+ // max commit time of current path.
+ private String maxCommitTime = "";
+ // the basePath of current hoodie table.
+ private String basePath = "";
+ // the base file belong to this path;
+ private String baseFilePath = "";
Review comment:
the base file path belong to this path.
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/BaseFileWithLogsSplit.java
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop;
+
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.apache.hadoop.mapred.FileSplit;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+public class BaseFileWithLogsSplit extends FileSplit {
Review comment:
`We need to encode additional information in split to track matching base and log files. Hence, this weird looking class which tracks a log/base file split`
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/RealtimeCompactedRecordReader.java
##########
@@ -102,97 +102,107 @@ private HoodieMergedLogRecordScanner getMergedLogRecordScanner() throws IOExcept
@Override
public boolean next(NullWritable aVoid, ArrayWritable arrayWritable) throws IOException {
+ // deal with DeltaOnlySplits
+ if (logReader.isPresent()) {
+ return logReader.get().next(aVoid, arrayWritable);
+ }
// Call the underlying parquetReader.next - which may replace the passed in ArrayWritable
// with a new block of values
- while (this.parquetReader.next(aVoid, arrayWritable)) {
- if (!deltaRecordMap.isEmpty()) {
- String key = arrayWritable.get()[recordKeyIndex].toString();
- if (deltaRecordMap.containsKey(key)) {
- // mark the key as handled
- this.deltaRecordKeys.remove(key);
- // TODO(NA): Invoke preCombine here by converting arrayWritable to Avro. This is required since the
- // deltaRecord may not be a full record and needs values of columns from the parquet
- Option<GenericRecord> rec = buildGenericRecordwithCustomPayload(deltaRecordMap.get(key));
- // If the record is not present, this is a delete record using an empty payload so skip this base record
- // and move to the next record
- if (!rec.isPresent()) {
- continue;
+ boolean result = this.parquetReader.next(aVoid, arrayWritable);
+ if (!result) {
+ // if the result is false, then there are no more records
+ return false;
+ }
+ if (!deltaRecordMap.isEmpty()) {
+ // TODO(VC): Right now, we assume all records in log, have a matching base record. (which
+ // would be true until we have a way to index logs too)
Review comment:
No only hbase index, the flink MOR table can also index log files, we need to support that: pure log file groups and base + log files file group.
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieParquetRealtimeInputFormat.java
##########
@@ -66,6 +91,139 @@
return HoodieRealtimeInputFormatUtils.getRealtimeSplits(job, fileSplits);
}
+ /**
+ * keep the logical of mor_incr_view as same as spark datasource.
+ * to do: unify the incremental view code between hive/spark-sql and spark datasource
+ */
+ @Override
+ protected List<FileStatus> listStatusForIncrementalMode(
+ JobConf job, HoodieTableMetaClient tableMetaClient, List<Path> inputPaths) throws IOException {
+ List<FileStatus> result = new ArrayList<>();
+ String tableName = tableMetaClient.getTableConfig().getTableName();
+ Job jobContext = Job.getInstance(job);
+
+ Option<HoodieTimeline> timeline = HoodieInputFormatUtils.getFilteredCommitsTimeline(jobContext, tableMetaClient);
+ if (!timeline.isPresent()) {
+ return result;
+ }
+ String lastIncrementalTs = HoodieHiveUtils.readStartCommitTime(jobContext, tableName);
+ // Total number of commits to return in this batch. Set this to -1 to get all the commits.
+ Integer maxCommits = HoodieHiveUtils.readMaxCommits(jobContext, tableName);
+ HoodieTimeline commitsTimelineToReturn = timeline.get().findInstantsAfter(lastIncrementalTs, maxCommits);
+ Option<List<HoodieInstant>> commitsToCheck = Option.of(commitsTimelineToReturn.getInstants().collect(Collectors.toList()));
+ if (!commitsToCheck.isPresent()) {
+ return result;
+ }
+ Map<String, HashMap<String, FileStatus>> partitionsWithFileStatus = HoodieInputFormatUtils
+ .listAffectedFilesForCommits(new Path(tableMetaClient.getBasePath()), commitsToCheck.get(), commitsTimelineToReturn);
+ // build fileGroup from fsView
+ List<FileStatus> affectedFileStatus = new ArrayList<>();
+ partitionsWithFileStatus.forEach((key, value) -> value.forEach((k, v) -> affectedFileStatus.add(v)));
+ HoodieTableFileSystemView fsView = new HoodieTableFileSystemView(tableMetaClient, commitsTimelineToReturn, affectedFileStatus.toArray(new FileStatus[0]));
+ // build fileGroup from fsView
+ String basePath = tableMetaClient.getBasePath();
+ // filter affectedPartition by inputPaths
+ List<String> affectedPartition = partitionsWithFileStatus.keySet().stream()
+ .filter(k -> k.isEmpty() ? inputPaths.contains(new Path(basePath)) : inputPaths.contains(new Path(basePath, k))).collect(Collectors.toList());
+ if (affectedPartition.isEmpty()) {
+ return result;
+ }
+ List<HoodieFileGroup> fileGroups = affectedPartition.stream()
+ .flatMap(partitionPath -> fsView.getAllFileGroups(partitionPath)).collect(Collectors.toList());
+ setInputPaths(job, affectedPartition.stream()
+ .map(p -> p.isEmpty() ? basePath : new Path(basePath, p).toUri().toString()).collect(Collectors.joining(",")));
+
+ // find all file status in current partitionPath
+ FileStatus[] fileStatuses = getStatus(job);
+ Map<String, FileStatus> candidateFileStatus = new HashMap<>();
+ for (int i = 0; i < fileStatuses.length; i++) {
+ String key = fileStatuses[i].getPath().toString();
+ candidateFileStatus.put(key, fileStatuses[i]);
+ }
+
+ String maxCommitTime = fsView.getLastInstant().get().getTimestamp();
+ fileGroups.stream().forEach(f -> {
+ try {
+ List<FileSlice> baseFiles = f.getAllFileSlices().filter(slice -> slice.getBaseFile().isPresent()).collect(Collectors.toList());
+ if (!baseFiles.isEmpty()) {
+ FileStatus baseFileStatus = HoodieInputFormatUtils.getFileStatus(baseFiles.get(0).getBaseFile().get());
+ String baseFilePath = baseFileStatus.getPath().toUri().toString();
+ if (!candidateFileStatus.containsKey(baseFilePath)) {
+ throw new HoodieException("Error obtaining fileStatus for file: " + baseFilePath);
+ }
+ RealtimeFileStatus fileStatus = new RealtimeFileStatus(candidateFileStatus.get(baseFilePath));
+ fileStatus.setMaxCommitTime(maxCommitTime);
+ fileStatus.setBelongToIncrementalFileStatus(true);
+ fileStatus.setBasePath(basePath);
+ fileStatus.setBaseFilePath(baseFilePath);
+ fileStatus.setDeltaLogPaths(f.getLatestFileSlice().get().getLogFiles().map(l -> l.getPath().toString()).collect(Collectors.toList()));
+ // try to set bootstrapfileStatus
+ if (baseFileStatus instanceof LocatedFileStatusWithBootstrapBaseFile || baseFileStatus instanceof FileStatusWithBootstrapBaseFile) {
+ fileStatus.setBootStrapFileStatus(baseFileStatus);
+ }
+ result.add(fileStatus);
+ }
+ // add file group which has only logs.
+ if (f.getLatestFileSlice().isPresent() && baseFiles.isEmpty()) {
+ List<FileStatus> logFileStatus = f.getLatestFileSlice().get().getLogFiles().map(logFile -> logFile.getFileStatus()).collect(Collectors.toList());
+ if (logFileStatus.size() > 0) {
+ RealtimeFileStatus fileStatus = new RealtimeFileStatus(logFileStatus.get(0));
+ fileStatus.setBelongToIncrementalFileStatus(true);
+ fileStatus.setDeltaLogPaths(logFileStatus.stream().map(l -> l.getPath().toString()).collect(Collectors.toList()));
+ fileStatus.setMaxCommitTime(maxCommitTime);
+ fileStatus.setBasePath(basePath);
+ result.add(fileStatus);
+ }
+ }
+ } catch (IOException e) {
+ throw new HoodieException("Error obtaining data file/log file grouping ", e);
+ }
+ });
+ return result;
+ }
+
+ @Override
+ protected boolean includeLogFilesForSnapShotView() {
+ return true;
+ }
+
+ @Override
+ protected boolean isSplitable(FileSystem fs, Path filename) {
+ if (filename instanceof PathWithLogFilePath) {
+ return ((PathWithLogFilePath)filename).splitable();
+ }
+ return super.isSplitable(fs, filename);
+ }
+
+ @Override
+ protected FileSplit makeSplit(Path file, long start, long length, String[] hosts) {
+ if (file instanceof PathWithLogFilePath) {
+ return doMakeSplitForPathWithLogFilePath((PathWithLogFilePath) file, start, length, hosts, null);
+ }
+ return super.makeSplit(file, start, length, hosts);
+ }
+
+ @Override
+ protected FileSplit makeSplit(Path file, long start, long length, String[] hosts, String[] inMemoryHosts) {
+ if (file instanceof PathWithLogFilePath) {
Review comment:
Okey, please add the explanation to the code comments.
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/HoodieEmptyRecordReader.java
##########
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop;
+
+import org.apache.hadoop.io.ArrayWritable;
+import org.apache.hadoop.io.NullWritable;
+import org.apache.hadoop.io.Writable;
+import org.apache.hadoop.mapred.RecordReader;
+
+import java.io.IOException;
+
+public class HoodieEmptyRecordReader implements RecordReader<NullWritable, ArrayWritable> {
+ @Override
Review comment:
Clazz documents.
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieParquetRealtimeInputFormat.java
##########
@@ -66,6 +91,139 @@
return HoodieRealtimeInputFormatUtils.getRealtimeSplits(job, fileSplits);
}
+ /**
+ * keep the logical of mor_incr_view as same as spark datasource.
+ * to do: unify the incremental view code between hive/spark-sql and spark datasource
+ */
+ @Override
+ protected List<FileStatus> listStatusForIncrementalMode(
+ JobConf job, HoodieTableMetaClient tableMetaClient, List<Path> inputPaths) throws IOException {
+ List<FileStatus> result = new ArrayList<>();
+ String tableName = tableMetaClient.getTableConfig().getTableName();
+ Job jobContext = Job.getInstance(job);
+
+ Option<HoodieTimeline> timeline = HoodieInputFormatUtils.getFilteredCommitsTimeline(jobContext, tableMetaClient);
+ if (!timeline.isPresent()) {
+ return result;
+ }
+ String lastIncrementalTs = HoodieHiveUtils.readStartCommitTime(jobContext, tableName);
+ // Total number of commits to return in this batch. Set this to -1 to get all the commits.
+ Integer maxCommits = HoodieHiveUtils.readMaxCommits(jobContext, tableName);
+ HoodieTimeline commitsTimelineToReturn = timeline.get().findInstantsAfter(lastIncrementalTs, maxCommits);
+ Option<List<HoodieInstant>> commitsToCheck = Option.of(commitsTimelineToReturn.getInstants().collect(Collectors.toList()));
+ if (!commitsToCheck.isPresent()) {
+ return result;
+ }
+ Map<String, HashMap<String, FileStatus>> partitionsWithFileStatus = HoodieInputFormatUtils
+ .listAffectedFilesForCommits(new Path(tableMetaClient.getBasePath()), commitsToCheck.get(), commitsTimelineToReturn);
+ // build fileGroup from fsView
+ List<FileStatus> affectedFileStatus = new ArrayList<>();
+ partitionsWithFileStatus.forEach((key, value) -> value.forEach((k, v) -> affectedFileStatus.add(v)));
+ HoodieTableFileSystemView fsView = new HoodieTableFileSystemView(tableMetaClient, commitsTimelineToReturn, affectedFileStatus.toArray(new FileStatus[0]));
+ // build fileGroup from fsView
+ String basePath = tableMetaClient.getBasePath();
+ // filter affectedPartition by inputPaths
+ List<String> affectedPartition = partitionsWithFileStatus.keySet().stream()
+ .filter(k -> k.isEmpty() ? inputPaths.contains(new Path(basePath)) : inputPaths.contains(new Path(basePath, k))).collect(Collectors.toList());
+ if (affectedPartition.isEmpty()) {
+ return result;
+ }
+ List<HoodieFileGroup> fileGroups = affectedPartition.stream()
+ .flatMap(partitionPath -> fsView.getAllFileGroups(partitionPath)).collect(Collectors.toList());
+ setInputPaths(job, affectedPartition.stream()
+ .map(p -> p.isEmpty() ? basePath : new Path(basePath, p).toUri().toString()).collect(Collectors.joining(",")));
+
+ // find all file status in current partitionPath
+ FileStatus[] fileStatuses = getStatus(job);
+ Map<String, FileStatus> candidateFileStatus = new HashMap<>();
+ for (int i = 0; i < fileStatuses.length; i++) {
+ String key = fileStatuses[i].getPath().toString();
+ candidateFileStatus.put(key, fileStatuses[i]);
+ }
+
+ String maxCommitTime = fsView.getLastInstant().get().getTimestamp();
+ fileGroups.stream().forEach(f -> {
+ try {
+ List<FileSlice> baseFiles = f.getAllFileSlices().filter(slice -> slice.getBaseFile().isPresent()).collect(Collectors.toList());
+ if (!baseFiles.isEmpty()) {
+ FileStatus baseFileStatus = HoodieInputFormatUtils.getFileStatus(baseFiles.get(0).getBaseFile().get());
+ String baseFilePath = baseFileStatus.getPath().toUri().toString();
+ if (!candidateFileStatus.containsKey(baseFilePath)) {
+ throw new HoodieException("Error obtaining fileStatus for file: " + baseFilePath);
+ }
+ RealtimeFileStatus fileStatus = new RealtimeFileStatus(candidateFileStatus.get(baseFilePath));
+ fileStatus.setMaxCommitTime(maxCommitTime);
+ fileStatus.setBelongToIncrementalFileStatus(true);
+ fileStatus.setBasePath(basePath);
+ fileStatus.setBaseFilePath(baseFilePath);
+ fileStatus.setDeltaLogPaths(f.getLatestFileSlice().get().getLogFiles().map(l -> l.getPath().toString()).collect(Collectors.toList()));
Review comment:
Also confused by this logic, why all the file slices here and why we take the first base file then ?
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/PathWithLogFilePath.java
##########
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop;
+
+import org.apache.hadoop.fs.Path;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * we need to encode additional information in Path to track matching log file and base files.
+ * Hence, this weird looking class which tracks an log/base file status
+ */
+public class PathWithLogFilePath extends Path {
+
+ private boolean belongToIncrementalPath = false;
+ private List<String> deltaLogPaths = new ArrayList<>();
Review comment:
Encode additional information in Path to track matching base and log files.
Hence, this clazz tracks both base and log file statuses.
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/BaseFileWithLogsSplit.java
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop;
+
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.apache.hadoop.mapred.FileSplit;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * we need to encode additional information in split to track matching log file and base files.
+ * Hence, this weird looking class which tracks an log/base file split
+ */
+public class BaseFileWithLogsSplit extends FileSplit {
+ // a flag to mark this split is produced by incremental query or not.
+ private boolean belongToIncrementalSplit = false;
+ // the log files of this split.
+ private List<String> deltaLogPaths = new ArrayList<>();
Review comment:
`the log files of this split.` => `the log file paths of this split.`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r725419094
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieMergedLogReader.java
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop.realtime;
+
+import java.io.IOException;
+import java.text.MessageFormat;
+import java.util.Iterator;
+
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.io.ArrayWritable;
+import org.apache.hadoop.io.NullWritable;
+import org.apache.hadoop.io.Writable;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.RecordReader;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.hadoop.utils.HoodieRealtimeRecordReaderUtils;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+/**
+ * Record Reader implementation to read avro data, to support inc queries.
+ */
+public class HoodieMergedLogReader extends AbstractRealtimeRecordReader
+ implements RecordReader<NullWritable, ArrayWritable> {
+ private static final Logger LOG = LogManager.getLogger(AbstractRealtimeRecordReader.class);
+ private final HoodieMergedLogRecordScanner logRecordScanner;
+ private final Iterator<HoodieRecord<? extends HoodieRecordPayload>> logRecordsKeyIterator;
+ private ArrayWritable valueObj;
+
+ private int end;
+ private int offset;
+
+ public HoodieMergedLogReader(RealtimeSplit split, JobConf job, HoodieMergedLogRecordScanner logRecordScanner) {
+ super(split, job);
+ this.logRecordScanner = logRecordScanner;
+ this.end = logRecordScanner.getRecords().size();
+ this.logRecordsKeyIterator = logRecordScanner.iterator();
+ this.valueObj = new ArrayWritable(Writable.class, new Writable[getHiveSchema().getFields().size()]);
+ }
+
+ private Option buildGenericRecord(HoodieRecord record) throws IOException {
+ if (usesCustomPayload) {
+ return record.getData().getInsertValue(getWriterSchema());
+ } else {
+ return record.getData().getInsertValue(getReaderSchema());
+ }
+ }
+
+ @Override
+ public boolean next(NullWritable key, ArrayWritable arrayWritable) throws IOException {
+ if (!logRecordsKeyIterator.hasNext()) {
+ return false;
+ }
+ Option<GenericRecord> rec;
+ HoodieRecord currentRecord = logRecordsKeyIterator.next();
+
+ rec = buildGenericRecord(currentRecord);
+ // try to skip delete record
+ while (!rec.isPresent() && logRecordsKeyIterator.hasNext()) {
+ offset++;
+ rec = buildGenericRecord(logRecordsKeyIterator.next());
+ }
+ if (!rec.isPresent()) {
+ return false;
+ }
+
+ GenericRecord recordToReturn = rec.get();
+ if (usesCustomPayload) {
+ // If using a custom payload, return only the projection fields. The readerSchema is a schema derived from
Review comment:
yes， keep the logical continuity with HoodieMergedLogRecordScanner。 when we build HoodieMergedLogRecordScanner we use usesCustomPayload to deternime the read schema for log file， see follow code from RealTimeCompactedRecordReader。
_return HoodieMergedLogRecordScanner.newBuilder()
.withFileSystem(FSUtils.getFs(split.getPath().toString(), jobConf))
.withBasePath(split.getBasePath())
.withLogFilePaths(split.getDeltaLogPaths())
.withReaderSchema(**usesCustomPayload ? getWriterSchema() : getReaderSchema()**)_
of course, Due to the schema evolution ability of avro， use getWriterSchema() or getReaderSchema() is not different。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a change in pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
danny0405 commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r718116742
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieMergedLogReader.java
##########
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop.realtime;
+
+import java.io.IOException;
+import java.text.MessageFormat;
+import java.util.Iterator;
+
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.io.ArrayWritable;
+import org.apache.hadoop.io.NullWritable;
+import org.apache.hadoop.io.Writable;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.RecordReader;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.hadoop.utils.HoodieRealtimeRecordReaderUtils;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+/**
+ * Record Reader implementation to read avro data, to support inc queries.
+ */
+public class HoodieMergedLogReader extends AbstractRealtimeRecordReader
+ implements RecordReader<NullWritable, ArrayWritable> {
+ private static final Logger LOG = LogManager.getLogger(AbstractRealtimeRecordReader.class);
+ private final HoodieMergedLogRecordScanner logRecordScanner;
+ private final Iterator<HoodieRecord<? extends HoodieRecordPayload>> logRecordsKeyIterator;
+ private ArrayWritable valueObj;
+ private boolean supportTimeStamp = false;
+
+ private int end;
+ private int offset;
+
+ public HoodieMergedLogReader(RealtimeSplit split, JobConf job, HoodieMergedLogRecordScanner logRecordScanner) {
+ super(split, job);
+ this.logRecordScanner = logRecordScanner;
+ this.end = logRecordScanner.getRecords().size();
+ this.logRecordsKeyIterator = logRecordScanner.iterator();
+ this.valueObj = new ArrayWritable(Writable.class, new Writable[getHiveSchema().getFields().size()]);
+ }
+
+ private Option buildGenericRecordwithCustomPayload(HoodieRecord record) throws IOException {
+ if (usesCustomPayload) {
+ return record.getData().getInsertValue(getWriterSchema());
+ } else {
+ return record.getData().getInsertValue(getReaderSchema());
+ }
+ }
+
+ @Override
+ public boolean next(NullWritable key, ArrayWritable arrayWritable) throws IOException {
+ if (!logRecordsKeyIterator.hasNext()) {
+ return false;
+ }
+ Option<GenericRecord> rec;
+ HoodieRecord currentRecord = logRecordsKeyIterator.next();
+
+ rec = buildGenericRecordwithCustomPayload(currentRecord);
+ // try to skip delete record
+ while (!rec.isPresent() && logRecordsKeyIterator.hasNext()) {
+ offset++;
+ rec = buildGenericRecordwithCustomPayload(logRecordsKeyIterator.next());
+ }
+ if (!rec.isPresent()) {
+ return false;
+ }
+
+ GenericRecord recordToReturn = rec.get();
+ if (usesCustomPayload) {
+ // If using a custom payload, return only the projection fields. The readerSchema is a schema derived from
+ // the writerSchema with only the projection fields
+ recordToReturn = HoodieAvroUtils.rewriteRecord(rec.get(), getReaderSchema());
+ }
+ ArrayWritable curWritable = (ArrayWritable) HoodieRealtimeRecordReaderUtils.avroToArrayWritable(recordToReturn, getHiveSchema());
+
+ if (arrayWritable != curWritable) {
+ final Writable[] arrValue = arrayWritable.get();
+ final Writable[] arrCurrent = curWritable.get();
+ if (arrayWritable != null && arrValue.length == arrCurrent.length) {
+ System.arraycopy(arrCurrent, 0, arrValue, 0, arrCurrent.length);
Review comment:
`arrayWritable != null` should never reach because `arrayWritable.get()` is invoked.
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieMergedLogReader.java
##########
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop.realtime;
+
+import java.io.IOException;
+import java.text.MessageFormat;
+import java.util.Iterator;
+
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.io.ArrayWritable;
+import org.apache.hadoop.io.NullWritable;
+import org.apache.hadoop.io.Writable;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.RecordReader;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.hadoop.utils.HoodieRealtimeRecordReaderUtils;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+/**
+ * Record Reader implementation to read avro data, to support inc queries.
+ */
+public class HoodieMergedLogReader extends AbstractRealtimeRecordReader
+ implements RecordReader<NullWritable, ArrayWritable> {
+ private static final Logger LOG = LogManager.getLogger(AbstractRealtimeRecordReader.class);
+ private final HoodieMergedLogRecordScanner logRecordScanner;
+ private final Iterator<HoodieRecord<? extends HoodieRecordPayload>> logRecordsKeyIterator;
+ private ArrayWritable valueObj;
+ private boolean supportTimeStamp = false;
+
+ private int end;
+ private int offset;
+
+ public HoodieMergedLogReader(RealtimeSplit split, JobConf job, HoodieMergedLogRecordScanner logRecordScanner) {
+ super(split, job);
+ this.logRecordScanner = logRecordScanner;
+ this.end = logRecordScanner.getRecords().size();
+ this.logRecordsKeyIterator = logRecordScanner.iterator();
+ this.valueObj = new ArrayWritable(Writable.class, new Writable[getHiveSchema().getFields().size()]);
+ }
+
+ private Option buildGenericRecordwithCustomPayload(HoodieRecord record) throws IOException {
+ if (usesCustomPayload) {
Review comment:
buildGenericRecord
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieMergedLogReader.java
##########
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop.realtime;
+
+import java.io.IOException;
+import java.text.MessageFormat;
+import java.util.Iterator;
+
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.io.ArrayWritable;
+import org.apache.hadoop.io.NullWritable;
+import org.apache.hadoop.io.Writable;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.RecordReader;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.hadoop.utils.HoodieRealtimeRecordReaderUtils;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+/**
+ * Record Reader implementation to read avro data, to support inc queries.
+ */
+public class HoodieMergedLogReader extends AbstractRealtimeRecordReader
+ implements RecordReader<NullWritable, ArrayWritable> {
+ private static final Logger LOG = LogManager.getLogger(AbstractRealtimeRecordReader.class);
+ private final HoodieMergedLogRecordScanner logRecordScanner;
+ private final Iterator<HoodieRecord<? extends HoodieRecordPayload>> logRecordsKeyIterator;
+ private ArrayWritable valueObj;
+ private boolean supportTimeStamp = false;
+
+ private int end;
+ private int offset;
+
+ public HoodieMergedLogReader(RealtimeSplit split, JobConf job, HoodieMergedLogRecordScanner logRecordScanner) {
+ super(split, job);
+ this.logRecordScanner = logRecordScanner;
+ this.end = logRecordScanner.getRecords().size();
+ this.logRecordsKeyIterator = logRecordScanner.iterator();
+ this.valueObj = new ArrayWritable(Writable.class, new Writable[getHiveSchema().getFields().size()]);
+ }
+
+ private Option buildGenericRecordwithCustomPayload(HoodieRecord record) throws IOException {
+ if (usesCustomPayload) {
+ return record.getData().getInsertValue(getWriterSchema());
+ } else {
+ return record.getData().getInsertValue(getReaderSchema());
+ }
+ }
+
+ @Override
+ public boolean next(NullWritable key, ArrayWritable arrayWritable) throws IOException {
+ if (!logRecordsKeyIterator.hasNext()) {
+ return false;
+ }
+ Option<GenericRecord> rec;
+ HoodieRecord currentRecord = logRecordsKeyIterator.next();
+
+ rec = buildGenericRecordwithCustomPayload(currentRecord);
+ // try to skip delete record
+ while (!rec.isPresent() && logRecordsKeyIterator.hasNext()) {
+ offset++;
+ rec = buildGenericRecordwithCustomPayload(logRecordsKeyIterator.next());
+ }
+ if (!rec.isPresent()) {
+ return false;
+ }
+
+ GenericRecord recordToReturn = rec.get();
+ if (usesCustomPayload) {
+ // If using a custom payload, return only the projection fields. The readerSchema is a schema derived from
+ // the writerSchema with only the projection fields
+ recordToReturn = HoodieAvroUtils.rewriteRecord(rec.get(), getReaderSchema());
+ }
+ ArrayWritable curWritable = (ArrayWritable) HoodieRealtimeRecordReaderUtils.avroToArrayWritable(recordToReturn, getHiveSchema());
+
+ if (arrayWritable != curWritable) {
+ final Writable[] arrValue = arrayWritable.get();
+ final Writable[] arrCurrent = curWritable.get();
+ if (arrayWritable != null && arrValue.length == arrCurrent.length) {
+ System.arraycopy(arrCurrent, 0, arrValue, 0, arrCurrent.length);
+ } else {
+ if (arrValue.length != arrCurrent.length) {
+ throw new IOException(MessageFormat.format("HoodieMergeLogReader : size of object differs. Value size : {0}, Current Object size : {1}",
+ arrValue.length, arrCurrent.length));
+ } else {
+ throw new IOException("HoodieMergeLogReader can not support RecordReaders that don't return same key & value & value is null");
+ }
+ }
+ }
+
+ offset++;
+ return true;
+ }
+
+ @Override
+ public NullWritable createKey() {
+ return null;
+ }
+
+ @Override
+ public ArrayWritable createValue() {
+ return valueObj;
+ }
+
+ @Override
+ public long getPos() throws IOException {
+ return offset;
+ }
+
+ @Override
+ public void close() throws IOException {
+ if (logRecordScanner != null) {
+ logRecordScanner.close();
+ }
+ }
+
+ @Override
+ public float getProgress() throws IOException {
+ if (end == offset) {
+ return 0.0f;
Review comment:
Is this correct ?
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/fs/FSUtils.java
##########
@@ -336,6 +336,11 @@ public static String getFileExtensionFromLog(Path logPath) {
return matcher.group(3);
}
+ public static String getLogFileExtension(String fullName) {
+ Matcher matcher = LOG_FILE_PATTERN.matcher(fullName);
Review comment:
What's the difference with `getFileExtensionFromLog` ?
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieInputFormatUtils.java
##########
@@ -445,10 +457,21 @@ public static HoodieMetadataConfig buildMetadataConfig(Configuration conf) {
HoodieTableFileSystemView fsView = fsViewCache.computeIfAbsent(metaClient, tableMetaClient ->
FileSystemViewManager.createInMemoryFileSystemViewWithTimeline(engineContext, tableMetaClient, buildMetadataConfig(job), timeline));
List<HoodieBaseFile> filteredBaseFiles = new ArrayList<>();
+ Map<HoodieLogFile, List<String>> filteredLogs = new HashMap<>();
for (Path p : entry.getValue()) {
Review comment:
The cache key is weird.
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/BaseFileWithLogsSplit.java
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop;
+
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.apache.hadoop.mapred.FileSplit;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+public class BaseFileWithLogsSplit extends FileSplit {
Review comment:
Add class document.
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieParquetRealtimeInputFormat.java
##########
@@ -119,6 +275,11 @@ void addProjectionToJobConf(final RealtimeSplit realtimeSplit, final JobConf job
addProjectionToJobConf(realtimeSplit, jobConf);
LOG.info("Creating record reader with readCols :" + jobConf.get(ColumnProjectionUtils.READ_COLUMN_NAMES_CONF_STR)
+ ", Ids :" + jobConf.get(ColumnProjectionUtils.READ_COLUMN_IDS_CONF_STR));
+
+ // for log only split, we no need parquet reader, set it to empty
+ if (realtimeSplit.getPath().toString().contains(".log.")) {
+ return new HoodieRealtimeRecordReader(realtimeSplit, jobConf, new HoodieEmptyRecordReader());
+ }
Review comment:
Use `FSUtils.isLogFile` ?
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/PathWithLogFilePath.java
##########
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop;
+
+import org.apache.hadoop.fs.Path;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * we need to encode additional information in Path to track matching log file and base files.
+ * Hence, this weird looking class which tracks an log/base file status
+ */
+public class PathWithLogFilePath extends Path {
+
+ private boolean belongToIncrementalPath = false;
+ private List<String> deltaLogPaths = new ArrayList<>();
Review comment:
Ad comments for each member.
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/RealtimeCompactedRecordReader.java
##########
@@ -55,13 +55,20 @@
private final Set<String> deltaRecordKeys;
private int recordKeyIndex = HoodieInputFormatUtils.HOODIE_RECORD_KEY_COL_POS;
private Iterator<String> deltaItr;
+ private boolean logFileOnlySplit;
+ private HoodieMergedLogReader logReader;
public RealtimeCompactedRecordReader(RealtimeSplit split, JobConf job,
RecordReader<NullWritable, ArrayWritable> realReader) throws IOException {
super(split, job);
this.parquetReader = realReader;
- this.deltaRecordMap = getMergedLogRecordScanner().getRecords();
+ HoodieMergedLogRecordScanner hoodieMergedLogRecordScanner = getMergedLogRecordScanner();
+ this.deltaRecordMap = hoodieMergedLogRecordScanner.getRecords();
this.deltaRecordKeys = new HashSet<>(this.deltaRecordMap.keySet());
+ if (split.getPath().toString().contains(".log.")) {
+ this.logFileOnlySplit = true;
Review comment:
Keep `logReader` as `Option< HoodieMergedLogReader >` then we can eliminate the flag `logFileOnlySplit`.
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieParquetRealtimeInputFormat.java
##########
@@ -66,6 +90,138 @@
return HoodieRealtimeInputFormatUtils.getRealtimeSplits(job, fileSplits);
}
+ /**
+ * keep the logical of mor_incr_view as same as spark datasource.
+ * to do: unify the incremental view code between hive/spark-sql and spark datasource
Review comment:
Can we unify the logic in this PR ?
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/RealtimeCompactedRecordReader.java
##########
@@ -55,13 +55,20 @@
private final Set<String> deltaRecordKeys;
private int recordKeyIndex = HoodieInputFormatUtils.HOODIE_RECORD_KEY_COL_POS;
private Iterator<String> deltaItr;
+ private boolean logFileOnlySplit;
+ private HoodieMergedLogReader logReader;
public RealtimeCompactedRecordReader(RealtimeSplit split, JobConf job,
RecordReader<NullWritable, ArrayWritable> realReader) throws IOException {
super(split, job);
this.parquetReader = realReader;
- this.deltaRecordMap = getMergedLogRecordScanner().getRecords();
+ HoodieMergedLogRecordScanner hoodieMergedLogRecordScanner = getMergedLogRecordScanner();
+ this.deltaRecordMap = hoodieMergedLogRecordScanner.getRecords();
this.deltaRecordKeys = new HashSet<>(this.deltaRecordMap.keySet());
+ if (split.getPath().toString().contains(".log.")) {
+ this.logFileOnlySplit = true;
Review comment:
Use `FSUtils.isLogFile` ?
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieRealtimeInputFormatUtils.java
##########
@@ -120,28 +147,32 @@
.orElse(Stream.empty());
// subgroup splits again by file id & match with log files.
- Map<String, List<FileSplit>> groupedInputSplits = partitionsToParquetSplits.get(partitionPath).stream()
- .collect(Collectors.groupingBy(split -> FSUtils.getFileId(split.getPath().getName())));
+ Map<String, List<FileSplit>> groupedInputSplits = filteredPartitionsToParquetSplits.get(partitionPath).stream()
+ .collect(Collectors.groupingBy(split -> {
+ String fileName = split.getPath().getName();
+ // mor snapsht view may contains some raw log files, for example: if user use Hbase index, hudi will produce only log Files
+ if (split instanceof BaseFileWithLogsSplit && ((BaseFileWithLogsSplit) split).getBaseFilePath().isEmpty()) {
+ fileName = split.getPath().getName().substring(1, fileName.length() - 1);
+ }
Review comment:
`FSUtils.getFileIdFromLogPath` ?
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieParquetRealtimeInputFormat.java
##########
@@ -66,6 +90,138 @@
return HoodieRealtimeInputFormatUtils.getRealtimeSplits(job, fileSplits);
}
+ /**
+ * keep the logical of mor_incr_view as same as spark datasource.
+ * to do: unify the incremental view code between hive/spark-sql and spark datasource
+ */
+ @Override
+ protected List<FileStatus> listStatusForIncrementalMode(
+ JobConf job, HoodieTableMetaClient tableMetaClient, List<Path> inputPaths) throws IOException {
+ List<FileStatus> result = new ArrayList<>();
+ String tableName = tableMetaClient.getTableConfig().getTableName();
+ Job jobContext = Job.getInstance(job);
+
+ Option<HoodieTimeline> timeline = HoodieInputFormatUtils.getFilteredCommitsTimeline(jobContext, tableMetaClient);
+ if (!timeline.isPresent()) {
+ return result;
+ }
+ String lastIncrementalTs = HoodieHiveUtils.readStartCommitTime(jobContext, tableName);
+ // Total number of commits to return in this batch. Set this to -1 to get all the commits.
+ Integer maxCommits = HoodieHiveUtils.readMaxCommits(jobContext, tableName);
+ HoodieTimeline commitsTimelineToReturn = timeline.get().findInstantsAfter(lastIncrementalTs, maxCommits);
+ Option<List<HoodieInstant>> commitsToCheck = Option.of(commitsTimelineToReturn.getInstants().collect(Collectors.toList()));
+ if (!commitsToCheck.isPresent()) {
+ return result;
+ }
+ Map<String, HashMap<String, FileStatus>> partitionsWithFileStatus = HoodieInputFormatUtils
+ .listAffectedFilesForCommits(new Path(tableMetaClient.getBasePath()), commitsToCheck.get(), commitsTimelineToReturn);
+ // build fileGroup from fsView
+ List<FileStatus> affectedFileStatus = new ArrayList<>();
+ partitionsWithFileStatus.forEach((key, value) -> value.forEach((k, v) -> affectedFileStatus.add(v)));
+ HoodieTableFileSystemView fsView = new HoodieTableFileSystemView(tableMetaClient, commitsTimelineToReturn, affectedFileStatus.toArray(new FileStatus[0]));
+ // build fileGroup from fsView
+ String basePath = tableMetaClient.getBasePath();
+ // filter affectedPartition by inputPaths
+ List<String> affectedPartition = partitionsWithFileStatus.keySet().stream()
+ .filter(k -> k.isEmpty() ? inputPaths.contains(new Path(basePath)) : inputPaths.contains(new Path(basePath, k))).collect(Collectors.toList());
+ if (affectedPartition.isEmpty()) {
+ return result;
+ }
+ List<HoodieFileGroup> fileGroups = affectedPartition.stream()
+ .flatMap(partitionPath -> fsView.getAllFileGroups(partitionPath)).collect(Collectors.toList());
+ setInputPaths(job, affectedPartition.stream()
+ .map(p -> p.isEmpty() ? basePath : new Path(basePath, p).toUri().toString()).collect(Collectors.joining(",")));
+
+ // find all file status in current partitionPath
+ FileStatus[] fileStatuses = getStatus(job);
+ Map<String, FileStatus> candidateFileStatus = new HashMap<>();
+ for (int i = 0; i < fileStatuses.length; i++) {
+ String key = fileStatuses[i].getPath().toString();
+ candidateFileStatus.put(key, fileStatuses[i]);
+ }
+
+ String maxCommitTime = fsView.getLastInstant().get().getTimestamp();
+ fileGroups.stream().forEach(f -> {
+ try {
+ List<FileSlice> baseFiles = f.getAllFileSlices().filter(slice -> slice.getBaseFile().isPresent()).collect(Collectors.toList());
+ if (!baseFiles.isEmpty()) {
+ FileStatus baseFileStatus = HoodieInputFormatUtils.getFileStatus(baseFiles.get(0).getBaseFile().get());
+ String baseFilePath = baseFileStatus.getPath().toUri().toString();
+ if (!candidateFileStatus.containsKey(baseFilePath)) {
+ throw new HoodieException("Error obtaining fileStatus for file: " + baseFilePath);
+ }
+ RealtimeFileStatus fileStatus = new RealtimeFileStatus(candidateFileStatus.get(baseFilePath));
+ fileStatus.setMaxCommitTime(maxCommitTime);
+ fileStatus.setBelongToIncrementalFileStatus(true);
+ fileStatus.setBasePath(basePath);
+ fileStatus.setBaseFilePath(baseFilePath);
+ fileStatus.setDeltaLogPaths(f.getLatestFileSlice().get().getLogFiles().map(l -> l.getPath().toString()).collect(Collectors.toList()));
+ // try to set bootstrapfileStatus
+ if (baseFileStatus instanceof LocatedFileStatusWithBootstrapBaseFile || baseFileStatus instanceof FileStatusWithBootstrapBaseFile) {
+ fileStatus.setBootStrapFileStatus(baseFileStatus);
+ }
+ result.add(fileStatus);
+ }
+ // add file group which has only logs.
+ if (f.getLatestFileSlice().isPresent() && baseFiles.isEmpty()) {
+ List<FileStatus> logFileStatus = f.getLatestFileSlice().get().getLogFiles().map(logFile -> logFile.getFileStatus()).collect(Collectors.toList());
+ if (logFileStatus.size() > 0) {
+ RealtimeFileStatus fileStatus = new RealtimeFileStatus(logFileStatus.get(0));
+ fileStatus.setBelongToIncrementalFileStatus(true);
+ fileStatus.setDeltaLogPaths(logFileStatus.stream().map(l -> l.getPath().toString()).collect(Collectors.toList()));
+ fileStatus.setMaxCommitTime(maxCommitTime);
+ fileStatus.setBasePath(basePath);
+ result.add(fileStatus);
+ }
+ }
+ } catch (IOException e) {
+ throw new HoodieException("Error obtaining data file/log file grouping ", e);
+ }
+ });
+ return result;
+ }
+
+ @Override
+ protected boolean includeLogFilesForSnapShotView() {
+ return true;
+ }
+
+ @Override
+ protected boolean isSplitable(FileSystem fs, Path filename) {
+ if (filename instanceof PathWithLogFilePath) {
+ return ((PathWithLogFilePath)filename).splitable();
+ }
+ return super.isSplitable(fs, filename);
+ }
+
+ @Override
+ protected FileSplit makeSplit(Path file, long start, long length, String[] hosts) {
+ if (file instanceof PathWithLogFilePath) {
+ PathWithLogFilePath pf = (PathWithLogFilePath) file;
+ if (pf.getPathWithBootstrapFileStatus() == null) {
+ return pf.buildSplit(file, start, length, hosts);
+ } else {
Review comment:
add a method `doMakeSplit` to abstract the variable `String[] inMemoryHosts`, then we only need one piece of code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
danny0405 commented on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-929775609
Changes the title and commit message to "[HUDI-2086] Redo the logic of mor incremental view for hive"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r718193779
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieMergedLogReader.java
##########
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop.realtime;
+
+import java.io.IOException;
+import java.text.MessageFormat;
+import java.util.Iterator;
+
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.io.ArrayWritable;
+import org.apache.hadoop.io.NullWritable;
+import org.apache.hadoop.io.Writable;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.RecordReader;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.hadoop.utils.HoodieRealtimeRecordReaderUtils;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+/**
+ * Record Reader implementation to read avro data, to support inc queries.
+ */
+public class HoodieMergedLogReader extends AbstractRealtimeRecordReader
+ implements RecordReader<NullWritable, ArrayWritable> {
+ private static final Logger LOG = LogManager.getLogger(AbstractRealtimeRecordReader.class);
+ private final HoodieMergedLogRecordScanner logRecordScanner;
+ private final Iterator<HoodieRecord<? extends HoodieRecordPayload>> logRecordsKeyIterator;
+ private ArrayWritable valueObj;
+ private boolean supportTimeStamp = false;
+
+ private int end;
+ private int offset;
+
+ public HoodieMergedLogReader(RealtimeSplit split, JobConf job, HoodieMergedLogRecordScanner logRecordScanner) {
+ super(split, job);
+ this.logRecordScanner = logRecordScanner;
+ this.end = logRecordScanner.getRecords().size();
+ this.logRecordsKeyIterator = logRecordScanner.iterator();
+ this.valueObj = new ArrayWritable(Writable.class, new Writable[getHiveSchema().getFields().size()]);
+ }
+
+ private Option buildGenericRecordwithCustomPayload(HoodieRecord record) throws IOException {
+ if (usesCustomPayload) {
Review comment:
ok
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/PathWithLogFilePath.java
##########
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop;
+
+import org.apache.hadoop.fs.Path;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * we need to encode additional information in Path to track matching log file and base files.
+ * Hence, this weird looking class which tracks an log/base file status
+ */
+public class PathWithLogFilePath extends Path {
+
+ private boolean belongToIncrementalPath = false;
+ private List<String> deltaLogPaths = new ArrayList<>();
Review comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-956216242
@codope thanks for you review. i will try to address all comments tomorrow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-960507145
@danny0405
This pr is try to solve the problem of incremental views in various scenarios.
These problems in https://issues.apache.org/jira/browse/HUDI-2649 have nothing to do with this pr, right?
first question：i test on our cluster it's ok. (hive3.1.1 flink1.12.x)
second question：hudi-hive-sync is not compatible withhive-2.1.1
third question: is a Typical classNotFound problem, i donnot think it's a problem of hudi. i test on our cluster it's ok.(hive3.1.1 flink1.12.x)
other question: Are these problems from the dingding group? Can you pull me into the dingding group?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao edited a comment on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-962975376
@danny0405 @nsivabalan @codope i test latest master code from hudi , there still exist this problem.
this problem has no relate to this pr.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a change in pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
danny0405 commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r736128496
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieRealtimeInputFormatUtils.java
##########
@@ -161,6 +162,46 @@
return rtSplits.toArray(new InputSplit[0]);
}
+ // pick all incremental files and add them to rtSplits then filter out those files.
+ private static Map<Path, List<FileSplit>> filterOutIncrementalSplits(
+ List<FileSplit> fileSplitList,
Review comment:
Thanks, got your idea, the input param `fileSplits` comes from the scanning of Hive input format, so it should include all the files on the query path because `HoodieParquetRealtimeInputFormat#collectAllIncrementalFiles` generates the input splits from all the file slices.
And the incremental query use the same code path here with the snapshot query, which complicates the code, maybe we can split the incremental query code out of the method `getRealtimeSplits` into a new method `getIncrementalRealtimeSplits` ? WDYT :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872264798
# [Codecov](https://codecov.io/gh/apache/hudi/pull/3203?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3203](https://codecov.io/gh/apache/hudi/pull/3203?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (792ae1d) into [master](https://codecov.io/gh/apache/hudi/commit/94f0f40fec44c7eecb4e886df62f52a28226974c?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (94f0f40) will **increase** coverage by `1.49%`.
> The diff coverage is `64.35%`.
[](https://codecov.io/gh/apache/hudi/pull/3203?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3203 +/- ##
============================================
+ Coverage 46.20% 47.69% +1.49%
- Complexity 5389 5503 +114
============================================
Files 921 927 +6
Lines 40053 41262 +1209
Branches 4297 4130 -167
============================================
+ Hits 18505 19679 +1174
- Misses 19664 19831 +167
+ Partials 1884 1752 -132
```
| Flag | Coverage Δ | |
|---|---|---|
| hudicli | `40.00% <ø> (+0.05%)` | :arrow_up: |
| hudiclient | `34.58% <ø> (+4.12%)` | :arrow_up: |
| hudicommon | `48.39% <0.00%> (+0.83%)` | :arrow_up: |
| hudiflink | `60.07% <ø> (+0.16%)` | :arrow_up: |
| hudihadoopmr | `54.29% <64.78%> (+3.00%)` | :arrow_up: |
| hudisparkdatasource | `67.10% <ø> (+0.04%)` | :arrow_up: |
| hudisync | `54.05% <ø> (ø)` | |
| huditimelineservice | `64.07% <ø> (-0.29%)` | :arrow_down: |
| hudiutilities | `58.05% <ø> (-0.54%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3203?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...c/main/java/org/apache/hudi/common/fs/FSUtils.java](https://codecov.io/gh/apache/hudi/pull/3203/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL2ZzL0ZTVXRpbHMuamF2YQ==) | `46.22% <0.00%> (-0.87%)` | :arrow_down: |
| [...adoop/realtime/RealtimeBootstrapBaseFileSplit.java](https://codecov.io/gh/apache/hudi/pull/3203/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL3JlYWx0aW1lL1JlYWx0aW1lQm9vdHN0cmFwQmFzZUZpbGVTcGxpdC5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...rg/apache/hudi/hadoop/HoodieEmptyRecordReader.java](https://codecov.io/gh/apache/hudi/pull/3203/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL0hvb2RpZUVtcHR5UmVjb3JkUmVhZGVyLmphdmE=) | `14.28% <14.28%> (ø)` | |
| [...ache/hudi/hadoop/utils/HoodieInputFormatUtils.java](https://codecov.io/gh/apache/hudi/pull/3203/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL3V0aWxzL0hvb2RpZUlucHV0Rm9ybWF0VXRpbHMuamF2YQ==) | `58.85% <25.00%> (+4.49%)` | :arrow_up: |
| [.../org/apache/hudi/hadoop/BaseFileWithLogsSplit.java](https://codecov.io/gh/apache/hudi/pull/3203/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL0Jhc2VGaWxlV2l0aExvZ3NTcGxpdC5qYXZh) | `51.16% <51.16%> (ø)` | |
| [...he/hudi/hadoop/realtime/HoodieMergedLogReader.java](https://codecov.io/gh/apache/hudi/pull/3203/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL3JlYWx0aW1lL0hvb2RpZU1lcmdlZExvZ1JlYWRlci5qYXZh) | `52.38% <52.38%> (ø)` | |
| [...i/hadoop/utils/HoodieRealtimeInputFormatUtils.java](https://codecov.io/gh/apache/hudi/pull/3203/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL3V0aWxzL0hvb2RpZVJlYWx0aW1lSW5wdXRGb3JtYXRVdGlscy5qYXZh) | `44.20% <59.25%> (+9.42%)` | :arrow_up: |
| [...oop/realtime/HoodieParquetRealtimeInputFormat.java](https://codecov.io/gh/apache/hudi/pull/3203/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL3JlYWx0aW1lL0hvb2RpZVBhcnF1ZXRSZWFsdGltZUlucHV0Rm9ybWF0LmphdmE=) | `80.80% <77.77%> (-0.68%)` | :arrow_down: |
| [...hadoop/realtime/RealtimeCompactedRecordReader.java](https://codecov.io/gh/apache/hudi/pull/3203/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL3JlYWx0aW1lL1JlYWx0aW1lQ29tcGFjdGVkUmVjb3JkUmVhZGVyLmphdmE=) | `67.50% <77.77%> (+1.87%)` | :arrow_up: |
| [...va/org/apache/hudi/hadoop/PathWithLogFilePath.java](https://codecov.io/gh/apache/hudi/pull/3203/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL1BhdGhXaXRoTG9nRmlsZVBhdGguamF2YQ==) | `80.00% <80.00%> (ø)` | |
| ... and [58 more](https://codecov.io/gh/apache/hudi/pull/3203/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/hudi/pull/3203?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/hudi/pull/3203?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [94f0f40...792ae1d](https://codecov.io/gh/apache/hudi/pull/3203?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
}, {
"hash" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325",
"triggerID" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"triggerType" : "PUSH"
}, {
"hash" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2343",
"triggerID" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"triggerType" : "PUSH"
}, {
"hash" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2424",
"triggerID" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2427",
"triggerID" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2429",
"triggerID" : "927749497",
"triggerType" : "MANUAL"
}, {
"hash" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2431",
"triggerID" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"triggerType" : "PUSH"
}, {
"hash" : "29e6acb65044597e35f9d499385dad940580d9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2432",
"triggerID" : "29e6acb65044597e35f9d499385dad940580d9df",
"triggerType" : "PUSH"
}, {
"hash" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2441",
"triggerID" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"triggerType" : "PUSH"
}, {
"hash" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2443",
"triggerID" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0fa6297ce58eb877fd5c4eba59fef20ad9335d26 UNKNOWN
* 423f047e00904b17735a5f0120959de87e4cb0a3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2443)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
}, {
"hash" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325",
"triggerID" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"triggerType" : "PUSH"
}, {
"hash" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2343",
"triggerID" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"triggerType" : "PUSH"
}, {
"hash" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2424",
"triggerID" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2427",
"triggerID" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2429",
"triggerID" : "927749497",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 814b7eb1e147197c29cc2701a08b6bc77bf6c2f8 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2427) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2429)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-928890183
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r716731093
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieParquetRealtimeInputFormat.java
##########
@@ -66,6 +90,139 @@
return HoodieRealtimeInputFormatUtils.getRealtimeSplits(job, fileSplits);
}
+ /**
+ * keep the logical of mor_incr_view as same as spark datasource.
+ * to do: unify the incremental view code between hive/spark-sql and spark datasource
+ */
+ @Override
+ protected List<FileStatus> listStatusForIncrementalMode(
+ JobConf job, HoodieTableMetaClient tableMetaClient, List<Path> inputPaths) throws IOException {
+ List<FileStatus> result = new ArrayList<>();
+ String tableName = tableMetaClient.getTableConfig().getTableName();
+ Job jobContext = Job.getInstance(job);
+
+ Option<HoodieTimeline> timeline = HoodieInputFormatUtils.getFilteredCommitsTimeline(jobContext, tableMetaClient);
+ if (!timeline.isPresent()) {
+ return result;
+ }
+ String lastIncrementalTs = HoodieHiveUtils.readStartCommitTime(jobContext, tableName);
+ // Total number of commits to return in this batch. Set this to -1 to get all the commits.
+ Integer maxCommits = HoodieHiveUtils.readMaxCommits(jobContext, tableName);
+ HoodieTimeline commitsTimelineToReturn = timeline.get().findInstantsAfter(lastIncrementalTs, maxCommits);
+ Option<List<HoodieInstant>> commitsToCheck = Option.of(commitsTimelineToReturn.getInstants().collect(Collectors.toList()));
+ if (!commitsToCheck.isPresent()) {
+ return result;
+ }
+ Map<String, HashMap<String, FileStatus>> partitionsWithFileStatus = HoodieInputFormatUtils
+ .listAffectedFilesForCommits(new Path(tableMetaClient.getBasePath()), commitsToCheck.get(), commitsTimelineToReturn);
+ // build fileGroup from fsView
+ List<FileStatus> affectedFileStatus = new ArrayList<>();
+ partitionsWithFileStatus.forEach((key, value) -> value.forEach((k, v) -> affectedFileStatus.add(v)));
+ HoodieTableFileSystemView fsView = new HoodieTableFileSystemView(tableMetaClient, commitsTimelineToReturn, affectedFileStatus.toArray(new FileStatus[0]));
+ // build fileGroup from fsView
+ String basePath = tableMetaClient.getBasePath();
+ // filter affectedPartition by inputPaths
+ List<String> affectedPartition = partitionsWithFileStatus.keySet().stream()
+ .filter(k -> k.isEmpty() ? inputPaths.contains(new Path(basePath)) : inputPaths.contains(new Path(basePath, k))).collect(Collectors.toList());
+ if (affectedPartition.isEmpty()) {
+ return result;
+ }
+ List<HoodieFileGroup> fileGroups = affectedPartition.stream()
+ .flatMap(partitionPath -> fsView.getAllFileGroups(partitionPath)).collect(Collectors.toList());
+ setInputPaths(job, affectedPartition.stream()
+ .map(p -> p.isEmpty() ? basePath : new Path(basePath, p).toUri().toString()).collect(Collectors.joining(",")));
+
+ // find all file status in current partitionPath
+ FileStatus[] fileStatuses = getStatus(job);
+ Map<String, FileStatus> candidateFileStatus = new HashMap<>();
+ for (int i = 0; i < fileStatuses.length; i++) {
+ String key = fileStatuses[i].getPath().toString();
+ candidateFileStatus.put(key, fileStatuses[i]);
+ }
+
+ String maxCommitTime = fsView.getLastInstant().get().getTimestamp();
+ fileGroups.stream().forEach(f -> {
+ try {
+ List<FileSlice> baseFiles = f.getAllFileSlices().filter(slice -> slice.getBaseFile().isPresent()).collect(Collectors.toList());
+ if (!baseFiles.isEmpty()) {
+ FileStatus baseFileStatus = HoodieInputFormatUtils.getFileStatus(baseFiles.get(0).getBaseFile().get());
+ String baseFilePath = baseFileStatus.getPath().toUri().toString();
+ if (!candidateFileStatus.containsKey(baseFilePath)) {
+ throw new HoodieException("Error obtaining fileStatus for file: " + baseFilePath);
+ }
+ RealtimeFileStatus fileStatus = new RealtimeFileStatus(candidateFileStatus.get(baseFilePath));
+ fileStatus.setMaxCommitTime(maxCommitTime);
+ fileStatus.setBelongToIncrementalFileStatus(true);
+ fileStatus.setBasePath(basePath);
+ fileStatus.setBaseFilePath(baseFilePath);
+ fileStatus.setDeltaLogPaths(f.getLatestFileSlice().get().getLogFiles().map(l -> l.getPath().toString()).collect(Collectors.toList()));
+ // try to set bootstrapfileStatus
+ if (baseFileStatus instanceof LocatedFileStatusWithBootstrapBaseFile || baseFileStatus instanceof FileStatusWithBootstrapBaseFile) {
+ fileStatus.setBootStrapFileStatus(baseFileStatus);
+ }
+ result.add(fileStatus);
+ }
+ // add file group which has only logs.
+ if (f.getLatestFileSlice().isPresent() && baseFiles.isEmpty()) {
+ List<FileStatus> logFileStatus = f.getLatestFileSlice().get().getLogFiles().map(logFile -> logFile.getFileStatus()).collect(Collectors.toList());
+ if (logFileStatus.size() > 0) {
+ RealtimeFileStatus fileStatus = new RealtimeFileStatus(logFileStatus.get(0));
+ fileStatus.setLogFileOnly(true);
+ fileStatus.setBelongToIncrementalFileStatus(true);
+ fileStatus.setDeltaLogPaths(logFileStatus.stream().map(l -> l.getPath().toString()).collect(Collectors.toList()));
+ fileStatus.setMaxCommitTime(maxCommitTime);
+ fileStatus.setBasePath(basePath);
+ result.add(fileStatus);
+ }
+ }
+ } catch (IOException e) {
+ throw new HoodieException("Error obtaining data file/log file grouping ", e);
+ }
+ });
+ return result;
+ }
+
+ @Override
+ protected boolean includeLogFilesForSnapShortView() {
+ return false;
Review comment:
here should return `true` since the RealtimeInputFormat contains log files?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
}, {
"hash" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325",
"triggerID" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"triggerType" : "PUSH"
}, {
"hash" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2343",
"triggerID" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"triggerType" : "PUSH"
}, {
"hash" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2424",
"triggerID" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2427",
"triggerID" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2429",
"triggerID" : "927749497",
"triggerType" : "MANUAL"
}, {
"hash" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2431",
"triggerID" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"triggerType" : "PUSH"
}, {
"hash" : "29e6acb65044597e35f9d499385dad940580d9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2432",
"triggerID" : "29e6acb65044597e35f9d499385dad940580d9df",
"triggerType" : "PUSH"
}, {
"hash" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2441",
"triggerID" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"triggerType" : "PUSH"
}, {
"hash" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2443",
"triggerID" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2447",
"triggerID" : "928890183",
"triggerType" : "MANUAL"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2470",
"triggerID" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"triggerType" : "PUSH"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2560",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562",
"triggerID" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"triggerType" : "PUSH"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
}, {
"hash" : "6fa031e1dcf6e21dfb24cf4296cb00d910b0a204",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2589",
"triggerID" : "6fa031e1dcf6e21dfb24cf4296cb00d910b0a204",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0fa6297ce58eb877fd5c4eba59fef20ad9335d26 UNKNOWN
* 6eebb1a711e3655061d471c2a96b54e205dac630 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562)
* 6fa031e1dcf6e21dfb24cf4296cb00d910b0a204 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2589)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
}, {
"hash" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325",
"triggerID" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"triggerType" : "PUSH"
}, {
"hash" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2343",
"triggerID" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"triggerType" : "PUSH"
}, {
"hash" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2424",
"triggerID" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2427",
"triggerID" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2429",
"triggerID" : "927749497",
"triggerType" : "MANUAL"
}, {
"hash" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2431",
"triggerID" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"triggerType" : "PUSH"
}, {
"hash" : "29e6acb65044597e35f9d499385dad940580d9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2432",
"triggerID" : "29e6acb65044597e35f9d499385dad940580d9df",
"triggerType" : "PUSH"
}, {
"hash" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2441",
"triggerID" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"triggerType" : "PUSH"
}, {
"hash" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 158bfbfd6991b583b3b708922459fb2c0eaf4f62 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2441)
* 0fa6297ce58eb877fd5c4eba59fef20ad9335d26 UNKNOWN
* 423f047e00904b17735a5f0120959de87e4cb0a3 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
}, {
"hash" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325",
"triggerID" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"triggerType" : "PUSH"
}, {
"hash" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2343",
"triggerID" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"triggerType" : "PUSH"
}, {
"hash" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c470b4c2c6a4d1d33a5facf729971b4331fe0771 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2343)
* 679a0104259b01a85be2def1b5cd396ae4f0e858 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 792ae1d9884eac5eb7004e4edb617ea6e86d79b7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
}, {
"hash" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325",
"triggerID" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"triggerType" : "PUSH"
}, {
"hash" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2343",
"triggerID" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"triggerType" : "PUSH"
}, {
"hash" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2424",
"triggerID" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2427",
"triggerID" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2429",
"triggerID" : "927749497",
"triggerType" : "MANUAL"
}, {
"hash" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2431",
"triggerID" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"triggerType" : "PUSH"
}, {
"hash" : "29e6acb65044597e35f9d499385dad940580d9df",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2432",
"triggerID" : "29e6acb65044597e35f9d499385dad940580d9df",
"triggerType" : "PUSH"
}, {
"hash" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 29e6acb65044597e35f9d499385dad940580d9df Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2432)
* 158bfbfd6991b583b3b708922459fb2c0eaf4f62 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
}, {
"hash" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325",
"triggerID" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"triggerType" : "PUSH"
}, {
"hash" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2343",
"triggerID" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"triggerType" : "PUSH"
}, {
"hash" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2424",
"triggerID" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2427",
"triggerID" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 814b7eb1e147197c29cc2701a08b6bc77bf6c2f8 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2427)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r716722524
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/RealtimeFileStatus.java
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+public class RealtimeFileStatus extends FileStatus {
+ private boolean belongToIncrementalFileStatus = false;
+ private boolean logFileOnly = false;
+ private List<String> deltaLogPaths = new ArrayList<>();
Review comment:
is non-empty `deltaLogPaths` and empty `baseFilePath` means `logFileOnly = true`, do we really need the field?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
}, {
"hash" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325",
"triggerID" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"triggerType" : "PUSH"
}, {
"hash" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2343",
"triggerID" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"triggerType" : "PUSH"
}, {
"hash" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2424",
"triggerID" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 679a0104259b01a85be2def1b5cd396ae4f0e858 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2424)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
}, {
"hash" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325",
"triggerID" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"triggerType" : "PUSH"
}, {
"hash" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2343",
"triggerID" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"triggerType" : "PUSH"
}, {
"hash" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2424",
"triggerID" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2427",
"triggerID" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2429",
"triggerID" : "927749497",
"triggerType" : "MANUAL"
}, {
"hash" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2431",
"triggerID" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"triggerType" : "PUSH"
}, {
"hash" : "29e6acb65044597e35f9d499385dad940580d9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2432",
"triggerID" : "29e6acb65044597e35f9d499385dad940580d9df",
"triggerType" : "PUSH"
}, {
"hash" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2441",
"triggerID" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"triggerType" : "PUSH"
}, {
"hash" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2443",
"triggerID" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2447",
"triggerID" : "928890183",
"triggerType" : "MANUAL"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2470",
"triggerID" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0fa6297ce58eb877fd5c4eba59fef20ad9335d26 UNKNOWN
* 423f047e00904b17735a5f0120959de87e4cb0a3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2443) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2447)
* c14bded2e903e9c55d86fddc77d40effccca5e01 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2470)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-953860153
cc @codope to also review this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
}, {
"hash" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325",
"triggerID" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"triggerType" : "PUSH"
}, {
"hash" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2343",
"triggerID" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"triggerType" : "PUSH"
}, {
"hash" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2424",
"triggerID" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2427",
"triggerID" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2429",
"triggerID" : "927749497",
"triggerType" : "MANUAL"
}, {
"hash" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2431",
"triggerID" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"triggerType" : "PUSH"
}, {
"hash" : "29e6acb65044597e35f9d499385dad940580d9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2432",
"triggerID" : "29e6acb65044597e35f9d499385dad940580d9df",
"triggerType" : "PUSH"
}, {
"hash" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2441",
"triggerID" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"triggerType" : "PUSH"
}, {
"hash" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2443",
"triggerID" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2447",
"triggerID" : "928890183",
"triggerType" : "MANUAL"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2470",
"triggerID" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"triggerType" : "PUSH"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2560",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562",
"triggerID" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"triggerType" : "PUSH"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
}, {
"hash" : "6fa031e1dcf6e21dfb24cf4296cb00d910b0a204",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2589",
"triggerID" : "6fa031e1dcf6e21dfb24cf4296cb00d910b0a204",
"triggerType" : "PUSH"
}, {
"hash" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2593",
"triggerID" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"triggerType" : "PUSH"
}, {
"hash" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2604",
"triggerID" : "941865271",
"triggerType" : "MANUAL"
}, {
"hash" : "6d67b68e19a43f8668e5773d27ca9c33a8de0a37",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2642",
"triggerID" : "6d67b68e19a43f8668e5773d27ca9c33a8de0a37",
"triggerType" : "PUSH"
}, {
"hash" : "1a80559bd98829552acffdaf20d3ea0384d1d936",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2645",
"triggerID" : "1a80559bd98829552acffdaf20d3ea0384d1d936",
"triggerType" : "PUSH"
}, {
"hash" : "c4af04dab3dab31ef05ba6007000738a2dfb81ce",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2649",
"triggerID" : "c4af04dab3dab31ef05ba6007000738a2dfb81ce",
"triggerType" : "PUSH"
}, {
"hash" : "369c5d6f1f4b6eed3c86f838fb6148d127cc6703",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2660",
"triggerID" : "369c5d6f1f4b6eed3c86f838fb6148d127cc6703",
"triggerType" : "PUSH"
}, {
"hash" : "369c5d6f1f4b6eed3c86f838fb6148d127cc6703",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2898",
"triggerID" : "952859446",
"triggerType" : "MANUAL"
}, {
"hash" : "196a40e3273e98f2317856a55de3bd1888e9f754",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2899",
"triggerID" : "196a40e3273e98f2317856a55de3bd1888e9f754",
"triggerType" : "PUSH"
}, {
"hash" : "196a40e3273e98f2317856a55de3bd1888e9f754",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2899",
"triggerID" : "952859446",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 0fa6297ce58eb877fd5c4eba59fef20ad9335d26 UNKNOWN
* 196a40e3273e98f2317856a55de3bd1888e9f754 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2899)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r732502115
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieRealtimeInputFormatUtils.java
##########
@@ -161,6 +162,46 @@
return rtSplits.toArray(new InputSplit[0]);
}
+ // pick all incremental files and add them to rtSplits then filter out those files.
+ private static Map<Path, List<FileSplit>> filterOutIncrementalSplits(
+ List<FileSplit> fileSplitList,
Review comment:
@danny0405 thanks。 i will try and update the code。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
}, {
"hash" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325",
"triggerID" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"triggerType" : "PUSH"
}, {
"hash" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2343",
"triggerID" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"triggerType" : "PUSH"
}, {
"hash" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2424",
"triggerID" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2427",
"triggerID" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2429",
"triggerID" : "927749497",
"triggerType" : "MANUAL"
}, {
"hash" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2431",
"triggerID" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"triggerType" : "PUSH"
}, {
"hash" : "29e6acb65044597e35f9d499385dad940580d9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2432",
"triggerID" : "29e6acb65044597e35f9d499385dad940580d9df",
"triggerType" : "PUSH"
}, {
"hash" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2441",
"triggerID" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"triggerType" : "PUSH"
}, {
"hash" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2443",
"triggerID" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2447",
"triggerID" : "928890183",
"triggerType" : "MANUAL"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2470",
"triggerID" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"triggerType" : "PUSH"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2560",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562",
"triggerID" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"triggerType" : "PUSH"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
}, {
"hash" : "6fa031e1dcf6e21dfb24cf4296cb00d910b0a204",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2589",
"triggerID" : "6fa031e1dcf6e21dfb24cf4296cb00d910b0a204",
"triggerType" : "PUSH"
}, {
"hash" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0fa6297ce58eb877fd5c4eba59fef20ad9335d26 UNKNOWN
* 6fa031e1dcf6e21dfb24cf4296cb00d910b0a204 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2589)
* b5649459e5e8a5ebf5a140418c16951294a29689 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-960486647
@danny0405 of cource, let me check this issue later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao edited a comment on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-960507145
@danny0405
This pr is try to solve the problem of incremental views in various scenarios.
These problems in https://issues.apache.org/jira/browse/HUDI-2649 have nothing to do with this pr, right?
first question：i test on our cluster it's ok. (hive3.1.1 flink1.12.x)
second question：hudi-hive-sync is not compatible withhive-2.1.1
third question: is a Typical classNotFound problem, i donnot think it's a problem of hudi. i test on our cluster it's ok.(hive3.1.1 flink1.12.x) select count(*) from xx will trigger mr task, but select * from xx will run in local if the data is small enought
other question: Are these problems from the dingding group? Can you pull me into the dingding group?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-962957956
@danny0405 @codope i repeated this problem. but i donnot think it's relate to this patch, let me fix this bug later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao removed a comment on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao removed a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-962957956
@danny0405 @codope i repeated this problem. but i donnot think it's relate to this patch, let me fix this bug later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r725736091
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieParquetRealtimeInputFormat.java
##########
@@ -66,6 +91,139 @@
return HoodieRealtimeInputFormatUtils.getRealtimeSplits(job, fileSplits);
}
+ /**
+ * keep the logical of mor_incr_view as same as spark datasource.
+ * to do: unify the incremental view code between hive/spark-sql and spark datasource
+ */
+ @Override
+ protected List<FileStatus> listStatusForIncrementalMode(
+ JobConf job, HoodieTableMetaClient tableMetaClient, List<Path> inputPaths) throws IOException {
+ List<FileStatus> result = new ArrayList<>();
+ String tableName = tableMetaClient.getTableConfig().getTableName();
+ Job jobContext = Job.getInstance(job);
+
+ Option<HoodieTimeline> timeline = HoodieInputFormatUtils.getFilteredCommitsTimeline(jobContext, tableMetaClient);
+ if (!timeline.isPresent()) {
+ return result;
+ }
+ String lastIncrementalTs = HoodieHiveUtils.readStartCommitTime(jobContext, tableName);
+ // Total number of commits to return in this batch. Set this to -1 to get all the commits.
+ Integer maxCommits = HoodieHiveUtils.readMaxCommits(jobContext, tableName);
+ HoodieTimeline commitsTimelineToReturn = timeline.get().findInstantsAfter(lastIncrementalTs, maxCommits);
+ Option<List<HoodieInstant>> commitsToCheck = Option.of(commitsTimelineToReturn.getInstants().collect(Collectors.toList()));
+ if (!commitsToCheck.isPresent()) {
+ return result;
+ }
+ Map<String, HashMap<String, FileStatus>> partitionsWithFileStatus = HoodieInputFormatUtils
+ .listAffectedFilesForCommits(new Path(tableMetaClient.getBasePath()), commitsToCheck.get(), commitsTimelineToReturn);
+ // build fileGroup from fsView
+ List<FileStatus> affectedFileStatus = new ArrayList<>();
+ partitionsWithFileStatus.forEach((key, value) -> value.forEach((k, v) -> affectedFileStatus.add(v)));
+ HoodieTableFileSystemView fsView = new HoodieTableFileSystemView(tableMetaClient, commitsTimelineToReturn, affectedFileStatus.toArray(new FileStatus[0]));
+ // build fileGroup from fsView
+ String basePath = tableMetaClient.getBasePath();
+ // filter affectedPartition by inputPaths
+ List<String> affectedPartition = partitionsWithFileStatus.keySet().stream()
+ .filter(k -> k.isEmpty() ? inputPaths.contains(new Path(basePath)) : inputPaths.contains(new Path(basePath, k))).collect(Collectors.toList());
+ if (affectedPartition.isEmpty()) {
+ return result;
+ }
+ List<HoodieFileGroup> fileGroups = affectedPartition.stream()
+ .flatMap(partitionPath -> fsView.getAllFileGroups(partitionPath)).collect(Collectors.toList());
+ setInputPaths(job, affectedPartition.stream()
+ .map(p -> p.isEmpty() ? basePath : new Path(basePath, p).toUri().toString()).collect(Collectors.joining(",")));
+
+ // find all file status in current partitionPath
+ FileStatus[] fileStatuses = getStatus(job);
+ Map<String, FileStatus> candidateFileStatus = new HashMap<>();
+ for (int i = 0; i < fileStatuses.length; i++) {
+ String key = fileStatuses[i].getPath().toString();
+ candidateFileStatus.put(key, fileStatuses[i]);
+ }
+
+ String maxCommitTime = fsView.getLastInstant().get().getTimestamp();
+ fileGroups.stream().forEach(f -> {
+ try {
+ List<FileSlice> baseFiles = f.getAllFileSlices().filter(slice -> slice.getBaseFile().isPresent()).collect(Collectors.toList());
+ if (!baseFiles.isEmpty()) {
+ FileStatus baseFileStatus = HoodieInputFormatUtils.getFileStatus(baseFiles.get(0).getBaseFile().get());
+ String baseFilePath = baseFileStatus.getPath().toUri().toString();
+ if (!candidateFileStatus.containsKey(baseFilePath)) {
+ throw new HoodieException("Error obtaining fileStatus for file: " + baseFilePath);
+ }
+ RealtimeFileStatus fileStatus = new RealtimeFileStatus(candidateFileStatus.get(baseFilePath));
+ fileStatus.setMaxCommitTime(maxCommitTime);
+ fileStatus.setBelongToIncrementalFileStatus(true);
+ fileStatus.setBasePath(basePath);
+ fileStatus.setBaseFilePath(baseFilePath);
+ fileStatus.setDeltaLogPaths(f.getLatestFileSlice().get().getLogFiles().map(l -> l.getPath().toString()).collect(Collectors.toList()));
+ // try to set bootstrapfileStatus
+ if (baseFileStatus instanceof LocatedFileStatusWithBootstrapBaseFile || baseFileStatus instanceof FileStatusWithBootstrapBaseFile) {
+ fileStatus.setBootStrapFileStatus(baseFileStatus);
+ }
+ result.add(fileStatus);
+ }
+ // add file group which has only logs.
+ if (f.getLatestFileSlice().isPresent() && baseFiles.isEmpty()) {
+ List<FileStatus> logFileStatus = f.getLatestFileSlice().get().getLogFiles().map(logFile -> logFile.getFileStatus()).collect(Collectors.toList());
+ if (logFileStatus.size() > 0) {
+ RealtimeFileStatus fileStatus = new RealtimeFileStatus(logFileStatus.get(0));
+ fileStatus.setBelongToIncrementalFileStatus(true);
+ fileStatus.setDeltaLogPaths(logFileStatus.stream().map(l -> l.getPath().toString()).collect(Collectors.toList()));
+ fileStatus.setMaxCommitTime(maxCommitTime);
+ fileStatus.setBasePath(basePath);
+ result.add(fileStatus);
+ }
+ }
+ } catch (IOException e) {
+ throw new HoodieException("Error obtaining data file/log file grouping ", e);
+ }
+ });
+ return result;
+ }
+
+ @Override
+ protected boolean includeLogFilesForSnapShotView() {
Review comment:
agree
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieParquetRealtimeInputFormat.java
##########
@@ -66,6 +91,139 @@
return HoodieRealtimeInputFormatUtils.getRealtimeSplits(job, fileSplits);
}
+ /**
+ * keep the logical of mor_incr_view as same as spark datasource.
+ * to do: unify the incremental view code between hive/spark-sql and spark datasource
Review comment:
agree
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r725420475
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieRealtimeInputFormatUtils.java
##########
@@ -99,7 +101,32 @@
}
}
Option<HoodieVirtualKeyInfo> finalHoodieVirtualKeyInfo = hoodieVirtualKeyInfo;
- partitionsToParquetSplits.keySet().forEach(partitionPath -> {
+ // deal with incremental query
+ candidateFileSplits.stream().forEach(s -> {
+ try {
Review comment:
sorry, i cannot get your point。could you explain it in detail， thanks。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r725742520
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieParquetRealtimeInputFormat.java
##########
@@ -66,6 +91,139 @@
return HoodieRealtimeInputFormatUtils.getRealtimeSplits(job, fileSplits);
}
+ /**
+ * keep the logical of mor_incr_view as same as spark datasource.
+ * to do: unify the incremental view code between hive/spark-sql and spark datasource
+ */
+ @Override
+ protected List<FileStatus> listStatusForIncrementalMode(
+ JobConf job, HoodieTableMetaClient tableMetaClient, List<Path> inputPaths) throws IOException {
Review comment:
yes only partitions, this method is override from HoodieParquetInputFormat. we can see the doc of HoodieParquetInputFormat.listStatusForIncrementalMode
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r725746607
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/RealtimeCompactedRecordReader.java
##########
@@ -102,97 +102,107 @@ private HoodieMergedLogRecordScanner getMergedLogRecordScanner() throws IOExcept
@Override
public boolean next(NullWritable aVoid, ArrayWritable arrayWritable) throws IOException {
+ // deal with DeltaOnlySplits
+ if (logReader.isPresent()) {
+ return logReader.get().next(aVoid, arrayWritable);
+ }
// Call the underlying parquetReader.next - which may replace the passed in ArrayWritable
// with a new block of values
- while (this.parquetReader.next(aVoid, arrayWritable)) {
- if (!deltaRecordMap.isEmpty()) {
- String key = arrayWritable.get()[recordKeyIndex].toString();
- if (deltaRecordMap.containsKey(key)) {
- // mark the key as handled
- this.deltaRecordKeys.remove(key);
- // TODO(NA): Invoke preCombine here by converting arrayWritable to Avro. This is required since the
- // deltaRecord may not be a full record and needs values of columns from the parquet
- Option<GenericRecord> rec = buildGenericRecordwithCustomPayload(deltaRecordMap.get(key));
- // If the record is not present, this is a delete record using an empty payload so skip this base record
- // and move to the next record
- if (!rec.isPresent()) {
- continue;
+ boolean result = this.parquetReader.next(aVoid, arrayWritable);
+ if (!result) {
+ // if the result is false, then there are no more records
+ return false;
Review comment:
same comments as https://github.com/apache/hudi/pull/3203#discussion_r725746415
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a change in pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
danny0405 commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r728651340
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieParquetRealtimeInputFormat.java
##########
@@ -66,6 +90,170 @@
return HoodieRealtimeInputFormatUtils.getRealtimeSplits(job, fileSplits);
}
+ /**
+ * Keep the logical of mor_incr_view as same as spark datasource.
+ * Step1: Get list of commits to be fetched based on start commit and max commits(for snapshot max commits is -1).
Review comment:
`logical` is an adjective, please use noun `logic` instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-941865271
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5d49bd4e8c638ffb9ced102dc6771d6291c199e3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603)
* 792ae1d9884eac5eb7004e4edb617ea6e86d79b7 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
}, {
"hash" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325",
"triggerID" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"triggerType" : "PUSH"
}, {
"hash" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2343",
"triggerID" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"triggerType" : "PUSH"
}, {
"hash" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2424",
"triggerID" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c470b4c2c6a4d1d33a5facf729971b4331fe0771 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2343)
* 679a0104259b01a85be2def1b5cd396ae4f0e858 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2424)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r718194530
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieInputFormatUtils.java
##########
@@ -445,10 +457,21 @@ public static HoodieMetadataConfig buildMetadataConfig(Configuration conf) {
HoodieTableFileSystemView fsView = fsViewCache.computeIfAbsent(metaClient, tableMetaClient ->
FileSystemViewManager.createInMemoryFileSystemViewWithTimeline(engineContext, tableMetaClient, buildMetadataConfig(job), timeline));
List<HoodieBaseFile> filteredBaseFiles = new ArrayList<>();
+ Map<HoodieLogFile, List<String>> filteredLogs = new HashMap<>();
for (Path p : entry.getValue()) {
Review comment:
will change it。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r718189715
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieMergedLogReader.java
##########
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop.realtime;
+
+import java.io.IOException;
+import java.text.MessageFormat;
+import java.util.Iterator;
+
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.io.ArrayWritable;
+import org.apache.hadoop.io.NullWritable;
+import org.apache.hadoop.io.Writable;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.RecordReader;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.hadoop.utils.HoodieRealtimeRecordReaderUtils;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+/**
+ * Record Reader implementation to read avro data, to support inc queries.
+ */
+public class HoodieMergedLogReader extends AbstractRealtimeRecordReader
+ implements RecordReader<NullWritable, ArrayWritable> {
+ private static final Logger LOG = LogManager.getLogger(AbstractRealtimeRecordReader.class);
+ private final HoodieMergedLogRecordScanner logRecordScanner;
+ private final Iterator<HoodieRecord<? extends HoodieRecordPayload>> logRecordsKeyIterator;
+ private ArrayWritable valueObj;
+ private boolean supportTimeStamp = false;
+
+ private int end;
+ private int offset;
+
+ public HoodieMergedLogReader(RealtimeSplit split, JobConf job, HoodieMergedLogRecordScanner logRecordScanner) {
+ super(split, job);
+ this.logRecordScanner = logRecordScanner;
+ this.end = logRecordScanner.getRecords().size();
+ this.logRecordsKeyIterator = logRecordScanner.iterator();
+ this.valueObj = new ArrayWritable(Writable.class, new Writable[getHiveSchema().getFields().size()]);
+ }
+
+ private Option buildGenericRecordwithCustomPayload(HoodieRecord record) throws IOException {
+ if (usesCustomPayload) {
+ return record.getData().getInsertValue(getWriterSchema());
+ } else {
+ return record.getData().getInsertValue(getReaderSchema());
+ }
+ }
+
+ @Override
+ public boolean next(NullWritable key, ArrayWritable arrayWritable) throws IOException {
+ if (!logRecordsKeyIterator.hasNext()) {
+ return false;
+ }
+ Option<GenericRecord> rec;
+ HoodieRecord currentRecord = logRecordsKeyIterator.next();
+
+ rec = buildGenericRecordwithCustomPayload(currentRecord);
+ // try to skip delete record
+ while (!rec.isPresent() && logRecordsKeyIterator.hasNext()) {
+ offset++;
+ rec = buildGenericRecordwithCustomPayload(logRecordsKeyIterator.next());
+ }
+ if (!rec.isPresent()) {
+ return false;
+ }
+
+ GenericRecord recordToReturn = rec.get();
+ if (usesCustomPayload) {
+ // If using a custom payload, return only the projection fields. The readerSchema is a schema derived from
+ // the writerSchema with only the projection fields
+ recordToReturn = HoodieAvroUtils.rewriteRecord(rec.get(), getReaderSchema());
+ }
+ ArrayWritable curWritable = (ArrayWritable) HoodieRealtimeRecordReaderUtils.avroToArrayWritable(recordToReturn, getHiveSchema());
+
+ if (arrayWritable != curWritable) {
+ final Writable[] arrValue = arrayWritable.get();
+ final Writable[] arrCurrent = curWritable.get();
+ if (arrayWritable != null && arrValue.length == arrCurrent.length) {
+ System.arraycopy(arrCurrent, 0, arrValue, 0, arrCurrent.length);
Review comment:
yes， i will remove it。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r718194813
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieRealtimeInputFormatUtils.java
##########
@@ -120,28 +147,32 @@
.orElse(Stream.empty());
// subgroup splits again by file id & match with log files.
- Map<String, List<FileSplit>> groupedInputSplits = partitionsToParquetSplits.get(partitionPath).stream()
- .collect(Collectors.groupingBy(split -> FSUtils.getFileId(split.getPath().getName())));
+ Map<String, List<FileSplit>> groupedInputSplits = filteredPartitionsToParquetSplits.get(partitionPath).stream()
+ .collect(Collectors.groupingBy(split -> {
+ String fileName = split.getPath().getName();
+ // mor snapsht view may contains some raw log files, for example: if user use Hbase index, hudi will produce only log Files
+ if (split instanceof BaseFileWithLogsSplit && ((BaseFileWithLogsSplit) split).getBaseFilePath().isEmpty()) {
+ fileName = split.getPath().getName().substring(1, fileName.length() - 1);
+ }
Review comment:
agree
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/BaseFileWithLogsSplit.java
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop;
+
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.apache.hadoop.mapred.FileSplit;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+public class BaseFileWithLogsSplit extends FileSplit {
Review comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r725492944
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieParquetRealtimeInputFormat.java
##########
@@ -66,6 +91,139 @@
return HoodieRealtimeInputFormatUtils.getRealtimeSplits(job, fileSplits);
}
+ /**
+ * keep the logical of mor_incr_view as same as spark datasource.
+ * to do: unify the incremental view code between hive/spark-sql and spark datasource
+ */
+ @Override
+ protected List<FileStatus> listStatusForIncrementalMode(
+ JobConf job, HoodieTableMetaClient tableMetaClient, List<Path> inputPaths) throws IOException {
+ List<FileStatus> result = new ArrayList<>();
+ String tableName = tableMetaClient.getTableConfig().getTableName();
+ Job jobContext = Job.getInstance(job);
+
+ Option<HoodieTimeline> timeline = HoodieInputFormatUtils.getFilteredCommitsTimeline(jobContext, tableMetaClient);
+ if (!timeline.isPresent()) {
+ return result;
+ }
+ String lastIncrementalTs = HoodieHiveUtils.readStartCommitTime(jobContext, tableName);
+ // Total number of commits to return in this batch. Set this to -1 to get all the commits.
+ Integer maxCommits = HoodieHiveUtils.readMaxCommits(jobContext, tableName);
+ HoodieTimeline commitsTimelineToReturn = timeline.get().findInstantsAfter(lastIncrementalTs, maxCommits);
+ Option<List<HoodieInstant>> commitsToCheck = Option.of(commitsTimelineToReturn.getInstants().collect(Collectors.toList()));
+ if (!commitsToCheck.isPresent()) {
+ return result;
+ }
+ Map<String, HashMap<String, FileStatus>> partitionsWithFileStatus = HoodieInputFormatUtils
+ .listAffectedFilesForCommits(new Path(tableMetaClient.getBasePath()), commitsToCheck.get(), commitsTimelineToReturn);
+ // build fileGroup from fsView
+ List<FileStatus> affectedFileStatus = new ArrayList<>();
+ partitionsWithFileStatus.forEach((key, value) -> value.forEach((k, v) -> affectedFileStatus.add(v)));
+ HoodieTableFileSystemView fsView = new HoodieTableFileSystemView(tableMetaClient, commitsTimelineToReturn, affectedFileStatus.toArray(new FileStatus[0]));
+ // build fileGroup from fsView
+ String basePath = tableMetaClient.getBasePath();
+ // filter affectedPartition by inputPaths
+ List<String> affectedPartition = partitionsWithFileStatus.keySet().stream()
+ .filter(k -> k.isEmpty() ? inputPaths.contains(new Path(basePath)) : inputPaths.contains(new Path(basePath, k))).collect(Collectors.toList());
+ if (affectedPartition.isEmpty()) {
+ return result;
+ }
+ List<HoodieFileGroup> fileGroups = affectedPartition.stream()
+ .flatMap(partitionPath -> fsView.getAllFileGroups(partitionPath)).collect(Collectors.toList());
+ setInputPaths(job, affectedPartition.stream()
+ .map(p -> p.isEmpty() ? basePath : new Path(basePath, p).toUri().toString()).collect(Collectors.joining(",")));
+
+ // find all file status in current partitionPath
+ FileStatus[] fileStatuses = getStatus(job);
+ Map<String, FileStatus> candidateFileStatus = new HashMap<>();
+ for (int i = 0; i < fileStatuses.length; i++) {
+ String key = fileStatuses[i].getPath().toString();
+ candidateFileStatus.put(key, fileStatuses[i]);
+ }
+
+ String maxCommitTime = fsView.getLastInstant().get().getTimestamp();
+ fileGroups.stream().forEach(f -> {
+ try {
+ List<FileSlice> baseFiles = f.getAllFileSlices().filter(slice -> slice.getBaseFile().isPresent()).collect(Collectors.toList());
+ if (!baseFiles.isEmpty()) {
+ FileStatus baseFileStatus = HoodieInputFormatUtils.getFileStatus(baseFiles.get(0).getBaseFile().get());
+ String baseFilePath = baseFileStatus.getPath().toUri().toString();
+ if (!candidateFileStatus.containsKey(baseFilePath)) {
+ throw new HoodieException("Error obtaining fileStatus for file: " + baseFilePath);
+ }
+ RealtimeFileStatus fileStatus = new RealtimeFileStatus(candidateFileStatus.get(baseFilePath));
+ fileStatus.setMaxCommitTime(maxCommitTime);
+ fileStatus.setBelongToIncrementalFileStatus(true);
+ fileStatus.setBasePath(basePath);
+ fileStatus.setBaseFilePath(baseFilePath);
+ fileStatus.setDeltaLogPaths(f.getLatestFileSlice().get().getLogFiles().map(l -> l.getPath().toString()).collect(Collectors.toList()));
+ // try to set bootstrapfileStatus
+ if (baseFileStatus instanceof LocatedFileStatusWithBootstrapBaseFile || baseFileStatus instanceof FileStatusWithBootstrapBaseFile) {
+ fileStatus.setBootStrapFileStatus(baseFileStatus);
+ }
+ result.add(fileStatus);
+ }
+ // add file group which has only logs.
+ if (f.getLatestFileSlice().isPresent() && baseFiles.isEmpty()) {
+ List<FileStatus> logFileStatus = f.getLatestFileSlice().get().getLogFiles().map(logFile -> logFile.getFileStatus()).collect(Collectors.toList());
+ if (logFileStatus.size() > 0) {
+ RealtimeFileStatus fileStatus = new RealtimeFileStatus(logFileStatus.get(0));
+ fileStatus.setBelongToIncrementalFileStatus(true);
+ fileStatus.setDeltaLogPaths(logFileStatus.stream().map(l -> l.getPath().toString()).collect(Collectors.toList()));
+ fileStatus.setMaxCommitTime(maxCommitTime);
+ fileStatus.setBasePath(basePath);
+ result.add(fileStatus);
+ }
+ }
+ } catch (IOException e) {
+ throw new HoodieException("Error obtaining data file/log file grouping ", e);
+ }
+ });
+ return result;
+ }
+
+ @Override
+ protected boolean includeLogFilesForSnapShotView() {
Review comment:
nit: Snapshot
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r725419860
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieInputFormatUtils.java
##########
@@ -445,10 +457,21 @@ public static HoodieMetadataConfig buildMetadataConfig(Configuration conf) {
HoodieTableFileSystemView fsView = fsViewCache.computeIfAbsent(metaClient, tableMetaClient ->
FileSystemViewManager.createInMemoryFileSystemViewWithTimeline(engineContext, tableMetaClient, buildMetadataConfig(job), timeline));
List<HoodieBaseFile> filteredBaseFiles = new ArrayList<>();
+ Map<HoodieLogFile, List<String>> filteredLogs = new HashMap<>();
for (Path p : entry.getValue()) {
Review comment:
we need map here（Map<FileStatus, List<String>> filteredLogs)， since when we build ReatimeFilesStatus we need the fileStatus of current log. of course we can use List<List<String>>, but there is no need.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-922421084
@danny0405 Thank you for your attention to this pr。 This pr has not been updated for a long time. Do we still need this now？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r725742066
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieParquetRealtimeInputFormat.java
##########
@@ -66,6 +91,139 @@
return HoodieRealtimeInputFormatUtils.getRealtimeSplits(job, fileSplits);
}
+ /**
+ * keep the logical of mor_incr_view as same as spark datasource.
+ * to do: unify the incremental view code between hive/spark-sql and spark datasource
+ */
+ @Override
+ protected List<FileStatus> listStatusForIncrementalMode(
+ JobConf job, HoodieTableMetaClient tableMetaClient, List<Path> inputPaths) throws IOException {
+ List<FileStatus> result = new ArrayList<>();
+ String tableName = tableMetaClient.getTableConfig().getTableName();
+ Job jobContext = Job.getInstance(job);
+
+ Option<HoodieTimeline> timeline = HoodieInputFormatUtils.getFilteredCommitsTimeline(jobContext, tableMetaClient);
+ if (!timeline.isPresent()) {
+ return result;
+ }
+ String lastIncrementalTs = HoodieHiveUtils.readStartCommitTime(jobContext, tableName);
+ // Total number of commits to return in this batch. Set this to -1 to get all the commits.
+ Integer maxCommits = HoodieHiveUtils.readMaxCommits(jobContext, tableName);
+ HoodieTimeline commitsTimelineToReturn = timeline.get().findInstantsAfter(lastIncrementalTs, maxCommits);
+ Option<List<HoodieInstant>> commitsToCheck = Option.of(commitsTimelineToReturn.getInstants().collect(Collectors.toList()));
+ if (!commitsToCheck.isPresent()) {
+ return result;
+ }
+ Map<String, HashMap<String, FileStatus>> partitionsWithFileStatus = HoodieInputFormatUtils
+ .listAffectedFilesForCommits(new Path(tableMetaClient.getBasePath()), commitsToCheck.get(), commitsTimelineToReturn);
+ // build fileGroup from fsView
+ List<FileStatus> affectedFileStatus = new ArrayList<>();
+ partitionsWithFileStatus.forEach((key, value) -> value.forEach((k, v) -> affectedFileStatus.add(v)));
+ HoodieTableFileSystemView fsView = new HoodieTableFileSystemView(tableMetaClient, commitsTimelineToReturn, affectedFileStatus.toArray(new FileStatus[0]));
+ // build fileGroup from fsView
+ String basePath = tableMetaClient.getBasePath();
+ // filter affectedPartition by inputPaths
+ List<String> affectedPartition = partitionsWithFileStatus.keySet().stream()
+ .filter(k -> k.isEmpty() ? inputPaths.contains(new Path(basePath)) : inputPaths.contains(new Path(basePath, k))).collect(Collectors.toList());
+ if (affectedPartition.isEmpty()) {
+ return result;
+ }
+ List<HoodieFileGroup> fileGroups = affectedPartition.stream()
+ .flatMap(partitionPath -> fsView.getAllFileGroups(partitionPath)).collect(Collectors.toList());
+ setInputPaths(job, affectedPartition.stream()
+ .map(p -> p.isEmpty() ? basePath : new Path(basePath, p).toUri().toString()).collect(Collectors.joining(",")));
+
+ // find all file status in current partitionPath
Review comment:
no, all partition paths will be include, see line 133 all partitionPath will be added to inputPaths, The comment caused a misunderstanding， i will fixed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r725737689
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieRealtimeInputFormatUtils.java
##########
@@ -161,6 +185,21 @@
return rtSplits.toArray(new InputSplit[0]);
}
+ public static RealtimeBootstrapBaseFileSplit createRealimeBootstrapBaseFileSplit(
+ BootstrapBaseFileSplit split, String basePath, List<String> deltaLogPaths, String maxInstantTime) {
+ try {
+ String[] hosts = split.getLocationInfo() != null ? Arrays.stream(split.getLocationInfo())
+ .filter(x -> !x.isInMemory()).toArray(String[]::new) : new String[0];
+ String[] inMemoryHosts = split.getLocationInfo() != null ? Arrays.stream(split.getLocationInfo())
+ .filter(SplitLocationInfo::isInMemory).toArray(String[]::new) : new String[0];
+ FileSplit baseSplit = new FileSplit(split.getPath(), split.getStart(), split.getLength(),
Review comment:
maybe no need。
by the way。those codes come from line 137 — line 142 in original HoodieRealtimeInputFormatUtils.java。 i just extract those code to make a function。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r725751050
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/BaseFileWithLogsSplit.java
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop;
+
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.apache.hadoop.mapred.FileSplit;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+public class BaseFileWithLogsSplit extends FileSplit {
Review comment:
@nsivabalan sorry for that, i will fixed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r726731332
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/BaseFileWithLogsSplit.java
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop;
+
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.apache.hadoop.mapred.FileSplit;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * we need to encode additional information in split to track matching log file and base files.
+ * Hence, this weird looking class which tracks an log/base file split
+ */
+public class BaseFileWithLogsSplit extends FileSplit {
+ // a flag to mark this split is produced by incremental query or not.
+ private boolean belongToIncrementalSplit = false;
Review comment:
agree, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r726732125
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/BaseFileWithLogsSplit.java
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop;
+
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.apache.hadoop.mapred.FileSplit;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * we need to encode additional information in split to track matching log file and base files.
Review comment:
This is a bridge class，all BaseFileWithLogSplit will convert to HoodieRealtimeFileSplit finally。 i think we should not add too many things in HoodieRealtimeFileSplit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r725781031
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieParquetRealtimeInputFormat.java
##########
@@ -66,6 +91,139 @@
return HoodieRealtimeInputFormatUtils.getRealtimeSplits(job, fileSplits);
}
+ /**
+ * keep the logical of mor_incr_view as same as spark datasource.
+ * to do: unify the incremental view code between hive/spark-sql and spark datasource
+ */
+ @Override
+ protected List<FileStatus> listStatusForIncrementalMode(
+ JobConf job, HoodieTableMetaClient tableMetaClient, List<Path> inputPaths) throws IOException {
+ List<FileStatus> result = new ArrayList<>();
+ String tableName = tableMetaClient.getTableConfig().getTableName();
+ Job jobContext = Job.getInstance(job);
+
+ Option<HoodieTimeline> timeline = HoodieInputFormatUtils.getFilteredCommitsTimeline(jobContext, tableMetaClient);
+ if (!timeline.isPresent()) {
+ return result;
+ }
+ String lastIncrementalTs = HoodieHiveUtils.readStartCommitTime(jobContext, tableName);
+ // Total number of commits to return in this batch. Set this to -1 to get all the commits.
+ Integer maxCommits = HoodieHiveUtils.readMaxCommits(jobContext, tableName);
+ HoodieTimeline commitsTimelineToReturn = timeline.get().findInstantsAfter(lastIncrementalTs, maxCommits);
+ Option<List<HoodieInstant>> commitsToCheck = Option.of(commitsTimelineToReturn.getInstants().collect(Collectors.toList()));
+ if (!commitsToCheck.isPresent()) {
+ return result;
+ }
+ Map<String, HashMap<String, FileStatus>> partitionsWithFileStatus = HoodieInputFormatUtils
+ .listAffectedFilesForCommits(new Path(tableMetaClient.getBasePath()), commitsToCheck.get(), commitsTimelineToReturn);
+ // build fileGroup from fsView
+ List<FileStatus> affectedFileStatus = new ArrayList<>();
+ partitionsWithFileStatus.forEach((key, value) -> value.forEach((k, v) -> affectedFileStatus.add(v)));
+ HoodieTableFileSystemView fsView = new HoodieTableFileSystemView(tableMetaClient, commitsTimelineToReturn, affectedFileStatus.toArray(new FileStatus[0]));
+ // build fileGroup from fsView
+ String basePath = tableMetaClient.getBasePath();
+ // filter affectedPartition by inputPaths
+ List<String> affectedPartition = partitionsWithFileStatus.keySet().stream()
+ .filter(k -> k.isEmpty() ? inputPaths.contains(new Path(basePath)) : inputPaths.contains(new Path(basePath, k))).collect(Collectors.toList());
+ if (affectedPartition.isEmpty()) {
+ return result;
+ }
+ List<HoodieFileGroup> fileGroups = affectedPartition.stream()
+ .flatMap(partitionPath -> fsView.getAllFileGroups(partitionPath)).collect(Collectors.toList());
+ setInputPaths(job, affectedPartition.stream()
+ .map(p -> p.isEmpty() ? basePath : new Path(basePath, p).toUri().toString()).collect(Collectors.joining(",")));
+
+ // find all file status in current partitionPath
+ FileStatus[] fileStatuses = getStatus(job);
+ Map<String, FileStatus> candidateFileStatus = new HashMap<>();
+ for (int i = 0; i < fileStatuses.length; i++) {
+ String key = fileStatuses[i].getPath().toString();
+ candidateFileStatus.put(key, fileStatuses[i]);
+ }
+
+ String maxCommitTime = fsView.getLastInstant().get().getTimestamp();
+ fileGroups.stream().forEach(f -> {
+ try {
+ List<FileSlice> baseFiles = f.getAllFileSlices().filter(slice -> slice.getBaseFile().isPresent()).collect(Collectors.toList());
+ if (!baseFiles.isEmpty()) {
+ FileStatus baseFileStatus = HoodieInputFormatUtils.getFileStatus(baseFiles.get(0).getBaseFile().get());
+ String baseFilePath = baseFileStatus.getPath().toUri().toString();
+ if (!candidateFileStatus.containsKey(baseFilePath)) {
+ throw new HoodieException("Error obtaining fileStatus for file: " + baseFilePath);
+ }
+ RealtimeFileStatus fileStatus = new RealtimeFileStatus(candidateFileStatus.get(baseFilePath));
+ fileStatus.setMaxCommitTime(maxCommitTime);
+ fileStatus.setBelongToIncrementalFileStatus(true);
+ fileStatus.setBasePath(basePath);
+ fileStatus.setBaseFilePath(baseFilePath);
+ fileStatus.setDeltaLogPaths(f.getLatestFileSlice().get().getLogFiles().map(l -> l.getPath().toString()).collect(Collectors.toList()));
Review comment:
// Ensure get the base file when there is a pending compaction, which means the base file
// won't be in the latest file slice.
see the doc of MergeOnReadIncrementalRelation https://github.com/apache/hudi/blob/ceace1c653a3ce3c97e6ee5a244d71ff1806be4f/hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/hudi/MergeOnReadIncrementalRelation.scala#L203
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r725746415
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/RealtimeCompactedRecordReader.java
##########
@@ -102,97 +102,107 @@ private HoodieMergedLogRecordScanner getMergedLogRecordScanner() throws IOExcept
@Override
public boolean next(NullWritable aVoid, ArrayWritable arrayWritable) throws IOException {
+ // deal with DeltaOnlySplits
+ if (logReader.isPresent()) {
+ return logReader.get().next(aVoid, arrayWritable);
+ }
// Call the underlying parquetReader.next - which may replace the passed in ArrayWritable
// with a new block of values
- while (this.parquetReader.next(aVoid, arrayWritable)) {
- if (!deltaRecordMap.isEmpty()) {
- String key = arrayWritable.get()[recordKeyIndex].toString();
- if (deltaRecordMap.containsKey(key)) {
- // mark the key as handled
- this.deltaRecordKeys.remove(key);
- // TODO(NA): Invoke preCombine here by converting arrayWritable to Avro. This is required since the
- // deltaRecord may not be a full record and needs values of columns from the parquet
- Option<GenericRecord> rec = buildGenericRecordwithCustomPayload(deltaRecordMap.get(key));
- // If the record is not present, this is a delete record using an empty payload so skip this base record
- // and move to the next record
- if (!rec.isPresent()) {
- continue;
+ boolean result = this.parquetReader.next(aVoid, arrayWritable);
+ if (!result) {
+ // if the result is false, then there are no more records
+ return false;
+ }
+ if (!deltaRecordMap.isEmpty()) {
+ // TODO(VC): Right now, we assume all records in log, have a matching base record. (which
+ // would be true until we have a way to index logs too)
Review comment:
I need to clarify two points on this issue。
first： those code is produced by revert [HUDI-1969] Support reading logs for MOR Hive rt table (#3033) and those code are original hudi kernal code。
second： At present, we cannot distinguish between the newly added data and the updated data in a log file， so hbase index will not be the case。 HUDI-1969 try to solve this problem but introduce a new bug。 see https://github.com/apache/hudi/pull/3203#issuecomment-927820027
@nsivabalan what's your suggest about hbase index， maybe we cannot handle this case util we have a way to index logs。 for hbase index， we can only deal with log only hudi table， if the hudi table contains both parquet files（user do cluster） and logfiles we cannot handle。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a change in pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
danny0405 commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r728695624
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieRealtimeInputFormatUtils.java
##########
@@ -69,17 +70,18 @@
private static final Logger LOG = LogManager.getLogger(HoodieRealtimeInputFormatUtils.class);
public static InputSplit[] getRealtimeSplits(Configuration conf, Stream<FileSplit> fileSplits) throws IOException {
- Map<Path, List<FileSplit>> partitionsToParquetSplits =
- fileSplits.collect(Collectors.groupingBy(split -> split.getPath().getParent()));
+ // for all unique split parents, obtain all delta files based on delta commit timeline,
+ // grouped on file id
+ List<InputSplit> rtSplits = new ArrayList<>();
+ List<FileSplit> candidateFileSplits = fileSplits.collect(Collectors.toList());
+ Map<Path, List<FileSplit>> partitionsToParquetSplits = candidateFileSplits
Review comment:
`candidateFileSplits` => `fileSplitList`
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieParquetRealtimeInputFormat.java
##########
@@ -119,6 +307,11 @@ void addProjectionToJobConf(final RealtimeSplit realtimeSplit, final JobConf job
addProjectionToJobConf(realtimeSplit, jobConf);
LOG.info("Creating record reader with readCols :" + jobConf.get(ColumnProjectionUtils.READ_COLUMN_NAMES_CONF_STR)
+ ", Ids :" + jobConf.get(ColumnProjectionUtils.READ_COLUMN_IDS_CONF_STR));
+
+ // for log only split, we no need parquet reader, set it to empty
+ if (FSUtils.isLogFile(realtimeSplit.getPath())) {
Review comment:
`we no need parquet reader, set it to empty` => `set the parquet reader as empty`
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieEmptyRecordReader.java
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop.realtime;
+
+import org.apache.hadoop.io.ArrayWritable;
+import org.apache.hadoop.io.NullWritable;
+import org.apache.hadoop.io.Writable;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.RecordReader;
+
+import java.io.IOException;
+
+/**
+ * dummy record for log only realtime split.
+ */
Review comment:
`dummy record` => `Dummy record reader`
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieRealtimeInputFormatUtils.java
##########
@@ -99,7 +101,32 @@
}
}
Option<HoodieVirtualKeyInfo> finalHoodieVirtualKeyInfo = hoodieVirtualKeyInfo;
- partitionsToParquetSplits.keySet().forEach(partitionPath -> {
+ // deal with incremental query
+ candidateFileSplits.stream().forEach(s -> {
+ try {
Review comment:
Something like this:
```java
private static Map<Path, List<FileSplit>> filterOutIncrementalSplits(
List<FileSplit> fileSplitList,
List<InputSplit> rtSplits,
final Option<HoodieVirtualKeyInfo> finalHoodieVirtualKeyInfo) {
return fileSplitList.stream().filter(s -> {
// deal with incremental query.
try {
if (s instanceof BaseFileWithLogsSplit) {
BaseFileWithLogsSplit bs = (BaseFileWithLogsSplit)s;
if (bs.getBelongToIncrementalSplit()) {
rtSplits.add(new HoodieRealtimeFileSplit(bs, bs.getBasePath(), bs.getDeltaLogPaths(), bs.getMaxCommitTime(), finalHoodieVirtualKeyInfo));
return false;
}
} else if (s instanceof RealtimeBootstrapBaseFileSplit) {
rtSplits.add(s);
return false;
}
} catch (IOException e) {
throw new HoodieIOException("Error creating hoodie real time split ", e);
}
return true;
}).collect(Collectors.groupingBy(split -> split.getPath().getParent()));
}
```
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieRealtimeInputFormatUtils.java
##########
@@ -99,7 +101,32 @@
}
}
Option<HoodieVirtualKeyInfo> finalHoodieVirtualKeyInfo = hoodieVirtualKeyInfo;
- partitionsToParquetSplits.keySet().forEach(partitionPath -> {
+ // deal with incremental query
+ candidateFileSplits.stream().forEach(s -> {
+ try {
Review comment:
Can we abstract this filtering logic out as a separate tool method ?
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieRealtimeInputFormatUtils.java
##########
@@ -99,7 +101,32 @@
}
}
Option<HoodieVirtualKeyInfo> finalHoodieVirtualKeyInfo = hoodieVirtualKeyInfo;
- partitionsToParquetSplits.keySet().forEach(partitionPath -> {
+ // deal with incremental query
+ candidateFileSplits.stream().forEach(s -> {
+ try {
Review comment:
And the only usage of `partitionsToParquetSplits` it so collect the partition paths, we can simplify it as a partition path set.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
}, {
"hash" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325",
"triggerID" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"triggerType" : "PUSH"
}, {
"hash" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2343",
"triggerID" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"triggerType" : "PUSH"
}, {
"hash" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2424",
"triggerID" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2427",
"triggerID" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2429",
"triggerID" : "927749497",
"triggerType" : "MANUAL"
}, {
"hash" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2431",
"triggerID" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"triggerType" : "PUSH"
}, {
"hash" : "29e6acb65044597e35f9d499385dad940580d9df",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2432",
"triggerID" : "29e6acb65044597e35f9d499385dad940580d9df",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 40b81794cbd4d6e7b7356b42ece3290b0904e987 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2431)
* 29e6acb65044597e35f9d499385dad940580d9df Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2432)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r725419137
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieMergedLogReader.java
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop.realtime;
+
+import java.io.IOException;
+import java.text.MessageFormat;
+import java.util.Iterator;
+
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.io.ArrayWritable;
+import org.apache.hadoop.io.NullWritable;
+import org.apache.hadoop.io.Writable;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.RecordReader;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.hadoop.utils.HoodieRealtimeRecordReaderUtils;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+/**
+ * Record Reader implementation to read avro data, to support inc queries.
+ */
+public class HoodieMergedLogReader extends AbstractRealtimeRecordReader
+ implements RecordReader<NullWritable, ArrayWritable> {
+ private static final Logger LOG = LogManager.getLogger(AbstractRealtimeRecordReader.class);
+ private final HoodieMergedLogRecordScanner logRecordScanner;
+ private final Iterator<HoodieRecord<? extends HoodieRecordPayload>> logRecordsKeyIterator;
+ private ArrayWritable valueObj;
+
+ private int end;
+ private int offset;
+
+ public HoodieMergedLogReader(RealtimeSplit split, JobConf job, HoodieMergedLogRecordScanner logRecordScanner) {
+ super(split, job);
+ this.logRecordScanner = logRecordScanner;
+ this.end = logRecordScanner.getRecords().size();
+ this.logRecordsKeyIterator = logRecordScanner.iterator();
+ this.valueObj = new ArrayWritable(Writable.class, new Writable[getHiveSchema().getFields().size()]);
+ }
+
+ private Option buildGenericRecord(HoodieRecord record) throws IOException {
+ if (usesCustomPayload) {
+ return record.getData().getInsertValue(getWriterSchema());
+ } else {
+ return record.getData().getInsertValue(getReaderSchema());
+ }
+ }
+
+ @Override
+ public boolean next(NullWritable key, ArrayWritable arrayWritable) throws IOException {
+ if (!logRecordsKeyIterator.hasNext()) {
+ return false;
+ }
+ Option<GenericRecord> rec;
+ HoodieRecord currentRecord = logRecordsKeyIterator.next();
+
+ rec = buildGenericRecord(currentRecord);
+ // try to skip delete record
+ while (!rec.isPresent() && logRecordsKeyIterator.hasNext()) {
+ offset++;
+ rec = buildGenericRecord(logRecordsKeyIterator.next());
+ }
+ if (!rec.isPresent()) {
+ return false;
+ }
+
+ GenericRecord recordToReturn = rec.get();
+ if (usesCustomPayload) {
+ // If using a custom payload, return only the projection fields. The readerSchema is a schema derived from
+ // the writerSchema with only the projection fields
+ recordToReturn = HoodieAvroUtils.rewriteRecord(rec.get(), getReaderSchema());
+ }
+ ArrayWritable curWritable = (ArrayWritable) HoodieRealtimeRecordReaderUtils.avroToArrayWritable(recordToReturn, getHiveSchema());
+
+ if (arrayWritable != curWritable) {
+ final Writable[] arrValue = arrayWritable.get();
Review comment:
agree， will remove redundant judgment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a change in pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
danny0405 commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r718116742
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieMergedLogReader.java
##########
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop.realtime;
+
+import java.io.IOException;
+import java.text.MessageFormat;
+import java.util.Iterator;
+
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.io.ArrayWritable;
+import org.apache.hadoop.io.NullWritable;
+import org.apache.hadoop.io.Writable;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.RecordReader;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.hadoop.utils.HoodieRealtimeRecordReaderUtils;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+/**
+ * Record Reader implementation to read avro data, to support inc queries.
+ */
+public class HoodieMergedLogReader extends AbstractRealtimeRecordReader
+ implements RecordReader<NullWritable, ArrayWritable> {
+ private static final Logger LOG = LogManager.getLogger(AbstractRealtimeRecordReader.class);
+ private final HoodieMergedLogRecordScanner logRecordScanner;
+ private final Iterator<HoodieRecord<? extends HoodieRecordPayload>> logRecordsKeyIterator;
+ private ArrayWritable valueObj;
+ private boolean supportTimeStamp = false;
+
+ private int end;
+ private int offset;
+
+ public HoodieMergedLogReader(RealtimeSplit split, JobConf job, HoodieMergedLogRecordScanner logRecordScanner) {
+ super(split, job);
+ this.logRecordScanner = logRecordScanner;
+ this.end = logRecordScanner.getRecords().size();
+ this.logRecordsKeyIterator = logRecordScanner.iterator();
+ this.valueObj = new ArrayWritable(Writable.class, new Writable[getHiveSchema().getFields().size()]);
+ }
+
+ private Option buildGenericRecordwithCustomPayload(HoodieRecord record) throws IOException {
+ if (usesCustomPayload) {
+ return record.getData().getInsertValue(getWriterSchema());
+ } else {
+ return record.getData().getInsertValue(getReaderSchema());
+ }
+ }
+
+ @Override
+ public boolean next(NullWritable key, ArrayWritable arrayWritable) throws IOException {
+ if (!logRecordsKeyIterator.hasNext()) {
+ return false;
+ }
+ Option<GenericRecord> rec;
+ HoodieRecord currentRecord = logRecordsKeyIterator.next();
+
+ rec = buildGenericRecordwithCustomPayload(currentRecord);
+ // try to skip delete record
+ while (!rec.isPresent() && logRecordsKeyIterator.hasNext()) {
+ offset++;
+ rec = buildGenericRecordwithCustomPayload(logRecordsKeyIterator.next());
+ }
+ if (!rec.isPresent()) {
+ return false;
+ }
+
+ GenericRecord recordToReturn = rec.get();
+ if (usesCustomPayload) {
+ // If using a custom payload, return only the projection fields. The readerSchema is a schema derived from
+ // the writerSchema with only the projection fields
+ recordToReturn = HoodieAvroUtils.rewriteRecord(rec.get(), getReaderSchema());
+ }
+ ArrayWritable curWritable = (ArrayWritable) HoodieRealtimeRecordReaderUtils.avroToArrayWritable(recordToReturn, getHiveSchema());
+
+ if (arrayWritable != curWritable) {
+ final Writable[] arrValue = arrayWritable.get();
+ final Writable[] arrCurrent = curWritable.get();
+ if (arrayWritable != null && arrValue.length == arrCurrent.length) {
+ System.arraycopy(arrCurrent, 0, arrValue, 0, arrCurrent.length);
Review comment:
`arrayWritable != null` should never reach because `arrayWritable.get()` is invoked.
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieMergedLogReader.java
##########
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop.realtime;
+
+import java.io.IOException;
+import java.text.MessageFormat;
+import java.util.Iterator;
+
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.io.ArrayWritable;
+import org.apache.hadoop.io.NullWritable;
+import org.apache.hadoop.io.Writable;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.RecordReader;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.hadoop.utils.HoodieRealtimeRecordReaderUtils;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+/**
+ * Record Reader implementation to read avro data, to support inc queries.
+ */
+public class HoodieMergedLogReader extends AbstractRealtimeRecordReader
+ implements RecordReader<NullWritable, ArrayWritable> {
+ private static final Logger LOG = LogManager.getLogger(AbstractRealtimeRecordReader.class);
+ private final HoodieMergedLogRecordScanner logRecordScanner;
+ private final Iterator<HoodieRecord<? extends HoodieRecordPayload>> logRecordsKeyIterator;
+ private ArrayWritable valueObj;
+ private boolean supportTimeStamp = false;
+
+ private int end;
+ private int offset;
+
+ public HoodieMergedLogReader(RealtimeSplit split, JobConf job, HoodieMergedLogRecordScanner logRecordScanner) {
+ super(split, job);
+ this.logRecordScanner = logRecordScanner;
+ this.end = logRecordScanner.getRecords().size();
+ this.logRecordsKeyIterator = logRecordScanner.iterator();
+ this.valueObj = new ArrayWritable(Writable.class, new Writable[getHiveSchema().getFields().size()]);
+ }
+
+ private Option buildGenericRecordwithCustomPayload(HoodieRecord record) throws IOException {
+ if (usesCustomPayload) {
Review comment:
buildGenericRecord
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieMergedLogReader.java
##########
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop.realtime;
+
+import java.io.IOException;
+import java.text.MessageFormat;
+import java.util.Iterator;
+
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.io.ArrayWritable;
+import org.apache.hadoop.io.NullWritable;
+import org.apache.hadoop.io.Writable;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.RecordReader;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.hadoop.utils.HoodieRealtimeRecordReaderUtils;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+/**
+ * Record Reader implementation to read avro data, to support inc queries.
+ */
+public class HoodieMergedLogReader extends AbstractRealtimeRecordReader
+ implements RecordReader<NullWritable, ArrayWritable> {
+ private static final Logger LOG = LogManager.getLogger(AbstractRealtimeRecordReader.class);
+ private final HoodieMergedLogRecordScanner logRecordScanner;
+ private final Iterator<HoodieRecord<? extends HoodieRecordPayload>> logRecordsKeyIterator;
+ private ArrayWritable valueObj;
+ private boolean supportTimeStamp = false;
+
+ private int end;
+ private int offset;
+
+ public HoodieMergedLogReader(RealtimeSplit split, JobConf job, HoodieMergedLogRecordScanner logRecordScanner) {
+ super(split, job);
+ this.logRecordScanner = logRecordScanner;
+ this.end = logRecordScanner.getRecords().size();
+ this.logRecordsKeyIterator = logRecordScanner.iterator();
+ this.valueObj = new ArrayWritable(Writable.class, new Writable[getHiveSchema().getFields().size()]);
+ }
+
+ private Option buildGenericRecordwithCustomPayload(HoodieRecord record) throws IOException {
+ if (usesCustomPayload) {
+ return record.getData().getInsertValue(getWriterSchema());
+ } else {
+ return record.getData().getInsertValue(getReaderSchema());
+ }
+ }
+
+ @Override
+ public boolean next(NullWritable key, ArrayWritable arrayWritable) throws IOException {
+ if (!logRecordsKeyIterator.hasNext()) {
+ return false;
+ }
+ Option<GenericRecord> rec;
+ HoodieRecord currentRecord = logRecordsKeyIterator.next();
+
+ rec = buildGenericRecordwithCustomPayload(currentRecord);
+ // try to skip delete record
+ while (!rec.isPresent() && logRecordsKeyIterator.hasNext()) {
+ offset++;
+ rec = buildGenericRecordwithCustomPayload(logRecordsKeyIterator.next());
+ }
+ if (!rec.isPresent()) {
+ return false;
+ }
+
+ GenericRecord recordToReturn = rec.get();
+ if (usesCustomPayload) {
+ // If using a custom payload, return only the projection fields. The readerSchema is a schema derived from
+ // the writerSchema with only the projection fields
+ recordToReturn = HoodieAvroUtils.rewriteRecord(rec.get(), getReaderSchema());
+ }
+ ArrayWritable curWritable = (ArrayWritable) HoodieRealtimeRecordReaderUtils.avroToArrayWritable(recordToReturn, getHiveSchema());
+
+ if (arrayWritable != curWritable) {
+ final Writable[] arrValue = arrayWritable.get();
+ final Writable[] arrCurrent = curWritable.get();
+ if (arrayWritable != null && arrValue.length == arrCurrent.length) {
+ System.arraycopy(arrCurrent, 0, arrValue, 0, arrCurrent.length);
+ } else {
+ if (arrValue.length != arrCurrent.length) {
+ throw new IOException(MessageFormat.format("HoodieMergeLogReader : size of object differs. Value size : {0}, Current Object size : {1}",
+ arrValue.length, arrCurrent.length));
+ } else {
+ throw new IOException("HoodieMergeLogReader can not support RecordReaders that don't return same key & value & value is null");
+ }
+ }
+ }
+
+ offset++;
+ return true;
+ }
+
+ @Override
+ public NullWritable createKey() {
+ return null;
+ }
+
+ @Override
+ public ArrayWritable createValue() {
+ return valueObj;
+ }
+
+ @Override
+ public long getPos() throws IOException {
+ return offset;
+ }
+
+ @Override
+ public void close() throws IOException {
+ if (logRecordScanner != null) {
+ logRecordScanner.close();
+ }
+ }
+
+ @Override
+ public float getProgress() throws IOException {
+ if (end == offset) {
+ return 0.0f;
Review comment:
Is this correct ?
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/fs/FSUtils.java
##########
@@ -336,6 +336,11 @@ public static String getFileExtensionFromLog(Path logPath) {
return matcher.group(3);
}
+ public static String getLogFileExtension(String fullName) {
+ Matcher matcher = LOG_FILE_PATTERN.matcher(fullName);
Review comment:
What's the difference with `getFileExtensionFromLog` ?
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieInputFormatUtils.java
##########
@@ -445,10 +457,21 @@ public static HoodieMetadataConfig buildMetadataConfig(Configuration conf) {
HoodieTableFileSystemView fsView = fsViewCache.computeIfAbsent(metaClient, tableMetaClient ->
FileSystemViewManager.createInMemoryFileSystemViewWithTimeline(engineContext, tableMetaClient, buildMetadataConfig(job), timeline));
List<HoodieBaseFile> filteredBaseFiles = new ArrayList<>();
+ Map<HoodieLogFile, List<String>> filteredLogs = new HashMap<>();
for (Path p : entry.getValue()) {
Review comment:
The cache key is weird.
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/BaseFileWithLogsSplit.java
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop;
+
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.apache.hadoop.mapred.FileSplit;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+public class BaseFileWithLogsSplit extends FileSplit {
Review comment:
Add class document.
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieParquetRealtimeInputFormat.java
##########
@@ -119,6 +275,11 @@ void addProjectionToJobConf(final RealtimeSplit realtimeSplit, final JobConf job
addProjectionToJobConf(realtimeSplit, jobConf);
LOG.info("Creating record reader with readCols :" + jobConf.get(ColumnProjectionUtils.READ_COLUMN_NAMES_CONF_STR)
+ ", Ids :" + jobConf.get(ColumnProjectionUtils.READ_COLUMN_IDS_CONF_STR));
+
+ // for log only split, we no need parquet reader, set it to empty
+ if (realtimeSplit.getPath().toString().contains(".log.")) {
+ return new HoodieRealtimeRecordReader(realtimeSplit, jobConf, new HoodieEmptyRecordReader());
+ }
Review comment:
Use `FSUtils.isLogFile` ?
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/PathWithLogFilePath.java
##########
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop;
+
+import org.apache.hadoop.fs.Path;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * we need to encode additional information in Path to track matching log file and base files.
+ * Hence, this weird looking class which tracks an log/base file status
+ */
+public class PathWithLogFilePath extends Path {
+
+ private boolean belongToIncrementalPath = false;
+ private List<String> deltaLogPaths = new ArrayList<>();
Review comment:
Ad comments for each member.
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/RealtimeCompactedRecordReader.java
##########
@@ -55,13 +55,20 @@
private final Set<String> deltaRecordKeys;
private int recordKeyIndex = HoodieInputFormatUtils.HOODIE_RECORD_KEY_COL_POS;
private Iterator<String> deltaItr;
+ private boolean logFileOnlySplit;
+ private HoodieMergedLogReader logReader;
public RealtimeCompactedRecordReader(RealtimeSplit split, JobConf job,
RecordReader<NullWritable, ArrayWritable> realReader) throws IOException {
super(split, job);
this.parquetReader = realReader;
- this.deltaRecordMap = getMergedLogRecordScanner().getRecords();
+ HoodieMergedLogRecordScanner hoodieMergedLogRecordScanner = getMergedLogRecordScanner();
+ this.deltaRecordMap = hoodieMergedLogRecordScanner.getRecords();
this.deltaRecordKeys = new HashSet<>(this.deltaRecordMap.keySet());
+ if (split.getPath().toString().contains(".log.")) {
+ this.logFileOnlySplit = true;
Review comment:
Keep `logReader` as `Option< HoodieMergedLogReader >` then we can eliminate the flag `logFileOnlySplit`.
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieParquetRealtimeInputFormat.java
##########
@@ -66,6 +90,138 @@
return HoodieRealtimeInputFormatUtils.getRealtimeSplits(job, fileSplits);
}
+ /**
+ * keep the logical of mor_incr_view as same as spark datasource.
+ * to do: unify the incremental view code between hive/spark-sql and spark datasource
Review comment:
Can we unify the logic in this PR ?
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/RealtimeCompactedRecordReader.java
##########
@@ -55,13 +55,20 @@
private final Set<String> deltaRecordKeys;
private int recordKeyIndex = HoodieInputFormatUtils.HOODIE_RECORD_KEY_COL_POS;
private Iterator<String> deltaItr;
+ private boolean logFileOnlySplit;
+ private HoodieMergedLogReader logReader;
public RealtimeCompactedRecordReader(RealtimeSplit split, JobConf job,
RecordReader<NullWritable, ArrayWritable> realReader) throws IOException {
super(split, job);
this.parquetReader = realReader;
- this.deltaRecordMap = getMergedLogRecordScanner().getRecords();
+ HoodieMergedLogRecordScanner hoodieMergedLogRecordScanner = getMergedLogRecordScanner();
+ this.deltaRecordMap = hoodieMergedLogRecordScanner.getRecords();
this.deltaRecordKeys = new HashSet<>(this.deltaRecordMap.keySet());
+ if (split.getPath().toString().contains(".log.")) {
+ this.logFileOnlySplit = true;
Review comment:
Use `FSUtils.isLogFile` ?
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieRealtimeInputFormatUtils.java
##########
@@ -120,28 +147,32 @@
.orElse(Stream.empty());
// subgroup splits again by file id & match with log files.
- Map<String, List<FileSplit>> groupedInputSplits = partitionsToParquetSplits.get(partitionPath).stream()
- .collect(Collectors.groupingBy(split -> FSUtils.getFileId(split.getPath().getName())));
+ Map<String, List<FileSplit>> groupedInputSplits = filteredPartitionsToParquetSplits.get(partitionPath).stream()
+ .collect(Collectors.groupingBy(split -> {
+ String fileName = split.getPath().getName();
+ // mor snapsht view may contains some raw log files, for example: if user use Hbase index, hudi will produce only log Files
+ if (split instanceof BaseFileWithLogsSplit && ((BaseFileWithLogsSplit) split).getBaseFilePath().isEmpty()) {
+ fileName = split.getPath().getName().substring(1, fileName.length() - 1);
+ }
Review comment:
`FSUtils.getFileIdFromLogPath` ?
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieParquetRealtimeInputFormat.java
##########
@@ -66,6 +90,138 @@
return HoodieRealtimeInputFormatUtils.getRealtimeSplits(job, fileSplits);
}
+ /**
+ * keep the logical of mor_incr_view as same as spark datasource.
+ * to do: unify the incremental view code between hive/spark-sql and spark datasource
+ */
+ @Override
+ protected List<FileStatus> listStatusForIncrementalMode(
+ JobConf job, HoodieTableMetaClient tableMetaClient, List<Path> inputPaths) throws IOException {
+ List<FileStatus> result = new ArrayList<>();
+ String tableName = tableMetaClient.getTableConfig().getTableName();
+ Job jobContext = Job.getInstance(job);
+
+ Option<HoodieTimeline> timeline = HoodieInputFormatUtils.getFilteredCommitsTimeline(jobContext, tableMetaClient);
+ if (!timeline.isPresent()) {
+ return result;
+ }
+ String lastIncrementalTs = HoodieHiveUtils.readStartCommitTime(jobContext, tableName);
+ // Total number of commits to return in this batch. Set this to -1 to get all the commits.
+ Integer maxCommits = HoodieHiveUtils.readMaxCommits(jobContext, tableName);
+ HoodieTimeline commitsTimelineToReturn = timeline.get().findInstantsAfter(lastIncrementalTs, maxCommits);
+ Option<List<HoodieInstant>> commitsToCheck = Option.of(commitsTimelineToReturn.getInstants().collect(Collectors.toList()));
+ if (!commitsToCheck.isPresent()) {
+ return result;
+ }
+ Map<String, HashMap<String, FileStatus>> partitionsWithFileStatus = HoodieInputFormatUtils
+ .listAffectedFilesForCommits(new Path(tableMetaClient.getBasePath()), commitsToCheck.get(), commitsTimelineToReturn);
+ // build fileGroup from fsView
+ List<FileStatus> affectedFileStatus = new ArrayList<>();
+ partitionsWithFileStatus.forEach((key, value) -> value.forEach((k, v) -> affectedFileStatus.add(v)));
+ HoodieTableFileSystemView fsView = new HoodieTableFileSystemView(tableMetaClient, commitsTimelineToReturn, affectedFileStatus.toArray(new FileStatus[0]));
+ // build fileGroup from fsView
+ String basePath = tableMetaClient.getBasePath();
+ // filter affectedPartition by inputPaths
+ List<String> affectedPartition = partitionsWithFileStatus.keySet().stream()
+ .filter(k -> k.isEmpty() ? inputPaths.contains(new Path(basePath)) : inputPaths.contains(new Path(basePath, k))).collect(Collectors.toList());
+ if (affectedPartition.isEmpty()) {
+ return result;
+ }
+ List<HoodieFileGroup> fileGroups = affectedPartition.stream()
+ .flatMap(partitionPath -> fsView.getAllFileGroups(partitionPath)).collect(Collectors.toList());
+ setInputPaths(job, affectedPartition.stream()
+ .map(p -> p.isEmpty() ? basePath : new Path(basePath, p).toUri().toString()).collect(Collectors.joining(",")));
+
+ // find all file status in current partitionPath
+ FileStatus[] fileStatuses = getStatus(job);
+ Map<String, FileStatus> candidateFileStatus = new HashMap<>();
+ for (int i = 0; i < fileStatuses.length; i++) {
+ String key = fileStatuses[i].getPath().toString();
+ candidateFileStatus.put(key, fileStatuses[i]);
+ }
+
+ String maxCommitTime = fsView.getLastInstant().get().getTimestamp();
+ fileGroups.stream().forEach(f -> {
+ try {
+ List<FileSlice> baseFiles = f.getAllFileSlices().filter(slice -> slice.getBaseFile().isPresent()).collect(Collectors.toList());
+ if (!baseFiles.isEmpty()) {
+ FileStatus baseFileStatus = HoodieInputFormatUtils.getFileStatus(baseFiles.get(0).getBaseFile().get());
+ String baseFilePath = baseFileStatus.getPath().toUri().toString();
+ if (!candidateFileStatus.containsKey(baseFilePath)) {
+ throw new HoodieException("Error obtaining fileStatus for file: " + baseFilePath);
+ }
+ RealtimeFileStatus fileStatus = new RealtimeFileStatus(candidateFileStatus.get(baseFilePath));
+ fileStatus.setMaxCommitTime(maxCommitTime);
+ fileStatus.setBelongToIncrementalFileStatus(true);
+ fileStatus.setBasePath(basePath);
+ fileStatus.setBaseFilePath(baseFilePath);
+ fileStatus.setDeltaLogPaths(f.getLatestFileSlice().get().getLogFiles().map(l -> l.getPath().toString()).collect(Collectors.toList()));
+ // try to set bootstrapfileStatus
+ if (baseFileStatus instanceof LocatedFileStatusWithBootstrapBaseFile || baseFileStatus instanceof FileStatusWithBootstrapBaseFile) {
+ fileStatus.setBootStrapFileStatus(baseFileStatus);
+ }
+ result.add(fileStatus);
+ }
+ // add file group which has only logs.
+ if (f.getLatestFileSlice().isPresent() && baseFiles.isEmpty()) {
+ List<FileStatus> logFileStatus = f.getLatestFileSlice().get().getLogFiles().map(logFile -> logFile.getFileStatus()).collect(Collectors.toList());
+ if (logFileStatus.size() > 0) {
+ RealtimeFileStatus fileStatus = new RealtimeFileStatus(logFileStatus.get(0));
+ fileStatus.setBelongToIncrementalFileStatus(true);
+ fileStatus.setDeltaLogPaths(logFileStatus.stream().map(l -> l.getPath().toString()).collect(Collectors.toList()));
+ fileStatus.setMaxCommitTime(maxCommitTime);
+ fileStatus.setBasePath(basePath);
+ result.add(fileStatus);
+ }
+ }
+ } catch (IOException e) {
+ throw new HoodieException("Error obtaining data file/log file grouping ", e);
+ }
+ });
+ return result;
+ }
+
+ @Override
+ protected boolean includeLogFilesForSnapShotView() {
+ return true;
+ }
+
+ @Override
+ protected boolean isSplitable(FileSystem fs, Path filename) {
+ if (filename instanceof PathWithLogFilePath) {
+ return ((PathWithLogFilePath)filename).splitable();
+ }
+ return super.isSplitable(fs, filename);
+ }
+
+ @Override
+ protected FileSplit makeSplit(Path file, long start, long length, String[] hosts) {
+ if (file instanceof PathWithLogFilePath) {
+ PathWithLogFilePath pf = (PathWithLogFilePath) file;
+ if (pf.getPathWithBootstrapFileStatus() == null) {
+ return pf.buildSplit(file, start, length, hosts);
+ } else {
Review comment:
add a method `doMakeSplit` to abstract the variable `String[] inMemoryHosts`, then we only need one piece of code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r733458463
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieRealtimeInputFormatUtils.java
##########
@@ -161,6 +162,46 @@
return rtSplits.toArray(new InputSplit[0]);
}
+ // pick all incremental files and add them to rtSplits then filter out those files.
+ private static Map<Path, List<FileSplit>> filterOutIncrementalSplits(
+ List<FileSplit> fileSplitList,
Review comment:
@danny0405
I remembered why I did this.
we cannot read incremental datas which before replacecommit.
line 122, return latest fileSlices
line 134 we use those latest fileSlices to filter the queried inputslit.
think that now our table has 3 commits:
commit1 file: (file1)
commit2 file: (file2)
replacecommit file: (file3)
now we want to query the incremental data of commit2 (file2 will be picked as inputSplit);
line122, will give us a wrong fileSlices, (this fileSlices only contains file3)
line 134, when we do filter by using those fileSlices, all the incremental inputSplit will be exclueded. since incremental inputSplit is file2, but fileSlices only contains file3.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
}, {
"hash" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325",
"triggerID" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"triggerType" : "PUSH"
}, {
"hash" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2343",
"triggerID" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"triggerType" : "PUSH"
}, {
"hash" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2424",
"triggerID" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2427",
"triggerID" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2429",
"triggerID" : "927749497",
"triggerType" : "MANUAL"
}, {
"hash" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2431",
"triggerID" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"triggerType" : "PUSH"
}, {
"hash" : "29e6acb65044597e35f9d499385dad940580d9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2432",
"triggerID" : "29e6acb65044597e35f9d499385dad940580d9df",
"triggerType" : "PUSH"
}, {
"hash" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2441",
"triggerID" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"triggerType" : "PUSH"
}, {
"hash" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2443",
"triggerID" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2447",
"triggerID" : "928890183",
"triggerType" : "MANUAL"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2470",
"triggerID" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"triggerType" : "PUSH"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2560",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562",
"triggerID" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"triggerType" : "PUSH"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
}, {
"hash" : "6fa031e1dcf6e21dfb24cf4296cb00d910b0a204",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2589",
"triggerID" : "6fa031e1dcf6e21dfb24cf4296cb00d910b0a204",
"triggerType" : "PUSH"
}, {
"hash" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2593",
"triggerID" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"triggerType" : "PUSH"
}, {
"hash" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2604",
"triggerID" : "941865271",
"triggerType" : "MANUAL"
}, {
"hash" : "6d67b68e19a43f8668e5773d27ca9c33a8de0a37",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2642",
"triggerID" : "6d67b68e19a43f8668e5773d27ca9c33a8de0a37",
"triggerType" : "PUSH"
}, {
"hash" : "1a80559bd98829552acffdaf20d3ea0384d1d936",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2645",
"triggerID" : "1a80559bd98829552acffdaf20d3ea0384d1d936",
"triggerType" : "PUSH"
}, {
"hash" : "c4af04dab3dab31ef05ba6007000738a2dfb81ce",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2649",
"triggerID" : "c4af04dab3dab31ef05ba6007000738a2dfb81ce",
"triggerType" : "PUSH"
}, {
"hash" : "369c5d6f1f4b6eed3c86f838fb6148d127cc6703",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2660",
"triggerID" : "369c5d6f1f4b6eed3c86f838fb6148d127cc6703",
"triggerType" : "PUSH"
}, {
"hash" : "196a40e3273e98f2317856a55de3bd1888e9f754",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "196a40e3273e98f2317856a55de3bd1888e9f754",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0fa6297ce58eb877fd5c4eba59fef20ad9335d26 UNKNOWN
* 369c5d6f1f4b6eed3c86f838fb6148d127cc6703 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2660)
* 196a40e3273e98f2317856a55de3bd1888e9f754 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-952871720
@danny0405 split getIncrementalRealtimeSplits from getRealtimeSplits.
@nsivabalan @danny0405 could you pls review again, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r740695703
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieRealtimeInputFormatUtils.java
##########
@@ -161,6 +148,82 @@
return rtSplits.toArray(new InputSplit[0]);
}
+ // get IncrementalRealtimeSplits
+ public static InputSplit[] getIncrementalRealtimeSplits(Configuration conf, Stream<FileSplit> fileSplits) throws IOException {
+ List<InputSplit> rtSplits = new ArrayList<>();
+ List<FileSplit> fileSplitList = fileSplits.collect(Collectors.toList());
+ Set<Path> partitionSet = fileSplitList.stream().map(f -> f.getPath().getParent()).collect(Collectors.toSet());
+ Map<Path, HoodieTableMetaClient> partitionsToMetaClient = getTableMetaClientByPartitionPath(conf, partitionSet);
+ // Pre process tableConfig from first partition to fetch virtual key info
+ Option<HoodieVirtualKeyInfo> hoodieVirtualKeyInfo = Option.empty();
+ if (partitionSet.size() > 0) {
+ hoodieVirtualKeyInfo = getHoodieVirtualKeyInfo(partitionsToMetaClient.get(partitionSet.iterator().next()));
+ }
+ Option<HoodieVirtualKeyInfo> finalHoodieVirtualKeyInfo = hoodieVirtualKeyInfo;
+ fileSplitList.stream().forEach(s -> {
+ // deal with incremental query.
+ try {
+ if (s instanceof BaseFileWithLogsSplit) {
+ BaseFileWithLogsSplit bs = (BaseFileWithLogsSplit)s;
+ if (bs.getBelongToIncrementalSplit()) {
+ rtSplits.add(new HoodieRealtimeFileSplit(bs, bs.getBasePath(), bs.getDeltaLogPaths(), bs.getMaxCommitTime(), finalHoodieVirtualKeyInfo));
+ }
+ } else if (s instanceof RealtimeBootstrapBaseFileSplit) {
+ rtSplits.add(s);
+ }
+ } catch (IOException e) {
+ throw new HoodieIOException("Error creating hoodie real time split ", e);
+ }
+ });
+ LOG.info("Returning a total splits of " + rtSplits.size());
+ return rtSplits.toArray(new InputSplit[0]);
+ }
+
+ public static Option<HoodieVirtualKeyInfo> getHoodieVirtualKeyInfo(HoodieTableMetaClient metaClient) {
+ HoodieTableConfig tableConfig = metaClient.getTableConfig();
+ if (!tableConfig.populateMetaFields()) {
+ TableSchemaResolver tableSchemaResolver = new TableSchemaResolver(metaClient);
+ try {
+ MessageType parquetSchema = tableSchemaResolver.getTableParquetSchema();
+ return Option.of(new HoodieVirtualKeyInfo(tableConfig.getRecordKeyFieldProp(),
+ tableConfig.getPartitionFieldProp(), parquetSchema.getFieldIndex(tableConfig.getRecordKeyFieldProp()),
+ parquetSchema.getFieldIndex(tableConfig.getPartitionFieldProp())));
+ } catch (Exception exception) {
+ throw new HoodieException("Fetching table schema failed with exception ", exception);
+ }
+ }
+ return Option.empty();
+ }
+
+ public static boolean isIncrementalQuerySplits(List<FileSplit> fileSplits) {
+ if (fileSplits == null || fileSplits.size() == 0) {
+ return false;
+ }
+ return fileSplits.stream().anyMatch(s -> {
+ if (s instanceof BaseFileWithLogsSplit) {
+ BaseFileWithLogsSplit bs = (BaseFileWithLogsSplit)s;
+ return bs.getBelongToIncrementalSplit();
+ } else {
+ return s instanceof RealtimeBootstrapBaseFileSplit;
+ }
+ });
+ }
+
+ public static RealtimeBootstrapBaseFileSplit createRealimeBootstrapBaseFileSplit(
Review comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-957530371
@codope thanks for your review.
Adressed all comments。
Add test case for compaction to verify that incremental reads do NOT show inserts after compaction timestamp， other situations are currently covered by hudi.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
}, {
"hash" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325",
"triggerID" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"triggerType" : "PUSH"
}, {
"hash" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2343",
"triggerID" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"triggerType" : "PUSH"
}, {
"hash" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2424",
"triggerID" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2427",
"triggerID" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2429",
"triggerID" : "927749497",
"triggerType" : "MANUAL"
}, {
"hash" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2431",
"triggerID" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"triggerType" : "PUSH"
}, {
"hash" : "29e6acb65044597e35f9d499385dad940580d9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2432",
"triggerID" : "29e6acb65044597e35f9d499385dad940580d9df",
"triggerType" : "PUSH"
}, {
"hash" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2441",
"triggerID" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"triggerType" : "PUSH"
}, {
"hash" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2443",
"triggerID" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2447",
"triggerID" : "928890183",
"triggerType" : "MANUAL"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2470",
"triggerID" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"triggerType" : "PUSH"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2560",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562",
"triggerID" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"triggerType" : "PUSH"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
}, {
"hash" : "6fa031e1dcf6e21dfb24cf4296cb00d910b0a204",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2589",
"triggerID" : "6fa031e1dcf6e21dfb24cf4296cb00d910b0a204",
"triggerType" : "PUSH"
}, {
"hash" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2593",
"triggerID" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"triggerType" : "PUSH"
}, {
"hash" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2604",
"triggerID" : "941865271",
"triggerType" : "MANUAL"
}, {
"hash" : "6d67b68e19a43f8668e5773d27ca9c33a8de0a37",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2642",
"triggerID" : "6d67b68e19a43f8668e5773d27ca9c33a8de0a37",
"triggerType" : "PUSH"
}, {
"hash" : "1a80559bd98829552acffdaf20d3ea0384d1d936",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2645",
"triggerID" : "1a80559bd98829552acffdaf20d3ea0384d1d936",
"triggerType" : "PUSH"
}, {
"hash" : "c4af04dab3dab31ef05ba6007000738a2dfb81ce",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2649",
"triggerID" : "c4af04dab3dab31ef05ba6007000738a2dfb81ce",
"triggerType" : "PUSH"
}, {
"hash" : "369c5d6f1f4b6eed3c86f838fb6148d127cc6703",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2660",
"triggerID" : "369c5d6f1f4b6eed3c86f838fb6148d127cc6703",
"triggerType" : "PUSH"
}, {
"hash" : "369c5d6f1f4b6eed3c86f838fb6148d127cc6703",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2898",
"triggerID" : "952859446",
"triggerType" : "MANUAL"
}, {
"hash" : "196a40e3273e98f2317856a55de3bd1888e9f754",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2899",
"triggerID" : "196a40e3273e98f2317856a55de3bd1888e9f754",
"triggerType" : "PUSH"
}, {
"hash" : "196a40e3273e98f2317856a55de3bd1888e9f754",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2899",
"triggerID" : "952859446",
"triggerType" : "MANUAL"
}, {
"hash" : "633b7370155185dead726ddfafd773e284ed510c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2935",
"triggerID" : "633b7370155185dead726ddfafd773e284ed510c",
"triggerType" : "PUSH"
}, {
"hash" : "2182f542acd4ae3d1fd239e6441e9761eb2d33a1",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3069",
"triggerID" : "2182f542acd4ae3d1fd239e6441e9761eb2d33a1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0fa6297ce58eb877fd5c4eba59fef20ad9335d26 UNKNOWN
* 2182f542acd4ae3d1fd239e6441e9761eb2d33a1 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3069)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-961560763
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
}, {
"hash" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325",
"triggerID" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"triggerType" : "PUSH"
}, {
"hash" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2343",
"triggerID" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"triggerType" : "PUSH"
}, {
"hash" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2424",
"triggerID" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2427",
"triggerID" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2429",
"triggerID" : "927749497",
"triggerType" : "MANUAL"
}, {
"hash" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2431",
"triggerID" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"triggerType" : "PUSH"
}, {
"hash" : "29e6acb65044597e35f9d499385dad940580d9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2432",
"triggerID" : "29e6acb65044597e35f9d499385dad940580d9df",
"triggerType" : "PUSH"
}, {
"hash" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2441",
"triggerID" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"triggerType" : "PUSH"
}, {
"hash" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2443",
"triggerID" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2447",
"triggerID" : "928890183",
"triggerType" : "MANUAL"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2470",
"triggerID" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"triggerType" : "PUSH"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2560",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562",
"triggerID" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"triggerType" : "PUSH"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
}, {
"hash" : "6fa031e1dcf6e21dfb24cf4296cb00d910b0a204",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2589",
"triggerID" : "6fa031e1dcf6e21dfb24cf4296cb00d910b0a204",
"triggerType" : "PUSH"
}, {
"hash" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2593",
"triggerID" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"triggerType" : "PUSH"
}, {
"hash" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2604",
"triggerID" : "941865271",
"triggerType" : "MANUAL"
}, {
"hash" : "6d67b68e19a43f8668e5773d27ca9c33a8de0a37",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2642",
"triggerID" : "6d67b68e19a43f8668e5773d27ca9c33a8de0a37",
"triggerType" : "PUSH"
}, {
"hash" : "1a80559bd98829552acffdaf20d3ea0384d1d936",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2645",
"triggerID" : "1a80559bd98829552acffdaf20d3ea0384d1d936",
"triggerType" : "PUSH"
}, {
"hash" : "c4af04dab3dab31ef05ba6007000738a2dfb81ce",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2649",
"triggerID" : "c4af04dab3dab31ef05ba6007000738a2dfb81ce",
"triggerType" : "PUSH"
}, {
"hash" : "369c5d6f1f4b6eed3c86f838fb6148d127cc6703",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2660",
"triggerID" : "369c5d6f1f4b6eed3c86f838fb6148d127cc6703",
"triggerType" : "PUSH"
}, {
"hash" : "369c5d6f1f4b6eed3c86f838fb6148d127cc6703",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2898",
"triggerID" : "952859446",
"triggerType" : "MANUAL"
}, {
"hash" : "196a40e3273e98f2317856a55de3bd1888e9f754",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2899",
"triggerID" : "196a40e3273e98f2317856a55de3bd1888e9f754",
"triggerType" : "PUSH"
}, {
"hash" : "196a40e3273e98f2317856a55de3bd1888e9f754",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2899",
"triggerID" : "952859446",
"triggerType" : "MANUAL"
}, {
"hash" : "633b7370155185dead726ddfafd773e284ed510c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2935",
"triggerID" : "633b7370155185dead726ddfafd773e284ed510c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0fa6297ce58eb877fd5c4eba59fef20ad9335d26 UNKNOWN
* 633b7370155185dead726ddfafd773e284ed510c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2935)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r725493388
##########
File path: hudi-hadoop-mr/src/test/java/org/apache/hudi/hadoop/realtime/TestHoodieRealtimeRecordReader.java
##########
@@ -535,4 +549,211 @@ public void testSchemaEvolutionAndRollbackBlockInLastLogFile(ExternalSpillableMa
arguments(ExternalSpillableMap.DiskMapType.ROCKS_DB, true, true)
);
}
+
+ @Test
+ public void testIncremetalWithOnlylog() throws Exception {
+ // initial commit
+ Schema schema = HoodieAvroUtils.addMetadataFields(SchemaTestUtil.getEvolvedSchema());
+ HoodieTestUtils.init(hadoopConf, basePath.toString(), HoodieTableType.MERGE_ON_READ);
+ String instantTime = "100";
+ final int numRecords = 1000;
+ File partitionDir = InputFormatTestUtil.prepareParquetTable(basePath, schema, 1, numRecords, instantTime,
+ HoodieTableType.MERGE_ON_READ);
+ //FileCreateUtils.createDeltaCommit(basePath.toString(), instantTime);
+ createDeltaCommitFile(basePath, instantTime,"2016/05/01", "2016/05/01/fileid0_1-0-1_100.parquet", "fileid0");
Review comment:
here created parquet file, is it testIncrementalWithOnlyLog?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao edited a comment on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-927820027
@danny0405 i revert this pr: [HUDI-1969] Support reading logs for MOR Hive rt table (#3033), since this pr introduce a critical bug which may cause the query result repeat。
give a example: now we have a base file(file1) which contains two record: key1, p1
key2, p2
then we update key2, the log file(logfile1) produced and contains record: key2, p3
when hive/presto query the table: file1 has been split into two file parts, one is file1_part1, which contains all the record; the other is file2_part2 wich contains no record. notice those two file parts will be bound to logfile1 and produce two realtimesplit。 then the log file will be read twice， repeat result produced。
The above is a very simple example， in fact query engine split one parquet file into small parts is very common。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a change in pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
danny0405 commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r724668250
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieMergedLogReader.java
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop.realtime;
+
+import java.io.IOException;
+import java.text.MessageFormat;
+import java.util.Iterator;
+
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.io.ArrayWritable;
+import org.apache.hadoop.io.NullWritable;
+import org.apache.hadoop.io.Writable;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.RecordReader;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.hadoop.utils.HoodieRealtimeRecordReaderUtils;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+/**
+ * Record Reader implementation to read avro data, to support inc queries.
+ */
+public class HoodieMergedLogReader extends AbstractRealtimeRecordReader
+ implements RecordReader<NullWritable, ArrayWritable> {
+ private static final Logger LOG = LogManager.getLogger(AbstractRealtimeRecordReader.class);
+ private final HoodieMergedLogRecordScanner logRecordScanner;
+ private final Iterator<HoodieRecord<? extends HoodieRecordPayload>> logRecordsKeyIterator;
+ private ArrayWritable valueObj;
Review comment:
`logRecordsKeyIterator ` => `iterator `
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieMergedLogReader.java
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop.realtime;
+
+import java.io.IOException;
+import java.text.MessageFormat;
+import java.util.Iterator;
+
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.io.ArrayWritable;
+import org.apache.hadoop.io.NullWritable;
+import org.apache.hadoop.io.Writable;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.RecordReader;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.hadoop.utils.HoodieRealtimeRecordReaderUtils;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+/**
+ * Record Reader implementation to read avro data, to support inc queries.
+ */
+public class HoodieMergedLogReader extends AbstractRealtimeRecordReader
+ implements RecordReader<NullWritable, ArrayWritable> {
+ private static final Logger LOG = LogManager.getLogger(AbstractRealtimeRecordReader.class);
+ private final HoodieMergedLogRecordScanner logRecordScanner;
+ private final Iterator<HoodieRecord<? extends HoodieRecordPayload>> logRecordsKeyIterator;
+ private ArrayWritable valueObj;
+
+ private int end;
+ private int offset;
+
+ public HoodieMergedLogReader(RealtimeSplit split, JobConf job, HoodieMergedLogRecordScanner logRecordScanner) {
+ super(split, job);
+ this.logRecordScanner = logRecordScanner;
+ this.end = logRecordScanner.getRecords().size();
+ this.logRecordsKeyIterator = logRecordScanner.iterator();
+ this.valueObj = new ArrayWritable(Writable.class, new Writable[getHiveSchema().getFields().size()]);
+ }
+
+ private Option buildGenericRecord(HoodieRecord record) throws IOException {
+ if (usesCustomPayload) {
+ return record.getData().getInsertValue(getWriterSchema());
+ } else {
+ return record.getData().getInsertValue(getReaderSchema());
+ }
+ }
+
+ @Override
+ public boolean next(NullWritable key, ArrayWritable arrayWritable) throws IOException {
+ if (!logRecordsKeyIterator.hasNext()) {
Review comment:
`arrayWritable` => `value`
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieMergedLogReader.java
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop.realtime;
+
+import java.io.IOException;
+import java.text.MessageFormat;
+import java.util.Iterator;
+
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.io.ArrayWritable;
+import org.apache.hadoop.io.NullWritable;
+import org.apache.hadoop.io.Writable;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.RecordReader;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.hadoop.utils.HoodieRealtimeRecordReaderUtils;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+/**
+ * Record Reader implementation to read avro data, to support inc queries.
+ */
+public class HoodieMergedLogReader extends AbstractRealtimeRecordReader
+ implements RecordReader<NullWritable, ArrayWritable> {
+ private static final Logger LOG = LogManager.getLogger(AbstractRealtimeRecordReader.class);
+ private final HoodieMergedLogRecordScanner logRecordScanner;
+ private final Iterator<HoodieRecord<? extends HoodieRecordPayload>> logRecordsKeyIterator;
+ private ArrayWritable valueObj;
+
+ private int end;
+ private int offset;
+
+ public HoodieMergedLogReader(RealtimeSplit split, JobConf job, HoodieMergedLogRecordScanner logRecordScanner) {
+ super(split, job);
+ this.logRecordScanner = logRecordScanner;
+ this.end = logRecordScanner.getRecords().size();
+ this.logRecordsKeyIterator = logRecordScanner.iterator();
+ this.valueObj = new ArrayWritable(Writable.class, new Writable[getHiveSchema().getFields().size()]);
+ }
+
+ private Option buildGenericRecord(HoodieRecord record) throws IOException {
+ if (usesCustomPayload) {
+ return record.getData().getInsertValue(getWriterSchema());
+ } else {
+ return record.getData().getInsertValue(getReaderSchema());
+ }
+ }
+
+ @Override
+ public boolean next(NullWritable key, ArrayWritable arrayWritable) throws IOException {
+ if (!logRecordsKeyIterator.hasNext()) {
+ return false;
+ }
+ Option<GenericRecord> rec;
+ HoodieRecord currentRecord = logRecordsKeyIterator.next();
+
+ rec = buildGenericRecord(currentRecord);
+ // try to skip delete record
+ while (!rec.isPresent() && logRecordsKeyIterator.hasNext()) {
+ offset++;
+ rec = buildGenericRecord(logRecordsKeyIterator.next());
+ }
+ if (!rec.isPresent()) {
+ return false;
+ }
+
+ GenericRecord recordToReturn = rec.get();
+ if (usesCustomPayload) {
+ // If using a custom payload, return only the projection fields. The readerSchema is a schema derived from
+ // the writerSchema with only the projection fields
+ recordToReturn = HoodieAvroUtils.rewriteRecord(rec.get(), getReaderSchema());
+ }
+ ArrayWritable curWritable = (ArrayWritable) HoodieRealtimeRecordReaderUtils.avroToArrayWritable(recordToReturn, getHiveSchema());
+
+ if (arrayWritable != curWritable) {
+ final Writable[] arrValue = arrayWritable.get();
Review comment:
I guess `arrayWritable != curWritable` is always true ?
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieRealtimeInputFormatUtils.java
##########
@@ -99,7 +101,32 @@
}
}
Option<HoodieVirtualKeyInfo> finalHoodieVirtualKeyInfo = hoodieVirtualKeyInfo;
- partitionsToParquetSplits.keySet().forEach(partitionPath -> {
+ // deal with incremental query
+ candidateFileSplits.stream().forEach(s -> {
+ try {
Review comment:
One for each loop is enough for mutually exclusive filtering condition.
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieMergedLogReader.java
##########
@@ -0,0 +1,144 @@
+/*
Review comment:
How about we name the clazz as `RealtiMeergedRecordReader` to keep the naming convention as the other two.
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieMergedLogReader.java
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop.realtime;
+
+import java.io.IOException;
+import java.text.MessageFormat;
+import java.util.Iterator;
+
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.io.ArrayWritable;
+import org.apache.hadoop.io.NullWritable;
+import org.apache.hadoop.io.Writable;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.RecordReader;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.hadoop.utils.HoodieRealtimeRecordReaderUtils;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+/**
+ * Record Reader implementation to read avro data, to support inc queries.
+ */
+public class HoodieMergedLogReader extends AbstractRealtimeRecordReader
+ implements RecordReader<NullWritable, ArrayWritable> {
+ private static final Logger LOG = LogManager.getLogger(AbstractRealtimeRecordReader.class);
+ private final HoodieMergedLogRecordScanner logRecordScanner;
+ private final Iterator<HoodieRecord<? extends HoodieRecordPayload>> logRecordsKeyIterator;
+ private ArrayWritable valueObj;
+
+ private int end;
+ private int offset;
+
+ public HoodieMergedLogReader(RealtimeSplit split, JobConf job, HoodieMergedLogRecordScanner logRecordScanner) {
+ super(split, job);
+ this.logRecordScanner = logRecordScanner;
+ this.end = logRecordScanner.getRecords().size();
+ this.logRecordsKeyIterator = logRecordScanner.iterator();
+ this.valueObj = new ArrayWritable(Writable.class, new Writable[getHiveSchema().getFields().size()]);
+ }
+
+ private Option buildGenericRecord(HoodieRecord record) throws IOException {
+ if (usesCustomPayload) {
+ return record.getData().getInsertValue(getWriterSchema());
+ } else {
+ return record.getData().getInsertValue(getReaderSchema());
+ }
+ }
+
+ @Override
+ public boolean next(NullWritable key, ArrayWritable arrayWritable) throws IOException {
+ if (!logRecordsKeyIterator.hasNext()) {
+ return false;
+ }
+ Option<GenericRecord> rec;
+ HoodieRecord currentRecord = logRecordsKeyIterator.next();
+
+ rec = buildGenericRecord(currentRecord);
+ // try to skip delete record
+ while (!rec.isPresent() && logRecordsKeyIterator.hasNext()) {
+ offset++;
+ rec = buildGenericRecord(logRecordsKeyIterator.next());
+ }
+ if (!rec.isPresent()) {
+ return false;
+ }
+
+ GenericRecord recordToReturn = rec.get();
+ if (usesCustomPayload) {
+ // If using a custom payload, return only the projection fields. The readerSchema is a schema derived from
Review comment:
Confused by this logic, why when `usesCustomPayload` is true, the code build the avro record with writer schema first then rewrite with reader schema ?
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieParquetRealtimeInputFormat.java
##########
@@ -66,6 +91,139 @@
return HoodieRealtimeInputFormatUtils.getRealtimeSplits(job, fileSplits);
}
+ /**
+ * keep the logical of mor_incr_view as same as spark datasource.
+ * to do: unify the incremental view code between hive/spark-sql and spark datasource
+ */
+ @Override
+ protected List<FileStatus> listStatusForIncrementalMode(
+ JobConf job, HoodieTableMetaClient tableMetaClient, List<Path> inputPaths) throws IOException {
+ List<FileStatus> result = new ArrayList<>();
+ String tableName = tableMetaClient.getTableConfig().getTableName();
+ Job jobContext = Job.getInstance(job);
+
+ Option<HoodieTimeline> timeline = HoodieInputFormatUtils.getFilteredCommitsTimeline(jobContext, tableMetaClient);
+ if (!timeline.isPresent()) {
+ return result;
+ }
+ String lastIncrementalTs = HoodieHiveUtils.readStartCommitTime(jobContext, tableName);
+ // Total number of commits to return in this batch. Set this to -1 to get all the commits.
+ Integer maxCommits = HoodieHiveUtils.readMaxCommits(jobContext, tableName);
+ HoodieTimeline commitsTimelineToReturn = timeline.get().findInstantsAfter(lastIncrementalTs, maxCommits);
+ Option<List<HoodieInstant>> commitsToCheck = Option.of(commitsTimelineToReturn.getInstants().collect(Collectors.toList()));
+ if (!commitsToCheck.isPresent()) {
+ return result;
+ }
+ Map<String, HashMap<String, FileStatus>> partitionsWithFileStatus = HoodieInputFormatUtils
+ .listAffectedFilesForCommits(new Path(tableMetaClient.getBasePath()), commitsToCheck.get(), commitsTimelineToReturn);
+ // build fileGroup from fsView
+ List<FileStatus> affectedFileStatus = new ArrayList<>();
+ partitionsWithFileStatus.forEach((key, value) -> value.forEach((k, v) -> affectedFileStatus.add(v)));
+ HoodieTableFileSystemView fsView = new HoodieTableFileSystemView(tableMetaClient, commitsTimelineToReturn, affectedFileStatus.toArray(new FileStatus[0]));
+ // build fileGroup from fsView
+ String basePath = tableMetaClient.getBasePath();
+ // filter affectedPartition by inputPaths
+ List<String> affectedPartition = partitionsWithFileStatus.keySet().stream()
+ .filter(k -> k.isEmpty() ? inputPaths.contains(new Path(basePath)) : inputPaths.contains(new Path(basePath, k))).collect(Collectors.toList());
+ if (affectedPartition.isEmpty()) {
+ return result;
+ }
+ List<HoodieFileGroup> fileGroups = affectedPartition.stream()
+ .flatMap(partitionPath -> fsView.getAllFileGroups(partitionPath)).collect(Collectors.toList());
+ setInputPaths(job, affectedPartition.stream()
+ .map(p -> p.isEmpty() ? basePath : new Path(basePath, p).toUri().toString()).collect(Collectors.joining(",")));
+
+ // find all file status in current partitionPath
+ FileStatus[] fileStatuses = getStatus(job);
+ Map<String, FileStatus> candidateFileStatus = new HashMap<>();
+ for (int i = 0; i < fileStatuses.length; i++) {
+ String key = fileStatuses[i].getPath().toString();
+ candidateFileStatus.put(key, fileStatuses[i]);
+ }
+
+ String maxCommitTime = fsView.getLastInstant().get().getTimestamp();
+ fileGroups.stream().forEach(f -> {
+ try {
+ List<FileSlice> baseFiles = f.getAllFileSlices().filter(slice -> slice.getBaseFile().isPresent()).collect(Collectors.toList());
+ if (!baseFiles.isEmpty()) {
+ FileStatus baseFileStatus = HoodieInputFormatUtils.getFileStatus(baseFiles.get(0).getBaseFile().get());
+ String baseFilePath = baseFileStatus.getPath().toUri().toString();
+ if (!candidateFileStatus.containsKey(baseFilePath)) {
+ throw new HoodieException("Error obtaining fileStatus for file: " + baseFilePath);
+ }
+ RealtimeFileStatus fileStatus = new RealtimeFileStatus(candidateFileStatus.get(baseFilePath));
+ fileStatus.setMaxCommitTime(maxCommitTime);
+ fileStatus.setBelongToIncrementalFileStatus(true);
+ fileStatus.setBasePath(basePath);
+ fileStatus.setBaseFilePath(baseFilePath);
+ fileStatus.setDeltaLogPaths(f.getLatestFileSlice().get().getLogFiles().map(l -> l.getPath().toString()).collect(Collectors.toList()));
+ // try to set bootstrapfileStatus
+ if (baseFileStatus instanceof LocatedFileStatusWithBootstrapBaseFile || baseFileStatus instanceof FileStatusWithBootstrapBaseFile) {
+ fileStatus.setBootStrapFileStatus(baseFileStatus);
+ }
+ result.add(fileStatus);
+ }
+ // add file group which has only logs.
+ if (f.getLatestFileSlice().isPresent() && baseFiles.isEmpty()) {
+ List<FileStatus> logFileStatus = f.getLatestFileSlice().get().getLogFiles().map(logFile -> logFile.getFileStatus()).collect(Collectors.toList());
+ if (logFileStatus.size() > 0) {
+ RealtimeFileStatus fileStatus = new RealtimeFileStatus(logFileStatus.get(0));
+ fileStatus.setBelongToIncrementalFileStatus(true);
+ fileStatus.setDeltaLogPaths(logFileStatus.stream().map(l -> l.getPath().toString()).collect(Collectors.toList()));
+ fileStatus.setMaxCommitTime(maxCommitTime);
+ fileStatus.setBasePath(basePath);
+ result.add(fileStatus);
+ }
+ }
+ } catch (IOException e) {
+ throw new HoodieException("Error obtaining data file/log file grouping ", e);
+ }
+ });
+ return result;
+ }
+
+ @Override
+ protected boolean includeLogFilesForSnapShotView() {
+ return true;
+ }
+
+ @Override
+ protected boolean isSplitable(FileSystem fs, Path filename) {
+ if (filename instanceof PathWithLogFilePath) {
+ return ((PathWithLogFilePath)filename).splitable();
+ }
+ return super.isSplitable(fs, filename);
+ }
+
+ @Override
+ protected FileSplit makeSplit(Path file, long start, long length, String[] hosts) {
+ if (file instanceof PathWithLogFilePath) {
+ return doMakeSplitForPathWithLogFilePath((PathWithLogFilePath) file, start, length, hosts, null);
+ }
+ return super.makeSplit(file, start, length, hosts);
+ }
+
+ @Override
+ protected FileSplit makeSplit(Path file, long start, long length, String[] hosts, String[] inMemoryHosts) {
+ if (file instanceof PathWithLogFilePath) {
Review comment:
Just curious: why the file is `PathWithLogFilePath` instead of `PathWithBootstrapFileStatus` for bootstrap table ?
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieInputFormatUtils.java
##########
@@ -445,10 +457,21 @@ public static HoodieMetadataConfig buildMetadataConfig(Configuration conf) {
HoodieTableFileSystemView fsView = fsViewCache.computeIfAbsent(metaClient, tableMetaClient ->
FileSystemViewManager.createInMemoryFileSystemViewWithTimeline(engineContext, tableMetaClient, buildMetadataConfig(job), timeline));
List<HoodieBaseFile> filteredBaseFiles = new ArrayList<>();
+ Map<HoodieLogFile, List<String>> filteredLogs = new HashMap<>();
for (Path p : entry.getValue()) {
Review comment:
There is no need to use a map here, `List<List<String>> ` is enough.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5d49bd4e8c638ffb9ced102dc6771d6291c199e3 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r725419147
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieMergedLogReader.java
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop.realtime;
+
+import java.io.IOException;
+import java.text.MessageFormat;
+import java.util.Iterator;
+
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.io.ArrayWritable;
+import org.apache.hadoop.io.NullWritable;
+import org.apache.hadoop.io.Writable;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.RecordReader;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.hadoop.utils.HoodieRealtimeRecordReaderUtils;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+/**
+ * Record Reader implementation to read avro data, to support inc queries.
+ */
+public class HoodieMergedLogReader extends AbstractRealtimeRecordReader
+ implements RecordReader<NullWritable, ArrayWritable> {
+ private static final Logger LOG = LogManager.getLogger(AbstractRealtimeRecordReader.class);
+ private final HoodieMergedLogRecordScanner logRecordScanner;
+ private final Iterator<HoodieRecord<? extends HoodieRecordPayload>> logRecordsKeyIterator;
+ private ArrayWritable valueObj;
Review comment:
agree
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r725736428
##########
File path: hudi-hadoop-mr/src/test/java/org/apache/hudi/hadoop/realtime/TestHoodieRealtimeRecordReader.java
##########
@@ -535,4 +549,211 @@ public void testSchemaEvolutionAndRollbackBlockInLastLogFile(ExternalSpillableMa
arguments(ExternalSpillableMap.DiskMapType.ROCKS_DB, true, true)
);
}
+
+ @Test
+ public void testIncremetalWithOnlylog() throws Exception {
+ // initial commit
+ Schema schema = HoodieAvroUtils.addMetadataFields(SchemaTestUtil.getEvolvedSchema());
+ HoodieTestUtils.init(hadoopConf, basePath.toString(), HoodieTableType.MERGE_ON_READ);
+ String instantTime = "100";
+ final int numRecords = 1000;
+ File partitionDir = InputFormatTestUtil.prepareParquetTable(basePath, schema, 1, numRecords, instantTime,
+ HoodieTableType.MERGE_ON_READ);
+ //FileCreateUtils.createDeltaCommit(basePath.toString(), instantTime);
+ createDeltaCommitFile(basePath, instantTime,"2016/05/01", "2016/05/01/fileid0_1-0-1_100.parquet", "fileid0");
Review comment:
yes, the incremental query only involved log files， no parquet file will read。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r725743160
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieParquetRealtimeInputFormat.java
##########
@@ -66,6 +91,139 @@
return HoodieRealtimeInputFormatUtils.getRealtimeSplits(job, fileSplits);
}
+ /**
+ * keep the logical of mor_incr_view as same as spark datasource.
+ * to do: unify the incremental view code between hive/spark-sql and spark datasource
+ */
+ @Override
+ protected List<FileStatus> listStatusForIncrementalMode(
+ JobConf job, HoodieTableMetaClient tableMetaClient, List<Path> inputPaths) throws IOException {
+ List<FileStatus> result = new ArrayList<>();
+ String tableName = tableMetaClient.getTableConfig().getTableName();
+ Job jobContext = Job.getInstance(job);
+
+ Option<HoodieTimeline> timeline = HoodieInputFormatUtils.getFilteredCommitsTimeline(jobContext, tableMetaClient);
+ if (!timeline.isPresent()) {
+ return result;
+ }
+ String lastIncrementalTs = HoodieHiveUtils.readStartCommitTime(jobContext, tableName);
+ // Total number of commits to return in this batch. Set this to -1 to get all the commits.
+ Integer maxCommits = HoodieHiveUtils.readMaxCommits(jobContext, tableName);
+ HoodieTimeline commitsTimelineToReturn = timeline.get().findInstantsAfter(lastIncrementalTs, maxCommits);
+ Option<List<HoodieInstant>> commitsToCheck = Option.of(commitsTimelineToReturn.getInstants().collect(Collectors.toList()));
+ if (!commitsToCheck.isPresent()) {
+ return result;
+ }
+ Map<String, HashMap<String, FileStatus>> partitionsWithFileStatus = HoodieInputFormatUtils
+ .listAffectedFilesForCommits(new Path(tableMetaClient.getBasePath()), commitsToCheck.get(), commitsTimelineToReturn);
+ // build fileGroup from fsView
+ List<FileStatus> affectedFileStatus = new ArrayList<>();
+ partitionsWithFileStatus.forEach((key, value) -> value.forEach((k, v) -> affectedFileStatus.add(v)));
+ HoodieTableFileSystemView fsView = new HoodieTableFileSystemView(tableMetaClient, commitsTimelineToReturn, affectedFileStatus.toArray(new FileStatus[0]));
+ // build fileGroup from fsView
+ String basePath = tableMetaClient.getBasePath();
+ // filter affectedPartition by inputPaths
+ List<String> affectedPartition = partitionsWithFileStatus.keySet().stream()
+ .filter(k -> k.isEmpty() ? inputPaths.contains(new Path(basePath)) : inputPaths.contains(new Path(basePath, k))).collect(Collectors.toList());
+ if (affectedPartition.isEmpty()) {
+ return result;
+ }
+ List<HoodieFileGroup> fileGroups = affectedPartition.stream()
+ .flatMap(partitionPath -> fsView.getAllFileGroups(partitionPath)).collect(Collectors.toList());
+ setInputPaths(job, affectedPartition.stream()
+ .map(p -> p.isEmpty() ? basePath : new Path(basePath, p).toUri().toString()).collect(Collectors.joining(",")));
+
+ // find all file status in current partitionPath
+ FileStatus[] fileStatuses = getStatus(job);
+ Map<String, FileStatus> candidateFileStatus = new HashMap<>();
+ for (int i = 0; i < fileStatuses.length; i++) {
+ String key = fileStatuses[i].getPath().toString();
+ candidateFileStatus.put(key, fileStatuses[i]);
+ }
+
+ String maxCommitTime = fsView.getLastInstant().get().getTimestamp();
Review comment:
great suggest, i will split this method and add docs. thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r725743160
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieParquetRealtimeInputFormat.java
##########
@@ -66,6 +91,139 @@
return HoodieRealtimeInputFormatUtils.getRealtimeSplits(job, fileSplits);
}
+ /**
+ * keep the logical of mor_incr_view as same as spark datasource.
+ * to do: unify the incremental view code between hive/spark-sql and spark datasource
+ */
+ @Override
+ protected List<FileStatus> listStatusForIncrementalMode(
+ JobConf job, HoodieTableMetaClient tableMetaClient, List<Path> inputPaths) throws IOException {
+ List<FileStatus> result = new ArrayList<>();
+ String tableName = tableMetaClient.getTableConfig().getTableName();
+ Job jobContext = Job.getInstance(job);
+
+ Option<HoodieTimeline> timeline = HoodieInputFormatUtils.getFilteredCommitsTimeline(jobContext, tableMetaClient);
+ if (!timeline.isPresent()) {
+ return result;
+ }
+ String lastIncrementalTs = HoodieHiveUtils.readStartCommitTime(jobContext, tableName);
+ // Total number of commits to return in this batch. Set this to -1 to get all the commits.
+ Integer maxCommits = HoodieHiveUtils.readMaxCommits(jobContext, tableName);
+ HoodieTimeline commitsTimelineToReturn = timeline.get().findInstantsAfter(lastIncrementalTs, maxCommits);
+ Option<List<HoodieInstant>> commitsToCheck = Option.of(commitsTimelineToReturn.getInstants().collect(Collectors.toList()));
+ if (!commitsToCheck.isPresent()) {
+ return result;
+ }
+ Map<String, HashMap<String, FileStatus>> partitionsWithFileStatus = HoodieInputFormatUtils
+ .listAffectedFilesForCommits(new Path(tableMetaClient.getBasePath()), commitsToCheck.get(), commitsTimelineToReturn);
+ // build fileGroup from fsView
+ List<FileStatus> affectedFileStatus = new ArrayList<>();
+ partitionsWithFileStatus.forEach((key, value) -> value.forEach((k, v) -> affectedFileStatus.add(v)));
+ HoodieTableFileSystemView fsView = new HoodieTableFileSystemView(tableMetaClient, commitsTimelineToReturn, affectedFileStatus.toArray(new FileStatus[0]));
+ // build fileGroup from fsView
+ String basePath = tableMetaClient.getBasePath();
+ // filter affectedPartition by inputPaths
+ List<String> affectedPartition = partitionsWithFileStatus.keySet().stream()
+ .filter(k -> k.isEmpty() ? inputPaths.contains(new Path(basePath)) : inputPaths.contains(new Path(basePath, k))).collect(Collectors.toList());
+ if (affectedPartition.isEmpty()) {
+ return result;
+ }
+ List<HoodieFileGroup> fileGroups = affectedPartition.stream()
+ .flatMap(partitionPath -> fsView.getAllFileGroups(partitionPath)).collect(Collectors.toList());
+ setInputPaths(job, affectedPartition.stream()
+ .map(p -> p.isEmpty() ? basePath : new Path(basePath, p).toUri().toString()).collect(Collectors.joining(",")));
+
+ // find all file status in current partitionPath
+ FileStatus[] fileStatuses = getStatus(job);
+ Map<String, FileStatus> candidateFileStatus = new HashMap<>();
+ for (int i = 0; i < fileStatuses.length; i++) {
+ String key = fileStatuses[i].getPath().toString();
+ candidateFileStatus.put(key, fileStatuses[i]);
+ }
+
+ String maxCommitTime = fsView.getLastInstant().get().getTimestamp();
Review comment:
great suggest, i will split this method. thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
}, {
"hash" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325",
"triggerID" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"triggerType" : "PUSH"
}, {
"hash" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2343",
"triggerID" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"triggerType" : "PUSH"
}, {
"hash" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2424",
"triggerID" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2427",
"triggerID" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2429",
"triggerID" : "927749497",
"triggerType" : "MANUAL"
}, {
"hash" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2431",
"triggerID" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"triggerType" : "PUSH"
}, {
"hash" : "29e6acb65044597e35f9d499385dad940580d9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2432",
"triggerID" : "29e6acb65044597e35f9d499385dad940580d9df",
"triggerType" : "PUSH"
}, {
"hash" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2441",
"triggerID" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"triggerType" : "PUSH"
}, {
"hash" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2443",
"triggerID" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2447",
"triggerID" : "928890183",
"triggerType" : "MANUAL"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2470",
"triggerID" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"triggerType" : "PUSH"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2560",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562",
"triggerID" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"triggerType" : "PUSH"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
}, {
"hash" : "6fa031e1dcf6e21dfb24cf4296cb00d910b0a204",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2589",
"triggerID" : "6fa031e1dcf6e21dfb24cf4296cb00d910b0a204",
"triggerType" : "PUSH"
}, {
"hash" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2593",
"triggerID" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0fa6297ce58eb877fd5c4eba59fef20ad9335d26 UNKNOWN
* b5649459e5e8a5ebf5a140418c16951294a29689 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2593)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
}, {
"hash" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325",
"triggerID" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"triggerType" : "PUSH"
}, {
"hash" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2343",
"triggerID" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"triggerType" : "PUSH"
}, {
"hash" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2424",
"triggerID" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2427",
"triggerID" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2429",
"triggerID" : "927749497",
"triggerType" : "MANUAL"
}, {
"hash" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2431",
"triggerID" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"triggerType" : "PUSH"
}, {
"hash" : "29e6acb65044597e35f9d499385dad940580d9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2432",
"triggerID" : "29e6acb65044597e35f9d499385dad940580d9df",
"triggerType" : "PUSH"
}, {
"hash" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2441",
"triggerID" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"triggerType" : "PUSH"
}, {
"hash" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2443",
"triggerID" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2447",
"triggerID" : "928890183",
"triggerType" : "MANUAL"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2470",
"triggerID" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"triggerType" : "PUSH"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2560",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562",
"triggerID" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"triggerType" : "PUSH"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
}, {
"hash" : "6fa031e1dcf6e21dfb24cf4296cb00d910b0a204",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2589",
"triggerID" : "6fa031e1dcf6e21dfb24cf4296cb00d910b0a204",
"triggerType" : "PUSH"
}, {
"hash" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2593",
"triggerID" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"triggerType" : "PUSH"
}, {
"hash" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2604",
"triggerID" : "941865271",
"triggerType" : "MANUAL"
}, {
"hash" : "6d67b68e19a43f8668e5773d27ca9c33a8de0a37",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6d67b68e19a43f8668e5773d27ca9c33a8de0a37",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0fa6297ce58eb877fd5c4eba59fef20ad9335d26 UNKNOWN
* b5649459e5e8a5ebf5a140418c16951294a29689 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2593) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2604)
* 6d67b68e19a43f8668e5773d27ca9c33a8de0a37 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
}, {
"hash" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325",
"triggerID" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"triggerType" : "PUSH"
}, {
"hash" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2343",
"triggerID" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"triggerType" : "PUSH"
}, {
"hash" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2424",
"triggerID" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2427",
"triggerID" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2429",
"triggerID" : "927749497",
"triggerType" : "MANUAL"
}, {
"hash" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2431",
"triggerID" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"triggerType" : "PUSH"
}, {
"hash" : "29e6acb65044597e35f9d499385dad940580d9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2432",
"triggerID" : "29e6acb65044597e35f9d499385dad940580d9df",
"triggerType" : "PUSH"
}, {
"hash" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2441",
"triggerID" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"triggerType" : "PUSH"
}, {
"hash" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2443",
"triggerID" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2447",
"triggerID" : "928890183",
"triggerType" : "MANUAL"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2470",
"triggerID" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"triggerType" : "PUSH"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2560",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562",
"triggerID" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"triggerType" : "PUSH"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
}, {
"hash" : "6fa031e1dcf6e21dfb24cf4296cb00d910b0a204",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2589",
"triggerID" : "6fa031e1dcf6e21dfb24cf4296cb00d910b0a204",
"triggerType" : "PUSH"
}, {
"hash" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2593",
"triggerID" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"triggerType" : "PUSH"
}, {
"hash" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2604",
"triggerID" : "941865271",
"triggerType" : "MANUAL"
}, {
"hash" : "6d67b68e19a43f8668e5773d27ca9c33a8de0a37",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2642",
"triggerID" : "6d67b68e19a43f8668e5773d27ca9c33a8de0a37",
"triggerType" : "PUSH"
}, {
"hash" : "1a80559bd98829552acffdaf20d3ea0384d1d936",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2645",
"triggerID" : "1a80559bd98829552acffdaf20d3ea0384d1d936",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0fa6297ce58eb877fd5c4eba59fef20ad9335d26 UNKNOWN
* 6d67b68e19a43f8668e5773d27ca9c33a8de0a37 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2642)
* 1a80559bd98829552acffdaf20d3ea0384d1d936 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2645)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r716729409
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/HoodieParquetInputFormat.java
##########
@@ -73,6 +73,14 @@ protected HoodieDefaultTimeline filterInstantsTimeline(HoodieDefaultTimeline tim
return HoodieInputFormatUtils.filterInstantsTimeline(timeline);
}
+ protected FileStatus[] getStatus(JobConf job) throws IOException {
+ return super.listStatus(job);
+ }
+
+ protected boolean includeLogFilesForSnapShortView() {
Review comment:
`SnapshotView`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r733521589
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieRealtimeInputFormatUtils.java
##########
@@ -161,6 +162,46 @@
return rtSplits.toArray(new InputSplit[0]);
}
+ // pick all incremental files and add them to rtSplits then filter out those files.
+ private static Map<Path, List<FileSplit>> filterOutIncrementalSplits(
+ List<FileSplit> fileSplitList,
Review comment:
The key point is not the number of commits， the key point is that at commit3 file3 replace the file1 and file2。 we can set startcommit a little smller than commit2 and max commit is 1.
set hoodie.testlog.consume.max.commits=1
set hoodie.testlog.consume.start.timestamp=commit2-1 then commit2 will be picked
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-953836748
@danny0405 update the code， test fail has no relate to this pr。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a change in pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
codope commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r739354289
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieRealtimeInputFormatUtils.java
##########
@@ -161,6 +148,82 @@
return rtSplits.toArray(new InputSplit[0]);
}
+ // get IncrementalRealtimeSplits
+ public static InputSplit[] getIncrementalRealtimeSplits(Configuration conf, Stream<FileSplit> fileSplits) throws IOException {
+ List<InputSplit> rtSplits = new ArrayList<>();
+ List<FileSplit> fileSplitList = fileSplits.collect(Collectors.toList());
+ Set<Path> partitionSet = fileSplitList.stream().map(f -> f.getPath().getParent()).collect(Collectors.toSet());
+ Map<Path, HoodieTableMetaClient> partitionsToMetaClient = getTableMetaClientByPartitionPath(conf, partitionSet);
+ // Pre process tableConfig from first partition to fetch virtual key info
+ Option<HoodieVirtualKeyInfo> hoodieVirtualKeyInfo = Option.empty();
+ if (partitionSet.size() > 0) {
+ hoodieVirtualKeyInfo = getHoodieVirtualKeyInfo(partitionsToMetaClient.get(partitionSet.iterator().next()));
+ }
+ Option<HoodieVirtualKeyInfo> finalHoodieVirtualKeyInfo = hoodieVirtualKeyInfo;
+ fileSplitList.stream().forEach(s -> {
+ // deal with incremental query.
+ try {
+ if (s instanceof BaseFileWithLogsSplit) {
+ BaseFileWithLogsSplit bs = (BaseFileWithLogsSplit)s;
+ if (bs.getBelongToIncrementalSplit()) {
+ rtSplits.add(new HoodieRealtimeFileSplit(bs, bs.getBasePath(), bs.getDeltaLogPaths(), bs.getMaxCommitTime(), finalHoodieVirtualKeyInfo));
+ }
+ } else if (s instanceof RealtimeBootstrapBaseFileSplit) {
+ rtSplits.add(s);
+ }
+ } catch (IOException e) {
+ throw new HoodieIOException("Error creating hoodie real time split ", e);
+ }
+ });
+ LOG.info("Returning a total splits of " + rtSplits.size());
+ return rtSplits.toArray(new InputSplit[0]);
+ }
+
+ public static Option<HoodieVirtualKeyInfo> getHoodieVirtualKeyInfo(HoodieTableMetaClient metaClient) {
+ HoodieTableConfig tableConfig = metaClient.getTableConfig();
+ if (!tableConfig.populateMetaFields()) {
+ TableSchemaResolver tableSchemaResolver = new TableSchemaResolver(metaClient);
+ try {
+ MessageType parquetSchema = tableSchemaResolver.getTableParquetSchema();
+ return Option.of(new HoodieVirtualKeyInfo(tableConfig.getRecordKeyFieldProp(),
+ tableConfig.getPartitionFieldProp(), parquetSchema.getFieldIndex(tableConfig.getRecordKeyFieldProp()),
+ parquetSchema.getFieldIndex(tableConfig.getPartitionFieldProp())));
+ } catch (Exception exception) {
+ throw new HoodieException("Fetching table schema failed with exception ", exception);
+ }
+ }
+ return Option.empty();
+ }
+
+ public static boolean isIncrementalQuerySplits(List<FileSplit> fileSplits) {
+ if (fileSplits == null || fileSplits.size() == 0) {
+ return false;
+ }
+ return fileSplits.stream().anyMatch(s -> {
+ if (s instanceof BaseFileWithLogsSplit) {
+ BaseFileWithLogsSplit bs = (BaseFileWithLogsSplit)s;
+ return bs.getBelongToIncrementalSplit();
+ } else {
+ return s instanceof RealtimeBootstrapBaseFileSplit;
+ }
+ });
+ }
+
+ public static RealtimeBootstrapBaseFileSplit createRealimeBootstrapBaseFileSplit(
Review comment:
minor: rename to createReal**t**imeBootstrapBaseFileSplit
##########
File path: hudi-hadoop-mr/src/test/java/org/apache/hudi/hadoop/realtime/TestHoodieRealtimeRecordReader.java
##########
@@ -535,4 +548,211 @@ public void testSchemaEvolutionAndRollbackBlockInLastLogFile(ExternalSpillableMa
arguments(ExternalSpillableMap.DiskMapType.ROCKS_DB, true, true)
);
}
+
+ @Test
+ public void testIncremetalWithOnlylog() throws Exception {
+ // initial commit
+ Schema schema = HoodieAvroUtils.addMetadataFields(SchemaTestUtil.getEvolvedSchema());
+ HoodieTestUtils.init(hadoopConf, basePath.toString(), HoodieTableType.MERGE_ON_READ);
+ String instantTime = "100";
+ final int numRecords = 1000;
+ File partitionDir = InputFormatTestUtil.prepareParquetTable(basePath, schema, 1, numRecords, instantTime,
+ HoodieTableType.MERGE_ON_READ);
+ //FileCreateUtils.createDeltaCommit(basePath.toString(), instantTime);
+ createDeltaCommitFile(basePath, instantTime,"2016/05/01", "2016/05/01/fileid0_1-0-1_100.parquet", "fileid0");
+ // Add the paths
+ FileInputFormat.setInputPaths(baseJobConf, partitionDir.getPath());
+
+ // insert new records to log file
+ try {
+ String newCommitTime = "102";
+ HoodieLogFormat.Writer writer =
+ InputFormatTestUtil.writeDataBlockToLogFile(partitionDir, fs, schema, "fileid0", instantTime, newCommitTime,
+ numRecords, numRecords, 0);
+ writer.close();
+ createDeltaCommitFile(basePath, newCommitTime,"2016/05/01", "2016/05/01/.fileid0_100.log.1_1-0-1", "fileid0");
+
+ InputFormatTestUtil.setupIncremental(baseJobConf, "101", 1);
+
+ HoodieParquetRealtimeInputFormat inputFormat = new HoodieParquetRealtimeInputFormat();
+ inputFormat.setConf(baseJobConf);
+ InputSplit[] splits = inputFormat.getSplits(baseJobConf, 1);
+ assertTrue(splits.length == 1);
+ JobConf newJobConf = new JobConf(baseJobConf);
+ List<Schema.Field> fields = schema.getFields();
+ setHiveColumnNameProps(fields, newJobConf, false);
+ RecordReader<NullWritable, ArrayWritable> reader = inputFormat.getRecordReader(splits[0], newJobConf, Reporter.NULL);
+ // use reader to read log file.
+ NullWritable key = reader.createKey();
+ ArrayWritable value = reader.createValue();
+ while (reader.next(key, value)) {
+ Writable[] values = value.get();
+ // check if the record written is with latest commit, here "101"
Review comment:
minor: let's fix the comment here and #L640. Latest commit time may not necessarily be "101" always. Like, here it is "102" the newCommitTime and in the test below it should be "100". More importantly, for the person reading the code, it's better to make it clear why latest commit time should be "100" or any other value that is expected in the assertion.
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieParquetRealtimeInputFormat.java
##########
@@ -61,9 +85,183 @@
@Override
public InputSplit[] getSplits(JobConf job, int numSplits) throws IOException {
- Stream<FileSplit> fileSplits = Arrays.stream(super.getSplits(job, numSplits)).map(is -> (FileSplit) is);
+ List<FileSplit> fileSplits = Arrays.stream(super.getSplits(job, numSplits)).map(is -> (FileSplit) is).collect(Collectors.toList());
+
+ boolean isIncrementalSplits = HoodieRealtimeInputFormatUtils.isIncrementalQuerySplits(fileSplits);
+
+ return isIncrementalSplits ? HoodieRealtimeInputFormatUtils.getIncrementalRealtimeSplits(job, fileSplits.stream()) : HoodieRealtimeInputFormatUtils.getRealtimeSplits(job, fileSplits.stream());
+ }
+
+ /**
+ * Keep the logic of mor_incr_view as same as spark datasource.
+ * Step1: Get list of commits to be fetched based on start commit and max commits(for snapshot max commits is -1).
+ * Step2: Get list of affected files status for these affected file status.
+ * Step3: Construct HoodieTableFileSystemView based on those affected file status.
+ * a. Filter affected partitions based on inputPaths.
+ * b. Get list of fileGroups based on affected partitions by fsView.getAllFileGroups.
+ * Step4: Set input paths based on filtered affected partition paths. changes that amony original input paths passed to
+ * this method. some partitions did not have commits as part of the trimmed down list of commits and hence we need this step.
+ * Step5: Find candidate fileStatus, since when we get baseFileStatus from HoodieTableFileSystemView,
+ * the BaseFileStatus will missing file size information.
+ * We should use candidate fileStatus to update the size information for BaseFileStatus.
+ * Step6: For every file group from step3(b)
+ * Get 1st available base file from all file slices. then we use candidate file status to update the baseFileStatus,
+ * and construct RealTimeFileStatus and add it to result along with log files.
+ * If file group just has log files, construct RealTimeFileStatus and add it to result.
+ * TODO: unify the incremental view code between hive/spark-sql and spark datasource
+ */
+ @Override
+ protected List<FileStatus> listStatusForIncrementalMode(
+ JobConf job, HoodieTableMetaClient tableMetaClient, List<Path> inputPaths) throws IOException {
+ List<FileStatus> result = new ArrayList<>();
+ String tableName = tableMetaClient.getTableConfig().getTableName();
+ Job jobContext = Job.getInstance(job);
+
+ // step1
+ Option<HoodieTimeline> timeline = HoodieInputFormatUtils.getFilteredCommitsTimeline(jobContext, tableMetaClient);
+ if (!timeline.isPresent()) {
+ return result;
+ }
+ String lastIncrementalTs = HoodieHiveUtils.readStartCommitTime(jobContext, tableName);
+ // Total number of commits to return in this batch. Set this to -1 to get all the commits.
+ Integer maxCommits = HoodieHiveUtils.readMaxCommits(jobContext, tableName);
+ HoodieTimeline commitsTimelineToReturn = timeline.get().findInstantsAfter(lastIncrementalTs, maxCommits);
+ Option<List<HoodieInstant>> commitsToCheck = Option.of(commitsTimelineToReturn.getInstants().collect(Collectors.toList()));
+ if (!commitsToCheck.isPresent()) {
+ return result;
+ }
+ // step2
+ commitsToCheck.get().sort(HoodieInstant::compareTo);
+ List<HoodieCommitMetadata> metadataList = commitsToCheck
+ .get().stream().map(instant -> {
+ try {
+ return HoodieInputFormatUtils.getCommitMetadata(instant, commitsTimelineToReturn);
+ } catch (IOException e) {
+ throw new HoodieException(String.format("cannot get metadata for instant: %s", instant));
+ }
+ }).collect(Collectors.toList());
+
+ // build fileGroup from fsView
+ List<FileStatus> affectedFileStatus = Arrays.asList(HoodieInputFormatUtils
+ .listAffectedFilesForCommits(new Path(tableMetaClient.getBasePath()), metadataList));
+ // step3
+ HoodieTableFileSystemView fsView = new HoodieTableFileSystemView(tableMetaClient, commitsTimelineToReturn, affectedFileStatus.toArray(new FileStatus[0]));
+ // build fileGroup from fsView
+ Path basePath = new Path(tableMetaClient.getBasePath());
+ // filter affectedPartition by inputPaths
+ List<String> affectedPartition = HoodieInputFormatUtils.getWritePartitionPaths(metadataList).stream()
+ .filter(k -> k.isEmpty() ? inputPaths.contains(basePath) : inputPaths.contains(new Path(basePath, k))).collect(Collectors.toList());
+ if (affectedPartition.isEmpty()) {
+ return result;
+ }
+ List<HoodieFileGroup> fileGroups = affectedPartition.stream()
Review comment:
This can be a costly operation as we need to take read lock every time. Will caching file groups by partition be helpful?
##########
File path: hudi-hadoop-mr/src/test/java/org/apache/hudi/hadoop/realtime/TestHoodieRealtimeRecordReader.java
##########
@@ -535,4 +548,211 @@ public void testSchemaEvolutionAndRollbackBlockInLastLogFile(ExternalSpillableMa
arguments(ExternalSpillableMap.DiskMapType.ROCKS_DB, true, true)
);
}
+
+ @Test
+ public void testIncremetalWithOnlylog() throws Exception {
Review comment:
minor: rename to testIncreme**n**talWithOnlylog (incremental spelling here and at #L602)
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieParquetRealtimeInputFormat.java
##########
@@ -61,9 +85,183 @@
@Override
public InputSplit[] getSplits(JobConf job, int numSplits) throws IOException {
- Stream<FileSplit> fileSplits = Arrays.stream(super.getSplits(job, numSplits)).map(is -> (FileSplit) is);
+ List<FileSplit> fileSplits = Arrays.stream(super.getSplits(job, numSplits)).map(is -> (FileSplit) is).collect(Collectors.toList());
+
+ boolean isIncrementalSplits = HoodieRealtimeInputFormatUtils.isIncrementalQuerySplits(fileSplits);
+
+ return isIncrementalSplits ? HoodieRealtimeInputFormatUtils.getIncrementalRealtimeSplits(job, fileSplits.stream()) : HoodieRealtimeInputFormatUtils.getRealtimeSplits(job, fileSplits.stream());
+ }
+
+ /**
+ * Keep the logic of mor_incr_view as same as spark datasource.
+ * Step1: Get list of commits to be fetched based on start commit and max commits(for snapshot max commits is -1).
+ * Step2: Get list of affected files status for these affected file status.
+ * Step3: Construct HoodieTableFileSystemView based on those affected file status.
+ * a. Filter affected partitions based on inputPaths.
+ * b. Get list of fileGroups based on affected partitions by fsView.getAllFileGroups.
+ * Step4: Set input paths based on filtered affected partition paths. changes that amony original input paths passed to
+ * this method. some partitions did not have commits as part of the trimmed down list of commits and hence we need this step.
+ * Step5: Find candidate fileStatus, since when we get baseFileStatus from HoodieTableFileSystemView,
+ * the BaseFileStatus will missing file size information.
+ * We should use candidate fileStatus to update the size information for BaseFileStatus.
+ * Step6: For every file group from step3(b)
+ * Get 1st available base file from all file slices. then we use candidate file status to update the baseFileStatus,
+ * and construct RealTimeFileStatus and add it to result along with log files.
+ * If file group just has log files, construct RealTimeFileStatus and add it to result.
+ * TODO: unify the incremental view code between hive/spark-sql and spark datasource
+ */
+ @Override
+ protected List<FileStatus> listStatusForIncrementalMode(
+ JobConf job, HoodieTableMetaClient tableMetaClient, List<Path> inputPaths) throws IOException {
+ List<FileStatus> result = new ArrayList<>();
+ String tableName = tableMetaClient.getTableConfig().getTableName();
+ Job jobContext = Job.getInstance(job);
+
+ // step1
+ Option<HoodieTimeline> timeline = HoodieInputFormatUtils.getFilteredCommitsTimeline(jobContext, tableMetaClient);
+ if (!timeline.isPresent()) {
+ return result;
+ }
+ String lastIncrementalTs = HoodieHiveUtils.readStartCommitTime(jobContext, tableName);
+ // Total number of commits to return in this batch. Set this to -1 to get all the commits.
+ Integer maxCommits = HoodieHiveUtils.readMaxCommits(jobContext, tableName);
Review comment:
Can we reuse `HoodieInputFormatUtils#getCommitsForIncrementalQuery`?
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieParquetRealtimeInputFormat.java
##########
@@ -61,9 +85,183 @@
@Override
public InputSplit[] getSplits(JobConf job, int numSplits) throws IOException {
- Stream<FileSplit> fileSplits = Arrays.stream(super.getSplits(job, numSplits)).map(is -> (FileSplit) is);
+ List<FileSplit> fileSplits = Arrays.stream(super.getSplits(job, numSplits)).map(is -> (FileSplit) is).collect(Collectors.toList());
+
+ boolean isIncrementalSplits = HoodieRealtimeInputFormatUtils.isIncrementalQuerySplits(fileSplits);
+
+ return isIncrementalSplits ? HoodieRealtimeInputFormatUtils.getIncrementalRealtimeSplits(job, fileSplits.stream()) : HoodieRealtimeInputFormatUtils.getRealtimeSplits(job, fileSplits.stream());
+ }
+
+ /**
+ * Keep the logic of mor_incr_view as same as spark datasource.
+ * Step1: Get list of commits to be fetched based on start commit and max commits(for snapshot max commits is -1).
+ * Step2: Get list of affected files status for these affected file status.
+ * Step3: Construct HoodieTableFileSystemView based on those affected file status.
+ * a. Filter affected partitions based on inputPaths.
+ * b. Get list of fileGroups based on affected partitions by fsView.getAllFileGroups.
+ * Step4: Set input paths based on filtered affected partition paths. changes that amony original input paths passed to
+ * this method. some partitions did not have commits as part of the trimmed down list of commits and hence we need this step.
+ * Step5: Find candidate fileStatus, since when we get baseFileStatus from HoodieTableFileSystemView,
+ * the BaseFileStatus will missing file size information.
+ * We should use candidate fileStatus to update the size information for BaseFileStatus.
+ * Step6: For every file group from step3(b)
+ * Get 1st available base file from all file slices. then we use candidate file status to update the baseFileStatus,
+ * and construct RealTimeFileStatus and add it to result along with log files.
+ * If file group just has log files, construct RealTimeFileStatus and add it to result.
+ * TODO: unify the incremental view code between hive/spark-sql and spark datasource
+ */
+ @Override
+ protected List<FileStatus> listStatusForIncrementalMode(
+ JobConf job, HoodieTableMetaClient tableMetaClient, List<Path> inputPaths) throws IOException {
+ List<FileStatus> result = new ArrayList<>();
+ String tableName = tableMetaClient.getTableConfig().getTableName();
+ Job jobContext = Job.getInstance(job);
+
+ // step1
+ Option<HoodieTimeline> timeline = HoodieInputFormatUtils.getFilteredCommitsTimeline(jobContext, tableMetaClient);
+ if (!timeline.isPresent()) {
+ return result;
+ }
+ String lastIncrementalTs = HoodieHiveUtils.readStartCommitTime(jobContext, tableName);
+ // Total number of commits to return in this batch. Set this to -1 to get all the commits.
+ Integer maxCommits = HoodieHiveUtils.readMaxCommits(jobContext, tableName);
+ HoodieTimeline commitsTimelineToReturn = timeline.get().findInstantsAfter(lastIncrementalTs, maxCommits);
+ Option<List<HoodieInstant>> commitsToCheck = Option.of(commitsTimelineToReturn.getInstants().collect(Collectors.toList()));
+ if (!commitsToCheck.isPresent()) {
+ return result;
+ }
+ // step2
+ commitsToCheck.get().sort(HoodieInstant::compareTo);
+ List<HoodieCommitMetadata> metadataList = commitsToCheck
+ .get().stream().map(instant -> {
+ try {
+ return HoodieInputFormatUtils.getCommitMetadata(instant, commitsTimelineToReturn);
+ } catch (IOException e) {
+ throw new HoodieException(String.format("cannot get metadata for instant: %s", instant));
+ }
+ }).collect(Collectors.toList());
+
+ // build fileGroup from fsView
+ List<FileStatus> affectedFileStatus = Arrays.asList(HoodieInputFormatUtils
+ .listAffectedFilesForCommits(new Path(tableMetaClient.getBasePath()), metadataList));
+ // step3
+ HoodieTableFileSystemView fsView = new HoodieTableFileSystemView(tableMetaClient, commitsTimelineToReturn, affectedFileStatus.toArray(new FileStatus[0]));
+ // build fileGroup from fsView
+ Path basePath = new Path(tableMetaClient.getBasePath());
+ // filter affectedPartition by inputPaths
+ List<String> affectedPartition = HoodieInputFormatUtils.getWritePartitionPaths(metadataList).stream()
+ .filter(k -> k.isEmpty() ? inputPaths.contains(basePath) : inputPaths.contains(new Path(basePath, k))).collect(Collectors.toList());
+ if (affectedPartition.isEmpty()) {
+ return result;
+ }
+ List<HoodieFileGroup> fileGroups = affectedPartition.stream()
+ .flatMap(partitionPath -> fsView.getAllFileGroups(partitionPath)).collect(Collectors.toList());
+ // step4
+ setInputPaths(job, affectedPartition.stream()
+ .map(p -> p.isEmpty() ? basePath.toString() : new Path(basePath, p).toString()).collect(Collectors.joining(",")));
+
+ // step5
+ // find all file status in partitionPaths.
+ FileStatus[] fileStatuses = getStatus(job);
+ Map<String, FileStatus> candidateFileStatus = new HashMap<>();
+ for (int i = 0; i < fileStatuses.length; i++) {
+ String key = fileStatuses[i].getPath().toString();
+ candidateFileStatus.put(key, fileStatuses[i]);
+ }
+
+ String maxCommitTime = fsView.getLastInstant().get().getTimestamp();
+ // step6
+ result.addAll(collectAllIncrementalFiles(fileGroups, maxCommitTime, basePath.toString(), candidateFileStatus));
+ return result;
+ }
+
+ private List<FileStatus> collectAllIncrementalFiles(List<HoodieFileGroup> fileGroups, String maxCommitTime, String basePath, Map<String, FileStatus> candidateFileStatus) {
+ List<FileStatus> result = new ArrayList<>();
+ fileGroups.stream().forEach(f -> {
+ try {
+ List<FileSlice> baseFiles = f.getAllFileSlices().filter(slice -> slice.getBaseFile().isPresent()).collect(Collectors.toList());
+ if (!baseFiles.isEmpty()) {
+ FileStatus baseFileStatus = HoodieInputFormatUtils.getFileStatus(baseFiles.get(0).getBaseFile().get());
+ String baseFilePath = baseFileStatus.getPath().toUri().toString();
+ if (!candidateFileStatus.containsKey(baseFilePath)) {
+ throw new HoodieException("Error obtaining fileStatus for file: " + baseFilePath);
+ }
+ // We cannot use baseFileStatus.getPath() here, since baseFileStatus.getPath() missing file size information.
+ // So we use candidateFileStatus.get(baseFileStatus.getPath()) to get a correct path.
+ RealtimeFileStatus fileStatus = new RealtimeFileStatus(candidateFileStatus.get(baseFilePath));
+ fileStatus.setMaxCommitTime(maxCommitTime);
+ fileStatus.setBelongToIncrementalFileStatus(true);
+ fileStatus.setBasePath(basePath.toString());
Review comment:
`toString()` is redundant as `basePath` is already a string. Same on #L211.
##########
File path: hudi-hadoop-mr/src/test/java/org/apache/hudi/hadoop/realtime/TestHoodieRealtimeRecordReader.java
##########
@@ -535,4 +548,211 @@ public void testSchemaEvolutionAndRollbackBlockInLastLogFile(ExternalSpillableMa
arguments(ExternalSpillableMap.DiskMapType.ROCKS_DB, true, true)
);
}
+
+ @Test
+ public void testIncremetalWithOnlylog() throws Exception {
+ // initial commit
+ Schema schema = HoodieAvroUtils.addMetadataFields(SchemaTestUtil.getEvolvedSchema());
+ HoodieTestUtils.init(hadoopConf, basePath.toString(), HoodieTableType.MERGE_ON_READ);
+ String instantTime = "100";
+ final int numRecords = 1000;
+ File partitionDir = InputFormatTestUtil.prepareParquetTable(basePath, schema, 1, numRecords, instantTime,
+ HoodieTableType.MERGE_ON_READ);
+ //FileCreateUtils.createDeltaCommit(basePath.toString(), instantTime);
+ createDeltaCommitFile(basePath, instantTime,"2016/05/01", "2016/05/01/fileid0_1-0-1_100.parquet", "fileid0");
+ // Add the paths
+ FileInputFormat.setInputPaths(baseJobConf, partitionDir.getPath());
+
+ // insert new records to log file
+ try {
+ String newCommitTime = "102";
+ HoodieLogFormat.Writer writer =
+ InputFormatTestUtil.writeDataBlockToLogFile(partitionDir, fs, schema, "fileid0", instantTime, newCommitTime,
+ numRecords, numRecords, 0);
+ writer.close();
+ createDeltaCommitFile(basePath, newCommitTime,"2016/05/01", "2016/05/01/.fileid0_100.log.1_1-0-1", "fileid0");
+
+ InputFormatTestUtil.setupIncremental(baseJobConf, "101", 1);
+
+ HoodieParquetRealtimeInputFormat inputFormat = new HoodieParquetRealtimeInputFormat();
+ inputFormat.setConf(baseJobConf);
+ InputSplit[] splits = inputFormat.getSplits(baseJobConf, 1);
+ assertTrue(splits.length == 1);
+ JobConf newJobConf = new JobConf(baseJobConf);
+ List<Schema.Field> fields = schema.getFields();
+ setHiveColumnNameProps(fields, newJobConf, false);
+ RecordReader<NullWritable, ArrayWritable> reader = inputFormat.getRecordReader(splits[0], newJobConf, Reporter.NULL);
+ // use reader to read log file.
+ NullWritable key = reader.createKey();
+ ArrayWritable value = reader.createValue();
+ while (reader.next(key, value)) {
+ Writable[] values = value.get();
+ // check if the record written is with latest commit, here "101"
+ assertEquals(newCommitTime, values[0].toString());
+ key = reader.createKey();
+ value = reader.createValue();
+ }
+ reader.close();
+ } catch (IOException e) {
+ throw new HoodieException(e.getMessage(), e);
+ }
+ }
+
+ @Test
+ public void testIncremetalWithReplace() throws Exception {
Review comment:
+1 for this test.
Shall we also validate the record reader with compaction? In that case, the latest commit time should match with compaction commit right. Is that already covered in these tests?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r740697734
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieParquetRealtimeInputFormat.java
##########
@@ -61,9 +85,183 @@
@Override
public InputSplit[] getSplits(JobConf job, int numSplits) throws IOException {
- Stream<FileSplit> fileSplits = Arrays.stream(super.getSplits(job, numSplits)).map(is -> (FileSplit) is);
+ List<FileSplit> fileSplits = Arrays.stream(super.getSplits(job, numSplits)).map(is -> (FileSplit) is).collect(Collectors.toList());
+
+ boolean isIncrementalSplits = HoodieRealtimeInputFormatUtils.isIncrementalQuerySplits(fileSplits);
+
+ return isIncrementalSplits ? HoodieRealtimeInputFormatUtils.getIncrementalRealtimeSplits(job, fileSplits.stream()) : HoodieRealtimeInputFormatUtils.getRealtimeSplits(job, fileSplits.stream());
+ }
+
+ /**
+ * Keep the logic of mor_incr_view as same as spark datasource.
+ * Step1: Get list of commits to be fetched based on start commit and max commits(for snapshot max commits is -1).
+ * Step2: Get list of affected files status for these affected file status.
+ * Step3: Construct HoodieTableFileSystemView based on those affected file status.
+ * a. Filter affected partitions based on inputPaths.
+ * b. Get list of fileGroups based on affected partitions by fsView.getAllFileGroups.
+ * Step4: Set input paths based on filtered affected partition paths. changes that amony original input paths passed to
+ * this method. some partitions did not have commits as part of the trimmed down list of commits and hence we need this step.
+ * Step5: Find candidate fileStatus, since when we get baseFileStatus from HoodieTableFileSystemView,
+ * the BaseFileStatus will missing file size information.
+ * We should use candidate fileStatus to update the size information for BaseFileStatus.
+ * Step6: For every file group from step3(b)
+ * Get 1st available base file from all file slices. then we use candidate file status to update the baseFileStatus,
+ * and construct RealTimeFileStatus and add it to result along with log files.
+ * If file group just has log files, construct RealTimeFileStatus and add it to result.
+ * TODO: unify the incremental view code between hive/spark-sql and spark datasource
+ */
+ @Override
+ protected List<FileStatus> listStatusForIncrementalMode(
+ JobConf job, HoodieTableMetaClient tableMetaClient, List<Path> inputPaths) throws IOException {
+ List<FileStatus> result = new ArrayList<>();
+ String tableName = tableMetaClient.getTableConfig().getTableName();
+ Job jobContext = Job.getInstance(job);
+
+ // step1
+ Option<HoodieTimeline> timeline = HoodieInputFormatUtils.getFilteredCommitsTimeline(jobContext, tableMetaClient);
+ if (!timeline.isPresent()) {
+ return result;
+ }
+ String lastIncrementalTs = HoodieHiveUtils.readStartCommitTime(jobContext, tableName);
+ // Total number of commits to return in this batch. Set this to -1 to get all the commits.
+ Integer maxCommits = HoodieHiveUtils.readMaxCommits(jobContext, tableName);
+ HoodieTimeline commitsTimelineToReturn = timeline.get().findInstantsAfter(lastIncrementalTs, maxCommits);
+ Option<List<HoodieInstant>> commitsToCheck = Option.of(commitsTimelineToReturn.getInstants().collect(Collectors.toList()));
+ if (!commitsToCheck.isPresent()) {
+ return result;
+ }
+ // step2
+ commitsToCheck.get().sort(HoodieInstant::compareTo);
+ List<HoodieCommitMetadata> metadataList = commitsToCheck
+ .get().stream().map(instant -> {
+ try {
+ return HoodieInputFormatUtils.getCommitMetadata(instant, commitsTimelineToReturn);
+ } catch (IOException e) {
+ throw new HoodieException(String.format("cannot get metadata for instant: %s", instant));
+ }
+ }).collect(Collectors.toList());
+
+ // build fileGroup from fsView
+ List<FileStatus> affectedFileStatus = Arrays.asList(HoodieInputFormatUtils
+ .listAffectedFilesForCommits(new Path(tableMetaClient.getBasePath()), metadataList));
+ // step3
+ HoodieTableFileSystemView fsView = new HoodieTableFileSystemView(tableMetaClient, commitsTimelineToReturn, affectedFileStatus.toArray(new FileStatus[0]));
+ // build fileGroup from fsView
+ Path basePath = new Path(tableMetaClient.getBasePath());
+ // filter affectedPartition by inputPaths
+ List<String> affectedPartition = HoodieInputFormatUtils.getWritePartitionPaths(metadataList).stream()
+ .filter(k -> k.isEmpty() ? inputPaths.contains(basePath) : inputPaths.contains(new Path(basePath, k))).collect(Collectors.toList());
+ if (affectedPartition.isEmpty()) {
+ return result;
+ }
+ List<HoodieFileGroup> fileGroups = affectedPartition.stream()
Review comment:
L148 has already added all files in HoodieTableFileSystemView， and partitionToFileGroupsMap will be filled and cached。
L157 take the read lock， but it can load fileGroups from partitionToFileGroupsMap directly no need fs rpc call， i thinks It won't cost much to read the lock。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-957530371
@codope thanks for your review.
Adressed all comments。
Add test case for compaction to verify that incremental reads do NOT show inserts after compaction timestamp， other situations are currently covered by hudi.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r718189151
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieParquetRealtimeInputFormat.java
##########
@@ -66,6 +90,138 @@
return HoodieRealtimeInputFormatUtils.getRealtimeSplits(job, fileSplits);
}
+ /**
+ * keep the logical of mor_incr_view as same as spark datasource.
+ * to do: unify the incremental view code between hive/spark-sql and spark datasource
Review comment:
the logic has already be unified，now the logic of hive incremental and spark incremental is same。 but to do unify the code has a little difficult。 may be can do it in another pr。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao edited a comment on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-962975376
@danny0405 @nsivabalan @codope i test latest master code from hudi , there still exist this problem.
this problem has no relate to this pr.
let me fixed it today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-962975376
@danny0405 @nsivabalan @codope i test latest master code from hudi , there still exist this problem.
it has no relate to this pr.
let me fixed it today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r740695703
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieRealtimeInputFormatUtils.java
##########
@@ -161,6 +148,82 @@
return rtSplits.toArray(new InputSplit[0]);
}
+ // get IncrementalRealtimeSplits
+ public static InputSplit[] getIncrementalRealtimeSplits(Configuration conf, Stream<FileSplit> fileSplits) throws IOException {
+ List<InputSplit> rtSplits = new ArrayList<>();
+ List<FileSplit> fileSplitList = fileSplits.collect(Collectors.toList());
+ Set<Path> partitionSet = fileSplitList.stream().map(f -> f.getPath().getParent()).collect(Collectors.toSet());
+ Map<Path, HoodieTableMetaClient> partitionsToMetaClient = getTableMetaClientByPartitionPath(conf, partitionSet);
+ // Pre process tableConfig from first partition to fetch virtual key info
+ Option<HoodieVirtualKeyInfo> hoodieVirtualKeyInfo = Option.empty();
+ if (partitionSet.size() > 0) {
+ hoodieVirtualKeyInfo = getHoodieVirtualKeyInfo(partitionsToMetaClient.get(partitionSet.iterator().next()));
+ }
+ Option<HoodieVirtualKeyInfo> finalHoodieVirtualKeyInfo = hoodieVirtualKeyInfo;
+ fileSplitList.stream().forEach(s -> {
+ // deal with incremental query.
+ try {
+ if (s instanceof BaseFileWithLogsSplit) {
+ BaseFileWithLogsSplit bs = (BaseFileWithLogsSplit)s;
+ if (bs.getBelongToIncrementalSplit()) {
+ rtSplits.add(new HoodieRealtimeFileSplit(bs, bs.getBasePath(), bs.getDeltaLogPaths(), bs.getMaxCommitTime(), finalHoodieVirtualKeyInfo));
+ }
+ } else if (s instanceof RealtimeBootstrapBaseFileSplit) {
+ rtSplits.add(s);
+ }
+ } catch (IOException e) {
+ throw new HoodieIOException("Error creating hoodie real time split ", e);
+ }
+ });
+ LOG.info("Returning a total splits of " + rtSplits.size());
+ return rtSplits.toArray(new InputSplit[0]);
+ }
+
+ public static Option<HoodieVirtualKeyInfo> getHoodieVirtualKeyInfo(HoodieTableMetaClient metaClient) {
+ HoodieTableConfig tableConfig = metaClient.getTableConfig();
+ if (!tableConfig.populateMetaFields()) {
+ TableSchemaResolver tableSchemaResolver = new TableSchemaResolver(metaClient);
+ try {
+ MessageType parquetSchema = tableSchemaResolver.getTableParquetSchema();
+ return Option.of(new HoodieVirtualKeyInfo(tableConfig.getRecordKeyFieldProp(),
+ tableConfig.getPartitionFieldProp(), parquetSchema.getFieldIndex(tableConfig.getRecordKeyFieldProp()),
+ parquetSchema.getFieldIndex(tableConfig.getPartitionFieldProp())));
+ } catch (Exception exception) {
+ throw new HoodieException("Fetching table schema failed with exception ", exception);
+ }
+ }
+ return Option.empty();
+ }
+
+ public static boolean isIncrementalQuerySplits(List<FileSplit> fileSplits) {
+ if (fileSplits == null || fileSplits.size() == 0) {
+ return false;
+ }
+ return fileSplits.stream().anyMatch(s -> {
+ if (s instanceof BaseFileWithLogsSplit) {
+ BaseFileWithLogsSplit bs = (BaseFileWithLogsSplit)s;
+ return bs.getBelongToIncrementalSplit();
+ } else {
+ return s instanceof RealtimeBootstrapBaseFileSplit;
+ }
+ });
+ }
+
+ public static RealtimeBootstrapBaseFileSplit createRealimeBootstrapBaseFileSplit(
Review comment:
ok
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieParquetRealtimeInputFormat.java
##########
@@ -61,9 +85,183 @@
@Override
public InputSplit[] getSplits(JobConf job, int numSplits) throws IOException {
- Stream<FileSplit> fileSplits = Arrays.stream(super.getSplits(job, numSplits)).map(is -> (FileSplit) is);
+ List<FileSplit> fileSplits = Arrays.stream(super.getSplits(job, numSplits)).map(is -> (FileSplit) is).collect(Collectors.toList());
+
+ boolean isIncrementalSplits = HoodieRealtimeInputFormatUtils.isIncrementalQuerySplits(fileSplits);
+
+ return isIncrementalSplits ? HoodieRealtimeInputFormatUtils.getIncrementalRealtimeSplits(job, fileSplits.stream()) : HoodieRealtimeInputFormatUtils.getRealtimeSplits(job, fileSplits.stream());
+ }
+
+ /**
+ * Keep the logic of mor_incr_view as same as spark datasource.
+ * Step1: Get list of commits to be fetched based on start commit and max commits(for snapshot max commits is -1).
+ * Step2: Get list of affected files status for these affected file status.
+ * Step3: Construct HoodieTableFileSystemView based on those affected file status.
+ * a. Filter affected partitions based on inputPaths.
+ * b. Get list of fileGroups based on affected partitions by fsView.getAllFileGroups.
+ * Step4: Set input paths based on filtered affected partition paths. changes that amony original input paths passed to
+ * this method. some partitions did not have commits as part of the trimmed down list of commits and hence we need this step.
+ * Step5: Find candidate fileStatus, since when we get baseFileStatus from HoodieTableFileSystemView,
+ * the BaseFileStatus will missing file size information.
+ * We should use candidate fileStatus to update the size information for BaseFileStatus.
+ * Step6: For every file group from step3(b)
+ * Get 1st available base file from all file slices. then we use candidate file status to update the baseFileStatus,
+ * and construct RealTimeFileStatus and add it to result along with log files.
+ * If file group just has log files, construct RealTimeFileStatus and add it to result.
+ * TODO: unify the incremental view code between hive/spark-sql and spark datasource
+ */
+ @Override
+ protected List<FileStatus> listStatusForIncrementalMode(
+ JobConf job, HoodieTableMetaClient tableMetaClient, List<Path> inputPaths) throws IOException {
+ List<FileStatus> result = new ArrayList<>();
+ String tableName = tableMetaClient.getTableConfig().getTableName();
+ Job jobContext = Job.getInstance(job);
+
+ // step1
+ Option<HoodieTimeline> timeline = HoodieInputFormatUtils.getFilteredCommitsTimeline(jobContext, tableMetaClient);
+ if (!timeline.isPresent()) {
+ return result;
+ }
+ String lastIncrementalTs = HoodieHiveUtils.readStartCommitTime(jobContext, tableName);
+ // Total number of commits to return in this batch. Set this to -1 to get all the commits.
+ Integer maxCommits = HoodieHiveUtils.readMaxCommits(jobContext, tableName);
Review comment:
good suggest， thanks
##########
File path: hudi-hadoop-mr/src/test/java/org/apache/hudi/hadoop/realtime/TestHoodieRealtimeRecordReader.java
##########
@@ -535,4 +548,211 @@ public void testSchemaEvolutionAndRollbackBlockInLastLogFile(ExternalSpillableMa
arguments(ExternalSpillableMap.DiskMapType.ROCKS_DB, true, true)
);
}
+
+ @Test
+ public void testIncremetalWithOnlylog() throws Exception {
+ // initial commit
+ Schema schema = HoodieAvroUtils.addMetadataFields(SchemaTestUtil.getEvolvedSchema());
+ HoodieTestUtils.init(hadoopConf, basePath.toString(), HoodieTableType.MERGE_ON_READ);
+ String instantTime = "100";
+ final int numRecords = 1000;
+ File partitionDir = InputFormatTestUtil.prepareParquetTable(basePath, schema, 1, numRecords, instantTime,
+ HoodieTableType.MERGE_ON_READ);
+ //FileCreateUtils.createDeltaCommit(basePath.toString(), instantTime);
+ createDeltaCommitFile(basePath, instantTime,"2016/05/01", "2016/05/01/fileid0_1-0-1_100.parquet", "fileid0");
+ // Add the paths
+ FileInputFormat.setInputPaths(baseJobConf, partitionDir.getPath());
+
+ // insert new records to log file
+ try {
+ String newCommitTime = "102";
+ HoodieLogFormat.Writer writer =
+ InputFormatTestUtil.writeDataBlockToLogFile(partitionDir, fs, schema, "fileid0", instantTime, newCommitTime,
+ numRecords, numRecords, 0);
+ writer.close();
+ createDeltaCommitFile(basePath, newCommitTime,"2016/05/01", "2016/05/01/.fileid0_100.log.1_1-0-1", "fileid0");
+
+ InputFormatTestUtil.setupIncremental(baseJobConf, "101", 1);
+
+ HoodieParquetRealtimeInputFormat inputFormat = new HoodieParquetRealtimeInputFormat();
+ inputFormat.setConf(baseJobConf);
+ InputSplit[] splits = inputFormat.getSplits(baseJobConf, 1);
+ assertTrue(splits.length == 1);
+ JobConf newJobConf = new JobConf(baseJobConf);
+ List<Schema.Field> fields = schema.getFields();
+ setHiveColumnNameProps(fields, newJobConf, false);
+ RecordReader<NullWritable, ArrayWritable> reader = inputFormat.getRecordReader(splits[0], newJobConf, Reporter.NULL);
+ // use reader to read log file.
+ NullWritable key = reader.createKey();
+ ArrayWritable value = reader.createValue();
+ while (reader.next(key, value)) {
+ Writable[] values = value.get();
+ // check if the record written is with latest commit, here "101"
+ assertEquals(newCommitTime, values[0].toString());
+ key = reader.createKey();
+ value = reader.createValue();
+ }
+ reader.close();
+ } catch (IOException e) {
+ throw new HoodieException(e.getMessage(), e);
+ }
+ }
+
+ @Test
+ public void testIncremetalWithReplace() throws Exception {
Review comment:
yes will add test case for compaction，
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieParquetRealtimeInputFormat.java
##########
@@ -61,9 +85,183 @@
@Override
public InputSplit[] getSplits(JobConf job, int numSplits) throws IOException {
- Stream<FileSplit> fileSplits = Arrays.stream(super.getSplits(job, numSplits)).map(is -> (FileSplit) is);
+ List<FileSplit> fileSplits = Arrays.stream(super.getSplits(job, numSplits)).map(is -> (FileSplit) is).collect(Collectors.toList());
+
+ boolean isIncrementalSplits = HoodieRealtimeInputFormatUtils.isIncrementalQuerySplits(fileSplits);
+
+ return isIncrementalSplits ? HoodieRealtimeInputFormatUtils.getIncrementalRealtimeSplits(job, fileSplits.stream()) : HoodieRealtimeInputFormatUtils.getRealtimeSplits(job, fileSplits.stream());
+ }
+
+ /**
+ * Keep the logic of mor_incr_view as same as spark datasource.
+ * Step1: Get list of commits to be fetched based on start commit and max commits(for snapshot max commits is -1).
+ * Step2: Get list of affected files status for these affected file status.
+ * Step3: Construct HoodieTableFileSystemView based on those affected file status.
+ * a. Filter affected partitions based on inputPaths.
+ * b. Get list of fileGroups based on affected partitions by fsView.getAllFileGroups.
+ * Step4: Set input paths based on filtered affected partition paths. changes that amony original input paths passed to
+ * this method. some partitions did not have commits as part of the trimmed down list of commits and hence we need this step.
+ * Step5: Find candidate fileStatus, since when we get baseFileStatus from HoodieTableFileSystemView,
+ * the BaseFileStatus will missing file size information.
+ * We should use candidate fileStatus to update the size information for BaseFileStatus.
+ * Step6: For every file group from step3(b)
+ * Get 1st available base file from all file slices. then we use candidate file status to update the baseFileStatus,
+ * and construct RealTimeFileStatus and add it to result along with log files.
+ * If file group just has log files, construct RealTimeFileStatus and add it to result.
+ * TODO: unify the incremental view code between hive/spark-sql and spark datasource
+ */
+ @Override
+ protected List<FileStatus> listStatusForIncrementalMode(
+ JobConf job, HoodieTableMetaClient tableMetaClient, List<Path> inputPaths) throws IOException {
+ List<FileStatus> result = new ArrayList<>();
+ String tableName = tableMetaClient.getTableConfig().getTableName();
+ Job jobContext = Job.getInstance(job);
+
+ // step1
+ Option<HoodieTimeline> timeline = HoodieInputFormatUtils.getFilteredCommitsTimeline(jobContext, tableMetaClient);
+ if (!timeline.isPresent()) {
+ return result;
+ }
+ String lastIncrementalTs = HoodieHiveUtils.readStartCommitTime(jobContext, tableName);
+ // Total number of commits to return in this batch. Set this to -1 to get all the commits.
+ Integer maxCommits = HoodieHiveUtils.readMaxCommits(jobContext, tableName);
+ HoodieTimeline commitsTimelineToReturn = timeline.get().findInstantsAfter(lastIncrementalTs, maxCommits);
+ Option<List<HoodieInstant>> commitsToCheck = Option.of(commitsTimelineToReturn.getInstants().collect(Collectors.toList()));
+ if (!commitsToCheck.isPresent()) {
+ return result;
+ }
+ // step2
+ commitsToCheck.get().sort(HoodieInstant::compareTo);
+ List<HoodieCommitMetadata> metadataList = commitsToCheck
+ .get().stream().map(instant -> {
+ try {
+ return HoodieInputFormatUtils.getCommitMetadata(instant, commitsTimelineToReturn);
+ } catch (IOException e) {
+ throw new HoodieException(String.format("cannot get metadata for instant: %s", instant));
+ }
+ }).collect(Collectors.toList());
+
+ // build fileGroup from fsView
+ List<FileStatus> affectedFileStatus = Arrays.asList(HoodieInputFormatUtils
+ .listAffectedFilesForCommits(new Path(tableMetaClient.getBasePath()), metadataList));
+ // step3
+ HoodieTableFileSystemView fsView = new HoodieTableFileSystemView(tableMetaClient, commitsTimelineToReturn, affectedFileStatus.toArray(new FileStatus[0]));
+ // build fileGroup from fsView
+ Path basePath = new Path(tableMetaClient.getBasePath());
+ // filter affectedPartition by inputPaths
+ List<String> affectedPartition = HoodieInputFormatUtils.getWritePartitionPaths(metadataList).stream()
+ .filter(k -> k.isEmpty() ? inputPaths.contains(basePath) : inputPaths.contains(new Path(basePath, k))).collect(Collectors.toList());
+ if (affectedPartition.isEmpty()) {
+ return result;
+ }
+ List<HoodieFileGroup> fileGroups = affectedPartition.stream()
Review comment:
L148 has already added all files in HoodieTableFileSystemView， and partitionToFileGroupsMap will be filled and cached。
L157 take the read lock， but it can load fileGroups from partitionToFileGroupsMap directly no need fs rpc call， i thinks It won't cost much to read the lock。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-961589519
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
}, {
"hash" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325",
"triggerID" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"triggerType" : "PUSH"
}, {
"hash" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2343",
"triggerID" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"triggerType" : "PUSH"
}, {
"hash" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2424",
"triggerID" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2427",
"triggerID" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2429",
"triggerID" : "927749497",
"triggerType" : "MANUAL"
}, {
"hash" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2431",
"triggerID" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"triggerType" : "PUSH"
}, {
"hash" : "29e6acb65044597e35f9d499385dad940580d9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2432",
"triggerID" : "29e6acb65044597e35f9d499385dad940580d9df",
"triggerType" : "PUSH"
}, {
"hash" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2441",
"triggerID" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"triggerType" : "PUSH"
}, {
"hash" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2443",
"triggerID" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2447",
"triggerID" : "928890183",
"triggerType" : "MANUAL"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2470",
"triggerID" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"triggerType" : "PUSH"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2560",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562",
"triggerID" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"triggerType" : "PUSH"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
}, {
"hash" : "6fa031e1dcf6e21dfb24cf4296cb00d910b0a204",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2589",
"triggerID" : "6fa031e1dcf6e21dfb24cf4296cb00d910b0a204",
"triggerType" : "PUSH"
}, {
"hash" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2593",
"triggerID" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"triggerType" : "PUSH"
}, {
"hash" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2604",
"triggerID" : "941865271",
"triggerType" : "MANUAL"
}, {
"hash" : "6d67b68e19a43f8668e5773d27ca9c33a8de0a37",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2642",
"triggerID" : "6d67b68e19a43f8668e5773d27ca9c33a8de0a37",
"triggerType" : "PUSH"
}, {
"hash" : "1a80559bd98829552acffdaf20d3ea0384d1d936",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2645",
"triggerID" : "1a80559bd98829552acffdaf20d3ea0384d1d936",
"triggerType" : "PUSH"
}, {
"hash" : "c4af04dab3dab31ef05ba6007000738a2dfb81ce",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2649",
"triggerID" : "c4af04dab3dab31ef05ba6007000738a2dfb81ce",
"triggerType" : "PUSH"
}, {
"hash" : "369c5d6f1f4b6eed3c86f838fb6148d127cc6703",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2660",
"triggerID" : "369c5d6f1f4b6eed3c86f838fb6148d127cc6703",
"triggerType" : "PUSH"
}, {
"hash" : "369c5d6f1f4b6eed3c86f838fb6148d127cc6703",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2898",
"triggerID" : "952859446",
"triggerType" : "MANUAL"
}, {
"hash" : "196a40e3273e98f2317856a55de3bd1888e9f754",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2899",
"triggerID" : "196a40e3273e98f2317856a55de3bd1888e9f754",
"triggerType" : "PUSH"
}, {
"hash" : "196a40e3273e98f2317856a55de3bd1888e9f754",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2899",
"triggerID" : "952859446",
"triggerType" : "MANUAL"
}, {
"hash" : "633b7370155185dead726ddfafd773e284ed510c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2935",
"triggerID" : "633b7370155185dead726ddfafd773e284ed510c",
"triggerType" : "PUSH"
}, {
"hash" : "2182f542acd4ae3d1fd239e6441e9761eb2d33a1",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3069",
"triggerID" : "2182f542acd4ae3d1fd239e6441e9761eb2d33a1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0fa6297ce58eb877fd5c4eba59fef20ad9335d26 UNKNOWN
* 2182f542acd4ae3d1fd239e6441e9761eb2d33a1 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3069)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-963790451
@codope Will #3630 fix this? this pr cannot solve this problem. let's open a new jira to track this issue. and try to resolve it in other pr。
@codope @danny0405
Since the above question has nothing to do with this PR, Does this PR still need to be verified before merging
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
}, {
"hash" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325",
"triggerID" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"triggerType" : "PUSH"
}, {
"hash" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2343",
"triggerID" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"triggerType" : "PUSH"
}, {
"hash" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2424",
"triggerID" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2427",
"triggerID" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2429",
"triggerID" : "927749497",
"triggerType" : "MANUAL"
}, {
"hash" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2431",
"triggerID" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"triggerType" : "PUSH"
}, {
"hash" : "29e6acb65044597e35f9d499385dad940580d9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2432",
"triggerID" : "29e6acb65044597e35f9d499385dad940580d9df",
"triggerType" : "PUSH"
}, {
"hash" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2441",
"triggerID" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"triggerType" : "PUSH"
}, {
"hash" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2443",
"triggerID" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2447",
"triggerID" : "928890183",
"triggerType" : "MANUAL"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2470",
"triggerID" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"triggerType" : "PUSH"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2560",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562",
"triggerID" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"triggerType" : "PUSH"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
}, {
"hash" : "6fa031e1dcf6e21dfb24cf4296cb00d910b0a204",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6fa031e1dcf6e21dfb24cf4296cb00d910b0a204",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0fa6297ce58eb877fd5c4eba59fef20ad9335d26 UNKNOWN
* 6eebb1a711e3655061d471c2a96b54e205dac630 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562)
* 6fa031e1dcf6e21dfb24cf4296cb00d910b0a204 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-943163714
@danny0405 @nsivabalan thanks you very much for your patience to review those code。 already rebase the code and addressed all comments
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
}, {
"hash" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325",
"triggerID" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"triggerType" : "PUSH"
}, {
"hash" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2343",
"triggerID" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"triggerType" : "PUSH"
}, {
"hash" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2424",
"triggerID" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2427",
"triggerID" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2429",
"triggerID" : "927749497",
"triggerType" : "MANUAL"
}, {
"hash" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2431",
"triggerID" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"triggerType" : "PUSH"
}, {
"hash" : "29e6acb65044597e35f9d499385dad940580d9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2432",
"triggerID" : "29e6acb65044597e35f9d499385dad940580d9df",
"triggerType" : "PUSH"
}, {
"hash" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2441",
"triggerID" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"triggerType" : "PUSH"
}, {
"hash" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2443",
"triggerID" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2447",
"triggerID" : "928890183",
"triggerType" : "MANUAL"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2470",
"triggerID" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"triggerType" : "PUSH"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2560",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562",
"triggerID" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"triggerType" : "PUSH"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
}, {
"hash" : "6fa031e1dcf6e21dfb24cf4296cb00d910b0a204",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2589",
"triggerID" : "6fa031e1dcf6e21dfb24cf4296cb00d910b0a204",
"triggerType" : "PUSH"
}, {
"hash" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2593",
"triggerID" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"triggerType" : "PUSH"
}, {
"hash" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2604",
"triggerID" : "941865271",
"triggerType" : "MANUAL"
}, {
"hash" : "6d67b68e19a43f8668e5773d27ca9c33a8de0a37",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2642",
"triggerID" : "6d67b68e19a43f8668e5773d27ca9c33a8de0a37",
"triggerType" : "PUSH"
}, {
"hash" : "1a80559bd98829552acffdaf20d3ea0384d1d936",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1a80559bd98829552acffdaf20d3ea0384d1d936",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0fa6297ce58eb877fd5c4eba59fef20ad9335d26 UNKNOWN
* 6d67b68e19a43f8668e5773d27ca9c33a8de0a37 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2642)
* 1a80559bd98829552acffdaf20d3ea0384d1d936 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
}, {
"hash" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325",
"triggerID" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"triggerType" : "PUSH"
}, {
"hash" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2343",
"triggerID" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"triggerType" : "PUSH"
}, {
"hash" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2424",
"triggerID" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2427",
"triggerID" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2429",
"triggerID" : "927749497",
"triggerType" : "MANUAL"
}, {
"hash" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2431",
"triggerID" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"triggerType" : "PUSH"
}, {
"hash" : "29e6acb65044597e35f9d499385dad940580d9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2432",
"triggerID" : "29e6acb65044597e35f9d499385dad940580d9df",
"triggerType" : "PUSH"
}, {
"hash" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2441",
"triggerID" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"triggerType" : "PUSH"
}, {
"hash" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2443",
"triggerID" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2447",
"triggerID" : "928890183",
"triggerType" : "MANUAL"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2470",
"triggerID" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"triggerType" : "PUSH"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2560",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562",
"triggerID" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"triggerType" : "PUSH"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
}, {
"hash" : "6fa031e1dcf6e21dfb24cf4296cb00d910b0a204",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2589",
"triggerID" : "6fa031e1dcf6e21dfb24cf4296cb00d910b0a204",
"triggerType" : "PUSH"
}, {
"hash" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2593",
"triggerID" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"triggerType" : "PUSH"
}, {
"hash" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2604",
"triggerID" : "941865271",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 0fa6297ce58eb877fd5c4eba59fef20ad9335d26 UNKNOWN
* b5649459e5e8a5ebf5a140418c16951294a29689 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2593) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2604)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao edited a comment on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-927820027
@danny0405 i revert this pr: [HUDI-1969] Support reading logs for MOR Hive rt table (#3033), since this pr introduce a critical bug which may cause the query result repeat。
give a example: now we have a base file(file1) which contains two record: key1, p1
key2, p2
then we update key2, the log file(logfile1) produced and contains record: key2, p3
when hive/presto query the table: file1 has been split into two file parts, one is file1_part1, which contains all the record; the other is file2_part2 wich contains no record. notice those two file parts will be bound to logfile1 and produce two realtimesplit。 then the log file will be read twice， repeat result produced。
The above is a very simple example， in fact query engine split one parquet file into small parts is very common。
This pr also covers the [HUDI-1969] 。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
}, {
"hash" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325",
"triggerID" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"triggerType" : "PUSH"
}, {
"hash" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2343",
"triggerID" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"triggerType" : "PUSH"
}, {
"hash" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2424",
"triggerID" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2427",
"triggerID" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2429",
"triggerID" : "927749497",
"triggerType" : "MANUAL"
}, {
"hash" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2431",
"triggerID" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"triggerType" : "PUSH"
}, {
"hash" : "29e6acb65044597e35f9d499385dad940580d9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2432",
"triggerID" : "29e6acb65044597e35f9d499385dad940580d9df",
"triggerType" : "PUSH"
}, {
"hash" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2441",
"triggerID" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"triggerType" : "PUSH"
}, {
"hash" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2443",
"triggerID" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2447",
"triggerID" : "928890183",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 0fa6297ce58eb877fd5c4eba59fef20ad9335d26 UNKNOWN
* 423f047e00904b17735a5f0120959de87e4cb0a3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2443) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2447)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
}, {
"hash" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325",
"triggerID" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"triggerType" : "PUSH"
}, {
"hash" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2343",
"triggerID" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"triggerType" : "PUSH"
}, {
"hash" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2424",
"triggerID" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2427",
"triggerID" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2429",
"triggerID" : "927749497",
"triggerType" : "MANUAL"
}, {
"hash" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2431",
"triggerID" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"triggerType" : "PUSH"
}, {
"hash" : "29e6acb65044597e35f9d499385dad940580d9df",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2432",
"triggerID" : "29e6acb65044597e35f9d499385dad940580d9df",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 29e6acb65044597e35f9d499385dad940580d9df Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2432)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
}, {
"hash" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325",
"triggerID" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"triggerType" : "PUSH"
}, {
"hash" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2343",
"triggerID" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"triggerType" : "PUSH"
}, {
"hash" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2424",
"triggerID" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2427",
"triggerID" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2429",
"triggerID" : "927749497",
"triggerType" : "MANUAL"
}, {
"hash" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2431",
"triggerID" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"triggerType" : "PUSH"
}, {
"hash" : "29e6acb65044597e35f9d499385dad940580d9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2432",
"triggerID" : "29e6acb65044597e35f9d499385dad940580d9df",
"triggerType" : "PUSH"
}, {
"hash" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2441",
"triggerID" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"triggerType" : "PUSH"
}, {
"hash" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2443",
"triggerID" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2447",
"triggerID" : "928890183",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 0fa6297ce58eb877fd5c4eba59fef20ad9335d26 UNKNOWN
* 423f047e00904b17735a5f0120959de87e4cb0a3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2443) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2447)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-941865271
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
}, {
"hash" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325",
"triggerID" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"triggerType" : "PUSH"
}, {
"hash" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2343",
"triggerID" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"triggerType" : "PUSH"
}, {
"hash" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2424",
"triggerID" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2427",
"triggerID" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2429",
"triggerID" : "927749497",
"triggerType" : "MANUAL"
}, {
"hash" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2431",
"triggerID" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 814b7eb1e147197c29cc2701a08b6bc77bf6c2f8 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2427) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2429)
* 40b81794cbd4d6e7b7356b42ece3290b0904e987 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2431)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-931167837
@danny0405 @leesf already update the code。 could you pls review again
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
danny0405 commented on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-929775609
Changes the title and commit message to "[HUDI-2086] Redo the logic of mor incremental view for hive"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a change in pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
danny0405 commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r725756894
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieRealtimeInputFormatUtils.java
##########
@@ -99,7 +101,32 @@
}
}
Option<HoodieVirtualKeyInfo> finalHoodieVirtualKeyInfo = hoodieVirtualKeyInfo;
- partitionsToParquetSplits.keySet().forEach(partitionPath -> {
+ // deal with incremental query
+ candidateFileSplits.stream().forEach(s -> {
+ try {
Review comment:
Yes, that i exactly what i mean.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
}, {
"hash" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325",
"triggerID" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"triggerType" : "PUSH"
}, {
"hash" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2343",
"triggerID" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"triggerType" : "PUSH"
}, {
"hash" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2424",
"triggerID" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2427",
"triggerID" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2429",
"triggerID" : "927749497",
"triggerType" : "MANUAL"
}, {
"hash" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2431",
"triggerID" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"triggerType" : "PUSH"
}, {
"hash" : "29e6acb65044597e35f9d499385dad940580d9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2432",
"triggerID" : "29e6acb65044597e35f9d499385dad940580d9df",
"triggerType" : "PUSH"
}, {
"hash" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2441",
"triggerID" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"triggerType" : "PUSH"
}, {
"hash" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2443",
"triggerID" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2447",
"triggerID" : "928890183",
"triggerType" : "MANUAL"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2470",
"triggerID" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"triggerType" : "PUSH"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"triggerType" : "PUSH"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2560",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 0fa6297ce58eb877fd5c4eba59fef20ad9335d26 UNKNOWN
* c14bded2e903e9c55d86fddc77d40effccca5e01 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2470) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2560)
* 6eebb1a711e3655061d471c2a96b54e205dac630 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r725737982
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieParquetRealtimeInputFormat.java
##########
@@ -66,6 +91,139 @@
return HoodieRealtimeInputFormatUtils.getRealtimeSplits(job, fileSplits);
}
+ /**
+ * keep the logical of mor_incr_view as same as spark datasource.
+ * to do: unify the incremental view code between hive/spark-sql and spark datasource
+ */
+ @Override
+ protected List<FileStatus> listStatusForIncrementalMode(
+ JobConf job, HoodieTableMetaClient tableMetaClient, List<Path> inputPaths) throws IOException {
+ List<FileStatus> result = new ArrayList<>();
+ String tableName = tableMetaClient.getTableConfig().getTableName();
+ Job jobContext = Job.getInstance(job);
+
+ Option<HoodieTimeline> timeline = HoodieInputFormatUtils.getFilteredCommitsTimeline(jobContext, tableMetaClient);
+ if (!timeline.isPresent()) {
+ return result;
+ }
+ String lastIncrementalTs = HoodieHiveUtils.readStartCommitTime(jobContext, tableName);
+ // Total number of commits to return in this batch. Set this to -1 to get all the commits.
+ Integer maxCommits = HoodieHiveUtils.readMaxCommits(jobContext, tableName);
+ HoodieTimeline commitsTimelineToReturn = timeline.get().findInstantsAfter(lastIncrementalTs, maxCommits);
+ Option<List<HoodieInstant>> commitsToCheck = Option.of(commitsTimelineToReturn.getInstants().collect(Collectors.toList()));
+ if (!commitsToCheck.isPresent()) {
+ return result;
+ }
+ Map<String, HashMap<String, FileStatus>> partitionsWithFileStatus = HoodieInputFormatUtils
+ .listAffectedFilesForCommits(new Path(tableMetaClient.getBasePath()), commitsToCheck.get(), commitsTimelineToReturn);
+ // build fileGroup from fsView
+ List<FileStatus> affectedFileStatus = new ArrayList<>();
+ partitionsWithFileStatus.forEach((key, value) -> value.forEach((k, v) -> affectedFileStatus.add(v)));
+ HoodieTableFileSystemView fsView = new HoodieTableFileSystemView(tableMetaClient, commitsTimelineToReturn, affectedFileStatus.toArray(new FileStatus[0]));
+ // build fileGroup from fsView
+ String basePath = tableMetaClient.getBasePath();
+ // filter affectedPartition by inputPaths
+ List<String> affectedPartition = partitionsWithFileStatus.keySet().stream()
+ .filter(k -> k.isEmpty() ? inputPaths.contains(new Path(basePath)) : inputPaths.contains(new Path(basePath, k))).collect(Collectors.toList());
Review comment:
yes, we apply this pr in our product env, and find that bug.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r728743555
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieParquetRealtimeInputFormat.java
##########
@@ -119,6 +307,11 @@ void addProjectionToJobConf(final RealtimeSplit realtimeSplit, final JobConf job
addProjectionToJobConf(realtimeSplit, jobConf);
LOG.info("Creating record reader with readCols :" + jobConf.get(ColumnProjectionUtils.READ_COLUMN_NAMES_CONF_STR)
+ ", Ids :" + jobConf.get(ColumnProjectionUtils.READ_COLUMN_IDS_CONF_STR));
+
+ // for log only split, we no need parquet reader, set it to empty
+ if (FSUtils.isLogFile(realtimeSplit.getPath())) {
Review comment:
agree
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieRealtimeInputFormatUtils.java
##########
@@ -69,17 +70,18 @@
private static final Logger LOG = LogManager.getLogger(HoodieRealtimeInputFormatUtils.class);
public static InputSplit[] getRealtimeSplits(Configuration conf, Stream<FileSplit> fileSplits) throws IOException {
- Map<Path, List<FileSplit>> partitionsToParquetSplits =
- fileSplits.collect(Collectors.groupingBy(split -> split.getPath().getParent()));
+ // for all unique split parents, obtain all delta files based on delta commit timeline,
+ // grouped on file id
+ List<InputSplit> rtSplits = new ArrayList<>();
+ List<FileSplit> candidateFileSplits = fileSplits.collect(Collectors.toList());
+ Map<Path, List<FileSplit>> partitionsToParquetSplits = candidateFileSplits
Review comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
}, {
"hash" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325",
"triggerID" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"triggerType" : "PUSH"
}, {
"hash" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2343",
"triggerID" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"triggerType" : "PUSH"
}, {
"hash" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2424",
"triggerID" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2427",
"triggerID" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2429",
"triggerID" : "927749497",
"triggerType" : "MANUAL"
}, {
"hash" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2431",
"triggerID" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"triggerType" : "PUSH"
}, {
"hash" : "29e6acb65044597e35f9d499385dad940580d9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2432",
"triggerID" : "29e6acb65044597e35f9d499385dad940580d9df",
"triggerType" : "PUSH"
}, {
"hash" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2441",
"triggerID" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"triggerType" : "PUSH"
}, {
"hash" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2443",
"triggerID" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2447",
"triggerID" : "928890183",
"triggerType" : "MANUAL"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2470",
"triggerID" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"triggerType" : "PUSH"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2560",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562",
"triggerID" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"triggerType" : "PUSH"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
}, {
"hash" : "6fa031e1dcf6e21dfb24cf4296cb00d910b0a204",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2589",
"triggerID" : "6fa031e1dcf6e21dfb24cf4296cb00d910b0a204",
"triggerType" : "PUSH"
}, {
"hash" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2593",
"triggerID" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"triggerType" : "PUSH"
}, {
"hash" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2604",
"triggerID" : "941865271",
"triggerType" : "MANUAL"
}, {
"hash" : "6d67b68e19a43f8668e5773d27ca9c33a8de0a37",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2642",
"triggerID" : "6d67b68e19a43f8668e5773d27ca9c33a8de0a37",
"triggerType" : "PUSH"
}, {
"hash" : "1a80559bd98829552acffdaf20d3ea0384d1d936",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2645",
"triggerID" : "1a80559bd98829552acffdaf20d3ea0384d1d936",
"triggerType" : "PUSH"
}, {
"hash" : "c4af04dab3dab31ef05ba6007000738a2dfb81ce",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c4af04dab3dab31ef05ba6007000738a2dfb81ce",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0fa6297ce58eb877fd5c4eba59fef20ad9335d26 UNKNOWN
* 1a80559bd98829552acffdaf20d3ea0384d1d936 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2645)
* c4af04dab3dab31ef05ba6007000738a2dfb81ce UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a change in pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
danny0405 commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r728830538
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieRealtimeInputFormatUtils.java
##########
@@ -161,6 +162,51 @@
return rtSplits.toArray(new InputSplit[0]);
}
+ // pick all incremental files and add them to rtSplits then filter out those files.
+ private static Map<Path, List<FileSplit>> filterOutIncrementalSplits(
+ List<FileSplit> fileSplitList,
+ List<InputSplit> rtSplits,
+ final Option<HoodieVirtualKeyInfo> finalHoodieVirtualKeyInfo) {
+ return fileSplitList.stream().filter(s -> {
+ // deal with incremental query.
+ try {
+ if (s instanceof BaseFileWithLogsSplit) {
+ BaseFileWithLogsSplit bs = (BaseFileWithLogsSplit)s;
+ if (bs.getBelongToIncrementalSplit()) {
+ rtSplits.add(new HoodieRealtimeFileSplit(bs, bs.getBasePath(), bs.getDeltaLogPaths(), bs.getMaxCommitTime(), finalHoodieVirtualKeyInfo));
+ }
+ } else if (s instanceof RealtimeBootstrapBaseFileSplit) {
+ rtSplits.add(s);
+ }
+ } catch (IOException e) {
+ throw new HoodieIOException("Error creating hoodie real time split ", e);
+ }
+ // filter the snapshot split.
+ if (s instanceof RealtimeBootstrapBaseFileSplit) {
+ return false;
+ } else if ((s instanceof BaseFileWithLogsSplit) && ((BaseFileWithLogsSplit) s).getBelongToIncrementalSplit()) {
Review comment:
Why not just return early, i have pasted the code. And Why we need to handle the incremental query first, can we handle them together with snapshot query ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r725743160
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieParquetRealtimeInputFormat.java
##########
@@ -66,6 +91,139 @@
return HoodieRealtimeInputFormatUtils.getRealtimeSplits(job, fileSplits);
}
+ /**
+ * keep the logical of mor_incr_view as same as spark datasource.
+ * to do: unify the incremental view code between hive/spark-sql and spark datasource
+ */
+ @Override
+ protected List<FileStatus> listStatusForIncrementalMode(
+ JobConf job, HoodieTableMetaClient tableMetaClient, List<Path> inputPaths) throws IOException {
+ List<FileStatus> result = new ArrayList<>();
+ String tableName = tableMetaClient.getTableConfig().getTableName();
+ Job jobContext = Job.getInstance(job);
+
+ Option<HoodieTimeline> timeline = HoodieInputFormatUtils.getFilteredCommitsTimeline(jobContext, tableMetaClient);
+ if (!timeline.isPresent()) {
+ return result;
+ }
+ String lastIncrementalTs = HoodieHiveUtils.readStartCommitTime(jobContext, tableName);
+ // Total number of commits to return in this batch. Set this to -1 to get all the commits.
+ Integer maxCommits = HoodieHiveUtils.readMaxCommits(jobContext, tableName);
+ HoodieTimeline commitsTimelineToReturn = timeline.get().findInstantsAfter(lastIncrementalTs, maxCommits);
+ Option<List<HoodieInstant>> commitsToCheck = Option.of(commitsTimelineToReturn.getInstants().collect(Collectors.toList()));
+ if (!commitsToCheck.isPresent()) {
+ return result;
+ }
+ Map<String, HashMap<String, FileStatus>> partitionsWithFileStatus = HoodieInputFormatUtils
+ .listAffectedFilesForCommits(new Path(tableMetaClient.getBasePath()), commitsToCheck.get(), commitsTimelineToReturn);
+ // build fileGroup from fsView
+ List<FileStatus> affectedFileStatus = new ArrayList<>();
+ partitionsWithFileStatus.forEach((key, value) -> value.forEach((k, v) -> affectedFileStatus.add(v)));
+ HoodieTableFileSystemView fsView = new HoodieTableFileSystemView(tableMetaClient, commitsTimelineToReturn, affectedFileStatus.toArray(new FileStatus[0]));
+ // build fileGroup from fsView
+ String basePath = tableMetaClient.getBasePath();
+ // filter affectedPartition by inputPaths
+ List<String> affectedPartition = partitionsWithFileStatus.keySet().stream()
+ .filter(k -> k.isEmpty() ? inputPaths.contains(new Path(basePath)) : inputPaths.contains(new Path(basePath, k))).collect(Collectors.toList());
+ if (affectedPartition.isEmpty()) {
+ return result;
+ }
+ List<HoodieFileGroup> fileGroups = affectedPartition.stream()
+ .flatMap(partitionPath -> fsView.getAllFileGroups(partitionPath)).collect(Collectors.toList());
+ setInputPaths(job, affectedPartition.stream()
+ .map(p -> p.isEmpty() ? basePath : new Path(basePath, p).toUri().toString()).collect(Collectors.joining(",")));
+
+ // find all file status in current partitionPath
+ FileStatus[] fileStatuses = getStatus(job);
+ Map<String, FileStatus> candidateFileStatus = new HashMap<>();
+ for (int i = 0; i < fileStatuses.length; i++) {
+ String key = fileStatuses[i].getPath().toString();
+ candidateFileStatus.put(key, fileStatuses[i]);
+ }
+
+ String maxCommitTime = fsView.getLastInstant().get().getTimestamp();
Review comment:
good suggest, i will split this method. thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r725744147
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/RealtimeCompactedRecordReader.java
##########
@@ -52,16 +49,19 @@
protected final RecordReader<NullWritable, ArrayWritable> parquetReader;
private final Map<String, HoodieRecord<? extends HoodieRecordPayload>> deltaRecordMap;
- private final Set<String> deltaRecordKeys;
private int recordKeyIndex = HoodieInputFormatUtils.HOODIE_RECORD_KEY_COL_POS;
- private Iterator<String> deltaItr;
+ private Option<RealTimeMergedRecordReader> logReader;
public RealtimeCompactedRecordReader(RealtimeSplit split, JobConf job,
RecordReader<NullWritable, ArrayWritable> realReader) throws IOException {
super(split, job);
this.parquetReader = realReader;
- this.deltaRecordMap = getMergedLogRecordScanner().getRecords();
- this.deltaRecordKeys = new HashSet<>(this.deltaRecordMap.keySet());
+ HoodieMergedLogRecordScanner hoodieMergedLogRecordScanner = getMergedLogRecordScanner();
+ this.deltaRecordMap = hoodieMergedLogRecordScanner.getRecords();
+ this.logReader = Option.empty();
+ if (FSUtils.isLogFile(split.getPath())) {
Review comment:
yes, A RealtimeSplit can be any of the 3 right. but we only add the processing logic to deal with case c ； since case a and case b will be handled correctly by original processing logic。
only case c ， FSutils.isLogFile will return true. case a and case b use the original processing logic
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
}, {
"hash" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325",
"triggerID" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"triggerType" : "PUSH"
}, {
"hash" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2343",
"triggerID" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"triggerType" : "PUSH"
}, {
"hash" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2424",
"triggerID" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2427",
"triggerID" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2429",
"triggerID" : "927749497",
"triggerType" : "MANUAL"
}, {
"hash" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2431",
"triggerID" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"triggerType" : "PUSH"
}, {
"hash" : "29e6acb65044597e35f9d499385dad940580d9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2432",
"triggerID" : "29e6acb65044597e35f9d499385dad940580d9df",
"triggerType" : "PUSH"
}, {
"hash" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2441",
"triggerID" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"triggerType" : "PUSH"
}, {
"hash" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2443",
"triggerID" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2447",
"triggerID" : "928890183",
"triggerType" : "MANUAL"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2470",
"triggerID" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"triggerType" : "PUSH"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2560",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562",
"triggerID" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"triggerType" : "PUSH"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 0fa6297ce58eb877fd5c4eba59fef20ad9335d26 UNKNOWN
* 6eebb1a711e3655061d471c2a96b54e205dac630 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
}, {
"hash" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325",
"triggerID" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"triggerType" : "PUSH"
}, {
"hash" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2343",
"triggerID" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"triggerType" : "PUSH"
}, {
"hash" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2424",
"triggerID" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2427",
"triggerID" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2429",
"triggerID" : "927749497",
"triggerType" : "MANUAL"
}, {
"hash" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2431",
"triggerID" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"triggerType" : "PUSH"
}, {
"hash" : "29e6acb65044597e35f9d499385dad940580d9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2432",
"triggerID" : "29e6acb65044597e35f9d499385dad940580d9df",
"triggerType" : "PUSH"
}, {
"hash" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2441",
"triggerID" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"triggerType" : "PUSH"
}, {
"hash" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 158bfbfd6991b583b3b708922459fb2c0eaf4f62 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2441)
* 0fa6297ce58eb877fd5c4eba59fef20ad9335d26 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
}, {
"hash" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325",
"triggerID" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"triggerType" : "PUSH"
}, {
"hash" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2343",
"triggerID" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"triggerType" : "PUSH"
}, {
"hash" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2424",
"triggerID" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2427",
"triggerID" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2429",
"triggerID" : "927749497",
"triggerType" : "MANUAL"
}, {
"hash" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2431",
"triggerID" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"triggerType" : "PUSH"
}, {
"hash" : "29e6acb65044597e35f9d499385dad940580d9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2432",
"triggerID" : "29e6acb65044597e35f9d499385dad940580d9df",
"triggerType" : "PUSH"
}, {
"hash" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2441",
"triggerID" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"triggerType" : "PUSH"
}, {
"hash" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2443",
"triggerID" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2447",
"triggerID" : "928890183",
"triggerType" : "MANUAL"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2470",
"triggerID" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0fa6297ce58eb877fd5c4eba59fef20ad9335d26 UNKNOWN
* c14bded2e903e9c55d86fddc77d40effccca5e01 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2470)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r718194228
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieParquetRealtimeInputFormat.java
##########
@@ -119,6 +275,11 @@ void addProjectionToJobConf(final RealtimeSplit realtimeSplit, final JobConf job
addProjectionToJobConf(realtimeSplit, jobConf);
LOG.info("Creating record reader with readCols :" + jobConf.get(ColumnProjectionUtils.READ_COLUMN_NAMES_CONF_STR)
+ ", Ids :" + jobConf.get(ColumnProjectionUtils.READ_COLUMN_IDS_CONF_STR));
+
+ // for log only split, we no need parquet reader, set it to empty
+ if (realtimeSplit.getPath().toString().contains(".log.")) {
+ return new HoodieRealtimeRecordReader(realtimeSplit, jobConf, new HoodieEmptyRecordReader());
+ }
Review comment:
agree
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/RealtimeCompactedRecordReader.java
##########
@@ -55,13 +55,20 @@
private final Set<String> deltaRecordKeys;
private int recordKeyIndex = HoodieInputFormatUtils.HOODIE_RECORD_KEY_COL_POS;
private Iterator<String> deltaItr;
+ private boolean logFileOnlySplit;
+ private HoodieMergedLogReader logReader;
public RealtimeCompactedRecordReader(RealtimeSplit split, JobConf job,
RecordReader<NullWritable, ArrayWritable> realReader) throws IOException {
super(split, job);
this.parquetReader = realReader;
- this.deltaRecordMap = getMergedLogRecordScanner().getRecords();
+ HoodieMergedLogRecordScanner hoodieMergedLogRecordScanner = getMergedLogRecordScanner();
+ this.deltaRecordMap = hoodieMergedLogRecordScanner.getRecords();
this.deltaRecordKeys = new HashSet<>(this.deltaRecordMap.keySet());
+ if (split.getPath().toString().contains(".log.")) {
+ this.logFileOnlySplit = true;
Review comment:
agree
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a change in pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
danny0405 commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r733479600
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieRealtimeInputFormatUtils.java
##########
@@ -161,6 +162,46 @@
return rtSplits.toArray(new InputSplit[0]);
}
+ // pick all incremental files and add them to rtSplits then filter out those files.
+ private static Map<Path, List<FileSplit>> filterOutIncrementalSplits(
+ List<FileSplit> fileSplitList,
Review comment:
> now we want to query the incremental data of commit2 (file2 will be picked as inputSplit);
What is the start commit time of the incremental query ? If the start time is commit2, i guess the incremental commit is not commit2(or file2) ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-872092745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
}, {
"hash" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325",
"triggerID" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"triggerType" : "PUSH"
}, {
"hash" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2343",
"triggerID" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"triggerType" : "PUSH"
}, {
"hash" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2424",
"triggerID" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2427",
"triggerID" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2429",
"triggerID" : "927749497",
"triggerType" : "MANUAL"
}, {
"hash" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2431",
"triggerID" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"triggerType" : "PUSH"
}, {
"hash" : "29e6acb65044597e35f9d499385dad940580d9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2432",
"triggerID" : "29e6acb65044597e35f9d499385dad940580d9df",
"triggerType" : "PUSH"
}, {
"hash" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2441",
"triggerID" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"triggerType" : "PUSH"
}, {
"hash" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2443",
"triggerID" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2447",
"triggerID" : "928890183",
"triggerType" : "MANUAL"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2470",
"triggerID" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"triggerType" : "PUSH"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2560",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562",
"triggerID" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"triggerType" : "PUSH"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
}, {
"hash" : "6fa031e1dcf6e21dfb24cf4296cb00d910b0a204",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2589",
"triggerID" : "6fa031e1dcf6e21dfb24cf4296cb00d910b0a204",
"triggerType" : "PUSH"
}, {
"hash" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2593",
"triggerID" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"triggerType" : "PUSH"
}, {
"hash" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2604",
"triggerID" : "941865271",
"triggerType" : "MANUAL"
}, {
"hash" : "6d67b68e19a43f8668e5773d27ca9c33a8de0a37",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2642",
"triggerID" : "6d67b68e19a43f8668e5773d27ca9c33a8de0a37",
"triggerType" : "PUSH"
}, {
"hash" : "1a80559bd98829552acffdaf20d3ea0384d1d936",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2645",
"triggerID" : "1a80559bd98829552acffdaf20d3ea0384d1d936",
"triggerType" : "PUSH"
}, {
"hash" : "c4af04dab3dab31ef05ba6007000738a2dfb81ce",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2649",
"triggerID" : "c4af04dab3dab31ef05ba6007000738a2dfb81ce",
"triggerType" : "PUSH"
}, {
"hash" : "369c5d6f1f4b6eed3c86f838fb6148d127cc6703",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2660",
"triggerID" : "369c5d6f1f4b6eed3c86f838fb6148d127cc6703",
"triggerType" : "PUSH"
}, {
"hash" : "369c5d6f1f4b6eed3c86f838fb6148d127cc6703",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2898",
"triggerID" : "952859446",
"triggerType" : "MANUAL"
}, {
"hash" : "196a40e3273e98f2317856a55de3bd1888e9f754",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2899",
"triggerID" : "196a40e3273e98f2317856a55de3bd1888e9f754",
"triggerType" : "PUSH"
}, {
"hash" : "196a40e3273e98f2317856a55de3bd1888e9f754",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2899",
"triggerID" : "952859446",
"triggerType" : "MANUAL"
}, {
"hash" : "633b7370155185dead726ddfafd773e284ed510c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2935",
"triggerID" : "633b7370155185dead726ddfafd773e284ed510c",
"triggerType" : "PUSH"
}, {
"hash" : "2182f542acd4ae3d1fd239e6441e9761eb2d33a1",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3069",
"triggerID" : "2182f542acd4ae3d1fd239e6441e9761eb2d33a1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0fa6297ce58eb877fd5c4eba59fef20ad9335d26 UNKNOWN
* 2182f542acd4ae3d1fd239e6441e9761eb2d33a1 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3069)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-961589519
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=603",
"triggerID" : "5d49bd4e8c638ffb9ced102dc6771d6291c199e3",
"triggerType" : "PUSH"
}, {
"hash" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=631",
"triggerID" : "792ae1d9884eac5eb7004e4edb617ea6e86d79b7",
"triggerType" : "PUSH"
}, {
"hash" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2325",
"triggerID" : "42735e78f24bd3a5f622026627d6ff8b95708705",
"triggerType" : "PUSH"
}, {
"hash" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2343",
"triggerID" : "c470b4c2c6a4d1d33a5facf729971b4331fe0771",
"triggerType" : "PUSH"
}, {
"hash" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2424",
"triggerID" : "679a0104259b01a85be2def1b5cd396ae4f0e858",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2427",
"triggerID" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"triggerType" : "PUSH"
}, {
"hash" : "814b7eb1e147197c29cc2701a08b6bc77bf6c2f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2429",
"triggerID" : "927749497",
"triggerType" : "MANUAL"
}, {
"hash" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2431",
"triggerID" : "40b81794cbd4d6e7b7356b42ece3290b0904e987",
"triggerType" : "PUSH"
}, {
"hash" : "29e6acb65044597e35f9d499385dad940580d9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2432",
"triggerID" : "29e6acb65044597e35f9d499385dad940580d9df",
"triggerType" : "PUSH"
}, {
"hash" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2441",
"triggerID" : "158bfbfd6991b583b3b708922459fb2c0eaf4f62",
"triggerType" : "PUSH"
}, {
"hash" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fa6297ce58eb877fd5c4eba59fef20ad9335d26",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2443",
"triggerID" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"triggerType" : "PUSH"
}, {
"hash" : "423f047e00904b17735a5f0120959de87e4cb0a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2447",
"triggerID" : "928890183",
"triggerType" : "MANUAL"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2470",
"triggerID" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"triggerType" : "PUSH"
}, {
"hash" : "c14bded2e903e9c55d86fddc77d40effccca5e01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2560",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562",
"triggerID" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"triggerType" : "PUSH"
}, {
"hash" : "6eebb1a711e3655061d471c2a96b54e205dac630",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2562",
"triggerID" : "939217992",
"triggerType" : "MANUAL"
}, {
"hash" : "6fa031e1dcf6e21dfb24cf4296cb00d910b0a204",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2589",
"triggerID" : "6fa031e1dcf6e21dfb24cf4296cb00d910b0a204",
"triggerType" : "PUSH"
}, {
"hash" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2593",
"triggerID" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"triggerType" : "PUSH"
}, {
"hash" : "b5649459e5e8a5ebf5a140418c16951294a29689",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2604",
"triggerID" : "941865271",
"triggerType" : "MANUAL"
}, {
"hash" : "6d67b68e19a43f8668e5773d27ca9c33a8de0a37",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2642",
"triggerID" : "6d67b68e19a43f8668e5773d27ca9c33a8de0a37",
"triggerType" : "PUSH"
}, {
"hash" : "1a80559bd98829552acffdaf20d3ea0384d1d936",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2645",
"triggerID" : "1a80559bd98829552acffdaf20d3ea0384d1d936",
"triggerType" : "PUSH"
}, {
"hash" : "c4af04dab3dab31ef05ba6007000738a2dfb81ce",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2649",
"triggerID" : "c4af04dab3dab31ef05ba6007000738a2dfb81ce",
"triggerType" : "PUSH"
}, {
"hash" : "369c5d6f1f4b6eed3c86f838fb6148d127cc6703",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2660",
"triggerID" : "369c5d6f1f4b6eed3c86f838fb6148d127cc6703",
"triggerType" : "PUSH"
}, {
"hash" : "369c5d6f1f4b6eed3c86f838fb6148d127cc6703",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2898",
"triggerID" : "952859446",
"triggerType" : "MANUAL"
}, {
"hash" : "196a40e3273e98f2317856a55de3bd1888e9f754",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2899",
"triggerID" : "196a40e3273e98f2317856a55de3bd1888e9f754",
"triggerType" : "PUSH"
}, {
"hash" : "196a40e3273e98f2317856a55de3bd1888e9f754",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2899",
"triggerID" : "952859446",
"triggerType" : "MANUAL"
}, {
"hash" : "633b7370155185dead726ddfafd773e284ed510c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2935",
"triggerID" : "633b7370155185dead726ddfafd773e284ed510c",
"triggerType" : "PUSH"
}, {
"hash" : "2182f542acd4ae3d1fd239e6441e9761eb2d33a1",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3069",
"triggerID" : "2182f542acd4ae3d1fd239e6441e9761eb2d33a1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0fa6297ce58eb877fd5c4eba59fef20ad9335d26 UNKNOWN
* 2182f542acd4ae3d1fd239e6441e9761eb2d33a1 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3069)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r740696192
##########
File path: hudi-hadoop-mr/src/test/java/org/apache/hudi/hadoop/realtime/TestHoodieRealtimeRecordReader.java
##########
@@ -535,4 +548,211 @@ public void testSchemaEvolutionAndRollbackBlockInLastLogFile(ExternalSpillableMa
arguments(ExternalSpillableMap.DiskMapType.ROCKS_DB, true, true)
);
}
+
+ @Test
+ public void testIncremetalWithOnlylog() throws Exception {
+ // initial commit
+ Schema schema = HoodieAvroUtils.addMetadataFields(SchemaTestUtil.getEvolvedSchema());
+ HoodieTestUtils.init(hadoopConf, basePath.toString(), HoodieTableType.MERGE_ON_READ);
+ String instantTime = "100";
+ final int numRecords = 1000;
+ File partitionDir = InputFormatTestUtil.prepareParquetTable(basePath, schema, 1, numRecords, instantTime,
+ HoodieTableType.MERGE_ON_READ);
+ //FileCreateUtils.createDeltaCommit(basePath.toString(), instantTime);
+ createDeltaCommitFile(basePath, instantTime,"2016/05/01", "2016/05/01/fileid0_1-0-1_100.parquet", "fileid0");
+ // Add the paths
+ FileInputFormat.setInputPaths(baseJobConf, partitionDir.getPath());
+
+ // insert new records to log file
+ try {
+ String newCommitTime = "102";
+ HoodieLogFormat.Writer writer =
+ InputFormatTestUtil.writeDataBlockToLogFile(partitionDir, fs, schema, "fileid0", instantTime, newCommitTime,
+ numRecords, numRecords, 0);
+ writer.close();
+ createDeltaCommitFile(basePath, newCommitTime,"2016/05/01", "2016/05/01/.fileid0_100.log.1_1-0-1", "fileid0");
+
+ InputFormatTestUtil.setupIncremental(baseJobConf, "101", 1);
+
+ HoodieParquetRealtimeInputFormat inputFormat = new HoodieParquetRealtimeInputFormat();
+ inputFormat.setConf(baseJobConf);
+ InputSplit[] splits = inputFormat.getSplits(baseJobConf, 1);
+ assertTrue(splits.length == 1);
+ JobConf newJobConf = new JobConf(baseJobConf);
+ List<Schema.Field> fields = schema.getFields();
+ setHiveColumnNameProps(fields, newJobConf, false);
+ RecordReader<NullWritable, ArrayWritable> reader = inputFormat.getRecordReader(splits[0], newJobConf, Reporter.NULL);
+ // use reader to read log file.
+ NullWritable key = reader.createKey();
+ ArrayWritable value = reader.createValue();
+ while (reader.next(key, value)) {
+ Writable[] values = value.get();
+ // check if the record written is with latest commit, here "101"
+ assertEquals(newCommitTime, values[0].toString());
+ key = reader.createKey();
+ value = reader.createValue();
+ }
+ reader.close();
+ } catch (IOException e) {
+ throw new HoodieException(e.getMessage(), e);
+ }
+ }
+
+ @Test
+ public void testIncremetalWithReplace() throws Exception {
Review comment:
yes will add test case for compaction，
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
danny0405 commented on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-961732455
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r725418146
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieMergedLogReader.java
##########
@@ -0,0 +1,144 @@
+/*
Review comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3203: [HUDI-2086] Redo the logical of mor_incremental_view for hive
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r725418164
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieMergedLogReader.java
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop.realtime;
+
+import java.io.IOException;
+import java.text.MessageFormat;
+import java.util.Iterator;
+
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.io.ArrayWritable;
+import org.apache.hadoop.io.NullWritable;
+import org.apache.hadoop.io.Writable;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.RecordReader;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.hadoop.utils.HoodieRealtimeRecordReaderUtils;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+/**
+ * Record Reader implementation to read avro data, to support inc queries.
+ */
+public class HoodieMergedLogReader extends AbstractRealtimeRecordReader
+ implements RecordReader<NullWritable, ArrayWritable> {
+ private static final Logger LOG = LogManager.getLogger(AbstractRealtimeRecordReader.class);
+ private final HoodieMergedLogRecordScanner logRecordScanner;
+ private final Iterator<HoodieRecord<? extends HoodieRecordPayload>> logRecordsKeyIterator;
+ private ArrayWritable valueObj;
+
+ private int end;
+ private int offset;
+
+ public HoodieMergedLogReader(RealtimeSplit split, JobConf job, HoodieMergedLogRecordScanner logRecordScanner) {
+ super(split, job);
+ this.logRecordScanner = logRecordScanner;
+ this.end = logRecordScanner.getRecords().size();
+ this.logRecordsKeyIterator = logRecordScanner.iterator();
+ this.valueObj = new ArrayWritable(Writable.class, new Writable[getHiveSchema().getFields().size()]);
+ }
+
+ private Option buildGenericRecord(HoodieRecord record) throws IOException {
+ if (usesCustomPayload) {
+ return record.getData().getInsertValue(getWriterSchema());
+ } else {
+ return record.getData().getInsertValue(getReaderSchema());
+ }
+ }
+
+ @Override
+ public boolean next(NullWritable key, ArrayWritable arrayWritable) throws IOException {
+ if (!logRecordsKeyIterator.hasNext()) {
Review comment:
agree
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao edited a comment on pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao edited a comment on pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#issuecomment-940624808
@codope yes, it will be ok after compact.
but consider that case: read incremental datas which before replacecommit, origin read logical cannot resolve this problem.
this pr also solve some hidden bugs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3203: [HUDI-2086] Refactor hive mor_incremental_view
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3203:
URL: https://github.com/apache/hudi/pull/3203#discussion_r726802648
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/BaseFileWithLogsSplit.java
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop;
+
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.apache.hadoop.mapred.FileSplit;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+public class BaseFileWithLogsSplit extends FileSplit {
Review comment:
agree
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org