You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/11/02 05:32:02 UTC
[GitHub] [hudi] trushev opened a new pull request, #7113: [HUDI-5147] Flink data skipping doesn't work when HepPlanner calls copy()…
trushev opened a new pull request, #7113:
URL: https://github.com/apache/hudi/pull/7113
… on HoodieTableSource
### Change Logs
When `HepPlanner` applies `PushFilterIntoSourceScanRuleBase` it copyies `HoodieTableSource` which leads to the loss of `List<ResolvedExpression> filters` in `FileIndex`.
This PR fixes adds `filters` to `FileIndex` when `HoodieTableSource` is copied.
### Impact
Flink data skipping feature works fine now
### Risk level (write none, low medium or high below)
Low because `read.data.skipping.enabled` is false by default
### Documentation Update
No need
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7113: [HUDI-5147] Flink data skipping doesn't work when HepPlanner calls copy()…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7113:
URL: https://github.com/apache/hudi/pull/7113#issuecomment-1301532033
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6f7af524ee4693252495a4ef100bc940b44b3599",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12718",
"triggerID" : "6f7af524ee4693252495a4ef100bc940b44b3599",
"triggerType" : "PUSH"
}, {
"hash" : "806ea1bd76b6212f7859fd3e33c34b8daf7526a1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12727",
"triggerID" : "806ea1bd76b6212f7859fd3e33c34b8daf7526a1",
"triggerType" : "PUSH"
}, {
"hash" : "aeabc3fa358e221807c6fdf50dc4d9da7e6c2b65",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12731",
"triggerID" : "aeabc3fa358e221807c6fdf50dc4d9da7e6c2b65",
"triggerType" : "PUSH"
}, {
"hash" : "7f6c9592fbe4213d9cfff5515dc77364d9a97a3d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7f6c9592fbe4213d9cfff5515dc77364d9a97a3d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aeabc3fa358e221807c6fdf50dc4d9da7e6c2b65 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12731)
* 7f6c9592fbe4213d9cfff5515dc77364d9a97a3d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] trushev commented on a diff in pull request #7113: [HUDI-5147] Flink data skipping doesn't work when HepPlanner calls copy()…
Posted by GitBox <gi...@apache.org>.
trushev commented on code in PR #7113:
URL: https://github.com/apache/hudi/pull/7113#discussion_r1011713031
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/FileIndex.java:
##########
@@ -80,6 +80,12 @@ public static FileIndex instance(Path path, Configuration conf, RowType rowType)
return new FileIndex(path, conf, rowType);
}
+ public static FileIndex instance(Path path, Configuration conf, RowType rowType, List<ResolvedExpression> filters) {
+ FileIndex fileIndex = instance(path, conf, rowType);
Review Comment:
Thanks, you are right
- Added `FileIndex` param in the constructor
- Replaced `this.filters` with local variable in `applyFilters(filters)`
- Removed field `List<ResolvedExpression> filters` as no need anymore
- Reverted `instance(path, conf, rowType, filters)` as no need anymore
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] trushev commented on a diff in pull request #7113: [HUDI-5147] Flink data skipping doesn't work when HepPlanner calls copy()…
Posted by GitBox <gi...@apache.org>.
trushev commented on code in PR #7113:
URL: https://github.com/apache/hudi/pull/7113#discussion_r1011312520
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/FileIndex.java:
##########
@@ -80,6 +80,12 @@ public static FileIndex instance(Path path, Configuration conf, RowType rowType)
return new FileIndex(path, conf, rowType);
}
+ public static FileIndex instance(Path path, Configuration conf, RowType rowType, List<ResolvedExpression> filters) {
+ FileIndex fileIndex = instance(path, conf, rowType);
Review Comment:
It is a really good question
Quick answer -- yes we should keep because the method is used in `IncrementalInputSplits` without filter at all.
But why there is no a filter in `IncrementalInputSplits`. Looks like `FileIndex` is useless in this way. I mean data skipping feature skips nothing
Mb we should pass filters from `HoodieTableSource` here:
```java
case FlinkOptions.QUERY_TYPE_INCREMENTAL:
IncrementalInputSplits incrementalInputSplits = IncrementalInputSplits.builder()
.conf(conf)
.path(FilePathUtils.toFlinkPath(path))
.rowType(this.tableRowType)
.maxCompactionMemoryInBytes(maxCompactionMemoryInBytes)
.requiredPartitions(getRequiredPartitionPaths()).build();
```
I think it is not part of this fix related to `copy()` by `HepPlanner`
WDYT about the separate ticket/PR for this problem if the one exists?:)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7113: [HUDI-5147] Flink data skipping doesn't work when HepPlanner calls copy()…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7113:
URL: https://github.com/apache/hudi/pull/7113#issuecomment-1299919028
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6f7af524ee4693252495a4ef100bc940b44b3599",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12718",
"triggerID" : "6f7af524ee4693252495a4ef100bc940b44b3599",
"triggerType" : "PUSH"
}, {
"hash" : "806ea1bd76b6212f7859fd3e33c34b8daf7526a1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "806ea1bd76b6212f7859fd3e33c34b8daf7526a1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6f7af524ee4693252495a4ef100bc940b44b3599 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12718)
* 806ea1bd76b6212f7859fd3e33c34b8daf7526a1 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7113: [HUDI-5147] Flink data skipping doesn't work when HepPlanner calls copy()…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7113:
URL: https://github.com/apache/hudi/pull/7113#issuecomment-1299926852
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6f7af524ee4693252495a4ef100bc940b44b3599",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12718",
"triggerID" : "6f7af524ee4693252495a4ef100bc940b44b3599",
"triggerType" : "PUSH"
}, {
"hash" : "806ea1bd76b6212f7859fd3e33c34b8daf7526a1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12727",
"triggerID" : "806ea1bd76b6212f7859fd3e33c34b8daf7526a1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6f7af524ee4693252495a4ef100bc940b44b3599 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12718)
* 806ea1bd76b6212f7859fd3e33c34b8daf7526a1 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12727)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #7113: [HUDI-5147] Flink data skipping doesn't work when HepPlanner calls copy()…
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #7113:
URL: https://github.com/apache/hudi/pull/7113#discussion_r1011470662
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/FileIndex.java:
##########
@@ -80,6 +80,12 @@ public static FileIndex instance(Path path, Configuration conf, RowType rowType)
return new FileIndex(path, conf, rowType);
}
+ public static FileIndex instance(Path path, Configuration conf, RowType rowType, List<ResolvedExpression> filters) {
+ FileIndex fileIndex = instance(path, conf, rowType);
Review Comment:
Agree, we can fix it in another PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7113: [HUDI-5147] Flink data skipping doesn't work when HepPlanner calls copy()…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7113:
URL: https://github.com/apache/hudi/pull/7113#issuecomment-1300383803
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6f7af524ee4693252495a4ef100bc940b44b3599",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12718",
"triggerID" : "6f7af524ee4693252495a4ef100bc940b44b3599",
"triggerType" : "PUSH"
}, {
"hash" : "806ea1bd76b6212f7859fd3e33c34b8daf7526a1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12727",
"triggerID" : "806ea1bd76b6212f7859fd3e33c34b8daf7526a1",
"triggerType" : "PUSH"
}, {
"hash" : "aeabc3fa358e221807c6fdf50dc4d9da7e6c2b65",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aeabc3fa358e221807c6fdf50dc4d9da7e6c2b65",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6f7af524ee4693252495a4ef100bc940b44b3599 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12718)
* 806ea1bd76b6212f7859fd3e33c34b8daf7526a1 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12727)
* aeabc3fa358e221807c6fdf50dc4d9da7e6c2b65 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7113: [HUDI-5147] Flink data skipping doesn't work when HepPlanner calls copy()…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7113:
URL: https://github.com/apache/hudi/pull/7113#issuecomment-1301601675
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6f7af524ee4693252495a4ef100bc940b44b3599",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12718",
"triggerID" : "6f7af524ee4693252495a4ef100bc940b44b3599",
"triggerType" : "PUSH"
}, {
"hash" : "806ea1bd76b6212f7859fd3e33c34b8daf7526a1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12727",
"triggerID" : "806ea1bd76b6212f7859fd3e33c34b8daf7526a1",
"triggerType" : "PUSH"
}, {
"hash" : "aeabc3fa358e221807c6fdf50dc4d9da7e6c2b65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12731",
"triggerID" : "aeabc3fa358e221807c6fdf50dc4d9da7e6c2b65",
"triggerType" : "PUSH"
}, {
"hash" : "7f6c9592fbe4213d9cfff5515dc77364d9a97a3d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12740",
"triggerID" : "7f6c9592fbe4213d9cfff5515dc77364d9a97a3d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7f6c9592fbe4213d9cfff5515dc77364d9a97a3d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12740)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 merged pull request #7113: [HUDI-5147] Flink data skipping doesn't work when HepPlanner calls copy()…
Posted by GitBox <gi...@apache.org>.
danny0405 merged PR #7113:
URL: https://github.com/apache/hudi/pull/7113
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7113: [HUDI-5147] Flink data skipping doesn't work when HepPlanner calls copy()…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7113:
URL: https://github.com/apache/hudi/pull/7113#issuecomment-1299644745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6f7af524ee4693252495a4ef100bc940b44b3599",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6f7af524ee4693252495a4ef100bc940b44b3599",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6f7af524ee4693252495a4ef100bc940b44b3599 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #7113: [HUDI-5147] Flink data skipping doesn't work when HepPlanner calls copy()…
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #7113:
URL: https://github.com/apache/hudi/pull/7113#discussion_r1011289748
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/FileIndex.java:
##########
@@ -80,6 +80,12 @@ public static FileIndex instance(Path path, Configuration conf, RowType rowType)
return new FileIndex(path, conf, rowType);
}
+ public static FileIndex instance(Path path, Configuration conf, RowType rowType, List<ResolvedExpression> filters) {
+ FileIndex fileIndex = instance(path, conf, rowType);
Review Comment:
Nice catch ~ Should we also keep this method: `instance(Path path, Configuration conf, RowType rowType)` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #7113: [HUDI-5147] Flink data skipping doesn't work when HepPlanner calls copy()…
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #7113:
URL: https://github.com/apache/hudi/pull/7113#discussion_r1011488715
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/FileIndex.java:
##########
@@ -80,6 +80,12 @@ public static FileIndex instance(Path path, Configuration conf, RowType rowType)
return new FileIndex(path, conf, rowType);
}
+ public static FileIndex instance(Path path, Configuration conf, RowType rowType, List<ResolvedExpression> filters) {
+ FileIndex fileIndex = instance(path, conf, rowType);
Review Comment:
And another solution is we add a `FileIndex` param in the constructor so that there is no need to re-construct it again and again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] trushev commented on a diff in pull request #7113: [HUDI-5147] Flink data skipping doesn't work when HepPlanner calls copy()…
Posted by GitBox <gi...@apache.org>.
trushev commented on code in PR #7113:
URL: https://github.com/apache/hudi/pull/7113#discussion_r1011312520
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/FileIndex.java:
##########
@@ -80,6 +80,12 @@ public static FileIndex instance(Path path, Configuration conf, RowType rowType)
return new FileIndex(path, conf, rowType);
}
+ public static FileIndex instance(Path path, Configuration conf, RowType rowType, List<ResolvedExpression> filters) {
+ FileIndex fileIndex = instance(path, conf, rowType);
Review Comment:
It is a really good question
Quick answer -- yes we should keep because the method is used in `IncrementalInputSplits` without filter at all.
But why there is no a filter in `IncrementalInputSplits`. Looks like `FileIndex` useless in this way. I mean data skipping feature skips nothing
Mb we should pass filters from `HoodieTableSource` here:
```java
case FlinkOptions.QUERY_TYPE_INCREMENTAL:
IncrementalInputSplits incrementalInputSplits = IncrementalInputSplits.builder()
.conf(conf)
.path(FilePathUtils.toFlinkPath(path))
.rowType(this.tableRowType)
.maxCompactionMemoryInBytes(maxCompactionMemoryInBytes)
.requiredPartitions(getRequiredPartitionPaths()).build();
```
I think it is not part of this fix related to `copy()` by `HepPlanner`
WDYT about the separate ticket/PR for this problem if the one exists?:)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7113: [HUDI-5147] Flink data skipping doesn't work when HepPlanner calls copy()…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7113:
URL: https://github.com/apache/hudi/pull/7113#issuecomment-1300391359
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6f7af524ee4693252495a4ef100bc940b44b3599",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12718",
"triggerID" : "6f7af524ee4693252495a4ef100bc940b44b3599",
"triggerType" : "PUSH"
}, {
"hash" : "806ea1bd76b6212f7859fd3e33c34b8daf7526a1",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12727",
"triggerID" : "806ea1bd76b6212f7859fd3e33c34b8daf7526a1",
"triggerType" : "PUSH"
}, {
"hash" : "aeabc3fa358e221807c6fdf50dc4d9da7e6c2b65",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12731",
"triggerID" : "aeabc3fa358e221807c6fdf50dc4d9da7e6c2b65",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 806ea1bd76b6212f7859fd3e33c34b8daf7526a1 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12727)
* aeabc3fa358e221807c6fdf50dc4d9da7e6c2b65 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12731)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7113: [HUDI-5147] Flink data skipping doesn't work when HepPlanner calls copy()…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7113:
URL: https://github.com/apache/hudi/pull/7113#issuecomment-1301534166
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6f7af524ee4693252495a4ef100bc940b44b3599",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12718",
"triggerID" : "6f7af524ee4693252495a4ef100bc940b44b3599",
"triggerType" : "PUSH"
}, {
"hash" : "806ea1bd76b6212f7859fd3e33c34b8daf7526a1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12727",
"triggerID" : "806ea1bd76b6212f7859fd3e33c34b8daf7526a1",
"triggerType" : "PUSH"
}, {
"hash" : "aeabc3fa358e221807c6fdf50dc4d9da7e6c2b65",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12731",
"triggerID" : "aeabc3fa358e221807c6fdf50dc4d9da7e6c2b65",
"triggerType" : "PUSH"
}, {
"hash" : "7f6c9592fbe4213d9cfff5515dc77364d9a97a3d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12740",
"triggerID" : "7f6c9592fbe4213d9cfff5515dc77364d9a97a3d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aeabc3fa358e221807c6fdf50dc4d9da7e6c2b65 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12731)
* 7f6c9592fbe4213d9cfff5515dc77364d9a97a3d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12740)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7113: [HUDI-5147] Flink data skipping doesn't work when HepPlanner calls copy()…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7113:
URL: https://github.com/apache/hudi/pull/7113#issuecomment-1299649595
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6f7af524ee4693252495a4ef100bc940b44b3599",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12718",
"triggerID" : "6f7af524ee4693252495a4ef100bc940b44b3599",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6f7af524ee4693252495a4ef100bc940b44b3599 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12718)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] trushev commented on a diff in pull request #7113: [HUDI-5147] Flink data skipping doesn't work when HepPlanner calls copy()…
Posted by GitBox <gi...@apache.org>.
trushev commented on code in PR #7113:
URL: https://github.com/apache/hudi/pull/7113#discussion_r1011312520
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/FileIndex.java:
##########
@@ -80,6 +80,12 @@ public static FileIndex instance(Path path, Configuration conf, RowType rowType)
return new FileIndex(path, conf, rowType);
}
+ public static FileIndex instance(Path path, Configuration conf, RowType rowType, List<ResolvedExpression> filters) {
+ FileIndex fileIndex = instance(path, conf, rowType);
Review Comment:
Is is a really good question
Quick answer -- yes we should keep because the method is used in `IncrementalInputSplits` without filter at all.
But why there is no a filter in `IncrementalInputSplits`. Looks like `FileIndex` useless in this way. I mean data skipping feature skips nothing
Mb we should pass filters from `HoodieTableSource` here:
```java
case FlinkOptions.QUERY_TYPE_INCREMENTAL:
IncrementalInputSplits incrementalInputSplits = IncrementalInputSplits.builder()
.conf(conf)
.path(FilePathUtils.toFlinkPath(path))
.rowType(this.tableRowType)
.maxCompactionMemoryInBytes(maxCompactionMemoryInBytes)
.requiredPartitions(getRequiredPartitionPaths()).build();
```
I think it is not part of this fix related to `copy()` by `HepPlanner`
WDYT about the separate ticket/PR for this problem if the one exists?:)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7113: [HUDI-5147] Flink data skipping doesn't work when HepPlanner calls copy()…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7113:
URL: https://github.com/apache/hudi/pull/7113#issuecomment-1299900202
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6f7af524ee4693252495a4ef100bc940b44b3599",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12718",
"triggerID" : "6f7af524ee4693252495a4ef100bc940b44b3599",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6f7af524ee4693252495a4ef100bc940b44b3599 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12718)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7113: [HUDI-5147] Flink data skipping doesn't work when HepPlanner calls copy()…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7113:
URL: https://github.com/apache/hudi/pull/7113#issuecomment-1300902947
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6f7af524ee4693252495a4ef100bc940b44b3599",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12718",
"triggerID" : "6f7af524ee4693252495a4ef100bc940b44b3599",
"triggerType" : "PUSH"
}, {
"hash" : "806ea1bd76b6212f7859fd3e33c34b8daf7526a1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12727",
"triggerID" : "806ea1bd76b6212f7859fd3e33c34b8daf7526a1",
"triggerType" : "PUSH"
}, {
"hash" : "aeabc3fa358e221807c6fdf50dc4d9da7e6c2b65",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12731",
"triggerID" : "aeabc3fa358e221807c6fdf50dc4d9da7e6c2b65",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aeabc3fa358e221807c6fdf50dc4d9da7e6c2b65 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12731)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org