You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/12/18 06:38:33 UTC
[GitHub] [hudi] sumihehe opened a new issue #2346: [SUPPORT]The rt view query returns a wrong result with predicate push down.
sumihehe opened a new issue #2346:
URL: https://github.com/apache/hudi/issues/2346
Hi all,
The rt view query returns a wrong result with predicate push down.
This is my query on a rt view of MOR table：
select count(1) from ***mor_rt where platform = "HYLOOP" and `_hoodie_commit_time`= '20201210213000';
| _c0 |
| -- |
0
But it do have records whose `_hoodie_commit_time` equal '20201210213000', it can be proved by the result of query:
select `_hoodie_commit_time`, count(1) from ***mor_rt where platform = "HYLOOP" group by `_hoodie_commit_time`;
_hoodie_commit_time | _c1
-- | --
20201210213000 | 16
I think it may be a bug and is caused by the wrong code logic in HoodieCombineRealtimeRecordReader .
```java
public boolean next(NullWritable key, ArrayWritable value) throws IOException {
if (this.currentRecordReader.next(key, value)) {
LOG.info("Reading from record reader");
LOG.info(HoodieRealtimeRecordReaderUtils.arrayWritableToString(value));
return true;
} else if (recordReaders.size() > 0) {
this.currentRecordReader.close();
this.currentRecordReader = recordReaders.remove(0);
return this.currentRecordReader.next(key, value); //TODO: bug? return this.next(key, value);
} else {
return false;
}
}
```
Hudi version : 0.6.1
Hive version : 2.4.3
Hadoop version : 2.7.3
Be glad to receive any help.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on issue #2346: [SUPPORT]The rt view query returns a wrong result with predicate push down.
Posted by GitBox <gi...@apache.org>.
bvaradar commented on issue #2346:
URL: https://github.com/apache/hudi/issues/2346#issuecomment-768102437
Closing this GH as we have a jira to track it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on issue #2346: [SUPPORT]The rt view query returns a wrong result with predicate push down.
Posted by GitBox <gi...@apache.org>.
bvaradar commented on issue #2346:
URL: https://github.com/apache/hudi/issues/2346#issuecomment-752988617
@satishkotha : Any update on this ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] satishkotha commented on issue #2346: [SUPPORT]The rt view query returns a wrong result with predicate push down.
Posted by GitBox <gi...@apache.org>.
satishkotha commented on issue #2346:
URL: https://github.com/apache/hudi/issues/2346#issuecomment-763163282
Thanks for that example. I filed https://issues.apache.org/jira/browse/HUDI-1539. The fix is probably straightforward as you suggested. But need to write a good test to catch this. Please feel free to open a pull request if you are interested. Otherwise, we'll get to it in the next week.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] satishkotha edited a comment on issue #2346: [SUPPORT]The rt view query returns a wrong result with predicate push down.
Posted by GitBox <gi...@apache.org>.
satishkotha edited a comment on issue #2346:
URL: https://github.com/apache/hudi/issues/2346#issuecomment-755791465
@sumihehe I'm not able to reproduce this issue using dataset in [docker demo](https://hudi.apache.org/docs/docker_demo.html)
0: jdbc:hive2://hiveserver:10000> select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_mor_rt where `_hoodie_commit_time`='20210106235525';
+----------------------+---------+----------------------+---------+------------+------------+--+
| _hoodie_commit_time | symbol | ts | volume | open | close |
+----------------------+---------+----------------------+---------+------------+------------+--+
| 20210106235525 | NIHD | 2018-08-31 10:29:00 | 2125 | 5.67 | 5.67 |
| 20210106235525 | CDMO | 2018-08-31 09:58:00 | 8577 | 7.02 | 7.07 |
| 20210106235525 | MRTX | 2018-08-31 10:28:00 | 1844 | 56.15 | 56.15 |
| 20210106235525 | NTRA | 2018-08-31 09:59:00 | 1326 | 27.45 | 27.45 |
| 20210106235525 | DRNA | 2018-08-31 09:51:00 | 334 | 16.0 | 16.0 |
| 20210106235525 | STAA | 2018-08-31 09:59:00 | 1709 | 47.95 | 47.95 |
...
197 rows selected (0.423 seconds)
Looked at code closely, I dont see anything obviously wrong. Could you help explain little more about your dataset so i can try to reproduce it?
- How many rows does it have?
- How many commits and how many deltacommits?
- Can you try the same query on **mor_ro table?
- Could you try other input format such as HoodieParquetRealtimeInputFormat. (as @bvaradar suggested)
It'd be awesome if you have a reproducible test using above docker demo or on simple spark shell
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on issue #2346: [SUPPORT]The rt view query returns a wrong result with predicate push down.
Posted by GitBox <gi...@apache.org>.
bvaradar commented on issue #2346:
URL: https://github.com/apache/hudi/issues/2346#issuecomment-749214160
@sumihehe : Sorry for missing this earlier. Can you check if you are seeing the issue when using HoodieParquetRealtimeInputFormat ?
@satishkotha : Since your production setup uses Combined Input format, can you please check if this is a bug ?
cc @n3nash
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] sumihehe commented on issue #2346: [SUPPORT]The rt view query returns a wrong result with predicate push down.
Posted by GitBox <gi...@apache.org>.
sumihehe commented on issue #2346:
URL: https://github.com/apache/hudi/issues/2346#issuecomment-758591316
> @sumihehe Did you get a chance to look at above? It'll be helpful if you can provide more information.
- Around 1w rows when the case happened
- Around 60 deltacommits and 0 commit when the case happened
- it returns correctly on the *mor_ro table
- It can not be reproduced when i made more deltacommit/commits on the table later. So it returns correctly now with HoodieParquetRealtimeInputFormat or
It think It will occur on condition that:
1. in a rt table
2. the hive query has predicate push down
3. there are no less than 3 splits (thus no less than 3 recordReaders in HoodieCombineRealtimeRecordReader), and the records satisfy the predicate are in the split which is in a relatively back position of the List
4. 2 recordReaders in succession with _this.currentRecordReader.next(key, value)_ returns false, as the predicate push down has filtered the baseFile.
In step 4, it leads to _HoodieCombineRealtimeRecordReader::next(NullWritable key, ArrayWritable value)_ return false and the reader will stop read next. So, records which satisfy the predicate are in the remanined recordReaders but can not be read.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar closed issue #2346: [SUPPORT]The rt view query returns a wrong result with predicate push down.
Posted by GitBox <gi...@apache.org>.
bvaradar closed issue #2346:
URL: https://github.com/apache/hudi/issues/2346
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] satishkotha commented on issue #2346: [SUPPORT]The rt view query returns a wrong result with predicate push down.
Posted by GitBox <gi...@apache.org>.
satishkotha commented on issue #2346:
URL: https://github.com/apache/hudi/issues/2346#issuecomment-755791465
@sumihehe I'm not able to reproduce this issue using dataset in [docker demo](https://hudi.apache.org/docs/docker_demo.html)
`0: jdbc:hive2://hiveserver:10000> select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_mor_rt where `_hoodie_commit_time`='20210106235525';
+----------------------+---------+----------------------+---------+------------+------------+--+
| _hoodie_commit_time | symbol | ts | volume | open | close |
+----------------------+---------+----------------------+---------+------------+------------+--+
| 20210106235525 | NIHD | 2018-08-31 10:29:00 | 2125 | 5.67 | 5.67 |
| 20210106235525 | CDMO | 2018-08-31 09:58:00 | 8577 | 7.02 | 7.07 |
| 20210106235525 | MRTX | 2018-08-31 10:28:00 | 1844 | 56.15 | 56.15 |
| 20210106235525 | NTRA | 2018-08-31 09:59:00 | 1326 | 27.45 | 27.45 |
| 20210106235525 | DRNA | 2018-08-31 09:51:00 | 334 | 16.0 | 16.0 |
| 20210106235525 | STAA | 2018-08-31 09:59:00 | 1709 | 47.95 | 47.95 |
...
197 rows selected (0.423 seconds)
`
Could you help explain little more about your dataset?
- How many rows does it have?
- How many commits and how many deltacommits?
- Can you try the same query on **mor_ro table?
- Could you try other input format such as HoodieParquetRealtimeInputFormat. (as @bvaradar suggested)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] satishkotha commented on issue #2346: [SUPPORT]The rt view query returns a wrong result with predicate push down.
Posted by GitBox <gi...@apache.org>.
satishkotha commented on issue #2346:
URL: https://github.com/apache/hudi/issues/2346#issuecomment-758169722
@sumihehe Did you get a chance to look at above? It'll be helpful if you can provide more information.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] satishkotha commented on issue #2346: [SUPPORT]The rt view query returns a wrong result with predicate push down.
Posted by GitBox <gi...@apache.org>.
satishkotha commented on issue #2346:
URL: https://github.com/apache/hudi/issues/2346#issuecomment-750941441
Sorry for delay. On a quick look, it does look like a bug. I'll take a closer look and fix it early next week.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org