[jira] [Created] (DRILL-5115) Metadata Cache Pruning randomly returns wrong results at higher concurrencies

["Rahul Challapalli (JIRA)" <ji...@apache.org>]

Rahul Challapalli created DRILL-5115:
----------------------------------------

             Summary: Metadata Cache Pruning randomly returns wrong results at higher concurrencies
                 Key: DRILL-5115
                 URL: https://issues.apache.org/jira/browse/DRILL-5115
             Project: Apache Drill
          Issue Type: Bug
          Components: Metadata, Query Planning & Optimization
    Affects Versions: 1.8.0, 1.9.0, 1.10
            Reporter: Rahul Challapalli


git.commit.id.abbrev=4312d65

When multiple queries are updating the metadata cache simultaneously the below query randomly returns wrong results. 

A single run includes executing a suite of 90 tests at a concurrency of 50. I encountered a wrong data scenario in my 10th run.
 
Query :
{code}
select l_orderkey from l_3level where dir0=1 and ((dir1='one' and dir2 IN ('2015-7-12', '2015-7-13')) or (dir1='two' and dir2='2015-8-12'))
{code}

Wrong Result Plan (based on the profile) : 
{code}
00-00    Screen : rowType = RecordType(ANY l_orderkey): rowcount = 310.0, cumulative cost = {341.0 rows, 341.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 205721
00-01      Project(l_orderkey=[$0]) : rowType = RecordType(ANY l_orderkey): rowcount = 310.0, cumulative cost = {310.0 rows, 310.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 205720
00-02        Scan(groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath [path=maprfs:/drill/testdata/metadata_caching_pp/l_3level/1/one/2015-7-13/20.parquet], ReadEntryWithPath [path=maprfs:/drill/testdata/metadata_caching_pp/l_3level/1/two/2015-8-12/30.parquet], ReadEntryWithPath [path=maprfs:/drill/testdata/metadata_caching_pp/l_3level/1/one/2015-7-12/10.parquet]], selectionRoot=maprfs:/drill/testdata/metadata_caching_pp/l_3level, numFiles=3, usedMetadataFile=true, cacheFileRoot=/drill/testdata/metadata_caching_pp/l_3level, columns=[`l_orderkey`]]]) : rowType = RecordType(ANY l_orderkey): rowcount = 310.0, cumulative cost = {310.0 rows, 310.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 205719
{code}

Correct Result Plan (based on the profile):
{code}
00-00    Screen : rowType = RecordType(ANY l_orderkey): rowcount = 2.25, cumulative cost = {122.475 rows, 527.475 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 226849
00-01      Project(l_orderkey=[$3]) : rowType = RecordType(ANY l_orderkey): rowcount = 2.25, cumulative cost = {122.25 rows, 527.25 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 226848
00-02        SelectionVectorRemover : rowType = RecordType(ANY dir0, ANY dir1, ANY dir2, ANY l_orderkey): rowcount = 2.25, cumulative cost = {122.25 rows, 527.25 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 226847
00-03          Filter(condition=[AND(=($0, 1), OR(AND(=($1, 'one'), OR(=($2, '2015-7-12'), =($2, '2015-7-13'))), AND(=($1, 'two'), =($2, '2015-8-12'))))]) : rowType = RecordType(ANY dir0, ANY dir1, ANY dir2, ANY l_orderkey): rowcount = 2.25, cumulative cost = {120.0 rows, 525.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 226846
00-04            Scan(groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath [path=/drill/testdata/metadata_caching_pp/l_3level/1/one/2015-7-13/20.parquet], ReadEntryWithPath [path=/drill/testdata/metadata_caching_pp/l_3level/1/two/2015-8-12/30.parquet], ReadEntryWithPath [path=/drill/testdata/metadata_caching_pp/l_3level/1/one/2015-7-12/10.parquet]], selectionRoot=/drill/testdata/metadata_caching_pp/l_3level, numFiles=3, usedMetadataFile=true, cacheFileRoot=/drill/testdata/metadata_caching_pp/l_3level/1, columns=[`dir0`, `dir1`, `dir2`, `l_orderkey`]]]) : rowType = RecordType(ANY dir0, ANY dir1, ANY dir2, ANY l_orderkey): rowcount = 60.0, cumulative cost = {60.0 rows, 240.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 226845
{code}

I attached the data set, log files and the query profiles. Let me know if you need anything



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)