[jira] [Commented] (DRILL-7570) Fix unstable statistics tests

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/DRILL-7570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17030721#comment-17030721 ] 

ASF GitHub Bot commented on DRILL-7570:
---------------------------------------

vvysotskyi commented on pull request #1968: DRILL-7570: Fix unstable statistics tests
URL: https://github.com/apache/drill/pull/1968
 
 
   # [DRILL-7570](https://issues.apache.org/jira/browse/DRILL-7570): Fix unstable statistics tests
   
   ## Description
   Updated sample statistics tests
   
   ## Documentation
   NA
   
   ## Testing
   Checked that updated unit tests pass
   
 
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> Fix unstable statistics tests
> -----------------------------
>
>                 Key: DRILL-7570
>                 URL: https://issues.apache.org/jira/browse/DRILL-7570
>             Project: Apache Drill
>          Issue Type: Task
>    Affects Versions: 1.17.0
>            Reporter: Vova Vysotskyi
>            Assignee: Vova Vysotskyi
>            Priority: Major
>             Fix For: 1.18.0
>
>
> Drill contains tests for checking that statistics is applied, some of them also use sampling to calculate statistics value.
> Sampling adds limit above scan, but tests check the value of the estimated row count to verify that statistics were applied. limit without sorting doesn't guarantee consistent results, so these tests may fail sometime:
> {noformat}
> [ERROR]   TestMetastoreCommands.testAnalyzeWithSampleStatistics:2739 Did not find expected pattern in plan: Filter\(condition.*\).*rowcount = 96.25,
> 00-00    Screen : rowType = RecordType(ANY employee_id): rowcount = 105.0, cumulative cost = {2530.5 rows, 7570.5 cpu, 2310.0 io, 0.0 network, 0.0 memory}, id = 336738
> 00-01      Project(employee_id=[$1]) : rowType = RecordType(ANY employee_id): rowcount = 105.0, cumulative cost = {2520.0 rows, 7560.0 cpu, 2310.0 io, 0.0 network, 0.0 memory}, id = 336737
> 00-02        SelectionVectorRemover : rowType = RecordType(ANY department_id, ANY employee_id): rowcount = 105.0, cumulative cost = {2415.0 rows, 7455.0 cpu, 2310.0 io, 0.0 network, 0.0 memory}, id = 336736
> 00-03          Filter(condition=[=($0, 2)]) : rowType = RecordType(ANY department_id, ANY employee_id): rowcount = 105.0, cumulative cost = {2310.0 rows, 7350.0 cpu, 2310.0 io, 0.0 network, 0.0 memory}, id = 336735
> 00-04            Scan(table=[[dfs, tmp, employeeWithStatsFile]], groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath [path=/home/runner/work/drill/drill/exec/java-exec/target/org.apache.drill.exec.sql.TestMetastoreCommands/dfsTestTmp/1580901135676-0/employeeWithStatsFile/0_0_0.parquet]], selectionRoot=/home/runner/work/drill/drill/exec/java-exec/target/org.apache.drill.exec.sql.TestMetastoreCommands/dfsTestTmp/1580901135676-0/employeeWithStatsFile, numFiles=1, numRowGroups=1, usedMetadataFile=false, usedMetastore=true, filter=equal(`department_id`, 2) , columns=[`department_id`, `employee_id`]]]) : rowType = RecordType(ANY department_id, ANY employee_id): rowcount = 1155.0, cumulative cost = {1155.0 rows, 2310.0 cpu, 2310.0 io, 0.0 network, 0.0 memory}, id = 336734
>  expected:<true> but was:<false> {noformat}
> List of tests to fix:
>  - TestMetastoreCommands.testAnalyzeWithSampleStatistics;
>  - TestAnalyze.basic3.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)