You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@impala.apache.org by "Tim Armstrong (Jira)" <ji...@apache.org> on 2020/06/13 00:09:00 UTC
[jira] [Resolved] (IMPALA-3617) Incorrect reporting of runtime
filters in scan node
[ https://issues.apache.org/jira/browse/IMPALA-3617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tim Armstrong resolved IMPALA-3617.
-----------------------------------
Resolution: Cannot Reproduce
> Incorrect reporting of runtime filters in scan node
> ---------------------------------------------------
>
> Key: IMPALA-3617
> URL: https://issues.apache.org/jira/browse/IMPALA-3617
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 2.6.0
> Reporter: Mostafa Mokhtar
> Assignee: Henry Robinson
> Priority: Minor
> Labels: runtime-filters
> Attachments: query31.sql.3.out
>
>
> Summary line reports one filter arriving while both filters were received.
> {code}
> HDFS_SCAN_NODE (id=6):(Total: 9s443ms, non-child: 9s443ms, % non-child: 100.00%)
> ExecOption: Expr Evaluation Codegen Disabled, Codegen enabled: 0 out of 14
> Hdfs split stats (<volume id>:<# splits>/<split lengths>): 2:39/8.86 GB 5:42/9.29 GB 4:28/6.50 GB 1:29/6.45 GB 6:32/7.38 GB 3:27/6.07 GB 10:34/7.63 GB 7:26/5.75 GB 0:33/7.24 GB 9:37/8.06 GB 8:22/4.86 GB
> Runtime filters: Only following filters arrived: 8, waited 8s991ms
> Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0% 17:0% 18:0% 19:0% 20:0% 21:0% 22:0% 23:0% 24:0% 25:0% 26:0% 27:0%
> File Formats: PARQUET/NONE:335 PARQUET/SNAPPY:28
> BytesRead(500.000ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 73.18 MB, 73.18 MB, 73.18 MB, 73.18 MB, 73.18 MB, 73.18 MB, 73.18 MB, 93.31 MB, 134.68 MB, 161.83 MB, 161.83 MB, 161.83 MB, 161.83 MB, 184.49 MB, 206.65 MB, 233.75 MB, 233.75 MB, 233.75 MB, 261.02 MB, 261.02 MB, 288.24 MB, 309.87 MB, 309.87 MB, 309.87 MB, 337.08 MB, 337.08 MB, 337.08 MB, 337.08 MB, 337.08 MB
> - AverageHdfsReadThreadConcurrency: 0.00
> - AverageScannerThreadConcurrency: 3.26
> - BytesRead: 337.08 MB (353454758)
> - BytesReadDataNodeCache: 0
> - BytesReadLocal: 337.08 MB (353454758)
> - BytesReadRemoteUnexpected: 0
> - BytesReadShortCircuit: 337.08 MB (353454758)
> - DecompressionTime: 784.603ms
> - MaxCompressedTextFileLength: 0
> - NumColumns: 2 (2)
> - NumDisksAccessed: 6 (6)
> - NumRowGroups: 14 (14)
> - NumScannerThreadsStarted: 4 (4)
> - PeakMemoryUsage: 134.29 MB (140810128)
> - PerReadThreadRawHdfsThroughput: 2.00 GB/sec
> - RemoteScanRanges: 0 (0)
> - RowsRead: 90.33M (90325159)
> - RowsReturned: 90.32M (90321194)
> - RowsReturnedRate: 9.56 M/sec
> - ScanRangesComplete: 349 (349)
> - ScannerThreadsInvoluntaryContextSwitches: 2.85K (2851)
> - ScannerThreadsTotalWallClockTime: 1m16s
> - MaterializeTupleTime(*): 50s026ms
> - ScannerThreadsSysTime: 887.864ms
> - ScannerThreadsUserTime: 6s995ms
> - ScannerThreadsVoluntaryContextSwitches: 176.78K (176783)
> - TotalRawHdfsReadTime(*): 164.437ms
> - TotalReadThroughput: 13.76 MB/sec
> Filter 7 (1.00 MB):
> - Rows processed: 229.36K (229362)
> - Rows rejected: 3.96K (3965)
> - Rows total: 229.38K (229376)
> Filter 8 (1.00 MB):
> - Files processed: 349 (349)
> - Files rejected: 335 (335)
> - Files total: 349 (349)
> - RowGroups processed: 88.21K (88214)
> - RowGroups rejected: 0 (0)
> - RowGroups total: 88.21K (88214)
> - Rows processed: 229.36K (229362)
> - Rows rejected: 0 (0)
> - Rows total: 229.38K (229376)
> - Splits processed: 14 (14)
> - Splits rejected: 0 (0)
> - Splits total: 14 (14)
> {code}
> Ditto in final routing table
> {code}
> ID Src. Node Tgt. Node(s) Targets Target type Partition filter Pending (Expected) First arrived Completed
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> 6 3 0 20 LOCAL true 0 (20) N/A N/A
> 5 4 0 20 LOCAL false 0 (20) N/A N/A
> 8 9 6 20 LOCAL true 0 (20) N/A N/A
> 7 10 6 20 LOCAL false 0 (20) N/A N/A
> 10 15 12 20 LOCAL true 0 (20) N/A N/A
> 9 16 12 20 LOCAL false 0 (20) N/A N/A
> 12 21 18 20 LOCAL true 0 (20) N/A N/A
> 11 22 18 20 LOCAL false 0 (20) N/A N/A
> 14 27 24 20 LOCAL true 0 (20) N/A N/A
> 13 28 24 20 LOCAL false 0 (20) N/A N/A
> 16 33 30 20 LOCAL true 0 (20) N/A N/A
> 15 34 30 20 LOCAL false 0 (20) N/A N/A
> 4 36 2 17 REMOTE false 0 (20) 25s414ms 34s022ms
> 3 37 8, 2 17, 17 REMOTE, REMOTE false, false 2 (20) 33s973ms N/A
> 2 38 14, 8, 2 17, 17, 17 REMOTE, REMOTE, REMOTE false, false, false 0 (20) 16s435ms 16s461ms
> 1 39 20, 14, 8, 2 17, 17, 17, 17 REMOTE, REMOTE, REMOTE, REMOTE false, false, false, false 0 (20) 15s442ms 15s473ms
> 0 40 26, 20, 14, 8, 2 17, 17, 17, 17, 17 REMOTE, REMOTE, REMOTE, REMOTE, REMOTE false, false, false, false, false 0 (20) 9s203ms 9s232ms
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)