You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@impala.apache.org by 俊杰陈 <cj...@gmail.com> on 2017/10/27 05:38:05 UTC
Re: performance issue on big table join
Thanks, let me put here.
Yes, the query is intended to verify parallelism of partitioned join. We
want to know how many fragment instances started for a hash join on a
single node, and how many threads in a fragment instance perform the join.
I'm not sure whether there is only one thread participate in hash join,
since thus it can not maximize the CPU utilization.
The compute stats met following error:
Query: compute stats store_sales
WARNINGS: ImpalaRuntimeException: Error making
'updateTableColumnStatistics' RPC to Hive Metastore:
CAUSED BY: MetaException: Insert of object
"org.apache.hadoop.hive.metastore.model.MTableColumnStatistics@49fe55a1"
using statement "INSERT INTO `TAB_COL_STATS`
(`CS_ID`,`AVG_COL_LEN`,`BIT_VECTOR`,`COLUMN_NAME`,`COLUMN_TYPE`,`DB_NAME`,`BIG_DECIMAL_HIGH_VALUE`,`BIG_DECIMAL_LOW_VALUE`,`DOUBLE_HIGH_VALUE`,`DOUBLE_LOW_VALUE`,`LAST_ANALYZED`,`LONG_HIGH_VALUE`,`LONG_LOW_VALUE`,`MAX_COL_LEN`,`NUM_DISTINCTS`,`NUM_FALSES`,`NUM_NULLS`,`NUM_TRUES`,`TBL_ID`,`TABLE_NAME`)
VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)" failed : Unknown column
'BIT_VECTOR' in 'field list'
2017-10-27 12:02 GMT+08:00 Mostafa Mokhtar <mm...@cloudera.com>:
> Hi,
>
> Looks like you are joining store_sales with catalog_sales on item_sk, this
> kind of join condition is a many to many, which means the output number of
> rows will be much larger then input number of rows, not sure if this is
> intended.
>
> Also did you run "compute stats [TABLE_NAME]" on both tables?
>
> For a more comprehensive query try TPCDS Q17
>
> select i_item_id
>
> ,i_item_desc
>
> ,s_state
>
> ,count(ss_quantity) as store_sales_quantitycount
>
> ,avg(ss_quantity) as store_sales_quantityave
>
> ,stddev_samp(ss_quantity) as store_sales_quantitystdev
>
> ,stddev_samp(ss_quantity)/avg(ss_quantity) as
> store_sales_quantitycov
>
> ,count(sr_return_quantity) as store_returns_quantitycount
>
> ,avg(sr_return_quantity) as store_returns_quantityave
>
> ,stddev_samp(sr_return_quantity) as store_returns_quantitystdev
>
> ,stddev_samp(sr_return_quantity)/avg(sr_return_quantity) as
> store_returns_quantitycov
>
> ,count(cs_quantity) as catalog_sales_quantitycount
> ,avg(cs_quantity) as catalog_sales_quantityave
>
> ,stddev_samp(cs_quantity) as catalog_sales_quantitystdev
>
> ,stddev_samp(cs_quantity)/avg(cs_quantity) as
> catalog_sales_quantitycov
>
> from store_sales
>
> ,store_returns
>
> ,catalog_sales
>
> ,date_dim d1
>
> ,date_dim d2
>
> ,date_dim d3
>
> ,store
>
> ,item
>
> where d1.d_quarter_name = '2000Q1'
>
> and d1.d_date_sk = ss_sold_date_sk
>
> and i_item_sk = ss_item_sk
>
> and s_store_sk = ss_store_sk
>
> and ss_customer_sk = sr_customer_sk
>
> and ss_item_sk = sr_item_sk
>
> and ss_ticket_number = sr_ticket_number
>
> and sr_returned_date_sk = d2.d_date_sk
>
> and d2.d_quarter_name in ('2000Q1','2000Q2','2000Q3')
>
> and sr_customer_sk = cs_bill_customer_sk
>
> and sr_item_sk = cs_item_sk
>
> and cs_sold_date_sk = d3.d_date_sk
>
> and d3.d_quarter_name in ('2000Q1','2000Q2','2000Q3')
>
> group by i_item_id
>
> ,i_item_desc
>
> ,s_state
>
> order by i_item_id
>
> ,i_item_desc
>
> ,s_state
>
> limit 100;
>
>
> I recommend moving this kind of discussion on
> user@impala.incubator.apache.org.
>
> On Thu, Oct 26, 2017 at 7:25 PM, 俊杰陈 <cj...@gmail.com> wrote:
>
> > The profile file is damaged. Here is a screenshot for exec summary
> >
> >
> >
> > 2017-10-27 10:04 GMT+08:00 俊杰陈 <cj...@gmail.com>:
> >
> >> Hi Devs
> >>
> >> I met a performance issue on big table join. The query takes more than 3
> >> hours on Impala and only 3 minutes on Spark SQL on the same 5 nodes
> >> cluster. when running query, the left scanner and exchange node are
> very
> >> slow. Did I miss some key arguments?
> >>
> >> you can see profile file in attachment.
> >>
> >>
> >>
> >> --
> >> Thanks & Best Regards
> >>
> >
> >
> >
> > --
> > Thanks & Best Regards
> >
>
--
Thanks & Best Regards
RE: performance issue on big table join
Posted by "Xu, Cheng A" <ch...@intel.com>.
Append the profile log and it looks probe is taking quite a long time here.
Query (id=2438b42f267ed44:3e1ac3c700000000):
DEBUG MODE WARNING: Query profile created while running a DEBUG build of Impala. Use RELEASE builds to measure query performance.
Summary:
Session ID: 214312dff6aa405d:71a5d38fb87083b2
Session Type: BEESWAX
Start Time: 2017-10-27 15:18:32.854201000
End Time:
Query Type: QUERY
Query State: RUNNING
Query Status: OK
Impala Version: impalad version 2.11.0-SNAPSHOT DEBUG (build 8ad6d03310825418b6aa1d427200b07bc8bdb0bc)
User: root
Connected User: root
Delegated User:
Network Address: ::ffff:10.239.47.102:33193
Default Db: tpcds_bin_partitioned_parquet_100
Sql Statement: select ss_item_sk, sum(cs_net_profit), sum(ss_net_profit) from
store_sales ss, catalog_sales cs
where ss.ss_item_sk = cs.cs_item_sk and cs.cs_net_profit > ss.ss_net_profit
group by ss_item_sk
order by ss_item_sk
limit 100
Coordinator: bdpe833n4:22000
Query Options (set by configuration):
Query Options (set by configuration and planner): MT_DOP=0
Plan:
----------------
Max Per-Host Resource Reservation: Memory=35.94MB
Per-Host Resource Estimates: Memory=589.39MB
F03:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1
| Per-Host Resources: mem-estimate=0B mem-reservation=0B
PLAN-ROOT SINK
| mem-estimate=0B mem-reservation=0B
|
07:MERGING-EXCHANGE [UNPARTITIONED]
| order by: ss_item_sk ASC
| limit: 100
| mem-estimate=0B mem-reservation=0B
| tuple-ids=3 row-size=24B cardinality=100
|
F02:PLAN FRAGMENT [HASH(ss.ss_item_sk)] hosts=5 instances=5
Per-Host Resources: mem-estimate=493.39MB mem-reservation=35.94MB
04:TOP-N [LIMIT=100]
| order by: ss_item_sk ASC
| mem-estimate=2.34KB mem-reservation=0B
| tuple-ids=3 row-size=24B cardinality=100
|
03:AGGREGATE [FINALIZE]
| output: sum(cs_net_profit), sum(ss_net_profit)
| group by: ss_item_sk
| mem-estimate=10.00MB mem-reservation=1.94MB spill-buffer=64.00KB
| tuple-ids=2 row-size=24B cardinality=204925
|
02:HASH JOIN [INNER JOIN, PARTITIONED]
| hash predicates: ss.ss_item_sk = cs.cs_item_sk
| fk/pk conjuncts: none
| other predicates: cs.cs_net_profit > ss.ss_net_profit
| runtime filters: RF000 <- cs.cs_item_sk
| mem-estimate=483.39MB mem-reservation=34.00MB spill-buffer=2.00MB
| tuple-ids=0,1 row-size=32B cardinality=202370263193
|
|--06:EXCHANGE [HASH(cs.cs_item_sk)]
| | mem-estimate=0B mem-reservation=0B
| | tuple-ids=1 row-size=16B cardinality=143997065
| |
| F01:PLAN FRAGMENT [RANDOM] hosts=5 instances=5
| Per-Host Resources: mem-estimate=48.00MB mem-reservation=0B
| 01:SCAN HDFS [tpcds_bin_partitioned_parquet_100.catalog_sales cs, RANDOM]
| partitions=1837/1837 files=1936 size=17.03GB
| stats-rows=143997065 extrapolated-rows=disabled
| table stats: rows=143997065 size=17.03GB
| column stats: all
| mem-estimate=48.00MB mem-reservation=0B
| tuple-ids=1 row-size=16B cardinality=143997065
|
05:EXCHANGE [HASH(ss.ss_item_sk)]
| mem-estimate=0B mem-reservation=0B
| tuple-ids=0 row-size=16B cardinality=287997024
|
F00:PLAN FRAGMENT [RANDOM] hosts=5 instances=5
Per-Host Resources: mem-estimate=48.00MB mem-reservation=0B
00:SCAN HDFS [tpcds_bin_partitioned_parquet_100.store_sales ss, RANDOM]
partitions=1824/1824 files=1923 size=20.89GB
runtime filters: RF000 -> ss.ss_item_sk
stats-rows=287997024 extrapolated-rows=disabled
table stats: rows=287997024 size=20.89GB
column stats: all
mem-estimate=48.00MB mem-reservation=0B
tuple-ids=0 row-size=16B cardinality=287997024
----------------
Estimated Per-Host Mem: 618018750
Per Host Min Reservation: bdpe611n1:22000(35.94 MB) bdpe611n2:22000(35.94 MB) bdpe611n3:22000(35.94 MB) bdpe611n4:22000(35.94 MB) bdpe722n4:22000(35.94 MB) bdpe833n4:22000(0)
Request Pool: default-pool
Admission result: Admitted immediately
Planner Timeline: 106.057ms
- Analysis finished: 4.687ms (4.687ms)
- Equivalence classes computed: 5.000ms (313.121us)
- Single node plan created: 79.940ms (74.939ms)
- Runtime filters computed: 80.184ms (244.176us)
- Distributed plan created: 80.437ms (253.115us)
- Planning finished: 106.057ms (25.620ms)
Query Timeline: 1h7m
- Query submitted: 115.266us (115.266us)
- Planning finished: 207.412ms (207.297ms)
- Submit for admission: 282.363ms (74.950ms)
- Completed admission: 282.516ms (152.998us)
- Ready to start on 6 backends: 318.275ms (35.759ms)
- All 6 execution backends (16 fragment instances) started: 495.692ms (177.416ms)
- First dynamic filter received: 21s281ms (20s785ms)
- ComputeScanRangeAssignmentTimer: 54.075ms
ImpalaServer:
- ClientFetchWaitTimer: 0.000ns
- RowMaterializationTimer: 0.000ns
Execution Profile 2438b42f267ed44:3e1ac3c700000000:(Total: 195.134ms, non-child: 0.000ns, % non-child: 0.00%)
Number of filters: 1
Filter routing table:
ID Src. Node Tgt. Node(s) Target type Partition filter Pending (Expected) First arrived Completed Enabled
-------------------------------------------------------------------------------------------------------------------
0 2 0 REMOTE false 5 (5) N/A N/A true
Backend startup latencies: Count: 6, min / max: 78ms / 131ms, 25th %-ile: 92ms, 50th %-ile: 126ms, 75th %-ile: 128ms, 90th %-ile: 128ms, 95th %-ile: 131ms, 99.9th %-ile: 131ms
- FiltersReceived: 5 (5)
- FinalizationTimer: 0.000ns
Averaged Fragment F03:
Coordinator Fragment F03:
Instance 2438b42f267ed44:3e1ac3c700000000 (host=bdpe833n4:22000):(Total: 26.461ms, non-child: 0.000ns, % non-child: 0.00%)
MemoryUsage(1m4s): 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB, 8.28 KB
- AverageThreadTokens: 0.00
- BloomFilterBytes: 0
- PeakMemoryUsage: 232.00 KB (237568)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 16.01 MB (16785690)
- RowsProduced: 0 (0)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 0.000ns
- TotalStorageWaitTime: 0.000ns
- TotalThreadsInvoluntaryContextSwitches: 0 (0)
- TotalThreadsTotalWallClockTime: 0.000ns
- TotalThreadsSysTime: 0.000ns
- TotalThreadsUserTime: 0.000ns
- TotalThreadsVoluntaryContextSwitches: 0 (0)
Fragment Instance Lifecycle Timings:
- OpenTime: 0.000ns
- ExecTreeOpenTime: 0.000ns
- PrepareTime: 26.380ms
- ExecTreePrepareTime: 83.673us
PLAN_ROOT_SINK:
- PeakMemoryUsage: 0
CodeGen:(Total: 40.138ms, non-child: 40.138ms, % non-child: 100.00%)
- CodegenTime: 2.170ms
- CompileTime: 2.239ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 1.84 MB (1929896)
- NumFunctions: 29 (29)
- NumInstructions: 448 (448)
- OptimizationTime: 11.294ms
- PeakMemoryUsage: 224.00 KB (229376)
- PrepareTime: 25.840ms
EXCHANGE_NODE (id=7):
ExecOption: Codegen Enabled
- ConvertRowBatchTime: 0.000ns
- MergeGetNext: 0.000ns
- MergeGetNextBatch: 0.000ns
- PeakMemoryUsage: 0
- RowsReturned: 0 (0)
- RowsReturnedRate: 0
DataStreamReceiver:
BytesReceived(1m4s): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
- BytesReceived: 0
- DeserializeRowBatchTimer: 0.000ns
- FirstBatchArrivalWaitTime: 0.000ns
- PeakMemoryUsage: 0
- SendersBlockedTimer: 0.000ns
- SendersBlockedTotalTimer(*): 0.000ns
Averaged Fragment F02:
Fragment F02:
Instance 2438b42f267ed44:3e1ac3c70000000e (host=bdpe611n1:22000):(Total: 60.735ms, non-child: 0.000ns, % non-child: 0.00%)
MemoryUsage(1m4s): 1.51 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB
ThreadUsage(1m4s): 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1
- AverageThreadTokens: 1.01
- BloomFilterBytes: 16.00 MB (16777216)
- PeakMemoryUsage: 2.15 GB (2303872045)
- PeakReservation: 2.12 GB (2274426880)
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 2.36 GB (2530721031)
- RowsProduced: 0 (0)
- TotalNetworkReceiveTime: 10s323ms
- TotalNetworkSendTime: 0.000ns
- TotalStorageWaitTime: 0.000ns
- TotalThreadsInvoluntaryContextSwitches: 74 (74)
- TotalThreadsTotalWallClockTime: 23s913ms
- TotalThreadsSysTime: 1s022ms
- TotalThreadsUserTime: 13s752ms
- TotalThreadsVoluntaryContextSwitches: 17.21K (17214)
Fragment Instance Lifecycle Timings:
- OpenTime: 0.000ns
- ExecTreeOpenTime: 0.000ns
- PrepareTime: 60.675ms
- ExecTreePrepareTime: 1.038ms
DataStreamSender (dst_id=7):(Total: 277.868us, non-child: 277.868us, % non-child: 100.00%)
- BytesSent: 0
- NetworkThroughput(*): 0.00 /sec
- OverallThroughput: 0.00 /sec
- PeakMemoryUsage: 5.12 KB (5240)
- RowsReturned: 0 (0)
- SerializeBatchTime: 0.000ns
- TransmitDataRPCTime: 0.000ns
- UncompressedRowBatchSize: 0
CodeGen:(Total: 403.767ms, non-child: 403.767ms, % non-child: 100.00%)
- CodegenTime: 7.281ms
- CompileTime: 25.928ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 1.84 MB (1929896)
- NumFunctions: 149 (149)
- NumInstructions: 4.39K (4394)
- OptimizationTime: 338.848ms
- PeakMemoryUsage: 2.15 MB (2249728)
- PrepareTime: 37.870ms
SORT_NODE (id=4):(Total: 1.036ms, non-child: 113.376us, % non-child: 10.93%)
ExecOption: Codegen Enabled
- InsertBatchTime: 0.000ns
- PeakMemoryUsage: 4.00 KB (4096)
- RowsReturned: 0 (0)
- RowsReturnedRate: 0
- TuplePoolReclamations: 0 (0)
AGGREGATION_NODE (id=3):(Total: 923.490us, non-child: 0.000ns, % non-child: 0.00%)
ExecOption: Codegen Enabled
- BuildTime: 7m9s
- GetResultsTime: 0.000ns
- HTResizeTime: 9s233ms
- HashBuckets: 0 (0)
- LargestPartitionPercent: 0 (0)
- MaxPartitionLevel: 0 (0)
- NumRepartitions: 0 (0)
- PartitionsCreated: 16 (16)
- PeakMemoryUsage: 3.13 MB (3277926)
- RowsRepartitioned: 0 (0)
- RowsReturned: 0 (0)
- RowsReturnedRate: 0
- SpilledPartitions: 0 (0)
HASH_JOIN_NODE (id=2):(Total: 59m41s, non-child: 59m25s, % non-child: 99.56%)
ExecOption: Probe Side Codegen Enabled, Join Build-Side Prepared Asynchronously
- BuildRows: 28.78M (28780361)
- BuildTime: 11s291ms
- NumHashTableBuildsSkipped: 0 (0)
- PeakMemoryUsage: 2.12 GB (2271291622)
- ProbeRows: 19.04M (19038208)
- ProbeRowsPartitioned: 0 (0)
- ProbeTime: 59m6s
- RowsReturned: 9.19B (9192516608)
- RowsReturnedRate: 2.57 M/sec
Buffer pool:
- AllocTime: 41.602ms
- CumulativeAllocationBytes: 2.12 GB (2271215616)
- CumulativeAllocations: 587 (587)
- PeakReservation: 2.12 GB (2271215616)
- PeakUnpinnedBytes: 0
- PeakUsedReservation: 2.12 GB (2271215616)
- ReadIoBytes: 0
- ReadIoOps: 0 (0)
- ReadIoWaitTime: 0.000ns
- WriteIoBytes: 0
- WriteIoOps: 0 (0)
- WriteIoWaitTime: 0.000ns
Hash Join Builder (join_node_id=2):
ExecOption: Build Side Codegen Enabled, Hash Table Construction Codegen Enabled
Runtime filters: 1 of 1 Runtime Filter Published
- BuildRowsPartitionTime: 7s193ms
- BuildRowsPartitioned: 28.78M (28780361)
- HashBuckets: 67.11M (67108864)
- HashCollisions: 0 (0)
- HashTablesBuildTime: 4s069ms
- LargestPartitionPercent: 6 (6)
- MaxPartitionLevel: 0 (0)
- NumRepartitions: 0 (0)
- PartitionsCreated: 16 (16)
- PeakMemoryUsage: 21.12 KB (21632)
- RepartitionTime: 0.000ns
- SpilledPartitions: 0 (0)
EXCHANGE_NODE (id=6):(Total: 12s499ms, non-child: 12s499ms, % non-child: 100.00%)
- ConvertRowBatchTime: 2s961ms
- PeakMemoryUsage: 0
- RowsReturned: 28.78M (28780361)
- RowsReturnedRate: 2.30 M/sec
DataStreamReceiver:
BytesReceived(500.000ms): 4.83 MB, 13.21 MB, 20.00 MB, 25.94 MB, 34.47 MB, 43.07 MB, 53.29 MB, 60.58 MB, 69.64 MB, 79.96 MB, 90.95 MB, 102.65 MB, 114.45 MB, 126.38 MB, 138.11 MB, 149.89 MB, 160.54 MB, 171.15 MB, 183.00 MB, 194.76 MB, 206.61 MB, 218.42 MB, 230.30 MB, 241.95 MB, 252.93 MB, 263.84 MB, 275.62 MB, 287.39 MB, 299.12 MB, 310.70 MB, 322.34 MB, 333.89 MB, 345.54 MB, 357.32 MB, 368.84 MB, 378.03 MB, 387.25 MB, 395.88 MB, 400.75 MB, 403.03 MB, 404.79 MB, 404.79 MB, 404.79 MB, 404.79 MB, 404.79 MB, 404.79 MB, 404.79 MB, 404.79 MB
- BytesReceived: 404.79 MB (424455941)
- DeserializeRowBatchTimer: 1s093ms
- FirstBatchArrivalWaitTime: 0.000ns
- PeakMemoryUsage: 10.08 MB (10568925)
- SendersBlockedTimer: 0.000ns
- SendersBlockedTotalTimer(*): 0.000ns
EXCHANGE_NODE (id=5):(Total: 3s449ms, non-child: 3s449ms, % non-child: 100.00%)
- ConvertRowBatchTime: 2s103ms
- PeakMemoryUsage: 0
- RowsReturned: 19.04M (19038208)
- RowsReturnedRate: 5.52 M/sec
DataStreamReceiver:
BytesReceived(1m4s): 7.95 MB, 11.14 MB, 15.32 MB, 18.94 MB, 22.47 MB, 26.87 MB, 32.41 MB, 37.89 MB, 42.43 MB, 46.75 MB, 51.04 MB, 55.32 MB, 59.45 MB, 63.75 MB, 67.99 MB, 72.17 MB, 76.59 MB, 80.87 MB, 84.89 MB, 89.18 MB, 93.54 MB, 97.90 MB, 102.21 MB, 106.48 MB, 110.87 MB, 115.28 MB, 119.65 MB, 124.01 MB, 128.33 MB, 132.74 MB, 137.14 MB, 141.41 MB, 145.62 MB, 149.66 MB, 154.18 MB, 159.13 MB, 164.06 MB, 168.96 MB, 173.81 MB, 178.62 MB, 183.34 MB, 187.96 MB, 192.59 MB, 197.31 MB, 201.98 MB, 206.65 MB, 211.33 MB, 215.70 MB, 220.12 MB, 224.81 MB, 229.49 MB, 234.18 MB, 238.89 MB, 243.60 MB, 248.33 MB, 253.07 MB, 257.80 MB, 262.53 MB, 267.27 MB, 272.02 MB, 276.27 MB, 280.55 MB, 284.87 MB
- BytesReceived: 288.64 MB (302657277)
- DeserializeRowBatchTimer: 1s534ms
- FirstBatchArrivalWaitTime: 647.613ms
- PeakMemoryUsage: 11.92 MB (12494925)
- SendersBlockedTimer: 1h
- SendersBlockedTotalTimer(*): 4h58m
Buffer pool:
- AllocTime: 677.240us
- CumulativeAllocationBytes: 4.62 MB (4849664)
- CumulativeAllocations: 58 (58)
- PeakReservation: 3.06 MB (3211264)
- PeakUnpinnedBytes: 0
- PeakUsedReservation: 3.06 MB (3211264)
- ReadIoBytes: 0
- ReadIoOps: 0 (0)
- ReadIoWaitTime: 0.000ns
- WriteIoBytes: 0
- WriteIoOps: 0 (0)
- WriteIoWaitTime: 0.000ns
Instance 2438b42f267ed44:3e1ac3c70000000c (host=bdpe611n2:22000):(Total: 60.385ms, non-child: 0.000ns, % non-child: 0.00%)
MemoryUsage(1m4s): 1.51 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB
ThreadUsage(1m4s): 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1
- AverageThreadTokens: 1.01
- BloomFilterBytes: 16.00 MB (16777216)
- PeakMemoryUsage: 2.15 GB (2308066349)
- PeakReservation: 2.12 GB (2278621184)
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 2.35 GB (2523002980)
- RowsProduced: 0 (0)
- TotalNetworkReceiveTime: 4m28s
- TotalNetworkSendTime: 0.000ns
- TotalStorageWaitTime: 0.000ns
- TotalThreadsInvoluntaryContextSwitches: 46 (46)
- TotalThreadsTotalWallClockTime: 23s375ms
- TotalThreadsSysTime: 830.830ms
- TotalThreadsUserTime: 12s866ms
- TotalThreadsVoluntaryContextSwitches: 17.74K (17740)
Fragment Instance Lifecycle Timings:
- OpenTime: 0.000ns
- ExecTreeOpenTime: 0.000ns
- PrepareTime: 60.246ms
- ExecTreePrepareTime: 1.008ms
DataStreamSender (dst_id=7):(Total: 338.446us, non-child: 338.446us, % non-child: 100.00%)
- BytesSent: 0
- NetworkThroughput(*): 0.00 /sec
- OverallThroughput: 0.00 /sec
- PeakMemoryUsage: 5.12 KB (5240)
- RowsReturned: 0 (0)
- SerializeBatchTime: 0.000ns
- TransmitDataRPCTime: 0.000ns
- UncompressedRowBatchSize: 0
CodeGen:(Total: 418.144ms, non-child: 418.144ms, % non-child: 100.00%)
- CodegenTime: 6.477ms
- CompileTime: 26.204ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 1.84 MB (1929896)
- NumFunctions: 149 (149)
- NumInstructions: 4.39K (4394)
- OptimizationTime: 353.902ms
- PeakMemoryUsage: 2.15 MB (2249728)
- PrepareTime: 36.915ms
SORT_NODE (id=4):(Total: 1.007ms, non-child: 94.499us, % non-child: 9.38%)
ExecOption: Codegen Enabled
- InsertBatchTime: 0.000ns
- PeakMemoryUsage: 4.00 KB (4096)
- RowsReturned: 0 (0)
- RowsReturnedRate: 0
- TuplePoolReclamations: 0 (0)
AGGREGATION_NODE (id=3):(Total: 912.507us, non-child: 0.000ns, % non-child: 0.00%)
ExecOption: Codegen Enabled
- BuildTime: 7m55s
- GetResultsTime: 0.000ns
- HTResizeTime: 9s948ms
- HashBuckets: 0 (0)
- LargestPartitionPercent: 0 (0)
- MaxPartitionLevel: 0 (0)
- NumRepartitions: 0 (0)
- PartitionsCreated: 16 (16)
- PeakMemoryUsage: 3.13 MB (3277926)
- RowsRepartitioned: 0 (0)
- RowsReturned: 0 (0)
- RowsReturnedRate: 0
- SpilledPartitions: 0 (0)
HASH_JOIN_NODE (id=2):(Total: 58m52s, non-child: 54m19s, % non-child: 92.26%)
ExecOption: Probe Side Codegen Enabled, Join Build-Side Prepared Asynchronously
- BuildRows: 28.94M (28939765)
- BuildTime: 10s085ms
- NumHashTableBuildsSkipped: 0 (0)
- PeakMemoryUsage: 2.12 GB (2275485926)
- ProbeRows: 19.66M (19658752)
- ProbeRowsPartitioned: 0 (0)
- ProbeTime: 54m
- RowsReturned: 9.48B (9482271744)
- RowsReturnedRate: 2.68 M/sec
Buffer pool:
- AllocTime: 34.354ms
- CumulativeAllocationBytes: 2.12 GB (2275409920)
- CumulativeAllocations: 589 (589)
- PeakReservation: 2.12 GB (2275409920)
- PeakUnpinnedBytes: 0
- PeakUsedReservation: 2.12 GB (2275409920)
- ReadIoBytes: 0
- ReadIoOps: 0 (0)
- ReadIoWaitTime: 0.000ns
- WriteIoBytes: 0
- WriteIoOps: 0 (0)
- WriteIoWaitTime: 0.000ns
Hash Join Builder (join_node_id=2):
ExecOption: Build Side Codegen Enabled, Hash Table Construction Codegen Enabled
Runtime filters: 1 of 1 Runtime Filter Published
- BuildRowsPartitionTime: 6s484ms
- BuildRowsPartitioned: 28.94M (28939765)
- HashBuckets: 67.11M (67108864)
- HashCollisions: 0 (0)
- HashTablesBuildTime: 3s570ms
- LargestPartitionPercent: 6 (6)
- MaxPartitionLevel: 0 (0)
- NumRepartitions: 0 (0)
- PartitionsCreated: 16 (16)
- PeakMemoryUsage: 21.12 KB (21632)
- RepartitionTime: 0.000ns
- SpilledPartitions: 0 (0)
EXCHANGE_NODE (id=6):(Total: 13s166ms, non-child: 13s166ms, % non-child: 100.00%)
- ConvertRowBatchTime: 3s054ms
- PeakMemoryUsage: 0
- RowsReturned: 28.94M (28939765)
- RowsReturnedRate: 2.20 M/sec
DataStreamReceiver:
BytesReceived(500.000ms): 0, 8.31 MB, 16.07 MB, 21.92 MB, 29.09 MB, 36.98 MB, 46.76 MB, 55.98 MB, 63.87 MB, 73.19 MB, 83.86 MB, 95.62 MB, 107.31 MB, 119.33 MB, 131.28 MB, 142.99 MB, 154.89 MB, 164.96 MB, 176.31 MB, 188.16 MB, 200.09 MB, 211.93 MB, 223.91 MB, 235.83 MB, 247.31 MB, 257.80 MB, 269.44 MB, 281.31 MB, 293.16 MB, 304.77 MB, 316.58 MB, 328.36 MB, 339.80 MB, 351.73 MB, 363.51 MB, 374.18 MB, 383.41 MB, 392.72 MB, 400.63 MB, 403.84 MB, 406.17 MB, 407.11 MB, 407.11 MB, 407.11 MB, 407.11 MB, 407.11 MB, 407.11 MB, 407.11 MB
- BytesReceived: 407.11 MB (426884473)
- DeserializeRowBatchTimer: 1s128ms
- FirstBatchArrivalWaitTime: 0.000ns
- PeakMemoryUsage: 11.76 MB (12326400)
- SendersBlockedTimer: 0.000ns
- SendersBlockedTotalTimer(*): 0.000ns
EXCHANGE_NODE (id=5):(Total: 4m20s, non-child: 4m20s, % non-child: 100.00%)
- ConvertRowBatchTime: 2s176ms
- PeakMemoryUsage: 0
- RowsReturned: 19.66M (19658752)
- RowsReturnedRate: 75.42 K/sec
DataStreamReceiver:
BytesReceived(1m4s): 7.91 MB, 11.14 MB, 15.37 MB, 19.01 MB, 22.56 MB, 26.95 MB, 32.53 MB, 38.07 MB, 42.66 MB, 46.99 MB, 51.30 MB, 55.62 MB, 59.75 MB, 64.07 MB, 68.35 MB, 72.55 MB, 76.99 MB, 81.31 MB, 85.35 MB, 89.68 MB, 94.06 MB, 98.47 MB, 102.79 MB, 107.08 MB, 111.48 MB, 115.92 MB, 120.33 MB, 124.71 MB, 129.05 MB, 133.48 MB, 137.90 MB, 142.18 MB, 146.39 MB, 150.45 MB, 154.98 MB, 159.94 MB, 164.88 MB, 169.81 MB, 174.68 MB, 179.51 MB, 184.25 MB, 188.90 MB, 193.57 MB, 198.35 MB, 203.05 MB, 207.74 MB, 212.47 MB, 216.86 MB, 221.31 MB, 226.05 MB, 230.75 MB, 235.47 MB, 240.20 MB, 244.97 MB, 249.73 MB, 254.48 MB, 259.22 MB, 263.98 MB, 268.74 MB, 273.53 MB, 277.81 MB, 282.12 MB, 286.47 MB
- BytesReceived: 290.43 MB (304536877)
- DeserializeRowBatchTimer: 1s880ms
- FirstBatchArrivalWaitTime: 611.434ms
- PeakMemoryUsage: 11.92 MB (12494925)
- SendersBlockedTimer: 2m23s
- SendersBlockedTotalTimer(*): 11m9s
Buffer pool:
- AllocTime: 740.683us
- CumulativeAllocationBytes: 4.62 MB (4849664)
- CumulativeAllocations: 58 (58)
- PeakReservation: 3.06 MB (3211264)
- PeakUnpinnedBytes: 0
- PeakUsedReservation: 3.06 MB (3211264)
- ReadIoBytes: 0
- ReadIoOps: 0 (0)
- ReadIoWaitTime: 0.000ns
- WriteIoBytes: 0
- WriteIoOps: 0 (0)
- WriteIoWaitTime: 0.000ns
Instance 2438b42f267ed44:3e1ac3c70000000b (host=bdpe611n3:22000):(Total: 65.589ms, non-child: 0.000ns, % non-child: 0.00%)
MemoryUsage(1m4s): 1.50 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB
ThreadUsage(1m4s): 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1
- AverageThreadTokens: 1.01
- BloomFilterBytes: 16.00 MB (16777216)
- PeakMemoryUsage: 2.14 GB (2296330746)
- PeakReservation: 2.11 GB (2268135424)
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 2.33 GB (2505825451)
- RowsProduced: 0 (0)
- TotalNetworkReceiveTime: 3m38s
- TotalNetworkSendTime: 0.000ns
- TotalStorageWaitTime: 0.000ns
- TotalThreadsInvoluntaryContextSwitches: 24 (24)
- TotalThreadsTotalWallClockTime: 23s506ms
- TotalThreadsSysTime: 848.021ms
- TotalThreadsUserTime: 12s669ms
- TotalThreadsVoluntaryContextSwitches: 18.45K (18448)
Fragment Instance Lifecycle Timings:
- OpenTime: 0.000ns
- ExecTreeOpenTime: 0.000ns
- PrepareTime: 65.405ms
- ExecTreePrepareTime: 951.512us
DataStreamSender (dst_id=7):(Total: 280.611us, non-child: 280.611us, % non-child: 100.00%)
- BytesSent: 0
- NetworkThroughput(*): 0.00 /sec
- OverallThroughput: 0.00 /sec
- PeakMemoryUsage: 5.12 KB (5240)
- RowsReturned: 0 (0)
- SerializeBatchTime: 0.000ns
- TransmitDataRPCTime: 0.000ns
- UncompressedRowBatchSize: 0
CodeGen:(Total: 401.447ms, non-child: 401.447ms, % non-child: 100.00%)
- CodegenTime: 8.074ms
- CompileTime: 25.532ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 1.84 MB (1929896)
- NumFunctions: 149 (149)
- NumInstructions: 4.39K (4394)
- OptimizationTime: 335.130ms
- PeakMemoryUsage: 2.15 MB (2249728)
- PrepareTime: 39.643ms
SORT_NODE (id=4):(Total: 949.746us, non-child: 96.968us, % non-child: 10.21%)
ExecOption: Codegen Enabled
- InsertBatchTime: 0.000ns
- PeakMemoryUsage: 4.00 KB (4096)
- RowsReturned: 0 (0)
- RowsReturnedRate: 0
- TuplePoolReclamations: 0 (0)
AGGREGATION_NODE (id=3):(Total: 852.778us, non-child: 0.000ns, % non-child: 0.00%)
ExecOption: Codegen Enabled
- BuildTime: 7m48s
- GetResultsTime: 0.000ns
- HTResizeTime: 9s829ms
- HashBuckets: 0 (0)
- LargestPartitionPercent: 0 (0)
- MaxPartitionLevel: 0 (0)
- NumRepartitions: 0 (0)
- PartitionsCreated: 16 (16)
- PeakMemoryUsage: 3.13 MB (3277926)
- RowsRepartitioned: 0 (0)
- RowsReturned: 0 (0)
- RowsReturnedRate: 0
- SpilledPartitions: 0 (0)
HASH_JOIN_NODE (id=2):(Total: 58m59s, non-child: 55m16s, % non-child: 93.69%)
ExecOption: Probe Side Codegen Enabled, Join Build-Side Prepared Asynchronously
- BuildRows: 28.62M (28618322)
- BuildTime: 10s036ms
- NumHashTableBuildsSkipped: 0 (0)
- PeakMemoryUsage: 2.11 GB (2265000166)
- ProbeRows: 19.43M (19430400)
- ProbeRowsPartitioned: 0 (0)
- ProbeTime: 54m58s
- RowsReturned: 9.37B (9368660992)
- RowsReturnedRate: 2.65 M/sec
Buffer pool:
- AllocTime: 26.074ms
- CumulativeAllocationBytes: 2.11 GB (2264924160)
- CumulativeAllocations: 584 (584)
- PeakReservation: 2.11 GB (2264924160)
- PeakUnpinnedBytes: 0
- PeakUsedReservation: 2.11 GB (2264924160)
- ReadIoBytes: 0
- ReadIoOps: 0 (0)
- ReadIoWaitTime: 0.000ns
- WriteIoBytes: 0
- WriteIoOps: 0 (0)
- WriteIoWaitTime: 0.000ns
Hash Join Builder (join_node_id=2):
ExecOption: Build Side Codegen Enabled, Hash Table Construction Codegen Enabled
Runtime filters: 1 of 1 Runtime Filter Published
- BuildRowsPartitionTime: 6s314ms
- BuildRowsPartitioned: 28.62M (28618322)
- HashBuckets: 67.11M (67108864)
- HashCollisions: 0 (0)
- HashTablesBuildTime: 3s687ms
- LargestPartitionPercent: 6 (6)
- MaxPartitionLevel: 0 (0)
- NumRepartitions: 0 (0)
- PartitionsCreated: 16 (16)
- PeakMemoryUsage: 21.12 KB (21632)
- RepartitionTime: 0.000ns
- SpilledPartitions: 0 (0)
EXCHANGE_NODE (id=6):(Total: 13s349ms, non-child: 13s349ms, % non-child: 100.00%)
- ConvertRowBatchTime: 2s954ms
- PeakMemoryUsage: 0
- RowsReturned: 28.62M (28618322)
- RowsReturnedRate: 2.14 M/sec
DataStreamReceiver:
BytesReceived(500.000ms): 2.51 MB, 11.25 MB, 18.78 MB, 23.82 MB, 32.00 MB, 40.32 MB, 50.11 MB, 58.46 MB, 66.92 MB, 76.52 MB, 87.38 MB, 99.16 MB, 110.70 MB, 122.51 MB, 134.29 MB, 145.95 MB, 157.06 MB, 167.11 MB, 178.85 MB, 190.56 MB, 202.38 MB, 214.10 MB, 225.90 MB, 237.57 MB, 248.62 MB, 259.30 MB, 271.01 MB, 282.75 MB, 294.46 MB, 305.97 MB, 317.48 MB, 329.01 MB, 340.59 MB, 352.31 MB, 363.98 MB, 373.54 MB, 382.68 MB, 391.87 MB, 397.94 MB, 400.22 MB, 402.55 MB, 402.56 MB, 402.56 MB, 402.56 MB, 402.56 MB, 402.56 MB, 402.56 MB, 402.56 MB
- BytesReceived: 402.56 MB (422114759)
- DeserializeRowBatchTimer: 1s198ms
- FirstBatchArrivalWaitTime: 0.000ns
- PeakMemoryUsage: 11.20 MB (11748600)
- SendersBlockedTimer: 0.000ns
- SendersBlockedTotalTimer(*): 0.000ns
EXCHANGE_NODE (id=5):(Total: 3m30s, non-child: 3m30s, % non-child: 100.00%)
- ConvertRowBatchTime: 2s142ms
- PeakMemoryUsage: 0
- RowsReturned: 19.43M (19431414)
- RowsReturnedRate: 92.42 K/sec
DataStreamReceiver:
BytesReceived(1m4s): 7.86 MB, 11.07 MB, 15.21 MB, 18.81 MB, 22.33 MB, 26.69 MB, 32.22 MB, 37.69 MB, 42.22 MB, 46.49 MB, 50.75 MB, 55.01 MB, 59.10 MB, 63.39 MB, 67.63 MB, 71.77 MB, 76.16 MB, 80.43 MB, 84.41 MB, 88.70 MB, 93.02 MB, 97.37 MB, 101.64 MB, 105.88 MB, 110.22 MB, 114.59 MB, 118.93 MB, 123.26 MB, 127.55 MB, 131.96 MB, 136.33 MB, 140.60 MB, 144.78 MB, 148.79 MB, 153.29 MB, 158.21 MB, 163.08 MB, 167.95 MB, 172.76 MB, 177.55 MB, 182.25 MB, 186.84 MB, 191.45 MB, 196.16 MB, 200.84 MB, 205.50 MB, 210.16 MB, 214.50 MB, 218.89 MB, 223.58 MB, 228.22 MB, 232.87 MB, 237.56 MB, 242.26 MB, 246.97 MB, 251.68 MB, 256.38 MB, 261.09 MB, 265.81 MB, 270.55 MB, 274.74 MB, 279.01 MB, 283.31 MB
- BytesReceived: 287.04 MB (300979168)
- DeserializeRowBatchTimer: 1s831ms
- FirstBatchArrivalWaitTime: 621.536ms
- PeakMemoryUsage: 11.87 MB (12446775)
- SendersBlockedTimer: 0.000ns
- SendersBlockedTotalTimer(*): 0.000ns
Buffer pool:
- AllocTime: 677.596us
- CumulativeAllocationBytes: 4.62 MB (4849664)
- CumulativeAllocations: 58 (58)
- PeakReservation: 3.06 MB (3211264)
- PeakUnpinnedBytes: 0
- PeakUsedReservation: 3.06 MB (3211264)
- ReadIoBytes: 0
- ReadIoOps: 0 (0)
- ReadIoWaitTime: 0.000ns
- WriteIoBytes: 0
- WriteIoOps: 0 (0)
- WriteIoWaitTime: 0.000ns
Instance 2438b42f267ed44:3e1ac3c70000000f (host=bdpe611n4:22000):(Total: 51.527ms, non-child: 0.000ns, % non-child: 0.00%)
MemoryUsage(1m4s): 1.50 GB, 2.15 GB, 2.16 GB, 2.16 GB, 2.16 GB, 2.16 GB, 2.16 GB, 2.16 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.15 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB, 2.14 GB
ThreadUsage(1m4s): 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1
- AverageThreadTokens: 1.01
- BloomFilterBytes: 16.00 MB (16777216)
- PeakMemoryUsage: 2.16 GB (2314423341)
- PeakReservation: 2.13 GB (2284978176)
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 2.34 GB (2515248771)
- RowsProduced: 0 (0)
- TotalNetworkReceiveTime: 5m3s
- TotalNetworkSendTime: 0.000ns
- TotalStorageWaitTime: 0.000ns
- TotalThreadsInvoluntaryContextSwitches: 47 (47)
- TotalThreadsTotalWallClockTime: 23s986ms
- TotalThreadsSysTime: 930.342ms
- TotalThreadsUserTime: 13s892ms
- TotalThreadsVoluntaryContextSwitches: 17.42K (17421)
Fragment Instance Lifecycle Timings:
- OpenTime: 0.000ns
- ExecTreeOpenTime: 0.000ns
- PrepareTime: 51.455ms
- ExecTreePrepareTime: 875.534us
DataStreamSender (dst_id=7):(Total: 287.278us, non-child: 287.278us, % non-child: 100.00%)
- BytesSent: 0
- NetworkThroughput(*): 0.00 /sec
- OverallThroughput: 0.00 /sec
- PeakMemoryUsage: 5.12 KB (5240)
- RowsReturned: 0 (0)
- SerializeBatchTime: 0.000ns
- TransmitDataRPCTime: 0.000ns
- UncompressedRowBatchSize: 0
CodeGen:(Total: 406.407ms, non-child: 406.407ms, % non-child: 100.00%)
- CodegenTime: 6.204ms
- CompileTime: 25.846ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 1.84 MB (1929896)
- NumFunctions: 149 (149)
- NumInstructions: 4.39K (4394)
- OptimizationTime: 347.117ms
- PeakMemoryUsage: 2.15 MB (2249728)
- PrepareTime: 32.535ms
SORT_NODE (id=4):(Total: 859.398us, non-child: 119.435us, % non-child: 13.90%)
ExecOption: Codegen Enabled
- InsertBatchTime: 0.000ns
- PeakMemoryUsage: 4.00 KB (4096)
- RowsReturned: 0 (0)
- RowsReturnedRate: 0
- TuplePoolReclamations: 0 (0)
AGGREGATION_NODE (id=3):(Total: 739.963us, non-child: 0.000ns, % non-child: 0.00%)
ExecOption: Codegen Enabled
- BuildTime: 7m49s
- GetResultsTime: 0.000ns
- HTResizeTime: 9s866ms
- HashBuckets: 0 (0)
- LargestPartitionPercent: 0 (0)
- MaxPartitionLevel: 0 (0)
- NumRepartitions: 0 (0)
- PartitionsCreated: 16 (16)
- PeakMemoryUsage: 3.19 MB (3343462)
- RowsRepartitioned: 0 (0)
- RowsReturned: 0 (0)
- RowsReturnedRate: 0
- SpilledPartitions: 0 (0)
HASH_JOIN_NODE (id=2):(Total: 58m59s, non-child: 53m50s, % non-child: 91.27%)
ExecOption: Probe Side Codegen Enabled, Join Build-Side Prepared Asynchronously
- BuildRows: 29.07M (29066489)
- BuildTime: 11s309ms
- NumHashTableBuildsSkipped: 0 (0)
- PeakMemoryUsage: 2.13 GB (2281777382)
- ProbeRows: 19.74M (19742720)
- ProbeRowsPartitioned: 0 (0)
- ProbeTime: 53m31s
- RowsReturned: 9.52B (9516909568)
- RowsReturnedRate: 2.69 M/sec
Buffer pool:
- AllocTime: 43.372ms
- CumulativeAllocationBytes: 2.12 GB (2281701376)
- CumulativeAllocations: 592 (592)
- PeakReservation: 2.12 GB (2281701376)
- PeakUnpinnedBytes: 0
- PeakUsedReservation: 2.12 GB (2281701376)
- ReadIoBytes: 0
- ReadIoOps: 0 (0)
- ReadIoWaitTime: 0.000ns
- WriteIoBytes: 0
- WriteIoOps: 0 (0)
- WriteIoWaitTime: 0.000ns
Hash Join Builder (join_node_id=2):
ExecOption: Build Side Codegen Enabled, Hash Table Construction Codegen Enabled
Runtime filters: 1 of 1 Runtime Filter Published
- BuildRowsPartitionTime: 7s172ms
- BuildRowsPartitioned: 29.07M (29066489)
- HashBuckets: 67.11M (67108864)
- HashCollisions: 0 (0)
- HashTablesBuildTime: 4s108ms
- LargestPartitionPercent: 6 (6)
- MaxPartitionLevel: 0 (0)
- NumRepartitions: 0 (0)
- PartitionsCreated: 16 (16)
- PeakMemoryUsage: 21.12 KB (21632)
- RepartitionTime: 0.000ns
- SpilledPartitions: 0 (0)
EXCHANGE_NODE (id=6):(Total: 12s559ms, non-child: 12s559ms, % non-child: 100.00%)
- ConvertRowBatchTime: 2s958ms
- PeakMemoryUsage: 0
- RowsReturned: 29.07M (29066489)
- RowsReturnedRate: 2.31 M/sec
DataStreamReceiver:
BytesReceived(500.000ms): 0, 6.96 MB, 14.73 MB, 21.22 MB, 28.25 MB, 35.79 MB, 45.44 MB, 55.30 MB, 62.80 MB, 72.11 MB, 82.67 MB, 94.24 MB, 106.02 MB, 117.98 MB, 130.01 MB, 141.71 MB, 153.75 MB, 164.15 MB, 175.18 MB, 187.17 MB, 199.08 MB, 210.99 MB, 222.96 MB, 235.02 MB, 246.78 MB, 257.52 MB, 268.85 MB, 280.75 MB, 292.65 MB, 304.29 MB, 316.18 MB, 328.03 MB, 339.47 MB, 351.43 MB, 363.32 MB, 374.42 MB, 383.69 MB, 393.01 MB, 401.27 MB, 405.23 MB, 407.56 MB, 408.86 MB, 408.86 MB, 408.86 MB, 408.86 MB, 408.86 MB, 408.86 MB, 408.86 MB, 408.86 MB
- BytesReceived: 408.86 MB (428719786)
- DeserializeRowBatchTimer: 1s039ms
- FirstBatchArrivalWaitTime: 0.000ns
- PeakMemoryUsage: 9.16 MB (9605925)
- SendersBlockedTimer: 0.000ns
- SendersBlockedTotalTimer(*): 0.000ns
EXCHANGE_NODE (id=5):(Total: 4m56s, non-child: 4m56s, % non-child: 100.00%)
- ConvertRowBatchTime: 2s194ms
- PeakMemoryUsage: 0
- RowsReturned: 19.74M (19742720)
- RowsReturnedRate: 66.51 K/sec
DataStreamReceiver:
BytesReceived(1m4s): 7.95 MB, 11.24 MB, 15.42 MB, 19.05 MB, 22.62 MB, 27.06 MB, 32.66 MB, 38.24 MB, 42.86 MB, 47.22 MB, 51.55 MB, 55.88 MB, 60.06 MB, 64.38 MB, 68.68 MB, 72.89 MB, 77.33 MB, 81.66 MB, 85.73 MB, 90.08 MB, 94.49 MB, 98.90 MB, 103.25 MB, 107.56 MB, 111.98 MB, 116.40 MB, 120.83 MB, 125.23 MB, 129.58 MB, 134.04 MB, 138.47 MB, 142.79 MB, 147.04 MB, 151.12 MB, 155.68 MB, 160.66 MB, 165.62 MB, 170.58 MB, 175.48 MB, 180.33 MB, 185.10 MB, 189.79 MB, 194.46 MB, 199.24 MB, 203.96 MB, 208.67 MB, 213.39 MB, 217.78 MB, 222.25 MB, 227.00 MB, 231.75 MB, 236.47 MB, 241.24 MB, 246.00 MB, 250.79 MB, 255.57 MB, 260.35 MB, 265.13 MB, 269.91 MB, 274.72 MB, 279.00 MB, 283.33 MB, 287.69 MB
- BytesReceived: 291.66 MB (305826020)
- DeserializeRowBatchTimer: 1s758ms
- FirstBatchArrivalWaitTime: 684.861ms
- PeakMemoryUsage: 11.92 MB (12494925)
- SendersBlockedTimer: 6m37s
- SendersBlockedTotalTimer(*): 32m12s
Buffer pool:
- AllocTime: 528.670us
- CumulativeAllocationBytes: 4.69 MB (4915200)
- CumulativeAllocations: 59 (59)
- PeakReservation: 3.12 MB (3276800)
- PeakUnpinnedBytes: 0
- PeakUsedReservation: 3.12 MB (3276800)
- ReadIoBytes: 0
- ReadIoOps: 0 (0)
- ReadIoWaitTime: 0.000ns
- WriteIoBytes: 0
- WriteIoOps: 0 (0)
- WriteIoWaitTime: 0.000ns
Instance 2438b42f267ed44:3e1ac3c70000000d (host=bdpe722n4:22000):(Total: 60.833ms, non-child: 0.000ns, % non-child: 0.00%)
MemoryUsage(1m4s): 1.51 GB, 2.14 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB, 2.13 GB
ThreadUsage(1m4s): 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1
- AverageThreadTokens: 1.01
- BloomFilterBytes: 16.00 MB (16777216)
- PeakMemoryUsage: 2.14 GB (2296402971)
- PeakReservation: 2.11 GB (2268135424)
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 2.34 GB (2507859104)
- RowsProduced: 0 (0)
- TotalNetworkReceiveTime: 7m2s
- TotalNetworkSendTime: 0.000ns
- TotalStorageWaitTime: 0.000ns
- TotalThreadsInvoluntaryContextSwitches: 54 (54)
- TotalThreadsTotalWallClockTime: 23s068ms
- TotalThreadsSysTime: 648.538ms
- TotalThreadsUserTime: 12s929ms
- TotalThreadsVoluntaryContextSwitches: 17.43K (17431)
Fragment Instance Lifecycle Timings:
- OpenTime: 0.000ns
- ExecTreeOpenTime: 0.000ns
- PrepareTime: 60.685ms
- ExecTreePrepareTime: 893.957us
DataStreamSender (dst_id=7):(Total: 278.197us, non-child: 278.197us, % non-child: 100.00%)
- BytesSent: 0
- NetworkThroughput(*): 0.00 /sec
- OverallThroughput: 0.00 /sec
- PeakMemoryUsage: 5.12 KB (5240)
- RowsReturned: 0 (0)
- SerializeBatchTime: 0.000ns
- TransmitDataRPCTime: 0.000ns
- UncompressedRowBatchSize: 0
CodeGen:(Total: 407.057ms, non-child: 407.057ms, % non-child: 100.00%)
- CodegenTime: 7.258ms
- CompileTime: 26.797ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 1.84 MB (1929896)
- NumFunctions: 149 (149)
- NumInstructions: 4.39K (4394)
- OptimizationTime: 342.023ms
- PeakMemoryUsage: 2.15 MB (2249728)
- PrepareTime: 36.965ms
SORT_NODE (id=4):(Total: 892.289us, non-child: 84.246us, % non-child: 9.44%)
ExecOption: Codegen Enabled
- InsertBatchTime: 0.000ns
- PeakMemoryUsage: 4.00 KB (4096)
- RowsReturned: 0 (0)
- RowsReturnedRate: 0
- TuplePoolReclamations: 0 (0)
AGGREGATION_NODE (id=3):(Total: 808.043us, non-child: 0.000ns, % non-child: 0.00%)
ExecOption: Codegen Enabled
- BuildTime: 7m38s
- GetResultsTime: 0.000ns
- HTResizeTime: 9s600ms
- HashBuckets: 0 (0)
- LargestPartitionPercent: 0 (0)
- MaxPartitionLevel: 0 (0)
- NumRepartitions: 0 (0)
- PartitionsCreated: 16 (16)
- PeakMemoryUsage: 3.13 MB (3277926)
- RowsRepartitioned: 0 (0)
- RowsReturned: 0 (0)
- RowsReturnedRate: 0
- SpilledPartitions: 0 (0)
HASH_JOIN_NODE (id=2):(Total: 59m13s, non-child: 52m4s, % non-child: 87.95%)
ExecOption: Probe Side Codegen Enabled, Join Build-Side Prepared Asynchronously
- BuildRows: 28.59M (28592128)
- BuildTime: 9s937ms
- NumHashTableBuildsSkipped: 0 (0)
- PeakMemoryUsage: 2.11 GB (2265000166)
- ProbeRows: 19.42M (19423232)
- ProbeRowsPartitioned: 0 (0)
- ProbeTime: 51m47s
- RowsReturned: 9.36B (9362539520)
- RowsReturnedRate: 2.64 M/sec
Buffer pool:
- AllocTime: 9.410ms
- CumulativeAllocationBytes: 2.11 GB (2264924160)
- CumulativeAllocations: 584 (584)
- PeakReservation: 2.11 GB (2264924160)
- PeakUnpinnedBytes: 0
- PeakUsedReservation: 2.11 GB (2264924160)
- ReadIoBytes: 0
- ReadIoOps: 0 (0)
- ReadIoWaitTime: 0.000ns
- WriteIoBytes: 0
- WriteIoOps: 0 (0)
- WriteIoWaitTime: 0.000ns
Hash Join Builder (join_node_id=2):
ExecOption: Build Side Codegen Enabled, Hash Table Construction Codegen Enabled
Runtime filters: 1 of 1 Runtime Filter Published
- BuildRowsPartitionTime: 6s656ms
- BuildRowsPartitioned: 28.59M (28592128)
- HashBuckets: 67.11M (67108864)
- HashCollisions: 0 (0)
- HashTablesBuildTime: 3s246ms
- LargestPartitionPercent: 6 (6)
- MaxPartitionLevel: 0 (0)
- NumRepartitions: 0 (0)
- PartitionsCreated: 16 (16)
- PeakMemoryUsage: 21.12 KB (21632)
- RepartitionTime: 0.000ns
- SpilledPartitions: 0 (0)
EXCHANGE_NODE (id=6):(Total: 13s011ms, non-child: 13s011ms, % non-child: 100.00%)
- ConvertRowBatchTime: 3s049ms
- PeakMemoryUsage: 0
- RowsReturned: 28.59M (28592128)
- RowsReturnedRate: 2.20 M/sec
DataStreamReceiver:
BytesReceived(500.000ms): 7.57 MB, 15.13 MB, 21.41 MB, 28.33 MB, 35.80 MB, 45.54 MB, 54.95 MB, 62.50 MB, 71.47 MB, 81.98 MB, 93.58 MB, 105.10 MB, 117.00 MB, 128.77 MB, 140.32 MB, 152.05 MB, 162.20 MB, 173.23 MB, 185.00 MB, 196.73 MB, 208.40 MB, 220.22 MB, 232.05 MB, 243.48 MB, 253.96 MB, 265.26 MB, 277.00 MB, 288.71 MB, 300.16 MB, 311.85 MB, 323.50 MB, 334.73 MB, 346.51 MB, 358.21 MB, 368.93 MB, 378.08 MB, 387.25 MB, 395.23 MB, 398.76 MB, 401.06 MB, 402.18 MB, 402.18 MB, 402.18 MB, 402.18 MB, 402.18 MB, 402.18 MB, 402.18 MB
- BytesReceived: 402.18 MB (421714313)
- DeserializeRowBatchTimer: 1s124ms
- FirstBatchArrivalWaitTime: 0.000ns
- PeakMemoryUsage: 11.11 MB (11652300)
- SendersBlockedTimer: 0.000ns
- SendersBlockedTotalTimer(*): 0.000ns
EXCHANGE_NODE (id=5):(Total: 6m55s, non-child: 6m55s, % non-child: 100.00%)
- ConvertRowBatchTime: 2s171ms
- PeakMemoryUsage: 0
- RowsReturned: 19.42M (19423232)
- RowsReturnedRate: 46.76 K/sec
DataStreamReceiver:
BytesReceived(1m4s): 7.94 MB, 11.11 MB, 15.24 MB, 18.82 MB, 22.33 MB, 26.70 MB, 32.20 MB, 37.66 MB, 42.18 MB, 46.47 MB, 50.72 MB, 54.98 MB, 59.07 MB, 63.35 MB, 67.57 MB, 71.73 MB, 76.12 MB, 80.39 MB, 84.38 MB, 88.64 MB, 92.98 MB, 97.32 MB, 101.61 MB, 105.82 MB, 110.18 MB, 114.54 MB, 118.90 MB, 123.23 MB, 127.50 MB, 131.89 MB, 136.26 MB, 140.51 MB, 144.68 MB, 148.70 MB, 153.19 MB, 158.10 MB, 162.97 MB, 167.85 MB, 172.66 MB, 177.42 MB, 182.12 MB, 186.71 MB, 191.31 MB, 196.02 MB, 200.66 MB, 205.29 MB, 209.95 MB, 214.27 MB, 218.66 MB, 223.34 MB, 227.99 MB, 232.64 MB, 237.33 MB, 242.02 MB, 246.73 MB, 251.43 MB, 256.12 MB, 260.82 MB, 265.55 MB, 270.28 MB, 274.49 MB, 278.76 MB, 283.04 MB
- BytesReceived: 286.91 MB (300843911)
- DeserializeRowBatchTimer: 1s677ms
- FirstBatchArrivalWaitTime: 622.022ms
- PeakMemoryUsage: 11.92 MB (12494925)
- SendersBlockedTimer: 0.000ns
- SendersBlockedTotalTimer(*): 0.000ns
Buffer pool:
- AllocTime: 713.547us
- CumulativeAllocationBytes: 4.56 MB (4784128)
- CumulativeAllocations: 57 (57)
- PeakReservation: 3.06 MB (3211264)
- PeakUnpinnedBytes: 0
- PeakUsedReservation: 3.06 MB (3211264)
- ReadIoBytes: 0
- ReadIoOps: 0 (0)
- ReadIoWaitTime: 0.000ns
- WriteIoBytes: 0
- WriteIoOps: 0 (0)
- WriteIoWaitTime: 0.000ns
Averaged Fragment F00:
Fragment F00:
Instance 2438b42f267ed44:3e1ac3c700000004 (host=bdpe611n1:22000):(Total: 55.453ms, non-child: 0.000ns, % non-child: 0.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:386/4.18 GB
Filter 0 arrival: 21s284ms
MemoryUsage(1m4s): 143.94 MB, 154.84 MB, 154.86 MB, 154.84 MB, 154.85 MB, 154.86 MB, 154.28 MB, 154.73 MB, 154.85 MB, 153.83 MB, 166.99 MB, 191.46 MB, 191.65 MB, 191.81 MB, 196.52 MB, 195.88 MB, 195.24 MB, 195.23 MB, 195.07 MB, 195.59 MB, 201.53 MB, 205.04 MB, 199.80 MB, 199.99 MB, 200.11 MB, 201.09 MB, 202.76 MB, 200.50 MB, 203.26 MB, 210.82 MB, 210.56 MB, 212.15 MB, 212.13 MB, 209.29 MB, 207.42 MB, 207.43 MB, 207.43 MB, 207.41 MB, 207.14 MB, 211.74 MB, 210.14 MB, 209.88 MB, 207.49 MB, 188.98 MB, 189.98 MB, 191.98 MB, 189.05 MB, 189.91 MB, 187.61 MB, 187.70 MB, 186.83 MB, 187.44 MB, 187.41 MB, 185.74 MB, 192.02 MB, 192.00 MB, 190.85 MB, 193.70 MB, 192.45 MB, 191.96 MB, 191.04 MB, 195.83 MB, 195.18 MB
ThreadUsage(1m4s): 39, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40
- AverageThreadTokens: 39.99
- BloomFilterBytes: 16.00 MB (16777216)
- PeakMemoryUsage: 216.37 MB (226881728)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 2.36 GB (2530721031)
- RowsProduced: 21.88M (21879143)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 1h7m
- TotalStorageWaitTime: 52s749ms
- TotalThreadsInvoluntaryContextSwitches: 0 (0)
- TotalThreadsTotalWallClockTime: 32.249ms
- TotalThreadsSysTime: 0.000ns
- TotalThreadsUserTime: 31.550ms
- TotalThreadsVoluntaryContextSwitches: 3 (3)
Fragment Instance Lifecycle Timings:
- ExecTime: 0.000ns
- ExecTreeExecTime: 1s504ms
- OpenTime: 32.252ms
- ExecTreeOpenTime: 915.429us
- PrepareTime: 23.084ms
- ExecTreePrepareTime: 4.885ms
DataStreamSender (dst_id=5):(Total: 10s820ms, non-child: 10s820ms, % non-child: 100.00%)
- BytesSent: 323.04 MB (338729014)
- NetworkThroughput(*): 129.37 MB/sec
- OverallThroughput: 29.86 MB/sec
- PeakMemoryUsage: 177.62 KB (181880)
- RowsReturned: 21.88M (21878119)
- SerializeBatchTime: 8s315ms
- TransmitDataRPCTime: 2s497ms
- UncompressedRowBatchSize: 438.21 MB (459496759)
CodeGen:(Total: 47.723ms, non-child: 47.723ms, % non-child: 100.00%)
- CodegenTime: 1.816ms
- CompileTime: 4.683ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 1.84 MB (1929896)
- NumFunctions: 29 (29)
- NumInstructions: 437 (437)
- OptimizationTime: 25.986ms
- PeakMemoryUsage: 218.50 KB (223744)
- PrepareTime: 16.431ms
HDFS_SCAN_NODE (id=0):(Total: 1s476ms, non-child: 1s476ms, % non-child: 100.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:386/4.18 GB
ExecOption: PARQUET Codegen Enabled
Runtime filters: Not all filters arrived (arrived: [], missing [0]), waited for 963ms
BytesRead(1m4s): 78.87 MB, 82.05 MB, 82.05 MB, 82.05 MB, 82.05 MB, 82.05 MB, 82.05 MB, 84.65 MB, 84.73 MB, 85.20 MB, 113.53 MB, 136.74 MB, 136.88 MB, 138.24 MB, 141.97 MB, 141.99 MB, 143.33 MB, 143.33 MB, 143.33 MB, 149.84 MB, 158.62 MB, 165.59 MB, 173.98 MB, 179.09 MB, 181.20 MB, 183.28 MB, 184.44 MB, 186.89 MB, 192.38 MB, 203.07 MB, 209.82 MB, 221.26 MB, 230.63 MB, 234.38 MB, 237.83 MB, 237.87 MB, 237.87 MB, 237.87 MB, 237.87 MB, 242.55 MB, 245.86 MB, 252.57 MB, 253.92 MB, 274.27 MB, 285.76 MB, 287.18 MB, 288.10 MB, 289.84 MB, 290.32 MB, 291.22 MB, 291.68 MB, 292.60 MB, 292.60 MB, 293.47 MB, 299.67 MB, 302.75 MB, 309.14 MB, 317.08 MB, 321.24 MB, 324.43 MB, 325.38 MB, 331.44 MB, 332.20 MB
- FooterProcessingTime: (Avg: 123.978ms ; Min: 998.381us ; Max: 1s024ms ; Number of samples: 162)
- AverageHdfsReadThreadConcurrency: 0.00
- AverageScannerThreadConcurrency: 38.99
- BytesRead: 337.49 MB (353888074)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 0
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 0
- CachedFileHandlesHitCount: 0 (0)
- CachedFileHandlesMissCount: 0 (0)
- CollectionItemsRead: 0 (0)
- DecompressionTime: 0.000ns
- MaxCompressedTextFileLength: 0
- NumColumns: 2 (2)
- NumDictFilteredRowGroups: 0 (0)
- NumDisksAccessed: 0 (0)
- NumRowGroups: 162 (162)
- NumScannerThreadsStarted: 39 (39)
- NumScannersWithNoReads: 0 (0)
- NumStatsFilteredRowGroups: 0 (0)
- PeakMemoryUsage: 200.19 MB (209912215)
- PerReadThreadRawHdfsThroughput: 61.99 MB/sec
- RemoteScanRanges: 0 (0)
- RowBatchQueueGetWaitTime: 0.000ns
- RowBatchQueuePutWaitTime: 0.000ns
- RowsRead: 21.97M (21970270)
- RowsReturned: 21.88M (21879143)
- RowsReturnedRate: 14.82 M/sec
- ScanRangesComplete: 123 (123)
- ScannerThreadsInvoluntaryContextSwitches: 0 (0)
- ScannerThreadsTotalWallClockTime: 0.000ns
- MaterializeTupleTime(*): 6s513ms
- ScannerThreadsSysTime: 0.000ns
- ScannerThreadsUserTime: 0.000ns
- ScannerThreadsVoluntaryContextSwitches: 0 (0)
- TotalRawHdfsReadTime(*): 5s444ms
- TotalReadThroughput: 85.28 KB/sec
Filter 0 (16.00 MB):
- Rows processed: 1.93M (1931264)
- Rows rejected: 0 (0)
- Rows total: 17.32M (17318247)
Instance 2438b42f267ed44:3e1ac3c700000002 (host=bdpe611n2:22000):(Total: 84.539ms, non-child: 0.000ns, % non-child: 0.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:385/4.17 GB
Filter 0 arrival: 21s443ms
MemoryUsage(1m4s): 150.81 MB, 163.60 MB, 163.59 MB, 163.58 MB, 163.60 MB, 163.60 MB, 163.59 MB, 163.61 MB, 163.60 MB, 163.16 MB, 163.58 MB, 163.59 MB, 162.54 MB, 167.00 MB, 190.04 MB, 190.06 MB, 190.06 MB, 190.05 MB, 190.07 MB, 192.55 MB, 192.56 MB, 192.55 MB, 192.96 MB, 194.76 MB, 194.78 MB, 195.84 MB, 214.78 MB, 213.38 MB, 204.45 MB, 208.05 MB, 208.28 MB, 208.27 MB, 208.30 MB, 208.29 MB, 208.28 MB, 208.09 MB, 208.86 MB, 212.46 MB, 208.51 MB, 208.58 MB, 197.71 MB, 195.12 MB, 195.36 MB, 193.94 MB, 193.15 MB, 193.14 MB, 192.79 MB, 200.50 MB, 200.50 MB, 202.30 MB, 201.78 MB, 208.82 MB, 208.41 MB, 209.08 MB, 210.73 MB, 211.13 MB, 211.13 MB, 211.77 MB, 213.36 MB, 210.88 MB, 209.36 MB, 209.28 MB, 197.86 MB
ThreadUsage(1m4s): 39, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40
- AverageThreadTokens: 39.99
- BloomFilterBytes: 16.00 MB (16777216)
- PeakMemoryUsage: 216.79 MB (227319848)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 2.35 GB (2523002980)
- RowsProduced: 19.28M (19282007)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 1h7m
- TotalStorageWaitTime: 1m
- TotalThreadsInvoluntaryContextSwitches: 1 (1)
- TotalThreadsTotalWallClockTime: 40.998ms
- TotalThreadsSysTime: 970.000us
- TotalThreadsUserTime: 39.651ms
- TotalThreadsVoluntaryContextSwitches: 0 (0)
Fragment Instance Lifecycle Timings:
- ExecTime: 0.000ns
- ExecTreeExecTime: 1s617ms
- OpenTime: 41.001ms
- ExecTreeOpenTime: 722.978us
- PrepareTime: 43.375ms
- ExecTreePrepareTime: 9.931ms
DataStreamSender (dst_id=5):(Total: 12s640ms, non-child: 12s640ms, % non-child: 100.00%)
- BytesSent: 285.91 MB (299796899)
- NetworkThroughput(*): 90.33 MB/sec
- OverallThroughput: 22.62 MB/sec
- PeakMemoryUsage: 177.62 KB (181880)
- RowsReturned: 19.28M (19280983)
- SerializeBatchTime: 9s466ms
- TransmitDataRPCTime: 3s165ms
- UncompressedRowBatchSize: 386.17 MB (404924313)
CodeGen:(Total: 70.323ms, non-child: 70.323ms, % non-child: 100.00%)
- CodegenTime: 3.270ms
- CompileTime: 3.944ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 1.84 MB (1929896)
- NumFunctions: 29 (29)
- NumInstructions: 437 (437)
- OptimizationTime: 34.725ms
- PeakMemoryUsage: 218.50 KB (223744)
- PrepareTime: 30.589ms
HDFS_SCAN_NODE (id=0):(Total: 1s595ms, non-child: 1s595ms, % non-child: 100.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:385/4.17 GB
ExecOption: PARQUET Codegen Enabled
Runtime filters: Not all filters arrived (arrived: [], missing [0]), waited for 923ms
BytesRead(1m4s): 82.82 MB, 87.08 MB, 87.08 MB, 87.08 MB, 87.08 MB, 87.08 MB, 87.08 MB, 87.08 MB, 87.08 MB, 87.24 MB, 88.44 MB, 88.44 MB, 88.44 MB, 112.19 MB, 132.16 MB, 132.16 MB, 132.16 MB, 132.16 MB, 132.49 MB, 134.82 MB, 134.82 MB, 134.82 MB, 135.88 MB, 137.49 MB, 137.49 MB, 145.43 MB, 163.21 MB, 164.69 MB, 182.08 MB, 187.12 MB, 188.30 MB, 188.30 MB, 188.30 MB, 188.30 MB, 188.30 MB, 188.30 MB, 194.83 MB, 197.84 MB, 200.01 MB, 216.59 MB, 228.42 MB, 233.02 MB, 233.33 MB, 233.61 MB, 234.67 MB, 234.67 MB, 236.64 MB, 243.52 MB, 243.52 MB, 246.88 MB, 259.84 MB, 276.82 MB, 285.14 MB, 290.53 MB, 293.70 MB, 294.16 MB, 294.16 MB, 295.38 MB, 296.84 MB, 297.51 MB, 300.96 MB, 308.61 MB, 313.48 MB
- FooterProcessingTime: (Avg: 167.786ms ; Min: 1.095ms ; Max: 1s132ms ; Number of samples: 153)
- AverageHdfsReadThreadConcurrency: 0.00
- AverageScannerThreadConcurrency: 39.00
- BytesRead: 323.72 MB (339446916)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 0
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 0
- CachedFileHandlesHitCount: 0 (0)
- CachedFileHandlesMissCount: 0 (0)
- CollectionItemsRead: 0 (0)
- DecompressionTime: 0.000ns
- MaxCompressedTextFileLength: 0
- NumColumns: 2 (2)
- NumDictFilteredRowGroups: 0 (0)
- NumDisksAccessed: 0 (0)
- NumRowGroups: 153 (153)
- NumScannerThreadsStarted: 39 (39)
- NumScannersWithNoReads: 0 (0)
- NumStatsFilteredRowGroups: 0 (0)
- PeakMemoryUsage: 200.59 MB (210333951)
- PerReadThreadRawHdfsThroughput: 72.86 MB/sec
- RemoteScanRanges: 0 (0)
- RowBatchQueueGetWaitTime: 0.000ns
- RowBatchQueuePutWaitTime: 0.000ns
- RowsRead: 19.37M (19367427)
- RowsReturned: 19.28M (19282007)
- RowsReturnedRate: 12.08 M/sec
- ScanRangesComplete: 114 (114)
- ScannerThreadsInvoluntaryContextSwitches: 0 (0)
- ScannerThreadsTotalWallClockTime: 0.000ns
- MaterializeTupleTime(*): 7s403ms
- ScannerThreadsSysTime: 0.000ns
- ScannerThreadsUserTime: 0.000ns
- ScannerThreadsVoluntaryContextSwitches: 0 (0)
- TotalRawHdfsReadTime(*): 4s443ms
- TotalReadThroughput: 81.79 KB/sec
Filter 0 (16.00 MB):
- Rows processed: 1.75M (1754112)
- Rows rejected: 0 (0)
- Rows total: 16.09M (16090035)
Instance 2438b42f267ed44:3e1ac3c700000001 (host=bdpe611n3:22000):(Total: 85.005ms, non-child: 0.000ns, % non-child: 0.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:384/4.18 GB
Filter 0 arrival: 21s619ms
MemoryUsage(1m4s): 134.97 MB, 145.25 MB, 145.23 MB, 145.21 MB, 145.24 MB, 145.24 MB, 145.22 MB, 145.21 MB, 145.22 MB, 145.21 MB, 144.74 MB, 139.05 MB, 175.44 MB, 176.40 MB, 176.37 MB, 176.38 MB, 176.26 MB, 176.45 MB, 177.04 MB, 178.64 MB, 178.64 MB, 178.64 MB, 178.61 MB, 178.17 MB, 173.00 MB, 187.41 MB, 191.83 MB, 189.62 MB, 187.75 MB, 187.95 MB, 188.47 MB, 194.64 MB, 197.19 MB, 197.16 MB, 197.17 MB, 197.17 MB, 196.54 MB, 198.07 MB, 202.24 MB, 194.24 MB, 184.85 MB, 176.52 MB, 179.96 MB, 183.74 MB, 183.73 MB, 182.92 MB, 183.73 MB, 187.84 MB, 194.18 MB, 193.12 MB, 192.58 MB, 194.24 MB, 196.05 MB, 195.43 MB, 191.75 MB, 192.35 MB, 192.34 MB, 192.33 MB, 192.56 MB, 196.64 MB, 196.46 MB, 197.65 MB, 194.81 MB
ThreadUsage(1m4s): 39, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40
- AverageThreadTokens: 39.99
- BloomFilterBytes: 16.00 MB (16777216)
- PeakMemoryUsage: 205.33 MB (215307504)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 2.33 GB (2505825451)
- RowsProduced: 19.06M (19056293)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 1h7m
- TotalStorageWaitTime: 45s650ms
- TotalThreadsInvoluntaryContextSwitches: 0 (0)
- TotalThreadsTotalWallClockTime: 43.650ms
- TotalThreadsSysTime: 1.981ms
- TotalThreadsUserTime: 40.673ms
- TotalThreadsVoluntaryContextSwitches: 1 (1)
Fragment Instance Lifecycle Timings:
- ExecTime: 0.000ns
- ExecTreeExecTime: 1s611ms
- OpenTime: 43.660ms
- ExecTreeOpenTime: 770.272us
- PrepareTime: 41.072ms
- ExecTreePrepareTime: 10.438ms
DataStreamSender (dst_id=5):(Total: 13s402ms, non-child: 13s402ms, % non-child: 100.00%)
- BytesSent: 280.48 MB (294099431)
- NetworkThroughput(*): 92.33 MB/sec
- OverallThroughput: 20.93 MB/sec
- PeakMemoryUsage: 177.62 KB (181880)
- RowsReturned: 19.06M (19055269)
- SerializeBatchTime: 10s355ms
- TransmitDataRPCTime: 3s037ms
- UncompressedRowBatchSize: 381.63 MB (400170968)
CodeGen:(Total: 70.305ms, non-child: 70.305ms, % non-child: 100.00%)
- CodegenTime: 3.083ms
- CompileTime: 4.354ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 1.84 MB (1929896)
- NumFunctions: 29 (29)
- NumInstructions: 437 (437)
- OptimizationTime: 37.275ms
- PeakMemoryUsage: 218.50 KB (223744)
- PrepareTime: 27.772ms
HDFS_SCAN_NODE (id=0):(Total: 1s589ms, non-child: 1s589ms, % non-child: 100.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:384/4.18 GB
ExecOption: PARQUET Codegen Enabled
Runtime filters: Not all filters arrived (arrived: [], missing [0]), waited for 923ms
BytesRead(1m4s): 73.07 MB, 75.85 MB, 75.85 MB, 75.85 MB, 75.85 MB, 75.85 MB, 75.85 MB, 75.85 MB, 75.85 MB, 75.85 MB, 75.85 MB, 88.81 MB, 135.10 MB, 135.82 MB, 135.82 MB, 135.82 MB, 135.82 MB, 137.02 MB, 138.45 MB, 139.87 MB, 139.87 MB, 139.87 MB, 139.87 MB, 140.02 MB, 144.54 MB, 161.79 MB, 165.31 MB, 167.45 MB, 170.50 MB, 170.86 MB, 173.03 MB, 180.97 MB, 182.62 MB, 182.62 MB, 182.62 MB, 182.62 MB, 183.09 MB, 188.19 MB, 193.27 MB, 201.88 MB, 216.28 MB, 223.21 MB, 231.16 MB, 235.53 MB, 235.53 MB, 236.15 MB, 238.19 MB, 242.95 MB, 249.19 MB, 250.34 MB, 261.98 MB, 275.16 MB, 281.47 MB, 282.85 MB, 287.02 MB, 288.56 MB, 288.56 MB, 288.56 MB, 289.35 MB, 292.31 MB, 294.73 MB, 302.91 MB, 311.58 MB
- FooterProcessingTime: (Avg: 116.132ms ; Min: 1.215ms ; Max: 845.660ms ; Number of samples: 161)
- AverageHdfsReadThreadConcurrency: 0.00
- AverageScannerThreadConcurrency: 38.99
- BytesRead: 319.01 MB (334507959)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 0
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 0
- CachedFileHandlesHitCount: 0 (0)
- CachedFileHandlesMissCount: 0 (0)
- CollectionItemsRead: 0 (0)
- DecompressionTime: 0.000ns
- MaxCompressedTextFileLength: 0
- NumColumns: 2 (2)
- NumDictFilteredRowGroups: 0 (0)
- NumDisksAccessed: 0 (0)
- NumRowGroups: 161 (161)
- NumScannerThreadsStarted: 39 (39)
- NumScannersWithNoReads: 0 (0)
- NumStatsFilteredRowGroups: 0 (0)
- PeakMemoryUsage: 189.15 MB (198337991)
- PerReadThreadRawHdfsThroughput: 75.30 MB/sec
- RemoteScanRanges: 0 (0)
- RowBatchQueueGetWaitTime: 0.000ns
- RowBatchQueuePutWaitTime: 0.000ns
- RowsRead: 19.15M (19145381)
- RowsReturned: 19.06M (19056293)
- RowsReturnedRate: 11.99 M/sec
- ScanRangesComplete: 122 (122)
- ScannerThreadsInvoluntaryContextSwitches: 0 (0)
- ScannerThreadsTotalWallClockTime: 0.000ns
- MaterializeTupleTime(*): 9s050ms
- ScannerThreadsSysTime: 0.000ns
- ScannerThreadsUserTime: 0.000ns
- ScannerThreadsVoluntaryContextSwitches: 0 (0)
- TotalRawHdfsReadTime(*): 4s236ms
- TotalReadThroughput: 80.52 KB/sec
Filter 0 (16.00 MB):
- Rows processed: 1.90M (1902592)
- Rows rejected: 0 (0)
- Rows total: 15.55M (15552165)
Instance 2438b42f267ed44:3e1ac3c700000005 (host=bdpe611n4:22000):(Total: 71.177ms, non-child: 0.000ns, % non-child: 0.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:383/4.18 GB
Filter 0 arrival: 21s843ms
MemoryUsage(1m4s): 139.40 MB, 151.29 MB, 151.32 MB, 151.29 MB, 151.32 MB, 151.31 MB, 151.31 MB, 151.31 MB, 151.30 MB, 151.32 MB, 151.29 MB, 151.30 MB, 151.29 MB, 151.31 MB, 146.82 MB, 162.00 MB, 176.59 MB, 176.75 MB, 176.78 MB, 176.77 MB, 176.78 MB, 176.41 MB, 178.32 MB, 178.72 MB, 178.72 MB, 178.73 MB, 178.72 MB, 177.66 MB, 175.91 MB, 191.75 MB, 184.02 MB, 181.95 MB, 177.38 MB, 182.42 MB, 182.99 MB, 183.21 MB, 183.22 MB, 183.20 MB, 183.23 MB, 184.59 MB, 184.76 MB, 192.95 MB, 196.09 MB, 195.33 MB, 194.24 MB, 193.29 MB, 194.14 MB, 194.14 MB, 193.45 MB, 196.21 MB, 196.09 MB, 198.99 MB, 203.00 MB, 198.52 MB, 191.68 MB, 190.21 MB, 192.16 MB, 192.28 MB, 192.11 MB, 192.39 MB, 192.12 MB, 192.84 MB, 194.61 MB
ThreadUsage(1m4s): 39, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40
- AverageThreadTokens: 39.99
- BloomFilterBytes: 16.00 MB (16777216)
- PeakMemoryUsage: 203.41 MB (213288576)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 2.34 GB (2515248771)
- RowsProduced: 18.56M (18555600)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 1h7m
- TotalStorageWaitTime: 1m
- TotalThreadsInvoluntaryContextSwitches: 0 (0)
- TotalThreadsTotalWallClockTime: 35.550ms
- TotalThreadsSysTime: 0.000ns
- TotalThreadsUserTime: 35.291ms
- TotalThreadsVoluntaryContextSwitches: 0 (0)
Fragment Instance Lifecycle Timings:
- ExecTime: 0.000ns
- ExecTreeExecTime: 1s504ms
- OpenTime: 35.553ms
- ExecTreeOpenTime: 735.786us
- PrepareTime: 35.470ms
- ExecTreePrepareTime: 8.224ms
DataStreamSender (dst_id=5):(Total: 10s272ms, non-child: 10s272ms, % non-child: 100.00%)
- BytesSent: 273.41 MB (286691030)
- NetworkThroughput(*): 98.09 MB/sec
- OverallThroughput: 26.62 MB/sec
- PeakMemoryUsage: 177.62 KB (181880)
- RowsReturned: 18.55M (18554576)
- SerializeBatchTime: 7s477ms
- TransmitDataRPCTime: 2s787ms
- UncompressedRowBatchSize: 371.62 MB (389673155)
CodeGen:(Total: 58.194ms, non-child: 58.194ms, % non-child: 100.00%)
- CodegenTime: 2.512ms
- CompileTime: 3.759ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 1.84 MB (1929896)
- NumFunctions: 29 (29)
- NumInstructions: 437 (437)
- OptimizationTime: 30.129ms
- PeakMemoryUsage: 218.50 KB (223744)
- PrepareTime: 23.586ms
HDFS_SCAN_NODE (id=0):(Total: 1s489ms, non-child: 1s489ms, % non-child: 100.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:383/4.18 GB
ExecOption: PARQUET Codegen Enabled
Runtime filters: Not all filters arrived (arrived: [], missing [0]), waited for 943ms
BytesRead(1m4s): 80.26 MB, 84.15 MB, 84.15 MB, 84.15 MB, 84.15 MB, 84.15 MB, 84.15 MB, 84.15 MB, 84.15 MB, 84.15 MB, 84.15 MB, 84.15 MB, 84.15 MB, 84.15 MB, 85.37 MB, 120.85 MB, 133.32 MB, 133.46 MB, 133.46 MB, 133.46 MB, 133.46 MB, 133.46 MB, 135.87 MB, 136.14 MB, 136.14 MB, 136.14 MB, 136.14 MB, 136.19 MB, 141.98 MB, 161.14 MB, 162.47 MB, 171.68 MB, 175.43 MB, 182.38 MB, 184.33 MB, 185.02 MB, 185.02 MB, 185.02 MB, 185.02 MB, 186.91 MB, 192.14 MB, 202.80 MB, 208.74 MB, 220.51 MB, 233.96 MB, 240.41 MB, 241.97 MB, 241.97 MB, 242.08 MB, 245.66 MB, 245.66 MB, 252.62 MB, 256.43 MB, 261.55 MB, 271.20 MB, 283.57 MB, 286.51 MB, 287.54 MB, 288.70 MB, 288.94 MB, 288.94 MB, 291.34 MB, 292.95 MB
- FooterProcessingTime: (Avg: 161.210ms ; Min: 906.207us ; Max: 998.197ms ; Number of samples: 150)
- AverageHdfsReadThreadConcurrency: 0.00
- AverageScannerThreadConcurrency: 38.99
- BytesRead: 292.95 MB (307182818)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 0
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 0
- CachedFileHandlesHitCount: 0 (0)
- CachedFileHandlesMissCount: 0 (0)
- CollectionItemsRead: 0 (0)
- DecompressionTime: 0.000ns
- MaxCompressedTextFileLength: 0
- NumColumns: 2 (2)
- NumDictFilteredRowGroups: 0 (0)
- NumDisksAccessed: 0 (0)
- NumRowGroups: 150 (150)
- NumScannerThreadsStarted: 39 (39)
- NumScannersWithNoReads: 0 (0)
- NumStatsFilteredRowGroups: 0 (0)
- PeakMemoryUsage: 187.24 MB (196335447)
- PerReadThreadRawHdfsThroughput: 57.66 MB/sec
- RemoteScanRanges: 0 (0)
- RowBatchQueueGetWaitTime: 0.000ns
- RowBatchQueuePutWaitTime: 0.000ns
- RowsRead: 18.65M (18646736)
- RowsReturned: 18.56M (18555600)
- RowsReturnedRate: 12.46 M/sec
- ScanRangesComplete: 111 (111)
- ScannerThreadsInvoluntaryContextSwitches: 0 (0)
- ScannerThreadsTotalWallClockTime: 0.000ns
- MaterializeTupleTime(*): 5s686ms
- ScannerThreadsSysTime: 0.000ns
- ScannerThreadsUserTime: 0.000ns
- ScannerThreadsVoluntaryContextSwitches: 0 (0)
- TotalRawHdfsReadTime(*): 5s080ms
- TotalReadThroughput: 74.01 KB/sec
Filter 0 (16.00 MB):
- Rows processed: 1.68M (1679360)
- Rows rejected: 0 (0)
- Rows total: 13.77M (13769424)
Instance 2438b42f267ed44:3e1ac3c700000003 (host=bdpe722n4:22000):(Total: 82.178ms, non-child: 0.000ns, % non-child: 0.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:385/4.17 GB
Filter 0 arrival: 21s965ms
MemoryUsage(1m4s): 152.01 MB, 163.30 MB, 163.31 MB, 163.31 MB, 162.33 MB, 160.76 MB, 161.08 MB, 161.11 MB, 161.11 MB, 161.09 MB, 157.17 MB, 178.12 MB, 194.19 MB, 194.21 MB, 194.21 MB, 193.78 MB, 194.19 MB, 198.23 MB, 198.22 MB, 198.23 MB, 198.24 MB, 198.12 MB, 198.66 MB, 210.96 MB, 209.29 MB, 192.40 MB, 190.51 MB, 190.58 MB, 191.98 MB, 194.02 MB, 194.48 MB, 194.46 MB, 194.45 MB, 194.45 MB, 194.46 MB, 192.65 MB, 198.75 MB, 194.95 MB, 204.31 MB, 192.23 MB, 189.19 MB, 193.58 MB, 194.35 MB, 194.33 MB, 194.33 MB, 194.34 MB, 195.14 MB, 203.49 MB, 203.70 MB, 205.54 MB, 197.37 MB, 196.96 MB, 196.98 MB, 196.47 MB, 192.56 MB, 192.96 MB, 188.01 MB, 188.65 MB, 188.66 MB, 188.66 MB, 189.08 MB, 190.12 MB, 190.84 MB
ThreadUsage(1m4s): 39, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40
- AverageThreadTokens: 39.99
- BloomFilterBytes: 16.00 MB (16777216)
- PeakMemoryUsage: 212.08 MB (222381024)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 2.34 GB (2507859104)
- RowsProduced: 18.98M (18978066)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 1h7m
- TotalStorageWaitTime: 50s972ms
- TotalThreadsInvoluntaryContextSwitches: 1 (1)
- TotalThreadsTotalWallClockTime: 43.018ms
- TotalThreadsSysTime: 0.000ns
- TotalThreadsUserTime: 42.330ms
- TotalThreadsVoluntaryContextSwitches: 1 (1)
Fragment Instance Lifecycle Timings:
- ExecTime: 0.000ns
- ExecTreeExecTime: 1s645ms
- OpenTime: 43.020ms
- ExecTreeOpenTime: 685.669us
- PrepareTime: 39.009ms
- ExecTreePrepareTime: 9.755ms
DataStreamSender (dst_id=5):(Total: 13s018ms, non-child: 13s018ms, % non-child: 100.00%)
- BytesSent: 280.86 MB (294500459)
- NetworkThroughput(*): 120.91 MB/sec
- OverallThroughput: 21.57 MB/sec
- PeakMemoryUsage: 177.62 KB (181880)
- RowsReturned: 18.98M (18977042)
- SerializeBatchTime: 10s686ms
- TransmitDataRPCTime: 2s322ms
- UncompressedRowBatchSize: 380.09 MB (398552808)
CodeGen:(Total: 68.436ms, non-child: 68.436ms, % non-child: 100.00%)
- CodegenTime: 3.099ms
- CompileTime: 3.775ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 1.84 MB (1929896)
- NumFunctions: 29 (29)
- NumInstructions: 437 (437)
- OptimizationTime: 37.348ms
- PeakMemoryUsage: 218.50 KB (223744)
- PrepareTime: 26.399ms
HDFS_SCAN_NODE (id=0):(Total: 1s622ms, non-child: 1s622ms, % non-child: 100.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:385/4.17 GB
ExecOption: PARQUET Codegen Enabled
Runtime filters: Not all filters arrived (arrived: [], missing [0]), waited for 923ms
BytesRead(1m4s): 80.94 MB, 84.23 MB, 84.23 MB, 84.23 MB, 84.23 MB, 85.30 MB, 85.60 MB, 85.60 MB, 85.60 MB, 85.60 MB, 85.78 MB, 119.30 MB, 134.23 MB, 134.23 MB, 134.23 MB, 135.07 MB, 136.34 MB, 139.31 MB, 139.31 MB, 139.31 MB, 139.31 MB, 139.31 MB, 144.43 MB, 159.83 MB, 160.44 MB, 165.29 MB, 172.66 MB, 173.43 MB, 175.33 MB, 177.99 MB, 179.39 MB, 179.39 MB, 179.39 MB, 179.39 MB, 179.39 MB, 181.08 MB, 188.56 MB, 192.87 MB, 218.82 MB, 228.29 MB, 230.98 MB, 235.85 MB, 237.00 MB, 237.00 MB, 237.00 MB, 237.00 MB, 238.84 MB, 247.04 MB, 252.21 MB, 263.47 MB, 273.90 MB, 280.85 MB, 282.80 MB, 283.47 MB, 285.87 MB, 286.22 MB, 287.03 MB, 287.57 MB, 287.57 MB, 287.57 MB, 288.69 MB, 292.91 MB, 298.01 MB
- FooterProcessingTime: (Avg: 158.157ms ; Min: 1.028ms ; Max: 901.023ms ; Number of samples: 150)
- AverageHdfsReadThreadConcurrency: 0.00
- AverageScannerThreadConcurrency: 38.99
- BytesRead: 306.69 MB (321585641)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 0
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 0
- CachedFileHandlesHitCount: 0 (0)
- CachedFileHandlesMissCount: 0 (0)
- CollectionItemsRead: 0 (0)
- DecompressionTime: 0.000ns
- MaxCompressedTextFileLength: 0
- NumColumns: 2 (2)
- NumDictFilteredRowGroups: 0 (0)
- NumDisksAccessed: 0 (0)
- NumRowGroups: 150 (150)
- NumScannerThreadsStarted: 39 (39)
- NumScannersWithNoReads: 0 (0)
- NumStatsFilteredRowGroups: 0 (0)
- PeakMemoryUsage: 195.90 MB (205411511)
- PerReadThreadRawHdfsThroughput: 66.40 MB/sec
- RemoteScanRanges: 0 (0)
- RowBatchQueueGetWaitTime: 0.000ns
- RowBatchQueuePutWaitTime: 0.000ns
- RowsRead: 19.07M (19073298)
- RowsReturned: 18.98M (18978066)
- RowsReturnedRate: 11.70 M/sec
- ScanRangesComplete: 111 (111)
- ScannerThreadsInvoluntaryContextSwitches: 0 (0)
- ScannerThreadsTotalWallClockTime: 0.000ns
- MaterializeTupleTime(*): 7s622ms
- ScannerThreadsSysTime: 0.000ns
- ScannerThreadsUserTime: 0.000ns
- ScannerThreadsVoluntaryContextSwitches: 0 (0)
- TotalRawHdfsReadTime(*): 4s619ms
- TotalReadThroughput: 77.49 KB/sec
Filter 0 (16.00 MB):
- Rows processed: 1.74M (1743872)
- Rows rejected: 0 (0)
- Rows total: 14.88M (14875904)
Averaged Fragment F01:
Fragment F01:
Instance 2438b42f267ed44:3e1ac3c700000006 (host=bdpe611n1:22000):(Total: 18s739ms, non-child: 7s511ms, % non-child: 40.09%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:388/3.41 GB
MemoryUsage(500.000ms): 1.76 MB, 6.03 MB, 842.40 KB, 12.20 MB, 1.27 MB, 1.35 MB, 754.27 KB, 2.28 MB, 6.07 MB, 16.39 MB, 15.51 MB, 5.56 MB, 5.44 MB, 9.52 MB, 3.21 MB, 12.83 MB, 40.04 MB, 81.45 MB, 83.83 MB, 88.26 MB, 81.70 MB, 86.63 MB, 81.17 MB, 61.96 MB, 55.29 MB, 67.06 MB, 76.19 MB, 67.45 MB, 64.55 MB, 74.25 MB, 82.55 MB, 84.46 MB, 94.95 MB, 90.90 MB, 94.23 MB, 77.24 MB, 42.21 MB
ThreadUsage(500.000ms): 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 28, 14
- AverageThreadTokens: 38.97
- BloomFilterBytes: 0
- PeakMemoryUsage: 98.20 MB (102973750)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 710.68 MB (745201229)
- RowsProduced: 28.80M (28804452)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 12.756ms
- TotalStorageWaitTime: 5m38s
- TotalThreadsInvoluntaryContextSwitches: 143 (143)
- TotalThreadsTotalWallClockTime: 12m3s
- TotalThreadsSysTime: 494.514ms
- TotalThreadsUserTime: 23s480ms
- TotalThreadsVoluntaryContextSwitches: 27.76K (27760)
Fragment Instance Lifecycle Timings:
- ExecTime: 18s679ms
- ExecTreeExecTime: 1s708ms
- OpenTime: 19.453ms
- ExecTreeOpenTime: 793.045us
- PrepareTime: 40.244ms
- ExecTreePrepareTime: 11.587ms
DataStreamSender (dst_id=6):(Total: 9s474ms, non-child: 9s474ms, % non-child: 100.00%)
- BytesSent: 405.14 MB (424823878)
- NetworkThroughput(*): 232.29 MB/sec
- OverallThroughput: 42.76 MB/sec
- PeakMemoryUsage: 177.62 KB (181880)
- RowsReturned: 28.80M (28804452)
- SerializeBatchTime: 7s725ms
- TransmitDataRPCTime: 1s744ms
- UncompressedRowBatchSize: 576.99 MB (605013144)
CodeGen:(Total: 45.247ms, non-child: 45.247ms, % non-child: 100.00%)
- CodegenTime: 1.192ms
- CompileTime: 2.651ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 1.84 MB (1929896)
- NumFunctions: 7 (7)
- NumInstructions: 105 (105)
- OptimizationTime: 14.958ms
- PeakMemoryUsage: 52.50 KB (53760)
- PrepareTime: 26.660ms
HDFS_SCAN_NODE (id=1):(Total: 1s707ms, non-child: 1s707ms, % non-child: 100.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:388/3.41 GB
ExecOption: PARQUET Codegen Enabled, Codegen enabled: 388 out of 388
Hdfs Read Thread Concurrency Bucket: 0:29.73% 1:70.27% 2:0% 3:0% 4:0% 5:0%
File Formats: PARQUET/NONE:776
BytesRead(500.000ms): 7.00 MB, 19.43 MB, 28.44 MB, 39.55 MB, 48.13 MB, 56.26 MB, 65.00 MB, 71.00 MB, 84.05 MB, 98.90 MB, 110.39 MB, 120.82 MB, 134.11 MB, 147.25 MB, 157.92 MB, 175.50 MB, 199.36 MB, 231.07 MB, 238.86 MB, 247.73 MB, 255.24 MB, 271.52 MB, 280.39 MB, 291.89 MB, 307.52 MB, 322.65 MB, 336.39 MB, 344.78 MB, 359.34 MB, 376.17 MB, 401.50 MB, 410.95 MB, 426.16 MB, 434.93 MB, 448.52 MB, 452.67 MB, 452.67 MB
- FooterProcessingTime: (Avg: 429.574ms ; Min: 734.820us ; Max: 1s454ms ; Number of samples: 388)
- AverageHdfsReadThreadConcurrency: 0.70
- AverageScannerThreadConcurrency: 37.97
- BytesRead: 452.67 MB (474662103)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 452.67 MB (474662103)
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 0
- CachedFileHandlesHitCount: 933 (933)
- CachedFileHandlesMissCount: 231 (231)
- CollectionItemsRead: 0 (0)
- DecompressionTime: 0.000ns
- MaxCompressedTextFileLength: 0
- NumColumns: 2 (2)
- NumDictFilteredRowGroups: 0 (0)
- NumDisksAccessed: 1 (1)
- NumRowGroups: 388 (388)
- NumScannerThreadsStarted: 39 (39)
- NumScannersWithNoReads: 0 (0)
- NumStatsFilteredRowGroups: 0 (0)
- PeakMemoryUsage: 98.02 MB (102782239)
- PerReadThreadRawHdfsThroughput: 30.11 MB/sec
- RemoteScanRanges: 0 (0)
- RowBatchQueueGetWaitTime: 1s425ms
- RowBatchQueuePutWaitTime: 5m45s
- RowsRead: 28.80M (28804452)
- RowsReturned: 28.80M (28804452)
- RowsReturnedRate: 16.87 M/sec
- ScanRangesComplete: 388 (388)
- ScannerThreadsInvoluntaryContextSwitches: 83 (83)
- ScannerThreadsTotalWallClockTime: 11m45s
- MaterializeTupleTime(*): 5s156ms
- ScannerThreadsSysTime: 314.594ms
- ScannerThreadsUserTime: 6s399ms
- ScannerThreadsVoluntaryContextSwitches: 27.67K (27673)
- TotalRawHdfsReadTime(*): 15s034ms
- TotalReadThroughput: 24.47 MB/sec
Instance 2438b42f267ed44:3e1ac3c700000009 (host=bdpe611n2:22000):(Total: 20s236ms, non-child: 7s497ms, % non-child: 37.05%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:387/3.40 GB
MemoryUsage(500.000ms): 37.62 KB, 673.14 KB, 2.18 MB, 746.30 KB, 490.39 KB, 458.45 KB, 9.22 MB, 2.08 MB, 1.28 MB, 4.18 MB, 7.23 MB, 19.96 MB, 49.62 MB, 41.85 MB, 41.28 MB, 22.38 MB, 710.30 KB, 858.45 KB, 24.58 MB, 28.19 MB, 31.91 MB, 38.47 MB, 40.97 MB, 30.28 MB, 31.27 MB, 35.69 MB, 51.93 MB, 70.03 MB, 74.24 MB, 89.80 MB, 89.11 MB, 77.88 MB, 76.28 MB, 68.25 MB, 68.93 MB, 59.45 MB, 63.09 MB, 68.49 MB, 67.46 MB, 46.51 MB, 24.61 MB
ThreadUsage(500.000ms): 1, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 22, 8
- AverageThreadTokens: 37.83
- BloomFilterBytes: 0
- PeakMemoryUsage: 93.42 MB (97953773)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 727.59 MB (762937435)
- RowsProduced: 28.79M (28788883)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 12.764ms
- TotalStorageWaitTime: 7m21s
- TotalThreadsInvoluntaryContextSwitches: 388 (388)
- TotalThreadsTotalWallClockTime: 13m2s
- TotalThreadsSysTime: 487.302ms
- TotalThreadsUserTime: 23s689ms
- TotalThreadsVoluntaryContextSwitches: 27.90K (27903)
Fragment Instance Lifecycle Timings:
- ExecTime: 20s177ms
- ExecTreeExecTime: 3s275ms
- OpenTime: 19.161ms
- ExecTreeOpenTime: 777.817us
- PrepareTime: 39.621ms
- ExecTreePrepareTime: 10.609ms
DataStreamSender (dst_id=6):(Total: 9s423ms, non-child: 9s423ms, % non-child: 100.00%)
- BytesSent: 404.95 MB (424617979)
- NetworkThroughput(*): 231.84 MB/sec
- OverallThroughput: 42.97 MB/sec
- PeakMemoryUsage: 177.62 KB (181880)
- RowsReturned: 28.79M (28788883)
- SerializeBatchTime: 7s671ms
- TransmitDataRPCTime: 1s746ms
- UncompressedRowBatchSize: 576.67 MB (604686135)
CodeGen:(Total: 44.918ms, non-child: 44.918ms, % non-child: 100.00%)
- CodegenTime: 1.156ms
- CompileTime: 2.415ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 1.84 MB (1929896)
- NumFunctions: 7 (7)
- NumInstructions: 105 (105)
- OptimizationTime: 14.818ms
- PeakMemoryUsage: 52.50 KB (53760)
- PrepareTime: 26.800ms
HDFS_SCAN_NODE (id=1):(Total: 3s270ms, non-child: 3s270ms, % non-child: 100.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:387/3.40 GB
ExecOption: PARQUET Codegen Enabled, Codegen enabled: 387 out of 387
Hdfs Read Thread Concurrency Bucket: 0:7.5% 1:92.5% 2:0% 3:0% 4:0% 5:0%
File Formats: PARQUET/NONE:774
BytesRead(500.000ms): 0, 7.63 MB, 12.75 MB, 17.28 MB, 23.94 MB, 30.49 MB, 41.65 MB, 50.18 MB, 56.42 MB, 62.53 MB, 70.59 MB, 88.79 MB, 112.31 MB, 119.81 MB, 131.30 MB, 137.71 MB, 146.37 MB, 150.41 MB, 171.55 MB, 182.59 MB, 195.88 MB, 212.64 MB, 224.92 MB, 236.00 MB, 251.59 MB, 265.33 MB, 285.94 MB, 307.73 MB, 316.23 MB, 335.93 MB, 346.77 MB, 355.72 MB, 366.58 MB, 378.46 MB, 392.91 MB, 404.08 MB, 418.19 MB, 431.16 MB, 441.04 MB, 449.77 MB, 452.59 MB
- FooterProcessingTime: (Avg: 563.541ms ; Min: 880.869us ; Max: 1s508ms ; Number of samples: 387)
- AverageHdfsReadThreadConcurrency: 0.93
- AverageScannerThreadConcurrency: 37.75
- BytesRead: 452.59 MB (474576310)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 452.59 MB (474576310)
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 0
- CachedFileHandlesHitCount: 945 (945)
- CachedFileHandlesMissCount: 216 (216)
- CollectionItemsRead: 0 (0)
- DecompressionTime: 0.000ns
- MaxCompressedTextFileLength: 0
- NumColumns: 2 (2)
- NumDictFilteredRowGroups: 0 (0)
- NumDisksAccessed: 1 (1)
- NumRowGroups: 387 (387)
- NumScannerThreadsStarted: 39 (39)
- NumScannersWithNoReads: 0 (0)
- NumStatsFilteredRowGroups: 0 (0)
- PeakMemoryUsage: 93.23 MB (97762262)
- PerReadThreadRawHdfsThroughput: 24.96 MB/sec
- RemoteScanRanges: 0 (0)
- RowBatchQueueGetWaitTime: 2s988ms
- RowBatchQueuePutWaitTime: 4m49s
- RowsRead: 28.79M (28788883)
- RowsReturned: 28.79M (28788883)
- RowsReturnedRate: 8.80 M/sec
- ScanRangesComplete: 387 (387)
- ScannerThreadsInvoluntaryContextSwitches: 140 (140)
- ScannerThreadsTotalWallClockTime: 12m42s
- MaterializeTupleTime(*): 5s310ms
- ScannerThreadsSysTime: 360.294ms
- ScannerThreadsUserTime: 6s621ms
- ScannerThreadsVoluntaryContextSwitches: 27.80K (27802)
- TotalRawHdfsReadTime(*): 18s131ms
- TotalReadThroughput: 22.08 MB/sec
Instance 2438b42f267ed44:3e1ac3c700000007 (host=bdpe611n3:22000):(Total: 17s317ms, non-child: 7s255ms, % non-child: 41.90%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:386/3.40 GB
MemoryUsage(500.000ms): 13.78 MB, 14.74 MB, 23.47 MB, 24.69 MB, 7.56 MB, 786.34 KB, 11.93 MB, 9.53 MB, 19.48 MB, 27.44 MB, 37.02 MB, 51.12 MB, 61.02 MB, 77.49 MB, 89.82 MB, 91.64 MB, 95.21 MB, 95.21 MB, 96.97 MB, 91.19 MB, 92.26 MB, 95.63 MB, 87.48 MB, 86.90 MB, 86.48 MB, 67.31 MB, 43.66 MB, 54.83 MB, 74.72 MB, 65.69 MB, 58.68 MB, 64.94 MB, 69.01 MB, 50.47 MB, 16.42 MB
ThreadUsage(500.000ms): 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 39, 24, 5
- AverageThreadTokens: 38.51
- BloomFilterBytes: 0
- PeakMemoryUsage: 102.32 MB (107294254)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 640.03 MB (671116894)
- RowsProduced: 28.79M (28786856)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 17.008ms
- TotalStorageWaitTime: 3m55s
- TotalThreadsInvoluntaryContextSwitches: 77 (77)
- TotalThreadsTotalWallClockTime: 11m9s
- TotalThreadsSysTime: 427.669ms
- TotalThreadsUserTime: 23s514ms
- TotalThreadsVoluntaryContextSwitches: 30.17K (30173)
Fragment Instance Lifecycle Timings:
- ExecTime: 17s255ms
- ExecTreeExecTime: 630.366ms
- OpenTime: 21.983ms
- ExecTreeOpenTime: 794.560us
- PrepareTime: 40.089ms
- ExecTreePrepareTime: 10.105ms
DataStreamSender (dst_id=6):(Total: 9s393ms, non-child: 9s393ms, % non-child: 100.00%)
- BytesSent: 404.96 MB (424630554)
- NetworkThroughput(*): 223.64 MB/sec
- OverallThroughput: 43.11 MB/sec
- PeakMemoryUsage: 177.62 KB (181880)
- RowsReturned: 28.79M (28786856)
- SerializeBatchTime: 7s577ms
- TransmitDataRPCTime: 1s810ms
- UncompressedRowBatchSize: 576.63 MB (604643556)
CodeGen:(Total: 48.826ms, non-child: 48.826ms, % non-child: 100.00%)
- CodegenTime: 1.294ms
- CompileTime: 3.504ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 1.84 MB (1929896)
- NumFunctions: 7 (7)
- NumInstructions: 105 (105)
- OptimizationTime: 16.485ms
- PeakMemoryUsage: 52.50 KB (53760)
- PrepareTime: 27.871ms
HDFS_SCAN_NODE (id=1):(Total: 620.395ms, non-child: 620.395ms, % non-child: 100.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:386/3.40 GB
ExecOption: PARQUET Codegen Enabled, Codegen enabled: 386 out of 386
Hdfs Read Thread Concurrency Bucket: 0:34.29% 1:65.71% 2:0% 3:0% 4:0% 5:0%
File Formats: PARQUET/NONE:772
BytesRead(500.000ms): 9.30 MB, 22.92 MB, 39.97 MB, 53.28 MB, 59.49 MB, 67.43 MB, 82.92 MB, 95.58 MB, 112.51 MB, 129.79 MB, 143.33 MB, 163.12 MB, 180.09 MB, 197.56 MB, 212.59 MB, 228.42 MB, 240.94 MB, 248.82 MB, 264.78 MB, 278.10 MB, 296.45 MB, 308.04 MB, 316.73 MB, 326.30 MB, 335.87 MB, 344.96 MB, 358.22 MB, 375.37 MB, 394.93 MB, 403.47 MB, 416.35 MB, 429.51 MB, 452.42 MB, 452.42 MB, 452.42 MB
- FooterProcessingTime: (Avg: 298.680ms ; Min: 724.423us ; Max: 1s068ms ; Number of samples: 386)
- AverageHdfsReadThreadConcurrency: 0.66
- AverageScannerThreadConcurrency: 37.51
- BytesRead: 452.42 MB (474401701)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 452.42 MB (474401701)
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 0
- CachedFileHandlesHitCount: 958 (958)
- CachedFileHandlesMissCount: 200 (200)
- CollectionItemsRead: 0 (0)
- DecompressionTime: 0.000ns
- MaxCompressedTextFileLength: 0
- NumColumns: 2 (2)
- NumDictFilteredRowGroups: 0 (0)
- NumDisksAccessed: 1 (1)
- NumRowGroups: 386 (386)
- NumScannerThreadsStarted: 39 (39)
- NumScannersWithNoReads: 0 (0)
- NumStatsFilteredRowGroups: 0 (0)
- PeakMemoryUsage: 102.14 MB (107102743)
- PerReadThreadRawHdfsThroughput: 37.03 MB/sec
- RemoteScanRanges: 0 (0)
- RowBatchQueueGetWaitTime: 280.019ms
- RowBatchQueuePutWaitTime: 6m43s
- RowsRead: 28.79M (28786856)
- RowsReturned: 28.79M (28786856)
- RowsReturnedRate: 46.40 M/sec
- ScanRangesComplete: 386 (386)
- ScannerThreadsInvoluntaryContextSwitches: 45 (45)
- ScannerThreadsTotalWallClockTime: 10m52s
- MaterializeTupleTime(*): 5s044ms
- ScannerThreadsSysTime: 300.723ms
- ScannerThreadsUserTime: 6s655ms
- ScannerThreadsVoluntaryContextSwitches: 30.11K (30109)
- TotalRawHdfsReadTime(*): 12s217ms
- TotalReadThroughput: 25.85 MB/sec
Instance 2438b42f267ed44:3e1ac3c700000008 (host=bdpe611n4:22000):(Total: 19s105ms, non-child: 7s527ms, % non-child: 39.40%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:387/3.41 GB
MemoryUsage(500.000ms): 37.62 KB, 6.22 MB, 754.27 KB, 314.40 KB, 1.78 MB, 10.99 MB, 5.16 MB, 3.19 MB, 426.35 KB, 10.26 MB, 20.30 MB, 29.34 MB, 32.28 MB, 38.28 MB, 48.56 MB, 51.51 MB, 55.28 MB, 64.16 MB, 62.72 MB, 50.63 MB, 48.81 MB, 56.92 MB, 74.28 MB, 80.01 MB, 84.90 MB, 79.66 MB, 68.48 MB, 58.46 MB, 60.82 MB, 62.81 MB, 56.09 MB, 67.62 MB, 74.75 MB, 91.05 MB, 87.45 MB, 93.06 MB, 90.38 MB, 57.38 MB, 17.24 MB
ThreadUsage(500.000ms): 1, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 36, 24, 6
- AverageThreadTokens: 37.62
- BloomFilterBytes: 0
- PeakMemoryUsage: 97.56 MB (102297694)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 714.74 MB (749458108)
- RowsProduced: 28.83M (28834382)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 12.032ms
- TotalStorageWaitTime: 5m32s
- TotalThreadsInvoluntaryContextSwitches: 204 (204)
- TotalThreadsTotalWallClockTime: 12m22s
- TotalThreadsSysTime: 534.741ms
- TotalThreadsUserTime: 25s347ms
- TotalThreadsVoluntaryContextSwitches: 29.07K (29067)
Fragment Instance Lifecycle Timings:
- ExecTime: 19s054ms
- ExecTreeExecTime: 2s076ms
- OpenTime: 17.312ms
- ExecTreeOpenTime: 737.427us
- PrepareTime: 33.913ms
- ExecTreePrepareTime: 8.403ms
DataStreamSender (dst_id=6):(Total: 9s471ms, non-child: 9s471ms, % non-child: 100.00%)
- BytesSent: 405.60 MB (425303423)
- NetworkThroughput(*): 231.66 MB/sec
- OverallThroughput: 42.83 MB/sec
- PeakMemoryUsage: 177.62 KB (181880)
- RowsReturned: 28.83M (28834382)
- SerializeBatchTime: 7s713ms
- TransmitDataRPCTime: 1s750ms
- UncompressedRowBatchSize: 577.59 MB (605641802)
CodeGen:(Total: 38.865ms, non-child: 38.865ms, % non-child: 100.00%)
- CodegenTime: 982.787us
- CompileTime: 2.512ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 1.84 MB (1929896)
- NumFunctions: 7 (7)
- NumInstructions: 105 (105)
- OptimizationTime: 13.084ms
- PeakMemoryUsage: 52.50 KB (53760)
- PrepareTime: 22.489ms
HDFS_SCAN_NODE (id=1):(Total: 2s068ms, non-child: 2s068ms, % non-child: 100.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:387/3.41 GB
ExecOption: PARQUET Codegen Enabled, Codegen enabled: 387 out of 387
Hdfs Read Thread Concurrency Bucket: 0:15.79% 1:84.21% 2:0% 3:0% 4:0% 5:0%
File Formats: PARQUET/NONE:774
BytesRead(500.000ms): 0, 8.51 MB, 15.71 MB, 21.21 MB, 28.56 MB, 42.67 MB, 52.30 MB, 61.61 MB, 65.44 MB, 78.45 MB, 96.17 MB, 111.02 MB, 124.62 MB, 138.56 MB, 156.16 MB, 171.67 MB, 186.45 MB, 202.00 MB, 212.05 MB, 220.39 MB, 239.72 MB, 258.14 MB, 274.14 MB, 287.71 MB, 296.55 MB, 309.36 MB, 318.52 MB, 334.06 MB, 344.03 MB, 357.73 MB, 367.59 MB, 388.09 MB, 403.68 MB, 420.25 MB, 426.52 MB, 443.22 MB, 452.72 MB, 452.72 MB, 452.72 MB
- FooterProcessingTime: (Avg: 421.574ms ; Min: 767.385us ; Max: 1s455ms ; Number of samples: 387)
- AverageHdfsReadThreadConcurrency: 0.84
- AverageScannerThreadConcurrency: 37.58
- BytesRead: 452.72 MB (474706225)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 452.72 MB (474706225)
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 0
- CachedFileHandlesHitCount: 946 (946)
- CachedFileHandlesMissCount: 215 (215)
- CollectionItemsRead: 0 (0)
- DecompressionTime: 0.000ns
- MaxCompressedTextFileLength: 0
- NumColumns: 2 (2)
- NumDictFilteredRowGroups: 0 (0)
- NumDisksAccessed: 1 (1)
- NumRowGroups: 387 (387)
- NumScannerThreadsStarted: 39 (39)
- NumScannersWithNoReads: 0 (0)
- NumStatsFilteredRowGroups: 0 (0)
- PeakMemoryUsage: 97.39 MB (102122567)
- PerReadThreadRawHdfsThroughput: 28.93 MB/sec
- RemoteScanRanges: 0 (0)
- RowBatchQueueGetWaitTime: 1s774ms
- RowBatchQueuePutWaitTime: 6m4s
- RowsRead: 28.83M (28834382)
- RowsReturned: 28.83M (28834382)
- RowsReturnedRate: 13.94 M/sec
- ScanRangesComplete: 387 (387)
- ScannerThreadsInvoluntaryContextSwitches: 99 (99)
- ScannerThreadsTotalWallClockTime: 12m3s
- MaterializeTupleTime(*): 6s575ms
- ScannerThreadsSysTime: 367.774ms
- ScannerThreadsUserTime: 8s229ms
- ScannerThreadsVoluntaryContextSwitches: 28.99K (28988)
- TotalRawHdfsReadTime(*): 15s649ms
- TotalReadThroughput: 23.22 MB/sec
Instance 2438b42f267ed44:3e1ac3c70000000a (host=bdpe722n4:22000):(Total: 19s224ms, non-child: 7s613ms, % non-child: 39.60%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:388/3.40 GB
MemoryUsage(500.000ms): 1.21 MB, 6.36 MB, 2.28 MB, 490.42 KB, 3.44 MB, 2.52 MB, 32.22 MB, 34.53 MB, 41.90 MB, 58.63 MB, 80.14 MB, 84.30 MB, 61.27 MB, 55.51 MB, 47.88 MB, 43.02 MB, 48.51 MB, 42.41 MB, 40.27 MB, 40.90 MB, 36.74 MB, 24.47 MB, 3.74 MB, 1.31 MB, 434.39 KB, 16.05 MB, 9.53 MB, 22.27 MB, 8.35 MB, 7.53 MB, 826.35 KB, 11.06 MB, 14.73 MB, 19.32 MB, 15.94 MB, 16.71 MB, 31.90 MB, 23.77 MB
ThreadUsage(500.000ms): 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 28, 17, 8
- AverageThreadTokens: 38.24
- BloomFilterBytes: 0
- PeakMemoryUsage: 84.69 MB (88798822)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 680.93 MB (714007426)
- RowsProduced: 28.78M (28782492)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 11.845ms
- TotalStorageWaitTime: 8m15s
- TotalThreadsInvoluntaryContextSwitches: 179 (179)
- TotalThreadsTotalWallClockTime: 12m12s
- TotalThreadsSysTime: 449.831ms
- TotalThreadsUserTime: 23s998ms
- TotalThreadsVoluntaryContextSwitches: 27.49K (27492)
Fragment Instance Lifecycle Timings:
- ExecTime: 19s166ms
- ExecTreeExecTime: 2s209ms
- OpenTime: 20.637ms
- ExecTreeOpenTime: 833.069us
- PrepareTime: 37.334ms
- ExecTreePrepareTime: 9.378ms
DataStreamSender (dst_id=6):(Total: 9s361ms, non-child: 9s361ms, % non-child: 100.00%)
- BytesSent: 404.85 MB (424513438)
- NetworkThroughput(*): 247.99 MB/sec
- OverallThroughput: 43.25 MB/sec
- PeakMemoryUsage: 177.62 KB (181880)
- RowsReturned: 28.78M (28782492)
- SerializeBatchTime: 7s723ms
- TransmitDataRPCTime: 1s632ms
- UncompressedRowBatchSize: 576.55 MB (604551892)
CodeGen:(Total: 45.388ms, non-child: 45.388ms, % non-child: 100.00%)
- CodegenTime: 1.188ms
- CompileTime: 2.912ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 1.84 MB (1929896)
- NumFunctions: 7 (7)
- NumInstructions: 105 (105)
- OptimizationTime: 15.672ms
- PeakMemoryUsage: 52.50 KB (53760)
- PrepareTime: 25.893ms
HDFS_SCAN_NODE (id=1):(Total: 2s203ms, non-child: 2s203ms, % non-child: 100.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:388/3.40 GB
ExecOption: PARQUET Codegen Enabled, Codegen enabled: 388 out of 388
Hdfs Read Thread Concurrency Bucket: 0:5.263% 1:94.74% 2:0% 3:0% 4:0% 5:0% 6:0%
File Formats: PARQUET/NONE:776
BytesRead(500.000ms): 10.52 MB, 20.29 MB, 25.80 MB, 30.25 MB, 35.98 MB, 46.06 MB, 68.92 MB, 80.86 MB, 97.49 MB, 116.13 MB, 136.11 MB, 144.39 MB, 151.96 MB, 160.84 MB, 174.40 MB, 187.23 MB, 199.41 MB, 211.46 MB, 226.74 MB, 236.98 MB, 248.14 MB, 258.62 MB, 265.77 MB, 276.36 MB, 282.98 MB, 299.42 MB, 311.21 MB, 327.77 MB, 334.76 MB, 348.54 MB, 358.54 MB, 374.73 MB, 388.41 MB, 402.72 MB, 415.56 MB, 429.66 MB, 448.27 MB, 452.55 MB
- FooterProcessingTime: (Avg: 628.978ms ; Min: 1.447ms ; Max: 1s481ms ; Number of samples: 388)
- AverageHdfsReadThreadConcurrency: 0.95
- AverageScannerThreadConcurrency: 37.24
- BytesRead: 452.55 MB (474535168)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 452.55 MB (474535168)
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 0
- CachedFileHandlesHitCount: 944 (944)
- CachedFileHandlesMissCount: 220 (220)
- CollectionItemsRead: 0 (0)
- DecompressionTime: 0.000ns
- MaxCompressedTextFileLength: 0
- NumColumns: 2 (2)
- NumDictFilteredRowGroups: 0 (0)
- NumDisksAccessed: 1 (1)
- NumRowGroups: 388 (388)
- NumScannerThreadsStarted: 39 (39)
- NumScannersWithNoReads: 0 (0)
- NumStatsFilteredRowGroups: 0 (0)
- PeakMemoryUsage: 84.50 MB (88607311)
- PerReadThreadRawHdfsThroughput: 26.27 MB/sec
- RemoteScanRanges: 0 (0)
- RowBatchQueueGetWaitTime: 1s927ms
- RowBatchQueuePutWaitTime: 3m13s
- RowsRead: 28.78M (28782492)
- RowsReturned: 28.78M (28782492)
- RowsReturnedRate: 13.06 M/sec
- ScanRangesComplete: 388 (388)
- ScannerThreadsInvoluntaryContextSwitches: 82 (82)
- ScannerThreadsTotalWallClockTime: 11m53s
- MaterializeTupleTime(*): 5s432ms
- ScannerThreadsSysTime: 322.882ms
- ScannerThreadsUserTime: 6s878ms
- ScannerThreadsVoluntaryContextSwitches: 27.40K (27396)
- TotalRawHdfsReadTime(*): 17s225ms
- TotalReadThroughput: 23.82 MB/sec
Best Regards
Ferdinand Xu
From: Xu, Cheng A [mailto:cheng.a.xu@intel.com]
Sent: Friday, October 27, 2017 2:57 PM
To: user@impala.incubator.apache.org; mmokhtar@cloudera.com
Subject: RE: performance issue on big table join
Hi Junjie,
The following error is caused by low version of Hive metastore. Can be resolved by upgrade script.
Hi Mostafa
About “compute stats”, is it used to help Impala determine join type (broadcast join or partitioned join) since we have better stats?
https://www.cloudera.com/documentation/enterprise/5-6-x/topics/impala_perf_joins.html#perf_joins
From the plan picture as attached, the join type for store_sales and catalog_sales is already partitioned type.
Also from this picture, the broadcast for catalog_sales is very fast and I think this table is used for build and probe store_sales table. The MAX and avg for broadcast and exchange is much lower than that in Hash join(Max is 2h, avg 2h). Not sure whether we can increase the parallelism for the probe part. If only single thread mode is supported now, any tuning tips to improve this case (two big table join)? Thank you!
Best Regards
Ferdinand Xu
From: 俊杰陈 [mailto:cjjnjust@gmail.com]
Sent: Friday, October 27, 2017 1:38 PM
To: user@impala.incubator.apache.org<ma...@impala.incubator.apache.org>; mmokhtar@cloudera.com<ma...@cloudera.com>
Subject: Re: performance issue on big table join
Thanks, let me put here.
Yes, the query is intended to verify parallelism of partitioned join. We want to know how many fragment instances started for a hash join on a single node, and how many threads in a fragment instance perform the join. I'm not sure whether there is only one thread participate in hash join, since thus it can not maximize the CPU utilization.
The compute stats met following error:
Query: compute stats store_sales
WARNINGS: ImpalaRuntimeException: Error making 'updateTableColumnStatistics' RPC to Hive Metastore:
CAUSED BY: MetaException: Insert of object "org.apache.hadoop.hive.metastore.model.MTableColumnStatistics@49fe55a1<ma...@49fe55a1>" using statement "INSERT INTO `TAB_COL_STATS` (`CS_ID`,`AVG_COL_LEN`,`BIT_VECTOR`,`COLUMN_NAME`,`COLUMN_TYPE`,`DB_NAME`,`BIG_DECIMAL_HIGH_VALUE`,`BIG_DECIMAL_LOW_VALUE`,`DOUBLE_HIGH_VALUE`,`DOUBLE_LOW_VALUE`,`LAST_ANALYZED`,`LONG_HIGH_VALUE`,`LONG_LOW_VALUE`,`MAX_COL_LEN`,`NUM_DISTINCTS`,`NUM_FALSES`,`NUM_NULLS`,`NUM_TRUES`,`TBL_ID`,`TABLE_NAME`) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)" failed : Unknown column 'BIT_VECTOR' in 'field list'
2017-10-27 12:02 GMT+08:00 Mostafa Mokhtar <mm...@cloudera.com>>:
Hi,
Looks like you are joining store_sales with catalog_sales on item_sk, this
kind of join condition is a many to many, which means the output number of
rows will be much larger then input number of rows, not sure if this is
intended.
Also did you run "compute stats [TABLE_NAME]" on both tables?
For a more comprehensive query try TPCDS Q17
select i_item_id
,i_item_desc
,s_state
,count(ss_quantity) as store_sales_quantitycount
,avg(ss_quantity) as store_sales_quantityave
,stddev_samp(ss_quantity) as store_sales_quantitystdev
,stddev_samp(ss_quantity)/avg(ss_quantity) as store_sales_quantitycov
,count(sr_return_quantity) as store_returns_quantitycount
,avg(sr_return_quantity) as store_returns_quantityave
,stddev_samp(sr_return_quantity) as store_returns_quantitystdev
,stddev_samp(sr_return_quantity)/avg(sr_return_quantity) as
store_returns_quantitycov
,count(cs_quantity) as catalog_sales_quantitycount
,avg(cs_quantity) as catalog_sales_quantityave
,stddev_samp(cs_quantity) as catalog_sales_quantitystdev
,stddev_samp(cs_quantity)/avg(cs_quantity) as catalog_sales_quantitycov
from store_sales
,store_returns
,catalog_sales
,date_dim d1
,date_dim d2
,date_dim d3
,store
,item
where d1.d_quarter_name = '2000Q1'
and d1.d_date_sk = ss_sold_date_sk
and i_item_sk = ss_item_sk
and s_store_sk = ss_store_sk
and ss_customer_sk = sr_customer_sk
and ss_item_sk = sr_item_sk
and ss_ticket_number = sr_ticket_number
and sr_returned_date_sk = d2.d_date_sk
and d2.d_quarter_name in ('2000Q1','2000Q2','2000Q3')
and sr_customer_sk = cs_bill_customer_sk
and sr_item_sk = cs_item_sk
and cs_sold_date_sk = d3.d_date_sk
and d3.d_quarter_name in ('2000Q1','2000Q2','2000Q3')
group by i_item_id
,i_item_desc
,s_state
order by i_item_id
,i_item_desc
,s_state
limit 100;
I recommend moving this kind of discussion on
user@impala.incubator.apache.org<ma...@impala.incubator.apache.org>.
On Thu, Oct 26, 2017 at 7:25 PM, 俊杰陈 <cj...@gmail.com>> wrote:
> The profile file is damaged. Here is a screenshot for exec summary
>
>
>
> 2017-10-27 10:04 GMT+08:00 俊杰陈 <cj...@gmail.com>>:
>
>> Hi Devs
>>
>> I met a performance issue on big table join. The query takes more than 3
>> hours on Impala and only 3 minutes on Spark SQL on the same 5 nodes
>> cluster. when running query, the left scanner and exchange node are very
>> slow. Did I miss some key arguments?
>>
>> you can see profile file in attachment.
>>
>>
>>
>> --
>> Thanks & Best Regards
>>
>
>
>
> --
> Thanks & Best Regards
>
--
Thanks & Best Regards
RE: performance issue on big table join
Posted by "Xu, Cheng A" <ch...@intel.com>.
Hi Junjie,
The following error is caused by low version of Hive metastore. Can be resolved by upgrade script.
Hi Mostafa
About “compute stats”, is it used to help Impala determine join type (broadcast join or partitioned join) since we have better stats?
https://www.cloudera.com/documentation/enterprise/5-6-x/topics/impala_perf_joins.html#perf_joins
From the plan picture as attached, the join type for store_sales and catalog_sales is already partitioned type.
Also from this picture, the broadcast for catalog_sales is very fast and I think this table is used for build and probe store_sales table. The MAX and avg for broadcast and exchange is much lower than that in Hash join(Max is 2h, avg 2h). Not sure whether we can increase the parallelism for the probe part. If only single thread mode is supported now, any tuning tips to improve this case (two big table join)? Thank you!
Best Regards
Ferdinand Xu
From: 俊杰陈 [mailto:cjjnjust@gmail.com]
Sent: Friday, October 27, 2017 1:38 PM
To: user@impala.incubator.apache.org; mmokhtar@cloudera.com
Subject: Re: performance issue on big table join
Thanks, let me put here.
Yes, the query is intended to verify parallelism of partitioned join. We want to know how many fragment instances started for a hash join on a single node, and how many threads in a fragment instance perform the join. I'm not sure whether there is only one thread participate in hash join, since thus it can not maximize the CPU utilization.
The compute stats met following error:
Query: compute stats store_sales
WARNINGS: ImpalaRuntimeException: Error making 'updateTableColumnStatistics' RPC to Hive Metastore:
CAUSED BY: MetaException: Insert of object "org.apache.hadoop.hive.metastore.model.MTableColumnStatistics@49fe55a1<ma...@49fe55a1>" using statement "INSERT INTO `TAB_COL_STATS` (`CS_ID`,`AVG_COL_LEN`,`BIT_VECTOR`,`COLUMN_NAME`,`COLUMN_TYPE`,`DB_NAME`,`BIG_DECIMAL_HIGH_VALUE`,`BIG_DECIMAL_LOW_VALUE`,`DOUBLE_HIGH_VALUE`,`DOUBLE_LOW_VALUE`,`LAST_ANALYZED`,`LONG_HIGH_VALUE`,`LONG_LOW_VALUE`,`MAX_COL_LEN`,`NUM_DISTINCTS`,`NUM_FALSES`,`NUM_NULLS`,`NUM_TRUES`,`TBL_ID`,`TABLE_NAME`) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)" failed : Unknown column 'BIT_VECTOR' in 'field list'
2017-10-27 12:02 GMT+08:00 Mostafa Mokhtar <mm...@cloudera.com>>:
Hi,
Looks like you are joining store_sales with catalog_sales on item_sk, this
kind of join condition is a many to many, which means the output number of
rows will be much larger then input number of rows, not sure if this is
intended.
Also did you run "compute stats [TABLE_NAME]" on both tables?
For a more comprehensive query try TPCDS Q17
select i_item_id
,i_item_desc
,s_state
,count(ss_quantity) as store_sales_quantitycount
,avg(ss_quantity) as store_sales_quantityave
,stddev_samp(ss_quantity) as store_sales_quantitystdev
,stddev_samp(ss_quantity)/avg(ss_quantity) as store_sales_quantitycov
,count(sr_return_quantity) as store_returns_quantitycount
,avg(sr_return_quantity) as store_returns_quantityave
,stddev_samp(sr_return_quantity) as store_returns_quantitystdev
,stddev_samp(sr_return_quantity)/avg(sr_return_quantity) as
store_returns_quantitycov
,count(cs_quantity) as catalog_sales_quantitycount
,avg(cs_quantity) as catalog_sales_quantityave
,stddev_samp(cs_quantity) as catalog_sales_quantitystdev
,stddev_samp(cs_quantity)/avg(cs_quantity) as catalog_sales_quantitycov
from store_sales
,store_returns
,catalog_sales
,date_dim d1
,date_dim d2
,date_dim d3
,store
,item
where d1.d_quarter_name = '2000Q1'
and d1.d_date_sk = ss_sold_date_sk
and i_item_sk = ss_item_sk
and s_store_sk = ss_store_sk
and ss_customer_sk = sr_customer_sk
and ss_item_sk = sr_item_sk
and ss_ticket_number = sr_ticket_number
and sr_returned_date_sk = d2.d_date_sk
and d2.d_quarter_name in ('2000Q1','2000Q2','2000Q3')
and sr_customer_sk = cs_bill_customer_sk
and sr_item_sk = cs_item_sk
and cs_sold_date_sk = d3.d_date_sk
and d3.d_quarter_name in ('2000Q1','2000Q2','2000Q3')
group by i_item_id
,i_item_desc
,s_state
order by i_item_id
,i_item_desc
,s_state
limit 100;
I recommend moving this kind of discussion on
user@impala.incubator.apache.org<ma...@impala.incubator.apache.org>.
On Thu, Oct 26, 2017 at 7:25 PM, 俊杰陈 <cj...@gmail.com>> wrote:
> The profile file is damaged. Here is a screenshot for exec summary
>
>
>
> 2017-10-27 10:04 GMT+08:00 俊杰陈 <cj...@gmail.com>>:
>
>> Hi Devs
>>
>> I met a performance issue on big table join. The query takes more than 3
>> hours on Impala and only 3 minutes on Spark SQL on the same 5 nodes
>> cluster. when running query, the left scanner and exchange node are very
>> slow. Did I miss some key arguments?
>>
>> you can see profile file in attachment.
>>
>>
>>
>> --
>> Thanks & Best Regards
>>
>
>
>
> --
> Thanks & Best Regards
>
--
Thanks & Best Regards
Re: performance issue on big table join
Posted by Alexander Behm <al...@cloudera.com>.
Want to make sure there are no misunderstandings. When I say "builds can be
run in parallel" I mean that for queries with a series of joins, multiple
independent join builds can run in parallel. But each individual build is
still run in a singe thread.
The degree of build parallelism is dynamically adjusted by our
ThreadResourceMgr based on the availability of threads. You can control how
many threads are available for the ThreadResourceMgr to allocate based on
these paremeters:
--num_cores
--num_threads_per_core
The total number of threads for the ThreadResourceMgr is
num_cores*num_threads_per_core.
That said, I strongly advise against adjusting those config params for the
purpose of "controlling" the build parallelism because those params affect
many more things i the system (e.g. scan parallelism).
On Thu, Oct 26, 2017 at 11:26 PM, 俊杰陈 <cj...@gmail.com> wrote:
> Thank Alex.
>
> Cheng, build phase multithreading should depend on Disk IO manager.
>
>
> 2017-10-27 13:50 GMT+08:00 Xu, Cheng A <ch...@intel.com>:
>
>> Thanks Alexander for the reply. Is there any configuration we can use to
>> determine the parallelism level for build phase? Thank you!
>>
>>
>>
>> Thanks
>>
>> Ferdinand Xu
>>
>>
>>
>> *From:* Alexander Behm [mailto:alex.behm@cloudera.com]
>> *Sent:* Friday, October 27, 2017 1:46 PM
>> *To:* user@impala.incubator.apache.org
>> *Cc:* Mostafa Mokhtar <mm...@cloudera.com>
>> *Subject:* Re: performance issue on big table join
>>
>>
>>
>> See my response on the other thread you started. The probe side of joins
>> are are executed in a single thread per host. Impala can run multiple
>> builds in parallel - but each build uses only a single thread.
>>
>> A single query might not be able to max out your CPU, but most realistic
>> workloads run several queries concurrently.
>>
>>
>>
>>
>>
>> On Thu, Oct 26, 2017 at 10:38 PM, 俊杰陈 <cj...@gmail.com> wrote:
>>
>> Thanks, let me put here.
>>
>>
>>
>> Yes, the query is intended to verify parallelism of partitioned join. We
>> want to know how many fragment instances started for a hash join on a
>> single node, and how many threads in a fragment instance perform the join.
>> I'm not sure whether there is only one thread participate in hash join,
>> since thus it can not maximize the CPU utilization.
>>
>>
>>
>> The compute stats met following error:
>>
>>
>>
>> Query: compute stats store_sales
>>
>> WARNINGS: ImpalaRuntimeException: Error making
>> 'updateTableColumnStatistics' RPC to Hive Metastore:
>>
>> CAUSED BY: MetaException: Insert of object "org.apache.hadoop.hive.metast
>> ore.model.MTableColumnStatistics@49fe55a1" using statement "INSERT INTO
>> `TAB_COL_STATS` (`CS_ID`,`AVG_COL_LEN`,`BIT_VE
>> CTOR`,`COLUMN_NAME`,`COLUMN_TYPE`,`DB_NAME`,`BIG_DECIMAL_HIG
>> H_VALUE`,`BIG_DECIMAL_LOW_VALUE`,`DOUBLE_HIGH_VALUE`,`DOUBLE
>> _LOW_VALUE`,`LAST_ANALYZED`,`LONG_HIGH_VALUE`,`LONG_LOW_
>> VALUE`,`MAX_COL_LEN`,`NUM_DISTINCTS`,`NUM_FALSES`,`NUM_
>> NULLS`,`NUM_TRUES`,`TBL_ID`,`TABLE_NAME`) VALUES
>> (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)" failed : Unknown column
>> 'BIT_VECTOR' in 'field list'
>>
>>
>>
>>
>>
>> 2017-10-27 12:02 GMT+08:00 Mostafa Mokhtar <mm...@cloudera.com>:
>>
>> Hi,
>>
>> Looks like you are joining store_sales with catalog_sales on item_sk, this
>> kind of join condition is a many to many, which means the output number of
>> rows will be much larger then input number of rows, not sure if this is
>> intended.
>>
>> Also did you run "compute stats [TABLE_NAME]" on both tables?
>>
>> For a more comprehensive query try TPCDS Q17
>>
>> select i_item_id
>>
>> ,i_item_desc
>>
>> ,s_state
>>
>> ,count(ss_quantity) as store_sales_quantitycount
>>
>> ,avg(ss_quantity) as store_sales_quantityave
>>
>> ,stddev_samp(ss_quantity) as store_sales_quantitystdev
>>
>> ,stddev_samp(ss_quantity)/avg(ss_quantity) as
>> store_sales_quantitycov
>>
>> ,count(sr_return_quantity) as store_returns_quantitycount
>>
>> ,avg(sr_return_quantity) as store_returns_quantityave
>>
>> ,stddev_samp(sr_return_quantity) as store_returns_quantitystdev
>>
>> ,stddev_samp(sr_return_quantity)/avg(sr_return_quantity) as
>> store_returns_quantitycov
>>
>> ,count(cs_quantity) as catalog_sales_quantitycount
>> ,avg(cs_quantity) as catalog_sales_quantityave
>>
>> ,stddev_samp(cs_quantity) as catalog_sales_quantitystdev
>>
>> ,stddev_samp(cs_quantity)/avg(cs_quantity) as
>> catalog_sales_quantitycov
>>
>> from store_sales
>>
>> ,store_returns
>>
>> ,catalog_sales
>>
>> ,date_dim d1
>>
>> ,date_dim d2
>>
>> ,date_dim d3
>>
>> ,store
>>
>> ,item
>>
>> where d1.d_quarter_name = '2000Q1'
>>
>> and d1.d_date_sk = ss_sold_date_sk
>>
>> and i_item_sk = ss_item_sk
>>
>> and s_store_sk = ss_store_sk
>>
>> and ss_customer_sk = sr_customer_sk
>>
>> and ss_item_sk = sr_item_sk
>>
>> and ss_ticket_number = sr_ticket_number
>>
>> and sr_returned_date_sk = d2.d_date_sk
>>
>> and d2.d_quarter_name in ('2000Q1','2000Q2','2000Q3')
>>
>> and sr_customer_sk = cs_bill_customer_sk
>>
>> and sr_item_sk = cs_item_sk
>>
>> and cs_sold_date_sk = d3.d_date_sk
>>
>> and d3.d_quarter_name in ('2000Q1','2000Q2','2000Q3')
>>
>> group by i_item_id
>>
>> ,i_item_desc
>>
>> ,s_state
>>
>> order by i_item_id
>>
>> ,i_item_desc
>>
>> ,s_state
>>
>> limit 100;
>>
>>
>> I recommend moving this kind of discussion on
>> user@impala.incubator.apache.org.
>>
>> On Thu, Oct 26, 2017 at 7:25 PM, 俊杰陈 <cj...@gmail.com> wrote:
>>
>> > The profile file is damaged. Here is a screenshot for exec summary
>> >
>> >
>> >
>> > 2017-10-27 10:04 GMT+08:00 俊杰陈 <cj...@gmail.com>:
>> >
>> >> Hi Devs
>> >>
>> >> I met a performance issue on big table join. The query takes more than
>> 3
>> >> hours on Impala and only 3 minutes on Spark SQL on the same 5 nodes
>> >> cluster. when running query, the left scanner and exchange node are
>> very
>> >> slow. Did I miss some key arguments?
>> >>
>> >> you can see profile file in attachment.
>> >>
>> >>
>> >>
>>
>> >> --
>> >> Thanks & Best Regards
>> >>
>> >
>> >
>> >
>> > --
>> > Thanks & Best Regards
>> >
>>
>>
>>
>>
>>
>> --
>>
>> Thanks & Best Regards
>>
>>
>>
>
>
>
> --
> Thanks & Best Regards
>
Re: performance issue on big table join
Posted by 俊杰陈 <cj...@gmail.com>.
Thank Alex.
Cheng, build phase multithreading should depend on Disk IO manager.
2017-10-27 13:50 GMT+08:00 Xu, Cheng A <ch...@intel.com>:
> Thanks Alexander for the reply. Is there any configuration we can use to
> determine the parallelism level for build phase? Thank you!
>
>
>
> Thanks
>
> Ferdinand Xu
>
>
>
> *From:* Alexander Behm [mailto:alex.behm@cloudera.com]
> *Sent:* Friday, October 27, 2017 1:46 PM
> *To:* user@impala.incubator.apache.org
> *Cc:* Mostafa Mokhtar <mm...@cloudera.com>
> *Subject:* Re: performance issue on big table join
>
>
>
> See my response on the other thread you started. The probe side of joins
> are are executed in a single thread per host. Impala can run multiple
> builds in parallel - but each build uses only a single thread.
>
> A single query might not be able to max out your CPU, but most realistic
> workloads run several queries concurrently.
>
>
>
>
>
> On Thu, Oct 26, 2017 at 10:38 PM, 俊杰陈 <cj...@gmail.com> wrote:
>
> Thanks, let me put here.
>
>
>
> Yes, the query is intended to verify parallelism of partitioned join. We
> want to know how many fragment instances started for a hash join on a
> single node, and how many threads in a fragment instance perform the join.
> I'm not sure whether there is only one thread participate in hash join,
> since thus it can not maximize the CPU utilization.
>
>
>
> The compute stats met following error:
>
>
>
> Query: compute stats store_sales
>
> WARNINGS: ImpalaRuntimeException: Error making
> 'updateTableColumnStatistics' RPC to Hive Metastore:
>
> CAUSED BY: MetaException: Insert of object "org.apache.hadoop.hive.
> metastore.model.MTableColumnStatistics@49fe55a1" using statement "INSERT
> INTO `TAB_COL_STATS` (`CS_ID`,`AVG_COL_LEN`,`BIT_
> VECTOR`,`COLUMN_NAME`,`COLUMN_TYPE`,`DB_NAME`,`BIG_DECIMAL_
> HIGH_VALUE`,`BIG_DECIMAL_LOW_VALUE`,`DOUBLE_HIGH_VALUE`,`
> DOUBLE_LOW_VALUE`,`LAST_ANALYZED`,`LONG_HIGH_VALUE`,`
> LONG_LOW_VALUE`,`MAX_COL_LEN`,`NUM_DISTINCTS`,`NUM_FALSES`,`
> NUM_NULLS`,`NUM_TRUES`,`TBL_ID`,`TABLE_NAME`) VALUES
> (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)" failed : Unknown column
> 'BIT_VECTOR' in 'field list'
>
>
>
>
>
> 2017-10-27 12:02 GMT+08:00 Mostafa Mokhtar <mm...@cloudera.com>:
>
> Hi,
>
> Looks like you are joining store_sales with catalog_sales on item_sk, this
> kind of join condition is a many to many, which means the output number of
> rows will be much larger then input number of rows, not sure if this is
> intended.
>
> Also did you run "compute stats [TABLE_NAME]" on both tables?
>
> For a more comprehensive query try TPCDS Q17
>
> select i_item_id
>
> ,i_item_desc
>
> ,s_state
>
> ,count(ss_quantity) as store_sales_quantitycount
>
> ,avg(ss_quantity) as store_sales_quantityave
>
> ,stddev_samp(ss_quantity) as store_sales_quantitystdev
>
> ,stddev_samp(ss_quantity)/avg(ss_quantity) as
> store_sales_quantitycov
>
> ,count(sr_return_quantity) as store_returns_quantitycount
>
> ,avg(sr_return_quantity) as store_returns_quantityave
>
> ,stddev_samp(sr_return_quantity) as store_returns_quantitystdev
>
> ,stddev_samp(sr_return_quantity)/avg(sr_return_quantity) as
> store_returns_quantitycov
>
> ,count(cs_quantity) as catalog_sales_quantitycount
> ,avg(cs_quantity) as catalog_sales_quantityave
>
> ,stddev_samp(cs_quantity) as catalog_sales_quantitystdev
>
> ,stddev_samp(cs_quantity)/avg(cs_quantity) as
> catalog_sales_quantitycov
>
> from store_sales
>
> ,store_returns
>
> ,catalog_sales
>
> ,date_dim d1
>
> ,date_dim d2
>
> ,date_dim d3
>
> ,store
>
> ,item
>
> where d1.d_quarter_name = '2000Q1'
>
> and d1.d_date_sk = ss_sold_date_sk
>
> and i_item_sk = ss_item_sk
>
> and s_store_sk = ss_store_sk
>
> and ss_customer_sk = sr_customer_sk
>
> and ss_item_sk = sr_item_sk
>
> and ss_ticket_number = sr_ticket_number
>
> and sr_returned_date_sk = d2.d_date_sk
>
> and d2.d_quarter_name in ('2000Q1','2000Q2','2000Q3')
>
> and sr_customer_sk = cs_bill_customer_sk
>
> and sr_item_sk = cs_item_sk
>
> and cs_sold_date_sk = d3.d_date_sk
>
> and d3.d_quarter_name in ('2000Q1','2000Q2','2000Q3')
>
> group by i_item_id
>
> ,i_item_desc
>
> ,s_state
>
> order by i_item_id
>
> ,i_item_desc
>
> ,s_state
>
> limit 100;
>
>
> I recommend moving this kind of discussion on
> user@impala.incubator.apache.org.
>
> On Thu, Oct 26, 2017 at 7:25 PM, 俊杰陈 <cj...@gmail.com> wrote:
>
> > The profile file is damaged. Here is a screenshot for exec summary
> >
> >
> >
> > 2017-10-27 10:04 GMT+08:00 俊杰陈 <cj...@gmail.com>:
> >
> >> Hi Devs
> >>
> >> I met a performance issue on big table join. The query takes more than 3
> >> hours on Impala and only 3 minutes on Spark SQL on the same 5 nodes
> >> cluster. when running query, the left scanner and exchange node are
> very
> >> slow. Did I miss some key arguments?
> >>
> >> you can see profile file in attachment.
> >>
> >>
> >>
>
> >> --
> >> Thanks & Best Regards
> >>
> >
> >
> >
> > --
> > Thanks & Best Regards
> >
>
>
>
>
>
> --
>
> Thanks & Best Regards
>
>
>
--
Thanks & Best Regards
RE: performance issue on big table join
Posted by "Xu, Cheng A" <ch...@intel.com>.
Thanks Alexander for the reply. Is there any configuration we can use to determine the parallelism level for build phase? Thank you!
Thanks
Ferdinand Xu
From: Alexander Behm [mailto:alex.behm@cloudera.com]
Sent: Friday, October 27, 2017 1:46 PM
To: user@impala.incubator.apache.org
Cc: Mostafa Mokhtar <mm...@cloudera.com>
Subject: Re: performance issue on big table join
See my response on the other thread you started. The probe side of joins are are executed in a single thread per host. Impala can run multiple builds in parallel - but each build uses only a single thread.
A single query might not be able to max out your CPU, but most realistic workloads run several queries concurrently.
On Thu, Oct 26, 2017 at 10:38 PM, 俊杰陈 <cj...@gmail.com>> wrote:
Thanks, let me put here.
Yes, the query is intended to verify parallelism of partitioned join. We want to know how many fragment instances started for a hash join on a single node, and how many threads in a fragment instance perform the join. I'm not sure whether there is only one thread participate in hash join, since thus it can not maximize the CPU utilization.
The compute stats met following error:
Query: compute stats store_sales
WARNINGS: ImpalaRuntimeException: Error making 'updateTableColumnStatistics' RPC to Hive Metastore:
CAUSED BY: MetaException: Insert of object "org.apache.hadoop.hive.metastore.model.MTableColumnStatistics@49fe55a1<ma...@49fe55a1>" using statement "INSERT INTO `TAB_COL_STATS` (`CS_ID`,`AVG_COL_LEN`,`BIT_VECTOR`,`COLUMN_NAME`,`COLUMN_TYPE`,`DB_NAME`,`BIG_DECIMAL_HIGH_VALUE`,`BIG_DECIMAL_LOW_VALUE`,`DOUBLE_HIGH_VALUE`,`DOUBLE_LOW_VALUE`,`LAST_ANALYZED`,`LONG_HIGH_VALUE`,`LONG_LOW_VALUE`,`MAX_COL_LEN`,`NUM_DISTINCTS`,`NUM_FALSES`,`NUM_NULLS`,`NUM_TRUES`,`TBL_ID`,`TABLE_NAME`) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)" failed : Unknown column 'BIT_VECTOR' in 'field list'
2017-10-27 12:02 GMT+08:00 Mostafa Mokhtar <mm...@cloudera.com>>:
Hi,
Looks like you are joining store_sales with catalog_sales on item_sk, this
kind of join condition is a many to many, which means the output number of
rows will be much larger then input number of rows, not sure if this is
intended.
Also did you run "compute stats [TABLE_NAME]" on both tables?
For a more comprehensive query try TPCDS Q17
select i_item_id
,i_item_desc
,s_state
,count(ss_quantity) as store_sales_quantitycount
,avg(ss_quantity) as store_sales_quantityave
,stddev_samp(ss_quantity) as store_sales_quantitystdev
,stddev_samp(ss_quantity)/avg(ss_quantity) as store_sales_quantitycov
,count(sr_return_quantity) as store_returns_quantitycount
,avg(sr_return_quantity) as store_returns_quantityave
,stddev_samp(sr_return_quantity) as store_returns_quantitystdev
,stddev_samp(sr_return_quantity)/avg(sr_return_quantity) as
store_returns_quantitycov
,count(cs_quantity) as catalog_sales_quantitycount
,avg(cs_quantity) as catalog_sales_quantityave
,stddev_samp(cs_quantity) as catalog_sales_quantitystdev
,stddev_samp(cs_quantity)/avg(cs_quantity) as catalog_sales_quantitycov
from store_sales
,store_returns
,catalog_sales
,date_dim d1
,date_dim d2
,date_dim d3
,store
,item
where d1.d_quarter_name = '2000Q1'
and d1.d_date_sk = ss_sold_date_sk
and i_item_sk = ss_item_sk
and s_store_sk = ss_store_sk
and ss_customer_sk = sr_customer_sk
and ss_item_sk = sr_item_sk
and ss_ticket_number = sr_ticket_number
and sr_returned_date_sk = d2.d_date_sk
and d2.d_quarter_name in ('2000Q1','2000Q2','2000Q3')
and sr_customer_sk = cs_bill_customer_sk
and sr_item_sk = cs_item_sk
and cs_sold_date_sk = d3.d_date_sk
and d3.d_quarter_name in ('2000Q1','2000Q2','2000Q3')
group by i_item_id
,i_item_desc
,s_state
order by i_item_id
,i_item_desc
,s_state
limit 100;
I recommend moving this kind of discussion on
user@impala.incubator.apache.org<ma...@impala.incubator.apache.org>.
On Thu, Oct 26, 2017 at 7:25 PM, 俊杰陈 <cj...@gmail.com>> wrote:
> The profile file is damaged. Here is a screenshot for exec summary
>
>
>
> 2017-10-27 10:04 GMT+08:00 俊杰陈 <cj...@gmail.com>>:
>
>> Hi Devs
>>
>> I met a performance issue on big table join. The query takes more than 3
>> hours on Impala and only 3 minutes on Spark SQL on the same 5 nodes
>> cluster. when running query, the left scanner and exchange node are very
>> slow. Did I miss some key arguments?
>>
>> you can see profile file in attachment.
>>
>>
>>
>> --
>> Thanks & Best Regards
>>
>
>
>
> --
> Thanks & Best Regards
>
--
Thanks & Best Regards
Re: performance issue on big table join
Posted by Alexander Behm <al...@cloudera.com>.
See my response on the other thread you started. The probe side of joins
are are executed in a single thread per host. Impala can run multiple
builds in parallel - but each build uses only a single thread.
A single query might not be able to max out your CPU, but most realistic
workloads run several queries concurrently.
On Thu, Oct 26, 2017 at 10:38 PM, 俊杰陈 <cj...@gmail.com> wrote:
> Thanks, let me put here.
>
> Yes, the query is intended to verify parallelism of partitioned join. We
> want to know how many fragment instances started for a hash join on a
> single node, and how many threads in a fragment instance perform the join.
> I'm not sure whether there is only one thread participate in hash join,
> since thus it can not maximize the CPU utilization.
>
> The compute stats met following error:
>
> Query: compute stats store_sales
> WARNINGS: ImpalaRuntimeException: Error making
> 'updateTableColumnStatistics' RPC to Hive Metastore:
> CAUSED BY: MetaException: Insert of object "org.apache.hadoop.hive.
> metastore.model.MTableColumnStatistics@49fe55a1" using statement "INSERT
> INTO `TAB_COL_STATS` (`CS_ID`,`AVG_COL_LEN`,`BIT_
> VECTOR`,`COLUMN_NAME`,`COLUMN_TYPE`,`DB_NAME`,`BIG_DECIMAL_
> HIGH_VALUE`,`BIG_DECIMAL_LOW_VALUE`,`DOUBLE_HIGH_VALUE`,`
> DOUBLE_LOW_VALUE`,`LAST_ANALYZED`,`LONG_HIGH_VALUE`,`
> LONG_LOW_VALUE`,`MAX_COL_LEN`,`NUM_DISTINCTS`,`NUM_FALSES`,`
> NUM_NULLS`,`NUM_TRUES`,`TBL_ID`,`TABLE_NAME`) VALUES
> (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)" failed : Unknown column
> 'BIT_VECTOR' in 'field list'
>
>
> 2017-10-27 12:02 GMT+08:00 Mostafa Mokhtar <mm...@cloudera.com>:
>
>> Hi,
>>
>> Looks like you are joining store_sales with catalog_sales on item_sk, this
>> kind of join condition is a many to many, which means the output number of
>> rows will be much larger then input number of rows, not sure if this is
>> intended.
>>
>> Also did you run "compute stats [TABLE_NAME]" on both tables?
>>
>> For a more comprehensive query try TPCDS Q17
>>
>> select i_item_id
>>
>> ,i_item_desc
>>
>> ,s_state
>>
>> ,count(ss_quantity) as store_sales_quantitycount
>>
>> ,avg(ss_quantity) as store_sales_quantityave
>>
>> ,stddev_samp(ss_quantity) as store_sales_quantitystdev
>>
>> ,stddev_samp(ss_quantity)/avg(ss_quantity) as
>> store_sales_quantitycov
>>
>> ,count(sr_return_quantity) as store_returns_quantitycount
>>
>> ,avg(sr_return_quantity) as store_returns_quantityave
>>
>> ,stddev_samp(sr_return_quantity) as store_returns_quantitystdev
>>
>> ,stddev_samp(sr_return_quantity)/avg(sr_return_quantity) as
>> store_returns_quantitycov
>>
>> ,count(cs_quantity) as catalog_sales_quantitycount
>> ,avg(cs_quantity) as catalog_sales_quantityave
>>
>> ,stddev_samp(cs_quantity) as catalog_sales_quantitystdev
>>
>> ,stddev_samp(cs_quantity)/avg(cs_quantity) as
>> catalog_sales_quantitycov
>>
>> from store_sales
>>
>> ,store_returns
>>
>> ,catalog_sales
>>
>> ,date_dim d1
>>
>> ,date_dim d2
>>
>> ,date_dim d3
>>
>> ,store
>>
>> ,item
>>
>> where d1.d_quarter_name = '2000Q1'
>>
>> and d1.d_date_sk = ss_sold_date_sk
>>
>> and i_item_sk = ss_item_sk
>>
>> and s_store_sk = ss_store_sk
>>
>> and ss_customer_sk = sr_customer_sk
>>
>> and ss_item_sk = sr_item_sk
>>
>> and ss_ticket_number = sr_ticket_number
>>
>> and sr_returned_date_sk = d2.d_date_sk
>>
>> and d2.d_quarter_name in ('2000Q1','2000Q2','2000Q3')
>>
>> and sr_customer_sk = cs_bill_customer_sk
>>
>> and sr_item_sk = cs_item_sk
>>
>> and cs_sold_date_sk = d3.d_date_sk
>>
>> and d3.d_quarter_name in ('2000Q1','2000Q2','2000Q3')
>>
>> group by i_item_id
>>
>> ,i_item_desc
>>
>> ,s_state
>>
>> order by i_item_id
>>
>> ,i_item_desc
>>
>> ,s_state
>>
>> limit 100;
>>
>>
>> I recommend moving this kind of discussion on
>> user@impala.incubator.apache.org.
>>
>> On Thu, Oct 26, 2017 at 7:25 PM, 俊杰陈 <cj...@gmail.com> wrote:
>>
>> > The profile file is damaged. Here is a screenshot for exec summary
>> >
>> >
>> >
>> > 2017-10-27 10:04 GMT+08:00 俊杰陈 <cj...@gmail.com>:
>> >
>> >> Hi Devs
>> >>
>> >> I met a performance issue on big table join. The query takes more than
>> 3
>> >> hours on Impala and only 3 minutes on Spark SQL on the same 5 nodes
>> >> cluster. when running query, the left scanner and exchange node are
>> very
>> >> slow. Did I miss some key arguments?
>> >>
>> >> you can see profile file in attachment.
>> >>
>> >>
>> >>
>> >> --
>> >> Thanks & Best Regards
>> >>
>> >
>> >
>> >
>> > --
>> > Thanks & Best Regards
>> >
>>
>
>
>
> --
> Thanks & Best Regards
>