You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@impala.apache.org by Sumudu Madushanka <ng...@cse.mrt.ac.lk> on 2022/05/02 09:31:02 UTC
RE: Impala Performance Issue with huge Kudu Data load
Query (id=test_id):
Summary:
Session ID: test_id_1:test_id_2
Session Type: HIVESERVER2
HiveServer2 Protocol Version: V6
Start Time: 2022-05-02 09:45:06.613721000
End Time: 2022-05-02 09:45:56.470552000
Query Type: QUERY
Query State: FINISHED
Impala Query State: FINISHED
Query Status: OK
Impala Version: impalad version 3.4.0-RELEASE RELEASE (build Could not obtain git hash)
User:
Connected User:
Delegated User:
Network Address: ***.***.***.***:******
Default Db: default
Sql Statement: SELECT * from table_abc where (col_1>=num_1 AND col_1<num_2) AND col_2 ='value_1' ORDER BY col_1 ASC, col_3 ASC LIMIT 250000
Coordinator: impala-coordinator-node-1:22000
Query Options (set by configuration): TIMEZONE=timezone
Query Options (set by configuration and planner): MT_DOP=0,TIMEZONE=timezone
Plan:
----------------
Max Per-Host Resource Reservation: Memory=0B Threads=3
Per-Host Resource Estimates: Memory=422MB
Dedicated Coordinator Resource Estimate: Memory=106MB
WARNING: The following tables are missing relevant table and/or column statistics.
default.table_abc
Analyzed query: SELECT * FROM `default`.table_abc WHERE
col_1 >= CAST(num_1 AS BIGINT) AND col_1 <
CAST(num_2 AS BIGINT) AND col_2 = 'value_1' ORDER BY
col_1 ASC, col_3 ASC LIMIT CAST(250000 AS INT)
F01:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1
| Per-Host Resources: mem-estimate=5.51MB mem-reservation=0B thread-reservation=1
PLAN-ROOT SINK
| output exprs: col_2, col_1, col_3, col_4, col_5, col_6, col_7, col_8, col_9, col_10, col_11, col_12, col_13, col_14, col_15, col_16, col_17, col_18, col_19, col_20, col_21, col_22, col_23, col_24, col_25, col_26, col_27, col_28, col_29, col_30, col_31, col_32, col_33, col_34, col_35, col_36, col_37, col_38, col_39, col_40, col_41, col_42, col_43, col_44, col_45, col_46, col_47, col_48, col_49, col_50, col_51, col_52, col_53, col_54, col_55, col_56, col_57, col_58, col_59, col_60, col_61, col_62, col_63, col_64, col_65, col_66, col_67, col_68, col_69, col_70, col_71, col_72, col_73, col_74, col_75, col_76, col_77, col_78, col_79, col_80, col_81, col_82, col_83, col_84, col_85, col_86, col_87, col_88, col_89, col_90, col_91, col_92, col_93, col_94, col_95, col_96, col_97, col_98, col_99, col_100, col_101, col_102, col_103, col_104, col_105, col_106, col_107, col_108, col_109, col_110, col_111, col_112, col_113, col_114, col_115, col_116, col_117
| mem-estimate=0B mem-reservation=0B thread-reservation=0
|
02:MERGING-EXCHANGE [UNPARTITIONED]
| order by: col_1 ASC, col_3 ASC
| limit: 250000
| mem-estimate=5.51MB mem-reservation=0B thread-reservation=0
| tuple-ids=1 row-size=936B cardinality=0
| in pipelines: 01(GETNEXT)
|
F00:PLAN FRAGMENT [RANDOM] hosts=6 instances=6
Per-Host Resources: mem-estimate=416.00MB mem-reservation=0B thread-reservation=2
01:TOP-N [LIMIT=250000]
| order by: col_1 ASC, col_3 ASC
| mem-estimate=0B mem-reservation=0B thread-reservation=0
| tuple-ids=1 row-size=936B cardinality=0
| in pipelines: 01(GETNEXT), 00(OPEN)
|
00:SCAN KUDU [default.table_abc]
kudu predicates: col_1 < CAST(num_2 AS BIGINT), col_1 >= CAST(num_1 AS BIGINT), col_2 = 'value_1'
mem-estimate=416.00MB mem-reservation=0B thread-reservation=1
tuple-ids=0 row-size=1.09KB cardinality=0
in pipelines: 00(GETNEXT)
----------------
Estimated Per-Host Mem: 441982976
Tables Missing Stats: default.table_abc
Request Pool: default-pool
Per Host Min Memory Reservation: node-1:22000(0) node-5:22000(0) node-6:22000(0) node-4:22000(0) node-3:22000(0) node-2:22000(0) impala-coordinator-node-1:22000(0)
Per Host Number of Fragment Instances: node-1:22000(1) node-5:22000(1) node-6:22000(1) node-4:22000(1) node-3:22000(1) node-2:22000(1) impala-coordinator-node-1:22000(1)
Admission result: Admitted immediately
Cluster Memory Admitted: 2.57 GB
Executor Group: default
ExecSummary:
Operator #Hosts #Inst Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
--------------------------------------------------------------------------------------------------------------------------------------------
F01:ROOT 1 1 625.002ms 625.002ms 0 0
02:MERGING-EXCHANGE 1 1 10.000ms 10.000ms 250.00K 0 44.60 MB 5.51 MB UNPARTITIONED
F00:EXCHANGE SENDER 6 6 872.503ms 884.003ms 136.00 B 0
01:TOP-N 6 6 15s490ms 16s627ms 1.50M 0 772.60 MB 0
00:SCAN KUDU 6 6 19s313ms 20s704ms 212.00M 0 33.28 MB 416.00 MB default.table_abc
Errors:
Query Compilation: 7s284ms
- Metadata load started: 86.966ms (86.966ms)
- Metadata load finished. loaded-tables=1/1 load-requests=1 catalog-updates=5 storage-load-time=996ms: 6s062ms (5s975ms)
- Analysis finished: 6s133ms (71.252ms)
- Authorization finished (noop): 6s134ms (1.170ms)
- Value transfer graph computed: 6s173ms (39.176ms)
- Single node plan created: 7s206ms (1s032ms)
- Runtime filters computed: 7s211ms (4.579ms)
- Distributed plan created: 7s212ms (1.348ms)
- Planning finished: 7s284ms (71.641ms)
Query Timeline: 49s896ms
- Query submitted: 1.000ms (1.000ms)
- Planning finished: 7s356ms (7s355ms)
- Submit for admission: 7s356ms (0.000ns)
- Completed admission: 7s356ms (0.000ns)
- Ready to start on 7 backends: 7s357ms (1.000ms)
- All 7 execution backends (7 fragment instances) started: 7s374ms (17.000ms)
- Rows available: 45s049ms (37s675ms)
- First row fetched: 45s095ms (46.000ms)
- Last row fetched: 49s126ms (4s031ms)
- Released admission control resources: 49s858ms (732.002ms)
- Unregister query: 49s896ms (38.000ms)
- AdmissionControlTimeSinceLastUpdate: 50.000ms
- ComputeScanRangeAssignmentTimer: 0.000ns
Frontend:
ImpalaServer:
- ClientFetchWaitTimer: 3s486ms
- NumRowsFetched: 250.00K (250000)
- NumRowsFetchedFromCache: 0 (0)
- RowMaterializationRate: 183.69 K/sec
- RowMaterializationTimer: 1s361ms
Execution Profile test_id:(Total: 39s053ms, non-child: 0.000ns, % non-child: 0.00%)
Number of filters: 0
Filter routing table:
ID Src. Node Tgt. Node(s) Target type Partition filter Pending (Expected) First arrived Completed Enabled
-------------------------------------------------------------------------------------------------------------------
Backend startup latencies: Count: 7, min / max: 4ms / 15ms, 25th %-ile: 8ms, 50th %-ile: 10ms, 75th %-ile: 15ms, 90th %-ile: 15ms, 95th %-ile: 15ms, 99.9th %-ile: 15ms
Slowest backend to start up: node-6:22000
Per Node Peak Memory Usage: node-1:22000(803.20 MB) node-5:22000(803.20 MB) node-6:22000(804.80 MB) node-4:22000(804.80 MB) node-3:22000(804.80 MB) node-2:22000(804.30 MB) impala-coordinator-node-1:22000(44.62 MB)
Per Node Bytes Read: node-1:22000(0) node-5:22000(0) node-6:22000(0) node-4:22000(0) node-3:22000(0) node-2:22000(0) impala-coordinator-node-1:22000(0)
Per Node User Time: node-1:22000(37s983ms) node-5:22000(38s197ms) node-6:22000(42s559ms) node-4:22000(42s054ms) node-3:22000(42s405ms) node-2:22000(41s349ms) impala-coordinator-node-1:22000(816.651ms)
Per Node System Time: node-1:22000(23s847ms) node-5:22000(24s348ms) node-6:22000(26s663ms) node-4:22000(26s685ms) node-3:22000(27s136ms) node-2:22000(26s544ms) impala-coordinator-node-1:22000(15.825ms)
- ExchangeScanRatio: 0.00
- FiltersReceived: 0 (0)
- FinalizationTimer: 0.000ns
- InnerNodeSelectivityRatio: 0.00
- NumBackends: 7 (7)
- NumCompletedBackends: 7 (7)
- NumFragmentInstances: 7 (7)
- NumFragments: 2 (2)
- TotalBytesRead: 0
- TotalBytesSent: 105.79 MB (110929234)
- TotalCpuTime: 6m40s
- TotalInnerBytesSent: 105.79 MB (110929234)
- TotalScanBytesSent: 0
Per Node Profiles:
node-1:22000:
- AdmissionSlots: 1 (1)
- BloomFilterBytes: 0
- ScratchBytesRead: 0
- ScratchBytesWritten: 0
- ScratchFileUsedBytes: 0
- ScratchReads: 0 (0)
- ScratchWrites: 0 (0)
- TotalEncryptionTime: 0.000ns
- TotalReadBlockTime: 0.000ns
Buffer pool:
- AllocTime: 0.000ns
- CumulativeAllocationBytes: 0
- CumulativeAllocations: 0 (0)
- PeakReservation: 0
- PeakUnpinnedBytes: 0
- PeakUsedReservation: 0
- ReadIoBytes: 0
- ReadIoOps: 0 (0)
- ReadIoWaitTime: 0.000ns
- ReservationLimit: 0
- SystemAllocTime: 0.000ns
- WriteIoBytes: 0
- WriteIoOps: 0 (0)
- WriteIoWaitTime: 0.000ns
node-5:22000:
- AdmissionSlots: 1 (1)
- BloomFilterBytes: 0
- ScratchBytesRead: 0
- ScratchBytesWritten: 0
- ScratchFileUsedBytes: 0
- ScratchReads: 0 (0)
- ScratchWrites: 0 (0)
- TotalEncryptionTime: 0.000ns
- TotalReadBlockTime: 0.000ns
Buffer pool:
- AllocTime: 0.000ns
- CumulativeAllocationBytes: 0
- CumulativeAllocations: 0 (0)
- PeakReservation: 0
- PeakUnpinnedBytes: 0
- PeakUsedReservation: 0
- ReadIoBytes: 0
- ReadIoOps: 0 (0)
- ReadIoWaitTime: 0.000ns
- ReservationLimit: 0
- SystemAllocTime: 0.000ns
- WriteIoBytes: 0
- WriteIoOps: 0 (0)
- WriteIoWaitTime: 0.000ns
node-6:22000:
- AdmissionSlots: 1 (1)
- BloomFilterBytes: 0
- ScratchBytesRead: 0
- ScratchBytesWritten: 0
- ScratchFileUsedBytes: 0
- ScratchReads: 0 (0)
- ScratchWrites: 0 (0)
- TotalEncryptionTime: 0.000ns
- TotalReadBlockTime: 0.000ns
Buffer pool:
- AllocTime: 0.000ns
- CumulativeAllocationBytes: 0
- CumulativeAllocations: 0 (0)
- PeakReservation: 0
- PeakUnpinnedBytes: 0
- PeakUsedReservation: 0
- ReadIoBytes: 0
- ReadIoOps: 0 (0)
- ReadIoWaitTime: 0.000ns
- ReservationLimit: 0
- SystemAllocTime: 0.000ns
- WriteIoBytes: 0
- WriteIoOps: 0 (0)
- WriteIoWaitTime: 0.000ns
node-4:22000:
- AdmissionSlots: 1 (1)
- BloomFilterBytes: 0
- ScratchBytesRead: 0
- ScratchBytesWritten: 0
- ScratchFileUsedBytes: 0
- ScratchReads: 0 (0)
- ScratchWrites: 0 (0)
- TotalEncryptionTime: 0.000ns
- TotalReadBlockTime: 0.000ns
Buffer pool:
- AllocTime: 0.000ns
- CumulativeAllocationBytes: 0
- CumulativeAllocations: 0 (0)
- PeakReservation: 0
- PeakUnpinnedBytes: 0
- PeakUsedReservation: 0
- ReadIoBytes: 0
- ReadIoOps: 0 (0)
- ReadIoWaitTime: 0.000ns
- ReservationLimit: 0
- SystemAllocTime: 0.000ns
- WriteIoBytes: 0
- WriteIoOps: 0 (0)
- WriteIoWaitTime: 0.000ns
node-3:22000:
- AdmissionSlots: 1 (1)
- BloomFilterBytes: 0
- ScratchBytesRead: 0
- ScratchBytesWritten: 0
- ScratchFileUsedBytes: 0
- ScratchReads: 0 (0)
- ScratchWrites: 0 (0)
- TotalEncryptionTime: 0.000ns
- TotalReadBlockTime: 0.000ns
Buffer pool:
- AllocTime: 0.000ns
- CumulativeAllocationBytes: 0
- CumulativeAllocations: 0 (0)
- PeakReservation: 0
- PeakUnpinnedBytes: 0
- PeakUsedReservation: 0
- ReadIoBytes: 0
- ReadIoOps: 0 (0)
- ReadIoWaitTime: 0.000ns
- ReservationLimit: 0
- SystemAllocTime: 0.000ns
- WriteIoBytes: 0
- WriteIoOps: 0 (0)
- WriteIoWaitTime: 0.000ns
node-2:22000:
- AdmissionSlots: 1 (1)
- BloomFilterBytes: 0
- ScratchBytesRead: 0
- ScratchBytesWritten: 0
- ScratchFileUsedBytes: 0
- ScratchReads: 0 (0)
- ScratchWrites: 0 (0)
- TotalEncryptionTime: 0.000ns
- TotalReadBlockTime: 0.000ns
Buffer pool:
- AllocTime: 0.000ns
- CumulativeAllocationBytes: 0
- CumulativeAllocations: 0 (0)
- PeakReservation: 0
- PeakUnpinnedBytes: 0
- PeakUsedReservation: 0
- ReadIoBytes: 0
- ReadIoOps: 0 (0)
- ReadIoWaitTime: 0.000ns
- ReservationLimit: 0
- SystemAllocTime: 0.000ns
- WriteIoBytes: 0
- WriteIoOps: 0 (0)
- WriteIoWaitTime: 0.000ns
impala-coordinator-node-1:22000:
- AdmissionSlots: 1 (1)
- BloomFilterBytes: 0
- ScratchBytesRead: 0
- ScratchBytesWritten: 0
- ScratchFileUsedBytes: 0
- ScratchReads: 0 (0)
- ScratchWrites: 0 (0)
- TotalEncryptionTime: 0.000ns
- TotalReadBlockTime: 0.000ns
Buffer pool:
- AllocTime: 0.000ns
- CumulativeAllocationBytes: 0
- CumulativeAllocations: 0 (0)
- PeakReservation: 0
- PeakUnpinnedBytes: 0
- PeakUsedReservation: 0
- ReadIoBytes: 0
- ReadIoOps: 0 (0)
- ReadIoWaitTime: 0.000ns
- ReservationLimit: 0
- SystemAllocTime: 0.000ns
- WriteIoBytes: 0
- WriteIoOps: 0 (0)
- WriteIoWaitTime: 0.000ns
Averaged Fragment F01:(Total: 41s751ms, non-child: 0.000ns, % non-child: 0.00%)
split sizes: min: 0, max: 0, avg: 0, stddev: 0
completion times: min:41s752ms max:41s752ms mean: 41s752ms stddev:0.000ns
execution rates: min:0.00 /sec max:0.00 /sec mean:0.00 /sec stddev:0.00 /sec
num instances: 1
- AverageThreadTokens: 0.00
- ExchangeScanRatio: 0.00
- PeakMemoryUsage: 44.62 MB (46784837)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 44.62 MB (46784837)
- RowsProduced: 250.00K (250000)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 0.000ns
- TotalStorageWaitTime: 0.000ns
- TotalThreadsInvoluntaryContextSwitches: 0 (0)
- TotalThreadsTotalWallClockTime: 41s748ms
- TotalThreadsSysTime: 15.825ms
- TotalThreadsUserTime: 816.651ms
- TotalThreadsVoluntaryContextSwitches: 584 (584)
Fragment Instance Lifecycle Timings:
- ExecTime: 4s065ms
- ExecTreeExecTime: 10.000ms
- OpenTime: 37s683ms
- ExecTreeOpenTime: 37s466ms
- PrepareTime: 2.000ms
- ExecTreePrepareTime: 0.000ns
PLAN_ROOT_SINK:(Total: 4s055ms, non-child: 0.000ns, % non-child: 0.00%)
- PeakMemoryUsage: 0
- RowsSent: 250.00K (250000)
- RowsSentRate: 61.65 K/sec
EXCHANGE_NODE (id=2):(Total: 37s476ms, non-child: 0.000ns, % non-child: 0.00%)
- ConvertRowBatchTime: 0.000ns
- MergeGetNext: 10.000ms
- MergeGetNextBatch: 1.000ms
- PeakMemoryUsage: 44.60 MB (46771278)
- RowsReturned: 250.00K (250000)
- RowsReturnedRate: 6.67 K/sec
Buffer pool:
- AllocTime: 4.000ms
- CumulativeAllocationBytes: 511.99 MB (536862720)
- CumulativeAllocations: 510 (510)
- PeakReservation: 42.14 MB (44187648)
- PeakUnpinnedBytes: 0
- PeakUsedReservation: 42.14 MB (44187648)
- ReadIoBytes: 0
- ReadIoOps: 0 (0)
- ReadIoWaitTime: 0.000ns
- SystemAllocTime: 4.000ms
- WriteIoBytes: 0
- WriteIoOps: 0 (0)
- WriteIoWaitTime: 0.000ns
Dequeue:
- FirstBatchWaitTime: 37s466ms
- TotalBytesDequeued: 293.49 MB (307742393)
- TotalGetBatchTime: 37s467ms
- DataWaitTime: 37s466ms
Enqueue:
- DeserializeRowBatchTime: 242.000ms
- TotalBatchesEnqueued: 255 (255)
- TotalBatchesReceived: 261 (261)
- TotalBytesReceived: 105.79 MB (110929234)
- TotalEarlySenders: 0 (0)
- TotalEosReceived: 0 (0)
- TotalHasDeferredRPCsTime: 37s114ms
- TotalRPCsDeferred: 246 (246)
CodeGen:(Total: 217.000ms, non-child: 0.000ns, % non-child: 0.00%)
- CodegenInvoluntaryContextSwitches: 0 (0)
- CodegenTotalWallClockTime: 217.000ms
- CodegenSysTime: 5.794ms
- CodegenUserTime: 136.811ms
- CodegenVoluntaryContextSwitches: 93 (93)
- CompileTime: 96.000ms
- IrGenerationTime: 10.000ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 2.48 MB (2595844)
- NumFunctions: 134 (134)
- NumInstructions: 3.50K (3499)
- OptimizationTime: 97.000ms
- PeakMemoryUsage: 1.71 MB (1791488)
- PrepareTime: 13.000ms
Coordinator Fragment F01:
Instance test_id (host=impala-coordinator-node-1:22000):(Total: 41s751ms, non-child: 0.000ns, % non-child: 0.00%)
Last report received time: 2022-05-02 09:45:55.728
Fragment Instance Lifecycle Event Timeline: 41s751ms
- Prepare Finished: 3.000ms (3.000ms)
- Open Finished: 37s686ms (37s683ms)
- First Batch Produced: 37s686ms (0.000ns)
- First Batch Sent: 37s812ms (126.000ms)
- ExecInternal Finished: 41s751ms (3s939ms)
- MemoryUsage (1s000ms): 13.24 KB, 13.24 KB, 13.24 KB, 13.24 KB, 13.24 KB, 13.24 KB, 13.24 KB, 13.24 KB, 13.24 KB, 13.24 KB, 13.24 KB, 13.24 KB, 13.24 KB, 13.24 KB, 13.24 KB, 13.24 KB, 13.24 KB, 13.24 KB, 13.24 KB, 13.24 KB, 13.24 KB, 13.24 KB, 13.24 KB, 13.24 KB, 13.24 KB, 13.24 KB, 13.24 KB, 13.24 KB, 13.24 KB, 13.24 KB, 13.24 KB, 13.24 KB, 13.24 KB, 13.24 KB, 21.07 MB, 24.47 MB, 28.06 MB, 33.28 MB, 44.39 MB, 44.59 MB, 44.59 MB
- AverageThreadTokens: 0.00
- ExchangeScanRatio: 0.00
- PeakMemoryUsage: 44.62 MB (46784837)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 44.62 MB (46784837)
- RowsProduced: 250.00K (250000)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 0.000ns
- TotalStorageWaitTime: 0.000ns
- TotalThreadsInvoluntaryContextSwitches: 0 (0)
- TotalThreadsTotalWallClockTime: 41s748ms
- TotalThreadsSysTime: 15.825ms
- TotalThreadsUserTime: 816.651ms
- TotalThreadsVoluntaryContextSwitches: 584 (584)
Fragment Instance Lifecycle Timings:
- ExecTime: 4s065ms
- ExecTreeExecTime: 10.000ms
- OpenTime: 37s683ms
- ExecTreeOpenTime: 37s466ms
- PrepareTime: 2.000ms
- ExecTreePrepareTime: 0.000ns
PLAN_ROOT_SINK:(Total: 4s055ms, non-child: 0.000ns, % non-child: 0.00%)
- PeakMemoryUsage: 0
- RowsSent: 250.00K (250000)
- RowsSentRate: 61.65 K/sec
EXCHANGE_NODE (id=2):(Total: 37s476ms, non-child: 0.000ns, % non-child: 0.00%)
ExecOption: Codegen Enabled
Node Lifecycle Event Timeline: 41s751ms
- Open Started: 220.000ms (220.000ms)
- Open Finished: 37s686ms (37s466ms)
- First Batch Requested: 37s686ms (0.000ns)
- First Batch Returned: 37s686ms (0.000ns)
- Last Batch Returned: 41s748ms (4s062ms)
- Closed: 41s751ms (3.000ms)
- ConvertRowBatchTime: 0.000ns
- MergeGetNext: 10.000ms
- MergeGetNextBatch: 1.000ms
- PeakMemoryUsage: 44.60 MB (46771278)
- RowsReturned: 250.00K (250000)
- RowsReturnedRate: 6.67 K/sec
Buffer pool:
- AllocTime: 4.000ms
- CumulativeAllocationBytes: 511.99 MB (536862720)
- CumulativeAllocations: 510 (510)
- PeakReservation: 42.14 MB (44187648)
- PeakUnpinnedBytes: 0
- PeakUsedReservation: 42.14 MB (44187648)
- ReadIoBytes: 0
- ReadIoOps: 0 (0)
- ReadIoWaitTime: 0.000ns
- SystemAllocTime: 4.000ms
- WriteIoBytes: 0
- WriteIoOps: 0 (0)
- WriteIoWaitTime: 0.000ns
Dequeue:
- BytesDequeued (1s000ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 601.98 KB, 1.18 MB, 1.18 MB, 4.12 MB, 58.01 MB, 132.81 MB, 205.43 MB
- FirstBatchWaitTime: 37s466ms
- TotalBytesDequeued: 293.49 MB (307742393)
- TotalGetBatchTime: 37s467ms
- DataWaitTime: 37s466ms
Enqueue:
- BytesReceived (1s000ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4.91 MB, 5.88 MB, 7.01 MB, 8.17 MB, 25.02 MB, 50.82 MB, 75.68 MB
- DeferredQueueSize (1s000ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 3, 5, 5, 5, 6, 6
- DispatchTime: (Avg: -22246.000ns ; Min: -71104.000ns ; Max: -13189.000ns ; Number of samples: 261)
- DeserializeRowBatchTime: 242.000ms
- TotalBatchesEnqueued: 255 (255)
- TotalBatchesReceived: 261 (261)
- TotalBytesReceived: 105.79 MB (110929234)
- TotalEarlySenders: 0 (0)
- TotalEosReceived: 0 (0)
- TotalHasDeferredRPCsTime: 37s114ms
- TotalRPCsDeferred: 246 (246)
CodeGen:(Total: 217.000ms, non-child: 0.000ns, % non-child: 0.00%)
- CodegenInvoluntaryContextSwitches: 0 (0)
- CodegenTotalWallClockTime: 217.000ms
- CodegenSysTime: 5.794ms
- CodegenUserTime: 136.811ms
- CodegenVoluntaryContextSwitches: 93 (93)
- CompileTime: 96.000ms
- IrGenerationTime: 10.000ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 2.48 MB (2595844)
- NumFunctions: 134 (134)
- NumInstructions: 3.50K (3499)
- OptimizationTime: 97.000ms
- PeakMemoryUsage: 1.71 MB (1791488)
- PrepareTime: 13.000ms
Averaged Fragment F00:(Total: 42s478ms, non-child: 0.000ns, % non-child: 0.00%)
split sizes: min: 0, max: 0, avg: 0, stddev: 0
completion times: min:42s473ms max:42s490ms mean: 42s480ms stddev:5.335ms
execution rates: min:0.00 /sec max:0.00 /sec mean:0.00 /sec stddev:0.00 /sec
num instances: 6
- AverageThreadTokens: 9.55
- ExchangeScanRatio: 0.00
- PeakMemoryUsage: 804.18 MB (843248438)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 804.18 MB (843248438)
- RowsProduced: 250.00K (250000)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 6s151ms
- TotalStorageWaitTime: 4m50s
- TotalThreadsInvoluntaryContextSwitches: 24 (24)
- TotalThreadsTotalWallClockTime: 6m45s
- TotalThreadsSysTime: 25s871ms
- TotalThreadsUserTime: 40s758ms
- TotalThreadsVoluntaryContextSwitches: 88.47K (88472)
Fragment Instance Lifecycle Timings:
- ExecTime: 7s063ms
- ExecTreeExecTime: 1.833ms
- OpenTime: 35s413ms
- ExecTreeOpenTime: 34s801ms
- PrepareTime: 1.333ms
- ExecTreePrepareTime: 166.666us
KrpcDataStreamSender (dst_id=2):(Total: 7s023ms, non-child: 0.000ns, % non-child: 0.00%)
- EosSent: 0 (0)
- PeakMemoryUsage: 136.00 B (136)
- RowsSent: 250.00K (250000)
- RpcFailure: 0 (0)
- RpcRetry: 0 (0)
- SerializeBatchTime: 872.003ms
- TotalBytesSent: 17.63 MB (18488205)
- UncompressedRowBatchSize: 290.62 MB (304736180)
SORT_NODE (id=1):(Total: 34s803ms, non-child: 0.000ns, % non-child: 0.00%)
- InsertBatchTime: 10s491ms
- PeakMemoryUsage: 772.52 MB (810043050)
- RowsReturned: 250.00K (250000)
- RowsReturnedRate: 7.19 K/sec
- TuplePoolReclamations: 2 (2)
KUDU_SCAN_NODE (id=0):(Total: 19s313ms, non-child: 0.000ns, % non-child: 0.00%)
- AverageScannerThreadConcurrency: 10.45
- KuduClientTime: 4m50s
- KuduRemoteScanTokens: 0 (0)
- MaterializeTupleTime: 5m28s
- NumScannerThreadMemUnavailable: 0 (0)
- NumScannerThreadsStarted: 10 (10)
- PeakMemoryUsage: 32.56 MB (34138282)
- PeakScannerThreadConcurrency: 10 (10)
- RowBatchBytesEnqueued: 53.94 GB (57921613141)
- RowBatchQueueGetWaitTime: 19s171ms
- RowBatchQueuePeakMemoryUsage: 31.63 MB (33168384)
- RowBatchQueuePutWaitTime: 31s818ms
- RowBatchesEnqueued: 34.51K (34511)
- RowsRead: 35.33M (35333550)
- RowsReturned: 35.33M (35333550)
- RowsReturnedRate: 1.83 M/sec
- ScanRangesComplete: 10 (10)
- ScannerThreadsInvoluntaryContextSwitches: 11 (11)
- ScannerThreadsTotalWallClockTime: 6m2s
- ScannerThreadsSysTime: 20s596ms
- ScannerThreadsUserTime: 28s914ms
- ScannerThreadsVoluntaryContextSwitches: 58.55K (58545)
- TotalKuduScanRoundTrips: 46.03K (46029)
CodeGen:(Total: 611.501ms, non-child: 0.000ns, % non-child: 0.00%)
- CodegenInvoluntaryContextSwitches: 0 (0)
- CodegenTotalWallClockTime: 611.334ms
- CodegenSysTime: 15.765ms
- CodegenUserTime: 517.740ms
- CodegenVoluntaryContextSwitches: 92 (92)
- CompileTime: 198.667ms
- IrGenerationTime: 15.666ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 2.48 MB (2595844)
- NumFunctions: 168 (168)
- NumInstructions: 8.93K (8927)
- OptimizationTime: 384.167ms
- PeakMemoryUsage: 4.36 MB (4570624)
- PrepareTime: 12.500ms
Fragment F00:
Instance 6044b4a39c7f7962:eed7043600000006 (host=node-1:22000):(Total: 42s487ms, non-child: 0.000ns, % non-child: 0.00%)
Last report received time: 2022-05-02 09:45:56.470
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
Fragment Instance Lifecycle Event Timeline: 42s487ms
- Prepare Finished: 999.998us (999.998us)
- Open Finished: 34s009ms (34s008ms)
- First Batch Produced: 34s009ms (0.000ns)
- First Batch Sent: 34s015ms (6.000ms)
- ExecInternal Finished: 42s487ms (8s472ms)
- MemoryUsage (1s000ms): 2.21 MB, 382.34 MB, 453.66 MB, 389.32 MB, 545.66 MB, 398.46 MB, 450.41 MB, 341.66 MB, 364.66 MB, 374.42 MB, 376.68 MB, 376.12 MB, 376.92 MB, 376.12 MB, 376.12 MB, 376.12 MB, 376.92 MB, 376.12 MB, 376.12 MB, 378.42 MB, 379.92 MB, 379.12 MB, 379.92 MB, 379.12 MB, 379.92 MB, 379.92 MB, 379.12 MB, 379.92 MB, 379.12 MB, 379.92 MB, 379.12 MB, 379.12 MB, 378.87 MB, 371.08 MB, 363.03 MB, 363.03 MB, 363.03 MB, 363.03 MB, 363.03 MB, 363.03 MB, 363.03 MB, 363.03 MB
- ThreadUsage (1s000ms): 6, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 5, 1, 1, 1, 1, 1, 1, 1, 1
- AverageThreadTokens: 8.75
- ExchangeScanRatio: 0.00
- PeakMemoryUsage: 803.20 MB (842211468)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 803.20 MB (842211468)
- RowsProduced: 250.00K (250000)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 7s563ms
- TotalStorageWaitTime: 4m22s
- TotalThreadsInvoluntaryContextSwitches: 20 (20)
- TotalThreadsTotalWallClockTime: 6m10s
- TotalThreadsSysTime: 23s847ms
- TotalThreadsUserTime: 37s983ms
- TotalThreadsVoluntaryContextSwitches: 83.55K (83551)
Fragment Instance Lifecycle Timings:
- ExecTime: 8s478ms
- ExecTreeExecTime: 1.000ms
- OpenTime: 34s008ms
- ExecTreeOpenTime: 33s379ms
- PrepareTime: 999.998us
- ExecTreePrepareTime: 0.000ns
KrpcDataStreamSender (dst_id=2):(Total: 8s433ms, non-child: 0.000ns, % non-child: 0.00%)
ExecOption: Unpartitioned Sender Codegen Disabled: not needed
- BytesSent (1s000ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3.18 MB, 3.18 MB, 3.18 MB, 3.18 MB, 4.19 MB, 7.82 MB, 11.87 MB, 15.92 MB
- NetworkThroughput: (Avg: 22.40 MB/sec ; Min: 47.87 KB/sec ; Max: 121.23 MB/sec ; Number of samples: 43)
- RpcNetworkTime: (Avg: 357.597ms ; Min: 3.356ms ; Max: 8s565ms ; Number of samples: 43)
- RpcRecvrTime: (Avg: -178206067.000ns ; Min: -4282486732.000ns ; Max: -1261033.000ns ; Number of samples: 43)
- EosSent: 0 (0)
- PeakMemoryUsage: 136.00 B (136)
- RowsSent: 250.00K (250000)
- RpcFailure: 0 (0)
- RpcRetry: 0 (0)
- SerializeBatchTime: 870.010ms
- TotalBytesSent: 17.33 MB (18170174)
- UncompressedRowBatchSize: 290.61 MB (304728781)
SORT_NODE (id=1):(Total: 33s380ms, non-child: 0.000ns, % non-child: 0.00%)
SortType: TopN
ExecOption: Codegen Enabled
Node Lifecycle Event Timeline: 42s487ms
- Open Started: 629.999ms (629.999ms)
- Open Finished: 34s009ms (33s379ms)
- First Batch Requested: 34s009ms (0.000ns)
- First Batch Returned: 34s009ms (0.000ns)
- Last Batch Returned: 42s442ms (8s433ms)
- Closed: 42s487ms (45.000ms)
- InsertBatchTime: 9s756ms
- PeakMemoryUsage: 772.60 MB (810130432)
- RowsReturned: 250.00K (250000)
- RowsReturnedRate: 7.49 K/sec
- TuplePoolReclamations: 3 (3)
KUDU_SCAN_NODE (id=0):(Total: 18s935ms, non-child: 0.000ns, % non-child: 0.00%)
Table Name: default.table_abc
Node Lifecycle Event Timeline: 34s009ms
- Open Started: 629.999ms (629.999ms)
- Open Finished: 649.999ms (19.999ms)
- First Batch Requested: 649.999ms (0.000ns)
- First Batch Returned: 840.998ms (190.999ms)
- Last Batch Returned: 33s810ms (32s969ms)
- Closed: 34s009ms (199.000ms)
- AverageScannerThreadConcurrency: 9.84
- KuduClientTime: 4m22s
- KuduRemoteScanTokens: 0 (0)
- MaterializeTupleTime: 4m57s
- NumScannerThreadMemUnavailable: 0 (0)
- NumScannerThreadsStarted: 10 (10)
- PeakMemoryUsage: 30.57 MB (32052224)
- PeakScannerThreadConcurrency: 10 (10)
- RowBatchBytesEnqueued: 50.56 GB (54285777920)
- RowBatchQueueGetWaitTime: 18s802ms
- RowBatchQueuePeakMemoryUsage: 30.56 MB (32044032)
- RowBatchQueuePutWaitTime: 28s692ms
- RowBatchesEnqueued: 32.34K (32345)
- RowsRead: 33.12M (33115154)
- RowsReturned: 33.12M (33115154)
- RowsReturnedRate: 1.75 M/sec
- ScanRangesComplete: 10 (10)
- ScannerThreadsInvoluntaryContextSwitches: 5 (5)
- ScannerThreadsTotalWallClockTime: 5m28s
- ScannerThreadsSysTime: 18s948ms
- ScannerThreadsUserTime: 26s852ms
- ScannerThreadsVoluntaryContextSwitches: 55.05K (55045)
- TotalKuduScanRoundTrips: 43.14K (43142)
CodeGen:(Total: 628.999ms, non-child: 0.000ns, % non-child: 0.00%)
- CodegenInvoluntaryContextSwitches: 0 (0)
- CodegenTotalWallClockTime: 628.999ms
- CodegenSysTime: 10.024ms
- CodegenUserTime: 546.104ms
- CodegenVoluntaryContextSwitches: 92 (92)
- CompileTime: 195.999ms
- IrGenerationTime: 14.999ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 2.48 MB (2595844)
- NumFunctions: 168 (168)
- NumInstructions: 8.93K (8927)
- OptimizationTime: 404.999ms
- PeakMemoryUsage: 4.36 MB (4570624)
- PrepareTime: 11.999ms
Instance 6044b4a39c7f7962:eed7043600000004 (host=node-6:22000):(Total: 42s480ms, non-child: 0.000ns, % non-child: 0.00%)
Last report received time: 2022-05-02 09:45:56.468
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
Fragment Instance Lifecycle Event Timeline: 42s480ms
- Prepare Finished: 2.000ms (2.000ms)
- Open Finished: 36s213ms (36s211ms)
- First Batch Produced: 36s213ms (0.000ns)
- First Batch Sent: 36s220ms (7.000ms)
- ExecInternal Finished: 42s480ms (6s260ms)
- MemoryUsage (1s000ms): 2.21 MB, 365.37 MB, 438.32 MB, 594.37 MB, 403.02 MB, 423.31 MB, 370.23 MB, 458.56 MB, 580.07 MB, 389.77 MB, 417.73 MB, 423.64 MB, 421.23 MB, 422.03 MB, 421.23 MB, 421.23 MB, 422.03 MB, 421.23 MB, 422.84 MB, 422.03 MB, 448.22 MB, 453.73 MB, 461.53 MB, 460.73 MB, 460.73 MB, 461.53 MB, 460.73 MB, 460.73 MB, 460.73 MB, 461.53 MB, 461.53 MB, 462.33 MB, 460.73 MB, 460.73 MB, 459.93 MB, 451.08 MB, 443.03 MB, 443.03 MB, 443.03 MB, 443.03 MB, 443.03 MB, 443.03 MB
- ThreadUsage (1s000ms): 6, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 11, 6, 1, 1, 1, 1, 1, 1
- AverageThreadTokens: 10.04
- ExchangeScanRatio: 0.00
- PeakMemoryUsage: 804.80 MB (843897996)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 804.80 MB (843897996)
- RowsProduced: 250.00K (250000)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 5s350ms
- TotalStorageWaitTime: 4m58s
- TotalThreadsInvoluntaryContextSwitches: 25 (25)
- TotalThreadsTotalWallClockTime: 7m5s
- TotalThreadsSysTime: 26s663ms
- TotalThreadsUserTime: 42s559ms
- TotalThreadsVoluntaryContextSwitches: 91.66K (91656)
Fragment Instance Lifecycle Timings:
- ExecTime: 6s267ms
- ExecTreeExecTime: 0.000ns
- OpenTime: 36s211ms
- ExecTreeOpenTime: 35s576ms
- PrepareTime: 1.000ms
- ExecTreePrepareTime: 0.000ns
KrpcDataStreamSender (dst_id=2):(Total: 6s223ms, non-child: 0.000ns, % non-child: 0.00%)
ExecOption: Unpartitioned Sender Codegen Disabled: not needed
- BytesSent (1s000ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 379.67 KB, 574.66 KB, 3.36 MB, 7.80 MB, 12.25 MB, 16.50 MB
- NetworkThroughput: (Avg: 2.78 MB/sec ; Min: 134.99 KB/sec ; Max: 30.34 MB/sec ; Number of samples: 44)
- RpcNetworkTime: (Avg: 248.116ms ; Min: 12.222ms ; Max: 2s889ms ; Number of samples: 44)
- RpcRecvrTime: (Avg: -122756291.000ns ; Min: -1443300748.000ns ; Max: -1258097.000ns ; Number of samples: 44)
- EosSent: 0 (0)
- PeakMemoryUsage: 136.00 B (136)
- RowsSent: 250.00K (250000)
- RpcFailure: 0 (0)
- RpcRetry: 0 (0)
- SerializeBatchTime: 873.010ms
- TotalBytesSent: 17.71 MB (18573770)
- UncompressedRowBatchSize: 290.63 MB (304745705)
SORT_NODE (id=1):(Total: 35s576ms, non-child: 0.000ns, % non-child: 0.00%)
SortType: TopN
ExecOption: Codegen Enabled
Node Lifecycle Event Timeline: 42s480ms
- Open Started: 637.007ms (637.007ms)
- Open Finished: 36s213ms (35s576ms)
- First Batch Requested: 36s213ms (0.000ns)
- First Batch Returned: 36s213ms (0.000ns)
- Last Batch Returned: 42s436ms (6s223ms)
- Closed: 42s480ms (44.000ms)
- InsertBatchTime: 11s497ms
- PeakMemoryUsage: 772.60 MB (810130432)
- RowsReturned: 250.00K (250000)
- RowsReturnedRate: 7.03 K/sec
- TuplePoolReclamations: 3 (3)
KUDU_SCAN_NODE (id=0):(Total: 18s949ms, non-child: 0.000ns, % non-child: 0.00%)
Table Name: default.table_abc
Node Lifecycle Event Timeline: 36s213ms
- Open Started: 637.007ms (637.007ms)
- Open Finished: 664.008ms (27.000ms)
- First Batch Requested: 664.008ms (0.000ns)
- First Batch Returned: 888.011ms (224.002ms)
- Last Batch Returned: 36s004ms (35s116ms)
- Closed: 36s213ms (209.002ms)
- AverageScannerThreadConcurrency: 10.82
- KuduClientTime: 4m58s
- KuduRemoteScanTokens: 0 (0)
- MaterializeTupleTime: 5m37s
- NumScannerThreadMemUnavailable: 0 (0)
- NumScannerThreadsStarted: 11 (11)
- PeakMemoryUsage: 33.28 MB (34892800)
- PeakScannerThreadConcurrency: 11 (11)
- RowBatchBytesEnqueued: 55.62 GB (59725264896)
- RowBatchQueueGetWaitTime: 18s796ms
- RowBatchQueuePeakMemoryUsage: 32.17 MB (33730560)
- RowBatchQueuePutWaitTime: 42s614ms
- RowBatchesEnqueued: 35.59K (35586)
- RowsRead: 36.43M (36433200)
- RowsReturned: 36.43M (36433200)
- RowsReturnedRate: 1.92 M/sec
- ScanRangesComplete: 11 (11)
- ScannerThreadsInvoluntaryContextSwitches: 13 (13)
- ScannerThreadsTotalWallClockTime: 6m22s
- ScannerThreadsSysTime: 21s219ms
- ScannerThreadsUserTime: 29s716ms
- ScannerThreadsVoluntaryContextSwitches: 60.79K (60793)
- TotalKuduScanRoundTrips: 47.46K (47457)
CodeGen:(Total: 635.007ms, non-child: 0.000ns, % non-child: 0.00%)
- CodegenInvoluntaryContextSwitches: 1 (1)
- CodegenTotalWallClockTime: 635.007ms
- CodegenSysTime: 15.991ms
- CodegenUserTime: 528.504ms
- CodegenVoluntaryContextSwitches: 92 (92)
- CompileTime: 207.002ms
- IrGenerationTime: 16.000ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 2.48 MB (2595844)
- NumFunctions: 168 (168)
- NumInstructions: 8.93K (8927)
- OptimizationTime: 399.004ms
- PeakMemoryUsage: 4.36 MB (4570624)
- PrepareTime: 12.000ms
Instance 6044b4a39c7f7962:eed7043600000002 (host=node-3:22000):(Total: 42s479ms, non-child: 0.000ns, % non-child: 0.00%)
Last report received time: 2022-05-02 09:45:56.461
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
Fragment Instance Lifecycle Event Timeline: 42s480ms
- Prepare Finished: 1.999ms (1.999ms)
- Open Finished: 37s670ms (37s668ms)
- First Batch Produced: 37s670ms (0.000ns)
- First Batch Sent: 37s676ms (5.999ms)
- ExecInternal Finished: 42s480ms (4s803ms)
- MemoryUsage (1s000ms): 64.34 MB, 401.77 MB, 518.52 MB, 405.93 MB, 564.48 MB, 418.06 MB, 470.89 MB, 600.56 MB, 396.48 MB, 413.73 MB, 413.73 MB, 413.73 MB, 414.53 MB, 413.73 MB, 415.33 MB, 414.53 MB, 414.53 MB, 414.53 MB, 413.73 MB, 413.73 MB, 413.73 MB, 421.53 MB, 420.73 MB, 420.73 MB, 420.73 MB, 421.28 MB, 420.48 MB, 421.53 MB, 420.73 MB, 422.33 MB, 421.53 MB, 420.73 MB, 420.73 MB, 422.34 MB, 420.73 MB, 421.53 MB, 416.71 MB, 404.64 MB, 403.03 MB, 403.03 MB, 403.03 MB, 403.03 MB
- ThreadUsage (1s000ms): 6, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 9, 2, 1, 1, 1, 1
- AverageThreadTokens: 10.41
- ExchangeScanRatio: 0.00
- PeakMemoryUsage: 804.80 MB (843897996)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 804.80 MB (843897996)
- RowsProduced: 250.00K (250000)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 3s903ms
- TotalStorageWaitTime: 5m23s
- TotalThreadsInvoluntaryContextSwitches: 34 (34)
- TotalThreadsTotalWallClockTime: 7m22s
- TotalThreadsSysTime: 27s136ms
- TotalThreadsUserTime: 42s405ms
- TotalThreadsVoluntaryContextSwitches: 90.68K (90680)
Fragment Instance Lifecycle Timings:
- ExecTime: 4s808ms
- ExecTreeExecTime: 5.999ms
- OpenTime: 37s668ms
- ExecTreeOpenTime: 37s097ms
- PrepareTime: 1.999ms
- ExecTreePrepareTime: 999.999us
KrpcDataStreamSender (dst_id=2):(Total: 4s769ms, non-child: 0.000ns, % non-child: 0.00%)
ExecOption: Unpartitioned Sender Codegen Disabled: not needed
- BytesSent (1s000ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 585.66 KB, 4.18 MB, 8.62 MB, 12.68 MB, 16.92 MB
- NetworkThroughput: (Avg: 5.50 MB/sec ; Min: 902.23 KB/sec ; Max: 93.65 MB/sec ; Number of samples: 44)
- RpcNetworkTime: (Avg: 183.510ms ; Min: 4.097ms ; Max: 438.466ms ; Number of samples: 44)
- RpcRecvrTime: (Avg: -91124357.000ns ; Min: -218711905.000ns ; Max: -1292804.000ns ; Number of samples: 44)
- EosSent: 0 (0)
- PeakMemoryUsage: 136.00 B (136)
- RowsSent: 250.00K (250000)
- RpcFailure: 0 (0)
- RpcRetry: 0 (0)
- SerializeBatchTime: 865.999ms
- TotalBytesSent: 17.73 MB (18590995)
- UncompressedRowBatchSize: 290.63 MB (304743618)
SORT_NODE (id=1):(Total: 37s103ms, non-child: 0.000ns, % non-child: 0.00%)
SortType: TopN
ExecOption: Codegen Enabled
Node Lifecycle Event Timeline: 42s479ms
- Open Started: 572.999ms (572.999ms)
- Open Finished: 37s670ms (37s097ms)
- First Batch Requested: 37s670ms (0.000ns)
- First Batch Returned: 37s670ms (0.000ns)
- Last Batch Returned: 42s446ms (4s775ms)
- Closed: 42s479ms (32.999ms)
- InsertBatchTime: 11s157ms
- PeakMemoryUsage: 772.60 MB (810130432)
- RowsReturned: 250.00K (250000)
- RowsReturnedRate: 6.74 K/sec
- TuplePoolReclamations: 3 (3)
KUDU_SCAN_NODE (id=0):(Total: 20s704ms, non-child: 0.000ns, % non-child: 0.00%)
Table Name: default.table_abc
Node Lifecycle Event Timeline: 37s670ms
- Open Started: 572.999ms (572.999ms)
- Open Finished: 589.999ms (16.999ms)
- First Batch Requested: 589.999ms (0.000ns)
- First Batch Returned: 788.999ms (198.999ms)
- Last Batch Returned: 37s468ms (36s679ms)
- Closed: 37s670ms (201.999ms)
- AverageScannerThreadConcurrency: 10.81
- KuduClientTime: 5m23s
- KuduRemoteScanTokens: 0 (0)
- MaterializeTupleTime: 6m2s
- NumScannerThreadMemUnavailable: 0 (0)
- NumScannerThreadsStarted: 11 (11)
- PeakMemoryUsage: 33.28 MB (34892800)
- PeakScannerThreadConcurrency: 11 (11)
- RowBatchBytesEnqueued: 55.63 GB (59735334912)
- RowBatchQueueGetWaitTime: 20s562ms
- RowBatchQueuePeakMemoryUsage: 32.17 MB (33730560)
- RowBatchQueuePutWaitTime: 35s451ms
- RowBatchesEnqueued: 35.59K (35592)
- RowsRead: 36.44M (36442638)
- RowsReturned: 36.44M (36442638)
- RowsReturnedRate: 1.76 M/sec
- ScanRangesComplete: 11 (11)
- ScannerThreadsInvoluntaryContextSwitches: 18 (18)
- ScannerThreadsTotalWallClockTime: 6m40s
- ScannerThreadsSysTime: 21s588ms
- ScannerThreadsUserTime: 29s950ms
- ScannerThreadsVoluntaryContextSwitches: 60.14K (60141)
- TotalKuduScanRoundTrips: 47.49K (47488)
CodeGen:(Total: 570.999ms, non-child: 0.000ns, % non-child: 0.00%)
- CodegenInvoluntaryContextSwitches: 1 (1)
- CodegenTotalWallClockTime: 570.999ms
- CodegenSysTime: 23.250ms
- CodegenUserTime: 490.702ms
- CodegenVoluntaryContextSwitches: 92 (92)
- CompileTime: 185.999ms
- IrGenerationTime: 11.999ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 2.48 MB (2595844)
- NumFunctions: 168 (168)
- NumInstructions: 8.93K (8927)
- OptimizationTime: 359.999ms
- PeakMemoryUsage: 4.36 MB (4570624)
- PrepareTime: 11.999ms
Instance 6044b4a39c7f7962:eed7043600000003 (host=node-4:22000):(Total: 42s476ms, non-child: 0.000ns, % non-child: 0.00%)
Last report received time: 2022-05-02 09:45:56.464
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
Fragment Instance Lifecycle Event Timeline: 42s476ms
- Prepare Finished: 999.994us (999.994us)
- Open Finished: 35s745ms (35s744ms)
- First Batch Produced: 35s745ms (0.000ns)
- First Batch Sent: 35s752ms (6.999ms)
- ExecInternal Finished: 42s476ms (6s723ms)
- MemoryUsage (1s000ms): 8.13 KB, 260.07 MB, 429.23 MB, 608.06 MB, 461.59 MB, 591.06 MB, 454.66 MB, 427.68 MB, 392.59 MB, 418.77 MB, 419.73 MB, 420.53 MB, 421.08 MB, 420.03 MB, 420.28 MB, 419.73 MB, 419.73 MB, 420.53 MB, 425.03 MB, 429.34 MB, 445.16 MB, 450.23 MB, 450.23 MB, 450.23 MB, 450.23 MB, 450.23 MB, 451.03 MB, 450.23 MB, 450.23 MB, 450.23 MB, 451.03 MB, 450.24 MB, 451.03 MB, 451.03 MB, 450.23 MB, 436.56 MB, 432.53 MB, 432.53 MB, 432.53 MB, 432.53 MB, 432.53 MB, 432.53 MB
- ThreadUsage (1s000ms): 1, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 11, 3, 1, 1, 1, 1, 1, 1
- AverageThreadTokens: 9.84
- ExchangeScanRatio: 0.00
- PeakMemoryUsage: 804.80 MB (843897996)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 804.80 MB (843897996)
- RowsProduced: 250.00K (250000)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 5s805ms
- TotalStorageWaitTime: 5m4s
- TotalThreadsInvoluntaryContextSwitches: 30 (30)
- TotalThreadsTotalWallClockTime: 7m
- TotalThreadsSysTime: 26s685ms
- TotalThreadsUserTime: 42s054ms
- TotalThreadsVoluntaryContextSwitches: 90.92K (90924)
Fragment Instance Lifecycle Timings:
- ExecTime: 6s730ms
- ExecTreeExecTime: 0.000ns
- OpenTime: 35s744ms
- ExecTreeOpenTime: 35s123ms
- PrepareTime: 999.994us
- ExecTreePrepareTime: 0.000ns
KrpcDataStreamSender (dst_id=2):(Total: 6s687ms, non-child: 0.000ns, % non-child: 0.00%)
ExecOption: Unpartitioned Sender Codegen Disabled: not needed
- BytesSent (1s000ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 388.12 KB, 388.12 KB, 2.61 MB, 7.15 MB, 11.49 MB, 16.04 MB
- NetworkThroughput: (Avg: 3.05 MB/sec ; Min: 104.92 KB/sec ; Max: 29.33 MB/sec ; Number of samples: 45)
- RpcNetworkTime: (Avg: 263.329ms ; Min: 12.921ms ; Max: 3s823ms ; Number of samples: 45)
- RpcRecvrTime: (Avg: -130347565.000ns ; Min: -1910204152.000ns ; Max: -1429922.000ns ; Number of samples: 45)
- EosSent: 0 (0)
- PeakMemoryUsage: 136.00 B (136)
- RowsSent: 250.00K (250000)
- RpcFailure: 0 (0)
- RpcRetry: 0 (0)
- SerializeBatchTime: 880.994ms
- TotalBytesSent: 18.52 MB (19416128)
- UncompressedRowBatchSize: 290.60 MB (304721211)
SORT_NODE (id=1):(Total: 35s123ms, non-child: 0.000ns, % non-child: 0.00%)
SortType: TopN
ExecOption: Codegen Enabled
Node Lifecycle Event Timeline: 42s476ms
- Open Started: 621.995ms (621.995ms)
- Open Finished: 35s745ms (35s123ms)
- First Batch Requested: 35s745ms (0.000ns)
- First Batch Returned: 35s745ms (0.000ns)
- Last Batch Returned: 42s432ms (6s686ms)
- Closed: 42s476ms (43.999ms)
- InsertBatchTime: 10s954ms
- PeakMemoryUsage: 772.60 MB (810130432)
- RowsReturned: 250.00K (250000)
- RowsReturnedRate: 7.12 K/sec
- TuplePoolReclamations: 3 (3)
KUDU_SCAN_NODE (id=0):(Total: 18s931ms, non-child: 0.000ns, % non-child: 0.00%)
Table Name: default.table_abc
Node Lifecycle Event Timeline: 35s745ms
- Open Started: 621.995ms (621.995ms)
- Open Finished: 645.995ms (23.999ms)
- First Batch Requested: 645.995ms (0.000ns)
- First Batch Returned: 791.994ms (145.999ms)
- Last Batch Returned: 35s544ms (34s752ms)
- Closed: 35s745ms (200.998ms)
- AverageScannerThreadConcurrency: 10.73
- KuduClientTime: 5m4s
- KuduRemoteScanTokens: 0 (0)
- MaterializeTupleTime: 5m42s
- NumScannerThreadMemUnavailable: 0 (0)
- NumScannerThreadsStarted: 11 (11)
- PeakMemoryUsage: 33.28 MB (34892800)
- PeakScannerThreadConcurrency: 11 (11)
- RowBatchBytesEnqueued: 55.63 GB (59728621568)
- RowBatchQueueGetWaitTime: 18s785ms
- RowBatchQueuePeakMemoryUsage: 32.17 MB (33730560)
- RowBatchQueuePutWaitTime: 31s936ms
- RowBatchesEnqueued: 35.59K (35588)
- RowsRead: 36.44M (36435994)
- RowsReturned: 36.44M (36435994)
- RowsReturnedRate: 1.92 M/sec
- ScanRangesComplete: 11 (11)
- ScannerThreadsInvoluntaryContextSwitches: 15 (15)
- ScannerThreadsTotalWallClockTime: 6m17s
- ScannerThreadsSysTime: 21s145ms
- ScannerThreadsUserTime: 29s800ms
- ScannerThreadsVoluntaryContextSwitches: 60.77K (60774)
- TotalKuduScanRoundTrips: 47.47K (47465)
CodeGen:(Total: 620.995ms, non-child: 0.000ns, % non-child: 0.00%)
- CodegenInvoluntaryContextSwitches: 2 (2)
- CodegenTotalWallClockTime: 620.995ms
- CodegenSysTime: 14.067ms
- CodegenUserTime: 522.296ms
- CodegenVoluntaryContextSwitches: 92 (92)
- CompileTime: 202.998ms
- IrGenerationTime: 18.999ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 2.48 MB (2595844)
- NumFunctions: 168 (168)
- NumInstructions: 8.93K (8927)
- OptimizationTime: 383.997ms
- PeakMemoryUsage: 4.36 MB (4570624)
- PrepareTime: 14.999ms
Instance 6044b4a39c7f7962:eed7043600000005 (host=node-5:22000):(Total: 42s474ms, non-child: 0.000ns, % non-child: 0.00%)
Last report received time: 2022-05-02 09:45:56.457
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
Fragment Instance Lifecycle Event Timeline: 42s474ms
- Prepare Finished: 1.000ms (1.000ms)
- Open Finished: 34s258ms (34s257ms)
- First Batch Produced: 34s258ms (0.000ns)
- First Batch Sent: 34s264ms (6.000ms)
- ExecInternal Finished: 42s474ms (8s210ms)
- MemoryUsage (1s000ms): 3.01 MB, 379.71 MB, 464.66 MB, 383.02 MB, 466.51 MB, 373.96 MB, 411.66 MB, 464.23 MB, 474.72 MB, 534.46 MB, 325.42 MB, 325.42 MB, 324.62 MB, 324.62 MB, 324.62 MB, 324.62 MB, 324.62 MB, 325.42 MB, 346.16 MB, 340.62 MB, 340.62 MB, 340.62 MB, 341.43 MB, 340.62 MB, 340.62 MB, 340.62 MB, 340.62 MB, 341.42 MB, 341.42 MB, 341.42 MB, 340.62 MB, 340.62 MB, 339.82 MB, 326.95 MB, 324.53 MB, 324.53 MB, 324.53 MB, 324.53 MB, 324.53 MB, 324.53 MB, 324.53 MB, 324.53 MB
- ThreadUsage (1s000ms): 6, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 10, 2, 1, 1, 1, 1, 1, 1, 1, 1
- AverageThreadTokens: 8.67
- ExchangeScanRatio: 0.00
- PeakMemoryUsage: 803.20 MB (842211468)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 803.20 MB (842211468)
- RowsProduced: 250.00K (250000)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 7s322ms
- TotalStorageWaitTime: 4m19s
- TotalThreadsInvoluntaryContextSwitches: 20 (20)
- TotalThreadsTotalWallClockTime: 6m7s
- TotalThreadsSysTime: 24s348ms
- TotalThreadsUserTime: 38s197ms
- TotalThreadsVoluntaryContextSwitches: 83.07K (83068)
Fragment Instance Lifecycle Timings:
- ExecTime: 8s216ms
- ExecTreeExecTime: 1.000ms
- OpenTime: 34s257ms
- ExecTreeOpenTime: 33s651ms
- PrepareTime: 1.000ms
- ExecTreePrepareTime: 0.000ns
KrpcDataStreamSender (dst_id=2):(Total: 8s182ms, non-child: 0.000ns, % non-child: 0.00%)
ExecOption: Unpartitioned Sender Codegen Disabled: not needed
- BytesSent (1s000ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 384.21 KB, 384.21 KB, 384.21 KB, 579.52 KB, 3.38 MB, 7.44 MB, 11.30 MB, 15.36 MB
- NetworkThroughput: (Avg: 3.51 MB/sec ; Min: 57.24 KB/sec ; Max: 55.82 MB/sec ; Number of samples: 41)
- RpcNetworkTime: (Avg: 363.379ms ; Min: 6.721ms ; Max: 6s824ms ; Number of samples: 41)
- RpcRecvrTime: (Avg: -181019758.000ns ; Min: -3411512967.000ns ; Max: -1296478.000ns ; Number of samples: 41)
- EosSent: 0 (0)
- PeakMemoryUsage: 136.00 B (136)
- RowsSent: 250.00K (250000)
- RpcFailure: 0 (0)
- RpcRetry: 0 (0)
- SerializeBatchTime: 859.005ms
- TotalBytesSent: 16.58 MB (17383444)
- UncompressedRowBatchSize: 290.62 MB (304741907)
SORT_NODE (id=1):(Total: 33s652ms, non-child: 0.000ns, % non-child: 0.00%)
SortType: TopN
ExecOption: Codegen Enabled
Node Lifecycle Event Timeline: 42s474ms
- Open Started: 607.003ms (607.003ms)
- Open Finished: 34s258ms (33s651ms)
- First Batch Requested: 34s258ms (0.000ns)
- First Batch Returned: 34s258ms (0.000ns)
- Last Batch Returned: 42s441ms (8s183ms)
- Closed: 42s474ms (33.000ms)
- InsertBatchTime: 9s819ms
- PeakMemoryUsage: 772.60 MB (810130432)
- RowsReturned: 250.00K (250000)
- RowsReturnedRate: 7.43 K/sec
- TuplePoolReclamations: 3 (3)
KUDU_SCAN_NODE (id=0):(Total: 19s128ms, non-child: 0.000ns, % non-child: 0.00%)
Table Name: default.table_abc
Node Lifecycle Event Timeline: 34s258ms
- Open Started: 607.003ms (607.003ms)
- Open Finished: 628.003ms (21.000ms)
- First Batch Requested: 628.003ms (0.000ns)
- First Batch Returned: 849.004ms (221.001ms)
- Last Batch Returned: 34s065ms (33s216ms)
- Closed: 34s258ms (193.001ms)
- AverageScannerThreadConcurrency: 9.73
- KuduClientTime: 4m19s
- KuduRemoteScanTokens: 0 (0)
- MaterializeTupleTime: 4m54s
- NumScannerThreadMemUnavailable: 0 (0)
- NumScannerThreadsStarted: 10 (10)
- PeakMemoryUsage: 31.67 MB (33206272)
- PeakScannerThreadConcurrency: 10 (10)
- RowBatchBytesEnqueued: 50.58 GB (54312631296)
- RowBatchQueueGetWaitTime: 18s999ms
- RowBatchQueuePeakMemoryUsage: 30.56 MB (32044032)
- RowBatchQueuePutWaitTime: 27s649ms
- RowBatchesEnqueued: 32.36K (32361)
- RowsRead: 33.13M (33131362)
- RowsReturned: 33.13M (33131362)
- RowsReturnedRate: 1.73 M/sec
- ScanRangesComplete: 10 (10)
- ScannerThreadsInvoluntaryContextSwitches: 9 (9)
- ScannerThreadsTotalWallClockTime: 5m24s
- ScannerThreadsSysTime: 19s336ms
- ScannerThreadsUserTime: 27s112ms
- ScannerThreadsVoluntaryContextSwitches: 54.64K (54636)
- TotalKuduScanRoundTrips: 43.15K (43154)
CodeGen:(Total: 606.003ms, non-child: 0.000ns, % non-child: 0.00%)
- CodegenInvoluntaryContextSwitches: 0 (0)
- CodegenTotalWallClockTime: 606.003ms
- CodegenSysTime: 16.129ms
- CodegenUserTime: 516.952ms
- CodegenVoluntaryContextSwitches: 92 (92)
- CompileTime: 199.001ms
- IrGenerationTime: 15.000ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 2.48 MB (2595844)
- NumFunctions: 168 (168)
- NumInstructions: 8.93K (8927)
- OptimizationTime: 380.002ms
- PeakMemoryUsage: 4.36 MB (4570624)
- PrepareTime: 12.000ms
Instance 6044b4a39c7f7962:eed7043600000001 (host=node-2:22000):(Total: 42s472ms, non-child: 0.000ns, % non-child: 0.00%)
Last report received time: 2022-05-02 09:45:56.460
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
Fragment Instance Lifecycle Event Timeline: 42s472ms
- Prepare Finished: 2.000ms (2.000ms)
- Open Finished: 34s592ms (34s590ms)
- First Batch Produced: 34s592ms (0.000ns)
- First Batch Sent: 34s598ms (6.000ms)
- ExecInternal Finished: 42s472ms (7s874ms)
- MemoryUsage (1s000ms): 2.21 MB, 372.32 MB, 468.82 MB, 394.56 MB, 459.07 MB, 598.06 MB, 394.52 MB, 444.49 MB, 474.27 MB, 472.73 MB, 474.09 MB, 472.73 MB, 473.28 MB, 473.53 MB, 472.73 MB, 472.73 MB, 472.73 MB, 472.73 MB, 473.53 MB, 475.78 MB, 478.03 MB, 477.23 MB, 477.23 MB, 477.23 MB, 478.03 MB, 478.83 MB, 477.23 MB, 477.23 MB, 477.23 MB, 477.23 MB, 478.03 MB, 478.59 MB, 477.23 MB, 471.60 MB, 461.14 MB, 459.53 MB, 459.53 MB, 459.53 MB, 459.53 MB, 459.53 MB, 459.53 MB, 459.53 MB
- ThreadUsage (1s000ms): 6, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 8, 2, 1, 1, 1, 1, 1, 1, 1
- AverageThreadTokens: 9.61
- ExchangeScanRatio: 0.00
- PeakMemoryUsage: 804.30 MB (843373708)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 804.30 MB (843373708)
- RowsProduced: 250.00K (250000)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 6s962ms
- TotalStorageWaitTime: 4m57s
- TotalThreadsInvoluntaryContextSwitches: 19 (19)
- TotalThreadsTotalWallClockTime: 6m46s
- TotalThreadsSysTime: 26s544ms
- TotalThreadsUserTime: 41s349ms
- TotalThreadsVoluntaryContextSwitches: 90.95K (90954)
Fragment Instance Lifecycle Timings:
- ExecTime: 7s880ms
- ExecTreeExecTime: 3.000ms
- OpenTime: 34s590ms
- ExecTreeOpenTime: 33s983ms
- PrepareTime: 2.000ms
- ExecTreePrepareTime: 0.000ns
KrpcDataStreamSender (dst_id=2):(Total: 7s846ms, non-child: 0.000ns, % non-child: 0.00%)
ExecOption: Unpartitioned Sender Codegen Disabled: not needed
- BytesSent (1s000ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 390.60 KB, 781.21 KB, 781.21 KB, 781.21 KB, 3.42 MB, 7.91 MB, 12.00 MB, 16.69 MB
- NetworkThroughput: (Avg: 4.51 MB/sec ; Min: 61.24 KB/sec ; Max: 79.02 MB/sec ; Number of samples: 44)
- RpcNetworkTime: (Avg: 321.779ms ; Min: 4.883ms ; Max: 6s562ms ; Number of samples: 44)
- RpcRecvrTime: (Avg: -159590226.000ns ; Min: -3280054040.000ns ; Max: -1265135.000ns ; Number of samples: 44)
- EosSent: 0 (0)
- PeakMemoryUsage: 136.00 B (136)
- RowsSent: 250.00K (250000)
- RpcFailure: 0 (0)
- RpcRetry: 0 (0)
- SerializeBatchTime: 883.003ms
- TotalBytesSent: 17.92 MB (18794723)
- UncompressedRowBatchSize: 290.62 MB (304735863)
SORT_NODE (id=1):(Total: 33s985ms, non-child: 0.000ns, % non-child: 0.00%)
SortType: TopN
ExecOption: Codegen Enabled
Node Lifecycle Event Timeline: 42s472ms
- Open Started: 609.002ms (609.002ms)
- Open Finished: 34s592ms (33s983ms)
- First Batch Requested: 34s592ms (0.000ns)
- First Batch Returned: 34s592ms (0.000ns)
- Last Batch Returned: 42s441ms (7s849ms)
- Closed: 42s472ms (31.000ms)
- InsertBatchTime: 9s765ms
- PeakMemoryUsage: 772.10 MB (809606144)
- RowsReturned: 250.00K (250000)
- RowsReturnedRate: 7.36 K/sec
- TuplePoolReclamations: 2 (2)
KUDU_SCAN_NODE (id=0):(Total: 19s229ms, non-child: 0.000ns, % non-child: 0.00%)
Table Name: default.table_abc
Node Lifecycle Event Timeline: 34s592ms
- Open Started: 609.002ms (609.002ms)
- Open Finished: 636.002ms (27.000ms)
- First Batch Requested: 636.002ms (0.000ns)
- First Batch Returned: 769.002ms (133.000ms)
- Last Batch Returned: 34s393ms (33s624ms)
- Closed: 34s592ms (199.000ms)
- AverageScannerThreadConcurrency: 10.76
- KuduClientTime: 4m57s
- KuduRemoteScanTokens: 0 (0)
- MaterializeTupleTime: 5m36s
- NumScannerThreadMemUnavailable: 0 (0)
- NumScannerThreadsStarted: 11 (11)
- PeakMemoryUsage: 33.28 MB (34892800)
- PeakScannerThreadConcurrency: 11 (11)
- RowBatchBytesEnqueued: 55.64 GB (59742048256)
- RowBatchQueueGetWaitTime: 19s081ms
- RowBatchQueuePeakMemoryUsage: 32.17 MB (33730560)
- RowBatchQueuePutWaitTime: 24s568ms
- RowBatchesEnqueued: 35.60K (35596)
- RowsRead: 36.44M (36442952)
- RowsReturned: 36.44M (36442952)
- RowsReturnedRate: 1.90 M/sec
- ScanRangesComplete: 11 (11)
- ScannerThreadsInvoluntaryContextSwitches: 8 (8)
- ScannerThreadsTotalWallClockTime: 6m3s
- ScannerThreadsSysTime: 21s343ms
- ScannerThreadsUserTime: 30s051ms
- ScannerThreadsVoluntaryContextSwitches: 59.88K (59885)
- TotalKuduScanRoundTrips: 47.47K (47471)
CodeGen:(Total: 607.002ms, non-child: 0.000ns, % non-child: 0.00%)
- CodegenInvoluntaryContextSwitches: 1 (1)
- CodegenTotalWallClockTime: 606.002ms
- CodegenSysTime: 15.132ms
- CodegenUserTime: 501.887ms
- CodegenVoluntaryContextSwitches: 92 (92)
- CompileTime: 201.000ms
- IrGenerationTime: 17.000ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 2.48 MB (2595844)
- NumFunctions: 168 (168)
- NumInstructions: 8.93K (8927)
- OptimizationTime: 377.001ms
- PeakMemoryUsage: 4.36 MB (4570624)
- PrepareTime: 12.000ms
Re: Impala Performance Issue with huge Kudu Data load
Posted by Sumudu Madushanka <ng...@cse.mrt.ac.lk>.
Impala version - impala-3.4.0
Kudu version - kudu-1.15.0
On Fri, 6 May 2022, 06:01 Quanlong Huang, <hu...@gmail.com> wrote:
> Hi Sumudu,
>
> Did you miss these questions?
>
> The TopN operator in the query also takes a significant amount of time.
>> How do you test the performance in Kudu level? It'd be helpful if you could
>> share the test codes.
>
>
> Another question is about the versions. The profile doesn't show the exact
>> Impala version. What's the kudu client version used in your Impala? What's
>> the version of your kudu cluster?
>
>
>
> On Tue, May 3, 2022 at 9:31 PM Sumudu Madushanka <
> ngsmadushanka.16@cse.mrt.ac.lk> wrote:
>
>> Hi @Quanlong Huang <hu...@gmail.com>
>>
>>
>>
>> CREATE TABLE `table_abc` (
>>
>> `col_2` STRING NOT NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_1` BIGINT NOT NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_3` BIGINT NOT NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_4` BIGINT NOT NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_5` INT NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_6` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_7` DOUBLE NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_8` INT NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_9` INT NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_10` DOUBLE NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_11` INT NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_12` BIGINT NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_13` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_14` INT NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_15` INT NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_16` INT NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_17` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_18` INT NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_19` DOUBLE NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_20` INT NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_21` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_22` DOUBLE NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_23` INT NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_24` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_25` BIGINT NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_26` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_27` INT NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_28` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_29` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_30` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_31` INT NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_32` INT NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_33` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_34` DOUBLE NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_35` INT NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_36` DOUBLE NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_37` INT NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_38` INT NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_39` INT NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_40` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_41` BIGINT NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_42` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_43` INT NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_44` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_45` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_46` INT NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_47` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_48` DOUBLE NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_49` INT NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_50` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_51` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_52` INT NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_53` INT NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_54` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_55` INT NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_56` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_57` INT NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_58` INT NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_59` INT NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_60` BIGINT NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_61` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_62` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_63` INT NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_64` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_65` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_66` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_67` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_68` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_69` INT NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_70` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_71` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_72` INT NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_73` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_74` DOUBLE NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_75` INT NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_76` INT NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_77` INT NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_78` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_79` DOUBLE NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_80` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_81` INT NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_82` INT NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_83` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_84` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_85` INT NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_86` DOUBLE NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_87` INT NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_88` DOUBLE NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_89` BIGINT NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_90` INT NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_91` DOUBLE NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_92` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_93` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_94` DOUBLE NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_95` DOUBLE NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_96` DOUBLE NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_97` INT NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_98` DOUBLE NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_99` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_100` INT NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_101` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_102` DOUBLE NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_103` DOUBLE NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_104` INT NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_105` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_106` DOUBLE NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_107` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_108` BIGINT NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_109` DOUBLE NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_110` INT NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_111` INT NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_112` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_113` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_114` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_115` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_116` INT NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> `col_117` INT NULL ENCODING AUTO_ENCODING COMPRESSION
>> DEFAULT_COMPRESSION,
>>
>> PRIMARY KEY (col_2, col_1, col_3, col_4)
>>
>> )
>>
>> PARTITION BY HASH (col_2) PARTITIONS 4, HASH (col_3) PARTITIONS 64, RANGE
>> (col_1) (PARTITION 1622505600000000000 <= VALUES < 1625097599999999999)
>>
>> STORED AS KUDU
>>
>> TBLPROPERTIES
>> ('kudu.master_addresses'='node-1:8051,node-2:8051,node-3:8051,node-4:8051,node-5:8051',
>> 'kudu.num_tablet_replicas'='3', 'numFiles'='0', 'totalSize'='0')
>>
>>
>>
>> Thank you!
>>
>> *From: *Quanlong Huang <hu...@gmail.com>
>> *Sent: *Tuesday, May 3, 2022 2:41 PM
>> *To: *user@impala.apache.org
>> *Subject: *Re: Impala Performance Issue with huge Kudu Data load
>>
>>
>>
>> Hi Sumudu,
>>
>>
>>
>> Could you share the DDL of the table, i.e. "show create table
>>
>> default.table_abc"? Just want to check the primary keys.
>>
>>
>>
>> > We have achieved the performance as we expected in Kudu level, however,
>> in Impala we haven’t reached to the performance we expected.
>>
>>
>>
>> The TopN operator in the query also takes a significant amount of
>>
>> time. How do you test the performance in Kudu level? It'd be helpful
>>
>> if you could share the test codes.
>>
>>
>>
>> Another question is about the versions. The profile doesn't show the
>>
>> exact Impala version. What's the kudu client version used in your
>>
>> Impala? What's the version of your kudu cluster?
>>
>>
>>
>> Regards,
>>
>> Quanlong
>>
>>
>>
>> On Mon, May 2, 2022 at 5:31 PM Sumudu Madushanka
>>
>> <ng...@cse.mrt.ac.lk> wrote:
>>
>> >
>>
>> > Hi,
>>
>> > Here is the profile of the query
>>
>> >
>>
>> >
>>
>> >
>>
>> > Thank you!
>>
>> >
>>
>> >
>>
>> >
>>
>> > Best Regards
>>
>> >
>>
>> > Sumudu Madushanka
>>
>> >
>>
>> >
>>
>> >
>>
>> > From: Tim Armstrong
>>
>> > Sent: Friday, April 29, 2022 8:55 PM
>>
>> > To: user@impala.apache.org
>>
>> > Subject: Re: Impala Performance Issue with huge Kudu Data load
>>
>> >
>>
>> >
>>
>> >
>>
>> > Hi Sumudu,
>>
>> >
>>
>> > We might be able to help more if you attach a query profile. You can
>> get it from impala-shell with the "profile" command or from the Impala
>> debug web UI "queries" tab.
>>
>> >
>>
>> >
>>
>> >
>>
>> > - Tim
>>
>> >
>>
>> >
>>
>> >
>>
>> > On Fri, 29 Apr 2022 at 07:07, Sumudu Madushanka <
>> ngsmadushanka.16@cse.mrt.ac.lk> wrote:
>>
>> >
>>
>> > Hi Team,
>>
>> >
>>
>> > This is regarding a performance issue me and my team have on a huge
>> data load in Kudu. We are hoping you can guide us on a solution to the
>> below mentioned concerns.
>>
>> >
>>
>> > We have 212 million data loads in Kudu. Currently for such a data load,
>> when loading through impala, 47 seconds are spent for query processing and
>> loading overall. We have used default configurations in Kudu and Impala
>> with 6 node clusters to get these numbers. We have achieved the performance
>> as we expected in Kudu level, however, in Impala we haven’t reached to the
>> performance we expected.
>>
>> >
>>
>> >
>>
>> > What can we do to reduce the time spent for loading 212 million data
>> loads from 47 seconds to 10 seconds through impala?
>>
>> >
>>
>> > We would be much obliged if you can provide us with some solutions.
>>
>> >
>>
>> > Thank You!
>>
>> >
>>
>> >
>>
>> > Best Regards
>>
>> >
>>
>> > Sumudu Madushanka
>>
>> >
>>
>> >
>>
>>
>>
>
RE: Impala Performance Issue with huge Kudu Data load
Posted by Sumudu Madushanka <ng...@cse.mrt.ac.lk>.
Hi [@Quanlong Huang](mailto:huangquanlong@gmail.com)
CREATE TABLE `table_abc` (
`col_2` STRING NOT NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_1` BIGINT NOT NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_3` BIGINT NOT NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_4` BIGINT NOT NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_5` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_6` STRING NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_7` DOUBLE NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_8` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_9` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_10` DOUBLE NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_11` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_12` BIGINT NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_13` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_14` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_15` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_16` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_17` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_18` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_19` DOUBLE NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_20` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_21` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_22` DOUBLE NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_23` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_24` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_25` BIGINT NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_26` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_27` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_28` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_29` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_30` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_31` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_32` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_33` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_34` DOUBLE NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_35` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_36` DOUBLE NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_37` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_38` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_39` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_40` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_41` BIGINT NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_42` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_43` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_44` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_45` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_46` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_47` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_48` DOUBLE NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_49` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_50` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_51` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_52` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_53` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_54` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_55` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_56` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_57` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_58` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_59` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_60` BIGINT NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_61` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_62` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_63` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_64` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_65` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_66` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_67` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_68` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_69` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_70` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_71` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_72` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_73` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_74` DOUBLE NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_75` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_76` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_77` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_78` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_79` DOUBLE NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_80` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_81` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_82` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_83` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_84` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_85` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_86` DOUBLE NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_87` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_88` DOUBLE NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_89` BIGINT NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_90` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_91` DOUBLE NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_92` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_93` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_94` DOUBLE NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_95` DOUBLE NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_96` DOUBLE NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_97` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_98` DOUBLE NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_99` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_100` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_101` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_102` DOUBLE NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_103` DOUBLE NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_104` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_105` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_106` DOUBLE NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_107` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_108` BIGINT NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_109` DOUBLE NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_110` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_111` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_112` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_113` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_114` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_115` STRING NULL ENCODING AUTO_ENCODING COMPRESSION
DEFAULT_COMPRESSION,
`col_116` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
`col_117` INT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION,
PRIMARY KEY (col_2, col_1, col_3, col_4)
)
PARTITION BY HASH (col_2) PARTITIONS 4, HASH (col_3) PARTITIONS 64, RANGE
(col_1) (PARTITION 1622505600000000000 <= VALUES < 1625097599999999999)
STORED AS KUDU
TBLPROPERTIES
('kudu.master_addresses'='node-1:8051,node-2:8051,node-3:8051,node-4:8051,node-5:8051',
'kudu.num_tablet_replicas'='3', 'numFiles'='0', 'totalSize'='0')
Thank you!
**From:**[Quanlong Huang](mailto:huangquanlong@gmail.com)
**Sent:** Tuesday, May 3, 2022 2:41 PM
**To:**[user@impala.apache.org](mailto:user@impala.apache.org)
**Subject:** Re: Impala Performance Issue with huge Kudu Data load
Hi Sumudu,
Could you share the DDL of the table, i.e. "show create table
default.table_abc"? Just want to check the primary keys.
> We have achieved the performance as we expected in Kudu level, however, in
> Impala we haven’t reached to the performance we expected.
The TopN operator in the query also takes a significant amount of
time. How do you test the performance in Kudu level? It'd be helpful
if you could share the test codes.
Another question is about the versions. The profile doesn't show the
exact Impala version. What's the kudu client version used in your
Impala? What's the version of your kudu cluster?
Regards,
Quanlong
On Mon, May 2, 2022 at 5:31 PM Sumudu Madushanka
<ng...@cse.mrt.ac.lk> wrote:
>
> Hi,
> Here is the profile of the query
>
>
>
> Thank you!
>
>
>
> Best Regards
>
> Sumudu Madushanka
>
>
>
> From: Tim Armstrong
> Sent: Friday, April 29, 2022 8:55 PM
> To: user@impala.apache.org
> Subject: Re: Impala Performance Issue with huge Kudu Data load
>
>
>
> Hi Sumudu,
>
> We might be able to help more if you attach a query profile. You can get it
> from impala-shell with the "profile" command or from the Impala debug web UI
> "queries" tab.
>
>
>
> \- Tim
>
>
>
> On Fri, 29 Apr 2022 at 07:07, Sumudu Madushanka
> <ng...@cse.mrt.ac.lk> wrote:
>
> Hi Team,
>
> This is regarding a performance issue me and my team have on a huge data
> load in Kudu. We are hoping you can guide us on a solution to the below
> mentioned concerns.
>
> We have 212 million data loads in Kudu. Currently for such a data load, when
> loading through impala, 47 seconds are spent for query processing and
> loading overall. We have used default configurations in Kudu and Impala with
> 6 node clusters to get these numbers. We have achieved the performance as we
> expected in Kudu level, however, in Impala we haven’t reached to the
> performance we expected.
>
>
> What can we do to reduce the time spent for loading 212 million data loads
> from 47 seconds to 10 seconds through impala?
>
> We would be much obliged if you can provide us with some solutions.
>
> Thank You!
>
>
> Best Regards
>
> Sumudu Madushanka
>
>
Re: Impala Performance Issue with huge Kudu Data load
Posted by Quanlong Huang <hu...@gmail.com>.
Hi Sumudu,
Could you share the DDL of the table, i.e. "show create table
default.table_abc"? Just want to check the primary keys.
> We have achieved the performance as we expected in Kudu level, however, in Impala we haven’t reached to the performance we expected.
The TopN operator in the query also takes a significant amount of
time. How do you test the performance in Kudu level? It'd be helpful
if you could share the test codes.
Another question is about the versions. The profile doesn't show the
exact Impala version. What's the kudu client version used in your
Impala? What's the version of your kudu cluster?
Regards,
Quanlong
On Mon, May 2, 2022 at 5:31 PM Sumudu Madushanka
<ng...@cse.mrt.ac.lk> wrote:
>
> Hi,
> Here is the profile of the query
>
>
>
> Thank you!
>
>
>
> Best Regards
>
> Sumudu Madushanka
>
>
>
> From: Tim Armstrong
> Sent: Friday, April 29, 2022 8:55 PM
> To: user@impala.apache.org
> Subject: Re: Impala Performance Issue with huge Kudu Data load
>
>
>
> Hi Sumudu,
>
> We might be able to help more if you attach a query profile. You can get it from impala-shell with the "profile" command or from the Impala debug web UI "queries" tab.
>
>
>
> - Tim
>
>
>
> On Fri, 29 Apr 2022 at 07:07, Sumudu Madushanka <ng...@cse.mrt.ac.lk> wrote:
>
> Hi Team,
>
> This is regarding a performance issue me and my team have on a huge data load in Kudu. We are hoping you can guide us on a solution to the below mentioned concerns.
>
> We have 212 million data loads in Kudu. Currently for such a data load, when loading through impala, 47 seconds are spent for query processing and loading overall. We have used default configurations in Kudu and Impala with 6 node clusters to get these numbers. We have achieved the performance as we expected in Kudu level, however, in Impala we haven’t reached to the performance we expected.
>
>
> What can we do to reduce the time spent for loading 212 million data loads from 47 seconds to 10 seconds through impala?
>
> We would be much obliged if you can provide us with some solutions.
>
> Thank You!
>
>
> Best Regards
>
> Sumudu Madushanka
>
>