You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@impala.apache.org by Geetika Gupta <ge...@knoldus.in> on 2018/05/07 06:04:24 UTC
Issue in data loading in Impala + Kudu
Hi community,
I was trying to load 500GB of TPCH data into kudu table using the following
query:
*insert into lineitem select * from PARQUETIMPALA500.LINEITEM*
While executing the query for around 17 hrs it got cancelled as the impalad
process of that machine got aborted. Here are the logs of the impalad
process.
*impalad.ERROR*
Log file created at: 2018/05/06 13:40:34
Running on machine: slave2
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E0506 13:40:34.097759 28730 logging.cc:121] stderr will be logged to this
file.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/root/softwares/
impala/fe/target/dependency/slf4j-log4j12-1.7.25.jar!/org/
slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/root/softwares/impala/testdata/target/
dependency/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/
StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
18/05/06 13:40:34 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
18/05/06 13:40:36 WARN shortcircuit.DomainSocketFactory: The short-circuit
local reads feature cannot be used because libhadoop cannot be loaded.
tcmalloc: large alloc 1073741824 bytes == 0x484434000 @ 0x4135176
0x7fd9e9fc3929
tcmalloc: large alloc 2147483648 bytes == 0x7fd540f18000 @ 0x4135176
0x7fd9e9fc3929
F0507 09:46:12.673912 29258 error-util.cc:148] Check failed:
log_entry.count > 0 (-1831809966 vs. 0)
*** Check failure stack trace: ***
@ 0x3fc0c0d google::LogMessage::Fail()
@ 0x3fc24b2 google::LogMessage::SendToLog()
@ 0x3fc05e7 google::LogMessage::Flush()
@ 0x3fc3bae google::LogMessageFatal::~LogMessageFatal()
@ 0x1bbcb31 impala::PrintErrorMap()
@ 0x1bbcd07 impala::PrintErrorMapToString()
@ 0x2decbd7 impala::Coordinator::GetErrorLog()
@ 0x1a8d634 impala::ImpalaServer::UnregisterQuery()
@ 0x1b29264 impala::ImpalaServer::CloseOperation()
@ 0x2c5ce86 apache::hive::service::cli::
thrift::TCLIServiceProcessor::process_CloseOperation()
@ 0x2c56b8c apache::hive::service::cli::
thrift::TCLIServiceProcessor::dispatchCall()
@ 0x2c2fcb1 impala::ImpalaHiveServer2ServiceProces
sor::dispatchCall()
@ 0x16fdb20 apache::thrift::TDispatchProcessor::process()
@ 0x18ea6b3 apache::thrift::server::
TAcceptQueueServer::Task::run()
@ 0x18e2181 impala::ThriftThread::RunRunnable()
@ 0x18e3885 boost::_mfi::mf2<>::operator()()
@ 0x18e371b boost::_bi::list3<>::operator()<>()
@ 0x18e3467 boost::_bi::bind_t<>::operator()()
@ 0x18e337a boost::detail::function::void_
function_obj_invoker0<>::invoke()
@ 0x192761c boost::function0<>::operator()()
@ 0x1c3ebf7 impala::Thread::SuperviseThread()
@ 0x1c470cd boost::_bi::list5<>::operator()<>()
@ 0x1c46ff1 boost::_bi::bind_t<>::operator()()
@ 0x1c46fb4 boost::detail::thread_data<>::run()
@ 0x2eedb4a thread_proxy
@ 0x7fda1dbb16ba start_thread
@ 0x7fda1d8e741d clone
Wrote minidump to /tmp/minidumps/impalad/a9113d9b-bc3d-488a-1feebf9b-
47b42022.dmp
*impalad.FATAL*
Log file created at: 2018/05/07 09:46:12
Running on machine: slave2
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
F0507 09:46:12.673912 29258 error-util.cc:148] Check failed:
log_entry.count > 0 (-1831809966 vs. 0)
*Impalad.INFO*
edentials={real_user=root}} blocked reactor thread for 34288.6us
I0507 09:38:14.943245 29882 outbound_call.cc:288] RPC callback for RPC call
kudu.tserver.TabletServerService.Write -> {remote=136.243.74.42:7050 (slave5),
user_credentials={real_user=root}} blocked reactor thread for 35859.8us
I0507 09:38:15.942150 29882 outbound_call.cc:288] RPC callback for RPC call
kudu.tserver.TabletServerService.Write -> {remote=136.243.74.42:7050 (slave5),
user_credentials={real_user=root}} blocked reactor thread for 40664.9us
I0507 09:38:17.495046 29882 outbound_call.cc:288] RPC callback for RPC call
kudu.tserver.TabletServerService.Write -> {remote=136.243.74.42:7050 (slave5),
user_credentials={real_user=root}} blocked reactor thread for 49514.6us
I0507 09:46:12.664149 4507 coordinator.cc:783] Release admission control
resources for query_id=3e4a4c646800e1d9:c859bb7f00000000
F0507 09:46:12.673912 29258 error-util.cc:148] Check failed:
log_entry.count > 0 (-1831809966 vs. 0)
Wrote minidump to /tmp/minidumps/impalad/a9113d9b-bc3d-488a-1feebf9b-
47b42022.dmp
*Note*:
We are executing the queries on 8 node cluster with the following
configuration
Cluster : 8 Node Cluster (48 GB RAM , 8 CPU Core and 2 TB hard-disk each,
Intel(R) Core(TM) i7 CPU 950 @ 3.07GHz
--
Regards,
Geetika Gupta
Re: Issue in data loading in Impala + Kudu
Posted by Jeszy <je...@gmail.com>.
As suggested over on the Kudu list, this is likely due to key
duplication (which is fine on HDFS, but won't work for Kudu). The
profile has the following error that confirms this:
Errors: Key already present in Kudu table
'impala::kudu_impala_500.LINEITEM'. (1 of -1831809966 similar)
Raised IMPALA-7007 to address the overflow(?) of the counter.
On 10 May 2018 at 08:00, Mostafa Mokhtar <mm...@cloudera.com> wrote:
> Can you try rerunning the query again against the Kudu database instead of
> default?
>
> select count(*) from kudu_impala_500.LINEITEM;
>
>
> On Wed, May 9, 2018 at 10:13 PM, Geetika Gupta <ge...@knoldus.in>
> wrote:
>>
>> Please find below the query profile :
>>
>> Query (id=9649f7ab3bcc5fb8:f4d6a60700000000):
>> Summary:
>> Session ID: b04a9080d1e1724d:41f2b0d261e8f280
>> Session Type: HIVESERVER2
>> HiveServer2 Protocol Version: V6
>> Start Time: 2018-05-08 17:55:23.181264000
>> End Time: 2018-05-10 00:34:17.784273000
>> Query Type: DML
>> Query State: FINISHED
>> Query Status: OK
>> Impala Version: impalad version 3.0.0-SNAPSHOT RELEASE (build
>> b68e06997c1f49f6b723d78e217efddec4f56f3a)
>> User: root
>> Connected User: root
>> Delegated User:
>> Network Address: ::ffff:46.4.88.233:59862
>> Default Db: kudu_impala_500
>> Sql Statement: insert into LINEITEM select * from
>> PARQUETIMPALA500.LINEITEM
>> Coordinator: slave2:22000
>> Query Options (set by configuration):
>> Query Options (set by configuration and planner): MT_DOP=0
>> Plan:
>> ----------------
>> Max Per-Host Resource Reservation: Memory=0B
>> Per-Host Resource Estimates: Memory=704.00MB
>> WARNING: The following tables are missing relevant table and/or column
>> statistics.
>> parquetimpala500.lineitem
>>
>> F00:PLAN FRAGMENT [RANDOM] hosts=7 instances=7
>> | Per-Host Resources: mem-estimate=704.00MB mem-reservation=0B
>> INSERT INTO KUDU [kudu_impala_500.lineitem]
>> | mem-estimate=0B mem-reservation=0B
>> |
>> 00:SCAN HDFS [parquetimpala500.lineitem, RANDOM]
>> partitions=1/1 files=396 size=97.29GB
>> stored statistics:
>> table: rows=unavailable size=unavailable
>> columns: unavailable
>> extrapolated-rows=disabled
>> mem-estimate=704.00MB mem-reservation=0B
>> tuple-ids=0 row-size=171B cardinality=unavailable
>> ----------------
>> Estimated Per-Host Mem: 738197504
>> Tables Missing Stats: parquetimpala500.lineitem
>> Per Host Min Reservation: slave1:22000(0) slave2:22000(0)
>> slave3:22000(0) slave4:22000(0) slave5:22000(0) slave6:22000(0)
>> slave7:22000(0)
>> Request Pool: default-pool
>> Admission result: Admitted immediately
>> ExecSummary:
>> Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem
>> Est. Peak Mem Detail
>>
>> -----------------------------------------------------------------------------------------------------------------
>> 00:SCAN HDFS 7 4s417ms 6s154ms 3.00B -1 1.39 GB
>> 704.00 MB parquetimpala500.lineitem
>> Errors: Key already present in Kudu table
>> 'impala::kudu_impala_500.LINEITEM'. (1 of -1831809966 similar)
>>
>> Query Compilation: 6s413ms
>> - Metadata load started: 14.443ms (14.443ms)
>> - Metadata load finished. loaded-tables=2/2 load-requests=1
>> catalog-updates=7: 6s298ms (6s283ms)
>> - Analysis finished: 6s301ms (3.659ms)
>> - Value transfer graph computed: 6s302ms (282.554us)
>> - Single node plan created: 6s363ms (61.624ms)
>> - Runtime filters computed: 6s363ms (98.878us)
>> - Distributed plan created: 6s366ms (2.832ms)
>> - Planning finished: 6s413ms (46.751ms)
>> Query Timeline: 30h38m
>> - Query submitted: 56.829us (56.829us)
>> - Planning finished: 6s431ms (6s431ms)
>> - Submit for admission: 6s432ms (821.867us)
>> - Completed admission: 6s432ms (14.519us)
>> - Ready to start on 7 backends: 6s432ms (99.515us)
>> - All 7 execution backends (7 fragment instances) started: 6s535ms
>> (103.320ms)
>> - Released admission control resources: 30h38m (30h38m)
>> - DML data written: 30h38m (934.739us)
>> - DML Metastore update finished: 30h38m (157.938us)
>> - Request finished: 30h38m (41.379us)
>> - First row fetched: 30h38m (173.124us)
>> - First row fetched: 30h38m (1.750ms)
>> - First row fetched: 30h38m (1.361ms)
>> - Unregister query: 30h38m (1.456ms)
>> - ComputeScanRangeAssignmentTimer: 558.440us
>> ImpalaServer:
>> - ClientFetchWaitTimer: 4.721ms
>> - MetastoreUpdateTimer: 191.424us
>> - RowMaterializationTimer: 0.000ns
>> Execution Profile 9649f7ab3bcc5fb8:f4d6a60700000000:(Total: 30h38m,
>> non-child: 0.000ns, % non-child: 0.00%)
>> Number of filters: 0
>> Filter routing table:
>> ID Src. Node Tgt. Node(s) Target type Partition filter Pending
>> (Expected) First arrived Completed Enabled
>>
>> -------------------------------------------------------------------------------------------------------------------
>>
>> Backend startup latencies: Count: 7, min / max: 1ms / 103ms, 25th
>> %-ile: 1ms, 50th %-ile: 67ms, 75th %-ile: 67ms, 90th %-ile: 80ms, 95th
>> %-ile: 103ms, 99.9th %-ile: 103ms
>> DML Stats:
>> Partition: Default
>> NumModifiedRows: 536870912
>> NumRowErrors: 2463157330
>> BytesWritten: 0
>> Per Node Peak Memory Usage: slave1:22000(1.40 GB) slave2:22000(1.41
>> GB) slave3:22000(1.39 GB) slave4:22000(1.41 GB) slave5:22000(1.41 GB)
>> slave6:22000(1.40 GB) slave7:22000(1.40 GB)
>> - FiltersReceived: 0 (0)
>> - FinalizationTimer: 0.000ns
>> - NumBackends: 7 (7)
>> - NumFragmentInstances: 7 (7)
>> - NumFragments: 1 (1)
>> Averaged Fragment F00:(Total: 29h51m, non-child: 14s973ms, %
>> non-child: 0.01%)
>> split sizes: min: 13.63 GB, max: 14.10 GB, avg: 13.90 GB, stddev:
>> 143.22 MB
>> completion times: min:28h47m max:30h38m mean: 29h51m
>> stddev:41m26s
>> execution rates: min:129.50 KB/sec max:140.31 KB/sec mean:135.67
>> KB/sec stddev:3.50 KB/sec
>> num instances: 7
>> - AverageThreadTokens: 8.89
>> - BloomFilterBytes: 0
>> - PeakMemoryUsage: 1.40 GB (1503884428)
>> - PeakReservation: 0
>> - PeakUsedReservation: 0
>> - PerHostPeakMemUsage: 1.40 GB (1503884428)
>> - RowsProduced: 428.58M (428575463)
>> - TotalNetworkReceiveTime: 0.000ns
>> - TotalNetworkSendTime: 0.000ns
>> - TotalStorageWaitTime: 2m10s
>> - TotalThreadsInvoluntaryContextSwitches: 369.81K (369812)
>> - TotalThreadsTotalWallClockTime: 265h20m
>> - TotalThreadsSysTime: 1m59s
>> - TotalThreadsUserTime: 1h9m
>> - TotalThreadsVoluntaryContextSwitches: 433.92K (433917)
>> Buffer pool:
>> - AllocTime: 0.000ns
>> - CumulativeAllocationBytes: 0
>> - CumulativeAllocations: 0 (0)
>> - PeakReservation: 0
>> - PeakUnpinnedBytes: 0
>> - PeakUsedReservation: 0
>> - ReadIoBytes: 0
>> - ReadIoOps: 0 (0)
>> - ReadIoWaitTime: 0.000ns
>> - ReservationLimit: 0
>> - WriteIoBytes: 0
>> - WriteIoOps: 0 (0)
>> - WriteIoWaitTime: 0.000ns
>> Fragment Instance Lifecycle Timings:
>> - ExecTime: 29h51m
>> - ExecTreeExecTime: 4s724ms
>> - OpenTime: 519.095ms
>> - ExecTreeOpenTime: 36.350us
>> - PrepareTime: 60.445ms
>> - ExecTreePrepareTime: 19.494ms
>> KuduTableSink:(Total: 29h50m, non-child: 29h50m, % non-child:
>> 100.00%)
>> - KuduApplyTimer: 28h54m
>> - NumRowErrors: 351.88M (351879618)
>> - PeakMemoryUsage: 20.00 MB (20971520)
>> - RowsProcessedRate: 3.99 K/sec
>> - TotalNumRows: 428.58M (428575463)
>> HDFS_SCAN_NODE (id=0):(Total: 4s417ms, non-child: 4s417ms, %
>> non-child: 100.00%)
>> - AverageHdfsReadThreadConcurrency: 0.00
>> - AverageScannerThreadConcurrency: 7.89
>> - BytesRead: 13.90 GB (14928723218)
>> - BytesReadDataNodeCache: 0
>> - BytesReadLocal: 13.90 GB (14928723218)
>> - BytesReadRemoteUnexpected: 0
>> - BytesReadShortCircuit: 0
>> - CachedFileHandlesHitCount: 2.31K (2305)
>> - CachedFileHandlesMissCount: 56 (56)
>> - CollectionItemsRead: 0 (0)
>> - DecompressionTime: 22s321ms
>> - MaxCompressedTextFileLength: 0
>> - NumColumns: 16 (16)
>> - NumDictFilteredRowGroups: 0 (0)
>> - NumDisksAccessed: 1 (1)
>> - NumRowGroups: 56 (56)
>> - NumScannerThreadsStarted: 8 (8)
>> - NumScannersWithNoReads: 0 (0)
>> - NumStatsFilteredRowGroups: 0 (0)
>> - PeakMemoryUsage: 1.38 GB (1482717084)
>> - PerReadThreadRawHdfsThroughput: 69.65 MB/sec
>> - RemoteScanRanges: 0 (0)
>> - RowBatchQueueGetWaitTime: 1s754ms
>> - RowBatchQueuePutWaitTime: 235h24m
>> - RowsRead: 428.58M (428575463)
>> - RowsReturned: 428.58M (428575463)
>> - RowsReturnedRate: 99.08 M/sec
>> - ScanRangesComplete: 56 (56)
>> - ScannerThreadsInvoluntaryContextSwitches: 12.33K (12332)
>> - ScannerThreadsTotalWallClockTime: 235h28m
>> - MaterializeTupleTime(*): 2m3s
>> - ScannerThreadsSysTime: 3s896ms
>> - ScannerThreadsUserTime: 2m39s
>> - ScannerThreadsVoluntaryContextSwitches: 419.69K (419690)
>> - TotalRawHdfsOpenFileTime(*): 3s995ms
>> - TotalRawHdfsReadTime(*): 4m2s
>> - TotalReadThroughput: 135.78 KB/sec
>> CodeGen:(Total: 436.431ms, non-child: 436.431ms, % non-child:
>> 100.00%)
>> - CodegenInvoluntaryContextSwitches: 5 (5)
>> - CodegenTotalWallClockTime: 436.422ms
>> - CodegenSysTime: 8.585ms
>> - CodegenUserTime: 41.806ms
>> - CodegenVoluntaryContextSwitches: 76 (76)
>> - CompileTime: 179.278ms
>> - IrGenerationTime: 90.176ms
>> - LoadTime: 0.000ns
>> - ModuleBitcodeSize: 2.26 MB (2373832)
>> - NumFunctions: 19 (19)
>> - NumInstructions: 706 (706)
>> - OptimizationTime: 151.857ms
>> - PeakMemoryUsage: 353.00 KB (361472)
>> - PrepareTime: 14.499ms
>> Fragment F00:
>> Instance 9649f7ab3bcc5fb8:f4d6a60700000000
>> (host=slave3:22000):(Total: 30h38m, non-child: 11s364ms, % non-child: 0.01%)
>> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>> 0:56/13.63 GB
>> Fragment Instance Lifecycle Event Timeline: 30h38m
>> - Prepare Finished: 86.126ms (86.126ms)
>> - Open Finished: 1s157ms (1s070ms)
>> - First Batch Produced: 3s031ms (1s874ms)
>> - First Batch Sent: 3s037ms (5.724ms)
>> - ExecInternal Finished: 30h38m (30h38m)
>> MemoryUsage(34m8s): 1.28 GB, 1.34 GB, 1.37 GB, 1.35 GB, 1.21 GB,
>> 925.43 MB, 727.39 MB, 1.17 GB, 1.35 GB, 1.35 GB, 1.35 GB, 1.35 GB, 1.26 GB,
>> 1.06 GB, 836.27 MB, 792.34 MB, 1.26 GB, 1.36 GB, 1.36 GB, 1.30 GB, 1.10 GB,
>> 827.96 MB, 1.01 GB, 1.38 GB, 1.38 GB, 1.32 GB, 1.12 GB, 875.56 MB, 979.49
>> MB, 1.34 GB, 1.37 GB, 1.37 GB, 1.28 GB, 1.10 GB, 950.36 MB, 805.45 MB,
>> 963.12 MB, 1.28 GB, 1.34 GB, 1.34 GB, 1.30 GB, 1.04 GB, 808.54 MB, 994.36
>> MB, 1.17 GB, 1.18 GB, 1.18 GB, 1.18 GB, 1.17 GB, 1.13 GB, 1.04 GB, 954.52
>> MB, 811.51 MB
>> ThreadUsage(34m8s): 8, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9,
>> 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9,
>> 9, 9, 9, 9, 8, 8, 8, 8, 8, 8, 8, 8, 8
>> - AverageThreadTokens: 8.80
>> - BloomFilterBytes: 0
>> - PeakMemoryUsage: 1.39 GB (1487275684)
>> - PeakReservation: 0
>> - PeakUsedReservation: 0
>> - PerHostPeakMemUsage: 1.39 GB (1487275684)
>> - RowsProduced: 424.29M (424287956)
>> - TotalNetworkReceiveTime: 0.000ns
>> - TotalNetworkSendTime: 0.000ns
>> - TotalStorageWaitTime: 1m32s
>> - TotalThreadsInvoluntaryContextSwitches: 349.35K (349347)
>> - TotalThreadsTotalWallClockTime: 269h37m
>> - TotalThreadsSysTime: 1m54s
>> - TotalThreadsUserTime: 1h6m
>> - TotalThreadsVoluntaryContextSwitches: 438.17K (438165)
>> Buffer pool:
>> - AllocTime: 0.000ns
>> - CumulativeAllocationBytes: 0
>> - CumulativeAllocations: 0 (0)
>> - PeakReservation: 0
>> - PeakUnpinnedBytes: 0
>> - PeakUsedReservation: 0
>> - ReadIoBytes: 0
>> - ReadIoOps: 0 (0)
>> - ReadIoWaitTime: 0.000ns
>> - ReservationLimit: 0
>> - WriteIoBytes: 0
>> - WriteIoOps: 0 (0)
>> - WriteIoWaitTime: 0.000ns
>> Fragment Instance Lifecycle Timings:
>> - ExecTime: 30h38m
>> - ExecTreeExecTime: 4s586ms
>> - OpenTime: 1s070ms
>> - ExecTreeOpenTime: 32.390us
>> - PrepareTime: 85.903ms
>> - ExecTreePrepareTime: 16.283ms
>> KuduTableSink:(Total: 30h38m, non-child: 30h38m, % non-child:
>> 100.00%)
>> - KuduApplyTimer: 29h44m
>> - NumRowErrors: 371.48M (371483541)
>> - PeakMemoryUsage: 20.00 MB (20971520)
>> - RowsProcessedRate: 3.85 K/sec
>> - TotalNumRows: 424.29M (424287956)
>> HDFS_SCAN_NODE (id=0):(Total: 4s304ms, non-child: 4s304ms, %
>> non-child: 100.00%)
>> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>> 0:56/13.63 GB
>> ExecOption: PARQUET Codegen Enabled, Codegen enabled: 56 out of
>> 56
>> Hdfs Read Thread Concurrency Bucket: 0:99.87% 1:0.1279% 2:0%
>> 3:0% 4:0% 5:0% 6:0%
>> File Formats: PARQUET/SNAPPY:56
>> BytesRead(34m8s): 1.19 GB, 1.46 GB, 1.65 GB, 1.78 GB, 1.88 GB,
>> 1.97 GB, 2.01 GB, 2.91 GB, 3.27 GB, 3.46 GB, 3.61 GB, 3.72 GB, 3.80 GB, 3.90
>> GB, 3.96 GB, 4.16 GB, 5.05 GB, 5.36 GB, 5.60 GB, 5.76 GB, 5.87 GB, 5.94 GB,
>> 6.57 GB, 7.32 GB, 7.57 GB, 7.75 GB, 7.86 GB, 7.92 GB, 8.43 GB, 9.18 GB, 9.43
>> GB, 9.62 GB, 9.75 GB, 9.84 GB, 9.89 GB, 9.94 GB, 10.41 GB, 11.07 GB, 11.24
>> GB, 11.42 GB, 11.66 GB, 11.82 GB, 12.05 GB, 12.48 GB, 12.96 GB, 13.07 GB,
>> 13.19 GB, 13.31 GB, 13.38 GB, 13.46 GB, 13.51 GB, 13.57 GB, 13.61 GB
>> - FooterProcessingTime: (Avg: 381.833ms ; Min: 46.831ms ; Max:
>> 4s214ms ; Number of samples: 56)
>> - AverageHdfsReadThreadConcurrency: 0.00
>> - AverageScannerThreadConcurrency: 7.80
>> - BytesRead: 13.63 GB (14635606518)
>> - BytesReadDataNodeCache: 0
>> - BytesReadLocal: 13.63 GB (14635606518)
>> - BytesReadRemoteUnexpected: 0
>> - BytesReadShortCircuit: 0
>> - CachedFileHandlesHitCount: 2.27K (2268)
>> - CachedFileHandlesMissCount: 56 (56)
>> - CollectionItemsRead: 0 (0)
>> - DecompressionTime: 19s612ms
>> - MaxCompressedTextFileLength: 0
>> - NumColumns: 16 (16)
>> - NumDictFilteredRowGroups: 0 (0)
>> - NumDisksAccessed: 1 (1)
>> - NumRowGroups: 56 (56)
>> - NumScannerThreadsStarted: 8 (8)
>> - NumScannersWithNoReads: 0 (0)
>> - NumStatsFilteredRowGroups: 0 (0)
>> - PeakMemoryUsage: 1.37 GB (1466047697)
>> - PerReadThreadRawHdfsThroughput: 100.45 MB/sec
>> - RemoteScanRanges: 0 (0)
>> - RowBatchQueueGetWaitTime: 1s872ms
>> - RowBatchQueuePutWaitTime: 238h55m
>> - RowsRead: 424.29M (424287956)
>> - RowsReturned: 424.29M (424287956)
>> - RowsReturnedRate: 98.57 M/sec
>> - ScanRangesComplete: 56 (56)
>> - ScannerThreadsInvoluntaryContextSwitches: 12.67K (12674)
>> - ScannerThreadsTotalWallClockTime: 238h58m
>> - MaterializeTupleTime(*): 1m57s
>> - ScannerThreadsSysTime: 3s421ms
>> - ScannerThreadsUserTime: 2m31s
>> - ScannerThreadsVoluntaryContextSwitches: 415.49K (415490)
>> - TotalRawHdfsOpenFileTime(*): 2s904ms
>> - TotalRawHdfsReadTime(*): 2m18s
>> - TotalReadThroughput: 129.60 KB/sec
>> CodeGen:(Total: 915.511ms, non-child: 915.511ms, % non-child:
>> 100.00%)
>> - CodegenInvoluntaryContextSwitches: 10 (10)
>> - CodegenTotalWallClockTime: 915.504ms
>> - CodegenSysTime: 8.256ms
>> - CodegenUserTime: 45.032ms
>> - CodegenVoluntaryContextSwitches: 87 (87)
>> - CompileTime: 319.788ms
>> - IrGenerationTime: 276.446ms
>> - LoadTime: 0.000ns
>> - ModuleBitcodeSize: 2.26 MB (2373832)
>> - NumFunctions: 19 (19)
>> - NumInstructions: 706 (706)
>> - OptimizationTime: 301.622ms
>> - PeakMemoryUsage: 353.00 KB (361472)
>> - PrepareTime: 16.858ms
>> Instance 9649f7ab3bcc5fb8:f4d6a60700000002
>> (host=slave1:22000):(Total: 30h17m, non-child: 1s508ms, % non-child: 0.00%)
>> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>> 0:57/14.10 GB
>> Fragment Instance Lifecycle Event Timeline: 30h17m
>> - Prepare Finished: 32.853ms (32.853ms)
>> - Open Finished: 362.691ms (329.838ms)
>> - First Batch Produced: 1s725ms (1s362ms)
>> - First Batch Sent: 1s728ms (3.415ms)
>> - ExecInternal Finished: 30h17m (30h17m)
>> MemoryUsage(34m8s): 1.39 GB, 1.40 GB, 1.40 GB, 1.27 GB, 1.07 GB,
>> 798.75 MB, 1.36 GB, 1.38 GB, 1.38 GB, 1.38 GB, 1.32 GB, 1.19 GB, 974.24 MB,
>> 800.43 MB, 1.27 GB, 1.38 GB, 1.38 GB, 1.38 GB, 1.26 GB, 1010.99 MB, 805.20
>> MB, 1.29 GB, 1.38 GB, 1.38 GB, 1.27 GB, 987.74 MB, 811.33 MB, 1.27 GB, 1.38
>> GB, 1.38 GB, 1.34 GB, 1.18 GB, 922.96 MB, 765.77 MB, 1.13 GB, 1.36 GB, 1.36
>> GB, 1.36 GB, 1.37 GB, 1.30 GB, 1.03 GB, 919.88 MB, 1.38 GB, 1.38 GB, 1.38
>> GB, 1.39 GB, 1.35 GB, 1.24 GB, 1.17 GB, 1005.37 MB, 844.55 MB, 751.70 MB,
>> 518.60 MB
>> ThreadUsage(34m8s): 8, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9,
>> 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9,
>> 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 5
>> - AverageThreadTokens: 8.91
>> - BloomFilterBytes: 0
>> - PeakMemoryUsage: 1.40 GB (1500404835)
>> - PeakReservation: 0
>> - PeakUsedReservation: 0
>> - PerHostPeakMemUsage: 1.40 GB (1500404835)
>> - RowsProduced: 434.96M (434955041)
>> - TotalNetworkReceiveTime: 0.000ns
>> - TotalNetworkSendTime: 0.000ns
>> - TotalStorageWaitTime: 2m56s
>> - TotalThreadsInvoluntaryContextSwitches: 370.59K (370587)
>> - TotalThreadsTotalWallClockTime: 269h58m
>> - TotalThreadsSysTime: 1m58s
>> - TotalThreadsUserTime: 1h9m
>> - TotalThreadsVoluntaryContextSwitches: 437.63K (437630)
>> Buffer pool:
>> - AllocTime: 0.000ns
>> - CumulativeAllocationBytes: 0
>> - CumulativeAllocations: 0 (0)
>> - PeakReservation: 0
>> - PeakUnpinnedBytes: 0
>> - PeakUsedReservation: 0
>> - ReadIoBytes: 0
>> - ReadIoOps: 0 (0)
>> - ReadIoWaitTime: 0.000ns
>> - ReservationLimit: 0
>> - WriteIoBytes: 0
>> - WriteIoOps: 0 (0)
>> - WriteIoWaitTime: 0.000ns
>> Fragment Instance Lifecycle Timings:
>> - ExecTime: 30h17m
>> - ExecTreeExecTime: 4s575ms
>> - OpenTime: 329.834ms
>> - ExecTreeOpenTime: 45.986us
>> - PrepareTime: 32.581ms
>> - ExecTreePrepareTime: 18.647ms
>> KuduTableSink:(Total: 30h17m, non-child: 30h17m, % non-child:
>> 100.00%)
>> - KuduApplyTimer: 29h22m
>> - NumRowErrors: 357.86M (357862755)
>> - PeakMemoryUsage: 20.00 MB (20971520)
>> - RowsProcessedRate: 3.99 K/sec
>> - TotalNumRows: 434.96M (434955041)
>> HDFS_SCAN_NODE (id=0):(Total: 4s229ms, non-child: 4s229ms, %
>> non-child: 100.00%)
>> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>> 0:57/14.10 GB
>> ExecOption: PARQUET Codegen Enabled, Codegen enabled: 57 out of
>> 57
>> Hdfs Read Thread Concurrency Bucket: 0:99.63% 1:0.3706% 2:0%
>> 3:0% 4:0% 5:0% 6:0%
>> File Formats: PARQUET/SNAPPY:57
>> BytesRead(34m8s): 1.32 GB, 1.59 GB, 1.74 GB, 1.85 GB, 1.94 GB,
>> 2.11 GB, 3.21 GB, 3.31 GB, 3.48 GB, 3.67 GB, 3.79 GB, 3.88 GB, 3.94 GB, 4.11
>> GB, 5.04 GB, 5.32 GB, 5.50 GB, 5.67 GB, 5.81 GB, 5.92 GB, 6.09 GB, 7.06 GB,
>> 7.40 GB, 7.63 GB, 7.79 GB, 7.90 GB, 8.08 GB, 9.02 GB, 9.35 GB, 9.55 GB, 9.70
>> GB, 9.81 GB, 9.90 GB, 10.01 GB, 10.74 GB, 11.19 GB, 11.34 GB, 11.48 GB,
>> 11.57 GB, 11.69 GB, 11.84 GB, 12.25 GB, 13.25 GB, 13.41 GB, 13.50 GB, 13.60
>> GB, 13.67 GB, 13.73 GB, 13.79 GB, 13.83 GB, 13.86 GB, 13.92 GB, 14.03 GB
>> - FooterProcessingTime: (Avg: 414.198ms ; Min: 51.291ms ; Max:
>> 2s894ms ; Number of samples: 57)
>> - AverageHdfsReadThreadConcurrency: 0.00
>> - AverageScannerThreadConcurrency: 7.91
>> - BytesRead: 14.11 GB (15148002597)
>> - BytesReadDataNodeCache: 0
>> - BytesReadLocal: 14.11 GB (15148002597)
>> - BytesReadRemoteUnexpected: 0
>> - BytesReadShortCircuit: 0
>> - CachedFileHandlesHitCount: 2.33K (2333)
>> - CachedFileHandlesMissCount: 57 (57)
>> - CollectionItemsRead: 0 (0)
>> - DecompressionTime: 23s330ms
>> - MaxCompressedTextFileLength: 0
>> - NumColumns: 16 (16)
>> - NumDictFilteredRowGroups: 0 (0)
>> - NumDisksAccessed: 1 (1)
>> - NumRowGroups: 57 (57)
>> - NumScannerThreadsStarted: 8 (8)
>> - NumScannersWithNoReads: 0 (0)
>> - NumStatsFilteredRowGroups: 0 (0)
>> - PeakMemoryUsage: 1.38 GB (1479367012)
>> - PerReadThreadRawHdfsThroughput: 36.38 MB/sec
>> - RemoteScanRanges: 0 (0)
>> - RowBatchQueueGetWaitTime: 1s354ms
>> - RowBatchQueuePutWaitTime: 239h35m
>> - RowsRead: 434.96M (434955041)
>> - RowsReturned: 434.96M (434955041)
>> - RowsReturnedRate: 102.83 M/sec
>> - ScanRangesComplete: 57 (57)
>> - ScannerThreadsInvoluntaryContextSwitches: 7.23K (7231)
>> - ScannerThreadsTotalWallClockTime: 239h40m
>> - MaterializeTupleTime(*): 2m1s
>> - ScannerThreadsSysTime: 4s044ms
>> - ScannerThreadsUserTime: 2m42s
>> - ScannerThreadsVoluntaryContextSwitches: 425.94K (425935)
>> - TotalRawHdfsOpenFileTime(*): 5s555ms
>> - TotalRawHdfsReadTime(*): 6m37s
>> - TotalReadThroughput: 135.68 KB/sec
>> CodeGen:(Total: 274.452ms, non-child: 274.452ms, % non-child:
>> 100.00%)
>> - CodegenInvoluntaryContextSwitches: 4 (4)
>> - CodegenTotalWallClockTime: 274.443ms
>> - CodegenSysTime: 7.440ms
>> - CodegenUserTime: 40.295ms
>> - CodegenVoluntaryContextSwitches: 86 (86)
>> - CompileTime: 103.404ms
>> - IrGenerationTime: 32.648ms
>> - LoadTime: 0.000ns
>> - ModuleBitcodeSize: 2.26 MB (2373832)
>> - NumFunctions: 19 (19)
>> - NumInstructions: 706 (706)
>> - OptimizationTime: 122.033ms
>> - PeakMemoryUsage: 353.00 KB (361472)
>> - PrepareTime: 15.943ms
>> Instance 9649f7ab3bcc5fb8:f4d6a60700000001
>> (host=slave4:22000):(Total: 30h16m, non-child: 2s019ms, % non-child: 0.00%)
>> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>> 0:56/13.87 GB
>> Fragment Instance Lifecycle Event Timeline: 30h16m
>> - Prepare Finished: 105.379ms (105.379ms)
>> - Open Finished: 519.128ms (413.748ms)
>> - First Batch Produced: 1s789ms (1s270ms)
>> - First Batch Sent: 1s795ms (6.364ms)
>> - ExecInternal Finished: 30h16m (30h16m)
>> MemoryUsage(34m8s): 1.39 GB, 1.40 GB, 1.40 GB, 1.38 GB, 1.23 GB,
>> 924.30 MB, 715.63 MB, 1.26 GB, 1.40 GB, 1.40 GB, 1.40 GB, 1.40 GB, 1.30 GB,
>> 1.08 GB, 843.90 MB, 887.06 MB, 1.40 GB, 1.41 GB, 1.39 GB, 1.19 GB, 826.21
>> MB, 1.19 GB, 1.38 GB, 1.38 GB, 1.31 GB, 1.05 GB, 748.49 MB, 1.17 GB, 1.38
>> GB, 1.38 GB, 1.38 GB, 1.24 GB, 999.76 MB, 733.27 MB, 995.28 MB, 1.35 GB,
>> 1.38 GB, 1.38 GB, 1.38 GB, 1.35 GB, 1.10 GB, 861.01 MB, 1.39 GB, 1.40 GB,
>> 1.40 GB, 1.40 GB, 1.39 GB, 1.33 GB, 1.24 GB, 1.13 GB, 1007.59 MB, 820.51 MB,
>> 655.33 MB
>> ThreadUsage(34m8s): 8, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9,
>> 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9,
>> 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 8
>> - AverageThreadTokens: 8.97
>> - BloomFilterBytes: 0
>> - PeakMemoryUsage: 1.41 GB (1513166691)
>> - PeakReservation: 0
>> - PeakUsedReservation: 0
>> - PerHostPeakMemUsage: 1.41 GB (1513166691)
>> - RowsProduced: 425.55M (425550934)
>> - TotalNetworkReceiveTime: 0.000ns
>> - TotalNetworkSendTime: 0.000ns
>> - TotalStorageWaitTime: 1m49s
>> - TotalThreadsInvoluntaryContextSwitches: 364.52K (364523)
>> - TotalThreadsTotalWallClockTime: 271h36m
>> - TotalThreadsSysTime: 1m56s
>> - TotalThreadsUserTime: 1h9m
>> - TotalThreadsVoluntaryContextSwitches: 431.64K (431643)
>> Buffer pool:
>> - AllocTime: 0.000ns
>> - CumulativeAllocationBytes: 0
>> - CumulativeAllocations: 0 (0)
>> - PeakReservation: 0
>> - PeakUnpinnedBytes: 0
>> - PeakUsedReservation: 0
>> - ReadIoBytes: 0
>> - ReadIoOps: 0 (0)
>> - ReadIoWaitTime: 0.000ns
>> - ReservationLimit: 0
>> - WriteIoBytes: 0
>> - WriteIoOps: 0 (0)
>> - WriteIoWaitTime: 0.000ns
>> Fragment Instance Lifecycle Timings:
>> - ExecTime: 30h16m
>> - ExecTreeExecTime: 4s185ms
>> - OpenTime: 413.744ms
>> - ExecTreeOpenTime: 31.615us
>> - PrepareTime: 105.138ms
>> - ExecTreePrepareTime: 17.849ms
>> KuduTableSink:(Total: 30h16m, non-child: 30h16m, % non-child:
>> 100.00%)
>> - KuduApplyTimer: 29h20m
>> - NumRowErrors: 336.92M (336918666)
>> - PeakMemoryUsage: 20.00 MB (20971520)
>> - RowsProcessedRate: 3.90 K/sec
>> - TotalNumRows: 425.55M (425550934)
>> HDFS_SCAN_NODE (id=0):(Total: 3s886ms, non-child: 3s886ms, %
>> non-child: 100.00%)
>> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>> 0:56/13.87 GB
>> ExecOption: PARQUET Codegen Enabled, Codegen enabled: 56 out of
>> 56
>> Hdfs Read Thread Concurrency Bucket: 0:99.86% 1:0.1368% 2:0%
>> 3:0% 4:0% 5:0% 6:0%
>> File Formats: PARQUET/SNAPPY:56
>> BytesRead(34m8s): 1.29 GB, 1.51 GB, 1.66 GB, 1.79 GB, 1.88 GB,
>> 1.97 GB, 2.00 GB, 3.01 GB, 3.30 GB, 3.49 GB, 3.63 GB, 3.74 GB, 3.82 GB, 3.92
>> GB, 3.96 GB, 4.29 GB, 5.26 GB, 5.47 GB, 5.71 GB, 5.87 GB, 5.94 GB, 6.88 GB,
>> 7.34 GB, 7.59 GB, 7.76 GB, 7.89 GB, 7.93 GB, 8.83 GB, 9.30 GB, 9.51 GB, 9.67
>> GB, 9.79 GB, 9.89 GB, 9.91 GB, 10.45 GB, 11.14 GB, 11.30 GB, 11.46 GB, 11.55
>> GB, 11.68 GB, 11.83 GB, 12.10 GB, 13.21 GB, 13.35 GB, 13.47 GB, 13.56 GB,
>> 13.64 GB, 13.71 GB, 13.76 GB, 13.81 GB, 13.86 GB, 13.87 GB, 13.88 GB
>> - FooterProcessingTime: (Avg: 331.439ms ; Min: 46.760ms ; Max:
>> 2s508ms ; Number of samples: 56)
>> - AverageHdfsReadThreadConcurrency: 0.00
>> - AverageScannerThreadConcurrency: 7.97
>> - BytesRead: 13.88 GB (14898393824)
>> - BytesReadDataNodeCache: 0
>> - BytesReadLocal: 13.88 GB (14898393824)
>> - BytesReadRemoteUnexpected: 0
>> - BytesReadShortCircuit: 0
>> - CachedFileHandlesHitCount: 2.30K (2296)
>> - CachedFileHandlesMissCount: 56 (56)
>> - CollectionItemsRead: 0 (0)
>> - DecompressionTime: 22s686ms
>> - MaxCompressedTextFileLength: 0
>> - NumColumns: 16 (16)
>> - NumDictFilteredRowGroups: 0 (0)
>> - NumDisksAccessed: 1 (1)
>> - NumRowGroups: 56 (56)
>> - NumScannerThreadsStarted: 8 (8)
>> - NumScannersWithNoReads: 0 (0)
>> - NumStatsFilteredRowGroups: 0 (0)
>> - PeakMemoryUsage: 1.39 GB (1491938707)
>> - PerReadThreadRawHdfsThroughput: 95.20 MB/sec
>> - RemoteScanRanges: 0 (0)
>> - RowBatchQueueGetWaitTime: 1s255ms
>> - RowBatchQueuePutWaitTime: 241h15m
>> - RowsRead: 425.55M (425550934)
>> - RowsReturned: 425.55M (425550934)
>> - RowsReturnedRate: 109.50 M/sec
>> - ScanRangesComplete: 56 (56)
>> - ScannerThreadsInvoluntaryContextSwitches: 12.02K (12019)
>> - ScannerThreadsTotalWallClockTime: 241h20m
>> - MaterializeTupleTime(*): 2m6s
>> - ScannerThreadsSysTime: 3s548ms
>> - ScannerThreadsUserTime: 2m39s
>> - ScannerThreadsVoluntaryContextSwitches: 416.75K (416750)
>> - TotalRawHdfsOpenFileTime(*): 3s230ms
>> - TotalRawHdfsReadTime(*): 2m29s
>> - TotalReadThroughput: 133.55 KB/sec
>> CodeGen:(Total: 315.226ms, non-child: 315.226ms, % non-child:
>> 100.00%)
>> - CodegenInvoluntaryContextSwitches: 9 (9)
>> - CodegenTotalWallClockTime: 315.217ms
>> - CodegenSysTime: 5.142ms
>> - CodegenUserTime: 43.310ms
>> - CodegenVoluntaryContextSwitches: 91 (91)
>> - CompileTime: 129.393ms
>> - IrGenerationTime: 60.406ms
>> - LoadTime: 0.000ns
>> - ModuleBitcodeSize: 2.26 MB (2373832)
>> - NumFunctions: 19 (19)
>> - NumInstructions: 706 (706)
>> - OptimizationTime: 111.534ms
>> - PeakMemoryUsage: 353.00 KB (361472)
>> - PrepareTime: 13.425ms
>> Instance 9649f7ab3bcc5fb8:f4d6a60700000004
>> (host=slave7:22000):(Total: 30h10m, non-child: 16s427ms, % non-child: 0.02%)
>> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>> 0:58/14.03 GB
>> Fragment Instance Lifecycle Event Timeline: 30h10m
>> - Prepare Finished: 105.361ms (105.361ms)
>> - Open Finished: 980.610ms (875.249ms)
>> - First Batch Produced: 4s553ms (3s573ms)
>> - First Batch Sent: 4s559ms (5.668ms)
>> - ExecInternal Finished: 30h10m (30h10m)
>> MemoryUsage(34m8s): 1.38 GB, 1.40 GB, 1.40 GB, 1.28 GB, 1.08 GB,
>> 796.55 MB, 1.34 GB, 1.37 GB, 1.35 GB, 1.34 GB, 1.27 GB, 1.21 GB, 983.86 MB,
>> 1.08 GB, 1.38 GB, 1.36 GB, 1.35 GB, 1.28 GB, 1.17 GB, 921.06 MB, 1.02 GB,
>> 1.35 GB, 1.33 GB, 1.28 GB, 1.15 GB, 905.74 MB, 1.01 GB, 1.36 GB, 1.34 GB,
>> 1.32 GB, 1.21 GB, 1.10 GB, 877.34 MB, 1.04 GB, 1.32 GB, 1.38 GB, 1.36 GB,
>> 1.35 GB, 1.32 GB, 1.21 GB, 1000.76 MB, 1.11 GB, 1.37 GB, 1.36 GB, 1.33 GB,
>> 1.29 GB, 1.20 GB, 1.09 GB, 1.07 GB, 937.32 MB, 814.51 MB, 480.87 MB, 155.08
>> MB
>> ThreadUsage(34m8s): 8, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9,
>> 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9,
>> 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 8, 5, 2
>> - AverageThreadTokens: 8.80
>> - BloomFilterBytes: 0
>> - PeakMemoryUsage: 1.40 GB (1500511831)
>> - PeakReservation: 0
>> - PeakUsedReservation: 0
>> - PerHostPeakMemUsage: 1.40 GB (1500511831)
>> - RowsProduced: 433.04M (433043954)
>> - TotalNetworkReceiveTime: 0.000ns
>> - TotalNetworkSendTime: 0.000ns
>> - TotalStorageWaitTime: 3m18s
>> - TotalThreadsInvoluntaryContextSwitches: 374.99K (374993)
>> - TotalThreadsTotalWallClockTime: 265h25m
>> - TotalThreadsSysTime: 1m57s
>> - TotalThreadsUserTime: 1h10m
>> - TotalThreadsVoluntaryContextSwitches: 435.67K (435665)
>> Buffer pool:
>> - AllocTime: 0.000ns
>> - CumulativeAllocationBytes: 0
>> - CumulativeAllocations: 0 (0)
>> - PeakReservation: 0
>> - PeakUnpinnedBytes: 0
>> - PeakUsedReservation: 0
>> - ReadIoBytes: 0
>> - ReadIoOps: 0 (0)
>> - ReadIoWaitTime: 0.000ns
>> - ReservationLimit: 0
>> - WriteIoBytes: 0
>> - WriteIoOps: 0 (0)
>> - WriteIoWaitTime: 0.000ns
>> Fragment Instance Lifecycle Timings:
>> - ExecTime: 30h10m
>> - ExecTreeExecTime: 6s426ms
>> - OpenTime: 875.240ms
>> - ExecTreeOpenTime: 41.994us
>> - PrepareTime: 105.102ms
>> - ExecTreePrepareTime: 24.800ms
>> KuduTableSink:(Total: 30h9m, non-child: 30h9m, % non-child:
>> 100.00%)
>> - KuduApplyTimer: 29h12m
>> - NumRowErrors: 357.43M (357426783)
>> - PeakMemoryUsage: 20.00 MB (20971520)
>> - RowsProcessedRate: 3.99 K/sec
>> - TotalNumRows: 433.04M (433043954)
>> HDFS_SCAN_NODE (id=0):(Total: 6s154ms, non-child: 6s154ms, %
>> non-child: 100.00%)
>> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>> 0:58/14.03 GB
>> ExecOption: PARQUET Codegen Enabled, Codegen enabled: 58 out of
>> 58
>> Hdfs Read Thread Concurrency Bucket: 0:99.68% 1:0.3169% 2:0%
>> 3:0% 4:0% 5:0% 6:0%
>> File Formats: PARQUET/SNAPPY:58
>> BytesRead(34m8s): 1.31 GB, 1.58 GB, 1.74 GB, 1.85 GB, 1.94 GB,
>> 2.10 GB, 3.18 GB, 3.29 GB, 3.44 GB, 3.62 GB, 3.72 GB, 3.95 GB, 4.03 GB, 4.61
>> GB, 5.22 GB, 5.37 GB, 5.52 GB, 5.66 GB, 5.90 GB, 6.01 GB, 6.53 GB, 7.22 GB,
>> 7.43 GB, 7.62 GB, 7.83 GB, 7.99 GB, 8.49 GB, 9.18 GB, 9.38 GB, 9.55 GB, 9.67
>> GB, 9.88 GB, 9.98 GB, 10.53 GB, 11.04 GB, 11.22 GB, 11.39 GB, 11.50 GB,
>> 11.58 GB, 11.70 GB, 11.95 GB, 12.61 GB, 13.30 GB, 13.43 GB, 13.52 GB, 13.59
>> GB, 13.65 GB, 13.72 GB, 13.91 GB, 13.94 GB, 13.96 GB, 13.98 GB, 14.03 GB
>> - FooterProcessingTime: (Avg: 615.194ms ; Min: 50.813ms ; Max:
>> 6s740ms ; Number of samples: 58)
>> - AverageHdfsReadThreadConcurrency: 0.00
>> - AverageScannerThreadConcurrency: 7.80
>> - BytesRead: 14.04 GB (15072987184)
>> - BytesReadDataNodeCache: 0
>> - BytesReadLocal: 14.04 GB (15072987184)
>> - BytesReadRemoteUnexpected: 0
>> - BytesReadShortCircuit: 0
>> - CachedFileHandlesHitCount: 2.34K (2340)
>> - CachedFileHandlesMissCount: 58 (58)
>> - CollectionItemsRead: 0 (0)
>> - DecompressionTime: 21s821ms
>> - MaxCompressedTextFileLength: 0
>> - NumColumns: 16 (16)
>> - NumDictFilteredRowGroups: 0 (0)
>> - NumDisksAccessed: 1 (1)
>> - NumRowGroups: 58 (58)
>> - NumScannerThreadsStarted: 8 (8)
>> - NumScannersWithNoReads: 0 (0)
>> - NumStatsFilteredRowGroups: 0 (0)
>> - PeakMemoryUsage: 1.38 GB (1479283937)
>> - PerReadThreadRawHdfsThroughput: 41.17 MB/sec
>> - RemoteScanRanges: 0 (0)
>> - RowBatchQueueGetWaitTime: 3s555ms
>> - RowBatchQueuePutWaitTime: 235h9m
>> - RowsRead: 433.04M (433043954)
>> - RowsReturned: 433.04M (433043954)
>> - RowsReturnedRate: 70.36 M/sec
>> - ScanRangesComplete: 58 (58)
>> - ScannerThreadsInvoluntaryContextSwitches: 16.45K (16447)
>> - ScannerThreadsTotalWallClockTime: 235h15m
>> - MaterializeTupleTime(*): 2m5s
>> - ScannerThreadsSysTime: 3s924ms
>> - ScannerThreadsUserTime: 2m40s
>> - ScannerThreadsVoluntaryContextSwitches: 424.08K (424080)
>> - TotalRawHdfsOpenFileTime(*): 4s852ms
>> - TotalRawHdfsReadTime(*): 5m49s
>> - TotalReadThroughput: 135.58 KB/sec
>> CodeGen:(Total: 771.273ms, non-child: 771.273ms, % non-child:
>> 100.00%)
>> - CodegenInvoluntaryContextSwitches: 5 (5)
>> - CodegenTotalWallClockTime: 771.263ms
>> - CodegenSysTime: 19.916ms
>> - CodegenUserTime: 41.600ms
>> - CodegenVoluntaryContextSwitches: 91 (91)
>> - CompileTime: 404.632ms
>> - IrGenerationTime: 153.195ms
>> - LoadTime: 0.000ns
>> - ModuleBitcodeSize: 2.26 MB (2373832)
>> - NumFunctions: 19 (19)
>> - NumInstructions: 706 (706)
>> - OptimizationTime: 199.856ms
>> - PeakMemoryUsage: 353.00 KB (361472)
>> - PrepareTime: 12.885ms
>> Instance 9649f7ab3bcc5fb8:f4d6a60700000006
>> (host=slave6:22000):(Total: 29h59m, non-child: 21s225ms, % non-child: 0.02%)
>> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>> 0:57/13.92 GB
>> Fragment Instance Lifecycle Event Timeline: 29h59m
>> - Prepare Finished: 55.600ms (55.600ms)
>> - Open Finished: 598.215ms (542.614ms)
>> - First Batch Produced: 2s350ms (1s752ms)
>> - First Batch Sent: 2s356ms (5.853ms)
>> - ExecInternal Finished: 29h59m (29h59m)
>> MemoryUsage(34m8s): 1.37 GB, 1.37 GB, 1.36 GB, 1.24 GB, 1.04 GB,
>> 864.64 MB, 1.37 GB, 1.40 GB, 1.40 GB, 1.38 GB, 1.31 GB, 1.17 GB, 965.56 MB,
>> 876.58 MB, 1.32 GB, 1.39 GB, 1.39 GB, 1.38 GB, 1.23 GB, 995.12 MB, 871.65
>> MB, 1.35 GB, 1.40 GB, 1.38 GB, 1.24 GB, 977.74 MB, 850.26 MB, 1.27 GB, 1.29
>> GB, 1.30 GB, 1.31 GB, 1.16 GB, 954.46 MB, 906.32 MB, 1.24 GB, 1.34 GB, 1.30
>> GB, 1.31 GB, 1.36 GB, 1.29 GB, 1.02 GB, 1.02 GB, 1.32 GB, 1.30 GB, 1.31 GB,
>> 1.38 GB, 1.33 GB, 1.23 GB, 1.14 GB, 1017.39 MB, 822.59 MB, 631.09 MB
>> ThreadUsage(34m8s): 8, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9,
>> 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9,
>> 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 8, 7
>> - AverageThreadTokens: 8.86
>> - BloomFilterBytes: 0
>> - PeakMemoryUsage: 1.40 GB (1499916960)
>> - PeakReservation: 0
>> - PeakUsedReservation: 0
>> - PerHostPeakMemUsage: 1.40 GB (1499916960)
>> - RowsProduced: 428.25M (428247756)
>> - TotalNetworkReceiveTime: 0.000ns
>> - TotalNetworkSendTime: 0.000ns
>> - TotalStorageWaitTime: 1m52s
>> - TotalThreadsInvoluntaryContextSwitches: 388.30K (388300)
>> - TotalThreadsTotalWallClockTime: 265h50m
>> - TotalThreadsSysTime: 1m58s
>> - TotalThreadsUserTime: 1h13m
>> - TotalThreadsVoluntaryContextSwitches: 431.32K (431323)
>> Buffer pool:
>> - AllocTime: 0.000ns
>> - CumulativeAllocationBytes: 0
>> - CumulativeAllocations: 0 (0)
>> - PeakReservation: 0
>> - PeakUnpinnedBytes: 0
>> - PeakUsedReservation: 0
>> - ReadIoBytes: 0
>> - ReadIoOps: 0 (0)
>> - ReadIoWaitTime: 0.000ns
>> - ReservationLimit: 0
>> - WriteIoBytes: 0
>> - WriteIoOps: 0 (0)
>> - WriteIoWaitTime: 0.000ns
>> Fragment Instance Lifecycle Timings:
>> - ExecTime: 29h59m
>> - ExecTreeExecTime: 4s703ms
>> - OpenTime: 542.606ms
>> - ExecTreeOpenTime: 32.022us
>> - PrepareTime: 55.377ms
>> - ExecTreePrepareTime: 20.617ms
>> KuduTableSink:(Total: 29h59m, non-child: 29h59m, % non-child:
>> 100.00%)
>> - KuduApplyTimer: 29h
>> - NumRowErrors: 346.33M (346333617)
>> - PeakMemoryUsage: 20.00 MB (20971520)
>> - RowsProcessedRate: 3.97 K/sec
>> - TotalNumRows: 428.25M (428247756)
>> HDFS_SCAN_NODE (id=0):(Total: 4s407ms, non-child: 4s407ms, %
>> non-child: 100.00%)
>> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>> 0:57/13.92 GB
>> ExecOption: PARQUET Codegen Enabled, Codegen enabled: 57 out of
>> 57
>> Hdfs Read Thread Concurrency Bucket: 0:99.81% 1:0.1895% 2:0%
>> 3:0% 4:0% 5:0% 6:0%
>> File Formats: PARQUET/SNAPPY:57
>> BytesRead(34m8s): 1.29 GB, 1.56 GB, 1.71 GB, 1.81 GB, 1.90 GB,
>> 2.18 GB, 3.19 GB, 3.30 GB, 3.46 GB, 3.65 GB, 3.76 GB, 3.84 GB, 3.91 GB, 4.19
>> GB, 5.09 GB, 5.31 GB, 5.48 GB, 5.64 GB, 5.78 GB, 5.90 GB, 6.16 GB, 7.13 GB,
>> 7.39 GB, 7.61 GB, 7.76 GB, 7.91 GB, 8.15 GB, 9.04 GB, 9.25 GB, 9.49 GB, 9.72
>> GB, 9.83 GB, 9.95 GB, 10.16 GB, 10.86 GB, 11.12 GB, 11.26 GB, 11.41 GB,
>> 11.60 GB, 11.71 GB, 11.87 GB, 12.39 GB, 13.18 GB, 13.32 GB, 13.42 GB, 13.63
>> GB, 13.70 GB, 13.76 GB, 13.82 GB, 13.87 GB, 13.89 GB, 13.90 GB
>> - FooterProcessingTime: (Avg: 368.827ms ; Min: 61.227ms ; Max:
>> 3s788ms ; Number of samples: 57)
>> - AverageHdfsReadThreadConcurrency: 0.00
>> - AverageScannerThreadConcurrency: 7.86
>> - BytesRead: 13.92 GB (14949018844)
>> - BytesReadDataNodeCache: 0
>> - BytesReadLocal: 13.92 GB (14949018844)
>> - BytesReadRemoteUnexpected: 0
>> - BytesReadShortCircuit: 0
>> - CachedFileHandlesHitCount: 2.31K (2315)
>> - CachedFileHandlesMissCount: 57 (57)
>> - CollectionItemsRead: 0 (0)
>> - DecompressionTime: 22s734ms
>> - MaxCompressedTextFileLength: 0
>> - NumColumns: 16 (16)
>> - NumDictFilteredRowGroups: 0 (0)
>> - NumDisksAccessed: 1 (1)
>> - NumRowGroups: 57 (57)
>> - NumScannerThreadsStarted: 8 (8)
>> - NumScannersWithNoReads: 0 (0)
>> - NumStatsFilteredRowGroups: 0 (0)
>> - PeakMemoryUsage: 1.38 GB (1478857724)
>> - PerReadThreadRawHdfsThroughput: 69.53 MB/sec
>> - RemoteScanRanges: 0 (0)
>> - RowBatchQueueGetWaitTime: 1s749ms
>> - RowBatchQueuePutWaitTime: 235h46m
>> - RowsRead: 428.25M (428247756)
>> - RowsReturned: 428.25M (428247756)
>> - RowsReturnedRate: 97.15 M/sec
>> - ScanRangesComplete: 57 (57)
>> - ScannerThreadsInvoluntaryContextSwitches: 15.99K (15992)
>> - ScannerThreadsTotalWallClockTime: 235h50m
>> - MaterializeTupleTime(*): 2m7s
>> - ScannerThreadsSysTime: 3s685ms
>> - ScannerThreadsUserTime: 2m42s
>> - ScannerThreadsVoluntaryContextSwitches: 419.35K (419351)
>> - TotalRawHdfsOpenFileTime(*): 3s380ms
>> - TotalRawHdfsReadTime(*): 3m25s
>> - TotalReadThroughput: 135.25 KB/sec
>> CodeGen:(Total: 405.466ms, non-child: 405.466ms, % non-child:
>> 100.00%)
>> - CodegenInvoluntaryContextSwitches: 2 (2)
>> - CodegenTotalWallClockTime: 405.458ms
>> - CodegenSysTime: 0.000ns
>> - CodegenUserTime: 49.456ms
>> - CodegenVoluntaryContextSwitches: 91 (91)
>> - CompileTime: 170.012ms
>> - IrGenerationTime: 52.154ms
>> - LoadTime: 0.000ns
>> - ModuleBitcodeSize: 2.26 MB (2373832)
>> - NumFunctions: 19 (19)
>> - NumInstructions: 706 (706)
>> - OptimizationTime: 168.310ms
>> - PeakMemoryUsage: 353.00 KB (361472)
>> - PrepareTime: 14.493ms
>> Instance 9649f7ab3bcc5fb8:f4d6a60700000003
>> (host=slave2:22000):(Total: 28h48m, non-child: 21s449ms, % non-child: 0.02%)
>> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>> 0:56/13.87 GB
>> Fragment Instance Lifecycle Event Timeline: 28h48m
>> - Prepare Finished: 13.348ms (13.348ms)
>> - Open Finished: 348.443ms (335.094ms)
>> - First Batch Produced: 1s505ms (1s157ms)
>> - First Batch Sent: 1s508ms (3.334ms)
>> - ExecInternal Finished: 28h48m (28h48m)
>> MemoryUsage(34m8s): 1.38 GB, 1.38 GB, 1.39 GB, 1.26 GB, 1.06 GB,
>> 777.91 MB, 1.37 GB, 1.38 GB, 1.38 GB, 1.39 GB, 1.33 GB, 1.20 GB, 985.85 MB,
>> 781.77 MB, 1.25 GB, 1.37 GB, 1.37 GB, 1.37 GB, 1.25 GB, 1015.91 MB, 756.94
>> MB, 1.27 GB, 1.37 GB, 1.37 GB, 1.27 GB, 1012.70 MB, 775.19 MB, 1.25 GB, 1.41
>> GB, 1.41 GB, 1.30 GB, 1.01 GB, 871.44 MB, 1.34 GB, 1.38 GB, 1.38 GB, 1.38
>> GB, 1.37 GB, 1.31 GB, 1.14 GB, 825.57 MB, 1.32 GB, 1.40 GB, 1.40 GB, 1.37
>> GB, 1.28 GB, 1.17 GB, 1.02 GB, 856.05 MB, 694.00 MB
>> ThreadUsage(34m8s): 8, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9,
>> 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9,
>> 9, 9, 9, 9, 9, 9, 9, 9, 9, 8
>> - AverageThreadTokens: 8.94
>> - BloomFilterBytes: 0
>> - PeakMemoryUsage: 1.41 GB (1512855058)
>> - PeakReservation: 0
>> - PeakUsedReservation: 0
>> - PerHostPeakMemUsage: 1.41 GB (1512855058)
>> - RowsProduced: 427.11M (427110203)
>> - TotalNetworkReceiveTime: 0.000ns
>> - TotalNetworkSendTime: 0.000ns
>> - TotalStorageWaitTime: 2m27s
>> - TotalThreadsInvoluntaryContextSwitches: 375.92K (375918)
>> - TotalThreadsTotalWallClockTime: 257h33m
>> - TotalThreadsSysTime: 2m12s
>> - TotalThreadsUserTime: 1h9m
>> - TotalThreadsVoluntaryContextSwitches: 431.39K (431386)
>> Buffer pool:
>> - AllocTime: 0.000ns
>> - CumulativeAllocationBytes: 0
>> - CumulativeAllocations: 0 (0)
>> - PeakReservation: 0
>> - PeakUnpinnedBytes: 0
>> - PeakUsedReservation: 0
>> - ReadIoBytes: 0
>> - ReadIoOps: 0 (0)
>> - ReadIoWaitTime: 0.000ns
>> - ReservationLimit: 0
>> - WriteIoBytes: 0
>> - WriteIoOps: 0 (0)
>> - WriteIoWaitTime: 0.000ns
>> Fragment Instance Lifecycle Timings:
>> - ExecTime: 28h48m
>> - ExecTreeExecTime: 4s366ms
>> - OpenTime: 335.089ms
>> - ExecTreeOpenTime: 40.091us
>> - PrepareTime: 13.257ms
>> - ExecTreePrepareTime: 12.776ms
>> KuduTableSink:(Total: 28h48m, non-child: 28h48m, % non-child:
>> 100.00%)
>> - KuduApplyTimer: 27h51m
>> - NumRowErrors: 347.46M (347457539)
>> - PeakMemoryUsage: 20.00 MB (20971520)
>> - RowsProcessedRate: 4.12 K/sec
>> - TotalNumRows: 427.11M (427110203)
>> HDFS_SCAN_NODE (id=0):(Total: 4s001ms, non-child: 4s001ms, %
>> non-child: 100.00%)
>> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>> 0:56/13.87 GB
>> ExecOption: PARQUET Codegen Enabled, Codegen enabled: 56 out of
>> 56
>> Hdfs Read Thread Concurrency Bucket: 0:99.69% 1:0.3106% 2:0%
>> 3:0% 4:0% 5:0% 6:0%
>> File Formats: PARQUET/SNAPPY:56
>> BytesRead(34m8s): 1.31 GB, 1.58 GB, 1.74 GB, 1.85 GB, 1.94 GB,
>> 2.08 GB, 3.22 GB, 3.31 GB, 3.48 GB, 3.67 GB, 3.79 GB, 3.88 GB, 3.94 GB, 4.08
>> GB, 5.02 GB, 5.30 GB, 5.48 GB, 5.66 GB, 5.80 GB, 5.91 GB, 6.00 GB, 7.04 GB,
>> 7.37 GB, 7.61 GB, 7.77 GB, 7.90 GB, 8.01 GB, 8.95 GB, 9.39 GB, 9.61 GB, 9.77
>> GB, 9.88 GB, 10.17 GB, 11.12 GB, 11.26 GB, 11.40 GB, 11.56 GB, 11.66 GB,
>> 11.73 GB, 11.82 GB, 11.97 GB, 13.08 GB, 13.48 GB, 13.60 GB, 13.68 GB, 13.74
>> GB, 13.80 GB, 13.85 GB, 13.87 GB, 13.88 GB
>> - FooterProcessingTime: (Avg: 339.758ms ; Min: 55.620ms ; Max:
>> 2s575ms ; Number of samples: 56)
>> - AverageHdfsReadThreadConcurrency: 0.00
>> - AverageScannerThreadConcurrency: 7.94
>> - BytesRead: 13.88 GB (14898474484)
>> - BytesReadDataNodeCache: 0
>> - BytesReadLocal: 13.88 GB (14898474484)
>> - BytesReadRemoteUnexpected: 0
>> - BytesReadShortCircuit: 0
>> - CachedFileHandlesHitCount: 2.29K (2294)
>> - CachedFileHandlesMissCount: 56 (56)
>> - CollectionItemsRead: 0 (0)
>> - DecompressionTime: 23s940ms
>> - MaxCompressedTextFileLength: 0
>> - NumColumns: 16 (16)
>> - NumDictFilteredRowGroups: 0 (0)
>> - NumDisksAccessed: 1 (1)
>> - NumRowGroups: 56 (56)
>> - NumScannerThreadsStarted: 8 (8)
>> - NumScannersWithNoReads: 0 (0)
>> - NumStatsFilteredRowGroups: 0 (0)
>> - PeakMemoryUsage: 1.39 GB (1491692569)
>> - PerReadThreadRawHdfsThroughput: 44.62 MB/sec
>> - RemoteScanRanges: 0 (0)
>> - RowBatchQueueGetWaitTime: 1s156ms
>> - RowBatchQueuePutWaitTime: 228h40m
>> - RowsRead: 427.11M (427110203)
>> - RowsReturned: 427.11M (427110203)
>> - RowsReturnedRate: 106.74 M/sec
>> - ScanRangesComplete: 56 (56)
>> - ScannerThreadsInvoluntaryContextSwitches: 7.41K (7412)
>> - ScannerThreadsTotalWallClockTime: 228h45m
>> - MaterializeTupleTime(*): 2m
>> - ScannerThreadsSysTime: 4s574ms
>> - ScannerThreadsUserTime: 2m39s
>> - ScannerThreadsVoluntaryContextSwitches: 418.25K (418255)
>> - TotalRawHdfsOpenFileTime(*): 4s930ms
>> - TotalRawHdfsReadTime(*): 5m18s
>> - TotalReadThroughput: 140.35 KB/sec
>> CodeGen:(Total: 316.986ms, non-child: 316.986ms, % non-child:
>> 100.00%)
>> - CodegenInvoluntaryContextSwitches: 7 (7)
>> - CodegenTotalWallClockTime: 316.976ms
>> - CodegenSysTime: 19.346ms
>> - CodegenUserTime: 35.434ms
>> - CodegenVoluntaryContextSwitches: 87 (87)
>> - CompileTime: 104.224ms
>> - IrGenerationTime: 54.920ms
>> - LoadTime: 0.000ns
>> - ModuleBitcodeSize: 2.26 MB (2373832)
>> - NumFunctions: 19 (19)
>> - NumInstructions: 706 (706)
>> - OptimizationTime: 143.706ms
>> - PeakMemoryUsage: 353.00 KB (361472)
>> - PrepareTime: 13.284ms
>> Instance 9649f7ab3bcc5fb8:f4d6a60700000005
>> (host=slave5:22000):(Total: 28h47m, non-child: 30s821ms, % non-child: 0.03%)
>> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>> 0:56/13.87 GB
>> Fragment Instance Lifecycle Event Timeline: 28h47m
>> - Prepare Finished: 25.961ms (25.961ms)
>> - Open Finished: 92.181ms (66.220ms)
>> - First Batch Produced: 1s428ms (1s336ms)
>> - First Batch Sent: 1s432ms (3.458ms)
>> - ExecInternal Finished: 28h47m (28h47m)
>> MemoryUsage(34m8s): 1.39 GB, 1.39 GB, 1.39 GB, 1.27 GB, 1.07 GB,
>> 786.07 MB, 1.37 GB, 1.37 GB, 1.37 GB, 1.37 GB, 1.31 GB, 1.19 GB, 973.49 MB,
>> 778.55 MB, 1.26 GB, 1.39 GB, 1.39 GB, 1.39 GB, 1.27 GB, 1.00 GB, 812.17 MB,
>> 1.29 GB, 1.41 GB, 1.39 GB, 1.14 GB, 881.39 MB, 1.30 GB, 1.37 GB, 1.37 GB,
>> 1.33 GB, 1.16 GB, 910.68 MB, 880.78 MB, 1.32 GB, 1.38 GB, 1.38 GB, 1.38 GB,
>> 1.37 GB, 1.29 GB, 1.13 GB, 843.04 MB, 1.32 GB, 1.39 GB, 1.39 GB, 1.35 GB,
>> 1.27 GB, 1.16 GB, 1.00 GB, 848.32 MB, 709.30 MB
>> ThreadUsage(34m8s): 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9,
>> 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9,
>> 9, 9, 9, 9, 9, 9, 9, 9, 9, 8
>> - AverageThreadTokens: 8.94
>> - BloomFilterBytes: 0
>> - PeakMemoryUsage: 1.41 GB (1513059939)
>> - PeakReservation: 0
>> - PeakUsedReservation: 0
>> - PerHostPeakMemUsage: 1.41 GB (1513059939)
>> - RowsProduced: 426.83M (426832398)
>> - TotalNetworkReceiveTime: 0.000ns
>> - TotalNetworkSendTime: 0.000ns
>> - TotalStorageWaitTime: 1m16s
>> - TotalThreadsInvoluntaryContextSwitches: 365.02K (365022)
>> - TotalThreadsTotalWallClockTime: 257h18m
>> - TotalThreadsSysTime: 1m56s
>> - TotalThreadsUserTime: 1h9m
>> - TotalThreadsVoluntaryContextSwitches: 431.61K (431609)
>> Buffer pool:
>> - AllocTime: 0.000ns
>> - CumulativeAllocationBytes: 0
>> - CumulativeAllocations: 0 (0)
>> - PeakReservation: 0
>> - PeakUnpinnedBytes: 0
>> - PeakUsedReservation: 0
>> - ReadIoBytes: 0
>> - ReadIoOps: 0 (0)
>> - ReadIoWaitTime: 0.000ns
>> - ReservationLimit: 0
>> - WriteIoBytes: 0
>> - WriteIoOps: 0 (0)
>> - WriteIoWaitTime: 0.000ns
>> Fragment Instance Lifecycle Timings:
>> - ExecTime: 28h47m
>> - ExecTreeExecTime: 4s226ms
>> - OpenTime: 66.213ms
>> - ExecTreeOpenTime: 30.358us
>> - PrepareTime: 25.760ms
>> - ExecTreePrepareTime: 25.484ms
>> KuduTableSink:(Total: 28h46m, non-child: 28h46m, % non-child:
>> 100.00%)
>> - KuduApplyTimer: 27h51m
>> - NumRowErrors: 345.67M (345674429)
>> - PeakMemoryUsage: 20.00 MB (20971520)
>> - RowsProcessedRate: 4.12 K/sec
>> - TotalNumRows: 426.83M (426832398)
>> HDFS_SCAN_NODE (id=0):(Total: 3s937ms, non-child: 3s937ms, %
>> non-child: 100.00%)
>> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>> 0:56/13.87 GB
>> ExecOption: PARQUET Codegen Enabled, Codegen enabled: 56 out of
>> 56
>> Hdfs Read Thread Concurrency Bucket: 0:99.86% 1:0.138% 2:0% 3:0%
>> 4:0% 5:0% 6:0%
>> File Formats: PARQUET/SNAPPY:56
>> BytesRead(34m8s): 1.31 GB, 1.58 GB, 1.74 GB, 1.85 GB, 1.94 GB,
>> 2.08 GB, 3.23 GB, 3.30 GB, 3.48 GB, 3.67 GB, 3.78 GB, 3.87 GB, 3.94 GB, 4.09
>> GB, 5.01 GB, 5.31 GB, 5.49 GB, 5.66 GB, 5.81 GB, 5.92 GB, 6.09 GB, 7.05 GB,
>> 7.45 GB, 7.69 GB, 7.86 GB, 8.14 GB, 9.07 GB, 9.35 GB, 9.56 GB, 9.70 GB, 9.81
>> GB, 9.90 GB, 10.23 GB, 11.09 GB, 11.27 GB, 11.41 GB, 11.56 GB, 11.66 GB,
>> 11.73 GB, 11.82 GB, 12.00 GB, 13.08 GB, 13.48 GB, 13.61 GB, 13.68 GB, 13.74
>> GB, 13.80 GB, 13.85 GB, 13.87 GB, 13.88 GB
>> - FooterProcessingTime: (Avg: 256.473ms ; Min: 49.349ms ; Max:
>> 2s343ms ; Number of samples: 56)
>> - AverageHdfsReadThreadConcurrency: 0.00
>> - AverageScannerThreadConcurrency: 7.94
>> - BytesRead: 13.88 GB (14898579078)
>> - BytesReadDataNodeCache: 0
>> - BytesReadLocal: 13.88 GB (14898579078)
>> - BytesReadRemoteUnexpected: 0
>> - BytesReadShortCircuit: 0
>> - CachedFileHandlesHitCount: 2.29K (2293)
>> - CachedFileHandlesMissCount: 56 (56)
>> - CollectionItemsRead: 0 (0)
>> - DecompressionTime: 22s127ms
>> - MaxCompressedTextFileLength: 0
>> - NumColumns: 16 (16)
>> - NumDictFilteredRowGroups: 0 (0)
>> - NumDisksAccessed: 1 (1)
>> - NumRowGroups: 56 (56)
>> - NumScannerThreadsStarted: 8 (8)
>> - NumScannersWithNoReads: 0 (0)
>> - NumStatsFilteredRowGroups: 0 (0)
>> - PeakMemoryUsage: 1.39 GB (1491831945)
>> - PerReadThreadRawHdfsThroughput: 100.21 MB/sec
>> - RemoteScanRanges: 0 (0)
>> - RowBatchQueueGetWaitTime: 1s334ms
>> - RowBatchQueuePutWaitTime: 228h26m
>> - RowsRead: 426.83M (426832398)
>> - RowsReturned: 426.83M (426832398)
>> - RowsReturnedRate: 108.39 M/sec
>> - ScanRangesComplete: 56 (56)
>> - ScannerThreadsInvoluntaryContextSwitches: 14.55K (14553)
>> - ScannerThreadsTotalWallClockTime: 228h30m
>> - MaterializeTupleTime(*): 2m3s
>> - ScannerThreadsSysTime: 4s079ms
>> - ScannerThreadsUserTime: 2m39s
>> - ScannerThreadsVoluntaryContextSwitches: 417.98K (417975)
>> - TotalRawHdfsOpenFileTime(*): 3s114ms
>> - TotalRawHdfsReadTime(*): 2m21s
>> - TotalReadThroughput: 140.45 KB/sec
>> CodeGen:(Total: 56.104ms, non-child: 56.104ms, % non-child:
>> 100.00%)
>> - CodegenInvoluntaryContextSwitches: 3 (3)
>> - CodegenTotalWallClockTime: 56.095ms
>> - CodegenSysTime: 0.000ns
>> - CodegenUserTime: 37.520ms
>> - CodegenVoluntaryContextSwitches: 1 (1)
>> - CompileTime: 23.495ms
>> - IrGenerationTime: 1.462ms
>> - LoadTime: 0.000ns
>> - ModuleBitcodeSize: 2.26 MB (2373832)
>> - NumFunctions: 19 (19)
>> - NumInstructions: 706 (706)
>> - OptimizationTime: 15.936ms
>> - PeakMemoryUsage: 353.00 KB (361472)
>> - PrepareTime: 14.604ms
>>
>>
>>
>> On Thu, May 10, 2018 at 10:33 AM, Mostafa Mokhtar <mm...@cloudera.com>
>> wrote:
>>>
>>> Can you share the query profile for the successful insert query?
>>>
>>> Thanks
>>> Mostafa
>>>
>>> On May 9, 2018, at 9:55 PM, Geetika Gupta <ge...@knoldus.in>
>>> wrote:
>>>
>>> Thanks, Jeszy.
>>>
>>> We build impala again with --release flag and data load was successful
>>> after that.
>>>
>>> But now we are facing another issue. The table in which we loaded the
>>> data has less number of rows. We executed the following command:
>>>
>>> insert into LINEITEM select * from PARQUETIMPALA500.LINEITEM
>>>
>>> This query was successful, but when we tried the count(*) on both the
>>> tables, row count was different:
>>>
>>> 0: jdbc:hive2://slave2:21050/default> select count(*) from lineitem
>>> . . . . . . . . . . . . . . . . . . > ;
>>> 536870912
>>>
>>> 0: jdbc:hive2://slave2:21050/default> select count(*) from
>>> parquetimpala500.lineitem;
>>> 3000028242
>>>
>>> Do you have any idea about this issue.
>>>
>>>
>>> On Mon, May 7, 2018 at 12:06 PM, Jeszy <je...@gmail.com> wrote:
>>>>
>>>> Impala doesn't store the data itself, so you can switch versions
>>>> without rewriting data. But you don't have to do that, you would just
>>>> have to build impala using the -release flag (of buildall.sh) and run
>>>> it using the release binaries (versus the debug ones). If you would be
>>>> looking at performance, using the release version is highly
>>>> recommended anyway.
>>>>
>>>> On 7 May 2018 at 08:30, Geetika Gupta <ge...@knoldus.in> wrote:
>>>> > Hi Jeszy,
>>>> >
>>>> > Currently, we are using the apache impala's Github master branch code.
>>>> > We
>>>> > tried using the released version but we encountered some errors
>>>> > related to
>>>> > downloading of dependencies and could not complete the installation.
>>>> >
>>>> > The current version of impala we are using: 2.12
>>>> >
>>>> > We can't try with the new release as we have already loaded 500GB of
>>>> > TPCH
>>>> > data on our cluster.
>>>> >
>>>> > On Mon, May 7, 2018 at 11:43 AM, Jeszy <je...@gmail.com> wrote:
>>>> >>
>>>> >> What version of Impala are you using?
>>>> >> DCHECKs won't be triggered if you run a release build. Looking at the
>>>> >> code, it should work with bad values if not for the DCHECK. Can you
>>>> >> try using a release build?
>>>> >>
>>>> >> On 7 May 2018 at 08:04, Geetika Gupta <ge...@knoldus.in>
>>>> >> wrote:
>>>> >> > Hi community,
>>>> >> >
>>>> >> > I was trying to load 500GB of TPCH data into kudu table using the
>>>> >> > following
>>>> >> > query:
>>>> >> >
>>>> >> > insert into lineitem select * from PARQUETIMPALA500.LINEITEM
>>>> >> >
>>>> >> > While executing the query for around 17 hrs it got cancelled as the
>>>> >> > impalad
>>>> >> > process of that machine got aborted. Here are the logs of the
>>>> >> > impalad
>>>> >> > process.
>>>> >> >
>>>> >> > impalad.ERROR
>>>> >> >
>>>> >> > Log file created at: 2018/05/06 13:40:34
>>>> >> > Running on machine: slave2
>>>> >> > Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
>>>> >> > E0506 13:40:34.097759 28730 logging.cc:121] stderr will be logged
>>>> >> > to
>>>> >> > this
>>>> >> > file.
>>>> >> > SLF4J: Class path contains multiple SLF4J bindings.
>>>> >> > SLF4J: Found binding in
>>>> >> >
>>>> >> >
>>>> >> > [jar:file:/root/softwares/impala/fe/target/dependency/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>>>> >> > SLF4J: Found binding in
>>>> >> >
>>>> >> >
>>>> >> > [jar:file:/root/softwares/impala/testdata/target/dependency/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>>>> >> > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
>>>> >> > explanation.
>>>> >> > SLF4J: Actual binding is of type
>>>> >> > [org.slf4j.impl.Log4jLoggerFactory]
>>>> >> > 18/05/06 13:40:34 WARN util.NativeCodeLoader: Unable to load
>>>> >> > native-hadoop
>>>> >> > library for your platform... using builtin-java classes where
>>>> >> > applicable
>>>> >> > 18/05/06 13:40:36 WARN shortcircuit.DomainSocketFactory: The
>>>> >> > short-circuit
>>>> >> > local reads feature cannot be used because libhadoop cannot be
>>>> >> > loaded.
>>>> >> > tcmalloc: large alloc 1073741824 bytes == 0x484434000 @ 0x4135176
>>>> >> > 0x7fd9e9fc3929
>>>> >> > tcmalloc: large alloc 2147483648 bytes == 0x7fd540f18000 @
>>>> >> > 0x4135176
>>>> >> > 0x7fd9e9fc3929
>>>> >> > F0507 09:46:12.673912 29258 error-util.cc:148] Check failed:
>>>> >> > log_entry.count
>>>> >> >> 0 (-1831809966 vs. 0)
>>>> >> > *** Check failure stack trace: ***
>>>> >> > @ 0x3fc0c0d google::LogMessage::Fail()
>>>> >> > @ 0x3fc24b2 google::LogMessage::SendToLog()
>>>> >> > @ 0x3fc05e7 google::LogMessage::Flush()
>>>> >> > @ 0x3fc3bae
>>>> >> > google::LogMessageFatal::~LogMessageFatal()
>>>> >> > @ 0x1bbcb31 impala::PrintErrorMap()
>>>> >> > @ 0x1bbcd07 impala::PrintErrorMapToString()
>>>> >> > @ 0x2decbd7 impala::Coordinator::GetErrorLog()
>>>> >> > @ 0x1a8d634 impala::ImpalaServer::UnregisterQuery()
>>>> >> > @ 0x1b29264 impala::ImpalaServer::CloseOperation()
>>>> >> > @ 0x2c5ce86
>>>> >> >
>>>> >> >
>>>> >> > apache::hive::service::cli::thrift::TCLIServiceProcessor::process_CloseOperation()
>>>> >> > @ 0x2c56b8c
>>>> >> >
>>>> >> > apache::hive::service::cli::thrift::TCLIServiceProcessor::dispatchCall()
>>>> >> > @ 0x2c2fcb1
>>>> >> > impala::ImpalaHiveServer2ServiceProcessor::dispatchCall()
>>>> >> > @ 0x16fdb20
>>>> >> > apache::thrift::TDispatchProcessor::process()
>>>> >> > @ 0x18ea6b3
>>>> >> > apache::thrift::server::TAcceptQueueServer::Task::run()
>>>> >> > @ 0x18e2181 impala::ThriftThread::RunRunnable()
>>>> >> > @ 0x18e3885 boost::_mfi::mf2<>::operator()()
>>>> >> > @ 0x18e371b boost::_bi::list3<>::operator()<>()
>>>> >> > @ 0x18e3467 boost::_bi::bind_t<>::operator()()
>>>> >> > @ 0x18e337a
>>>> >> > boost::detail::function::void_function_obj_invoker0<>::invoke()
>>>> >> > @ 0x192761c boost::function0<>::operator()()
>>>> >> > @ 0x1c3ebf7 impala::Thread::SuperviseThread()
>>>> >> > @ 0x1c470cd boost::_bi::list5<>::operator()<>()
>>>> >> > @ 0x1c46ff1 boost::_bi::bind_t<>::operator()()
>>>> >> > @ 0x1c46fb4 boost::detail::thread_data<>::run()
>>>> >> > @ 0x2eedb4a thread_proxy
>>>> >> > @ 0x7fda1dbb16ba start_thread
>>>> >> > @ 0x7fda1d8e741d clone
>>>> >> > Wrote minidump to
>>>> >> > /tmp/minidumps/impalad/a9113d9b-bc3d-488a-1feebf9b-47b42022.dmp
>>>> >> >
>>>> >> > impalad.FATAL
>>>> >> >
>>>> >> > Log file created at: 2018/05/07 09:46:12
>>>> >> > Running on machine: slave2
>>>> >> > Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
>>>> >> > F0507 09:46:12.673912 29258 error-util.cc:148] Check failed:
>>>> >> > log_entry.count
>>>> >> >> 0 (-1831809966 vs. 0)
>>>> >> >
>>>> >> > Impalad.INFO
>>>> >> > edentials={real_user=root}} blocked reactor thread for 34288.6us
>>>> >> > I0507 09:38:14.943245 29882 outbound_call.cc:288] RPC callback for
>>>> >> > RPC
>>>> >> > call
>>>> >> > kudu.tserver.TabletServerService.Write ->
>>>> >> > {remote=136.243.74.42:7050
>>>> >> > (slave5), user_credentials={real_user=root}} blocked reactor thread
>>>> >> > for
>>>> >> > 35859.8us
>>>> >> > I0507 09:38:15.942150 29882 outbound_call.cc:288] RPC callback for
>>>> >> > RPC
>>>> >> > call
>>>> >> > kudu.tserver.TabletServerService.Write ->
>>>> >> > {remote=136.243.74.42:7050
>>>> >> > (slave5), user_credentials={real_user=root}} blocked reactor thread
>>>> >> > for
>>>> >> > 40664.9us
>>>> >> > I0507 09:38:17.495046 29882 outbound_call.cc:288] RPC callback for
>>>> >> > RPC
>>>> >> > call
>>>> >> > kudu.tserver.TabletServerService.Write ->
>>>> >> > {remote=136.243.74.42:7050
>>>> >> > (slave5), user_credentials={real_user=root}} blocked reactor thread
>>>> >> > for
>>>> >> > 49514.6us
>>>> >> > I0507 09:46:12.664149 4507 coordinator.cc:783] Release admission
>>>> >> > control
>>>> >> > resources for query_id=3e4a4c646800e1d9:c859bb7f00000000
>>>> >> > F0507 09:46:12.673912 29258 error-util.cc:148] Check failed:
>>>> >> > log_entry.count
>>>> >> >> 0 (-1831809966 vs. 0)
>>>> >> > Wrote minidump to
>>>> >> > /tmp/minidumps/impalad/a9113d9b-bc3d-488a-1feebf9b-47b42022.dmp
>>>> >> >
>>>> >> > Note:
>>>> >> > We are executing the queries on 8 node cluster with the following
>>>> >> > configuration
>>>> >> > Cluster : 8 Node Cluster (48 GB RAM , 8 CPU Core and 2 TB hard-disk
>>>> >> > each,
>>>> >> > Intel(R) Core(TM) i7 CPU 950 @ 3.07GHz
>>>> >> >
>>>> >> >
>>>> >> > --
>>>> >> > Regards,
>>>> >> > Geetika Gupta
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > Regards,
>>>> > Geetika Gupta
>>>
>>>
>>>
>>>
>>> --
>>> Regards,
>>> Geetika Gupta
>>
>>
>>
>>
>> --
>> Regards,
>> Geetika Gupta
>
>
Re: Issue in data loading in Impala + Kudu
Posted by Mostafa Mokhtar <mm...@cloudera.com>.
Can you try rerunning the query again against the Kudu database instead of
default?
select count(*) from kudu_impala_500.LINEITEM;
On Wed, May 9, 2018 at 10:13 PM, Geetika Gupta <ge...@knoldus.in>
wrote:
> Please find below the query profile :
>
> Query (id=9649f7ab3bcc5fb8:f4d6a60700000000):
> Summary:
> Session ID: b04a9080d1e1724d:41f2b0d261e8f280
> Session Type: HIVESERVER2
> HiveServer2 Protocol Version: V6
> Start Time: 2018-05-08 17:55:23.181264000
> End Time: 2018-05-10 00:34:17.784273000
> Query Type: DML
> Query State: FINISHED
> Query Status: OK
> Impala Version: impalad version 3.0.0-SNAPSHOT RELEASE (build b68e06997c1f49f6b723d78e217efddec4f56f3a)
> User: root
> Connected User: root
> Delegated User:
> Network Address: ::ffff:46.4.88.233:59862
> Default Db: kudu_impala_500
> Sql Statement: insert into LINEITEM select * from PARQUETIMPALA500.LINEITEM
> Coordinator: slave2:22000
> Query Options (set by configuration):
> Query Options (set by configuration and planner): MT_DOP=0
> Plan:
> ----------------
> Max Per-Host Resource Reservation: Memory=0B
> Per-Host Resource Estimates: Memory=704.00MB
> WARNING: The following tables are missing relevant table and/or column statistics.
> parquetimpala500.lineitem
>
> F00:PLAN FRAGMENT [RANDOM] hosts=7 instances=7
> | Per-Host Resources: mem-estimate=704.00MB mem-reservation=0B
> INSERT INTO KUDU [kudu_impala_500.lineitem]
> | mem-estimate=0B mem-reservation=0B
> |
> 00:SCAN HDFS [parquetimpala500.lineitem, RANDOM]
> partitions=1/1 files=396 size=97.29GB
> stored statistics:
> table: rows=unavailable size=unavailable
> columns: unavailable
> extrapolated-rows=disabled
> mem-estimate=704.00MB mem-reservation=0B
> tuple-ids=0 row-size=171B cardinality=unavailable
> ----------------
> Estimated Per-Host Mem: 738197504
> Tables Missing Stats: parquetimpala500.lineitem
> Per Host Min Reservation: slave1:22000(0) slave2:22000(0) slave3:22000(0) slave4:22000(0) slave5:22000(0) slave6:22000(0) slave7:22000(0)
> Request Pool: default-pool
> Admission result: Admitted immediately
> ExecSummary:
> Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
> -----------------------------------------------------------------------------------------------------------------
> 00:SCAN HDFS 7 4s417ms 6s154ms 3.00B -1 1.39 GB 704.00 MB parquetimpala500.lineitem
> Errors: Key already present in Kudu table 'impala::kudu_impala_500.LINEITEM'. (1 of -1831809966 similar)
>
> Query Compilation: 6s413ms
> - Metadata load started: 14.443ms (14.443ms)
> - Metadata load finished. loaded-tables=2/2 load-requests=1 catalog-updates=7: 6s298ms (6s283ms)
> - Analysis finished: 6s301ms (3.659ms)
> - Value transfer graph computed: 6s302ms (282.554us)
> - Single node plan created: 6s363ms (61.624ms)
> - Runtime filters computed: 6s363ms (98.878us)
> - Distributed plan created: 6s366ms (2.832ms)
> - Planning finished: 6s413ms (46.751ms)
> Query Timeline: 30h38m
> - Query submitted: 56.829us (56.829us)
> - Planning finished: 6s431ms (6s431ms)
> - Submit for admission: 6s432ms (821.867us)
> - Completed admission: 6s432ms (14.519us)
> - Ready to start on 7 backends: 6s432ms (99.515us)
> - All 7 execution backends (7 fragment instances) started: 6s535ms (103.320ms)
> - Released admission control resources: 30h38m (30h38m)
> - DML data written: 30h38m (934.739us)
> - DML Metastore update finished: 30h38m (157.938us)
> - Request finished: 30h38m (41.379us)
> - First row fetched: 30h38m (173.124us)
> - First row fetched: 30h38m (1.750ms)
> - First row fetched: 30h38m (1.361ms)
> - Unregister query: 30h38m (1.456ms)
> - ComputeScanRangeAssignmentTimer: 558.440us
> ImpalaServer:
> - ClientFetchWaitTimer: 4.721ms
> - MetastoreUpdateTimer: 191.424us
> - RowMaterializationTimer: 0.000ns
> Execution Profile 9649f7ab3bcc5fb8:f4d6a60700000000:(Total: 30h38m, non-child: 0.000ns, % non-child: 0.00%)
> Number of filters: 0
> Filter routing table:
> ID Src. Node Tgt. Node(s) Target type Partition filter Pending (Expected) First arrived Completed Enabled
> -------------------------------------------------------------------------------------------------------------------
>
> Backend startup latencies: Count: 7, min / max: 1ms / 103ms, 25th %-ile: 1ms, 50th %-ile: 67ms, 75th %-ile: 67ms, 90th %-ile: 80ms, 95th %-ile: 103ms, 99.9th %-ile: 103ms
> DML Stats:
> Partition: Default
> NumModifiedRows: 536870912
> NumRowErrors: 2463157330
> BytesWritten: 0
> Per Node Peak Memory Usage: slave1:22000(1.40 GB) slave2:22000(1.41 GB) slave3:22000(1.39 GB) slave4:22000(1.41 GB) slave5:22000(1.41 GB) slave6:22000(1.40 GB) slave7:22000(1.40 GB)
> - FiltersReceived: 0 (0)
> - FinalizationTimer: 0.000ns
> - NumBackends: 7 (7)
> - NumFragmentInstances: 7 (7)
> - NumFragments: 1 (1)
> Averaged Fragment F00:(Total: 29h51m, non-child: 14s973ms, % non-child: 0.01%)
> split sizes: min: 13.63 GB, max: 14.10 GB, avg: 13.90 GB, stddev: 143.22 MB
> completion times: min:28h47m max:30h38m mean: 29h51m stddev:41m26s
> execution rates: min:129.50 KB/sec max:140.31 KB/sec mean:135.67 KB/sec stddev:3.50 KB/sec
> num instances: 7
> - AverageThreadTokens: 8.89
> - BloomFilterBytes: 0
> - PeakMemoryUsage: 1.40 GB (1503884428)
> - PeakReservation: 0
> - PeakUsedReservation: 0
> - PerHostPeakMemUsage: 1.40 GB (1503884428)
> - RowsProduced: 428.58M (428575463)
> - TotalNetworkReceiveTime: 0.000ns
> - TotalNetworkSendTime: 0.000ns
> - TotalStorageWaitTime: 2m10s
> - TotalThreadsInvoluntaryContextSwitches: 369.81K (369812)
> - TotalThreadsTotalWallClockTime: 265h20m
> - TotalThreadsSysTime: 1m59s
> - TotalThreadsUserTime: 1h9m
> - TotalThreadsVoluntaryContextSwitches: 433.92K (433917)
> Buffer pool:
> - AllocTime: 0.000ns
> - CumulativeAllocationBytes: 0
> - CumulativeAllocations: 0 (0)
> - PeakReservation: 0
> - PeakUnpinnedBytes: 0
> - PeakUsedReservation: 0
> - ReadIoBytes: 0
> - ReadIoOps: 0 (0)
> - ReadIoWaitTime: 0.000ns
> - ReservationLimit: 0
> - WriteIoBytes: 0
> - WriteIoOps: 0 (0)
> - WriteIoWaitTime: 0.000ns
> Fragment Instance Lifecycle Timings:
> - ExecTime: 29h51m
> - ExecTreeExecTime: 4s724ms
> - OpenTime: 519.095ms
> - ExecTreeOpenTime: 36.350us
> - PrepareTime: 60.445ms
> - ExecTreePrepareTime: 19.494ms
> KuduTableSink:(Total: 29h50m, non-child: 29h50m, % non-child: 100.00%)
> - KuduApplyTimer: 28h54m
> - NumRowErrors: 351.88M (351879618)
> - PeakMemoryUsage: 20.00 MB (20971520)
> - RowsProcessedRate: 3.99 K/sec
> - TotalNumRows: 428.58M (428575463)
> HDFS_SCAN_NODE (id=0):(Total: 4s417ms, non-child: 4s417ms, % non-child: 100.00%)
> - AverageHdfsReadThreadConcurrency: 0.00
> - AverageScannerThreadConcurrency: 7.89
> - BytesRead: 13.90 GB (14928723218)
> - BytesReadDataNodeCache: 0
> - BytesReadLocal: 13.90 GB (14928723218)
> - BytesReadRemoteUnexpected: 0
> - BytesReadShortCircuit: 0
> - CachedFileHandlesHitCount: 2.31K (2305)
> - CachedFileHandlesMissCount: 56 (56)
> - CollectionItemsRead: 0 (0)
> - DecompressionTime: 22s321ms
> - MaxCompressedTextFileLength: 0
> - NumColumns: 16 (16)
> - NumDictFilteredRowGroups: 0 (0)
> - NumDisksAccessed: 1 (1)
> - NumRowGroups: 56 (56)
> - NumScannerThreadsStarted: 8 (8)
> - NumScannersWithNoReads: 0 (0)
> - NumStatsFilteredRowGroups: 0 (0)
> - PeakMemoryUsage: 1.38 GB (1482717084)
> - PerReadThreadRawHdfsThroughput: 69.65 MB/sec
> - RemoteScanRanges: 0 (0)
> - RowBatchQueueGetWaitTime: 1s754ms
> - RowBatchQueuePutWaitTime: 235h24m
> - RowsRead: 428.58M (428575463)
> - RowsReturned: 428.58M (428575463)
> - RowsReturnedRate: 99.08 M/sec
> - ScanRangesComplete: 56 (56)
> - ScannerThreadsInvoluntaryContextSwitches: 12.33K (12332)
> - ScannerThreadsTotalWallClockTime: 235h28m
> - MaterializeTupleTime(*): 2m3s
> - ScannerThreadsSysTime: 3s896ms
> - ScannerThreadsUserTime: 2m39s
> - ScannerThreadsVoluntaryContextSwitches: 419.69K (419690)
> - TotalRawHdfsOpenFileTime(*): 3s995ms
> - TotalRawHdfsReadTime(*): 4m2s
> - TotalReadThroughput: 135.78 KB/sec
> CodeGen:(Total: 436.431ms, non-child: 436.431ms, % non-child: 100.00%)
> - CodegenInvoluntaryContextSwitches: 5 (5)
> - CodegenTotalWallClockTime: 436.422ms
> - CodegenSysTime: 8.585ms
> - CodegenUserTime: 41.806ms
> - CodegenVoluntaryContextSwitches: 76 (76)
> - CompileTime: 179.278ms
> - IrGenerationTime: 90.176ms
> - LoadTime: 0.000ns
> - ModuleBitcodeSize: 2.26 MB (2373832)
> - NumFunctions: 19 (19)
> - NumInstructions: 706 (706)
> - OptimizationTime: 151.857ms
> - PeakMemoryUsage: 353.00 KB (361472)
> - PrepareTime: 14.499ms
> Fragment F00:
> Instance 9649f7ab3bcc5fb8:f4d6a60700000000 (host=slave3:22000):(Total: 30h38m, non-child: 11s364ms, % non-child: 0.01%)
> Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:56/13.63 GB
> Fragment Instance Lifecycle Event Timeline: 30h38m
> - Prepare Finished: 86.126ms (86.126ms)
> - Open Finished: 1s157ms (1s070ms)
> - First Batch Produced: 3s031ms (1s874ms)
> - First Batch Sent: 3s037ms (5.724ms)
> - ExecInternal Finished: 30h38m (30h38m)
> MemoryUsage(34m8s): 1.28 GB, 1.34 GB, 1.37 GB, 1.35 GB, 1.21 GB, 925.43 MB, 727.39 MB, 1.17 GB, 1.35 GB, 1.35 GB, 1.35 GB, 1.35 GB, 1.26 GB, 1.06 GB, 836.27 MB, 792.34 MB, 1.26 GB, 1.36 GB, 1.36 GB, 1.30 GB, 1.10 GB, 827.96 MB, 1.01 GB, 1.38 GB, 1.38 GB, 1.32 GB, 1.12 GB, 875.56 MB, 979.49 MB, 1.34 GB, 1.37 GB, 1.37 GB, 1.28 GB, 1.10 GB, 950.36 MB, 805.45 MB, 963.12 MB, 1.28 GB, 1.34 GB, 1.34 GB, 1.30 GB, 1.04 GB, 808.54 MB, 994.36 MB, 1.17 GB, 1.18 GB, 1.18 GB, 1.18 GB, 1.17 GB, 1.13 GB, 1.04 GB, 954.52 MB, 811.51 MB
> ThreadUsage(34m8s): 8, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 8, 8, 8, 8, 8, 8, 8, 8, 8
> - AverageThreadTokens: 8.80
> - BloomFilterBytes: 0
> - PeakMemoryUsage: 1.39 GB (1487275684)
> - PeakReservation: 0
> - PeakUsedReservation: 0
> - PerHostPeakMemUsage: 1.39 GB (1487275684)
> - RowsProduced: 424.29M (424287956)
> - TotalNetworkReceiveTime: 0.000ns
> - TotalNetworkSendTime: 0.000ns
> - TotalStorageWaitTime: 1m32s
> - TotalThreadsInvoluntaryContextSwitches: 349.35K (349347)
> - TotalThreadsTotalWallClockTime: 269h37m
> - TotalThreadsSysTime: 1m54s
> - TotalThreadsUserTime: 1h6m
> - TotalThreadsVoluntaryContextSwitches: 438.17K (438165)
> Buffer pool:
> - AllocTime: 0.000ns
> - CumulativeAllocationBytes: 0
> - CumulativeAllocations: 0 (0)
> - PeakReservation: 0
> - PeakUnpinnedBytes: 0
> - PeakUsedReservation: 0
> - ReadIoBytes: 0
> - ReadIoOps: 0 (0)
> - ReadIoWaitTime: 0.000ns
> - ReservationLimit: 0
> - WriteIoBytes: 0
> - WriteIoOps: 0 (0)
> - WriteIoWaitTime: 0.000ns
> Fragment Instance Lifecycle Timings:
> - ExecTime: 30h38m
> - ExecTreeExecTime: 4s586ms
> - OpenTime: 1s070ms
> - ExecTreeOpenTime: 32.390us
> - PrepareTime: 85.903ms
> - ExecTreePrepareTime: 16.283ms
> KuduTableSink:(Total: 30h38m, non-child: 30h38m, % non-child: 100.00%)
> - KuduApplyTimer: 29h44m
> - NumRowErrors: 371.48M (371483541)
> - PeakMemoryUsage: 20.00 MB (20971520)
> - RowsProcessedRate: 3.85 K/sec
> - TotalNumRows: 424.29M (424287956)
> HDFS_SCAN_NODE (id=0):(Total: 4s304ms, non-child: 4s304ms, % non-child: 100.00%)
> Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:56/13.63 GB
> ExecOption: PARQUET Codegen Enabled, Codegen enabled: 56 out of 56
> Hdfs Read Thread Concurrency Bucket: 0:99.87% 1:0.1279% 2:0% 3:0% 4:0% 5:0% 6:0%
> File Formats: PARQUET/SNAPPY:56
> BytesRead(34m8s): 1.19 GB, 1.46 GB, 1.65 GB, 1.78 GB, 1.88 GB, 1.97 GB, 2.01 GB, 2.91 GB, 3.27 GB, 3.46 GB, 3.61 GB, 3.72 GB, 3.80 GB, 3.90 GB, 3.96 GB, 4.16 GB, 5.05 GB, 5.36 GB, 5.60 GB, 5.76 GB, 5.87 GB, 5.94 GB, 6.57 GB, 7.32 GB, 7.57 GB, 7.75 GB, 7.86 GB, 7.92 GB, 8.43 GB, 9.18 GB, 9.43 GB, 9.62 GB, 9.75 GB, 9.84 GB, 9.89 GB, 9.94 GB, 10.41 GB, 11.07 GB, 11.24 GB, 11.42 GB, 11.66 GB, 11.82 GB, 12.05 GB, 12.48 GB, 12.96 GB, 13.07 GB, 13.19 GB, 13.31 GB, 13.38 GB, 13.46 GB, 13.51 GB, 13.57 GB, 13.61 GB
> - FooterProcessingTime: (Avg: 381.833ms ; Min: 46.831ms ; Max: 4s214ms ; Number of samples: 56)
> - AverageHdfsReadThreadConcurrency: 0.00
> - AverageScannerThreadConcurrency: 7.80
> - BytesRead: 13.63 GB (14635606518)
> - BytesReadDataNodeCache: 0
> - BytesReadLocal: 13.63 GB (14635606518)
> - BytesReadRemoteUnexpected: 0
> - BytesReadShortCircuit: 0
> - CachedFileHandlesHitCount: 2.27K (2268)
> - CachedFileHandlesMissCount: 56 (56)
> - CollectionItemsRead: 0 (0)
> - DecompressionTime: 19s612ms
> - MaxCompressedTextFileLength: 0
> - NumColumns: 16 (16)
> - NumDictFilteredRowGroups: 0 (0)
> - NumDisksAccessed: 1 (1)
> - NumRowGroups: 56 (56)
> - NumScannerThreadsStarted: 8 (8)
> - NumScannersWithNoReads: 0 (0)
> - NumStatsFilteredRowGroups: 0 (0)
> - PeakMemoryUsage: 1.37 GB (1466047697)
> - PerReadThreadRawHdfsThroughput: 100.45 MB/sec
> - RemoteScanRanges: 0 (0)
> - RowBatchQueueGetWaitTime: 1s872ms
> - RowBatchQueuePutWaitTime: 238h55m
> - RowsRead: 424.29M (424287956)
> - RowsReturned: 424.29M (424287956)
> - RowsReturnedRate: 98.57 M/sec
> - ScanRangesComplete: 56 (56)
> - ScannerThreadsInvoluntaryContextSwitches: 12.67K (12674)
> - ScannerThreadsTotalWallClockTime: 238h58m
> - MaterializeTupleTime(*): 1m57s
> - ScannerThreadsSysTime: 3s421ms
> - ScannerThreadsUserTime: 2m31s
> - ScannerThreadsVoluntaryContextSwitches: 415.49K (415490)
> - TotalRawHdfsOpenFileTime(*): 2s904ms
> - TotalRawHdfsReadTime(*): 2m18s
> - TotalReadThroughput: 129.60 KB/sec
> CodeGen:(Total: 915.511ms, non-child: 915.511ms, % non-child: 100.00%)
> - CodegenInvoluntaryContextSwitches: 10 (10)
> - CodegenTotalWallClockTime: 915.504ms
> - CodegenSysTime: 8.256ms
> - CodegenUserTime: 45.032ms
> - CodegenVoluntaryContextSwitches: 87 (87)
> - CompileTime: 319.788ms
> - IrGenerationTime: 276.446ms
> - LoadTime: 0.000ns
> - ModuleBitcodeSize: 2.26 MB (2373832)
> - NumFunctions: 19 (19)
> - NumInstructions: 706 (706)
> - OptimizationTime: 301.622ms
> - PeakMemoryUsage: 353.00 KB (361472)
> - PrepareTime: 16.858ms
> Instance 9649f7ab3bcc5fb8:f4d6a60700000002 (host=slave1:22000):(Total: 30h17m, non-child: 1s508ms, % non-child: 0.00%)
> Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:57/14.10 GB
> Fragment Instance Lifecycle Event Timeline: 30h17m
> - Prepare Finished: 32.853ms (32.853ms)
> - Open Finished: 362.691ms (329.838ms)
> - First Batch Produced: 1s725ms (1s362ms)
> - First Batch Sent: 1s728ms (3.415ms)
> - ExecInternal Finished: 30h17m (30h17m)
> MemoryUsage(34m8s): 1.39 GB, 1.40 GB, 1.40 GB, 1.27 GB, 1.07 GB, 798.75 MB, 1.36 GB, 1.38 GB, 1.38 GB, 1.38 GB, 1.32 GB, 1.19 GB, 974.24 MB, 800.43 MB, 1.27 GB, 1.38 GB, 1.38 GB, 1.38 GB, 1.26 GB, 1010.99 MB, 805.20 MB, 1.29 GB, 1.38 GB, 1.38 GB, 1.27 GB, 987.74 MB, 811.33 MB, 1.27 GB, 1.38 GB, 1.38 GB, 1.34 GB, 1.18 GB, 922.96 MB, 765.77 MB, 1.13 GB, 1.36 GB, 1.36 GB, 1.36 GB, 1.37 GB, 1.30 GB, 1.03 GB, 919.88 MB, 1.38 GB, 1.38 GB, 1.38 GB, 1.39 GB, 1.35 GB, 1.24 GB, 1.17 GB, 1005.37 MB, 844.55 MB, 751.70 MB, 518.60 MB
> ThreadUsage(34m8s): 8, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 5
> - AverageThreadTokens: 8.91
> - BloomFilterBytes: 0
> - PeakMemoryUsage: 1.40 GB (1500404835)
> - PeakReservation: 0
> - PeakUsedReservation: 0
> - PerHostPeakMemUsage: 1.40 GB (1500404835)
> - RowsProduced: 434.96M (434955041)
> - TotalNetworkReceiveTime: 0.000ns
> - TotalNetworkSendTime: 0.000ns
> - TotalStorageWaitTime: 2m56s
> - TotalThreadsInvoluntaryContextSwitches: 370.59K (370587)
> - TotalThreadsTotalWallClockTime: 269h58m
> - TotalThreadsSysTime: 1m58s
> - TotalThreadsUserTime: 1h9m
> - TotalThreadsVoluntaryContextSwitches: 437.63K (437630)
> Buffer pool:
> - AllocTime: 0.000ns
> - CumulativeAllocationBytes: 0
> - CumulativeAllocations: 0 (0)
> - PeakReservation: 0
> - PeakUnpinnedBytes: 0
> - PeakUsedReservation: 0
> - ReadIoBytes: 0
> - ReadIoOps: 0 (0)
> - ReadIoWaitTime: 0.000ns
> - ReservationLimit: 0
> - WriteIoBytes: 0
> - WriteIoOps: 0 (0)
> - WriteIoWaitTime: 0.000ns
> Fragment Instance Lifecycle Timings:
> - ExecTime: 30h17m
> - ExecTreeExecTime: 4s575ms
> - OpenTime: 329.834ms
> - ExecTreeOpenTime: 45.986us
> - PrepareTime: 32.581ms
> - ExecTreePrepareTime: 18.647ms
> KuduTableSink:(Total: 30h17m, non-child: 30h17m, % non-child: 100.00%)
> - KuduApplyTimer: 29h22m
> - NumRowErrors: 357.86M (357862755)
> - PeakMemoryUsage: 20.00 MB (20971520)
> - RowsProcessedRate: 3.99 K/sec
> - TotalNumRows: 434.96M (434955041)
> HDFS_SCAN_NODE (id=0):(Total: 4s229ms, non-child: 4s229ms, % non-child: 100.00%)
> Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:57/14.10 GB
> ExecOption: PARQUET Codegen Enabled, Codegen enabled: 57 out of 57
> Hdfs Read Thread Concurrency Bucket: 0:99.63% 1:0.3706% 2:0% 3:0% 4:0% 5:0% 6:0%
> File Formats: PARQUET/SNAPPY:57
> BytesRead(34m8s): 1.32 GB, 1.59 GB, 1.74 GB, 1.85 GB, 1.94 GB, 2.11 GB, 3.21 GB, 3.31 GB, 3.48 GB, 3.67 GB, 3.79 GB, 3.88 GB, 3.94 GB, 4.11 GB, 5.04 GB, 5.32 GB, 5.50 GB, 5.67 GB, 5.81 GB, 5.92 GB, 6.09 GB, 7.06 GB, 7.40 GB, 7.63 GB, 7.79 GB, 7.90 GB, 8.08 GB, 9.02 GB, 9.35 GB, 9.55 GB, 9.70 GB, 9.81 GB, 9.90 GB, 10.01 GB, 10.74 GB, 11.19 GB, 11.34 GB, 11.48 GB, 11.57 GB, 11.69 GB, 11.84 GB, 12.25 GB, 13.25 GB, 13.41 GB, 13.50 GB, 13.60 GB, 13.67 GB, 13.73 GB, 13.79 GB, 13.83 GB, 13.86 GB, 13.92 GB, 14.03 GB
> - FooterProcessingTime: (Avg: 414.198ms ; Min: 51.291ms ; Max: 2s894ms ; Number of samples: 57)
> - AverageHdfsReadThreadConcurrency: 0.00
> - AverageScannerThreadConcurrency: 7.91
> - BytesRead: 14.11 GB (15148002597)
> - BytesReadDataNodeCache: 0
> - BytesReadLocal: 14.11 GB (15148002597)
> - BytesReadRemoteUnexpected: 0
> - BytesReadShortCircuit: 0
> - CachedFileHandlesHitCount: 2.33K (2333)
> - CachedFileHandlesMissCount: 57 (57)
> - CollectionItemsRead: 0 (0)
> - DecompressionTime: 23s330ms
> - MaxCompressedTextFileLength: 0
> - NumColumns: 16 (16)
> - NumDictFilteredRowGroups: 0 (0)
> - NumDisksAccessed: 1 (1)
> - NumRowGroups: 57 (57)
> - NumScannerThreadsStarted: 8 (8)
> - NumScannersWithNoReads: 0 (0)
> - NumStatsFilteredRowGroups: 0 (0)
> - PeakMemoryUsage: 1.38 GB (1479367012)
> - PerReadThreadRawHdfsThroughput: 36.38 MB/sec
> - RemoteScanRanges: 0 (0)
> - RowBatchQueueGetWaitTime: 1s354ms
> - RowBatchQueuePutWaitTime: 239h35m
> - RowsRead: 434.96M (434955041)
> - RowsReturned: 434.96M (434955041)
> - RowsReturnedRate: 102.83 M/sec
> - ScanRangesComplete: 57 (57)
> - ScannerThreadsInvoluntaryContextSwitches: 7.23K (7231)
> - ScannerThreadsTotalWallClockTime: 239h40m
> - MaterializeTupleTime(*): 2m1s
> - ScannerThreadsSysTime: 4s044ms
> - ScannerThreadsUserTime: 2m42s
> - ScannerThreadsVoluntaryContextSwitches: 425.94K (425935)
> - TotalRawHdfsOpenFileTime(*): 5s555ms
> - TotalRawHdfsReadTime(*): 6m37s
> - TotalReadThroughput: 135.68 KB/sec
> CodeGen:(Total: 274.452ms, non-child: 274.452ms, % non-child: 100.00%)
> - CodegenInvoluntaryContextSwitches: 4 (4)
> - CodegenTotalWallClockTime: 274.443ms
> - CodegenSysTime: 7.440ms
> - CodegenUserTime: 40.295ms
> - CodegenVoluntaryContextSwitches: 86 (86)
> - CompileTime: 103.404ms
> - IrGenerationTime: 32.648ms
> - LoadTime: 0.000ns
> - ModuleBitcodeSize: 2.26 MB (2373832)
> - NumFunctions: 19 (19)
> - NumInstructions: 706 (706)
> - OptimizationTime: 122.033ms
> - PeakMemoryUsage: 353.00 KB (361472)
> - PrepareTime: 15.943ms
> Instance 9649f7ab3bcc5fb8:f4d6a60700000001 (host=slave4:22000):(Total: 30h16m, non-child: 2s019ms, % non-child: 0.00%)
> Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:56/13.87 GB
> Fragment Instance Lifecycle Event Timeline: 30h16m
> - Prepare Finished: 105.379ms (105.379ms)
> - Open Finished: 519.128ms (413.748ms)
> - First Batch Produced: 1s789ms (1s270ms)
> - First Batch Sent: 1s795ms (6.364ms)
> - ExecInternal Finished: 30h16m (30h16m)
> MemoryUsage(34m8s): 1.39 GB, 1.40 GB, 1.40 GB, 1.38 GB, 1.23 GB, 924.30 MB, 715.63 MB, 1.26 GB, 1.40 GB, 1.40 GB, 1.40 GB, 1.40 GB, 1.30 GB, 1.08 GB, 843.90 MB, 887.06 MB, 1.40 GB, 1.41 GB, 1.39 GB, 1.19 GB, 826.21 MB, 1.19 GB, 1.38 GB, 1.38 GB, 1.31 GB, 1.05 GB, 748.49 MB, 1.17 GB, 1.38 GB, 1.38 GB, 1.38 GB, 1.24 GB, 999.76 MB, 733.27 MB, 995.28 MB, 1.35 GB, 1.38 GB, 1.38 GB, 1.38 GB, 1.35 GB, 1.10 GB, 861.01 MB, 1.39 GB, 1.40 GB, 1.40 GB, 1.40 GB, 1.39 GB, 1.33 GB, 1.24 GB, 1.13 GB, 1007.59 MB, 820.51 MB, 655.33 MB
> ThreadUsage(34m8s): 8, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 8
> - AverageThreadTokens: 8.97
> - BloomFilterBytes: 0
> - PeakMemoryUsage: 1.41 GB (1513166691)
> - PeakReservation: 0
> - PeakUsedReservation: 0
> - PerHostPeakMemUsage: 1.41 GB (1513166691)
> - RowsProduced: 425.55M (425550934)
> - TotalNetworkReceiveTime: 0.000ns
> - TotalNetworkSendTime: 0.000ns
> - TotalStorageWaitTime: 1m49s
> - TotalThreadsInvoluntaryContextSwitches: 364.52K (364523)
> - TotalThreadsTotalWallClockTime: 271h36m
> - TotalThreadsSysTime: 1m56s
> - TotalThreadsUserTime: 1h9m
> - TotalThreadsVoluntaryContextSwitches: 431.64K (431643)
> Buffer pool:
> - AllocTime: 0.000ns
> - CumulativeAllocationBytes: 0
> - CumulativeAllocations: 0 (0)
> - PeakReservation: 0
> - PeakUnpinnedBytes: 0
> - PeakUsedReservation: 0
> - ReadIoBytes: 0
> - ReadIoOps: 0 (0)
> - ReadIoWaitTime: 0.000ns
> - ReservationLimit: 0
> - WriteIoBytes: 0
> - WriteIoOps: 0 (0)
> - WriteIoWaitTime: 0.000ns
> Fragment Instance Lifecycle Timings:
> - ExecTime: 30h16m
> - ExecTreeExecTime: 4s185ms
> - OpenTime: 413.744ms
> - ExecTreeOpenTime: 31.615us
> - PrepareTime: 105.138ms
> - ExecTreePrepareTime: 17.849ms
> KuduTableSink:(Total: 30h16m, non-child: 30h16m, % non-child: 100.00%)
> - KuduApplyTimer: 29h20m
> - NumRowErrors: 336.92M (336918666)
> - PeakMemoryUsage: 20.00 MB (20971520)
> - RowsProcessedRate: 3.90 K/sec
> - TotalNumRows: 425.55M (425550934)
> HDFS_SCAN_NODE (id=0):(Total: 3s886ms, non-child: 3s886ms, % non-child: 100.00%)
> Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:56/13.87 GB
> ExecOption: PARQUET Codegen Enabled, Codegen enabled: 56 out of 56
> Hdfs Read Thread Concurrency Bucket: 0:99.86% 1:0.1368% 2:0% 3:0% 4:0% 5:0% 6:0%
> File Formats: PARQUET/SNAPPY:56
> BytesRead(34m8s): 1.29 GB, 1.51 GB, 1.66 GB, 1.79 GB, 1.88 GB, 1.97 GB, 2.00 GB, 3.01 GB, 3.30 GB, 3.49 GB, 3.63 GB, 3.74 GB, 3.82 GB, 3.92 GB, 3.96 GB, 4.29 GB, 5.26 GB, 5.47 GB, 5.71 GB, 5.87 GB, 5.94 GB, 6.88 GB, 7.34 GB, 7.59 GB, 7.76 GB, 7.89 GB, 7.93 GB, 8.83 GB, 9.30 GB, 9.51 GB, 9.67 GB, 9.79 GB, 9.89 GB, 9.91 GB, 10.45 GB, 11.14 GB, 11.30 GB, 11.46 GB, 11.55 GB, 11.68 GB, 11.83 GB, 12.10 GB, 13.21 GB, 13.35 GB, 13.47 GB, 13.56 GB, 13.64 GB, 13.71 GB, 13.76 GB, 13.81 GB, 13.86 GB, 13.87 GB, 13.88 GB
> - FooterProcessingTime: (Avg: 331.439ms ; Min: 46.760ms ; Max: 2s508ms ; Number of samples: 56)
> - AverageHdfsReadThreadConcurrency: 0.00
> - AverageScannerThreadConcurrency: 7.97
> - BytesRead: 13.88 GB (14898393824)
> - BytesReadDataNodeCache: 0
> - BytesReadLocal: 13.88 GB (14898393824)
> - BytesReadRemoteUnexpected: 0
> - BytesReadShortCircuit: 0
> - CachedFileHandlesHitCount: 2.30K (2296)
> - CachedFileHandlesMissCount: 56 (56)
> - CollectionItemsRead: 0 (0)
> - DecompressionTime: 22s686ms
> - MaxCompressedTextFileLength: 0
> - NumColumns: 16 (16)
> - NumDictFilteredRowGroups: 0 (0)
> - NumDisksAccessed: 1 (1)
> - NumRowGroups: 56 (56)
> - NumScannerThreadsStarted: 8 (8)
> - NumScannersWithNoReads: 0 (0)
> - NumStatsFilteredRowGroups: 0 (0)
> - PeakMemoryUsage: 1.39 GB (1491938707)
> - PerReadThreadRawHdfsThroughput: 95.20 MB/sec
> - RemoteScanRanges: 0 (0)
> - RowBatchQueueGetWaitTime: 1s255ms
> - RowBatchQueuePutWaitTime: 241h15m
> - RowsRead: 425.55M (425550934)
> - RowsReturned: 425.55M (425550934)
> - RowsReturnedRate: 109.50 M/sec
> - ScanRangesComplete: 56 (56)
> - ScannerThreadsInvoluntaryContextSwitches: 12.02K (12019)
> - ScannerThreadsTotalWallClockTime: 241h20m
> - MaterializeTupleTime(*): 2m6s
> - ScannerThreadsSysTime: 3s548ms
> - ScannerThreadsUserTime: 2m39s
> - ScannerThreadsVoluntaryContextSwitches: 416.75K (416750)
> - TotalRawHdfsOpenFileTime(*): 3s230ms
> - TotalRawHdfsReadTime(*): 2m29s
> - TotalReadThroughput: 133.55 KB/sec
> CodeGen:(Total: 315.226ms, non-child: 315.226ms, % non-child: 100.00%)
> - CodegenInvoluntaryContextSwitches: 9 (9)
> - CodegenTotalWallClockTime: 315.217ms
> - CodegenSysTime: 5.142ms
> - CodegenUserTime: 43.310ms
> - CodegenVoluntaryContextSwitches: 91 (91)
> - CompileTime: 129.393ms
> - IrGenerationTime: 60.406ms
> - LoadTime: 0.000ns
> - ModuleBitcodeSize: 2.26 MB (2373832)
> - NumFunctions: 19 (19)
> - NumInstructions: 706 (706)
> - OptimizationTime: 111.534ms
> - PeakMemoryUsage: 353.00 KB (361472)
> - PrepareTime: 13.425ms
> Instance 9649f7ab3bcc5fb8:f4d6a60700000004 (host=slave7:22000):(Total: 30h10m, non-child: 16s427ms, % non-child: 0.02%)
> Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:58/14.03 GB
> Fragment Instance Lifecycle Event Timeline: 30h10m
> - Prepare Finished: 105.361ms (105.361ms)
> - Open Finished: 980.610ms (875.249ms)
> - First Batch Produced: 4s553ms (3s573ms)
> - First Batch Sent: 4s559ms (5.668ms)
> - ExecInternal Finished: 30h10m (30h10m)
> MemoryUsage(34m8s): 1.38 GB, 1.40 GB, 1.40 GB, 1.28 GB, 1.08 GB, 796.55 MB, 1.34 GB, 1.37 GB, 1.35 GB, 1.34 GB, 1.27 GB, 1.21 GB, 983.86 MB, 1.08 GB, 1.38 GB, 1.36 GB, 1.35 GB, 1.28 GB, 1.17 GB, 921.06 MB, 1.02 GB, 1.35 GB, 1.33 GB, 1.28 GB, 1.15 GB, 905.74 MB, 1.01 GB, 1.36 GB, 1.34 GB, 1.32 GB, 1.21 GB, 1.10 GB, 877.34 MB, 1.04 GB, 1.32 GB, 1.38 GB, 1.36 GB, 1.35 GB, 1.32 GB, 1.21 GB, 1000.76 MB, 1.11 GB, 1.37 GB, 1.36 GB, 1.33 GB, 1.29 GB, 1.20 GB, 1.09 GB, 1.07 GB, 937.32 MB, 814.51 MB, 480.87 MB, 155.08 MB
> ThreadUsage(34m8s): 8, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 8, 5, 2
> - AverageThreadTokens: 8.80
> - BloomFilterBytes: 0
> - PeakMemoryUsage: 1.40 GB (1500511831)
> - PeakReservation: 0
> - PeakUsedReservation: 0
> - PerHostPeakMemUsage: 1.40 GB (1500511831)
> - RowsProduced: 433.04M (433043954)
> - TotalNetworkReceiveTime: 0.000ns
> - TotalNetworkSendTime: 0.000ns
> - TotalStorageWaitTime: 3m18s
> - TotalThreadsInvoluntaryContextSwitches: 374.99K (374993)
> - TotalThreadsTotalWallClockTime: 265h25m
> - TotalThreadsSysTime: 1m57s
> - TotalThreadsUserTime: 1h10m
> - TotalThreadsVoluntaryContextSwitches: 435.67K (435665)
> Buffer pool:
> - AllocTime: 0.000ns
> - CumulativeAllocationBytes: 0
> - CumulativeAllocations: 0 (0)
> - PeakReservation: 0
> - PeakUnpinnedBytes: 0
> - PeakUsedReservation: 0
> - ReadIoBytes: 0
> - ReadIoOps: 0 (0)
> - ReadIoWaitTime: 0.000ns
> - ReservationLimit: 0
> - WriteIoBytes: 0
> - WriteIoOps: 0 (0)
> - WriteIoWaitTime: 0.000ns
> Fragment Instance Lifecycle Timings:
> - ExecTime: 30h10m
> - ExecTreeExecTime: 6s426ms
> - OpenTime: 875.240ms
> - ExecTreeOpenTime: 41.994us
> - PrepareTime: 105.102ms
> - ExecTreePrepareTime: 24.800ms
> KuduTableSink:(Total: 30h9m, non-child: 30h9m, % non-child: 100.00%)
> - KuduApplyTimer: 29h12m
> - NumRowErrors: 357.43M (357426783)
> - PeakMemoryUsage: 20.00 MB (20971520)
> - RowsProcessedRate: 3.99 K/sec
> - TotalNumRows: 433.04M (433043954)
> HDFS_SCAN_NODE (id=0):(Total: 6s154ms, non-child: 6s154ms, % non-child: 100.00%)
> Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:58/14.03 GB
> ExecOption: PARQUET Codegen Enabled, Codegen enabled: 58 out of 58
> Hdfs Read Thread Concurrency Bucket: 0:99.68% 1:0.3169% 2:0% 3:0% 4:0% 5:0% 6:0%
> File Formats: PARQUET/SNAPPY:58
> BytesRead(34m8s): 1.31 GB, 1.58 GB, 1.74 GB, 1.85 GB, 1.94 GB, 2.10 GB, 3.18 GB, 3.29 GB, 3.44 GB, 3.62 GB, 3.72 GB, 3.95 GB, 4.03 GB, 4.61 GB, 5.22 GB, 5.37 GB, 5.52 GB, 5.66 GB, 5.90 GB, 6.01 GB, 6.53 GB, 7.22 GB, 7.43 GB, 7.62 GB, 7.83 GB, 7.99 GB, 8.49 GB, 9.18 GB, 9.38 GB, 9.55 GB, 9.67 GB, 9.88 GB, 9.98 GB, 10.53 GB, 11.04 GB, 11.22 GB, 11.39 GB, 11.50 GB, 11.58 GB, 11.70 GB, 11.95 GB, 12.61 GB, 13.30 GB, 13.43 GB, 13.52 GB, 13.59 GB, 13.65 GB, 13.72 GB, 13.91 GB, 13.94 GB, 13.96 GB, 13.98 GB, 14.03 GB
> - FooterProcessingTime: (Avg: 615.194ms ; Min: 50.813ms ; Max: 6s740ms ; Number of samples: 58)
> - AverageHdfsReadThreadConcurrency: 0.00
> - AverageScannerThreadConcurrency: 7.80
> - BytesRead: 14.04 GB (15072987184)
> - BytesReadDataNodeCache: 0
> - BytesReadLocal: 14.04 GB (15072987184)
> - BytesReadRemoteUnexpected: 0
> - BytesReadShortCircuit: 0
> - CachedFileHandlesHitCount: 2.34K (2340)
> - CachedFileHandlesMissCount: 58 (58)
> - CollectionItemsRead: 0 (0)
> - DecompressionTime: 21s821ms
> - MaxCompressedTextFileLength: 0
> - NumColumns: 16 (16)
> - NumDictFilteredRowGroups: 0 (0)
> - NumDisksAccessed: 1 (1)
> - NumRowGroups: 58 (58)
> - NumScannerThreadsStarted: 8 (8)
> - NumScannersWithNoReads: 0 (0)
> - NumStatsFilteredRowGroups: 0 (0)
> - PeakMemoryUsage: 1.38 GB (1479283937)
> - PerReadThreadRawHdfsThroughput: 41.17 MB/sec
> - RemoteScanRanges: 0 (0)
> - RowBatchQueueGetWaitTime: 3s555ms
> - RowBatchQueuePutWaitTime: 235h9m
> - RowsRead: 433.04M (433043954)
> - RowsReturned: 433.04M (433043954)
> - RowsReturnedRate: 70.36 M/sec
> - ScanRangesComplete: 58 (58)
> - ScannerThreadsInvoluntaryContextSwitches: 16.45K (16447)
> - ScannerThreadsTotalWallClockTime: 235h15m
> - MaterializeTupleTime(*): 2m5s
> - ScannerThreadsSysTime: 3s924ms
> - ScannerThreadsUserTime: 2m40s
> - ScannerThreadsVoluntaryContextSwitches: 424.08K (424080)
> - TotalRawHdfsOpenFileTime(*): 4s852ms
> - TotalRawHdfsReadTime(*): 5m49s
> - TotalReadThroughput: 135.58 KB/sec
> CodeGen:(Total: 771.273ms, non-child: 771.273ms, % non-child: 100.00%)
> - CodegenInvoluntaryContextSwitches: 5 (5)
> - CodegenTotalWallClockTime: 771.263ms
> - CodegenSysTime: 19.916ms
> - CodegenUserTime: 41.600ms
> - CodegenVoluntaryContextSwitches: 91 (91)
> - CompileTime: 404.632ms
> - IrGenerationTime: 153.195ms
> - LoadTime: 0.000ns
> - ModuleBitcodeSize: 2.26 MB (2373832)
> - NumFunctions: 19 (19)
> - NumInstructions: 706 (706)
> - OptimizationTime: 199.856ms
> - PeakMemoryUsage: 353.00 KB (361472)
> - PrepareTime: 12.885ms
> Instance 9649f7ab3bcc5fb8:f4d6a60700000006 (host=slave6:22000):(Total: 29h59m, non-child: 21s225ms, % non-child: 0.02%)
> Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:57/13.92 GB
> Fragment Instance Lifecycle Event Timeline: 29h59m
> - Prepare Finished: 55.600ms (55.600ms)
> - Open Finished: 598.215ms (542.614ms)
> - First Batch Produced: 2s350ms (1s752ms)
> - First Batch Sent: 2s356ms (5.853ms)
> - ExecInternal Finished: 29h59m (29h59m)
> MemoryUsage(34m8s): 1.37 GB, 1.37 GB, 1.36 GB, 1.24 GB, 1.04 GB, 864.64 MB, 1.37 GB, 1.40 GB, 1.40 GB, 1.38 GB, 1.31 GB, 1.17 GB, 965.56 MB, 876.58 MB, 1.32 GB, 1.39 GB, 1.39 GB, 1.38 GB, 1.23 GB, 995.12 MB, 871.65 MB, 1.35 GB, 1.40 GB, 1.38 GB, 1.24 GB, 977.74 MB, 850.26 MB, 1.27 GB, 1.29 GB, 1.30 GB, 1.31 GB, 1.16 GB, 954.46 MB, 906.32 MB, 1.24 GB, 1.34 GB, 1.30 GB, 1.31 GB, 1.36 GB, 1.29 GB, 1.02 GB, 1.02 GB, 1.32 GB, 1.30 GB, 1.31 GB, 1.38 GB, 1.33 GB, 1.23 GB, 1.14 GB, 1017.39 MB, 822.59 MB, 631.09 MB
> ThreadUsage(34m8s): 8, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 8, 7
> - AverageThreadTokens: 8.86
> - BloomFilterBytes: 0
> - PeakMemoryUsage: 1.40 GB (1499916960)
> - PeakReservation: 0
> - PeakUsedReservation: 0
> - PerHostPeakMemUsage: 1.40 GB (1499916960)
> - RowsProduced: 428.25M (428247756)
> - TotalNetworkReceiveTime: 0.000ns
> - TotalNetworkSendTime: 0.000ns
> - TotalStorageWaitTime: 1m52s
> - TotalThreadsInvoluntaryContextSwitches: 388.30K (388300)
> - TotalThreadsTotalWallClockTime: 265h50m
> - TotalThreadsSysTime: 1m58s
> - TotalThreadsUserTime: 1h13m
> - TotalThreadsVoluntaryContextSwitches: 431.32K (431323)
> Buffer pool:
> - AllocTime: 0.000ns
> - CumulativeAllocationBytes: 0
> - CumulativeAllocations: 0 (0)
> - PeakReservation: 0
> - PeakUnpinnedBytes: 0
> - PeakUsedReservation: 0
> - ReadIoBytes: 0
> - ReadIoOps: 0 (0)
> - ReadIoWaitTime: 0.000ns
> - ReservationLimit: 0
> - WriteIoBytes: 0
> - WriteIoOps: 0 (0)
> - WriteIoWaitTime: 0.000ns
> Fragment Instance Lifecycle Timings:
> - ExecTime: 29h59m
> - ExecTreeExecTime: 4s703ms
> - OpenTime: 542.606ms
> - ExecTreeOpenTime: 32.022us
> - PrepareTime: 55.377ms
> - ExecTreePrepareTime: 20.617ms
> KuduTableSink:(Total: 29h59m, non-child: 29h59m, % non-child: 100.00%)
> - KuduApplyTimer: 29h
> - NumRowErrors: 346.33M (346333617)
> - PeakMemoryUsage: 20.00 MB (20971520)
> - RowsProcessedRate: 3.97 K/sec
> - TotalNumRows: 428.25M (428247756)
> HDFS_SCAN_NODE (id=0):(Total: 4s407ms, non-child: 4s407ms, % non-child: 100.00%)
> Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:57/13.92 GB
> ExecOption: PARQUET Codegen Enabled, Codegen enabled: 57 out of 57
> Hdfs Read Thread Concurrency Bucket: 0:99.81% 1:0.1895% 2:0% 3:0% 4:0% 5:0% 6:0%
> File Formats: PARQUET/SNAPPY:57
> BytesRead(34m8s): 1.29 GB, 1.56 GB, 1.71 GB, 1.81 GB, 1.90 GB, 2.18 GB, 3.19 GB, 3.30 GB, 3.46 GB, 3.65 GB, 3.76 GB, 3.84 GB, 3.91 GB, 4.19 GB, 5.09 GB, 5.31 GB, 5.48 GB, 5.64 GB, 5.78 GB, 5.90 GB, 6.16 GB, 7.13 GB, 7.39 GB, 7.61 GB, 7.76 GB, 7.91 GB, 8.15 GB, 9.04 GB, 9.25 GB, 9.49 GB, 9.72 GB, 9.83 GB, 9.95 GB, 10.16 GB, 10.86 GB, 11.12 GB, 11.26 GB, 11.41 GB, 11.60 GB, 11.71 GB, 11.87 GB, 12.39 GB, 13.18 GB, 13.32 GB, 13.42 GB, 13.63 GB, 13.70 GB, 13.76 GB, 13.82 GB, 13.87 GB, 13.89 GB, 13.90 GB
> - FooterProcessingTime: (Avg: 368.827ms ; Min: 61.227ms ; Max: 3s788ms ; Number of samples: 57)
> - AverageHdfsReadThreadConcurrency: 0.00
> - AverageScannerThreadConcurrency: 7.86
> - BytesRead: 13.92 GB (14949018844)
> - BytesReadDataNodeCache: 0
> - BytesReadLocal: 13.92 GB (14949018844)
> - BytesReadRemoteUnexpected: 0
> - BytesReadShortCircuit: 0
> - CachedFileHandlesHitCount: 2.31K (2315)
> - CachedFileHandlesMissCount: 57 (57)
> - CollectionItemsRead: 0 (0)
> - DecompressionTime: 22s734ms
> - MaxCompressedTextFileLength: 0
> - NumColumns: 16 (16)
> - NumDictFilteredRowGroups: 0 (0)
> - NumDisksAccessed: 1 (1)
> - NumRowGroups: 57 (57)
> - NumScannerThreadsStarted: 8 (8)
> - NumScannersWithNoReads: 0 (0)
> - NumStatsFilteredRowGroups: 0 (0)
> - PeakMemoryUsage: 1.38 GB (1478857724)
> - PerReadThreadRawHdfsThroughput: 69.53 MB/sec
> - RemoteScanRanges: 0 (0)
> - RowBatchQueueGetWaitTime: 1s749ms
> - RowBatchQueuePutWaitTime: 235h46m
> - RowsRead: 428.25M (428247756)
> - RowsReturned: 428.25M (428247756)
> - RowsReturnedRate: 97.15 M/sec
> - ScanRangesComplete: 57 (57)
> - ScannerThreadsInvoluntaryContextSwitches: 15.99K (15992)
> - ScannerThreadsTotalWallClockTime: 235h50m
> - MaterializeTupleTime(*): 2m7s
> - ScannerThreadsSysTime: 3s685ms
> - ScannerThreadsUserTime: 2m42s
> - ScannerThreadsVoluntaryContextSwitches: 419.35K (419351)
> - TotalRawHdfsOpenFileTime(*): 3s380ms
> - TotalRawHdfsReadTime(*): 3m25s
> - TotalReadThroughput: 135.25 KB/sec
> CodeGen:(Total: 405.466ms, non-child: 405.466ms, % non-child: 100.00%)
> - CodegenInvoluntaryContextSwitches: 2 (2)
> - CodegenTotalWallClockTime: 405.458ms
> - CodegenSysTime: 0.000ns
> - CodegenUserTime: 49.456ms
> - CodegenVoluntaryContextSwitches: 91 (91)
> - CompileTime: 170.012ms
> - IrGenerationTime: 52.154ms
> - LoadTime: 0.000ns
> - ModuleBitcodeSize: 2.26 MB (2373832)
> - NumFunctions: 19 (19)
> - NumInstructions: 706 (706)
> - OptimizationTime: 168.310ms
> - PeakMemoryUsage: 353.00 KB (361472)
> - PrepareTime: 14.493ms
> Instance 9649f7ab3bcc5fb8:f4d6a60700000003 (host=slave2:22000):(Total: 28h48m, non-child: 21s449ms, % non-child: 0.02%)
> Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:56/13.87 GB
> Fragment Instance Lifecycle Event Timeline: 28h48m
> - Prepare Finished: 13.348ms (13.348ms)
> - Open Finished: 348.443ms (335.094ms)
> - First Batch Produced: 1s505ms (1s157ms)
> - First Batch Sent: 1s508ms (3.334ms)
> - ExecInternal Finished: 28h48m (28h48m)
> MemoryUsage(34m8s): 1.38 GB, 1.38 GB, 1.39 GB, 1.26 GB, 1.06 GB, 777.91 MB, 1.37 GB, 1.38 GB, 1.38 GB, 1.39 GB, 1.33 GB, 1.20 GB, 985.85 MB, 781.77 MB, 1.25 GB, 1.37 GB, 1.37 GB, 1.37 GB, 1.25 GB, 1015.91 MB, 756.94 MB, 1.27 GB, 1.37 GB, 1.37 GB, 1.27 GB, 1012.70 MB, 775.19 MB, 1.25 GB, 1.41 GB, 1.41 GB, 1.30 GB, 1.01 GB, 871.44 MB, 1.34 GB, 1.38 GB, 1.38 GB, 1.38 GB, 1.37 GB, 1.31 GB, 1.14 GB, 825.57 MB, 1.32 GB, 1.40 GB, 1.40 GB, 1.37 GB, 1.28 GB, 1.17 GB, 1.02 GB, 856.05 MB, 694.00 MB
> ThreadUsage(34m8s): 8, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 8
> - AverageThreadTokens: 8.94
> - BloomFilterBytes: 0
> - PeakMemoryUsage: 1.41 GB (1512855058)
> - PeakReservation: 0
> - PeakUsedReservation: 0
> - PerHostPeakMemUsage: 1.41 GB (1512855058)
> - RowsProduced: 427.11M (427110203)
> - TotalNetworkReceiveTime: 0.000ns
> - TotalNetworkSendTime: 0.000ns
> - TotalStorageWaitTime: 2m27s
> - TotalThreadsInvoluntaryContextSwitches: 375.92K (375918)
> - TotalThreadsTotalWallClockTime: 257h33m
> - TotalThreadsSysTime: 2m12s
> - TotalThreadsUserTime: 1h9m
> - TotalThreadsVoluntaryContextSwitches: 431.39K (431386)
> Buffer pool:
> - AllocTime: 0.000ns
> - CumulativeAllocationBytes: 0
> - CumulativeAllocations: 0 (0)
> - PeakReservation: 0
> - PeakUnpinnedBytes: 0
> - PeakUsedReservation: 0
> - ReadIoBytes: 0
> - ReadIoOps: 0 (0)
> - ReadIoWaitTime: 0.000ns
> - ReservationLimit: 0
> - WriteIoBytes: 0
> - WriteIoOps: 0 (0)
> - WriteIoWaitTime: 0.000ns
> Fragment Instance Lifecycle Timings:
> - ExecTime: 28h48m
> - ExecTreeExecTime: 4s366ms
> - OpenTime: 335.089ms
> - ExecTreeOpenTime: 40.091us
> - PrepareTime: 13.257ms
> - ExecTreePrepareTime: 12.776ms
> KuduTableSink:(Total: 28h48m, non-child: 28h48m, % non-child: 100.00%)
> - KuduApplyTimer: 27h51m
> - NumRowErrors: 347.46M (347457539)
> - PeakMemoryUsage: 20.00 MB (20971520)
> - RowsProcessedRate: 4.12 K/sec
> - TotalNumRows: 427.11M (427110203)
> HDFS_SCAN_NODE (id=0):(Total: 4s001ms, non-child: 4s001ms, % non-child: 100.00%)
> Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:56/13.87 GB
> ExecOption: PARQUET Codegen Enabled, Codegen enabled: 56 out of 56
> Hdfs Read Thread Concurrency Bucket: 0:99.69% 1:0.3106% 2:0% 3:0% 4:0% 5:0% 6:0%
> File Formats: PARQUET/SNAPPY:56
> BytesRead(34m8s): 1.31 GB, 1.58 GB, 1.74 GB, 1.85 GB, 1.94 GB, 2.08 GB, 3.22 GB, 3.31 GB, 3.48 GB, 3.67 GB, 3.79 GB, 3.88 GB, 3.94 GB, 4.08 GB, 5.02 GB, 5.30 GB, 5.48 GB, 5.66 GB, 5.80 GB, 5.91 GB, 6.00 GB, 7.04 GB, 7.37 GB, 7.61 GB, 7.77 GB, 7.90 GB, 8.01 GB, 8.95 GB, 9.39 GB, 9.61 GB, 9.77 GB, 9.88 GB, 10.17 GB, 11.12 GB, 11.26 GB, 11.40 GB, 11.56 GB, 11.66 GB, 11.73 GB, 11.82 GB, 11.97 GB, 13.08 GB, 13.48 GB, 13.60 GB, 13.68 GB, 13.74 GB, 13.80 GB, 13.85 GB, 13.87 GB, 13.88 GB
> - FooterProcessingTime: (Avg: 339.758ms ; Min: 55.620ms ; Max: 2s575ms ; Number of samples: 56)
> - AverageHdfsReadThreadConcurrency: 0.00
> - AverageScannerThreadConcurrency: 7.94
> - BytesRead: 13.88 GB (14898474484)
> - BytesReadDataNodeCache: 0
> - BytesReadLocal: 13.88 GB (14898474484)
> - BytesReadRemoteUnexpected: 0
> - BytesReadShortCircuit: 0
> - CachedFileHandlesHitCount: 2.29K (2294)
> - CachedFileHandlesMissCount: 56 (56)
> - CollectionItemsRead: 0 (0)
> - DecompressionTime: 23s940ms
> - MaxCompressedTextFileLength: 0
> - NumColumns: 16 (16)
> - NumDictFilteredRowGroups: 0 (0)
> - NumDisksAccessed: 1 (1)
> - NumRowGroups: 56 (56)
> - NumScannerThreadsStarted: 8 (8)
> - NumScannersWithNoReads: 0 (0)
> - NumStatsFilteredRowGroups: 0 (0)
> - PeakMemoryUsage: 1.39 GB (1491692569)
> - PerReadThreadRawHdfsThroughput: 44.62 MB/sec
> - RemoteScanRanges: 0 (0)
> - RowBatchQueueGetWaitTime: 1s156ms
> - RowBatchQueuePutWaitTime: 228h40m
> - RowsRead: 427.11M (427110203)
> - RowsReturned: 427.11M (427110203)
> - RowsReturnedRate: 106.74 M/sec
> - ScanRangesComplete: 56 (56)
> - ScannerThreadsInvoluntaryContextSwitches: 7.41K (7412)
> - ScannerThreadsTotalWallClockTime: 228h45m
> - MaterializeTupleTime(*): 2m
> - ScannerThreadsSysTime: 4s574ms
> - ScannerThreadsUserTime: 2m39s
> - ScannerThreadsVoluntaryContextSwitches: 418.25K (418255)
> - TotalRawHdfsOpenFileTime(*): 4s930ms
> - TotalRawHdfsReadTime(*): 5m18s
> - TotalReadThroughput: 140.35 KB/sec
> CodeGen:(Total: 316.986ms, non-child: 316.986ms, % non-child: 100.00%)
> - CodegenInvoluntaryContextSwitches: 7 (7)
> - CodegenTotalWallClockTime: 316.976ms
> - CodegenSysTime: 19.346ms
> - CodegenUserTime: 35.434ms
> - CodegenVoluntaryContextSwitches: 87 (87)
> - CompileTime: 104.224ms
> - IrGenerationTime: 54.920ms
> - LoadTime: 0.000ns
> - ModuleBitcodeSize: 2.26 MB (2373832)
> - NumFunctions: 19 (19)
> - NumInstructions: 706 (706)
> - OptimizationTime: 143.706ms
> - PeakMemoryUsage: 353.00 KB (361472)
> - PrepareTime: 13.284ms
> Instance 9649f7ab3bcc5fb8:f4d6a60700000005 (host=slave5:22000):(Total: 28h47m, non-child: 30s821ms, % non-child: 0.03%)
> Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:56/13.87 GB
> Fragment Instance Lifecycle Event Timeline: 28h47m
> - Prepare Finished: 25.961ms (25.961ms)
> - Open Finished: 92.181ms (66.220ms)
> - First Batch Produced: 1s428ms (1s336ms)
> - First Batch Sent: 1s432ms (3.458ms)
> - ExecInternal Finished: 28h47m (28h47m)
> MemoryUsage(34m8s): 1.39 GB, 1.39 GB, 1.39 GB, 1.27 GB, 1.07 GB, 786.07 MB, 1.37 GB, 1.37 GB, 1.37 GB, 1.37 GB, 1.31 GB, 1.19 GB, 973.49 MB, 778.55 MB, 1.26 GB, 1.39 GB, 1.39 GB, 1.39 GB, 1.27 GB, 1.00 GB, 812.17 MB, 1.29 GB, 1.41 GB, 1.39 GB, 1.14 GB, 881.39 MB, 1.30 GB, 1.37 GB, 1.37 GB, 1.33 GB, 1.16 GB, 910.68 MB, 880.78 MB, 1.32 GB, 1.38 GB, 1.38 GB, 1.38 GB, 1.37 GB, 1.29 GB, 1.13 GB, 843.04 MB, 1.32 GB, 1.39 GB, 1.39 GB, 1.35 GB, 1.27 GB, 1.16 GB, 1.00 GB, 848.32 MB, 709.30 MB
> ThreadUsage(34m8s): 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 8
> - AverageThreadTokens: 8.94
> - BloomFilterBytes: 0
> - PeakMemoryUsage: 1.41 GB (1513059939)
> - PeakReservation: 0
> - PeakUsedReservation: 0
> - PerHostPeakMemUsage: 1.41 GB (1513059939)
> - RowsProduced: 426.83M (426832398)
> - TotalNetworkReceiveTime: 0.000ns
> - TotalNetworkSendTime: 0.000ns
> - TotalStorageWaitTime: 1m16s
> - TotalThreadsInvoluntaryContextSwitches: 365.02K (365022)
> - TotalThreadsTotalWallClockTime: 257h18m
> - TotalThreadsSysTime: 1m56s
> - TotalThreadsUserTime: 1h9m
> - TotalThreadsVoluntaryContextSwitches: 431.61K (431609)
> Buffer pool:
> - AllocTime: 0.000ns
> - CumulativeAllocationBytes: 0
> - CumulativeAllocations: 0 (0)
> - PeakReservation: 0
> - PeakUnpinnedBytes: 0
> - PeakUsedReservation: 0
> - ReadIoBytes: 0
> - ReadIoOps: 0 (0)
> - ReadIoWaitTime: 0.000ns
> - ReservationLimit: 0
> - WriteIoBytes: 0
> - WriteIoOps: 0 (0)
> - WriteIoWaitTime: 0.000ns
> Fragment Instance Lifecycle Timings:
> - ExecTime: 28h47m
> - ExecTreeExecTime: 4s226ms
> - OpenTime: 66.213ms
> - ExecTreeOpenTime: 30.358us
> - PrepareTime: 25.760ms
> - ExecTreePrepareTime: 25.484ms
> KuduTableSink:(Total: 28h46m, non-child: 28h46m, % non-child: 100.00%)
> - KuduApplyTimer: 27h51m
> - NumRowErrors: 345.67M (345674429)
> - PeakMemoryUsage: 20.00 MB (20971520)
> - RowsProcessedRate: 4.12 K/sec
> - TotalNumRows: 426.83M (426832398)
> HDFS_SCAN_NODE (id=0):(Total: 3s937ms, non-child: 3s937ms, % non-child: 100.00%)
> Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:56/13.87 GB
> ExecOption: PARQUET Codegen Enabled, Codegen enabled: 56 out of 56
> Hdfs Read Thread Concurrency Bucket: 0:99.86% 1:0.138% 2:0% 3:0% 4:0% 5:0% 6:0%
> File Formats: PARQUET/SNAPPY:56
> BytesRead(34m8s): 1.31 GB, 1.58 GB, 1.74 GB, 1.85 GB, 1.94 GB, 2.08 GB, 3.23 GB, 3.30 GB, 3.48 GB, 3.67 GB, 3.78 GB, 3.87 GB, 3.94 GB, 4.09 GB, 5.01 GB, 5.31 GB, 5.49 GB, 5.66 GB, 5.81 GB, 5.92 GB, 6.09 GB, 7.05 GB, 7.45 GB, 7.69 GB, 7.86 GB, 8.14 GB, 9.07 GB, 9.35 GB, 9.56 GB, 9.70 GB, 9.81 GB, 9.90 GB, 10.23 GB, 11.09 GB, 11.27 GB, 11.41 GB, 11.56 GB, 11.66 GB, 11.73 GB, 11.82 GB, 12.00 GB, 13.08 GB, 13.48 GB, 13.61 GB, 13.68 GB, 13.74 GB, 13.80 GB, 13.85 GB, 13.87 GB, 13.88 GB
> - FooterProcessingTime: (Avg: 256.473ms ; Min: 49.349ms ; Max: 2s343ms ; Number of samples: 56)
> - AverageHdfsReadThreadConcurrency: 0.00
> - AverageScannerThreadConcurrency: 7.94
> - BytesRead: 13.88 GB (14898579078)
> - BytesReadDataNodeCache: 0
> - BytesReadLocal: 13.88 GB (14898579078)
> - BytesReadRemoteUnexpected: 0
> - BytesReadShortCircuit: 0
> - CachedFileHandlesHitCount: 2.29K (2293)
> - CachedFileHandlesMissCount: 56 (56)
> - CollectionItemsRead: 0 (0)
> - DecompressionTime: 22s127ms
> - MaxCompressedTextFileLength: 0
> - NumColumns: 16 (16)
> - NumDictFilteredRowGroups: 0 (0)
> - NumDisksAccessed: 1 (1)
> - NumRowGroups: 56 (56)
> - NumScannerThreadsStarted: 8 (8)
> - NumScannersWithNoReads: 0 (0)
> - NumStatsFilteredRowGroups: 0 (0)
> - PeakMemoryUsage: 1.39 GB (1491831945)
> - PerReadThreadRawHdfsThroughput: 100.21 MB/sec
> - RemoteScanRanges: 0 (0)
> - RowBatchQueueGetWaitTime: 1s334ms
> - RowBatchQueuePutWaitTime: 228h26m
> - RowsRead: 426.83M (426832398)
> - RowsReturned: 426.83M (426832398)
> - RowsReturnedRate: 108.39 M/sec
> - ScanRangesComplete: 56 (56)
> - ScannerThreadsInvoluntaryContextSwitches: 14.55K (14553)
> - ScannerThreadsTotalWallClockTime: 228h30m
> - MaterializeTupleTime(*): 2m3s
> - ScannerThreadsSysTime: 4s079ms
> - ScannerThreadsUserTime: 2m39s
> - ScannerThreadsVoluntaryContextSwitches: 417.98K (417975)
> - TotalRawHdfsOpenFileTime(*): 3s114ms
> - TotalRawHdfsReadTime(*): 2m21s
> - TotalReadThroughput: 140.45 KB/sec
> CodeGen:(Total: 56.104ms, non-child: 56.104ms, % non-child: 100.00%)
> - CodegenInvoluntaryContextSwitches: 3 (3)
> - CodegenTotalWallClockTime: 56.095ms
> - CodegenSysTime: 0.000ns
> - CodegenUserTime: 37.520ms
> - CodegenVoluntaryContextSwitches: 1 (1)
> - CompileTime: 23.495ms
> - IrGenerationTime: 1.462ms
> - LoadTime: 0.000ns
> - ModuleBitcodeSize: 2.26 MB (2373832)
> - NumFunctions: 19 (19)
> - NumInstructions: 706 (706)
> - OptimizationTime: 15.936ms
> - PeakMemoryUsage: 353.00 KB (361472)
> - PrepareTime: 14.604ms
>
>
>
> On Thu, May 10, 2018 at 10:33 AM, Mostafa Mokhtar <mm...@cloudera.com>
> wrote:
>
>> Can you share the query profile for the successful insert query?
>>
>> Thanks
>> Mostafa
>>
>> On May 9, 2018, at 9:55 PM, Geetika Gupta <ge...@knoldus.in>
>> wrote:
>>
>> Thanks, Jeszy.
>>
>> We build impala again with --release flag and data load was successful
>> after that.
>>
>> But now we are facing another issue. The table in which we loaded the
>> data has less number of rows. We executed the following command:
>>
>> insert into LINEITEM select * from PARQUETIMPALA500.LINEITEM
>>
>> This query was successful, but when we tried the count(*) on both the
>> tables, row count was different:
>>
>> 0: jdbc:hive2://slave2:21050/default> select count(*) from lineitem
>> . . . . . . . . . . . . . . . . . . > ;
>> 536870912
>>
>> 0: jdbc:hive2://slave2:21050/default> select count(*) from
>> parquetimpala500.lineitem;
>> 3000028242
>>
>> Do you have any idea about this issue.
>>
>>
>> On Mon, May 7, 2018 at 12:06 PM, Jeszy <je...@gmail.com> wrote:
>>
>>> Impala doesn't store the data itself, so you can switch versions
>>> without rewriting data. But you don't have to do that, you would just
>>> have to build impala using the -release flag (of buildall.sh) and run
>>> it using the release binaries (versus the debug ones). If you would be
>>> looking at performance, using the release version is highly
>>> recommended anyway.
>>>
>>> On 7 May 2018 at 08:30, Geetika Gupta <ge...@knoldus.in> wrote:
>>> > Hi Jeszy,
>>> >
>>> > Currently, we are using the apache impala's Github master branch code.
>>> We
>>> > tried using the released version but we encountered some errors
>>> related to
>>> > downloading of dependencies and could not complete the installation.
>>> >
>>> > The current version of impala we are using: 2.12
>>> >
>>> > We can't try with the new release as we have already loaded 500GB of
>>> TPCH
>>> > data on our cluster.
>>> >
>>> > On Mon, May 7, 2018 at 11:43 AM, Jeszy <je...@gmail.com> wrote:
>>> >>
>>> >> What version of Impala are you using?
>>> >> DCHECKs won't be triggered if you run a release build. Looking at the
>>> >> code, it should work with bad values if not for the DCHECK. Can you
>>> >> try using a release build?
>>> >>
>>> >> On 7 May 2018 at 08:04, Geetika Gupta <ge...@knoldus.in>
>>> wrote:
>>> >> > Hi community,
>>> >> >
>>> >> > I was trying to load 500GB of TPCH data into kudu table using the
>>> >> > following
>>> >> > query:
>>> >> >
>>> >> > insert into lineitem select * from PARQUETIMPALA500.LINEITEM
>>> >> >
>>> >> > While executing the query for around 17 hrs it got cancelled as the
>>> >> > impalad
>>> >> > process of that machine got aborted. Here are the logs of the
>>> impalad
>>> >> > process.
>>> >> >
>>> >> > impalad.ERROR
>>> >> >
>>> >> > Log file created at: 2018/05/06 13:40:34
>>> >> > Running on machine: slave2
>>> >> > Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
>>> >> > E0506 13:40:34.097759 28730 logging.cc:121] stderr will be logged
>>> to
>>> >> > this
>>> >> > file.
>>> >> > SLF4J: Class path contains multiple SLF4J bindings.
>>> >> > SLF4J: Found binding in
>>> >> >
>>> >> > [jar:file:/root/softwares/impala/fe/target/dependency/slf4j-
>>> log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>>> >> > SLF4J: Found binding in
>>> >> >
>>> >> > [jar:file:/root/softwares/impala/testdata/target/dependency/
>>> slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>>> >> > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
>>> >> > explanation.
>>> >> > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFac
>>> tory]
>>> >> > 18/05/06 13:40:34 WARN util.NativeCodeLoader: Unable to load
>>> >> > native-hadoop
>>> >> > library for your platform... using builtin-java classes where
>>> applicable
>>> >> > 18/05/06 13:40:36 WARN shortcircuit.DomainSocketFactory: The
>>> >> > short-circuit
>>> >> > local reads feature cannot be used because libhadoop cannot be
>>> loaded.
>>> >> > tcmalloc: large alloc 1073741824 bytes == 0x484434000 @ 0x4135176
>>> >> > 0x7fd9e9fc3929
>>> >> > tcmalloc: large alloc 2147483648 bytes == 0x7fd540f18000 @
>>> 0x4135176
>>> >> > 0x7fd9e9fc3929
>>> >> > F0507 09:46:12.673912 29258 error-util.cc:148] Check failed:
>>> >> > log_entry.count
>>> >> >> 0 (-1831809966 vs. 0)
>>> >> > *** Check failure stack trace: ***
>>> >> > @ 0x3fc0c0d google::LogMessage::Fail()
>>> >> > @ 0x3fc24b2 google::LogMessage::SendToLog()
>>> >> > @ 0x3fc05e7 google::LogMessage::Flush()
>>> >> > @ 0x3fc3bae google::LogMessageFatal::~LogM
>>> essageFatal()
>>> >> > @ 0x1bbcb31 impala::PrintErrorMap()
>>> >> > @ 0x1bbcd07 impala::PrintErrorMapToString()
>>> >> > @ 0x2decbd7 impala::Coordinator::GetErrorLog()
>>> >> > @ 0x1a8d634 impala::ImpalaServer::UnregisterQuery()
>>> >> > @ 0x1b29264 impala::ImpalaServer::CloseOperation()
>>> >> > @ 0x2c5ce86
>>> >> >
>>> >> > apache::hive::service::cli::thrift::TCLIServiceProcessor::pr
>>> ocess_CloseOperation()
>>> >> > @ 0x2c56b8c
>>> >> > apache::hive::service::cli::thrift::TCLIServiceProcessor::di
>>> spatchCall()
>>> >> > @ 0x2c2fcb1
>>> >> > impala::ImpalaHiveServer2ServiceProcessor::dispatchCall()
>>> >> > @ 0x16fdb20 apache::thrift::TDispatchProce
>>> ssor::process()
>>> >> > @ 0x18ea6b3
>>> >> > apache::thrift::server::TAcceptQueueServer::Task::run()
>>> >> > @ 0x18e2181 impala::ThriftThread::RunRunnable()
>>> >> > @ 0x18e3885 boost::_mfi::mf2<>::operator()()
>>> >> > @ 0x18e371b boost::_bi::list3<>::operator()<>()
>>> >> > @ 0x18e3467 boost::_bi::bind_t<>::operator()()
>>> >> > @ 0x18e337a
>>> >> > boost::detail::function::void_function_obj_invoker0<>::invoke()
>>> >> > @ 0x192761c boost::function0<>::operator()()
>>> >> > @ 0x1c3ebf7 impala::Thread::SuperviseThread()
>>> >> > @ 0x1c470cd boost::_bi::list5<>::operator()<>()
>>> >> > @ 0x1c46ff1 boost::_bi::bind_t<>::operator()()
>>> >> > @ 0x1c46fb4 boost::detail::thread_data<>::run()
>>> >> > @ 0x2eedb4a thread_proxy
>>> >> > @ 0x7fda1dbb16ba start_thread
>>> >> > @ 0x7fda1d8e741d clone
>>> >> > Wrote minidump to
>>> >> > /tmp/minidumps/impalad/a9113d9b-bc3d-488a-1feebf9b-47b42022.dmp
>>> >> >
>>> >> > impalad.FATAL
>>> >> >
>>> >> > Log file created at: 2018/05/07 09:46:12
>>> >> > Running on machine: slave2
>>> >> > Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
>>> >> > F0507 09:46:12.673912 29258 error-util.cc:148] Check failed:
>>> >> > log_entry.count
>>> >> >> 0 (-1831809966 vs. 0)
>>> >> >
>>> >> > Impalad.INFO
>>> >> > edentials={real_user=root}} blocked reactor thread for 34288.6us
>>> >> > I0507 09:38:14.943245 29882 outbound_call.cc:288] RPC callback for
>>> RPC
>>> >> > call
>>> >> > kudu.tserver.TabletServerService.Write -> {remote=
>>> 136.243.74.42:7050
>>> >> > (slave5), user_credentials={real_user=root}} blocked reactor
>>> thread for
>>> >> > 35859.8us
>>> >> > I0507 09:38:15.942150 29882 outbound_call.cc:288] RPC callback for
>>> RPC
>>> >> > call
>>> >> > kudu.tserver.TabletServerService.Write -> {remote=
>>> 136.243.74.42:7050
>>> >> > (slave5), user_credentials={real_user=root}} blocked reactor
>>> thread for
>>> >> > 40664.9us
>>> >> > I0507 09:38:17.495046 29882 outbound_call.cc:288] RPC callback for
>>> RPC
>>> >> > call
>>> >> > kudu.tserver.TabletServerService.Write -> {remote=
>>> 136.243.74.42:7050
>>> >> > (slave5), user_credentials={real_user=root}} blocked reactor
>>> thread for
>>> >> > 49514.6us
>>> >> > I0507 09:46:12.664149 4507 coordinator.cc:783] Release admission
>>> >> > control
>>> >> > resources for query_id=3e4a4c646800e1d9:c859bb7f00000000
>>> >> > F0507 09:46:12.673912 29258 error-util.cc:148] Check failed:
>>> >> > log_entry.count
>>> >> >> 0 (-1831809966 vs. 0)
>>> >> > Wrote minidump to
>>> >> > /tmp/minidumps/impalad/a9113d9b-bc3d-488a-1feebf9b-47b42022.dmp
>>> >> >
>>> >> > Note:
>>> >> > We are executing the queries on 8 node cluster with the following
>>> >> > configuration
>>> >> > Cluster : 8 Node Cluster (48 GB RAM , 8 CPU Core and 2 TB hard-disk
>>> >> > each,
>>> >> > Intel(R) Core(TM) i7 CPU 950 @ 3.07GHz
>>> >> >
>>> >> >
>>> >> > --
>>> >> > Regards,
>>> >> > Geetika Gupta
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> > Regards,
>>> > Geetika Gupta
>>>
>>
>>
>>
>> --
>> Regards,
>> Geetika Gupta
>>
>>
>
>
> --
> Regards,
> Geetika Gupta
>
Re: Issue in data loading in Impala + Kudu
Posted by Geetika Gupta <ge...@knoldus.in>.
Please find below the query profile :
Query (id=9649f7ab3bcc5fb8:f4d6a60700000000):
Summary:
Session ID: b04a9080d1e1724d:41f2b0d261e8f280
Session Type: HIVESERVER2
HiveServer2 Protocol Version: V6
Start Time: 2018-05-08 17:55:23.181264000
End Time: 2018-05-10 00:34:17.784273000
Query Type: DML
Query State: FINISHED
Query Status: OK
Impala Version: impalad version 3.0.0-SNAPSHOT RELEASE (build
b68e06997c1f49f6b723d78e217efddec4f56f3a)
User: root
Connected User: root
Delegated User:
Network Address: ::ffff:46.4.88.233:59862
Default Db: kudu_impala_500
Sql Statement: insert into LINEITEM select * from PARQUETIMPALA500.LINEITEM
Coordinator: slave2:22000
Query Options (set by configuration):
Query Options (set by configuration and planner): MT_DOP=0
Plan:
----------------
Max Per-Host Resource Reservation: Memory=0B
Per-Host Resource Estimates: Memory=704.00MB
WARNING: The following tables are missing relevant table and/or column
statistics.
parquetimpala500.lineitem
F00:PLAN FRAGMENT [RANDOM] hosts=7 instances=7
| Per-Host Resources: mem-estimate=704.00MB mem-reservation=0B
INSERT INTO KUDU [kudu_impala_500.lineitem]
| mem-estimate=0B mem-reservation=0B
|
00:SCAN HDFS [parquetimpala500.lineitem, RANDOM]
partitions=1/1 files=396 size=97.29GB
stored statistics:
table: rows=unavailable size=unavailable
columns: unavailable
extrapolated-rows=disabled
mem-estimate=704.00MB mem-reservation=0B
tuple-ids=0 row-size=171B cardinality=unavailable
----------------
Estimated Per-Host Mem: 738197504
Tables Missing Stats: parquetimpala500.lineitem
Per Host Min Reservation: slave1:22000(0) slave2:22000(0)
slave3:22000(0) slave4:22000(0) slave5:22000(0) slave6:22000(0)
slave7:22000(0)
Request Pool: default-pool
Admission result: Admitted immediately
ExecSummary:
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem
Est. Peak Mem Detail
-----------------------------------------------------------------------------------------------------------------
00:SCAN HDFS 7 4s417ms 6s154ms 3.00B -1 1.39 GB
704.00 MB parquetimpala500.lineitem
Errors: Key already present in Kudu table
'impala::kudu_impala_500.LINEITEM'. (1 of -1831809966 similar)
Query Compilation: 6s413ms
- Metadata load started: 14.443ms (14.443ms)
- Metadata load finished. loaded-tables=2/2 load-requests=1
catalog-updates=7: 6s298ms (6s283ms)
- Analysis finished: 6s301ms (3.659ms)
- Value transfer graph computed: 6s302ms (282.554us)
- Single node plan created: 6s363ms (61.624ms)
- Runtime filters computed: 6s363ms (98.878us)
- Distributed plan created: 6s366ms (2.832ms)
- Planning finished: 6s413ms (46.751ms)
Query Timeline: 30h38m
- Query submitted: 56.829us (56.829us)
- Planning finished: 6s431ms (6s431ms)
- Submit for admission: 6s432ms (821.867us)
- Completed admission: 6s432ms (14.519us)
- Ready to start on 7 backends: 6s432ms (99.515us)
- All 7 execution backends (7 fragment instances) started:
6s535ms (103.320ms)
- Released admission control resources: 30h38m (30h38m)
- DML data written: 30h38m (934.739us)
- DML Metastore update finished: 30h38m (157.938us)
- Request finished: 30h38m (41.379us)
- First row fetched: 30h38m (173.124us)
- First row fetched: 30h38m (1.750ms)
- First row fetched: 30h38m (1.361ms)
- Unregister query: 30h38m (1.456ms)
- ComputeScanRangeAssignmentTimer: 558.440us
ImpalaServer:
- ClientFetchWaitTimer: 4.721ms
- MetastoreUpdateTimer: 191.424us
- RowMaterializationTimer: 0.000ns
Execution Profile 9649f7ab3bcc5fb8:f4d6a60700000000:(Total: 30h38m,
non-child: 0.000ns, % non-child: 0.00%)
Number of filters: 0
Filter routing table:
ID Src. Node Tgt. Node(s) Target type Partition filter Pending
(Expected) First arrived Completed Enabled
-------------------------------------------------------------------------------------------------------------------
Backend startup latencies: Count: 7, min / max: 1ms / 103ms, 25th
%-ile: 1ms, 50th %-ile: 67ms, 75th %-ile: 67ms, 90th %-ile: 80ms, 95th
%-ile: 103ms, 99.9th %-ile: 103ms
DML Stats:
Partition: Default
NumModifiedRows: 536870912
NumRowErrors: 2463157330
BytesWritten: 0
Per Node Peak Memory Usage: slave1:22000(1.40 GB)
slave2:22000(1.41 GB) slave3:22000(1.39 GB) slave4:22000(1.41 GB)
slave5:22000(1.41 GB) slave6:22000(1.40 GB) slave7:22000(1.40 GB)
- FiltersReceived: 0 (0)
- FinalizationTimer: 0.000ns
- NumBackends: 7 (7)
- NumFragmentInstances: 7 (7)
- NumFragments: 1 (1)
Averaged Fragment F00:(Total: 29h51m, non-child: 14s973ms, %
non-child: 0.01%)
split sizes: min: 13.63 GB, max: 14.10 GB, avg: 13.90 GB,
stddev: 143.22 MB
completion times: min:28h47m max:30h38m mean: 29h51m stddev:41m26s
execution rates: min:129.50 KB/sec max:140.31 KB/sec
mean:135.67 KB/sec stddev:3.50 KB/sec
num instances: 7
- AverageThreadTokens: 8.89
- BloomFilterBytes: 0
- PeakMemoryUsage: 1.40 GB (1503884428)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 1.40 GB (1503884428)
- RowsProduced: 428.58M (428575463)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 0.000ns
- TotalStorageWaitTime: 2m10s
- TotalThreadsInvoluntaryContextSwitches: 369.81K (369812)
- TotalThreadsTotalWallClockTime: 265h20m
- TotalThreadsSysTime: 1m59s
- TotalThreadsUserTime: 1h9m
- TotalThreadsVoluntaryContextSwitches: 433.92K (433917)
Buffer pool:
- AllocTime: 0.000ns
- CumulativeAllocationBytes: 0
- CumulativeAllocations: 0 (0)
- PeakReservation: 0
- PeakUnpinnedBytes: 0
- PeakUsedReservation: 0
- ReadIoBytes: 0
- ReadIoOps: 0 (0)
- ReadIoWaitTime: 0.000ns
- ReservationLimit: 0
- WriteIoBytes: 0
- WriteIoOps: 0 (0)
- WriteIoWaitTime: 0.000ns
Fragment Instance Lifecycle Timings:
- ExecTime: 29h51m
- ExecTreeExecTime: 4s724ms
- OpenTime: 519.095ms
- ExecTreeOpenTime: 36.350us
- PrepareTime: 60.445ms
- ExecTreePrepareTime: 19.494ms
KuduTableSink:(Total: 29h50m, non-child: 29h50m, % non-child: 100.00%)
- KuduApplyTimer: 28h54m
- NumRowErrors: 351.88M (351879618)
- PeakMemoryUsage: 20.00 MB (20971520)
- RowsProcessedRate: 3.99 K/sec
- TotalNumRows: 428.58M (428575463)
HDFS_SCAN_NODE (id=0):(Total: 4s417ms, non-child: 4s417ms, %
non-child: 100.00%)
- AverageHdfsReadThreadConcurrency: 0.00
- AverageScannerThreadConcurrency: 7.89
- BytesRead: 13.90 GB (14928723218)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 13.90 GB (14928723218)
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 0
- CachedFileHandlesHitCount: 2.31K (2305)
- CachedFileHandlesMissCount: 56 (56)
- CollectionItemsRead: 0 (0)
- DecompressionTime: 22s321ms
- MaxCompressedTextFileLength: 0
- NumColumns: 16 (16)
- NumDictFilteredRowGroups: 0 (0)
- NumDisksAccessed: 1 (1)
- NumRowGroups: 56 (56)
- NumScannerThreadsStarted: 8 (8)
- NumScannersWithNoReads: 0 (0)
- NumStatsFilteredRowGroups: 0 (0)
- PeakMemoryUsage: 1.38 GB (1482717084)
- PerReadThreadRawHdfsThroughput: 69.65 MB/sec
- RemoteScanRanges: 0 (0)
- RowBatchQueueGetWaitTime: 1s754ms
- RowBatchQueuePutWaitTime: 235h24m
- RowsRead: 428.58M (428575463)
- RowsReturned: 428.58M (428575463)
- RowsReturnedRate: 99.08 M/sec
- ScanRangesComplete: 56 (56)
- ScannerThreadsInvoluntaryContextSwitches: 12.33K (12332)
- ScannerThreadsTotalWallClockTime: 235h28m
- MaterializeTupleTime(*): 2m3s
- ScannerThreadsSysTime: 3s896ms
- ScannerThreadsUserTime: 2m39s
- ScannerThreadsVoluntaryContextSwitches: 419.69K (419690)
- TotalRawHdfsOpenFileTime(*): 3s995ms
- TotalRawHdfsReadTime(*): 4m2s
- TotalReadThroughput: 135.78 KB/sec
CodeGen:(Total: 436.431ms, non-child: 436.431ms, % non-child: 100.00%)
- CodegenInvoluntaryContextSwitches: 5 (5)
- CodegenTotalWallClockTime: 436.422ms
- CodegenSysTime: 8.585ms
- CodegenUserTime: 41.806ms
- CodegenVoluntaryContextSwitches: 76 (76)
- CompileTime: 179.278ms
- IrGenerationTime: 90.176ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 2.26 MB (2373832)
- NumFunctions: 19 (19)
- NumInstructions: 706 (706)
- OptimizationTime: 151.857ms
- PeakMemoryUsage: 353.00 KB (361472)
- PrepareTime: 14.499ms
Fragment F00:
Instance 9649f7ab3bcc5fb8:f4d6a60700000000
(host=slave3:22000):(Total: 30h38m, non-child: 11s364ms, % non-child:
0.01%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
0:56/13.63 GB
Fragment Instance Lifecycle Event Timeline: 30h38m
- Prepare Finished: 86.126ms (86.126ms)
- Open Finished: 1s157ms (1s070ms)
- First Batch Produced: 3s031ms (1s874ms)
- First Batch Sent: 3s037ms (5.724ms)
- ExecInternal Finished: 30h38m (30h38m)
MemoryUsage(34m8s): 1.28 GB, 1.34 GB, 1.37 GB, 1.35 GB, 1.21
GB, 925.43 MB, 727.39 MB, 1.17 GB, 1.35 GB, 1.35 GB, 1.35 GB, 1.35 GB,
1.26 GB, 1.06 GB, 836.27 MB, 792.34 MB, 1.26 GB, 1.36 GB, 1.36 GB,
1.30 GB, 1.10 GB, 827.96 MB, 1.01 GB, 1.38 GB, 1.38 GB, 1.32 GB, 1.12
GB, 875.56 MB, 979.49 MB, 1.34 GB, 1.37 GB, 1.37 GB, 1.28 GB, 1.10 GB,
950.36 MB, 805.45 MB, 963.12 MB, 1.28 GB, 1.34 GB, 1.34 GB, 1.30 GB,
1.04 GB, 808.54 MB, 994.36 MB, 1.17 GB, 1.18 GB, 1.18 GB, 1.18 GB,
1.17 GB, 1.13 GB, 1.04 GB, 954.52 MB, 811.51 MB
ThreadUsage(34m8s): 8, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9,
9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9,
9, 9, 9, 9, 9, 9, 9, 8, 8, 8, 8, 8, 8, 8, 8, 8
- AverageThreadTokens: 8.80
- BloomFilterBytes: 0
- PeakMemoryUsage: 1.39 GB (1487275684)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 1.39 GB (1487275684)
- RowsProduced: 424.29M (424287956)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 0.000ns
- TotalStorageWaitTime: 1m32s
- TotalThreadsInvoluntaryContextSwitches: 349.35K (349347)
- TotalThreadsTotalWallClockTime: 269h37m
- TotalThreadsSysTime: 1m54s
- TotalThreadsUserTime: 1h6m
- TotalThreadsVoluntaryContextSwitches: 438.17K (438165)
Buffer pool:
- AllocTime: 0.000ns
- CumulativeAllocationBytes: 0
- CumulativeAllocations: 0 (0)
- PeakReservation: 0
- PeakUnpinnedBytes: 0
- PeakUsedReservation: 0
- ReadIoBytes: 0
- ReadIoOps: 0 (0)
- ReadIoWaitTime: 0.000ns
- ReservationLimit: 0
- WriteIoBytes: 0
- WriteIoOps: 0 (0)
- WriteIoWaitTime: 0.000ns
Fragment Instance Lifecycle Timings:
- ExecTime: 30h38m
- ExecTreeExecTime: 4s586ms
- OpenTime: 1s070ms
- ExecTreeOpenTime: 32.390us
- PrepareTime: 85.903ms
- ExecTreePrepareTime: 16.283ms
KuduTableSink:(Total: 30h38m, non-child: 30h38m, % non-child: 100.00%)
- KuduApplyTimer: 29h44m
- NumRowErrors: 371.48M (371483541)
- PeakMemoryUsage: 20.00 MB (20971520)
- RowsProcessedRate: 3.85 K/sec
- TotalNumRows: 424.29M (424287956)
HDFS_SCAN_NODE (id=0):(Total: 4s304ms, non-child: 4s304ms, %
non-child: 100.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
0:56/13.63 GB
ExecOption: PARQUET Codegen Enabled, Codegen enabled: 56 out of 56
Hdfs Read Thread Concurrency Bucket: 0:99.87% 1:0.1279% 2:0%
3:0% 4:0% 5:0% 6:0%
File Formats: PARQUET/SNAPPY:56
BytesRead(34m8s): 1.19 GB, 1.46 GB, 1.65 GB, 1.78 GB, 1.88
GB, 1.97 GB, 2.01 GB, 2.91 GB, 3.27 GB, 3.46 GB, 3.61 GB, 3.72 GB,
3.80 GB, 3.90 GB, 3.96 GB, 4.16 GB, 5.05 GB, 5.36 GB, 5.60 GB, 5.76
GB, 5.87 GB, 5.94 GB, 6.57 GB, 7.32 GB, 7.57 GB, 7.75 GB, 7.86 GB,
7.92 GB, 8.43 GB, 9.18 GB, 9.43 GB, 9.62 GB, 9.75 GB, 9.84 GB, 9.89
GB, 9.94 GB, 10.41 GB, 11.07 GB, 11.24 GB, 11.42 GB, 11.66 GB, 11.82
GB, 12.05 GB, 12.48 GB, 12.96 GB, 13.07 GB, 13.19 GB, 13.31 GB, 13.38
GB, 13.46 GB, 13.51 GB, 13.57 GB, 13.61 GB
- FooterProcessingTime: (Avg: 381.833ms ; Min: 46.831ms ;
Max: 4s214ms ; Number of samples: 56)
- AverageHdfsReadThreadConcurrency: 0.00
- AverageScannerThreadConcurrency: 7.80
- BytesRead: 13.63 GB (14635606518)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 13.63 GB (14635606518)
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 0
- CachedFileHandlesHitCount: 2.27K (2268)
- CachedFileHandlesMissCount: 56 (56)
- CollectionItemsRead: 0 (0)
- DecompressionTime: 19s612ms
- MaxCompressedTextFileLength: 0
- NumColumns: 16 (16)
- NumDictFilteredRowGroups: 0 (0)
- NumDisksAccessed: 1 (1)
- NumRowGroups: 56 (56)
- NumScannerThreadsStarted: 8 (8)
- NumScannersWithNoReads: 0 (0)
- NumStatsFilteredRowGroups: 0 (0)
- PeakMemoryUsage: 1.37 GB (1466047697)
- PerReadThreadRawHdfsThroughput: 100.45 MB/sec
- RemoteScanRanges: 0 (0)
- RowBatchQueueGetWaitTime: 1s872ms
- RowBatchQueuePutWaitTime: 238h55m
- RowsRead: 424.29M (424287956)
- RowsReturned: 424.29M (424287956)
- RowsReturnedRate: 98.57 M/sec
- ScanRangesComplete: 56 (56)
- ScannerThreadsInvoluntaryContextSwitches: 12.67K (12674)
- ScannerThreadsTotalWallClockTime: 238h58m
- MaterializeTupleTime(*): 1m57s
- ScannerThreadsSysTime: 3s421ms
- ScannerThreadsUserTime: 2m31s
- ScannerThreadsVoluntaryContextSwitches: 415.49K (415490)
- TotalRawHdfsOpenFileTime(*): 2s904ms
- TotalRawHdfsReadTime(*): 2m18s
- TotalReadThroughput: 129.60 KB/sec
CodeGen:(Total: 915.511ms, non-child: 915.511ms, % non-child: 100.00%)
- CodegenInvoluntaryContextSwitches: 10 (10)
- CodegenTotalWallClockTime: 915.504ms
- CodegenSysTime: 8.256ms
- CodegenUserTime: 45.032ms
- CodegenVoluntaryContextSwitches: 87 (87)
- CompileTime: 319.788ms
- IrGenerationTime: 276.446ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 2.26 MB (2373832)
- NumFunctions: 19 (19)
- NumInstructions: 706 (706)
- OptimizationTime: 301.622ms
- PeakMemoryUsage: 353.00 KB (361472)
- PrepareTime: 16.858ms
Instance 9649f7ab3bcc5fb8:f4d6a60700000002
(host=slave1:22000):(Total: 30h17m, non-child: 1s508ms, % non-child:
0.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
0:57/14.10 GB
Fragment Instance Lifecycle Event Timeline: 30h17m
- Prepare Finished: 32.853ms (32.853ms)
- Open Finished: 362.691ms (329.838ms)
- First Batch Produced: 1s725ms (1s362ms)
- First Batch Sent: 1s728ms (3.415ms)
- ExecInternal Finished: 30h17m (30h17m)
MemoryUsage(34m8s): 1.39 GB, 1.40 GB, 1.40 GB, 1.27 GB, 1.07
GB, 798.75 MB, 1.36 GB, 1.38 GB, 1.38 GB, 1.38 GB, 1.32 GB, 1.19 GB,
974.24 MB, 800.43 MB, 1.27 GB, 1.38 GB, 1.38 GB, 1.38 GB, 1.26 GB,
1010.99 MB, 805.20 MB, 1.29 GB, 1.38 GB, 1.38 GB, 1.27 GB, 987.74 MB,
811.33 MB, 1.27 GB, 1.38 GB, 1.38 GB, 1.34 GB, 1.18 GB, 922.96 MB,
765.77 MB, 1.13 GB, 1.36 GB, 1.36 GB, 1.36 GB, 1.37 GB, 1.30 GB, 1.03
GB, 919.88 MB, 1.38 GB, 1.38 GB, 1.38 GB, 1.39 GB, 1.35 GB, 1.24 GB,
1.17 GB, 1005.37 MB, 844.55 MB, 751.70 MB, 518.60 MB
ThreadUsage(34m8s): 8, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9,
9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9,
9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 5
- AverageThreadTokens: 8.91
- BloomFilterBytes: 0
- PeakMemoryUsage: 1.40 GB (1500404835)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 1.40 GB (1500404835)
- RowsProduced: 434.96M (434955041)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 0.000ns
- TotalStorageWaitTime: 2m56s
- TotalThreadsInvoluntaryContextSwitches: 370.59K (370587)
- TotalThreadsTotalWallClockTime: 269h58m
- TotalThreadsSysTime: 1m58s
- TotalThreadsUserTime: 1h9m
- TotalThreadsVoluntaryContextSwitches: 437.63K (437630)
Buffer pool:
- AllocTime: 0.000ns
- CumulativeAllocationBytes: 0
- CumulativeAllocations: 0 (0)
- PeakReservation: 0
- PeakUnpinnedBytes: 0
- PeakUsedReservation: 0
- ReadIoBytes: 0
- ReadIoOps: 0 (0)
- ReadIoWaitTime: 0.000ns
- ReservationLimit: 0
- WriteIoBytes: 0
- WriteIoOps: 0 (0)
- WriteIoWaitTime: 0.000ns
Fragment Instance Lifecycle Timings:
- ExecTime: 30h17m
- ExecTreeExecTime: 4s575ms
- OpenTime: 329.834ms
- ExecTreeOpenTime: 45.986us
- PrepareTime: 32.581ms
- ExecTreePrepareTime: 18.647ms
KuduTableSink:(Total: 30h17m, non-child: 30h17m, % non-child: 100.00%)
- KuduApplyTimer: 29h22m
- NumRowErrors: 357.86M (357862755)
- PeakMemoryUsage: 20.00 MB (20971520)
- RowsProcessedRate: 3.99 K/sec
- TotalNumRows: 434.96M (434955041)
HDFS_SCAN_NODE (id=0):(Total: 4s229ms, non-child: 4s229ms, %
non-child: 100.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
0:57/14.10 GB
ExecOption: PARQUET Codegen Enabled, Codegen enabled: 57 out of 57
Hdfs Read Thread Concurrency Bucket: 0:99.63% 1:0.3706% 2:0%
3:0% 4:0% 5:0% 6:0%
File Formats: PARQUET/SNAPPY:57
BytesRead(34m8s): 1.32 GB, 1.59 GB, 1.74 GB, 1.85 GB, 1.94
GB, 2.11 GB, 3.21 GB, 3.31 GB, 3.48 GB, 3.67 GB, 3.79 GB, 3.88 GB,
3.94 GB, 4.11 GB, 5.04 GB, 5.32 GB, 5.50 GB, 5.67 GB, 5.81 GB, 5.92
GB, 6.09 GB, 7.06 GB, 7.40 GB, 7.63 GB, 7.79 GB, 7.90 GB, 8.08 GB,
9.02 GB, 9.35 GB, 9.55 GB, 9.70 GB, 9.81 GB, 9.90 GB, 10.01 GB, 10.74
GB, 11.19 GB, 11.34 GB, 11.48 GB, 11.57 GB, 11.69 GB, 11.84 GB, 12.25
GB, 13.25 GB, 13.41 GB, 13.50 GB, 13.60 GB, 13.67 GB, 13.73 GB, 13.79
GB, 13.83 GB, 13.86 GB, 13.92 GB, 14.03 GB
- FooterProcessingTime: (Avg: 414.198ms ; Min: 51.291ms ;
Max: 2s894ms ; Number of samples: 57)
- AverageHdfsReadThreadConcurrency: 0.00
- AverageScannerThreadConcurrency: 7.91
- BytesRead: 14.11 GB (15148002597)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 14.11 GB (15148002597)
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 0
- CachedFileHandlesHitCount: 2.33K (2333)
- CachedFileHandlesMissCount: 57 (57)
- CollectionItemsRead: 0 (0)
- DecompressionTime: 23s330ms
- MaxCompressedTextFileLength: 0
- NumColumns: 16 (16)
- NumDictFilteredRowGroups: 0 (0)
- NumDisksAccessed: 1 (1)
- NumRowGroups: 57 (57)
- NumScannerThreadsStarted: 8 (8)
- NumScannersWithNoReads: 0 (0)
- NumStatsFilteredRowGroups: 0 (0)
- PeakMemoryUsage: 1.38 GB (1479367012)
- PerReadThreadRawHdfsThroughput: 36.38 MB/sec
- RemoteScanRanges: 0 (0)
- RowBatchQueueGetWaitTime: 1s354ms
- RowBatchQueuePutWaitTime: 239h35m
- RowsRead: 434.96M (434955041)
- RowsReturned: 434.96M (434955041)
- RowsReturnedRate: 102.83 M/sec
- ScanRangesComplete: 57 (57)
- ScannerThreadsInvoluntaryContextSwitches: 7.23K (7231)
- ScannerThreadsTotalWallClockTime: 239h40m
- MaterializeTupleTime(*): 2m1s
- ScannerThreadsSysTime: 4s044ms
- ScannerThreadsUserTime: 2m42s
- ScannerThreadsVoluntaryContextSwitches: 425.94K (425935)
- TotalRawHdfsOpenFileTime(*): 5s555ms
- TotalRawHdfsReadTime(*): 6m37s
- TotalReadThroughput: 135.68 KB/sec
CodeGen:(Total: 274.452ms, non-child: 274.452ms, % non-child: 100.00%)
- CodegenInvoluntaryContextSwitches: 4 (4)
- CodegenTotalWallClockTime: 274.443ms
- CodegenSysTime: 7.440ms
- CodegenUserTime: 40.295ms
- CodegenVoluntaryContextSwitches: 86 (86)
- CompileTime: 103.404ms
- IrGenerationTime: 32.648ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 2.26 MB (2373832)
- NumFunctions: 19 (19)
- NumInstructions: 706 (706)
- OptimizationTime: 122.033ms
- PeakMemoryUsage: 353.00 KB (361472)
- PrepareTime: 15.943ms
Instance 9649f7ab3bcc5fb8:f4d6a60700000001
(host=slave4:22000):(Total: 30h16m, non-child: 2s019ms, % non-child:
0.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
0:56/13.87 GB
Fragment Instance Lifecycle Event Timeline: 30h16m
- Prepare Finished: 105.379ms (105.379ms)
- Open Finished: 519.128ms (413.748ms)
- First Batch Produced: 1s789ms (1s270ms)
- First Batch Sent: 1s795ms (6.364ms)
- ExecInternal Finished: 30h16m (30h16m)
MemoryUsage(34m8s): 1.39 GB, 1.40 GB, 1.40 GB, 1.38 GB, 1.23
GB, 924.30 MB, 715.63 MB, 1.26 GB, 1.40 GB, 1.40 GB, 1.40 GB, 1.40 GB,
1.30 GB, 1.08 GB, 843.90 MB, 887.06 MB, 1.40 GB, 1.41 GB, 1.39 GB,
1.19 GB, 826.21 MB, 1.19 GB, 1.38 GB, 1.38 GB, 1.31 GB, 1.05 GB,
748.49 MB, 1.17 GB, 1.38 GB, 1.38 GB, 1.38 GB, 1.24 GB, 999.76 MB,
733.27 MB, 995.28 MB, 1.35 GB, 1.38 GB, 1.38 GB, 1.38 GB, 1.35 GB,
1.10 GB, 861.01 MB, 1.39 GB, 1.40 GB, 1.40 GB, 1.40 GB, 1.39 GB, 1.33
GB, 1.24 GB, 1.13 GB, 1007.59 MB, 820.51 MB, 655.33 MB
ThreadUsage(34m8s): 8, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9,
9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9,
9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 8
- AverageThreadTokens: 8.97
- BloomFilterBytes: 0
- PeakMemoryUsage: 1.41 GB (1513166691)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 1.41 GB (1513166691)
- RowsProduced: 425.55M (425550934)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 0.000ns
- TotalStorageWaitTime: 1m49s
- TotalThreadsInvoluntaryContextSwitches: 364.52K (364523)
- TotalThreadsTotalWallClockTime: 271h36m
- TotalThreadsSysTime: 1m56s
- TotalThreadsUserTime: 1h9m
- TotalThreadsVoluntaryContextSwitches: 431.64K (431643)
Buffer pool:
- AllocTime: 0.000ns
- CumulativeAllocationBytes: 0
- CumulativeAllocations: 0 (0)
- PeakReservation: 0
- PeakUnpinnedBytes: 0
- PeakUsedReservation: 0
- ReadIoBytes: 0
- ReadIoOps: 0 (0)
- ReadIoWaitTime: 0.000ns
- ReservationLimit: 0
- WriteIoBytes: 0
- WriteIoOps: 0 (0)
- WriteIoWaitTime: 0.000ns
Fragment Instance Lifecycle Timings:
- ExecTime: 30h16m
- ExecTreeExecTime: 4s185ms
- OpenTime: 413.744ms
- ExecTreeOpenTime: 31.615us
- PrepareTime: 105.138ms
- ExecTreePrepareTime: 17.849ms
KuduTableSink:(Total: 30h16m, non-child: 30h16m, % non-child: 100.00%)
- KuduApplyTimer: 29h20m
- NumRowErrors: 336.92M (336918666)
- PeakMemoryUsage: 20.00 MB (20971520)
- RowsProcessedRate: 3.90 K/sec
- TotalNumRows: 425.55M (425550934)
HDFS_SCAN_NODE (id=0):(Total: 3s886ms, non-child: 3s886ms, %
non-child: 100.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
0:56/13.87 GB
ExecOption: PARQUET Codegen Enabled, Codegen enabled: 56 out of 56
Hdfs Read Thread Concurrency Bucket: 0:99.86% 1:0.1368% 2:0%
3:0% 4:0% 5:0% 6:0%
File Formats: PARQUET/SNAPPY:56
BytesRead(34m8s): 1.29 GB, 1.51 GB, 1.66 GB, 1.79 GB, 1.88
GB, 1.97 GB, 2.00 GB, 3.01 GB, 3.30 GB, 3.49 GB, 3.63 GB, 3.74 GB,
3.82 GB, 3.92 GB, 3.96 GB, 4.29 GB, 5.26 GB, 5.47 GB, 5.71 GB, 5.87
GB, 5.94 GB, 6.88 GB, 7.34 GB, 7.59 GB, 7.76 GB, 7.89 GB, 7.93 GB,
8.83 GB, 9.30 GB, 9.51 GB, 9.67 GB, 9.79 GB, 9.89 GB, 9.91 GB, 10.45
GB, 11.14 GB, 11.30 GB, 11.46 GB, 11.55 GB, 11.68 GB, 11.83 GB, 12.10
GB, 13.21 GB, 13.35 GB, 13.47 GB, 13.56 GB, 13.64 GB, 13.71 GB, 13.76
GB, 13.81 GB, 13.86 GB, 13.87 GB, 13.88 GB
- FooterProcessingTime: (Avg: 331.439ms ; Min: 46.760ms ;
Max: 2s508ms ; Number of samples: 56)
- AverageHdfsReadThreadConcurrency: 0.00
- AverageScannerThreadConcurrency: 7.97
- BytesRead: 13.88 GB (14898393824)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 13.88 GB (14898393824)
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 0
- CachedFileHandlesHitCount: 2.30K (2296)
- CachedFileHandlesMissCount: 56 (56)
- CollectionItemsRead: 0 (0)
- DecompressionTime: 22s686ms
- MaxCompressedTextFileLength: 0
- NumColumns: 16 (16)
- NumDictFilteredRowGroups: 0 (0)
- NumDisksAccessed: 1 (1)
- NumRowGroups: 56 (56)
- NumScannerThreadsStarted: 8 (8)
- NumScannersWithNoReads: 0 (0)
- NumStatsFilteredRowGroups: 0 (0)
- PeakMemoryUsage: 1.39 GB (1491938707)
- PerReadThreadRawHdfsThroughput: 95.20 MB/sec
- RemoteScanRanges: 0 (0)
- RowBatchQueueGetWaitTime: 1s255ms
- RowBatchQueuePutWaitTime: 241h15m
- RowsRead: 425.55M (425550934)
- RowsReturned: 425.55M (425550934)
- RowsReturnedRate: 109.50 M/sec
- ScanRangesComplete: 56 (56)
- ScannerThreadsInvoluntaryContextSwitches: 12.02K (12019)
- ScannerThreadsTotalWallClockTime: 241h20m
- MaterializeTupleTime(*): 2m6s
- ScannerThreadsSysTime: 3s548ms
- ScannerThreadsUserTime: 2m39s
- ScannerThreadsVoluntaryContextSwitches: 416.75K (416750)
- TotalRawHdfsOpenFileTime(*): 3s230ms
- TotalRawHdfsReadTime(*): 2m29s
- TotalReadThroughput: 133.55 KB/sec
CodeGen:(Total: 315.226ms, non-child: 315.226ms, % non-child: 100.00%)
- CodegenInvoluntaryContextSwitches: 9 (9)
- CodegenTotalWallClockTime: 315.217ms
- CodegenSysTime: 5.142ms
- CodegenUserTime: 43.310ms
- CodegenVoluntaryContextSwitches: 91 (91)
- CompileTime: 129.393ms
- IrGenerationTime: 60.406ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 2.26 MB (2373832)
- NumFunctions: 19 (19)
- NumInstructions: 706 (706)
- OptimizationTime: 111.534ms
- PeakMemoryUsage: 353.00 KB (361472)
- PrepareTime: 13.425ms
Instance 9649f7ab3bcc5fb8:f4d6a60700000004
(host=slave7:22000):(Total: 30h10m, non-child: 16s427ms, % non-child:
0.02%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
0:58/14.03 GB
Fragment Instance Lifecycle Event Timeline: 30h10m
- Prepare Finished: 105.361ms (105.361ms)
- Open Finished: 980.610ms (875.249ms)
- First Batch Produced: 4s553ms (3s573ms)
- First Batch Sent: 4s559ms (5.668ms)
- ExecInternal Finished: 30h10m (30h10m)
MemoryUsage(34m8s): 1.38 GB, 1.40 GB, 1.40 GB, 1.28 GB, 1.08
GB, 796.55 MB, 1.34 GB, 1.37 GB, 1.35 GB, 1.34 GB, 1.27 GB, 1.21 GB,
983.86 MB, 1.08 GB, 1.38 GB, 1.36 GB, 1.35 GB, 1.28 GB, 1.17 GB,
921.06 MB, 1.02 GB, 1.35 GB, 1.33 GB, 1.28 GB, 1.15 GB, 905.74 MB,
1.01 GB, 1.36 GB, 1.34 GB, 1.32 GB, 1.21 GB, 1.10 GB, 877.34 MB, 1.04
GB, 1.32 GB, 1.38 GB, 1.36 GB, 1.35 GB, 1.32 GB, 1.21 GB, 1000.76 MB,
1.11 GB, 1.37 GB, 1.36 GB, 1.33 GB, 1.29 GB, 1.20 GB, 1.09 GB, 1.07
GB, 937.32 MB, 814.51 MB, 480.87 MB, 155.08 MB
ThreadUsage(34m8s): 8, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9,
9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9,
9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 8, 5, 2
- AverageThreadTokens: 8.80
- BloomFilterBytes: 0
- PeakMemoryUsage: 1.40 GB (1500511831)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 1.40 GB (1500511831)
- RowsProduced: 433.04M (433043954)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 0.000ns
- TotalStorageWaitTime: 3m18s
- TotalThreadsInvoluntaryContextSwitches: 374.99K (374993)
- TotalThreadsTotalWallClockTime: 265h25m
- TotalThreadsSysTime: 1m57s
- TotalThreadsUserTime: 1h10m
- TotalThreadsVoluntaryContextSwitches: 435.67K (435665)
Buffer pool:
- AllocTime: 0.000ns
- CumulativeAllocationBytes: 0
- CumulativeAllocations: 0 (0)
- PeakReservation: 0
- PeakUnpinnedBytes: 0
- PeakUsedReservation: 0
- ReadIoBytes: 0
- ReadIoOps: 0 (0)
- ReadIoWaitTime: 0.000ns
- ReservationLimit: 0
- WriteIoBytes: 0
- WriteIoOps: 0 (0)
- WriteIoWaitTime: 0.000ns
Fragment Instance Lifecycle Timings:
- ExecTime: 30h10m
- ExecTreeExecTime: 6s426ms
- OpenTime: 875.240ms
- ExecTreeOpenTime: 41.994us
- PrepareTime: 105.102ms
- ExecTreePrepareTime: 24.800ms
KuduTableSink:(Total: 30h9m, non-child: 30h9m, % non-child: 100.00%)
- KuduApplyTimer: 29h12m
- NumRowErrors: 357.43M (357426783)
- PeakMemoryUsage: 20.00 MB (20971520)
- RowsProcessedRate: 3.99 K/sec
- TotalNumRows: 433.04M (433043954)
HDFS_SCAN_NODE (id=0):(Total: 6s154ms, non-child: 6s154ms, %
non-child: 100.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
0:58/14.03 GB
ExecOption: PARQUET Codegen Enabled, Codegen enabled: 58 out of 58
Hdfs Read Thread Concurrency Bucket: 0:99.68% 1:0.3169% 2:0%
3:0% 4:0% 5:0% 6:0%
File Formats: PARQUET/SNAPPY:58
BytesRead(34m8s): 1.31 GB, 1.58 GB, 1.74 GB, 1.85 GB, 1.94
GB, 2.10 GB, 3.18 GB, 3.29 GB, 3.44 GB, 3.62 GB, 3.72 GB, 3.95 GB,
4.03 GB, 4.61 GB, 5.22 GB, 5.37 GB, 5.52 GB, 5.66 GB, 5.90 GB, 6.01
GB, 6.53 GB, 7.22 GB, 7.43 GB, 7.62 GB, 7.83 GB, 7.99 GB, 8.49 GB,
9.18 GB, 9.38 GB, 9.55 GB, 9.67 GB, 9.88 GB, 9.98 GB, 10.53 GB, 11.04
GB, 11.22 GB, 11.39 GB, 11.50 GB, 11.58 GB, 11.70 GB, 11.95 GB, 12.61
GB, 13.30 GB, 13.43 GB, 13.52 GB, 13.59 GB, 13.65 GB, 13.72 GB, 13.91
GB, 13.94 GB, 13.96 GB, 13.98 GB, 14.03 GB
- FooterProcessingTime: (Avg: 615.194ms ; Min: 50.813ms ;
Max: 6s740ms ; Number of samples: 58)
- AverageHdfsReadThreadConcurrency: 0.00
- AverageScannerThreadConcurrency: 7.80
- BytesRead: 14.04 GB (15072987184)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 14.04 GB (15072987184)
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 0
- CachedFileHandlesHitCount: 2.34K (2340)
- CachedFileHandlesMissCount: 58 (58)
- CollectionItemsRead: 0 (0)
- DecompressionTime: 21s821ms
- MaxCompressedTextFileLength: 0
- NumColumns: 16 (16)
- NumDictFilteredRowGroups: 0 (0)
- NumDisksAccessed: 1 (1)
- NumRowGroups: 58 (58)
- NumScannerThreadsStarted: 8 (8)
- NumScannersWithNoReads: 0 (0)
- NumStatsFilteredRowGroups: 0 (0)
- PeakMemoryUsage: 1.38 GB (1479283937)
- PerReadThreadRawHdfsThroughput: 41.17 MB/sec
- RemoteScanRanges: 0 (0)
- RowBatchQueueGetWaitTime: 3s555ms
- RowBatchQueuePutWaitTime: 235h9m
- RowsRead: 433.04M (433043954)
- RowsReturned: 433.04M (433043954)
- RowsReturnedRate: 70.36 M/sec
- ScanRangesComplete: 58 (58)
- ScannerThreadsInvoluntaryContextSwitches: 16.45K (16447)
- ScannerThreadsTotalWallClockTime: 235h15m
- MaterializeTupleTime(*): 2m5s
- ScannerThreadsSysTime: 3s924ms
- ScannerThreadsUserTime: 2m40s
- ScannerThreadsVoluntaryContextSwitches: 424.08K (424080)
- TotalRawHdfsOpenFileTime(*): 4s852ms
- TotalRawHdfsReadTime(*): 5m49s
- TotalReadThroughput: 135.58 KB/sec
CodeGen:(Total: 771.273ms, non-child: 771.273ms, % non-child: 100.00%)
- CodegenInvoluntaryContextSwitches: 5 (5)
- CodegenTotalWallClockTime: 771.263ms
- CodegenSysTime: 19.916ms
- CodegenUserTime: 41.600ms
- CodegenVoluntaryContextSwitches: 91 (91)
- CompileTime: 404.632ms
- IrGenerationTime: 153.195ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 2.26 MB (2373832)
- NumFunctions: 19 (19)
- NumInstructions: 706 (706)
- OptimizationTime: 199.856ms
- PeakMemoryUsage: 353.00 KB (361472)
- PrepareTime: 12.885ms
Instance 9649f7ab3bcc5fb8:f4d6a60700000006
(host=slave6:22000):(Total: 29h59m, non-child: 21s225ms, % non-child:
0.02%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
0:57/13.92 GB
Fragment Instance Lifecycle Event Timeline: 29h59m
- Prepare Finished: 55.600ms (55.600ms)
- Open Finished: 598.215ms (542.614ms)
- First Batch Produced: 2s350ms (1s752ms)
- First Batch Sent: 2s356ms (5.853ms)
- ExecInternal Finished: 29h59m (29h59m)
MemoryUsage(34m8s): 1.37 GB, 1.37 GB, 1.36 GB, 1.24 GB, 1.04
GB, 864.64 MB, 1.37 GB, 1.40 GB, 1.40 GB, 1.38 GB, 1.31 GB, 1.17 GB,
965.56 MB, 876.58 MB, 1.32 GB, 1.39 GB, 1.39 GB, 1.38 GB, 1.23 GB,
995.12 MB, 871.65 MB, 1.35 GB, 1.40 GB, 1.38 GB, 1.24 GB, 977.74 MB,
850.26 MB, 1.27 GB, 1.29 GB, 1.30 GB, 1.31 GB, 1.16 GB, 954.46 MB,
906.32 MB, 1.24 GB, 1.34 GB, 1.30 GB, 1.31 GB, 1.36 GB, 1.29 GB, 1.02
GB, 1.02 GB, 1.32 GB, 1.30 GB, 1.31 GB, 1.38 GB, 1.33 GB, 1.23 GB,
1.14 GB, 1017.39 MB, 822.59 MB, 631.09 MB
ThreadUsage(34m8s): 8, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9,
9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9,
9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 8, 7
- AverageThreadTokens: 8.86
- BloomFilterBytes: 0
- PeakMemoryUsage: 1.40 GB (1499916960)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 1.40 GB (1499916960)
- RowsProduced: 428.25M (428247756)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 0.000ns
- TotalStorageWaitTime: 1m52s
- TotalThreadsInvoluntaryContextSwitches: 388.30K (388300)
- TotalThreadsTotalWallClockTime: 265h50m
- TotalThreadsSysTime: 1m58s
- TotalThreadsUserTime: 1h13m
- TotalThreadsVoluntaryContextSwitches: 431.32K (431323)
Buffer pool:
- AllocTime: 0.000ns
- CumulativeAllocationBytes: 0
- CumulativeAllocations: 0 (0)
- PeakReservation: 0
- PeakUnpinnedBytes: 0
- PeakUsedReservation: 0
- ReadIoBytes: 0
- ReadIoOps: 0 (0)
- ReadIoWaitTime: 0.000ns
- ReservationLimit: 0
- WriteIoBytes: 0
- WriteIoOps: 0 (0)
- WriteIoWaitTime: 0.000ns
Fragment Instance Lifecycle Timings:
- ExecTime: 29h59m
- ExecTreeExecTime: 4s703ms
- OpenTime: 542.606ms
- ExecTreeOpenTime: 32.022us
- PrepareTime: 55.377ms
- ExecTreePrepareTime: 20.617ms
KuduTableSink:(Total: 29h59m, non-child: 29h59m, % non-child: 100.00%)
- KuduApplyTimer: 29h
- NumRowErrors: 346.33M (346333617)
- PeakMemoryUsage: 20.00 MB (20971520)
- RowsProcessedRate: 3.97 K/sec
- TotalNumRows: 428.25M (428247756)
HDFS_SCAN_NODE (id=0):(Total: 4s407ms, non-child: 4s407ms, %
non-child: 100.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
0:57/13.92 GB
ExecOption: PARQUET Codegen Enabled, Codegen enabled: 57 out of 57
Hdfs Read Thread Concurrency Bucket: 0:99.81% 1:0.1895% 2:0%
3:0% 4:0% 5:0% 6:0%
File Formats: PARQUET/SNAPPY:57
BytesRead(34m8s): 1.29 GB, 1.56 GB, 1.71 GB, 1.81 GB, 1.90
GB, 2.18 GB, 3.19 GB, 3.30 GB, 3.46 GB, 3.65 GB, 3.76 GB, 3.84 GB,
3.91 GB, 4.19 GB, 5.09 GB, 5.31 GB, 5.48 GB, 5.64 GB, 5.78 GB, 5.90
GB, 6.16 GB, 7.13 GB, 7.39 GB, 7.61 GB, 7.76 GB, 7.91 GB, 8.15 GB,
9.04 GB, 9.25 GB, 9.49 GB, 9.72 GB, 9.83 GB, 9.95 GB, 10.16 GB, 10.86
GB, 11.12 GB, 11.26 GB, 11.41 GB, 11.60 GB, 11.71 GB, 11.87 GB, 12.39
GB, 13.18 GB, 13.32 GB, 13.42 GB, 13.63 GB, 13.70 GB, 13.76 GB, 13.82
GB, 13.87 GB, 13.89 GB, 13.90 GB
- FooterProcessingTime: (Avg: 368.827ms ; Min: 61.227ms ;
Max: 3s788ms ; Number of samples: 57)
- AverageHdfsReadThreadConcurrency: 0.00
- AverageScannerThreadConcurrency: 7.86
- BytesRead: 13.92 GB (14949018844)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 13.92 GB (14949018844)
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 0
- CachedFileHandlesHitCount: 2.31K (2315)
- CachedFileHandlesMissCount: 57 (57)
- CollectionItemsRead: 0 (0)
- DecompressionTime: 22s734ms
- MaxCompressedTextFileLength: 0
- NumColumns: 16 (16)
- NumDictFilteredRowGroups: 0 (0)
- NumDisksAccessed: 1 (1)
- NumRowGroups: 57 (57)
- NumScannerThreadsStarted: 8 (8)
- NumScannersWithNoReads: 0 (0)
- NumStatsFilteredRowGroups: 0 (0)
- PeakMemoryUsage: 1.38 GB (1478857724)
- PerReadThreadRawHdfsThroughput: 69.53 MB/sec
- RemoteScanRanges: 0 (0)
- RowBatchQueueGetWaitTime: 1s749ms
- RowBatchQueuePutWaitTime: 235h46m
- RowsRead: 428.25M (428247756)
- RowsReturned: 428.25M (428247756)
- RowsReturnedRate: 97.15 M/sec
- ScanRangesComplete: 57 (57)
- ScannerThreadsInvoluntaryContextSwitches: 15.99K (15992)
- ScannerThreadsTotalWallClockTime: 235h50m
- MaterializeTupleTime(*): 2m7s
- ScannerThreadsSysTime: 3s685ms
- ScannerThreadsUserTime: 2m42s
- ScannerThreadsVoluntaryContextSwitches: 419.35K (419351)
- TotalRawHdfsOpenFileTime(*): 3s380ms
- TotalRawHdfsReadTime(*): 3m25s
- TotalReadThroughput: 135.25 KB/sec
CodeGen:(Total: 405.466ms, non-child: 405.466ms, % non-child: 100.00%)
- CodegenInvoluntaryContextSwitches: 2 (2)
- CodegenTotalWallClockTime: 405.458ms
- CodegenSysTime: 0.000ns
- CodegenUserTime: 49.456ms
- CodegenVoluntaryContextSwitches: 91 (91)
- CompileTime: 170.012ms
- IrGenerationTime: 52.154ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 2.26 MB (2373832)
- NumFunctions: 19 (19)
- NumInstructions: 706 (706)
- OptimizationTime: 168.310ms
- PeakMemoryUsage: 353.00 KB (361472)
- PrepareTime: 14.493ms
Instance 9649f7ab3bcc5fb8:f4d6a60700000003
(host=slave2:22000):(Total: 28h48m, non-child: 21s449ms, % non-child:
0.02%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
0:56/13.87 GB
Fragment Instance Lifecycle Event Timeline: 28h48m
- Prepare Finished: 13.348ms (13.348ms)
- Open Finished: 348.443ms (335.094ms)
- First Batch Produced: 1s505ms (1s157ms)
- First Batch Sent: 1s508ms (3.334ms)
- ExecInternal Finished: 28h48m (28h48m)
MemoryUsage(34m8s): 1.38 GB, 1.38 GB, 1.39 GB, 1.26 GB, 1.06
GB, 777.91 MB, 1.37 GB, 1.38 GB, 1.38 GB, 1.39 GB, 1.33 GB, 1.20 GB,
985.85 MB, 781.77 MB, 1.25 GB, 1.37 GB, 1.37 GB, 1.37 GB, 1.25 GB,
1015.91 MB, 756.94 MB, 1.27 GB, 1.37 GB, 1.37 GB, 1.27 GB, 1012.70 MB,
775.19 MB, 1.25 GB, 1.41 GB, 1.41 GB, 1.30 GB, 1.01 GB, 871.44 MB,
1.34 GB, 1.38 GB, 1.38 GB, 1.38 GB, 1.37 GB, 1.31 GB, 1.14 GB, 825.57
MB, 1.32 GB, 1.40 GB, 1.40 GB, 1.37 GB, 1.28 GB, 1.17 GB, 1.02 GB,
856.05 MB, 694.00 MB
ThreadUsage(34m8s): 8, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9,
9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9,
9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 8
- AverageThreadTokens: 8.94
- BloomFilterBytes: 0
- PeakMemoryUsage: 1.41 GB (1512855058)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 1.41 GB (1512855058)
- RowsProduced: 427.11M (427110203)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 0.000ns
- TotalStorageWaitTime: 2m27s
- TotalThreadsInvoluntaryContextSwitches: 375.92K (375918)
- TotalThreadsTotalWallClockTime: 257h33m
- TotalThreadsSysTime: 2m12s
- TotalThreadsUserTime: 1h9m
- TotalThreadsVoluntaryContextSwitches: 431.39K (431386)
Buffer pool:
- AllocTime: 0.000ns
- CumulativeAllocationBytes: 0
- CumulativeAllocations: 0 (0)
- PeakReservation: 0
- PeakUnpinnedBytes: 0
- PeakUsedReservation: 0
- ReadIoBytes: 0
- ReadIoOps: 0 (0)
- ReadIoWaitTime: 0.000ns
- ReservationLimit: 0
- WriteIoBytes: 0
- WriteIoOps: 0 (0)
- WriteIoWaitTime: 0.000ns
Fragment Instance Lifecycle Timings:
- ExecTime: 28h48m
- ExecTreeExecTime: 4s366ms
- OpenTime: 335.089ms
- ExecTreeOpenTime: 40.091us
- PrepareTime: 13.257ms
- ExecTreePrepareTime: 12.776ms
KuduTableSink:(Total: 28h48m, non-child: 28h48m, % non-child: 100.00%)
- KuduApplyTimer: 27h51m
- NumRowErrors: 347.46M (347457539)
- PeakMemoryUsage: 20.00 MB (20971520)
- RowsProcessedRate: 4.12 K/sec
- TotalNumRows: 427.11M (427110203)
HDFS_SCAN_NODE (id=0):(Total: 4s001ms, non-child: 4s001ms, %
non-child: 100.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
0:56/13.87 GB
ExecOption: PARQUET Codegen Enabled, Codegen enabled: 56 out of 56
Hdfs Read Thread Concurrency Bucket: 0:99.69% 1:0.3106% 2:0%
3:0% 4:0% 5:0% 6:0%
File Formats: PARQUET/SNAPPY:56
BytesRead(34m8s): 1.31 GB, 1.58 GB, 1.74 GB, 1.85 GB, 1.94
GB, 2.08 GB, 3.22 GB, 3.31 GB, 3.48 GB, 3.67 GB, 3.79 GB, 3.88 GB,
3.94 GB, 4.08 GB, 5.02 GB, 5.30 GB, 5.48 GB, 5.66 GB, 5.80 GB, 5.91
GB, 6.00 GB, 7.04 GB, 7.37 GB, 7.61 GB, 7.77 GB, 7.90 GB, 8.01 GB,
8.95 GB, 9.39 GB, 9.61 GB, 9.77 GB, 9.88 GB, 10.17 GB, 11.12 GB, 11.26
GB, 11.40 GB, 11.56 GB, 11.66 GB, 11.73 GB, 11.82 GB, 11.97 GB, 13.08
GB, 13.48 GB, 13.60 GB, 13.68 GB, 13.74 GB, 13.80 GB, 13.85 GB, 13.87
GB, 13.88 GB
- FooterProcessingTime: (Avg: 339.758ms ; Min: 55.620ms ;
Max: 2s575ms ; Number of samples: 56)
- AverageHdfsReadThreadConcurrency: 0.00
- AverageScannerThreadConcurrency: 7.94
- BytesRead: 13.88 GB (14898474484)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 13.88 GB (14898474484)
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 0
- CachedFileHandlesHitCount: 2.29K (2294)
- CachedFileHandlesMissCount: 56 (56)
- CollectionItemsRead: 0 (0)
- DecompressionTime: 23s940ms
- MaxCompressedTextFileLength: 0
- NumColumns: 16 (16)
- NumDictFilteredRowGroups: 0 (0)
- NumDisksAccessed: 1 (1)
- NumRowGroups: 56 (56)
- NumScannerThreadsStarted: 8 (8)
- NumScannersWithNoReads: 0 (0)
- NumStatsFilteredRowGroups: 0 (0)
- PeakMemoryUsage: 1.39 GB (1491692569)
- PerReadThreadRawHdfsThroughput: 44.62 MB/sec
- RemoteScanRanges: 0 (0)
- RowBatchQueueGetWaitTime: 1s156ms
- RowBatchQueuePutWaitTime: 228h40m
- RowsRead: 427.11M (427110203)
- RowsReturned: 427.11M (427110203)
- RowsReturnedRate: 106.74 M/sec
- ScanRangesComplete: 56 (56)
- ScannerThreadsInvoluntaryContextSwitches: 7.41K (7412)
- ScannerThreadsTotalWallClockTime: 228h45m
- MaterializeTupleTime(*): 2m
- ScannerThreadsSysTime: 4s574ms
- ScannerThreadsUserTime: 2m39s
- ScannerThreadsVoluntaryContextSwitches: 418.25K (418255)
- TotalRawHdfsOpenFileTime(*): 4s930ms
- TotalRawHdfsReadTime(*): 5m18s
- TotalReadThroughput: 140.35 KB/sec
CodeGen:(Total: 316.986ms, non-child: 316.986ms, % non-child: 100.00%)
- CodegenInvoluntaryContextSwitches: 7 (7)
- CodegenTotalWallClockTime: 316.976ms
- CodegenSysTime: 19.346ms
- CodegenUserTime: 35.434ms
- CodegenVoluntaryContextSwitches: 87 (87)
- CompileTime: 104.224ms
- IrGenerationTime: 54.920ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 2.26 MB (2373832)
- NumFunctions: 19 (19)
- NumInstructions: 706 (706)
- OptimizationTime: 143.706ms
- PeakMemoryUsage: 353.00 KB (361472)
- PrepareTime: 13.284ms
Instance 9649f7ab3bcc5fb8:f4d6a60700000005
(host=slave5:22000):(Total: 28h47m, non-child: 30s821ms, % non-child:
0.03%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
0:56/13.87 GB
Fragment Instance Lifecycle Event Timeline: 28h47m
- Prepare Finished: 25.961ms (25.961ms)
- Open Finished: 92.181ms (66.220ms)
- First Batch Produced: 1s428ms (1s336ms)
- First Batch Sent: 1s432ms (3.458ms)
- ExecInternal Finished: 28h47m (28h47m)
MemoryUsage(34m8s): 1.39 GB, 1.39 GB, 1.39 GB, 1.27 GB, 1.07
GB, 786.07 MB, 1.37 GB, 1.37 GB, 1.37 GB, 1.37 GB, 1.31 GB, 1.19 GB,
973.49 MB, 778.55 MB, 1.26 GB, 1.39 GB, 1.39 GB, 1.39 GB, 1.27 GB,
1.00 GB, 812.17 MB, 1.29 GB, 1.41 GB, 1.39 GB, 1.14 GB, 881.39 MB,
1.30 GB, 1.37 GB, 1.37 GB, 1.33 GB, 1.16 GB, 910.68 MB, 880.78 MB,
1.32 GB, 1.38 GB, 1.38 GB, 1.38 GB, 1.37 GB, 1.29 GB, 1.13 GB, 843.04
MB, 1.32 GB, 1.39 GB, 1.39 GB, 1.35 GB, 1.27 GB, 1.16 GB, 1.00 GB,
848.32 MB, 709.30 MB
ThreadUsage(34m8s): 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9,
9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9,
9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 8
- AverageThreadTokens: 8.94
- BloomFilterBytes: 0
- PeakMemoryUsage: 1.41 GB (1513059939)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 1.41 GB (1513059939)
- RowsProduced: 426.83M (426832398)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 0.000ns
- TotalStorageWaitTime: 1m16s
- TotalThreadsInvoluntaryContextSwitches: 365.02K (365022)
- TotalThreadsTotalWallClockTime: 257h18m
- TotalThreadsSysTime: 1m56s
- TotalThreadsUserTime: 1h9m
- TotalThreadsVoluntaryContextSwitches: 431.61K (431609)
Buffer pool:
- AllocTime: 0.000ns
- CumulativeAllocationBytes: 0
- CumulativeAllocations: 0 (0)
- PeakReservation: 0
- PeakUnpinnedBytes: 0
- PeakUsedReservation: 0
- ReadIoBytes: 0
- ReadIoOps: 0 (0)
- ReadIoWaitTime: 0.000ns
- ReservationLimit: 0
- WriteIoBytes: 0
- WriteIoOps: 0 (0)
- WriteIoWaitTime: 0.000ns
Fragment Instance Lifecycle Timings:
- ExecTime: 28h47m
- ExecTreeExecTime: 4s226ms
- OpenTime: 66.213ms
- ExecTreeOpenTime: 30.358us
- PrepareTime: 25.760ms
- ExecTreePrepareTime: 25.484ms
KuduTableSink:(Total: 28h46m, non-child: 28h46m, % non-child: 100.00%)
- KuduApplyTimer: 27h51m
- NumRowErrors: 345.67M (345674429)
- PeakMemoryUsage: 20.00 MB (20971520)
- RowsProcessedRate: 4.12 K/sec
- TotalNumRows: 426.83M (426832398)
HDFS_SCAN_NODE (id=0):(Total: 3s937ms, non-child: 3s937ms, %
non-child: 100.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
0:56/13.87 GB
ExecOption: PARQUET Codegen Enabled, Codegen enabled: 56 out of 56
Hdfs Read Thread Concurrency Bucket: 0:99.86% 1:0.138% 2:0%
3:0% 4:0% 5:0% 6:0%
File Formats: PARQUET/SNAPPY:56
BytesRead(34m8s): 1.31 GB, 1.58 GB, 1.74 GB, 1.85 GB, 1.94
GB, 2.08 GB, 3.23 GB, 3.30 GB, 3.48 GB, 3.67 GB, 3.78 GB, 3.87 GB,
3.94 GB, 4.09 GB, 5.01 GB, 5.31 GB, 5.49 GB, 5.66 GB, 5.81 GB, 5.92
GB, 6.09 GB, 7.05 GB, 7.45 GB, 7.69 GB, 7.86 GB, 8.14 GB, 9.07 GB,
9.35 GB, 9.56 GB, 9.70 GB, 9.81 GB, 9.90 GB, 10.23 GB, 11.09 GB, 11.27
GB, 11.41 GB, 11.56 GB, 11.66 GB, 11.73 GB, 11.82 GB, 12.00 GB, 13.08
GB, 13.48 GB, 13.61 GB, 13.68 GB, 13.74 GB, 13.80 GB, 13.85 GB, 13.87
GB, 13.88 GB
- FooterProcessingTime: (Avg: 256.473ms ; Min: 49.349ms ;
Max: 2s343ms ; Number of samples: 56)
- AverageHdfsReadThreadConcurrency: 0.00
- AverageScannerThreadConcurrency: 7.94
- BytesRead: 13.88 GB (14898579078)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 13.88 GB (14898579078)
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 0
- CachedFileHandlesHitCount: 2.29K (2293)
- CachedFileHandlesMissCount: 56 (56)
- CollectionItemsRead: 0 (0)
- DecompressionTime: 22s127ms
- MaxCompressedTextFileLength: 0
- NumColumns: 16 (16)
- NumDictFilteredRowGroups: 0 (0)
- NumDisksAccessed: 1 (1)
- NumRowGroups: 56 (56)
- NumScannerThreadsStarted: 8 (8)
- NumScannersWithNoReads: 0 (0)
- NumStatsFilteredRowGroups: 0 (0)
- PeakMemoryUsage: 1.39 GB (1491831945)
- PerReadThreadRawHdfsThroughput: 100.21 MB/sec
- RemoteScanRanges: 0 (0)
- RowBatchQueueGetWaitTime: 1s334ms
- RowBatchQueuePutWaitTime: 228h26m
- RowsRead: 426.83M (426832398)
- RowsReturned: 426.83M (426832398)
- RowsReturnedRate: 108.39 M/sec
- ScanRangesComplete: 56 (56)
- ScannerThreadsInvoluntaryContextSwitches: 14.55K (14553)
- ScannerThreadsTotalWallClockTime: 228h30m
- MaterializeTupleTime(*): 2m3s
- ScannerThreadsSysTime: 4s079ms
- ScannerThreadsUserTime: 2m39s
- ScannerThreadsVoluntaryContextSwitches: 417.98K (417975)
- TotalRawHdfsOpenFileTime(*): 3s114ms
- TotalRawHdfsReadTime(*): 2m21s
- TotalReadThroughput: 140.45 KB/sec
CodeGen:(Total: 56.104ms, non-child: 56.104ms, % non-child: 100.00%)
- CodegenInvoluntaryContextSwitches: 3 (3)
- CodegenTotalWallClockTime: 56.095ms
- CodegenSysTime: 0.000ns
- CodegenUserTime: 37.520ms
- CodegenVoluntaryContextSwitches: 1 (1)
- CompileTime: 23.495ms
- IrGenerationTime: 1.462ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 2.26 MB (2373832)
- NumFunctions: 19 (19)
- NumInstructions: 706 (706)
- OptimizationTime: 15.936ms
- PeakMemoryUsage: 353.00 KB (361472)
- PrepareTime: 14.604ms
On Thu, May 10, 2018 at 10:33 AM, Mostafa Mokhtar <mm...@cloudera.com>
wrote:
> Can you share the query profile for the successful insert query?
>
> Thanks
> Mostafa
>
> On May 9, 2018, at 9:55 PM, Geetika Gupta <ge...@knoldus.in>
> wrote:
>
> Thanks, Jeszy.
>
> We build impala again with --release flag and data load was successful
> after that.
>
> But now we are facing another issue. The table in which we loaded the data
> has less number of rows. We executed the following command:
>
> insert into LINEITEM select * from PARQUETIMPALA500.LINEITEM
>
> This query was successful, but when we tried the count(*) on both the
> tables, row count was different:
>
> 0: jdbc:hive2://slave2:21050/default> select count(*) from lineitem
> . . . . . . . . . . . . . . . . . . > ;
> 536870912
>
> 0: jdbc:hive2://slave2:21050/default> select count(*) from
> parquetimpala500.lineitem;
> 3000028242
>
> Do you have any idea about this issue.
>
>
> On Mon, May 7, 2018 at 12:06 PM, Jeszy <je...@gmail.com> wrote:
>
>> Impala doesn't store the data itself, so you can switch versions
>> without rewriting data. But you don't have to do that, you would just
>> have to build impala using the -release flag (of buildall.sh) and run
>> it using the release binaries (versus the debug ones). If you would be
>> looking at performance, using the release version is highly
>> recommended anyway.
>>
>> On 7 May 2018 at 08:30, Geetika Gupta <ge...@knoldus.in> wrote:
>> > Hi Jeszy,
>> >
>> > Currently, we are using the apache impala's Github master branch code.
>> We
>> > tried using the released version but we encountered some errors related
>> to
>> > downloading of dependencies and could not complete the installation.
>> >
>> > The current version of impala we are using: 2.12
>> >
>> > We can't try with the new release as we have already loaded 500GB of
>> TPCH
>> > data on our cluster.
>> >
>> > On Mon, May 7, 2018 at 11:43 AM, Jeszy <je...@gmail.com> wrote:
>> >>
>> >> What version of Impala are you using?
>> >> DCHECKs won't be triggered if you run a release build. Looking at the
>> >> code, it should work with bad values if not for the DCHECK. Can you
>> >> try using a release build?
>> >>
>> >> On 7 May 2018 at 08:04, Geetika Gupta <ge...@knoldus.in>
>> wrote:
>> >> > Hi community,
>> >> >
>> >> > I was trying to load 500GB of TPCH data into kudu table using the
>> >> > following
>> >> > query:
>> >> >
>> >> > insert into lineitem select * from PARQUETIMPALA500.LINEITEM
>> >> >
>> >> > While executing the query for around 17 hrs it got cancelled as the
>> >> > impalad
>> >> > process of that machine got aborted. Here are the logs of the impalad
>> >> > process.
>> >> >
>> >> > impalad.ERROR
>> >> >
>> >> > Log file created at: 2018/05/06 13:40:34
>> >> > Running on machine: slave2
>> >> > Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
>> >> > E0506 13:40:34.097759 28730 logging.cc:121] stderr will be logged to
>> >> > this
>> >> > file.
>> >> > SLF4J: Class path contains multiple SLF4J bindings.
>> >> > SLF4J: Found binding in
>> >> >
>> >> > [jar:file:/root/softwares/impala/fe/target/dependency/slf4j-
>> log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>> >> > SLF4J: Found binding in
>> >> >
>> >> > [jar:file:/root/softwares/impala/testdata/target/dependency/
>> slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>> >> > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
>> >> > explanation.
>> >> > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
>> >> > 18/05/06 13:40:34 WARN util.NativeCodeLoader: Unable to load
>> >> > native-hadoop
>> >> > library for your platform... using builtin-java classes where
>> applicable
>> >> > 18/05/06 13:40:36 WARN shortcircuit.DomainSocketFactory: The
>> >> > short-circuit
>> >> > local reads feature cannot be used because libhadoop cannot be
>> loaded.
>> >> > tcmalloc: large alloc 1073741824 bytes == 0x484434000 @ 0x4135176
>> >> > 0x7fd9e9fc3929
>> >> > tcmalloc: large alloc 2147483648 bytes == 0x7fd540f18000 @ 0x4135176
>> >> > 0x7fd9e9fc3929
>> >> > F0507 09:46:12.673912 29258 error-util.cc:148] Check failed:
>> >> > log_entry.count
>> >> >> 0 (-1831809966 vs. 0)
>> >> > *** Check failure stack trace: ***
>> >> > @ 0x3fc0c0d google::LogMessage::Fail()
>> >> > @ 0x3fc24b2 google::LogMessage::SendToLog()
>> >> > @ 0x3fc05e7 google::LogMessage::Flush()
>> >> > @ 0x3fc3bae google::LogMessageFatal::~LogM
>> essageFatal()
>> >> > @ 0x1bbcb31 impala::PrintErrorMap()
>> >> > @ 0x1bbcd07 impala::PrintErrorMapToString()
>> >> > @ 0x2decbd7 impala::Coordinator::GetErrorLog()
>> >> > @ 0x1a8d634 impala::ImpalaServer::UnregisterQuery()
>> >> > @ 0x1b29264 impala::ImpalaServer::CloseOperation()
>> >> > @ 0x2c5ce86
>> >> >
>> >> > apache::hive::service::cli::thrift::TCLIServiceProcessor::pr
>> ocess_CloseOperation()
>> >> > @ 0x2c56b8c
>> >> > apache::hive::service::cli::thrift::TCLIServiceProcessor::di
>> spatchCall()
>> >> > @ 0x2c2fcb1
>> >> > impala::ImpalaHiveServer2ServiceProcessor::dispatchCall()
>> >> > @ 0x16fdb20 apache::thrift::TDispatchProce
>> ssor::process()
>> >> > @ 0x18ea6b3
>> >> > apache::thrift::server::TAcceptQueueServer::Task::run()
>> >> > @ 0x18e2181 impala::ThriftThread::RunRunnable()
>> >> > @ 0x18e3885 boost::_mfi::mf2<>::operator()()
>> >> > @ 0x18e371b boost::_bi::list3<>::operator()<>()
>> >> > @ 0x18e3467 boost::_bi::bind_t<>::operator()()
>> >> > @ 0x18e337a
>> >> > boost::detail::function::void_function_obj_invoker0<>::invoke()
>> >> > @ 0x192761c boost::function0<>::operator()()
>> >> > @ 0x1c3ebf7 impala::Thread::SuperviseThread()
>> >> > @ 0x1c470cd boost::_bi::list5<>::operator()<>()
>> >> > @ 0x1c46ff1 boost::_bi::bind_t<>::operator()()
>> >> > @ 0x1c46fb4 boost::detail::thread_data<>::run()
>> >> > @ 0x2eedb4a thread_proxy
>> >> > @ 0x7fda1dbb16ba start_thread
>> >> > @ 0x7fda1d8e741d clone
>> >> > Wrote minidump to
>> >> > /tmp/minidumps/impalad/a9113d9b-bc3d-488a-1feebf9b-47b42022.dmp
>> >> >
>> >> > impalad.FATAL
>> >> >
>> >> > Log file created at: 2018/05/07 09:46:12
>> >> > Running on machine: slave2
>> >> > Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
>> >> > F0507 09:46:12.673912 29258 error-util.cc:148] Check failed:
>> >> > log_entry.count
>> >> >> 0 (-1831809966 vs. 0)
>> >> >
>> >> > Impalad.INFO
>> >> > edentials={real_user=root}} blocked reactor thread for 34288.6us
>> >> > I0507 09:38:14.943245 29882 outbound_call.cc:288] RPC callback for
>> RPC
>> >> > call
>> >> > kudu.tserver.TabletServerService.Write -> {remote=136.243.74.42:7050
>> >> > (slave5), user_credentials={real_user=root}} blocked reactor thread
>> for
>> >> > 35859.8us
>> >> > I0507 09:38:15.942150 29882 outbound_call.cc:288] RPC callback for
>> RPC
>> >> > call
>> >> > kudu.tserver.TabletServerService.Write -> {remote=136.243.74.42:7050
>> >> > (slave5), user_credentials={real_user=root}} blocked reactor thread
>> for
>> >> > 40664.9us
>> >> > I0507 09:38:17.495046 29882 outbound_call.cc:288] RPC callback for
>> RPC
>> >> > call
>> >> > kudu.tserver.TabletServerService.Write -> {remote=136.243.74.42:7050
>> >> > (slave5), user_credentials={real_user=root}} blocked reactor thread
>> for
>> >> > 49514.6us
>> >> > I0507 09:46:12.664149 4507 coordinator.cc:783] Release admission
>> >> > control
>> >> > resources for query_id=3e4a4c646800e1d9:c859bb7f00000000
>> >> > F0507 09:46:12.673912 29258 error-util.cc:148] Check failed:
>> >> > log_entry.count
>> >> >> 0 (-1831809966 vs. 0)
>> >> > Wrote minidump to
>> >> > /tmp/minidumps/impalad/a9113d9b-bc3d-488a-1feebf9b-47b42022.dmp
>> >> >
>> >> > Note:
>> >> > We are executing the queries on 8 node cluster with the following
>> >> > configuration
>> >> > Cluster : 8 Node Cluster (48 GB RAM , 8 CPU Core and 2 TB hard-disk
>> >> > each,
>> >> > Intel(R) Core(TM) i7 CPU 950 @ 3.07GHz
>> >> >
>> >> >
>> >> > --
>> >> > Regards,
>> >> > Geetika Gupta
>> >
>> >
>> >
>> >
>> > --
>> > Regards,
>> > Geetika Gupta
>>
>
>
>
> --
> Regards,
> Geetika Gupta
>
>
--
Regards,
Geetika Gupta
Re: Issue in data loading in Impala + Kudu
Posted by Mostafa Mokhtar <mm...@cloudera.com>.
Can you share the query profile for the successful insert query?
Thanks
Mostafa
> On May 9, 2018, at 9:55 PM, Geetika Gupta <ge...@knoldus.in> wrote:
>
> Thanks, Jeszy.
>
> We build impala again with --release flag and data load was successful after that.
>
> But now we are facing another issue. The table in which we loaded the data has less number of rows. We executed the following command:
>
> insert into LINEITEM select * from PARQUETIMPALA500.LINEITEM
>
> This query was successful, but when we tried the count(*) on both the tables, row count was different:
>
> 0: jdbc:hive2://slave2:21050/default> select count(*) from lineitem
> . . . . . . . . . . . . . . . . . . > ;
> 536870912
>
> 0: jdbc:hive2://slave2:21050/default> select count(*) from parquetimpala500.lineitem;
> 3000028242
>
> Do you have any idea about this issue.
>
>
>> On Mon, May 7, 2018 at 12:06 PM, Jeszy <je...@gmail.com> wrote:
>> Impala doesn't store the data itself, so you can switch versions
>> without rewriting data. But you don't have to do that, you would just
>> have to build impala using the -release flag (of buildall.sh) and run
>> it using the release binaries (versus the debug ones). If you would be
>> looking at performance, using the release version is highly
>> recommended anyway.
>>
>> On 7 May 2018 at 08:30, Geetika Gupta <ge...@knoldus.in> wrote:
>> > Hi Jeszy,
>> >
>> > Currently, we are using the apache impala's Github master branch code. We
>> > tried using the released version but we encountered some errors related to
>> > downloading of dependencies and could not complete the installation.
>> >
>> > The current version of impala we are using: 2.12
>> >
>> > We can't try with the new release as we have already loaded 500GB of TPCH
>> > data on our cluster.
>> >
>> > On Mon, May 7, 2018 at 11:43 AM, Jeszy <je...@gmail.com> wrote:
>> >>
>> >> What version of Impala are you using?
>> >> DCHECKs won't be triggered if you run a release build. Looking at the
>> >> code, it should work with bad values if not for the DCHECK. Can you
>> >> try using a release build?
>> >>
>> >> On 7 May 2018 at 08:04, Geetika Gupta <ge...@knoldus.in> wrote:
>> >> > Hi community,
>> >> >
>> >> > I was trying to load 500GB of TPCH data into kudu table using the
>> >> > following
>> >> > query:
>> >> >
>> >> > insert into lineitem select * from PARQUETIMPALA500.LINEITEM
>> >> >
>> >> > While executing the query for around 17 hrs it got cancelled as the
>> >> > impalad
>> >> > process of that machine got aborted. Here are the logs of the impalad
>> >> > process.
>> >> >
>> >> > impalad.ERROR
>> >> >
>> >> > Log file created at: 2018/05/06 13:40:34
>> >> > Running on machine: slave2
>> >> > Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
>> >> > E0506 13:40:34.097759 28730 logging.cc:121] stderr will be logged to
>> >> > this
>> >> > file.
>> >> > SLF4J: Class path contains multiple SLF4J bindings.
>> >> > SLF4J: Found binding in
>> >> >
>> >> > [jar:file:/root/softwares/impala/fe/target/dependency/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>> >> > SLF4J: Found binding in
>> >> >
>> >> > [jar:file:/root/softwares/impala/testdata/target/dependency/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>> >> > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
>> >> > explanation.
>> >> > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
>> >> > 18/05/06 13:40:34 WARN util.NativeCodeLoader: Unable to load
>> >> > native-hadoop
>> >> > library for your platform... using builtin-java classes where applicable
>> >> > 18/05/06 13:40:36 WARN shortcircuit.DomainSocketFactory: The
>> >> > short-circuit
>> >> > local reads feature cannot be used because libhadoop cannot be loaded.
>> >> > tcmalloc: large alloc 1073741824 bytes == 0x484434000 @ 0x4135176
>> >> > 0x7fd9e9fc3929
>> >> > tcmalloc: large alloc 2147483648 bytes == 0x7fd540f18000 @ 0x4135176
>> >> > 0x7fd9e9fc3929
>> >> > F0507 09:46:12.673912 29258 error-util.cc:148] Check failed:
>> >> > log_entry.count
>> >> >> 0 (-1831809966 vs. 0)
>> >> > *** Check failure stack trace: ***
>> >> > @ 0x3fc0c0d google::LogMessage::Fail()
>> >> > @ 0x3fc24b2 google::LogMessage::SendToLog()
>> >> > @ 0x3fc05e7 google::LogMessage::Flush()
>> >> > @ 0x3fc3bae google::LogMessageFatal::~LogMessageFatal()
>> >> > @ 0x1bbcb31 impala::PrintErrorMap()
>> >> > @ 0x1bbcd07 impala::PrintErrorMapToString()
>> >> > @ 0x2decbd7 impala::Coordinator::GetErrorLog()
>> >> > @ 0x1a8d634 impala::ImpalaServer::UnregisterQuery()
>> >> > @ 0x1b29264 impala::ImpalaServer::CloseOperation()
>> >> > @ 0x2c5ce86
>> >> >
>> >> > apache::hive::service::cli::thrift::TCLIServiceProcessor::process_CloseOperation()
>> >> > @ 0x2c56b8c
>> >> > apache::hive::service::cli::thrift::TCLIServiceProcessor::dispatchCall()
>> >> > @ 0x2c2fcb1
>> >> > impala::ImpalaHiveServer2ServiceProcessor::dispatchCall()
>> >> > @ 0x16fdb20 apache::thrift::TDispatchProcessor::process()
>> >> > @ 0x18ea6b3
>> >> > apache::thrift::server::TAcceptQueueServer::Task::run()
>> >> > @ 0x18e2181 impala::ThriftThread::RunRunnable()
>> >> > @ 0x18e3885 boost::_mfi::mf2<>::operator()()
>> >> > @ 0x18e371b boost::_bi::list3<>::operator()<>()
>> >> > @ 0x18e3467 boost::_bi::bind_t<>::operator()()
>> >> > @ 0x18e337a
>> >> > boost::detail::function::void_function_obj_invoker0<>::invoke()
>> >> > @ 0x192761c boost::function0<>::operator()()
>> >> > @ 0x1c3ebf7 impala::Thread::SuperviseThread()
>> >> > @ 0x1c470cd boost::_bi::list5<>::operator()<>()
>> >> > @ 0x1c46ff1 boost::_bi::bind_t<>::operator()()
>> >> > @ 0x1c46fb4 boost::detail::thread_data<>::run()
>> >> > @ 0x2eedb4a thread_proxy
>> >> > @ 0x7fda1dbb16ba start_thread
>> >> > @ 0x7fda1d8e741d clone
>> >> > Wrote minidump to
>> >> > /tmp/minidumps/impalad/a9113d9b-bc3d-488a-1feebf9b-47b42022.dmp
>> >> >
>> >> > impalad.FATAL
>> >> >
>> >> > Log file created at: 2018/05/07 09:46:12
>> >> > Running on machine: slave2
>> >> > Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
>> >> > F0507 09:46:12.673912 29258 error-util.cc:148] Check failed:
>> >> > log_entry.count
>> >> >> 0 (-1831809966 vs. 0)
>> >> >
>> >> > Impalad.INFO
>> >> > edentials={real_user=root}} blocked reactor thread for 34288.6us
>> >> > I0507 09:38:14.943245 29882 outbound_call.cc:288] RPC callback for RPC
>> >> > call
>> >> > kudu.tserver.TabletServerService.Write -> {remote=136.243.74.42:7050
>> >> > (slave5), user_credentials={real_user=root}} blocked reactor thread for
>> >> > 35859.8us
>> >> > I0507 09:38:15.942150 29882 outbound_call.cc:288] RPC callback for RPC
>> >> > call
>> >> > kudu.tserver.TabletServerService.Write -> {remote=136.243.74.42:7050
>> >> > (slave5), user_credentials={real_user=root}} blocked reactor thread for
>> >> > 40664.9us
>> >> > I0507 09:38:17.495046 29882 outbound_call.cc:288] RPC callback for RPC
>> >> > call
>> >> > kudu.tserver.TabletServerService.Write -> {remote=136.243.74.42:7050
>> >> > (slave5), user_credentials={real_user=root}} blocked reactor thread for
>> >> > 49514.6us
>> >> > I0507 09:46:12.664149 4507 coordinator.cc:783] Release admission
>> >> > control
>> >> > resources for query_id=3e4a4c646800e1d9:c859bb7f00000000
>> >> > F0507 09:46:12.673912 29258 error-util.cc:148] Check failed:
>> >> > log_entry.count
>> >> >> 0 (-1831809966 vs. 0)
>> >> > Wrote minidump to
>> >> > /tmp/minidumps/impalad/a9113d9b-bc3d-488a-1feebf9b-47b42022.dmp
>> >> >
>> >> > Note:
>> >> > We are executing the queries on 8 node cluster with the following
>> >> > configuration
>> >> > Cluster : 8 Node Cluster (48 GB RAM , 8 CPU Core and 2 TB hard-disk
>> >> > each,
>> >> > Intel(R) Core(TM) i7 CPU 950 @ 3.07GHz
>> >> >
>> >> >
>> >> > --
>> >> > Regards,
>> >> > Geetika Gupta
>> >
>> >
>> >
>> >
>> > --
>> > Regards,
>> > Geetika Gupta
>
>
>
> --
> Regards,
> Geetika Gupta
Re: Issue in data loading in Impala + Kudu
Posted by Geetika Gupta <ge...@knoldus.in>.
Thanks, Jeszy.
We build impala again with --release flag and data load was successful
after that.
But now we are facing another issue. The table in which we loaded the data
has less number of rows. We executed the following command:
insert into LINEITEM select * from PARQUETIMPALA500.LINEITEM
This query was successful, but when we tried the count(*) on both the
tables, row count was different:
0: jdbc:hive2://slave2:21050/default> select count(*) from lineitem
. . . . . . . . . . . . . . . . . . > ;
536870912
0: jdbc:hive2://slave2:21050/default> select count(*) from
parquetimpala500.lineitem;
3000028242
Do you have any idea about this issue.
On Mon, May 7, 2018 at 12:06 PM, Jeszy <je...@gmail.com> wrote:
> Impala doesn't store the data itself, so you can switch versions
> without rewriting data. But you don't have to do that, you would just
> have to build impala using the -release flag (of buildall.sh) and run
> it using the release binaries (versus the debug ones). If you would be
> looking at performance, using the release version is highly
> recommended anyway.
>
> On 7 May 2018 at 08:30, Geetika Gupta <ge...@knoldus.in> wrote:
> > Hi Jeszy,
> >
> > Currently, we are using the apache impala's Github master branch code. We
> > tried using the released version but we encountered some errors related
> to
> > downloading of dependencies and could not complete the installation.
> >
> > The current version of impala we are using: 2.12
> >
> > We can't try with the new release as we have already loaded 500GB of TPCH
> > data on our cluster.
> >
> > On Mon, May 7, 2018 at 11:43 AM, Jeszy <je...@gmail.com> wrote:
> >>
> >> What version of Impala are you using?
> >> DCHECKs won't be triggered if you run a release build. Looking at the
> >> code, it should work with bad values if not for the DCHECK. Can you
> >> try using a release build?
> >>
> >> On 7 May 2018 at 08:04, Geetika Gupta <ge...@knoldus.in> wrote:
> >> > Hi community,
> >> >
> >> > I was trying to load 500GB of TPCH data into kudu table using the
> >> > following
> >> > query:
> >> >
> >> > insert into lineitem select * from PARQUETIMPALA500.LINEITEM
> >> >
> >> > While executing the query for around 17 hrs it got cancelled as the
> >> > impalad
> >> > process of that machine got aborted. Here are the logs of the impalad
> >> > process.
> >> >
> >> > impalad.ERROR
> >> >
> >> > Log file created at: 2018/05/06 13:40:34
> >> > Running on machine: slave2
> >> > Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
> >> > E0506 13:40:34.097759 28730 logging.cc:121] stderr will be logged to
> >> > this
> >> > file.
> >> > SLF4J: Class path contains multiple SLF4J bindings.
> >> > SLF4J: Found binding in
> >> >
> >> > [jar:file:/root/softwares/impala/fe/target/dependency/
> slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> >> > SLF4J: Found binding in
> >> >
> >> > [jar:file:/root/softwares/impala/testdata/target/
> dependency/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/
> StaticLoggerBinder.class]
> >> > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> >> > explanation.
> >> > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> >> > 18/05/06 13:40:34 WARN util.NativeCodeLoader: Unable to load
> >> > native-hadoop
> >> > library for your platform... using builtin-java classes where
> applicable
> >> > 18/05/06 13:40:36 WARN shortcircuit.DomainSocketFactory: The
> >> > short-circuit
> >> > local reads feature cannot be used because libhadoop cannot be loaded.
> >> > tcmalloc: large alloc 1073741824 bytes == 0x484434000 @ 0x4135176
> >> > 0x7fd9e9fc3929
> >> > tcmalloc: large alloc 2147483648 bytes == 0x7fd540f18000 @ 0x4135176
> >> > 0x7fd9e9fc3929
> >> > F0507 09:46:12.673912 29258 error-util.cc:148] Check failed:
> >> > log_entry.count
> >> >> 0 (-1831809966 vs. 0)
> >> > *** Check failure stack trace: ***
> >> > @ 0x3fc0c0d google::LogMessage::Fail()
> >> > @ 0x3fc24b2 google::LogMessage::SendToLog()
> >> > @ 0x3fc05e7 google::LogMessage::Flush()
> >> > @ 0x3fc3bae google::LogMessageFatal::~LogMessageFatal()
> >> > @ 0x1bbcb31 impala::PrintErrorMap()
> >> > @ 0x1bbcd07 impala::PrintErrorMapToString()
> >> > @ 0x2decbd7 impala::Coordinator::GetErrorLog()
> >> > @ 0x1a8d634 impala::ImpalaServer::UnregisterQuery()
> >> > @ 0x1b29264 impala::ImpalaServer::CloseOperation()
> >> > @ 0x2c5ce86
> >> >
> >> > apache::hive::service::cli::thrift::TCLIServiceProcessor::
> process_CloseOperation()
> >> > @ 0x2c56b8c
> >> > apache::hive::service::cli::thrift::TCLIServiceProcessor::
> dispatchCall()
> >> > @ 0x2c2fcb1
> >> > impala::ImpalaHiveServer2ServiceProcessor::dispatchCall()
> >> > @ 0x16fdb20 apache::thrift::
> TDispatchProcessor::process()
> >> > @ 0x18ea6b3
> >> > apache::thrift::server::TAcceptQueueServer::Task::run()
> >> > @ 0x18e2181 impala::ThriftThread::RunRunnable()
> >> > @ 0x18e3885 boost::_mfi::mf2<>::operator()()
> >> > @ 0x18e371b boost::_bi::list3<>::operator()<>()
> >> > @ 0x18e3467 boost::_bi::bind_t<>::operator()()
> >> > @ 0x18e337a
> >> > boost::detail::function::void_function_obj_invoker0<>::invoke()
> >> > @ 0x192761c boost::function0<>::operator()()
> >> > @ 0x1c3ebf7 impala::Thread::SuperviseThread()
> >> > @ 0x1c470cd boost::_bi::list5<>::operator()<>()
> >> > @ 0x1c46ff1 boost::_bi::bind_t<>::operator()()
> >> > @ 0x1c46fb4 boost::detail::thread_data<>::run()
> >> > @ 0x2eedb4a thread_proxy
> >> > @ 0x7fda1dbb16ba start_thread
> >> > @ 0x7fda1d8e741d clone
> >> > Wrote minidump to
> >> > /tmp/minidumps/impalad/a9113d9b-bc3d-488a-1feebf9b-47b42022.dmp
> >> >
> >> > impalad.FATAL
> >> >
> >> > Log file created at: 2018/05/07 09:46:12
> >> > Running on machine: slave2
> >> > Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
> >> > F0507 09:46:12.673912 29258 error-util.cc:148] Check failed:
> >> > log_entry.count
> >> >> 0 (-1831809966 vs. 0)
> >> >
> >> > Impalad.INFO
> >> > edentials={real_user=root}} blocked reactor thread for 34288.6us
> >> > I0507 09:38:14.943245 29882 outbound_call.cc:288] RPC callback for RPC
> >> > call
> >> > kudu.tserver.TabletServerService.Write -> {remote=136.243.74.42:7050
> >> > (slave5), user_credentials={real_user=root}} blocked reactor thread
> for
> >> > 35859.8us
> >> > I0507 09:38:15.942150 29882 outbound_call.cc:288] RPC callback for RPC
> >> > call
> >> > kudu.tserver.TabletServerService.Write -> {remote=136.243.74.42:7050
> >> > (slave5), user_credentials={real_user=root}} blocked reactor thread
> for
> >> > 40664.9us
> >> > I0507 09:38:17.495046 29882 outbound_call.cc:288] RPC callback for RPC
> >> > call
> >> > kudu.tserver.TabletServerService.Write -> {remote=136.243.74.42:7050
> >> > (slave5), user_credentials={real_user=root}} blocked reactor thread
> for
> >> > 49514.6us
> >> > I0507 09:46:12.664149 4507 coordinator.cc:783] Release admission
> >> > control
> >> > resources for query_id=3e4a4c646800e1d9:c859bb7f00000000
> >> > F0507 09:46:12.673912 29258 error-util.cc:148] Check failed:
> >> > log_entry.count
> >> >> 0 (-1831809966 vs. 0)
> >> > Wrote minidump to
> >> > /tmp/minidumps/impalad/a9113d9b-bc3d-488a-1feebf9b-47b42022.dmp
> >> >
> >> > Note:
> >> > We are executing the queries on 8 node cluster with the following
> >> > configuration
> >> > Cluster : 8 Node Cluster (48 GB RAM , 8 CPU Core and 2 TB hard-disk
> >> > each,
> >> > Intel(R) Core(TM) i7 CPU 950 @ 3.07GHz
> >> >
> >> >
> >> > --
> >> > Regards,
> >> > Geetika Gupta
> >
> >
> >
> >
> > --
> > Regards,
> > Geetika Gupta
>
--
Regards,
Geetika Gupta
Re: Issue in data loading in Impala + Kudu
Posted by Jeszy <je...@gmail.com>.
Impala doesn't store the data itself, so you can switch versions
without rewriting data. But you don't have to do that, you would just
have to build impala using the -release flag (of buildall.sh) and run
it using the release binaries (versus the debug ones). If you would be
looking at performance, using the release version is highly
recommended anyway.
On 7 May 2018 at 08:30, Geetika Gupta <ge...@knoldus.in> wrote:
> Hi Jeszy,
>
> Currently, we are using the apache impala's Github master branch code. We
> tried using the released version but we encountered some errors related to
> downloading of dependencies and could not complete the installation.
>
> The current version of impala we are using: 2.12
>
> We can't try with the new release as we have already loaded 500GB of TPCH
> data on our cluster.
>
> On Mon, May 7, 2018 at 11:43 AM, Jeszy <je...@gmail.com> wrote:
>>
>> What version of Impala are you using?
>> DCHECKs won't be triggered if you run a release build. Looking at the
>> code, it should work with bad values if not for the DCHECK. Can you
>> try using a release build?
>>
>> On 7 May 2018 at 08:04, Geetika Gupta <ge...@knoldus.in> wrote:
>> > Hi community,
>> >
>> > I was trying to load 500GB of TPCH data into kudu table using the
>> > following
>> > query:
>> >
>> > insert into lineitem select * from PARQUETIMPALA500.LINEITEM
>> >
>> > While executing the query for around 17 hrs it got cancelled as the
>> > impalad
>> > process of that machine got aborted. Here are the logs of the impalad
>> > process.
>> >
>> > impalad.ERROR
>> >
>> > Log file created at: 2018/05/06 13:40:34
>> > Running on machine: slave2
>> > Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
>> > E0506 13:40:34.097759 28730 logging.cc:121] stderr will be logged to
>> > this
>> > file.
>> > SLF4J: Class path contains multiple SLF4J bindings.
>> > SLF4J: Found binding in
>> >
>> > [jar:file:/root/softwares/impala/fe/target/dependency/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>> > SLF4J: Found binding in
>> >
>> > [jar:file:/root/softwares/impala/testdata/target/dependency/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>> > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
>> > explanation.
>> > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
>> > 18/05/06 13:40:34 WARN util.NativeCodeLoader: Unable to load
>> > native-hadoop
>> > library for your platform... using builtin-java classes where applicable
>> > 18/05/06 13:40:36 WARN shortcircuit.DomainSocketFactory: The
>> > short-circuit
>> > local reads feature cannot be used because libhadoop cannot be loaded.
>> > tcmalloc: large alloc 1073741824 bytes == 0x484434000 @ 0x4135176
>> > 0x7fd9e9fc3929
>> > tcmalloc: large alloc 2147483648 bytes == 0x7fd540f18000 @ 0x4135176
>> > 0x7fd9e9fc3929
>> > F0507 09:46:12.673912 29258 error-util.cc:148] Check failed:
>> > log_entry.count
>> >> 0 (-1831809966 vs. 0)
>> > *** Check failure stack trace: ***
>> > @ 0x3fc0c0d google::LogMessage::Fail()
>> > @ 0x3fc24b2 google::LogMessage::SendToLog()
>> > @ 0x3fc05e7 google::LogMessage::Flush()
>> > @ 0x3fc3bae google::LogMessageFatal::~LogMessageFatal()
>> > @ 0x1bbcb31 impala::PrintErrorMap()
>> > @ 0x1bbcd07 impala::PrintErrorMapToString()
>> > @ 0x2decbd7 impala::Coordinator::GetErrorLog()
>> > @ 0x1a8d634 impala::ImpalaServer::UnregisterQuery()
>> > @ 0x1b29264 impala::ImpalaServer::CloseOperation()
>> > @ 0x2c5ce86
>> >
>> > apache::hive::service::cli::thrift::TCLIServiceProcessor::process_CloseOperation()
>> > @ 0x2c56b8c
>> > apache::hive::service::cli::thrift::TCLIServiceProcessor::dispatchCall()
>> > @ 0x2c2fcb1
>> > impala::ImpalaHiveServer2ServiceProcessor::dispatchCall()
>> > @ 0x16fdb20 apache::thrift::TDispatchProcessor::process()
>> > @ 0x18ea6b3
>> > apache::thrift::server::TAcceptQueueServer::Task::run()
>> > @ 0x18e2181 impala::ThriftThread::RunRunnable()
>> > @ 0x18e3885 boost::_mfi::mf2<>::operator()()
>> > @ 0x18e371b boost::_bi::list3<>::operator()<>()
>> > @ 0x18e3467 boost::_bi::bind_t<>::operator()()
>> > @ 0x18e337a
>> > boost::detail::function::void_function_obj_invoker0<>::invoke()
>> > @ 0x192761c boost::function0<>::operator()()
>> > @ 0x1c3ebf7 impala::Thread::SuperviseThread()
>> > @ 0x1c470cd boost::_bi::list5<>::operator()<>()
>> > @ 0x1c46ff1 boost::_bi::bind_t<>::operator()()
>> > @ 0x1c46fb4 boost::detail::thread_data<>::run()
>> > @ 0x2eedb4a thread_proxy
>> > @ 0x7fda1dbb16ba start_thread
>> > @ 0x7fda1d8e741d clone
>> > Wrote minidump to
>> > /tmp/minidumps/impalad/a9113d9b-bc3d-488a-1feebf9b-47b42022.dmp
>> >
>> > impalad.FATAL
>> >
>> > Log file created at: 2018/05/07 09:46:12
>> > Running on machine: slave2
>> > Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
>> > F0507 09:46:12.673912 29258 error-util.cc:148] Check failed:
>> > log_entry.count
>> >> 0 (-1831809966 vs. 0)
>> >
>> > Impalad.INFO
>> > edentials={real_user=root}} blocked reactor thread for 34288.6us
>> > I0507 09:38:14.943245 29882 outbound_call.cc:288] RPC callback for RPC
>> > call
>> > kudu.tserver.TabletServerService.Write -> {remote=136.243.74.42:7050
>> > (slave5), user_credentials={real_user=root}} blocked reactor thread for
>> > 35859.8us
>> > I0507 09:38:15.942150 29882 outbound_call.cc:288] RPC callback for RPC
>> > call
>> > kudu.tserver.TabletServerService.Write -> {remote=136.243.74.42:7050
>> > (slave5), user_credentials={real_user=root}} blocked reactor thread for
>> > 40664.9us
>> > I0507 09:38:17.495046 29882 outbound_call.cc:288] RPC callback for RPC
>> > call
>> > kudu.tserver.TabletServerService.Write -> {remote=136.243.74.42:7050
>> > (slave5), user_credentials={real_user=root}} blocked reactor thread for
>> > 49514.6us
>> > I0507 09:46:12.664149 4507 coordinator.cc:783] Release admission
>> > control
>> > resources for query_id=3e4a4c646800e1d9:c859bb7f00000000
>> > F0507 09:46:12.673912 29258 error-util.cc:148] Check failed:
>> > log_entry.count
>> >> 0 (-1831809966 vs. 0)
>> > Wrote minidump to
>> > /tmp/minidumps/impalad/a9113d9b-bc3d-488a-1feebf9b-47b42022.dmp
>> >
>> > Note:
>> > We are executing the queries on 8 node cluster with the following
>> > configuration
>> > Cluster : 8 Node Cluster (48 GB RAM , 8 CPU Core and 2 TB hard-disk
>> > each,
>> > Intel(R) Core(TM) i7 CPU 950 @ 3.07GHz
>> >
>> >
>> > --
>> > Regards,
>> > Geetika Gupta
>
>
>
>
> --
> Regards,
> Geetika Gupta
Re: Issue in data loading in Impala + Kudu
Posted by Geetika Gupta <ge...@knoldus.in>.
Hi Jeszy,
Currently, we are using the apache impala's Github master branch code. We
tried using the released version but we encountered some errors related to
downloading of dependencies and could not complete the installation.
The current version of impala we are using: 2.12
We can't try with the new release as we have already loaded 500GB of TPCH
data on our cluster.
On Mon, May 7, 2018 at 11:43 AM, Jeszy <je...@gmail.com> wrote:
> What version of Impala are you using?
> DCHECKs won't be triggered if you run a release build. Looking at the
> code, it should work with bad values if not for the DCHECK. Can you
> try using a release build?
>
> On 7 May 2018 at 08:04, Geetika Gupta <ge...@knoldus.in> wrote:
> > Hi community,
> >
> > I was trying to load 500GB of TPCH data into kudu table using the
> following
> > query:
> >
> > insert into lineitem select * from PARQUETIMPALA500.LINEITEM
> >
> > While executing the query for around 17 hrs it got cancelled as the
> impalad
> > process of that machine got aborted. Here are the logs of the impalad
> > process.
> >
> > impalad.ERROR
> >
> > Log file created at: 2018/05/06 13:40:34
> > Running on machine: slave2
> > Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
> > E0506 13:40:34.097759 28730 logging.cc:121] stderr will be logged to this
> > file.
> > SLF4J: Class path contains multiple SLF4J bindings.
> > SLF4J: Found binding in
> > [jar:file:/root/softwares/impala/fe/target/dependency/
> slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> > SLF4J: Found binding in
> > [jar:file:/root/softwares/impala/testdata/target/
> dependency/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/
> StaticLoggerBinder.class]
> > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> > explanation.
> > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> > 18/05/06 13:40:34 WARN util.NativeCodeLoader: Unable to load
> native-hadoop
> > library for your platform... using builtin-java classes where applicable
> > 18/05/06 13:40:36 WARN shortcircuit.DomainSocketFactory: The
> short-circuit
> > local reads feature cannot be used because libhadoop cannot be loaded.
> > tcmalloc: large alloc 1073741824 bytes == 0x484434000 @ 0x4135176
> > 0x7fd9e9fc3929
> > tcmalloc: large alloc 2147483648 bytes == 0x7fd540f18000 @ 0x4135176
> > 0x7fd9e9fc3929
> > F0507 09:46:12.673912 29258 error-util.cc:148] Check failed:
> log_entry.count
> >> 0 (-1831809966 vs. 0)
> > *** Check failure stack trace: ***
> > @ 0x3fc0c0d google::LogMessage::Fail()
> > @ 0x3fc24b2 google::LogMessage::SendToLog()
> > @ 0x3fc05e7 google::LogMessage::Flush()
> > @ 0x3fc3bae google::LogMessageFatal::~LogMessageFatal()
> > @ 0x1bbcb31 impala::PrintErrorMap()
> > @ 0x1bbcd07 impala::PrintErrorMapToString()
> > @ 0x2decbd7 impala::Coordinator::GetErrorLog()
> > @ 0x1a8d634 impala::ImpalaServer::UnregisterQuery()
> > @ 0x1b29264 impala::ImpalaServer::CloseOperation()
> > @ 0x2c5ce86
> > apache::hive::service::cli::thrift::TCLIServiceProcessor::
> process_CloseOperation()
> > @ 0x2c56b8c
> > apache::hive::service::cli::thrift::TCLIServiceProcessor::dispatchCall()
> > @ 0x2c2fcb1
> > impala::ImpalaHiveServer2ServiceProcessor::dispatchCall()
> > @ 0x16fdb20 apache::thrift::TDispatchProcessor::process()
> > @ 0x18ea6b3
> > apache::thrift::server::TAcceptQueueServer::Task::run()
> > @ 0x18e2181 impala::ThriftThread::RunRunnable()
> > @ 0x18e3885 boost::_mfi::mf2<>::operator()()
> > @ 0x18e371b boost::_bi::list3<>::operator()<>()
> > @ 0x18e3467 boost::_bi::bind_t<>::operator()()
> > @ 0x18e337a
> > boost::detail::function::void_function_obj_invoker0<>::invoke()
> > @ 0x192761c boost::function0<>::operator()()
> > @ 0x1c3ebf7 impala::Thread::SuperviseThread()
> > @ 0x1c470cd boost::_bi::list5<>::operator()<>()
> > @ 0x1c46ff1 boost::_bi::bind_t<>::operator()()
> > @ 0x1c46fb4 boost::detail::thread_data<>::run()
> > @ 0x2eedb4a thread_proxy
> > @ 0x7fda1dbb16ba start_thread
> > @ 0x7fda1d8e741d clone
> > Wrote minidump to
> > /tmp/minidumps/impalad/a9113d9b-bc3d-488a-1feebf9b-47b42022.dmp
> >
> > impalad.FATAL
> >
> > Log file created at: 2018/05/07 09:46:12
> > Running on machine: slave2
> > Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
> > F0507 09:46:12.673912 29258 error-util.cc:148] Check failed:
> log_entry.count
> >> 0 (-1831809966 vs. 0)
> >
> > Impalad.INFO
> > edentials={real_user=root}} blocked reactor thread for 34288.6us
> > I0507 09:38:14.943245 29882 outbound_call.cc:288] RPC callback for RPC
> call
> > kudu.tserver.TabletServerService.Write -> {remote=136.243.74.42:7050
> > (slave5), user_credentials={real_user=root}} blocked reactor thread for
> > 35859.8us
> > I0507 09:38:15.942150 29882 outbound_call.cc:288] RPC callback for RPC
> call
> > kudu.tserver.TabletServerService.Write -> {remote=136.243.74.42:7050
> > (slave5), user_credentials={real_user=root}} blocked reactor thread for
> > 40664.9us
> > I0507 09:38:17.495046 29882 outbound_call.cc:288] RPC callback for RPC
> call
> > kudu.tserver.TabletServerService.Write -> {remote=136.243.74.42:7050
> > (slave5), user_credentials={real_user=root}} blocked reactor thread for
> > 49514.6us
> > I0507 09:46:12.664149 4507 coordinator.cc:783] Release admission control
> > resources for query_id=3e4a4c646800e1d9:c859bb7f00000000
> > F0507 09:46:12.673912 29258 error-util.cc:148] Check failed:
> log_entry.count
> >> 0 (-1831809966 vs. 0)
> > Wrote minidump to
> > /tmp/minidumps/impalad/a9113d9b-bc3d-488a-1feebf9b-47b42022.dmp
> >
> > Note:
> > We are executing the queries on 8 node cluster with the following
> > configuration
> > Cluster : 8 Node Cluster (48 GB RAM , 8 CPU Core and 2 TB hard-disk each,
> > Intel(R) Core(TM) i7 CPU 950 @ 3.07GHz
> >
> >
> > --
> > Regards,
> > Geetika Gupta
>
--
Regards,
Geetika Gupta
Re: Issue in data loading in Impala + Kudu
Posted by Jeszy <je...@gmail.com>.
What version of Impala are you using?
DCHECKs won't be triggered if you run a release build. Looking at the
code, it should work with bad values if not for the DCHECK. Can you
try using a release build?
On 7 May 2018 at 08:04, Geetika Gupta <ge...@knoldus.in> wrote:
> Hi community,
>
> I was trying to load 500GB of TPCH data into kudu table using the following
> query:
>
> insert into lineitem select * from PARQUETIMPALA500.LINEITEM
>
> While executing the query for around 17 hrs it got cancelled as the impalad
> process of that machine got aborted. Here are the logs of the impalad
> process.
>
> impalad.ERROR
>
> Log file created at: 2018/05/06 13:40:34
> Running on machine: slave2
> Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
> E0506 13:40:34.097759 28730 logging.cc:121] stderr will be logged to this
> file.
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/root/softwares/impala/fe/target/dependency/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/root/softwares/impala/testdata/target/dependency/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> 18/05/06 13:40:34 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 18/05/06 13:40:36 WARN shortcircuit.DomainSocketFactory: The short-circuit
> local reads feature cannot be used because libhadoop cannot be loaded.
> tcmalloc: large alloc 1073741824 bytes == 0x484434000 @ 0x4135176
> 0x7fd9e9fc3929
> tcmalloc: large alloc 2147483648 bytes == 0x7fd540f18000 @ 0x4135176
> 0x7fd9e9fc3929
> F0507 09:46:12.673912 29258 error-util.cc:148] Check failed: log_entry.count
>> 0 (-1831809966 vs. 0)
> *** Check failure stack trace: ***
> @ 0x3fc0c0d google::LogMessage::Fail()
> @ 0x3fc24b2 google::LogMessage::SendToLog()
> @ 0x3fc05e7 google::LogMessage::Flush()
> @ 0x3fc3bae google::LogMessageFatal::~LogMessageFatal()
> @ 0x1bbcb31 impala::PrintErrorMap()
> @ 0x1bbcd07 impala::PrintErrorMapToString()
> @ 0x2decbd7 impala::Coordinator::GetErrorLog()
> @ 0x1a8d634 impala::ImpalaServer::UnregisterQuery()
> @ 0x1b29264 impala::ImpalaServer::CloseOperation()
> @ 0x2c5ce86
> apache::hive::service::cli::thrift::TCLIServiceProcessor::process_CloseOperation()
> @ 0x2c56b8c
> apache::hive::service::cli::thrift::TCLIServiceProcessor::dispatchCall()
> @ 0x2c2fcb1
> impala::ImpalaHiveServer2ServiceProcessor::dispatchCall()
> @ 0x16fdb20 apache::thrift::TDispatchProcessor::process()
> @ 0x18ea6b3
> apache::thrift::server::TAcceptQueueServer::Task::run()
> @ 0x18e2181 impala::ThriftThread::RunRunnable()
> @ 0x18e3885 boost::_mfi::mf2<>::operator()()
> @ 0x18e371b boost::_bi::list3<>::operator()<>()
> @ 0x18e3467 boost::_bi::bind_t<>::operator()()
> @ 0x18e337a
> boost::detail::function::void_function_obj_invoker0<>::invoke()
> @ 0x192761c boost::function0<>::operator()()
> @ 0x1c3ebf7 impala::Thread::SuperviseThread()
> @ 0x1c470cd boost::_bi::list5<>::operator()<>()
> @ 0x1c46ff1 boost::_bi::bind_t<>::operator()()
> @ 0x1c46fb4 boost::detail::thread_data<>::run()
> @ 0x2eedb4a thread_proxy
> @ 0x7fda1dbb16ba start_thread
> @ 0x7fda1d8e741d clone
> Wrote minidump to
> /tmp/minidumps/impalad/a9113d9b-bc3d-488a-1feebf9b-47b42022.dmp
>
> impalad.FATAL
>
> Log file created at: 2018/05/07 09:46:12
> Running on machine: slave2
> Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
> F0507 09:46:12.673912 29258 error-util.cc:148] Check failed: log_entry.count
>> 0 (-1831809966 vs. 0)
>
> Impalad.INFO
> edentials={real_user=root}} blocked reactor thread for 34288.6us
> I0507 09:38:14.943245 29882 outbound_call.cc:288] RPC callback for RPC call
> kudu.tserver.TabletServerService.Write -> {remote=136.243.74.42:7050
> (slave5), user_credentials={real_user=root}} blocked reactor thread for
> 35859.8us
> I0507 09:38:15.942150 29882 outbound_call.cc:288] RPC callback for RPC call
> kudu.tserver.TabletServerService.Write -> {remote=136.243.74.42:7050
> (slave5), user_credentials={real_user=root}} blocked reactor thread for
> 40664.9us
> I0507 09:38:17.495046 29882 outbound_call.cc:288] RPC callback for RPC call
> kudu.tserver.TabletServerService.Write -> {remote=136.243.74.42:7050
> (slave5), user_credentials={real_user=root}} blocked reactor thread for
> 49514.6us
> I0507 09:46:12.664149 4507 coordinator.cc:783] Release admission control
> resources for query_id=3e4a4c646800e1d9:c859bb7f00000000
> F0507 09:46:12.673912 29258 error-util.cc:148] Check failed: log_entry.count
>> 0 (-1831809966 vs. 0)
> Wrote minidump to
> /tmp/minidumps/impalad/a9113d9b-bc3d-488a-1feebf9b-47b42022.dmp
>
> Note:
> We are executing the queries on 8 node cluster with the following
> configuration
> Cluster : 8 Node Cluster (48 GB RAM , 8 CPU Core and 2 TB hard-disk each,
> Intel(R) Core(TM) i7 CPU 950 @ 3.07GHz
>
>
> --
> Regards,
> Geetika Gupta