You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@impala.apache.org by Fawze Abujaber <fa...@gmail.com> on 2018/02/07 19:42:57 UTC
impala submission control
Hello All,
Will appreciate any insight of what i'm missing in my configuration to get
the impala submission control working as expected ....
a- Here is my placement Placement Rule:
1- Use the pool Specified at run time., only if the pool exists.
2- Use the pool root.[secondary group].
3- Use the pool root.default.
b- Here is the impala resource pools
Name Max Memory Max Running Queries Max Queued Queries Queue
Timeout Default Query Memory Limit Access Control
root.impiaas Unlimited Unlimited
200 1 minute(s) No
Default Custom
root.impreports Unlimited Unlimited
200 1 minute(s) No Default
Custom
C- Here is my Access control for root.impiaas pool:
Allow these users and groups to submit to this pool
Users: fawzea
The same for impreports the allowed user and groups to submit to pool is
only nira.
LDAP groups for fawzea user:
Member in groups:
- dba
- impiaas
- mysql
LDAP group for nira user
Member in groups:
- dba
- impreports
- mysql
Impala Queries UI shows:
1- select count(*) from dv_entersiteevent_prq_remote where year=2017
User: fawzea
Pool: root.impiaas
2- select count(*) from dv_entersiteevent_prq_remote where year=2017 and
month=10
User: nira
Pool: root.impreports
As shown above that the queries submitted based on the secondary group rule.
When i'm running the following with fawzea user:
set request_pool=impreports;
select count(*) from dv_entersiteevent_prq_remote where year=2017 and
month=10;
I can see that the query is running with impreports pool.
What i'm missing here in my configuration to get this works as expected.
Re: impala submission control
Posted by Tim Armstrong <ta...@cloudera.com>.
Circling back around in case anyone is reading this and wants to know the
conclusion. Fawze and I had a private conversation and it turns out that
it's possible to generate XML files like this from Cloudera Manager. I.e.
you could run into this both by generating an XML file with no ACL by hand
and through a tool like CM. I've been talking to the people who work on CM
to figure out how to make this less confusing.
On Mon, Mar 5, 2018 at 5:22 PM, Tim Armstrong <ta...@cloudera.com>
wrote:
> Hi Fawze,
> I finally had some time to dig into what was going on here. It looks
> like it's a config problem - there is no ACL for the root queue. Without
> specifying an ACL, everyone has access and and those permissions are
> inherited by all children of "root".
>
> - Tim
>
> On Thu, Feb 8, 2018 at 1:03 PM, Fawze Abujaber <fa...@gmail.com> wrote:
>
>> Sorry, but i'm able to replicate the issue on impala
>> version: 2.10.0-cdh5.13.0
>>
>> I will be happy if anyone who can try to replicate the issue on his test
>> cluster.
>>
>> On Thu, Feb 8, 2018 at 8:25 PM, Matthew Jacobs <ja...@gmail.com>
>> wrote:
>>
>>> I just filed one: https://issues.apache.org/jira/browse/IMPALA-6493
>>>
>>> On Thu, Feb 8, 2018 at 10:12 AM, Matthew Jacobs <ja...@gmail.com>
>>> wrote:
>>>
>>>> As I said, I don't see anything wrong with the configuration so there
>>>> could be a bug in Impala 2.11, likely due to IMPALA-5920. Can you file a
>>>> JIRA?
>>>>
>>>> On Thu, Feb 8, 2018 at 9:42 AM, Fawze Abujaber <fa...@gmail.com>
>>>> wrote:
>>>>
>>>>> This is not fitting my situation,
>>>>>
>>>>> I removed even the placement role of the secondary group, and other
>>>>> users can still submit to the pool that i restrict it to my user.
>>>>>
>>>>> Just thinking in loud and think the problem might be in using the
>>>>> placement rule:
>>>>>
>>>>> Use the pool Specified at run time., only if the pool exists.
>>>>> Edit
>>>>>
>>>>>
>>>>> as it it the first rule, it always successes.
>>>>>
>>>>>
>>>>> On Thu, Feb 8, 2018 at 7:12 PM, Matthew Jacobs <ja...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> The aclSubmitApps can specify users or groups, so if the user name is
>>>>>> also a group name, that can lead to confusion, e.g.
>>>>>> https://issues.apache.org/jira/browse/IMPALA-2578
>>>>>> I'm not saying that is necessarily the problem, but something to
>>>>>> check. If we still can't find anything configured incorrectly I'd
>>>>>> recommend that you file a JIRA.
>>>>>>
>>>>>> On Thu, Feb 8, 2018 at 9:05 AM, Fawze Abujaber <fa...@gmail.com>
>>>>>> wrote:
>>>>>> > Hi Matt,
>>>>>> >
>>>>>> > If this is the situation, how we can explain that when i submit the
>>>>>> query it
>>>>>> > runs on my secondary group pool and when nira do the same it runs
>>>>>> under her
>>>>>> > secondary group pool.
>>>>>> >
>>>>>> > More than this, the issue not related to LDAP group as i'm using the
>>>>>> > submission restriction to a user and not a group.
>>>>>> >
>>>>>> > On Thu, Feb 8, 2018 at 7:00 PM, Matthew Jacobs <ja...@gmail.com>
>>>>>> wrote:
>>>>>> >>
>>>>>> >> As far as I can tell from this information, your configuration
>>>>>> looks
>>>>>> >> fine. The only other potential issue I can think of is that the
>>>>>> >> secondary groups placement rule isn't picking up your LDAP groups
>>>>>> >> properly. Does the ACL behavior work if you remove the secondary
>>>>>> >> groups rule?
>>>>>> >>
>>>>>> >> On Wed, Feb 7, 2018 at 10:09 PM, Fawze Abujaber <fa...@gmail.com>
>>>>>> wrote:
>>>>>> >> > Hi Matt,
>>>>>> >> >
>>>>>> >> > v2.11.0-cdh5.14.0
>>>>>> >> >
>>>>>> >> > Query Runtime Profile:
>>>>>> >> > Query (id=c843dae48ce9b289:6ad3ffc000000000):
>>>>>> >> > Summary:
>>>>>> >> > Session ID: ca420ac477772b56:51ae460ce9e44999
>>>>>> >> > Session Type: BEESWAX
>>>>>> >> > Start Time: 2018-02-08 01:02:01.023558000
>>>>>> >> > End Time: 2018-02-08 01:02:01.501506000
>>>>>> >> > Query Type: QUERY
>>>>>> >> > Query State: FINISHED
>>>>>> >> > Query Status: OK
>>>>>> >> > Impala Version: impalad version 2.11.0-cdh5.14.0 RELEASE
>>>>>> (build
>>>>>> >> > d68206561bce6b26762d62c01a78e6cd27aa7690)
>>>>>> >> > User: fawzea
>>>>>> >> > Connected User: fawzea
>>>>>> >> > Delegated User:
>>>>>> >> > Network Address: ::ffff:10.16.144.131:50078
>>>>>> >> > Default Db: default
>>>>>> >> > Sql Statement: select count(*) from
>>>>>> dv_entersiteevent_prq_remote
>>>>>> >> > where
>>>>>> >> > year=2017 and month=10
>>>>>> >> > Coordinator: aopr-dhc001.lpdomain.com:22000
>>>>>> >> > Query Options (set by configuration): REQUEST_POOL=impreports
>>>>>> >> > Query Options (set by configuration and planner):
>>>>>> >> > REQUEST_POOL=impreports,MT_DOP=0
>>>>>> >> > Plan:
>>>>>> >> > ----------------
>>>>>> >> > Max Per-Host Resource Reservation: Memory=0B
>>>>>> >> > Per-Host Resource Estimates: Memory=21.00MB
>>>>>> >> > WARNING: The following tables are missing relevant table and/or
>>>>>> column
>>>>>> >> > statistics.
>>>>>> >> > default.dv_entersiteevent_prq_remote
>>>>>> >> >
>>>>>> >> > F01:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1
>>>>>> >> > | Per-Host Resources: mem-estimate=10.00MB mem-reservation=0B
>>>>>> >> > PLAN-ROOT SINK
>>>>>> >> > | mem-estimate=0B mem-reservation=0B
>>>>>> >> > |
>>>>>> >> > 03:AGGREGATE [FINALIZE]
>>>>>> >> > | output: count:merge(*)
>>>>>> >> > | mem-estimate=10.00MB mem-reservation=0B spill-buffer=2.00MB
>>>>>> >> > | tuple-ids=1 row-size=8B cardinality=1
>>>>>> >> > |
>>>>>> >> > 02:EXCHANGE [UNPARTITIONED]
>>>>>> >> > | mem-estimate=0B mem-reservation=0B
>>>>>> >> > | tuple-ids=1 row-size=8B cardinality=1
>>>>>> >> > |
>>>>>> >> > F00:PLAN FRAGMENT [RANDOM] hosts=5 instances=5
>>>>>> >> > Per-Host Resources: mem-estimate=11.00MB mem-reservation=0B
>>>>>> >> > 01:AGGREGATE
>>>>>> >> > | output:
>>>>>> >> > sum_init_zero(default.dv_entersiteevent_prq_remote.parquet-s
>>>>>> tats:
>>>>>> >> > num_rows)
>>>>>> >> > | mem-estimate=10.00MB mem-reservation=0B spill-buffer=2.00MB
>>>>>> >> > | tuple-ids=1 row-size=8B cardinality=1
>>>>>> >> > |
>>>>>> >> > 00:SCAN HDFS [default.dv_entersiteevent_prq_remote, RANDOM]
>>>>>> >> > partitions=31/441 files=2292 size=1.39GB
>>>>>> >> > stats-rows=unavailable extrapolated-rows=disabled
>>>>>> >> > table stats: rows=unavailable size=unavailable
>>>>>> >> > column stats: all
>>>>>> >> > mem-estimate=1.00MB mem-reservation=0B
>>>>>> >> > tuple-ids=0 row-size=8B cardinality=unavailable
>>>>>> >> > ----------------
>>>>>> >> > Estimated Per-Host Mem: 22020096
>>>>>> >> > Tables Missing Stats: default.dv_entersiteevent_prq_remote
>>>>>> >> > Per Host Min Reservation: aopr-dhc001.lpdomain.com:22000(0)
>>>>>> >> > aopr-dhc002.lpdomain.com:22000(0) aopr-dhc003.lpdomain.com:22000
>>>>>> (0)
>>>>>> >> > aopr-dhc004.lpdomain.com:22000(0) aopr-dhc005.lpdomain.com:22000
>>>>>> (0)
>>>>>> >> > Request Pool: root.impreports
>>>>>> >> > Admission result: Admitted immediately
>>>>>> >> > ExecSummary:
>>>>>> >> > Operator #Hosts Avg Time Max Time #Rows Est. #Rows
>>>>>> Peak Mem
>>>>>> >> > Est. Peak Mem Detail
>>>>>> >> >
>>>>>> >> > ------------------------------------------------------------
>>>>>> ------------------------------------------------------------------
>>>>>> >> > 03:AGGREGATE 1 222.958us 222.958us 1 1
>>>>>> 36.00 KB
>>>>>> >> > 10.00 MB FINALIZE
>>>>>> >> > 02:EXCHANGE 1 357.508ms 357.508ms 5 1
>>>>>> 0
>>>>>> >> > 0 UNPARTITIONED
>>>>>> >> > 01:AGGREGATE 5 1.027ms 1.541ms 5 1
>>>>>> 20.00 KB
>>>>>> >> > 10.00 MB
>>>>>> >> > 00:SCAN HDFS 5 282.797ms 333.715ms 2.29K -1
>>>>>> 2.62 MB
>>>>>> >> > 1.00 MB default.dv_entersiteevent_prq_remote
>>>>>> >> > Errors:
>>>>>> >> > Planner Timeline: 13.508ms
>>>>>> >> > - Analysis finished: 3.303ms (3.303ms)
>>>>>> >> > - Value transfer graph computed: 3.465ms (162.667us)
>>>>>> >> > - Single node plan created: 10.352ms (6.886ms)
>>>>>> >> > - Runtime filters computed: 10.677ms (325.364us)
>>>>>> >> > - Distributed plan created: 10.733ms (55.854us)
>>>>>> >> > - Lineage info computed: 10.898ms (164.937us)
>>>>>> >> > - Planning finished: 13.508ms (2.610ms)
>>>>>> >> > Query Timeline: 481.950ms
>>>>>> >> > - Query submitted: 79.971us (79.971us)
>>>>>> >> > - Planning finished: 33.947ms (33.867ms)
>>>>>> >> > - Submit for admission: 40.067ms (6.120ms)
>>>>>> >> > - Completed admission: 40.197ms (130.146us)
>>>>>> >> > - Ready to start on 5 backends: 40.905ms (707.852us)
>>>>>> >> > - All 5 execution backends (6 fragment instances) started:
>>>>>> >> > 46.417ms
>>>>>> >> > (5.512ms)
>>>>>> >> > - Rows available: 424.548ms (378.131ms)
>>>>>> >> > - First row fetched: 473.488ms (48.939ms)
>>>>>> >> > - Unregister query: 477.951ms (4.462ms)
>>>>>> >> > - ComputeScanRangeAssignmentTimer: 3.980ms
>>>>>> >> > ImpalaServer:
>>>>>> >> > - ClientFetchWaitTimer: 51.659ms
>>>>>> >> > - RowMaterializationTimer: 1.726ms
>>>>>> >> > Execution Profile c843dae48ce9b289:6ad3ffc000000000:(Total:
>>>>>> 385.029ms,
>>>>>> >> > non-child: 0.000ns, % non-child: 0.00%)
>>>>>> >> > Number of filters: 0
>>>>>> >> > Filter routing table:
>>>>>> >> > ID Src. Node Tgt. Node(s) Target type Partition filter
>>>>>> Pending
>>>>>> >> > (Expected) First arrived Completed Enabled
>>>>>> >> >
>>>>>> >> > ------------------------------------------------------------
>>>>>> -------------------------------------------------------
>>>>>> >> >
>>>>>> >> > Backend startup latencies: Count: 5, min / max: 3ms / 4ms,
>>>>>> 25th
>>>>>> >> > %-ile:
>>>>>> >> > 3ms, 50th %-ile: 4ms, 75th %-ile: 4ms, 90th %-ile: 4ms, 95th
>>>>>> %-ile: 4ms,
>>>>>> >> > 99.9th %-ile: 4ms
>>>>>> >> > Per Node Peak Memory Usage: aopr-dhc001.lpdomain.com:22000(2.68
>>>>>> MB)
>>>>>> >> > aopr-dhc002.lpdomain.com:22000(2.54 MB)
>>>>>> >> > aopr-dhc003.lpdomain.com:22000(2.54
>>>>>> >> > MB) aopr-dhc004.lpdomain.com:22000(2.41 MB)
>>>>>> >> > aopr-dhc005.lpdomain.com:22000(2.56 MB)
>>>>>> >> > - FiltersReceived: 0 (0)
>>>>>> >> > - FinalizationTimer: 0.000ns
>>>>>> >> > - NumBackends: 5 (5)
>>>>>> >> > - NumFragmentInstances: 6 (6)
>>>>>> >> > - NumFragments: 2 (2)
>>>>>> >> > Averaged Fragment F01:(Total: 427.530ms, non-child:
>>>>>> 49.428ms, %
>>>>>> >> > non-child: 11.56%)
>>>>>> >> > split sizes: min: 0, max: 0, avg: 0, stddev: 0
>>>>>> >> > completion times: min:428.693ms max:428.693ms mean:
>>>>>> 428.693ms
>>>>>> >> > stddev:0.000ns
>>>>>> >> > execution rates: min:0.00 /sec max:0.00 /sec mean:0.00
>>>>>> /sec
>>>>>> >> > stddev:0.00 /sec
>>>>>> >> > num instances: 1
>>>>>> >> > - AverageThreadTokens: 0.00
>>>>>> >> > - BloomFilterBytes: 0
>>>>>> >> > - PeakMemoryUsage: 44.02 KB (45072)
>>>>>> >> > - PeakReservation: 0
>>>>>> >> > - PeakUsedReservation: 0
>>>>>> >> > - PerHostPeakMemUsage: 2.68 MB (2806113)
>>>>>> >> > - RowsProduced: 1 (1)
>>>>>> >> > - TotalNetworkReceiveTime: 357.481ms
>>>>>> >> > - TotalNetworkSendTime: 0.000ns
>>>>>> >> > - TotalStorageWaitTime: 0.000ns
>>>>>> >> > - TotalThreadsInvoluntaryContextSwitches: 0 (0)
>>>>>> >> > - TotalThreadsTotalWallClockTime: 406.708ms
>>>>>> >> > - TotalThreadsSysTime: 0.000ns
>>>>>> >> > - TotalThreadsUserTime: 0.000ns
>>>>>> >> > - TotalThreadsVoluntaryContextSwitches: 7 (7)
>>>>>> >> > Fragment Instance Lifecycle Timings:
>>>>>> >> > - ExecTime: 49.055ms
>>>>>> >> > - ExecTreeExecTime: 4.251us
>>>>>> >> > - OpenTime: 357.663ms
>>>>>> >> > - ExecTreeOpenTime: 357.649ms
>>>>>> >> > - PrepareTime: 20.764ms
>>>>>> >> > - ExecTreePrepareTime: 81.759us
>>>>>> >> > PLAN_ROOT_SINK:
>>>>>> >> > - PeakMemoryUsage: 0
>>>>>> >> > CodeGen:(Total: 20.369ms, non-child: 20.369ms, % non-child:
>>>>>> >> > 100.00%)
>>>>>> >> > - CodegenTime: 0.000ns
>>>>>> >> > - CompileTime: 0.000ns
>>>>>> >> > - LoadTime: 0.000ns
>>>>>> >> > - ModuleBitcodeSize: 1.86 MB (1950880)
>>>>>> >> > - NumFunctions: 0 (0)
>>>>>> >> > - NumInstructions: 0 (0)
>>>>>> >> > - OptimizationTime: 0.000ns
>>>>>> >> > - PeakMemoryUsage: 0
>>>>>> >> > - PrepareTime: 19.685ms
>>>>>> >> > AGGREGATION_NODE (id=3):(Total: 357.731ms, non-child:
>>>>>> 222.958us, %
>>>>>> >> > non-child: 0.06%)
>>>>>> >> > - BuildTime: 7.368us
>>>>>> >> > - GetResultsTime: 0.000ns
>>>>>> >> > - HTResizeTime: 0.000ns
>>>>>> >> > - HashBuckets: 0 (0)
>>>>>> >> > - LargestPartitionPercent: 0 (0)
>>>>>> >> > - MaxPartitionLevel: 0 (0)
>>>>>> >> > - NumRepartitions: 0 (0)
>>>>>> >> > - PartitionsCreated: 0 (0)
>>>>>> >> > - PeakMemoryUsage: 36.00 KB (36864)
>>>>>> >> > - RowsRepartitioned: 0 (0)
>>>>>> >> > - RowsReturned: 1 (1)
>>>>>> >> > - RowsReturnedRate: 2.00 /sec
>>>>>> >> > - SpilledPartitions: 0 (0)
>>>>>> >> > EXCHANGE_NODE (id=2):(Total: 357.508ms, non-child:
>>>>>> 357.508ms, %
>>>>>> >> > non-child: 100.00%)
>>>>>> >> > - ConvertRowBatchTime: 8.881us
>>>>>> >> > - PeakMemoryUsage: 0
>>>>>> >> > - RowsReturned: 5 (5)
>>>>>> >> > - RowsReturnedRate: 13.00 /sec
>>>>>> >> > DataStreamReceiver:
>>>>>> >> > - BytesReceived: 80.00 B (80)
>>>>>> >> > - DeserializeRowBatchTimer: 24.165us
>>>>>> >> > - FirstBatchArrivalWaitTime: 277.194ms
>>>>>> >> > - PeakMemoryUsage: 4.02 KB (4112)
>>>>>> >> > - SendersBlockedTimer: 0.000ns
>>>>>> >> > - SendersBlockedTotalTimer(*): 0.000ns
>>>>>> >> > Coordinator Fragment F01:
>>>>>> >> > Instance c843dae48ce9b289:6ad3ffc000000000
>>>>>> >> > (host=aopr-dhc001.lpdomain.com:22000):(Total: 427.530ms,
>>>>>> non-child:
>>>>>> >> > 49.428ms, % non-child: 11.56%)
>>>>>> >> > - AverageThreadTokens: 0.00
>>>>>> >> > - BloomFilterBytes: 0
>>>>>> >> > - PeakMemoryUsage: 44.02 KB (45072)
>>>>>> >> > - PeakReservation: 0
>>>>>> >> > - PeakUsedReservation: 0
>>>>>> >> > - PerHostPeakMemUsage: 2.68 MB (2806113)
>>>>>> >> > - RowsProduced: 1 (1)
>>>>>> >> > - TotalNetworkReceiveTime: 357.481ms
>>>>>> >> > - TotalNetworkSendTime: 0.000ns
>>>>>> >> > - TotalStorageWaitTime: 0.000ns
>>>>>> >> > - TotalThreadsInvoluntaryContextSwitches: 0 (0)
>>>>>> >> > - TotalThreadsTotalWallClockTime: 406.708ms
>>>>>> >> > - TotalThreadsSysTime: 0.000ns
>>>>>> >> > - TotalThreadsUserTime: 0.000ns
>>>>>> >> > - TotalThreadsVoluntaryContextSwitches: 7 (7)
>>>>>> >> > Fragment Instance Lifecycle Timings:
>>>>>> >> > - ExecTime: 49.055ms
>>>>>> >> > - ExecTreeExecTime: 4.251us
>>>>>> >> > - OpenTime: 357.663ms
>>>>>> >> > - ExecTreeOpenTime: 357.649ms
>>>>>> >> > - PrepareTime: 20.764ms
>>>>>> >> > - ExecTreePrepareTime: 81.759us
>>>>>> >> > PLAN_ROOT_SINK:
>>>>>> >> > - PeakMemoryUsage: 0
>>>>>> >> > CodeGen:(Total: 20.369ms, non-child: 20.369ms, %
>>>>>> non-child:
>>>>>> >> > 100.00%)
>>>>>> >> > - CodegenTime: 0.000ns
>>>>>> >> > - CompileTime: 0.000ns
>>>>>> >> > - LoadTime: 0.000ns
>>>>>> >> > - ModuleBitcodeSize: 1.86 MB (1950880)
>>>>>> >> > - NumFunctions: 0 (0)
>>>>>> >> > - NumInstructions: 0 (0)
>>>>>> >> > - OptimizationTime: 0.000ns
>>>>>> >> > - PeakMemoryUsage: 0
>>>>>> >> > - PrepareTime: 19.685ms
>>>>>> >> > AGGREGATION_NODE (id=3):(Total: 357.731ms, non-child:
>>>>>> 222.958us,
>>>>>> >> > %
>>>>>> >> > non-child: 0.06%)
>>>>>> >> > - BuildTime: 7.368us
>>>>>> >> > - GetResultsTime: 0.000ns
>>>>>> >> > - HTResizeTime: 0.000ns
>>>>>> >> > - HashBuckets: 0 (0)
>>>>>> >> > - LargestPartitionPercent: 0 (0)
>>>>>> >> > - MaxPartitionLevel: 0 (0)
>>>>>> >> > - NumRepartitions: 0 (0)
>>>>>> >> > - PartitionsCreated: 0 (0)
>>>>>> >> > - PeakMemoryUsage: 36.00 KB (36864)
>>>>>> >> > - RowsRepartitioned: 0 (0)
>>>>>> >> > - RowsReturned: 1 (1)
>>>>>> >> > - RowsReturnedRate: 2.00 /sec
>>>>>> >> > - SpilledPartitions: 0 (0)
>>>>>> >> > EXCHANGE_NODE (id=2):(Total: 357.508ms, non-child:
>>>>>> 357.508ms, %
>>>>>> >> > non-child: 100.00%)
>>>>>> >> > - ConvertRowBatchTime: 8.881us
>>>>>> >> > - PeakMemoryUsage: 0
>>>>>> >> > - RowsReturned: 5 (5)
>>>>>> >> > - RowsReturnedRate: 13.00 /sec
>>>>>> >> > DataStreamReceiver:
>>>>>> >> > - BytesReceived: 80.00 B (80)
>>>>>> >> > - DeserializeRowBatchTimer: 24.165us
>>>>>> >> > - FirstBatchArrivalWaitTime: 277.194ms
>>>>>> >> > - PeakMemoryUsage: 4.02 KB (4112)
>>>>>> >> > - SendersBlockedTimer: 0.000ns
>>>>>> >> > - SendersBlockedTotalTimer(*): 0.000ns
>>>>>> >> > Averaged Fragment F00:(Total: 327.360ms, non-child: 1.578ms,
>>>>>> %
>>>>>> >> > non-child: 0.48%)
>>>>>> >> > split sizes: min: 283.31 MB, max: 284.33 MB, avg: 284.04
>>>>>> MB,
>>>>>> >> > stddev:
>>>>>> >> > 382.00 KB
>>>>>> >> > completion times: min:299.457ms max:378.950ms mean:
>>>>>> 328.881ms
>>>>>> >> > stddev:31.507ms
>>>>>> >> > execution rates: min:750.18 MB/sec max:949.01 MB/sec
>>>>>> mean:871.27
>>>>>> >> > MB/sec stddev:79.74 MB/sec
>>>>>> >> > num instances: 5
>>>>>> >> > - AverageThreadTokens: 15.00
>>>>>> >> > - BloomFilterBytes: 0
>>>>>> >> > - PeakMemoryUsage: 2.54 MB (2666852)
>>>>>> >> > - PeakReservation: 0
>>>>>> >> > - PeakUsedReservation: 0
>>>>>> >> > - PerHostPeakMemUsage: 2.55 MB (2669309)
>>>>>> >> > - RowsProduced: 1 (1)
>>>>>> >> > - TotalNetworkReceiveTime: 0.000ns
>>>>>> >> > - TotalNetworkSendTime: 303.541us
>>>>>> >> > - TotalStorageWaitTime: 4s821ms
>>>>>> >> > - TotalThreadsInvoluntaryContextSwitches: 104 (104)
>>>>>> >> > - TotalThreadsTotalWallClockTime: 6s835ms
>>>>>> >> > - TotalThreadsSysTime: 31.784ms
>>>>>> >> > - TotalThreadsUserTime: 682.084ms
>>>>>> >> > - TotalThreadsVoluntaryContextSwitches: 2.12K (2121)
>>>>>> >> > Fragment Instance Lifecycle Timings:
>>>>>> >> > - ExecTime: 500.816us
>>>>>> >> > - ExecTreeExecTime: 8.200us
>>>>>> >> > - OpenTime: 302.848ms
>>>>>> >> > - ExecTreeOpenTime: 281.309ms
>>>>>> >> > - PrepareTime: 23.946ms
>>>>>> >> > - ExecTreePrepareTime: 2.513ms
>>>>>> >> > DataStreamSender (dst_id=2):(Total: 269.902us, non-child:
>>>>>> >> > 269.902us, %
>>>>>> >> > non-child: 100.00%)
>>>>>> >> > - BytesSent: 16.00 B (16)
>>>>>> >> > - NetworkThroughput(*): 373.13 KB/sec
>>>>>> >> > - OverallThroughput: 63.53 KB/sec
>>>>>> >> > - PeakMemoryUsage: 16.00 KB (16384)
>>>>>> >> > - RowsReturned: 1 (1)
>>>>>> >> > - SerializeBatchTime: 14.525us
>>>>>> >> > - TransmitDataRPCTime: 43.122us
>>>>>> >> > - UncompressedRowBatchSize: 16.00 B (16)
>>>>>> >> > CodeGen:(Total: 41.687ms, non-child: 41.687ms, % non-child:
>>>>>> >> > 100.00%)
>>>>>> >> > - CodegenTime: 1.194ms
>>>>>> >> > - CompileTime: 7.360ms
>>>>>> >> > - LoadTime: 0.000ns
>>>>>> >> > - ModuleBitcodeSize: 1.86 MB (1950880)
>>>>>> >> > - NumFunctions: 15 (15)
>>>>>> >> > - NumInstructions: 214 (214)
>>>>>> >> > - OptimizationTime: 13.529ms
>>>>>> >> > - PeakMemoryUsage: 107.00 KB (109568)
>>>>>> >> > - PrepareTime: 20.084ms
>>>>>> >> > AGGREGATION_NODE (id=1):(Total: 283.825ms, non-child:
>>>>>> 1.027ms, %
>>>>>> >> > non-child: 0.36%)
>>>>>> >> > - BuildTime: 1.259ms
>>>>>> >> > - GetResultsTime: 0.000ns
>>>>>> >> > - HTResizeTime: 0.000ns
>>>>>> >> > - HashBuckets: 0 (0)
>>>>>> >> > - LargestPartitionPercent: 0 (0)
>>>>>> >> > - MaxPartitionLevel: 0 (0)
>>>>>> >> > - NumRepartitions: 0 (0)
>>>>>> >> > - PartitionsCreated: 0 (0)
>>>>>> >> > - PeakMemoryUsage: 20.00 KB (20480)
>>>>>> >> > - RowsRepartitioned: 0 (0)
>>>>>> >> > - RowsReturned: 1 (1)
>>>>>> >> > - RowsReturnedRate: 2.00 /sec
>>>>>> >> > - SpilledPartitions: 0 (0)
>>>>>> >> > HDFS_SCAN_NODE (id=0):(Total: 282.797ms, non-child:
>>>>>> 282.797ms, %
>>>>>> >> > non-child: 100.00%)
>>>>>> >> > - AverageHdfsReadThreadConcurrency: 0.00
>>>>>> >> > - AverageScannerThreadConcurrency: 14.40
>>>>>> >> > - BytesRead: 44.52 MB (46686028)
>>>>>> >> > - BytesReadDataNodeCache: 0
>>>>>> >> > - BytesReadLocal: 44.52 MB (46686028)
>>>>>> >> > - BytesReadRemoteUnexpected: 0
>>>>>> >> > - BytesReadShortCircuit: 44.52 MB (46686028)
>>>>>> >> > - CachedFileHandlesHitCount: 0 (0)
>>>>>> >> > - CachedFileHandlesMissCount: 458 (458)
>>>>>> >> > - CollectionItemsRead: 0 (0)
>>>>>> >> > - DecompressionTime: 0.000ns
>>>>>> >> > - MaxCompressedTextFileLength: 0
>>>>>> >> > - NumColumns: 0 (0)
>>>>>> >> > - NumDictFilteredRowGroups: 0 (0)
>>>>>> >> > - NumDisksAccessed: 12 (12)
>>>>>> >> > - NumRowGroups: 458 (458)
>>>>>> >> > - NumScannerThreadsStarted: 24 (24)
>>>>>> >> > - NumScannersWithNoReads: 0 (0)
>>>>>> >> > - NumStatsFilteredRowGroups: 0 (0)
>>>>>> >> > - PeakMemoryUsage: 2.50 MB (2625539)
>>>>>> >> > - PerReadThreadRawHdfsThroughput: 1.29 GB/sec
>>>>>> >> > - RemoteScanRanges: 0 (0)
>>>>>> >> > - RowBatchQueueGetWaitTime: 269.787ms
>>>>>> >> > - RowBatchQueuePutWaitTime: 0.000ns
>>>>>> >> > - RowsRead: 0 (0)
>>>>>> >> > - RowsReturned: 458 (458)
>>>>>> >> > - RowsReturnedRate: 1.64 K/sec
>>>>>> >> > - ScanRangesComplete: 458 (458)
>>>>>> >> > - ScannerThreadsInvoluntaryContextSwitches: 99 (99)
>>>>>> >> > - ScannerThreadsTotalWallClockTime: 6s532ms
>>>>>> >> > - MaterializeTupleTime(*): 70.664us
>>>>>> >> > - ScannerThreadsSysTime: 26.184ms
>>>>>> >> > - ScannerThreadsUserTime: 650.089ms
>>>>>> >> > - ScannerThreadsVoluntaryContextSwitches: 1.67K (1674)
>>>>>> >> > - TotalRawHdfsReadTime(*): 34.416ms
>>>>>> >> > - TotalReadThroughput: 36.86 MB/sec
>>>>>> >> > Fragment F00:
>>>>>> >> > Instance c843dae48ce9b289:6ad3ffc000000004
>>>>>> >> > (host=aopr-dhc004.lpdomain.com:22000):(Total: 377.384ms,
>>>>>> non-child:
>>>>>> >> > 1.619ms,
>>>>>> >> > % non-child: 0.43%)
>>>>>> >> > Hdfs split stats (<volume id>:<# splits>/<split
>>>>>> lengths>):
>>>>>> >> > 0:36/21.95 MB 11:33
>>>>>> <https://maps.google.com/?q=MB+11:33&entry=gmail&source=g>/19.87 MB
>>>>>> 9:27 <https://maps.google.com/?q=MB+9:27&entry=gmail&source=g>/15.16 MB
>>>>>> 5:44 <https://maps.google.com/?q=MB+5:44&entry=gmail&source=g>/28.07 MB
>>>>>> 6:56 <https://maps.google.com/?q=MB+6:56&entry=gmail&source=g>/33.32
>>>>>> MB
>>>>>> >> > 1:38/22.24 MB 10:44
>>>>>> <https://maps.google.com/?q=MB+10:44&entry=gmail&source=g>/27.90 MB
>>>>>> 3:34 <https://maps.google.com/?q=MB+3:34&entry=gmail&source=g>/20.26 MB
>>>>>> 2:45 <https://maps.google.com/?q=MB+2:45&entry=gmail&source=g>/28.21 MB
>>>>>> 7:31 <https://maps.google.com/?q=MB+7:31&entry=gmail&source=g>/19.33
>>>>>> MB
>>>>>> >> > 4:42/26.73 MB 8:34
>>>>>> <https://maps.google.com/?q=MB+8:34&entry=gmail&source=g>/21.25 MB
>>>>>> >> > MemoryUsage(500.000ms): 1.91 MB
>>>>>> >> > ThreadUsage(500.000ms): 25
>>>>>> >> > - AverageThreadTokens: 25.00
>>>>>> >> > - BloomFilterBytes: 0
>>>>>> >> > - PeakMemoryUsage: 2.41 MB (2531689)
>>>>>> >> > - PeakReservation: 0
>>>>>> >> > - PeakUsedReservation: 0
>>>>>> >> > - PerHostPeakMemUsage: 2.41 MB (2531689)
>>>>>> >> > - RowsProduced: 1 (1)
>>>>>> >> > - TotalNetworkReceiveTime: 0.000ns
>>>>>> >> > - TotalNetworkSendTime: 382.629us
>>>>>> >> > - TotalStorageWaitTime: 5s790ms
>>>>>> >> > - TotalThreadsInvoluntaryContextSwitches: 115 (115)
>>>>>> >> > - TotalThreadsTotalWallClockTime: 8s122ms
>>>>>> >> > - TotalThreadsSysTime: 33.984ms
>>>>>> >> > - TotalThreadsUserTime: 702.879ms
>>>>>> >> > - TotalThreadsVoluntaryContextSwitches: 2.04K (2037)
>>>>>> >> > Fragment Instance Lifecycle Timings:
>>>>>> >> > - ExecTime: 604.005us
>>>>>> >> > - ExecTreeExecTime: 8.338us
>>>>>> >> > - OpenTime: 352.927ms
>>>>>> >> > - ExecTreeOpenTime: 331.921ms
>>>>>> >> > - PrepareTime: 23.783ms
>>>>>> >> > - ExecTreePrepareTime: 2.559ms
>>>>>> >> > DataStreamSender (dst_id=2):(Total: 331.818us, non-child:
>>>>>> >> > 331.818us,
>>>>>> >> > % non-child: 100.00%)
>>>>>> >> > - BytesSent: 16.00 B (16)
>>>>>> >> > - NetworkThroughput(*): 289.00 KB/sec
>>>>>> >> > - OverallThroughput: 47.09 KB/sec
>>>>>> >> > - PeakMemoryUsage: 16.00 KB (16384)
>>>>>> >> > - RowsReturned: 1 (1)
>>>>>> >> > - SerializeBatchTime: 14.466us
>>>>>> >> > - TransmitDataRPCTime: 54.066us
>>>>>> >> > - UncompressedRowBatchSize: 16.00 B (16)
>>>>>> >> > CodeGen:(Total: 40.950ms, non-child: 40.950ms, %
>>>>>> non-child:
>>>>>> >> > 100.00%)
>>>>>> >> > - CodegenTime: 1.122ms
>>>>>> >> > - CompileTime: 7.318ms
>>>>>> >> > - LoadTime: 0.000ns
>>>>>> >> > - ModuleBitcodeSize: 1.86 MB (1950880)
>>>>>> >> > - NumFunctions: 15 (15)
>>>>>> >> > - NumInstructions: 214 (214)
>>>>>> >> > - OptimizationTime: 13.106ms
>>>>>> >> > - PeakMemoryUsage: 107.00 KB (109568)
>>>>>> >> > - PrepareTime: 19.773ms
>>>>>> >> > AGGREGATION_NODE (id=1):(Total: 334.482ms, non-child:
>>>>>> 766.527us,
>>>>>> >> > %
>>>>>> >> > non-child: 0.23%)
>>>>>> >> > ExecOption: Codegen Enabled
>>>>>> >> > - BuildTime: 1.192ms
>>>>>> >> > - GetResultsTime: 0.000ns
>>>>>> >> > - HTResizeTime: 0.000ns
>>>>>> >> > - HashBuckets: 0 (0)
>>>>>> >> > - LargestPartitionPercent: 0 (0)
>>>>>> >> > - MaxPartitionLevel: 0 (0)
>>>>>> >> > - NumRepartitions: 0 (0)
>>>>>> >> > - PartitionsCreated: 0 (0)
>>>>>> >> > - PeakMemoryUsage: 20.00 KB (20480)
>>>>>> >> > - RowsRepartitioned: 0 (0)
>>>>>> >> > - RowsReturned: 1 (1)
>>>>>> >> > - RowsReturnedRate: 2.00 /sec
>>>>>> >> > - SpilledPartitions: 0 (0)
>>>>>> >> > HDFS_SCAN_NODE (id=0):(Total: 333.715ms, non-child:
>>>>>> 333.715ms, %
>>>>>> >> > non-child: 100.00%)
>>>>>> >> > Hdfs split stats (<volume id>:<# splits>/<split
>>>>>> lengths>):
>>>>>> >> > 0:36/21.95 MB 11:33
>>>>>> <https://maps.google.com/?q=MB+11:33&entry=gmail&source=g>/19.87 MB
>>>>>> 9:27 <https://maps.google.com/?q=MB+9:27&entry=gmail&source=g>/15.16 MB
>>>>>> 5:44 <https://maps.google.com/?q=MB+5:44&entry=gmail&source=g>/28.07 MB
>>>>>> 6:56 <https://maps.google.com/?q=MB+6:56&entry=gmail&source=g>/33.32
>>>>>> MB
>>>>>> >> > 1:38/22.24 MB 10:44
>>>>>> <https://maps.google.com/?q=MB+10:44&entry=gmail&source=g>/27.90 MB
>>>>>> 3:34 <https://maps.google.com/?q=MB+3:34&entry=gmail&source=g>/20.26 MB
>>>>>> 2:45 <https://maps.google.com/?q=MB+2:45&entry=gmail&source=g>/28.21 MB
>>>>>> 7:31 <https://maps.google.com/?q=MB+7:31&entry=gmail&source=g>/19.33
>>>>>> MB
>>>>>> >> > 4:42/26.73 MB 8:34
>>>>>> <https://maps.google.com/?q=MB+8:34&entry=gmail&source=g>/21.25 MB
>>>>>> >> > ExecOption: PARQUET Codegen Enabled, Codegen enabled:
>>>>>> 464 out
>>>>>> >> > of
>>>>>> >> > 464
>>>>>> >> > Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0%
>>>>>> 3:0%
>>>>>> >> > 4:0%
>>>>>> >> > 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0%
>>>>>> 16:0%
>>>>>> >> > File Formats: PARQUET/Unknown(Skipped):464
>>>>>> >> > BytesRead(500.000ms): 12.27 MB
>>>>>> >> > - FooterProcessingTime: (Avg: 13.555ms ; Min: 5.433ms
>>>>>> ; Max:
>>>>>> >> > 62.957ms ; Number of samples: 464)
>>>>>> >> > - AverageHdfsReadThreadConcurrency: 0.00
>>>>>> >> > - AverageScannerThreadConcurrency: 24.00
>>>>>> >> > - BytesRead: 45.01 MB (47193988)
>>>>>> >> > - BytesReadDataNodeCache: 0
>>>>>> >> > - BytesReadLocal: 45.01 MB (47193988)
>>>>>> >> > - BytesReadRemoteUnexpected: 0
>>>>>> >> > - BytesReadShortCircuit: 45.01 MB (47193988)
>>>>>> >> > - CachedFileHandlesHitCount: 0 (0)
>>>>>> >> > - CachedFileHandlesMissCount: 464 (464)
>>>>>> >> > - CollectionItemsRead: 0 (0)
>>>>>> >> > - DecompressionTime: 0.000ns
>>>>>> >> > - MaxCompressedTextFileLength: 0
>>>>>> >> > - NumColumns: 0 (0)
>>>>>> >> > - NumDictFilteredRowGroups: 0 (0)
>>>>>> >> > - NumDisksAccessed: 12 (12)
>>>>>> >> > - NumRowGroups: 464 (464)
>>>>>> >> > - NumScannerThreadsStarted: 24 (24)
>>>>>> >> > - NumScannersWithNoReads: 0 (0)
>>>>>> >> > - NumStatsFilteredRowGroups: 0 (0)
>>>>>> >> > - PeakMemoryUsage: 2.38 MB (2490376)
>>>>>> >> > - PerReadThreadRawHdfsThroughput: 1016.44 MB/sec
>>>>>> >> > - RemoteScanRanges: 0 (0)
>>>>>> >> > - RowBatchQueueGetWaitTime: 320.860ms
>>>>>> >> > - RowBatchQueuePutWaitTime: 0.000ns
>>>>>> >> > - RowsRead: 0 (0)
>>>>>> >> > - RowsReturned: 464 (464)
>>>>>> >> > - RowsReturnedRate: 1.39 K/sec
>>>>>> >> > - ScanRangesComplete: 464 (464)
>>>>>> >> > - ScannerThreadsInvoluntaryContextSwitches: 108 (108)
>>>>>> >> > - ScannerThreadsTotalWallClockTime: 7s769ms
>>>>>> >> > - MaterializeTupleTime(*): 75.514us
>>>>>> >> > - ScannerThreadsSysTime: 29.984ms
>>>>>> >> > - ScannerThreadsUserTime: 669.884ms
>>>>>> >> > - ScannerThreadsVoluntaryContextSwitches: 1.60K
>>>>>> (1597)
>>>>>> >> > - TotalRawHdfsReadTime(*): 44.279ms
>>>>>> >> > - TotalReadThroughput: 24.53 MB/sec
>>>>>> >> > Instance c843dae48ce9b289:6ad3ffc000000002
>>>>>> >> > (host=aopr-dhc003.lpdomain.com:22000):(Total: 351.014ms,
>>>>>> non-child:
>>>>>> >> > 1.641ms,
>>>>>> >> > % non-child: 0.47%)
>>>>>> >> > Hdfs split stats (<volume id>:<# splits>/<split
>>>>>> lengths>):
>>>>>> >> > 0:35/20.89 MB 7:44
>>>>>> <https://maps.google.com/?q=MB+7:44&entry=gmail&source=g>/26.64 MB
>>>>>> 11:27 <https://maps.google.com/?q=MB+11:27&entry=gmail&source=g>/17.21
>>>>>> MB 9:32 <https://maps.google.com/?q=MB+9:32&entry=gmail&source=g>/18.61
>>>>>> MB 5:25 <https://maps.google.com/?q=MB+5:25&entry=gmail&source=g>/14.56
>>>>>> MB
>>>>>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+2:38&entry=gmail&source=g>
>>>>>> >> > 2:38
>>>>>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+2:38&entry=gmail&source=g>/24.32
>>>>>> MB 6:31 <https://maps.google.com/?q=MB+6:31&entry=gmail&source=g>/19.85
>>>>>> MB 8:50 <https://maps.google.com/?q=MB+8:50&entry=gmail&source=g>/30.76
>>>>>> MB 10:41 <https://maps.google.com/?q=MB+10:41&entry=gmail&source=g>/25.92
>>>>>> MB 3:31 <https://maps.google.com/?q=MB+3:31&entry=gmail&source=g>/19.78
>>>>>> MB
>>>>>> >> > 4:60/37.59 MB 1:44/27.19 MB
>>>>>> >> > MemoryUsage(500.000ms): 1.16 MB
>>>>>> >> > ThreadUsage(500.000ms): 25
>>>>>> >> > - AverageThreadTokens: 25.00
>>>>>> >> > - BloomFilterBytes: 0
>>>>>> >> > - PeakMemoryUsage: 2.54 MB (2662753)
>>>>>> >> > - PeakReservation: 0
>>>>>> >> > - PeakUsedReservation: 0
>>>>>> >> > - PerHostPeakMemUsage: 2.54 MB (2662753)
>>>>>> >> > - RowsProduced: 1 (1)
>>>>>> >> > - TotalNetworkReceiveTime: 0.000ns
>>>>>> >> > - TotalNetworkSendTime: 342.746us
>>>>>> >> > - TotalStorageWaitTime: 4s796ms
>>>>>> >> > - TotalThreadsInvoluntaryContextSwitches: 93 (93)
>>>>>> >> > - TotalThreadsTotalWallClockTime: 7s286ms
>>>>>> >> > - TotalThreadsSysTime: 27.988ms
>>>>>> >> > - TotalThreadsUserTime: 671.887ms
>>>>>> >> > - TotalThreadsVoluntaryContextSwitches: 2.08K (2079)
>>>>>> >> > Fragment Instance Lifecycle Timings:
>>>>>> >> > - ExecTime: 524.006us
>>>>>> >> > - ExecTreeExecTime: 7.920us
>>>>>> >> > - OpenTime: 327.802ms
>>>>>> >> > - ExecTreeOpenTime: 307.154ms
>>>>>> >> > - PrepareTime: 22.617ms
>>>>>> >> > - ExecTreePrepareTime: 2.650ms
>>>>>> >> > DataStreamSender (dst_id=2):(Total: 348.990us, non-child:
>>>>>> >> > 348.990us,
>>>>>> >> > % non-child: 100.00%)
>>>>>> >> > - BytesSent: 16.00 B (16)
>>>>>> >> > - NetworkThroughput(*): 354.47 KB/sec
>>>>>> >> > - OverallThroughput: 44.77 KB/sec
>>>>>> >> > - PeakMemoryUsage: 16.00 KB (16384)
>>>>>> >> > - RowsReturned: 1 (1)
>>>>>> >> > - SerializeBatchTime: 15.602us
>>>>>> >> > - TransmitDataRPCTime: 44.080us
>>>>>> >> > - UncompressedRowBatchSize: 16.00 B (16)
>>>>>> >> > CodeGen:(Total: 39.215ms, non-child: 39.215ms, %
>>>>>> non-child:
>>>>>> >> > 100.00%)
>>>>>> >> > - CodegenTime: 1.183ms
>>>>>> >> > - CompileTime: 7.168ms
>>>>>> >> > - LoadTime: 0.000ns
>>>>>> >> > - ModuleBitcodeSize: 1.86 MB (1950880)
>>>>>> >> > - NumFunctions: 15 (15)
>>>>>> >> > - NumInstructions: 214 (214)
>>>>>> >> > - OptimizationTime: 12.929ms
>>>>>> >> > - PeakMemoryUsage: 107.00 KB (109568)
>>>>>> >> > - PrepareTime: 18.453ms
>>>>>> >> > AGGREGATION_NODE (id=1):(Total: 309.807ms, non-child:
>>>>>> 948.677us,
>>>>>> >> > %
>>>>>> >> > non-child: 0.31%)
>>>>>> >> > ExecOption: Codegen Enabled
>>>>>> >> > - BuildTime: 1.552ms
>>>>>> >> > - GetResultsTime: 0.000ns
>>>>>> >> > - HTResizeTime: 0.000ns
>>>>>> >> > - HashBuckets: 0 (0)
>>>>>> >> > - LargestPartitionPercent: 0 (0)
>>>>>> >> > - MaxPartitionLevel: 0 (0)
>>>>>> >> > - NumRepartitions: 0 (0)
>>>>>> >> > - PartitionsCreated: 0 (0)
>>>>>> >> > - PeakMemoryUsage: 20.00 KB (20480)
>>>>>> >> > - RowsRepartitioned: 0 (0)
>>>>>> >> > - RowsReturned: 1 (1)
>>>>>> >> > - RowsReturnedRate: 3.00 /sec
>>>>>> >> > - SpilledPartitions: 0 (0)
>>>>>> >> > HDFS_SCAN_NODE (id=0):(Total: 308.859ms, non-child:
>>>>>> 308.859ms, %
>>>>>> >> > non-child: 100.00%)
>>>>>> >> > Hdfs split stats (<volume id>:<# splits>/<split
>>>>>> lengths>):
>>>>>> >> > 0:35/20.89 MB 7:44
>>>>>> <https://maps.google.com/?q=MB+7:44&entry=gmail&source=g>/26.64 MB
>>>>>> 11:27 <https://maps.google.com/?q=MB+11:27&entry=gmail&source=g>/17.21
>>>>>> MB 9:32 <https://maps.google.com/?q=MB+9:32&entry=gmail&source=g>/18.61
>>>>>> MB 5:25 <https://maps.google.com/?q=MB+5:25&entry=gmail&source=g>/14.56
>>>>>> MB
>>>>>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+2:38&entry=gmail&source=g>
>>>>>> >> > 2:38
>>>>>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+2:38&entry=gmail&source=g>/24.32
>>>>>> MB 6:31 <https://maps.google.com/?q=MB+6:31&entry=gmail&source=g>/19.85
>>>>>> MB 8:50 <https://maps.google.com/?q=MB+8:50&entry=gmail&source=g>/30.76
>>>>>> MB 10:41 <https://maps.google.com/?q=MB+10:41&entry=gmail&source=g>/25.92
>>>>>> MB 3:31 <https://maps.google.com/?q=MB+3:31&entry=gmail&source=g>/19.78
>>>>>> MB
>>>>>> >> > 4:60/37.59 MB 1:44/27.19 MB
>>>>>> >> > ExecOption: PARQUET Codegen Enabled, Codegen enabled:
>>>>>> 458 out
>>>>>> >> > of
>>>>>> >> > 458
>>>>>> >> > Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0%
>>>>>> 3:0%
>>>>>> >> > 4:0%
>>>>>> >> > 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0%
>>>>>> 16:0%
>>>>>> >> > File Formats: PARQUET/Unknown(Skipped):458
>>>>>> >> > BytesRead(500.000ms): 36.74 MB
>>>>>> >> > - FooterProcessingTime: (Avg: 11.502ms ; Min: 3.668ms
>>>>>> ; Max:
>>>>>> >> > 23.564ms ; Number of samples: 458)
>>>>>> >> > - AverageHdfsReadThreadConcurrency: 0.00
>>>>>> >> > - AverageScannerThreadConcurrency: 24.00
>>>>>> >> > - BytesRead: 44.51 MB (46668432)
>>>>>> >> > - BytesReadDataNodeCache: 0
>>>>>> >> > - BytesReadLocal: 44.51 MB (46668432)
>>>>>> >> > - BytesReadRemoteUnexpected: 0
>>>>>> >> > - BytesReadShortCircuit: 44.51 MB (46668432)
>>>>>> >> > - CachedFileHandlesHitCount: 0 (0)
>>>>>> >> > - CachedFileHandlesMissCount: 458 (458)
>>>>>> >> > - CollectionItemsRead: 0 (0)
>>>>>> >> > - DecompressionTime: 0.000ns
>>>>>> >> > - MaxCompressedTextFileLength: 0
>>>>>> >> > - NumColumns: 0 (0)
>>>>>> >> > - NumDictFilteredRowGroups: 0 (0)
>>>>>> >> > - NumDisksAccessed: 12 (12)
>>>>>> >> > - NumRowGroups: 458 (458)
>>>>>> >> > - NumScannerThreadsStarted: 24 (24)
>>>>>> >> > - NumScannersWithNoReads: 0 (0)
>>>>>> >> > - NumStatsFilteredRowGroups: 0 (0)
>>>>>> >> > - PeakMemoryUsage: 2.50 MB (2621440)
>>>>>> >> > - PerReadThreadRawHdfsThroughput: 1.26 GB/sec
>>>>>> >> > - RemoteScanRanges: 0 (0)
>>>>>> >> > - RowBatchQueueGetWaitTime: 295.595ms
>>>>>> >> > - RowBatchQueuePutWaitTime: 0.000ns
>>>>>> >> > - RowsRead: 0 (0)
>>>>>> >> > - RowsReturned: 458 (458)
>>>>>> >> > - RowsReturnedRate: 1.48 K/sec
>>>>>> >> > - ScanRangesComplete: 458 (458)
>>>>>> >> > - ScannerThreadsInvoluntaryContextSwitches: 91 (91)
>>>>>> >> > - ScannerThreadsTotalWallClockTime: 6s958ms
>>>>>> >> > - MaterializeTupleTime(*): 71.630us
>>>>>> >> > - ScannerThreadsSysTime: 16.989ms
>>>>>> >> > - ScannerThreadsUserTime: 645.891ms
>>>>>> >> > - ScannerThreadsVoluntaryContextSwitches: 1.63K
>>>>>> (1630)
>>>>>> >> > - TotalRawHdfsReadTime(*): 34.377ms
>>>>>> >> > - TotalReadThroughput: 73.49 MB/sec
>>>>>> >> > Instance c843dae48ce9b289:6ad3ffc000000001
>>>>>> >> > (host=aopr-dhc005.lpdomain.com:22000):(Total: 308.195ms,
>>>>>> non-child:
>>>>>> >> > 1.784ms,
>>>>>> >> > % non-child: 0.58%)
>>>>>> >> > Hdfs split stats (<volume id>:<# splits>/<split
>>>>>> lengths>):
>>>>>> >> > 2:46/28.97 MB 8:39
>>>>>> <https://maps.google.com/?q=MB+8:39&entry=gmail&source=g>/24.34 MB
>>>>>> 10:33 <https://maps.google.com/?q=MB+10:33&entry=gmail&source=g>/21.03
>>>>>> MB 3:43 <https://maps.google.com/?q=MB+3:43&entry=gmail&source=g>/26.50
>>>>>> MB 5:32 <https://maps.google.com/?q=MB+5:32&entry=gmail&source=g>/20.27
>>>>>> MB
>>>>>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+9:32&entry=gmail&source=g>
>>>>>> >> > 9:32
>>>>>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+9:32&entry=gmail&source=g>/19.86
>>>>>> MB 6:25 <https://maps.google.com/?q=MB+6:25&entry=gmail&source=g>/16.01
>>>>>> MB 7:41 <https://maps.google.com/?q=MB+7:41&entry=gmail&source=g>/24.77
>>>>>> MB 1:42/25.01 MB 0:55/32.51 MB
>>>>>> >> > 4:40/24.29 MB 11:33
>>>>>> <https://maps.google.com/?q=MB+11:33&entry=gmail&source=g>/20.77 MB
>>>>>> >> > - AverageThreadTokens: 0.00
>>>>>> >> > - BloomFilterBytes: 0
>>>>>> >> > - PeakMemoryUsage: 2.56 MB (2683241)
>>>>>> >> > - PeakReservation: 0
>>>>>> >> > - PeakUsedReservation: 0
>>>>>> >> > - PerHostPeakMemUsage: 2.56 MB (2683241)
>>>>>> >> > - RowsProduced: 1 (1)
>>>>>> >> > - TotalNetworkReceiveTime: 0.000ns
>>>>>> >> > - TotalNetworkSendTime: 292.765us
>>>>>> >> > - TotalStorageWaitTime: 4s482ms
>>>>>> >> > - TotalThreadsInvoluntaryContextSwitches: 113 (113)
>>>>>> >> > - TotalThreadsTotalWallClockTime: 6s310ms
>>>>>> >> > - TotalThreadsSysTime: 35.983ms
>>>>>> >> > - TotalThreadsUserTime: 682.884ms
>>>>>> >> > - TotalThreadsVoluntaryContextSwitches: 2.14K (2143)
>>>>>> >> > Fragment Instance Lifecycle Timings:
>>>>>> >> > - ExecTime: 532.572us
>>>>>> >> > - ExecTreeExecTime: 8.377us
>>>>>> >> > - OpenTime: 280.241ms
>>>>>> >> > - ExecTreeOpenTime: 258.652ms
>>>>>> >> > - PrepareTime: 27.342ms
>>>>>> >> > - ExecTreePrepareTime: 2.863ms
>>>>>> >> > DataStreamSender (dst_id=2):(Total: 309.794us, non-child:
>>>>>> >> > 309.794us,
>>>>>> >> > % non-child: 100.00%)
>>>>>> >> > - BytesSent: 16.00 B (16)
>>>>>> >> > - NetworkThroughput(*): 356.91 KB/sec
>>>>>> >> > - OverallThroughput: 50.44 KB/sec
>>>>>> >> > - PeakMemoryUsage: 16.00 KB (16384)
>>>>>> >> > - RowsReturned: 1 (1)
>>>>>> >> > - SerializeBatchTime: 14.262us
>>>>>> >> > - TransmitDataRPCTime: 43.778us
>>>>>> >> > - UncompressedRowBatchSize: 16.00 B (16)
>>>>>> >> > CodeGen:(Total: 44.584ms, non-child: 44.584ms, %
>>>>>> non-child:
>>>>>> >> > 100.00%)
>>>>>> >> > - CodegenTime: 1.382ms
>>>>>> >> > - CompileTime: 7.346ms
>>>>>> >> > - LoadTime: 0.000ns
>>>>>> >> > - ModuleBitcodeSize: 1.86 MB (1950880)
>>>>>> >> > - NumFunctions: 15 (15)
>>>>>> >> > - NumInstructions: 214 (214)
>>>>>> >> > - OptimizationTime: 13.462ms
>>>>>> >> > - PeakMemoryUsage: 107.00 KB (109568)
>>>>>> >> > - PrepareTime: 23.006ms
>>>>>> >> > AGGREGATION_NODE (id=1):(Total: 261.516ms, non-child:
>>>>>> 836.656us,
>>>>>> >> > %
>>>>>> >> > non-child: 0.32%)
>>>>>> >> > ExecOption: Codegen Enabled
>>>>>> >> > - BuildTime: 1.280ms
>>>>>> >> > - GetResultsTime: 0.000ns
>>>>>> >> > - HTResizeTime: 0.000ns
>>>>>> >> > - HashBuckets: 0 (0)
>>>>>> >> > - LargestPartitionPercent: 0 (0)
>>>>>> >> > - MaxPartitionLevel: 0 (0)
>>>>>> >> > - NumRepartitions: 0 (0)
>>>>>> >> > - PartitionsCreated: 0 (0)
>>>>>> >> > - PeakMemoryUsage: 20.00 KB (20480)
>>>>>> >> > - RowsRepartitioned: 0 (0)
>>>>>> >> > - RowsReturned: 1 (1)
>>>>>> >> > - RowsReturnedRate: 3.00 /sec
>>>>>> >> > - SpilledPartitions: 0 (0)
>>>>>> >> > HDFS_SCAN_NODE (id=0):(Total: 260.680ms, non-child:
>>>>>> 260.680ms, %
>>>>>> >> > non-child: 100.00%)
>>>>>> >> > Hdfs split stats (<volume id>:<# splits>/<split
>>>>>> lengths>):
>>>>>> >> > 2:46/28.97 MB 8:39
>>>>>> <https://maps.google.com/?q=MB+8:39&entry=gmail&source=g>/24.34 MB
>>>>>> 10:33 <https://maps.google.com/?q=MB+10:33&entry=gmail&source=g>/21.03
>>>>>> MB 3:43 <https://maps.google.com/?q=MB+3:43&entry=gmail&source=g>/26.50
>>>>>> MB 5:32 <https://maps.google.com/?q=MB+5:32&entry=gmail&source=g>/20.27
>>>>>> MB
>>>>>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+9:32&entry=gmail&source=g>
>>>>>> >> > 9:32
>>>>>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+9:32&entry=gmail&source=g>/19.86
>>>>>> MB 6:25 <https://maps.google.com/?q=MB+6:25&entry=gmail&source=g>/16.01
>>>>>> MB 7:41 <https://maps.google.com/?q=MB+7:41&entry=gmail&source=g>/24.77
>>>>>> MB 1:42/25.01 MB 0:55/32.51 MB
>>>>>> >> > 4:40/24.29 MB 11:33
>>>>>> <https://maps.google.com/?q=MB+11:33&entry=gmail&source=g>/20.77 MB
>>>>>> >> > ExecOption: PARQUET Codegen Enabled, Codegen enabled:
>>>>>> 461 out
>>>>>> >> > of
>>>>>> >> > 461
>>>>>> >> > Hdfs Read Thread Concurrency Bucket: 0:0% 1:0% 2:0%
>>>>>> 3:0% 4:0%
>>>>>> >> > 5:0%
>>>>>> >> > 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
>>>>>> >> > File Formats: PARQUET/Unknown(Skipped):461
>>>>>> >> > - FooterProcessingTime: (Avg: 10.793ms ; Min: 3.087ms
>>>>>> ; Max:
>>>>>> >> > 20.253ms ; Number of samples: 461)
>>>>>> >> > - AverageHdfsReadThreadConcurrency: 0.00
>>>>>> >> > - AverageScannerThreadConcurrency: 0.00
>>>>>> >> > - BytesRead: 44.73 MB (46906407)
>>>>>> >> > - BytesReadDataNodeCache: 0
>>>>>> >> > - BytesReadLocal: 44.73 MB (46906407)
>>>>>> >> > - BytesReadRemoteUnexpected: 0
>>>>>> >> > - BytesReadShortCircuit: 44.73 MB (46906407)
>>>>>> >> > - CachedFileHandlesHitCount: 0 (0)
>>>>>> >> > - CachedFileHandlesMissCount: 461 (461)
>>>>>> >> > - CollectionItemsRead: 0 (0)
>>>>>> >> > - DecompressionTime: 0.000ns
>>>>>> >> > - MaxCompressedTextFileLength: 0
>>>>>> >> > - NumColumns: 0 (0)
>>>>>> >> > - NumDictFilteredRowGroups: 0 (0)
>>>>>> >> > - NumDisksAccessed: 12 (12)
>>>>>> >> > - NumRowGroups: 461 (461)
>>>>>> >> > - NumScannerThreadsStarted: 24 (24)
>>>>>> >> > - NumScannersWithNoReads: 0 (0)
>>>>>> >> > - NumStatsFilteredRowGroups: 0 (0)
>>>>>> >> > - PeakMemoryUsage: 2.52 MB (2641928)
>>>>>> >> > - PerReadThreadRawHdfsThroughput: 1.31 GB/sec
>>>>>> >> > - RemoteScanRanges: 0 (0)
>>>>>> >> > - RowBatchQueueGetWaitTime: 246.691ms
>>>>>> >> > - RowBatchQueuePutWaitTime: 0.000ns
>>>>>> >> > - RowsRead: 0 (0)
>>>>>> >> > - RowsReturned: 461 (461)
>>>>>> >> > - RowsReturnedRate: 1.77 K/sec
>>>>>> >> > - ScanRangesComplete: 461 (461)
>>>>>> >> > - ScannerThreadsInvoluntaryContextSwitches: 109 (109)
>>>>>> >> > - ScannerThreadsTotalWallClockTime: 6s030ms
>>>>>> >> > - MaterializeTupleTime(*): 68.735us
>>>>>> >> > - ScannerThreadsSysTime: 33.983ms
>>>>>> >> > - ScannerThreadsUserTime: 647.889ms
>>>>>> >> > - ScannerThreadsVoluntaryContextSwitches: 1.70K
>>>>>> (1696)
>>>>>> >> > - TotalRawHdfsReadTime(*): 33.405ms
>>>>>> >> > - TotalReadThroughput: 0.00 /sec
>>>>>> >> > Instance c843dae48ce9b289:6ad3ffc000000003
>>>>>> >> > (host=aopr-dhc002.lpdomain.com:22000):(Total: 302.116ms,
>>>>>> non-child:
>>>>>> >> > 1.604ms,
>>>>>> >> > % non-child: 0.53%)
>>>>>> >> > Hdfs split stats (<volume id>:<# splits>/<split
>>>>>> lengths>):
>>>>>> >> > 0:27/16.21 MB 11:33
>>>>>> <https://maps.google.com/?q=MB+11:33&entry=gmail&source=g>/21.23 MB
>>>>>> 6:36 <https://maps.google.com/?q=MB+6:36&entry=gmail&source=g>/23.15 MB
>>>>>> 10:41 <https://maps.google.com/?q=MB+10:41&entry=gmail&source=g>/25.09
>>>>>> MB 1:41/24.13 MB
>>>>>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+9:32&entry=gmail&source=g>
>>>>>> >> > 9:32
>>>>>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+9:32&entry=gmail&source=g>/18.79
>>>>>> MB 5:41 <https://maps.google.com/?q=MB+5:41&entry=gmail&source=g>/25.31
>>>>>> MB 2:42 <https://maps.google.com/?q=MB+2:42&entry=gmail&source=g>/26.49
>>>>>> MB 7:36 <https://maps.google.com/?q=MB+7:36&entry=gmail&source=g>/22.87
>>>>>> MB 3:51 <https://maps.google.com/?q=MB+3:51&entry=gmail&source=g>/31.00
>>>>>> MB
>>>>>> >> > 4:41/25.99 MB 8:38
>>>>>> <https://maps.google.com/?q=MB+8:38&entry=gmail&source=g>/23.79 MB
>>>>>> >> > MemoryUsage(500.000ms): 700.35 KB
>>>>>> >> > ThreadUsage(500.000ms): 25
>>>>>> >> > - AverageThreadTokens: 25.00
>>>>>> >> > - BloomFilterBytes: 0
>>>>>> >> > - PeakMemoryUsage: 2.54 MB (2662753)
>>>>>> >> > - PeakReservation: 0
>>>>>> >> > - PeakUsedReservation: 0
>>>>>> >> > - PerHostPeakMemUsage: 2.54 MB (2662753)
>>>>>> >> > - RowsProduced: 1 (1)
>>>>>> >> > - TotalNetworkReceiveTime: 0.000ns
>>>>>> >> > - TotalNetworkSendTime: 300.675us
>>>>>> >> > - TotalStorageWaitTime: 4s661ms
>>>>>> >> > - TotalThreadsInvoluntaryContextSwitches: 90 (90)
>>>>>> >> > - TotalThreadsTotalWallClockTime: 6s256ms
>>>>>> >> > - TotalThreadsSysTime: 30.983ms
>>>>>> >> > - TotalThreadsUserTime: 709.880ms
>>>>>> >> > - TotalThreadsVoluntaryContextSwitches: 2.17K (2175)
>>>>>> >> > Fragment Instance Lifecycle Timings:
>>>>>> >> > - ExecTime: 492.627us
>>>>>> >> > - ExecTreeExecTime: 8.191us
>>>>>> >> > - OpenTime: 277.922ms
>>>>>> >> > - ExecTreeOpenTime: 255.395ms
>>>>>> >> > - PrepareTime: 23.647ms
>>>>>> >> > - ExecTreePrepareTime: 2.243ms
>>>>>> >> > DataStreamSender (dst_id=2):(Total: 194.517us, non-child:
>>>>>> >> > 194.517us,
>>>>>> >> > % non-child: 100.00%)
>>>>>> >> > - BytesSent: 16.00 B (16)
>>>>>> >> > - NetworkThroughput(*): 371.86 KB/sec
>>>>>> >> > - OverallThroughput: 80.33 KB/sec
>>>>>> >> > - PeakMemoryUsage: 16.00 KB (16384)
>>>>>> >> > - RowsReturned: 1 (1)
>>>>>> >> > - SerializeBatchTime: 14.035us
>>>>>> >> > - TransmitDataRPCTime: 42.019us
>>>>>> >> > - UncompressedRowBatchSize: 16.00 B (16)
>>>>>> >> > CodeGen:(Total: 42.676ms, non-child: 42.676ms, %
>>>>>> non-child:
>>>>>> >> > 100.00%)
>>>>>> >> > - CodegenTime: 1.236ms
>>>>>> >> > - CompileTime: 7.717ms
>>>>>> >> > - LoadTime: 0.000ns
>>>>>> >> > - ModuleBitcodeSize: 1.86 MB (1950880)
>>>>>> >> > - NumFunctions: 15 (15)
>>>>>> >> > - NumInstructions: 214 (214)
>>>>>> >> > - OptimizationTime: 14.130ms
>>>>>> >> > - PeakMemoryUsage: 107.00 KB (109568)
>>>>>> >> > - PrepareTime: 20.109ms
>>>>>> >> > AGGREGATION_NODE (id=1):(Total: 257.640ms, non-child:
>>>>>> 1.541ms, %
>>>>>> >> > non-child: 0.60%)
>>>>>> >> > ExecOption: Codegen Enabled
>>>>>> >> > - BuildTime: 1.252ms
>>>>>> >> > - GetResultsTime: 0.000ns
>>>>>> >> > - HTResizeTime: 0.000ns
>>>>>> >> > - HashBuckets: 0 (0)
>>>>>> >> > - LargestPartitionPercent: 0 (0)
>>>>>> >> > - MaxPartitionLevel: 0 (0)
>>>>>> >> > - NumRepartitions: 0 (0)
>>>>>> >> > - PartitionsCreated: 0 (0)
>>>>>> >> > - PeakMemoryUsage: 20.00 KB (20480)
>>>>>> >> > - RowsRepartitioned: 0 (0)
>>>>>> >> > - RowsReturned: 1 (1)
>>>>>> >> > - RowsReturnedRate: 3.00 /sec
>>>>>> >> > - SpilledPartitions: 0 (0)
>>>>>> >> > HDFS_SCAN_NODE (id=0):(Total: 256.099ms, non-child:
>>>>>> 256.099ms, %
>>>>>> >> > non-child: 100.00%)
>>>>>> >> > Hdfs split stats (<volume id>:<# splits>/<split
>>>>>> lengths>):
>>>>>> >> > 0:27/16.21 MB 11:33
>>>>>> <https://maps.google.com/?q=MB+11:33&entry=gmail&source=g>/21.23 MB
>>>>>> 6:36 <https://maps.google.com/?q=MB+6:36&entry=gmail&source=g>/23.15 MB
>>>>>> 10:41 <https://maps.google.com/?q=MB+10:41&entry=gmail&source=g>/25.09
>>>>>> MB 1:41/24.13 MB
>>>>>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+9:32&entry=gmail&source=g>
>>>>>> >> > 9:32
>>>>>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+9:32&entry=gmail&source=g>/18.79
>>>>>> MB 5:41 <https://maps.google.com/?q=MB+5:41&entry=gmail&source=g>/25.31
>>>>>> MB 2:42 <https://maps.google.com/?q=MB+2:42&entry=gmail&source=g>/26.49
>>>>>> MB 7:36 <https://maps.google.com/?q=MB+7:36&entry=gmail&source=g>/22.87
>>>>>> MB 3:51 <https://maps.google.com/?q=MB+3:51&entry=gmail&source=g>/31.00
>>>>>> MB
>>>>>> >> > 4:41/25.99 MB 8:38
>>>>>> <https://maps.google.com/?q=MB+8:38&entry=gmail&source=g>/23.79 MB
>>>>>> >> > ExecOption: PARQUET Codegen Enabled, Codegen enabled:
>>>>>> 459 out
>>>>>> >> > of
>>>>>> >> > 459
>>>>>> >> > Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0%
>>>>>> 3:0%
>>>>>> >> > 4:0%
>>>>>> >> > 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0%
>>>>>> 16:0%
>>>>>> >> > File Formats: PARQUET/Unknown(Skipped):459
>>>>>> >> > BytesRead(500.000ms): 43.15 MB
>>>>>> >> > - FooterProcessingTime: (Avg: 11.257ms ; Min: 3.989ms
>>>>>> ; Max:
>>>>>> >> > 23.004ms ; Number of samples: 459)
>>>>>> >> > - AverageHdfsReadThreadConcurrency: 0.00
>>>>>> >> > - AverageScannerThreadConcurrency: 24.00
>>>>>> >> > - BytesRead: 44.52 MB (46678031)
>>>>>> >> > - BytesReadDataNodeCache: 0
>>>>>> >> > - BytesReadLocal: 44.52 MB (46678031)
>>>>>> >> > - BytesReadRemoteUnexpected: 0
>>>>>> >> > - BytesReadShortCircuit: 44.52 MB (46678031)
>>>>>> >> > - CachedFileHandlesHitCount: 0 (0)
>>>>>> >> > - CachedFileHandlesMissCount: 459 (459)
>>>>>> >> > - CollectionItemsRead: 0 (0)
>>>>>> >> > - DecompressionTime: 0.000ns
>>>>>> >> > - MaxCompressedTextFileLength: 0
>>>>>> >> > - NumColumns: 0 (0)
>>>>>> >> > - NumDictFilteredRowGroups: 0 (0)
>>>>>> >> > - NumDisksAccessed: 12 (12)
>>>>>> >> > - NumRowGroups: 459 (459)
>>>>>> >> > - NumScannerThreadsStarted: 24 (24)
>>>>>> >> > - NumScannersWithNoReads: 0 (0)
>>>>>> >> > - NumStatsFilteredRowGroups: 0 (0)
>>>>>> >> > - PeakMemoryUsage: 2.50 MB (2621440)
>>>>>> >> > - PerReadThreadRawHdfsThroughput: 1.33 GB/sec
>>>>>> >> > - RemoteScanRanges: 0 (0)
>>>>>> >> > - RowBatchQueueGetWaitTime: 243.426ms
>>>>>> >> > - RowBatchQueuePutWaitTime: 0.000ns
>>>>>> >> > - RowsRead: 0 (0)
>>>>>> >> > - RowsReturned: 459 (459)
>>>>>> >> > - RowsReturnedRate: 1.79 K/sec
>>>>>> >> > - ScanRangesComplete: 459 (459)
>>>>>> >> > - ScannerThreadsInvoluntaryContextSwitches: 85 (85)
>>>>>> >> > - ScannerThreadsTotalWallClockTime: 5s978ms
>>>>>> >> > - MaterializeTupleTime(*): 67.997us
>>>>>> >> > - ScannerThreadsSysTime: 22.984ms
>>>>>> >> > - ScannerThreadsUserTime: 678.884ms
>>>>>> >> > - ScannerThreadsVoluntaryContextSwitches: 1.72K
>>>>>> (1724)
>>>>>> >> > - TotalRawHdfsReadTime(*): 32.626ms
>>>>>> >> > - TotalReadThroughput: 86.30 MB/sec
>>>>>> >> > Instance c843dae48ce9b289:6ad3ffc000000005
>>>>>> >> > (host=aopr-dhc001.lpdomain.com:22000):(Total: 298.093ms,
>>>>>> non-child:
>>>>>> >> > 1.242ms,
>>>>>> >> > % non-child: 0.42%)
>>>>>> >> > Hdfs split stats (<volume id>:<# splits>/<split
>>>>>> lengths>):
>>>>>> >> > 9:55/35.38 MB 0:37/24.23 MB 5:37
>>>>>> <https://maps.google.com/?q=MB+5:37&entry=gmail&source=g>/23.31 MB
>>>>>> 11:22 <https://maps.google.com/?q=MB+11:22&entry=gmail&source=g>/13.79
>>>>>> MB 8:30 <https://maps.google.com/?q=MB+8:30&entry=gmail&source=g>/18.53
>>>>>> MB
>>>>>> >> > 4:33/20.45 MB 1:38/24.06 MB 10:29
>>>>>> <https://maps.google.com/?q=MB+10:29&entry=gmail&source=g>/18.40 MB
>>>>>> 3:39 <https://maps.google.com/?q=MB+3:39&entry=gmail&source=g>/24.35 MB
>>>>>> 2:41 <https://maps.google.com/?q=MB+2:41&entry=gmail&source=g>/25.45
>>>>>> MB
>>>>>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+6:42&entry=gmail&source=g>
>>>>>> >> > 6:42
>>>>>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+6:42&entry=gmail&source=g>/27.03
>>>>>> MB 7:47 <https://maps.google.com/?q=MB+7:47&entry=gmail&source=g>/29.22
>>>>>> MB
>>>>>> >> > - AverageThreadTokens: 0.00
>>>>>> >> > - BloomFilterBytes: 0
>>>>>> >> > - PeakMemoryUsage: 2.66 MB (2793825)
>>>>>> >> > - PeakReservation: 0
>>>>>> >> > - PeakUsedReservation: 0
>>>>>> >> > - PerHostPeakMemUsage: 2.68 MB (2806113)
>>>>>> >> > - RowsProduced: 1 (1)
>>>>>> >> > - TotalNetworkReceiveTime: 0.000ns
>>>>>> >> > - TotalNetworkSendTime: 198.894us
>>>>>> >> > - TotalStorageWaitTime: 4s374ms
>>>>>> >> > - TotalThreadsInvoluntaryContextSwitches: 110 (110)
>>>>>> >> > - TotalThreadsTotalWallClockTime: 6s201ms
>>>>>> >> > - TotalThreadsSysTime: 29.983ms
>>>>>> >> > - TotalThreadsUserTime: 642.892ms
>>>>>> >> > - TotalThreadsVoluntaryContextSwitches: 2.17K (2174)
>>>>>> >> > Fragment Instance Lifecycle Timings:
>>>>>> >> > - ExecTime: 350.873us
>>>>>> >> > - ExecTreeExecTime: 8.173us
>>>>>> >> > - OpenTime: 275.347ms
>>>>>> >> > - ExecTreeOpenTime: 253.425ms
>>>>>> >> > - PrepareTime: 22.341ms
>>>>>> >> > - ExecTreePrepareTime: 2.251ms
>>>>>> >> > DataStreamSender (dst_id=2):(Total: 164.394us, non-child:
>>>>>> >> > 164.394us,
>>>>>> >> > % non-child: 100.00%)
>>>>>> >> > - BytesSent: 16.00 B (16)
>>>>>> >> > - NetworkThroughput(*): 493.42 KB/sec
>>>>>> >> > - OverallThroughput: 95.05 KB/sec
>>>>>> >> > - PeakMemoryUsage: 16.00 KB (16384)
>>>>>> >> > - RowsReturned: 1 (1)
>>>>>> >> > - SerializeBatchTime: 14.262us
>>>>>> >> > - TransmitDataRPCTime: 31.667us
>>>>>> >> > - UncompressedRowBatchSize: 16.00 B (16)
>>>>>> >> > CodeGen:(Total: 41.008ms, non-child: 41.008ms, %
>>>>>> non-child:
>>>>>> >> > 100.00%)
>>>>>> >> > - CodegenTime: 1.049ms
>>>>>> >> > - CompileTime: 7.248ms
>>>>>> >> > - LoadTime: 0.000ns
>>>>>> >> > - ModuleBitcodeSize: 1.86 MB (1950880)
>>>>>> >> > - NumFunctions: 15 (15)
>>>>>> >> > - NumInstructions: 214 (214)
>>>>>> >> > - OptimizationTime: 14.018ms
>>>>>> >> > - PeakMemoryUsage: 107.00 KB (109568)
>>>>>> >> > - PrepareTime: 19.080ms
>>>>>> >> > AGGREGATION_NODE (id=1):(Total: 255.678ms, non-child:
>>>>>> 1.046ms, %
>>>>>> >> > non-child: 0.41%)
>>>>>> >> > ExecOption: Codegen Enabled
>>>>>> >> > - BuildTime: 1.015ms
>>>>>> >> > - GetResultsTime: 0.000ns
>>>>>> >> > - HTResizeTime: 0.000ns
>>>>>> >> > - HashBuckets: 0 (0)
>>>>>> >> > - LargestPartitionPercent: 0 (0)
>>>>>> >> > - MaxPartitionLevel: 0 (0)
>>>>>> >> > - NumRepartitions: 0 (0)
>>>>>> >> > - PartitionsCreated: 0 (0)
>>>>>> >> > - PeakMemoryUsage: 20.00 KB (20480)
>>>>>> >> > - RowsRepartitioned: 0 (0)
>>>>>> >> > - RowsReturned: 1 (1)
>>>>>> >> > - RowsReturnedRate: 3.00 /sec
>>>>>> >> > - SpilledPartitions: 0 (0)
>>>>>> >> > HDFS_SCAN_NODE (id=0):(Total: 254.631ms, non-child:
>>>>>> 254.631ms, %
>>>>>> >> > non-child: 100.00%)
>>>>>> >> > Hdfs split stats (<volume id>:<# splits>/<split
>>>>>> lengths>):
>>>>>> >> > 9:55/35.38 MB 0:37/24.23 MB 5:37
>>>>>> <https://maps.google.com/?q=MB+5:37&entry=gmail&source=g>/23.31 MB
>>>>>> 11:22 <https://maps.google.com/?q=MB+11:22&entry=gmail&source=g>/13.79
>>>>>> MB 8:30 <https://maps.google.com/?q=MB+8:30&entry=gmail&source=g>/18.53
>>>>>> MB
>>>>>> >> > 4:33/20.45 MB 1:38/24.06 MB 10:29
>>>>>> <https://maps.google.com/?q=MB+10:29&entry=gmail&source=g>/18.40 MB
>>>>>> 3:39 <https://maps.google.com/?q=MB+3:39&entry=gmail&source=g>/24.35 MB
>>>>>> 2:41 <https://maps.google.com/?q=MB+2:41&entry=gmail&source=g>/25.45
>>>>>> MB
>>>>>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+6:42&entry=gmail&source=g>
>>>>>> >> > 6:42
>>>>>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+6:42&entry=gmail&source=g>/27.03
>>>>>> MB 7:47 <https://maps.google.com/?q=MB+7:47&entry=gmail&source=g>/29.22
>>>>>> MB
>>>>>> >> > ExecOption: PARQUET Codegen Enabled, Codegen enabled:
>>>>>> 450 out
>>>>>> >> > of
>>>>>> >> > 450
>>>>>> >> > Hdfs Read Thread Concurrency Bucket: 0:0% 1:0% 2:0%
>>>>>> 3:0% 4:0%
>>>>>> >> > 5:0%
>>>>>> >> > 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
>>>>>> >> > File Formats: PARQUET/Unknown(Skipped):450
>>>>>> >> > - FooterProcessingTime: (Avg: 10.727ms ; Min: 3.813ms
>>>>>> ; Max:
>>>>>> >> > 23.755ms ; Number of samples: 450)
>>>>>> >> > - AverageHdfsReadThreadConcurrency: 0.00
>>>>>> >> > - AverageScannerThreadConcurrency: 0.00
>>>>>> >> > - BytesRead: 43.85 MB (45983284)
>>>>>> >> > - BytesReadDataNodeCache: 0
>>>>>> >> > - BytesReadLocal: 43.85 MB (45983284)
>>>>>> >> > - BytesReadRemoteUnexpected: 0
>>>>>> >> > - BytesReadShortCircuit: 43.85 MB (45983284)
>>>>>> >> > - CachedFileHandlesHitCount: 0 (0)
>>>>>> >> > - CachedFileHandlesMissCount: 450 (450)
>>>>>> >> > - CollectionItemsRead: 0 (0)
>>>>>> >> > - DecompressionTime: 0.000ns
>>>>>> >> > - MaxCompressedTextFileLength: 0
>>>>>> >> > - NumColumns: 0 (0)
>>>>>> >> > - NumDictFilteredRowGroups: 0 (0)
>>>>>> >> > - NumDisksAccessed: 12 (12)
>>>>>> >> > - NumRowGroups: 450 (450)
>>>>>> >> > - NumScannerThreadsStarted: 24 (24)
>>>>>> >> > - NumScannersWithNoReads: 0 (0)
>>>>>> >> > - NumStatsFilteredRowGroups: 0 (0)
>>>>>> >> > - PeakMemoryUsage: 2.62 MB (2752512)
>>>>>> >> > - PerReadThreadRawHdfsThroughput: 1.56 GB/sec
>>>>>> >> > - RemoteScanRanges: 0 (0)
>>>>>> >> > - RowBatchQueueGetWaitTime: 242.362ms
>>>>>> >> > - RowBatchQueuePutWaitTime: 0.000ns
>>>>>> >> > - RowsRead: 0 (0)
>>>>>> >> > - RowsReturned: 450 (450)
>>>>>> >> > - RowsReturnedRate: 1.77 K/sec
>>>>>> >> > - ScanRangesComplete: 450 (450)
>>>>>> >> > - ScannerThreadsInvoluntaryContextSwitches: 106 (106)
>>>>>> >> > - ScannerThreadsTotalWallClockTime: 5s926ms
>>>>>> >> > - MaterializeTupleTime(*): 69.447us
>>>>>> >> > - ScannerThreadsSysTime: 26.984ms
>>>>>> >> > - ScannerThreadsUserTime: 607.897ms
>>>>>> >> > - ScannerThreadsVoluntaryContextSwitches: 1.72K
>>>>>> (1723)
>>>>>> >> > - TotalRawHdfsReadTime(*): 27.393ms
>>>>>> >> > - TotalReadThroughput: 0.00 /sec
>>>>>> >> > ===================================================
>>>>>> >> >
>>>>>> >> > Coordinator log:
>>>>>> >> >
>>>>>> >> > 01: query (string) = "select count(*) from
>>>>>> dv_entersiteevent_prq_remote
>>>>>> >> > where year=2017 and month=10",
>>>>>> >> > 03: configuration (list) = list<string>[1] {
>>>>>> >> > [0] = "REQUEST_POOL=impreports",
>>>>>> >> > },
>>>>>> >> > 04: hadoop_user (string) = "fawzea",
>>>>>> >> > }
>>>>>> >> > I0208 01:07:06.321281 7154 impala-beeswax-server.cc:459]
>>>>>> >> > TClientRequest.queryOptions: TQueryOptions {
>>>>>> >> > 01: abort_on_error (bool) = false,
>>>>>> >> > 02: max_errors (i32) = 100,
>>>>>> >> > 03: disable_codegen (bool) = false,
>>>>>> >> > 04: batch_size (i32) = 0,
>>>>>> >> > 05: num_nodes (i32) = 0,
>>>>>> >> > 06: max_scan_range_length (i64) = 0,
>>>>>> >> > 07: num_scanner_threads (i32) = 0,
>>>>>> >> > 08: max_io_buffers (i32) = 0,
>>>>>> >> > 09: allow_unsupported_formats (bool) = false,
>>>>>> >> > 10: default_order_by_limit (i64) = -1,
>>>>>> >> > 11: debug_action (string) = "",
>>>>>> >> > 12: mem_limit (i64) = 0,
>>>>>> >> > 13: abort_on_default_limit_exceeded (bool) = false,
>>>>>> >> > 15: hbase_caching (i32) = 0,
>>>>>> >> > 16: hbase_cache_blocks (bool) = false,
>>>>>> >> > 17: parquet_file_size (i64) = 0,
>>>>>> >> > 18: explain_level (i32) = 1,
>>>>>> >> > 19: sync_ddl (bool) = false,
>>>>>> >> > 20: request_pool (string) = "impreports",
>>>>>> >> > 23: disable_cached_reads (bool) = false,
>>>>>> >> > 24: disable_outermost_topn (bool) = false,
>>>>>> >> > 25: rm_initial_mem (i64) = 0,
>>>>>> >> > 26: query_timeout_s (i32) = 0,
>>>>>> >> > 28: appx_count_distinct (bool) = false,
>>>>>> >> > 29: disable_unsafe_spills (bool) = false,
>>>>>> >> > 31: exec_single_node_rows_threshold (i32) = 100,
>>>>>> >> > 32: optimize_partition_key_scans (bool) = false,
>>>>>> >> > 33: replica_preference (i32) = 0,
>>>>>> >> > 34: schedule_random_replica (bool) = false,
>>>>>> >> > 35: scan_node_codegen_threshold (i64) = 1800000,
>>>>>> >> > 36: disable_streaming_preaggregations (bool) = false,
>>>>>> >> > 37: runtime_filter_mode (i32) = 2,
>>>>>> >> > 38: runtime_bloom_filter_size (i32) = 1048576,
>>>>>> >> > 39: runtime_filter_wait_time_ms (i32) = 0,
>>>>>> >> > 40: disable_row_runtime_filtering (bool) = false,
>>>>>> >> > 41: max_num_runtime_filters (i32) = 10,
>>>>>> >> > 42: parquet_annotate_strings_utf8 (bool) = false,
>>>>>> >> > 43: parquet_fallback_schema_resolution (i32) = 0,
>>>>>> >> > 45: s3_skip_insert_staging (bool) = true,
>>>>>> >> > 46: runtime_filter_min_size (i32) = 1048576,
>>>>>> >> > 47: runtime_filter_max_size (i32) = 16777216,
>>>>>> >> > 48: prefetch_mode (i32) = 1,
>>>>>> >> > 49: strict_mode (bool) = false,
>>>>>> >> > 50: scratch_limit (i64) = -1,
>>>>>> >> > 51: enable_expr_rewrites (bool) = true,
>>>>>> >> > 52: decimal_v2 (bool) = false,
>>>>>> >> > 53: parquet_dictionary_filtering (bool) = true,
>>>>>> >> > 54: parquet_array_resolution (i32) = 2,
>>>>>> >> > 55: parquet_read_statistics (bool) = true,
>>>>>> >> > 56: default_join_distribution_mode (i32) = 0,
>>>>>> >> > 57: disable_codegen_rows_threshold (i32) = 50000,
>>>>>> >> > 58: default_spillable_buffer_size (i64) = 2097152,
>>>>>> >> > 59: min_spillable_buffer_size (i64) = 65536,
>>>>>> >> > 60: max_row_size (i64) = 524288,
>>>>>> >> > }
>>>>>> >> > I0208 01:07:06.324410 7154 Frontend.java:909] Compiling query:
>>>>>> select
>>>>>> >> > count(*) from dv_entersiteevent_prq_remote where year=2017 and
>>>>>> month=10
>>>>>> >> > I0208 01:07:06.327203 7154 Frontend.java:948] Compiled query.
>>>>>> >> > I0208 01:07:06.359748 7154 admission-controller.cc:510]
>>>>>> Schedule for
>>>>>> >> > id=5f49e927591a9042:faf8f28600000000 in
>>>>>> pool_name=root.impreports
>>>>>> >> > cluster_mem_needed=105.00 MB PoolConfig: max_requests=-1
>>>>>> max_queued=200
>>>>>> >> > max_mem=-1.00 B
>>>>>> >> > I0208 01:07:06.359776 7154 admission-controller.cc:515] Stats:
>>>>>> >> > agg_num_running=0, agg_num_queued=0, agg_mem_reserved=0,
>>>>>> >> > local_host(local_mem_admitted=0, num_admitted_running=0,
>>>>>> num_queued=0,
>>>>>> >> > backend_mem_reserved=0)
>>>>>> >> > I0208 01:07:06.359784 7154 admission-controller.cc:531]
>>>>>> Admitted query
>>>>>> >> > id=5f49e927591a9042:faf8f28600000000
>>>>>> >> > I0208 01:07:06.359807 7154 coordinator.cc:99] Exec()
>>>>>> >> > query_id=5f49e927591a9042:faf8f28600000000 stmt=select count(*)
>>>>>> from
>>>>>> >> > dv_entersiteevent_prq_remote where year=2017 and month=10
>>>>>> >> > I0208 01:07:06.360694 7154 coordinator.cc:357] starting
>>>>>> execution on 5
>>>>>> >> > backends for query 5f49e927591a9042:faf8f28600000000
>>>>>> >> > I0208 01:07:06.363255 63583 impala-internal-service.cc:44]
>>>>>> >> > ExecQueryFInstances(): query_id=5f49e927591a9042:faf8
>>>>>> f28600000000
>>>>>> >> > I0208 01:07:06.363282 63583 query-exec-mgr.cc:46]
>>>>>> StartQueryFInstances()
>>>>>> >> > query_id=5f49e927591a9042:faf8f28600000000
>>>>>> >> > coord=aopr-dhc001.lpdomain.com:22000
>>>>>> >> > I0208 01:07:06.363296 63583 query-state.cc:173] Buffer pool
>>>>>> limit for
>>>>>> >> > 5f49e927591a9042:faf8f28600000000: 10802010521
>>>>>> >> > I0208 01:07:06.363364 63583 initial-reservations.cc:60]
>>>>>> Successfully
>>>>>> >> > claimed
>>>>>> >> > initial reservations (0) for query 5f49e927591a9042:faf8f28600000
>>>>>> 000
>>>>>> >> > I0208 01:07:06.363544 7155 query-state.cc:286]
>>>>>> StartFInstances():
>>>>>> >> > query_id=5f49e927591a9042:faf8f28600000000 #instances=2
>>>>>> >> > I0208 01:07:06.363844 7155 query-state.cc:299] descriptor table
>>>>>> for
>>>>>> >> > query=5f49e927591a9042:faf8f28600000000
>>>>>> >> > tuples:
>>>>>> >> > Tuple(id=2 size=0 slots=[] tuple_path=[])
>>>>>> >> > Tuple(id=1 size=8 slots=[Slot(id=2 type=BIGINT col_path=[]
>>>>>> offset=0
>>>>>> >> > null=(offset=0 mask=0) slot_idx=0 field_idx=-1)] tuple_path=[])
>>>>>> >> > Tuple(id=0 size=8 slots=[Slot(id=3 type=BIGINT col_path=[]
>>>>>> offset=0
>>>>>> >> > null=(offset=0 mask=0) slot_idx=0 field_idx=-1)] tuple_path=[])
>>>>>> >> > I0208 01:07:06.364013 7156 query-state.cc:377] Executing
>>>>>> instance.
>>>>>> >> > instance_id=5f49e927591a9042:faf8f28600000000 fragment_idx=0
>>>>>> >> > per_fragment_instance_idx=0 coord_state_idx=0 #in-flight=1
>>>>>> >> > I0208 01:07:06.364135 7157 query-state.cc:377] Executing
>>>>>> instance.
>>>>>> >> > instance_id=5f49e927591a9042:faf8f28600000005 fragment_idx=1
>>>>>> >> > per_fragment_instance_idx=4 coord_state_idx=0 #in-flight=2
>>>>>> >> > I0208 01:07:06.364241 7157 hdfs-scan-node.cc:160] Max row batch
>>>>>> queue
>>>>>> >> > size
>>>>>> >> > for scan node '0' in fragment instance
>>>>>> >> > '5f49e927591a9042:faf8f28600000005':
>>>>>> >> > 160
>>>>>> >> > I0208 01:07:06.366775 7154 coordinator.cc:370] started
>>>>>> execution on 5
>>>>>> >> > backends for query 5f49e927591a9042:faf8f28600000000
>>>>>> >> > I0208 01:07:06.385625 7155 query-exec-mgr.cc:149]
>>>>>> ReleaseQueryState():
>>>>>> >> > query_id=5f49e927591a9042:faf8f28600000000 refcnt=4
>>>>>> >> > I0208 01:07:06.671193 7559 impala-server.cc:1796] Connection
>>>>>> from
>>>>>> >> > client
>>>>>> >> > ::ffff:10.16.136.3:55598 closed, closing 1 associated session(s)
>>>>>> >> > I0208 01:07:06.676138 544 data-stream-mgr.cc:228] Reduced
>>>>>> stream ID
>>>>>> >> > cache
>>>>>> >> > from 2 items, to 0, eviction took: 0
>>>>>> >> > I0208 01:07:06.677202 544 coordinator.cc:955] Backend
>>>>>> completed:
>>>>>> >> > host=aopr-dhc002.lpdomain.com:22000 remaining=4
>>>>>> >> > I0208 01:07:06.677214 544 coordinator-backend-state.cc:229]
>>>>>> >> > query_id=5f49e927591a9042:faf8f28600000000: first in-progress
>>>>>> backend:
>>>>>> >> > aopr-dhc001.lpdomain.com:22000
>>>>>> >> > I0208 01:07:06.684221 7157 query-state.cc:384] Instance
>>>>>> completed.
>>>>>> >> > instance_id=5f49e927591a9042:faf8f28600000005 #in-flight=1
>>>>>> status=OK
>>>>>> >> > I0208 01:07:06.684237 7157 query-exec-mgr.cc:149]
>>>>>> ReleaseQueryState():
>>>>>> >> > query_id=5f49e927591a9042:faf8f28600000000 refcnt=3
>>>>>> >> > I0208 01:07:06.700275 520 coordinator.cc:955] Backend
>>>>>> completed:
>>>>>> >> > host=aopr-dhc005.lpdomain.com:22000 remaining=3
>>>>>> >> > I0208 01:07:06.700289 520 coordinator-backend-state.cc:229]
>>>>>> >> > query_id=5f49e927591a9042:faf8f28600000000: first in-progress
>>>>>> backend:
>>>>>> >> > aopr-dhc001.lpdomain.com:22000
>>>>>> >> > I0208 01:07:06.704620 65258 coordinator.cc:955] Backend
>>>>>> completed:
>>>>>> >> > host=aopr-dhc003.lpdomain.com:22000 remaining=2
>>>>>> >> > I0208 01:07:06.704666 65258 coordinator-backend-state.cc:229]
>>>>>> >> > query_id=5f49e927591a9042:faf8f28600000000: first in-progress
>>>>>> backend:
>>>>>> >> > aopr-dhc001.lpdomain.com:22000
>>>>>> >> > I0208 01:07:06.706041 7156 data-stream-mgr.cc:238]
>>>>>> DeregisterRecvr():
>>>>>> >> > fragment_instance_id=5f49e927591a9042:faf8f28600000000, node=2
>>>>>> >> > I0208 01:07:06.706070 7156 data-stream-recvr.cc:235] cancelled
>>>>>> stream:
>>>>>> >> > fragment_instance_id_=5f49e927591a9042:faf8f28600000000
>>>>>> node_id=2
>>>>>> >> > I0208 01:07:06.706809 64916 coordinator.cc:955] Backend
>>>>>> completed:
>>>>>> >> > host=aopr-dhc004.lpdomain.com:22000 remaining=1
>>>>>> >> > I0208 01:07:06.706821 64916 coordinator-backend-state.cc:229]
>>>>>> >> > query_id=5f49e927591a9042:faf8f28600000000: first in-progress
>>>>>> backend:
>>>>>> >> > aopr-dhc001.lpdomain.com:22000
>>>>>> >> > I0208 01:07:06.789568 7154 impala-beeswax-server.cc:193]
>>>>>> >> > get_results_metadata(): query_id=5f49e927591a9042:faf8
>>>>>> f28600000000
>>>>>> >> > I0208 01:07:06.790439 7154 coordinator.cc:789] Coordinator
>>>>>> waiting for
>>>>>> >> > backends to finish, 1 remaining
>>>>>> >> > I0208 01:07:06.791052 7154 coordinator.cc:794] All backends
>>>>>> finished
>>>>>> >> > successfully.
>>>>>> >> > I0208 01:07:06.791086 7154 coordinator.cc:1090] Release admssion
>>>>>> >> > control
>>>>>> >> > resources for query 5f49e927591a9042:faf8f28600000000
>>>>>> >> > I0208 01:07:06.791363 7156 query-state.cc:384] Instance
>>>>>> completed.
>>>>>> >> > instance_id=5f49e927591a9042:faf8f28600000000 #in-flight=0
>>>>>> status=OK
>>>>>> >> > I0208 01:07:06.791414 7156 query-exec-mgr.cc:149]
>>>>>> ReleaseQueryState():
>>>>>> >> > query_id=5f49e927591a9042:faf8f28600000000 refcnt=2
>>>>>> >> > I0208 01:07:06.793586 7154 impala-beeswax-server.cc:235]
>>>>>> close():
>>>>>> >> > query_id=5f49e927591a9042:faf8f28600000000
>>>>>> >> > I0208 01:07:06.793602 7154 impala-server.cc:992]
>>>>>> UnregisterQuery():
>>>>>> >> > query_id=5f49e927591a9042:faf8f28600000000
>>>>>> >> > I0208 01:07:06.793614 7154 impala-server.cc:1075] Cancel():
>>>>>> >> > query_id=5f49e927591a9042:faf8f28600000000
>>>>>> >> > I0208 01:07:06.793624 7154 coordinator.cc:895] Cancel()
>>>>>> >> > query_id=5f49e927591a9042:faf8f28600000000
>>>>>> >> > I0208 01:07:06.793637 7154 coordinator.cc:905] CancelBackends()
>>>>>> >> > query_id=5f49e927591a9042:faf8f28600000000, tried to cancel 0
>>>>>> backends
>>>>>> >> > I0208 01:07:06.802776 7154 query-exec-mgr.cc:149]
>>>>>> ReleaseQueryState():
>>>>>> >> > query_id=5f49e927591a9042:faf8f28600000000 refcnt=1
>>>>>> >> > I0208 01:07:06.814214 7154 impala-server.cc:1796] Connection
>>>>>> from
>>>>>> >> > client
>>>>>> >> > ::ffff:10.16.144.131:51012 closed, closing 1 associated
>>>>>> session(s)
>>>>>> >> > I0208 01:07:07.258891 7567 impala-server.cc:1796] Connection
>>>>>> from
>>>>>> >> > client
>>>>>> >> > ::ffff:10.16.136.2:56375 closed, closing 1 associated session(s)
>>>>>> >> > I0208 01:07:11.600131 7819 impala-server.cc:1796] Connection
>>>>>> from
>>>>>> >> > client
>>>>>> >> > ::ffff:10.16.136.3:27460 closed, closing 1 associated session(s)
>>>>>> >> > I0208 01:07:12.262739 7821 impala-server.cc:1796] Connection
>>>>>> from
>>>>>> >> > client
>>>>>> >> > ::ffff:10.16.136.2:52672 closed, closing 1 associated session(s)
>>>>>> >> > I0208 01:07:16.665451 7822 impala-server.cc:1796] Connection
>>>>>> from
>>>>>> >> > client
>>>>>> >> > ::ffff:10.16.136.3:10693 closed, closing 1 associated session(s)
>>>>>> >> > I0208 01:07:17.235321 7825 impala-server.cc:1796] Connection
>>>>>> from
>>>>>> >> > client
>>>>>> >> > ::ffff:10.16.136.2:34752 closed, closing 1 associated session(s)
>>>>>> >> > I0208 01:07:21.410611 5858 impala-beeswax-server.cc:193]
>>>>>> >> > get_results_metadata(): query_id=1d4fbf0f2571f1e9:ea34
>>>>>> 6f7900000000
>>>>>> >> > I0208 01:07:21.414175 5858 impala-beeswax-server.cc:235]
>>>>>> close():
>>>>>> >> > query_id=1d4fbf0f2571f1e9:ea346f7900000000
>>>>>> >> > I0208 01:07:21.414185 5858 impala-server.cc:992]
>>>>>> UnregisterQuery():
>>>>>> >> > query_id=1d4fbf0f2571f1e9:ea346f7900000000
>>>>>> >> > I0208 01:07:21.414191 5858 impala-server.cc:1075] Cancel():
>>>>>> >> > query_id=1d4fbf0f2571f1e9:ea346f7900000000
>>>>>> >> > I0208 01:07:21.421505 5858 impala-server.cc:1796] Connection
>>>>>> from
>>>>>> >> > client
>>>>>> >> > ::ffff:10.26.180.75:60072 closed, closing 1 associated
>>>>>> session(s)
>>>>>> >> > I0208 01:07:21.605690 7830 impala-server.cc:1796] Connection
>>>>>> from
>>>>>> >> > client
>>>>>> >> > ::ffff:10.16.136.3:46660 closed, closing 1 associated session(s)
>>>>>> >> >
>>>>>> >> >
>>>>>> >> > On Thu, Feb 8, 2018 at 6:20 AM, Matthew Jacobs <
>>>>>> mjacobs@apache.org>
>>>>>> >> > wrote:
>>>>>> >> >>
>>>>>> >> >> Thanks! I think that configuration looks OK.
>>>>>> >> >>
>>>>>> >> >> 1) What version of Impala is this?
>>>>>> >> >>
>>>>>> >> >> 2) Can you send the query profile and the Impala coordinator
>>>>>> log for
>>>>>> >> >> the query where it's not behaving as expected?
>>>>>> >> >>
>>>>>> >> >> > When i'm running the following with fawzea user:
>>>>>> >> >> > set request_pool=impreports;
>>>>>> >> >> > select count(*) from dv_entersiteevent_prq_remote where
>>>>>> year=2017 and
>>>>>> >> >> > month=10;
>>>>>> >> >> > I can see that the query is running with impreports pool.
>>>>>> >> >>
>>>>>> >> >> On Wed, Feb 7, 2018 at 7:59 PM, Fawze Abujaber <
>>>>>> fawzeaj@gmail.com>
>>>>>> >> >> wrote:
>>>>>> >> >> > Hi Matt,
>>>>>> >> >> >
>>>>>> >> >> > [root@aopr-dhc001 ~]# cat
>>>>>> >> >> >
>>>>>> >> >> >
>>>>>> >> >> > /var/run/cloudera-scm-agent/process/4558-impala-IMPALAD/impa
>>>>>> la-conf/fair-scheduler.xml
>>>>>> >> >> > <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
>>>>>> >> >> > <allocations>
>>>>>> >> >> > <queue name="root">
>>>>>> >> >> > <weight>1.0</weight>
>>>>>> >> >> > <schedulingPolicy>fair</schedulingPolicy>
>>>>>> >> >> > <queue name="impiaas">
>>>>>> >> >> > <weight>1.0</weight>
>>>>>> >> >> > <schedulingPolicy>fair</schedulingPolicy>
>>>>>> >> >> > <aclSubmitApps>fawzea </aclSubmitApps>
>>>>>> >> >> > </queue>
>>>>>> >> >> > <queue name="impreports">
>>>>>> >> >> > <weight>1.0</weight>
>>>>>> >> >> > <schedulingPolicy>fair</schedulingPolicy>
>>>>>> >> >> > <aclSubmitApps>nira </aclSubmitApps>
>>>>>> >> >> > </queue>
>>>>>> >> >> > </queue>
>>>>>> >> >> > <defaultQueueSchedulingPolicy
>>>>>> >fair</defaultQueueSchedulingPolicy>
>>>>>> >> >> > <queuePlacementPolicy>
>>>>>> >> >> > <rule name="specified" create="false"/>
>>>>>> >> >> > <rule name="secondaryGroupExistingQueue"/>
>>>>>> >> >> > <rule name="default"/>
>>>>>> >> >> > </queuePlacementPolicy>
>>>>>> >> >> > </allocations>
>>>>>> >> >> >
>>>>>> >> >> >
>>>>>> >> >> > On Thu, Feb 8, 2018 at 4:47 AM, Matthew Jacobs <
>>>>>> jacobs.mh@gmail.com>
>>>>>> >> >> > wrote:
>>>>>> >> >> >>
>>>>>> >> >> >> Yes, I saw that, but I'd like to see the actual
>>>>>> configuration that
>>>>>> >> >> >> is
>>>>>> >> >> >> passed to Impala because it may not be specified in the
>>>>>> right way to
>>>>>> >> >> >> accomplish what you're trying to do in "b".
>>>>>> >> >> >>
>>>>>> >> >> >> On Wed, Feb 7, 2018 at 4:32 PM, Fawze Abujaber <
>>>>>> fawzeaj@gmail.com>
>>>>>> >> >> >> wrote:
>>>>>> >> >> >> > Hi Matt,
>>>>>> >> >> >> >
>>>>>> >> >> >> > I created the pools manually and it’s completely
>>>>>> nfigurations as
>>>>>> >> >> >> > shown
>>>>>> >> >> >> > in b.
>>>>>> >> >> >> >
>>>>>> >> >> >> > On Wed, 7 Feb 2018 at 23:17 Matthew Jacobs <
>>>>>> jacobs.mh@gmail.com>
>>>>>> >> >> >> > wrote:
>>>>>> >> >> >> >>
>>>>>> >> >> >> >> How, more specifically, are you configuring these pools?
>>>>>> Can you
>>>>>> >> >> >> >> share
>>>>>> >> >> >> >> the configuration files?
>>>>>> >> >> >> >>
>>>>>> >> >> >> >> On Wed, Feb 7, 2018 at 11:42 AM, Fawze Abujaber
>>>>>> >> >> >> >> <fa...@gmail.com>
>>>>>> >> >> >> >> wrote:
>>>>>> >> >> >> >> > Hello All,
>>>>>> >> >> >> >> >
>>>>>> >> >> >> >> > Will appreciate any insight of what i'm missing in my
>>>>>> >> >> >> >> > configuration
>>>>>> >> >> >> >> > to
>>>>>> >> >> >> >> > get
>>>>>> >> >> >> >> > the impala submission control working as expected ....
>>>>>> >> >> >> >> >
>>>>>> >> >> >> >> > a- Here is my placement Placement Rule:
>>>>>> >> >> >> >> > 1- Use the pool Specified at run time., only if the pool
>>>>>> >> >> >> >> > exists.
>>>>>> >> >> >> >> > 2- Use the pool root.[secondary group].
>>>>>> >> >> >> >> > 3- Use the pool root.default.
>>>>>> >> >> >> >> >
>>>>>> >> >> >> >> > b- Here is the impala resource pools
>>>>>> >> >> >> >> >
>>>>>> >> >> >> >> > Name Max Memory Max Running Queries
>>>>>> Max
>>>>>> >> >> >> >> > Queued
>>>>>> >> >> >> >> > Queries
>>>>>> >> >> >> >> > Queue Timeout Default Query Memory Limit Access Control
>>>>>> >> >> >> >> > root.impiaas Unlimited
>>>>>> Unlimited
>>>>>> >> >> >> >> > 200 1 minute(s) No
>>>>>> Default
>>>>>> >> >> >> >> > Custom
>>>>>> >> >> >> >> > root.impreports Unlimited
>>>>>> Unlimited
>>>>>> >> >> >> >> > 200 1 minute(s) No
>>>>>> Default
>>>>>> >> >> >> >> > Custom
>>>>>> >> >> >> >> >
>>>>>> >> >> >> >> >
>>>>>> >> >> >> >> > C- Here is my Access control for root.impiaas pool:
>>>>>> >> >> >> >> >
>>>>>> >> >> >> >> > Allow these users and groups to submit to this pool
>>>>>> >> >> >> >> >
>>>>>> >> >> >> >> > Users: fawzea
>>>>>> >> >> >> >> >
>>>>>> >> >> >> >> > The same for impreports the allowed user and groups to
>>>>>> submit
>>>>>> >> >> >> >> > to
>>>>>> >> >> >> >> > pool
>>>>>> >> >> >> >> > is
>>>>>> >> >> >> >> > only nira.
>>>>>> >> >> >> >> >
>>>>>> >> >> >> >> >
>>>>>> >> >> >> >> > LDAP groups for fawzea user:
>>>>>> >> >> >> >> >
>>>>>> >> >> >> >> > Member in groups:
>>>>>> >> >> >> >> >
>>>>>> >> >> >> >> > - dba
>>>>>> >> >> >> >> > - impiaas
>>>>>> >> >> >> >> > - mysql
>>>>>> >> >> >> >> >
>>>>>> >> >> >> >> > LDAP group for nira user
>>>>>> >> >> >> >> > Member in groups:
>>>>>> >> >> >> >> > - dba
>>>>>> >> >> >> >> > - impreports
>>>>>> >> >> >> >> > - mysql
>>>>>> >> >> >> >> >
>>>>>> >> >> >> >> >
>>>>>> >> >> >> >> >
>>>>>> >> >> >> >> > Impala Queries UI shows:
>>>>>> >> >> >> >> >
>>>>>> >> >> >> >> > 1- select count(*) from dv_entersiteevent_prq_remote
>>>>>> where
>>>>>> >> >> >> >> > year=2017
>>>>>> >> >> >> >> >
>>>>>> >> >> >> >> > User: fawzea
>>>>>> >> >> >> >> > Pool: root.impiaas
>>>>>> >> >> >> >> >
>>>>>> >> >> >> >> > 2- select count(*) from dv_entersiteevent_prq_remote
>>>>>> where
>>>>>> >> >> >> >> > year=2017
>>>>>> >> >> >> >> > and
>>>>>> >> >> >> >> > month=10
>>>>>> >> >> >> >> > User: nira
>>>>>> >> >> >> >> > Pool: root.impreports
>>>>>> >> >> >> >> >
>>>>>> >> >> >> >> > As shown above that the queries submitted based on the
>>>>>> >> >> >> >> > secondary
>>>>>> >> >> >> >> > group
>>>>>> >> >> >> >> > rule.
>>>>>> >> >> >> >> >
>>>>>> >> >> >> >> > When i'm running the following with fawzea user:
>>>>>> >> >> >> >> >
>>>>>> >> >> >> >> > set request_pool=impreports;
>>>>>> >> >> >> >> > select count(*) from dv_entersiteevent_prq_remote where
>>>>>> >> >> >> >> > year=2017
>>>>>> >> >> >> >> > and
>>>>>> >> >> >> >> > month=10;
>>>>>> >> >> >> >> >
>>>>>> >> >> >> >> > I can see that the query is running with impreports
>>>>>> pool.
>>>>>> >> >> >> >> >
>>>>>> >> >> >> >> >
>>>>>> >> >> >> >> > What i'm missing here in my configuration to get this
>>>>>> works as
>>>>>> >> >> >> >> > expected.
>>>>>> >> >> >> >> >
>>>>>> >> >> >
>>>>>> >> >> >
>>>>>> >> >
>>>>>> >> >
>>>>>> >
>>>>>> >
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: impala submission control
Posted by Tim Armstrong <ta...@cloudera.com>.
Hi Fawze,
I finally had some time to dig into what was going on here. It looks like
it's a config problem - there is no ACL for the root queue. Without
specifying an ACL, everyone has access and and those permissions are
inherited by all children of "root".
- Tim
On Thu, Feb 8, 2018 at 1:03 PM, Fawze Abujaber <fa...@gmail.com> wrote:
> Sorry, but i'm able to replicate the issue on impala
> version: 2.10.0-cdh5.13.0
>
> I will be happy if anyone who can try to replicate the issue on his test
> cluster.
>
> On Thu, Feb 8, 2018 at 8:25 PM, Matthew Jacobs <ja...@gmail.com>
> wrote:
>
>> I just filed one: https://issues.apache.org/jira/browse/IMPALA-6493
>>
>> On Thu, Feb 8, 2018 at 10:12 AM, Matthew Jacobs <ja...@gmail.com>
>> wrote:
>>
>>> As I said, I don't see anything wrong with the configuration so there
>>> could be a bug in Impala 2.11, likely due to IMPALA-5920. Can you file a
>>> JIRA?
>>>
>>> On Thu, Feb 8, 2018 at 9:42 AM, Fawze Abujaber <fa...@gmail.com>
>>> wrote:
>>>
>>>> This is not fitting my situation,
>>>>
>>>> I removed even the placement role of the secondary group, and other
>>>> users can still submit to the pool that i restrict it to my user.
>>>>
>>>> Just thinking in loud and think the problem might be in using the
>>>> placement rule:
>>>>
>>>> Use the pool Specified at run time., only if the pool exists.
>>>> Edit
>>>>
>>>>
>>>> as it it the first rule, it always successes.
>>>>
>>>>
>>>> On Thu, Feb 8, 2018 at 7:12 PM, Matthew Jacobs <ja...@gmail.com>
>>>> wrote:
>>>>
>>>>> The aclSubmitApps can specify users or groups, so if the user name is
>>>>> also a group name, that can lead to confusion, e.g.
>>>>> https://issues.apache.org/jira/browse/IMPALA-2578
>>>>> I'm not saying that is necessarily the problem, but something to
>>>>> check. If we still can't find anything configured incorrectly I'd
>>>>> recommend that you file a JIRA.
>>>>>
>>>>> On Thu, Feb 8, 2018 at 9:05 AM, Fawze Abujaber <fa...@gmail.com>
>>>>> wrote:
>>>>> > Hi Matt,
>>>>> >
>>>>> > If this is the situation, how we can explain that when i submit the
>>>>> query it
>>>>> > runs on my secondary group pool and when nira do the same it runs
>>>>> under her
>>>>> > secondary group pool.
>>>>> >
>>>>> > More than this, the issue not related to LDAP group as i'm using the
>>>>> > submission restriction to a user and not a group.
>>>>> >
>>>>> > On Thu, Feb 8, 2018 at 7:00 PM, Matthew Jacobs <ja...@gmail.com>
>>>>> wrote:
>>>>> >>
>>>>> >> As far as I can tell from this information, your configuration looks
>>>>> >> fine. The only other potential issue I can think of is that the
>>>>> >> secondary groups placement rule isn't picking up your LDAP groups
>>>>> >> properly. Does the ACL behavior work if you remove the secondary
>>>>> >> groups rule?
>>>>> >>
>>>>> >> On Wed, Feb 7, 2018 at 10:09 PM, Fawze Abujaber <fa...@gmail.com>
>>>>> wrote:
>>>>> >> > Hi Matt,
>>>>> >> >
>>>>> >> > v2.11.0-cdh5.14.0
>>>>> >> >
>>>>> >> > Query Runtime Profile:
>>>>> >> > Query (id=c843dae48ce9b289:6ad3ffc000000000):
>>>>> >> > Summary:
>>>>> >> > Session ID: ca420ac477772b56:51ae460ce9e44999
>>>>> >> > Session Type: BEESWAX
>>>>> >> > Start Time: 2018-02-08 01:02:01.023558000
>>>>> >> > End Time: 2018-02-08 01:02:01.501506000
>>>>> >> > Query Type: QUERY
>>>>> >> > Query State: FINISHED
>>>>> >> > Query Status: OK
>>>>> >> > Impala Version: impalad version 2.11.0-cdh5.14.0 RELEASE
>>>>> (build
>>>>> >> > d68206561bce6b26762d62c01a78e6cd27aa7690)
>>>>> >> > User: fawzea
>>>>> >> > Connected User: fawzea
>>>>> >> > Delegated User:
>>>>> >> > Network Address: ::ffff:10.16.144.131:50078
>>>>> >> > Default Db: default
>>>>> >> > Sql Statement: select count(*) from
>>>>> dv_entersiteevent_prq_remote
>>>>> >> > where
>>>>> >> > year=2017 and month=10
>>>>> >> > Coordinator: aopr-dhc001.lpdomain.com:22000
>>>>> >> > Query Options (set by configuration): REQUEST_POOL=impreports
>>>>> >> > Query Options (set by configuration and planner):
>>>>> >> > REQUEST_POOL=impreports,MT_DOP=0
>>>>> >> > Plan:
>>>>> >> > ----------------
>>>>> >> > Max Per-Host Resource Reservation: Memory=0B
>>>>> >> > Per-Host Resource Estimates: Memory=21.00MB
>>>>> >> > WARNING: The following tables are missing relevant table and/or
>>>>> column
>>>>> >> > statistics.
>>>>> >> > default.dv_entersiteevent_prq_remote
>>>>> >> >
>>>>> >> > F01:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1
>>>>> >> > | Per-Host Resources: mem-estimate=10.00MB mem-reservation=0B
>>>>> >> > PLAN-ROOT SINK
>>>>> >> > | mem-estimate=0B mem-reservation=0B
>>>>> >> > |
>>>>> >> > 03:AGGREGATE [FINALIZE]
>>>>> >> > | output: count:merge(*)
>>>>> >> > | mem-estimate=10.00MB mem-reservation=0B spill-buffer=2.00MB
>>>>> >> > | tuple-ids=1 row-size=8B cardinality=1
>>>>> >> > |
>>>>> >> > 02:EXCHANGE [UNPARTITIONED]
>>>>> >> > | mem-estimate=0B mem-reservation=0B
>>>>> >> > | tuple-ids=1 row-size=8B cardinality=1
>>>>> >> > |
>>>>> >> > F00:PLAN FRAGMENT [RANDOM] hosts=5 instances=5
>>>>> >> > Per-Host Resources: mem-estimate=11.00MB mem-reservation=0B
>>>>> >> > 01:AGGREGATE
>>>>> >> > | output:
>>>>> >> > sum_init_zero(default.dv_entersiteevent_prq_remote.parquet-stats:
>>>>> >> > num_rows)
>>>>> >> > | mem-estimate=10.00MB mem-reservation=0B spill-buffer=2.00MB
>>>>> >> > | tuple-ids=1 row-size=8B cardinality=1
>>>>> >> > |
>>>>> >> > 00:SCAN HDFS [default.dv_entersiteevent_prq_remote, RANDOM]
>>>>> >> > partitions=31/441 files=2292 size=1.39GB
>>>>> >> > stats-rows=unavailable extrapolated-rows=disabled
>>>>> >> > table stats: rows=unavailable size=unavailable
>>>>> >> > column stats: all
>>>>> >> > mem-estimate=1.00MB mem-reservation=0B
>>>>> >> > tuple-ids=0 row-size=8B cardinality=unavailable
>>>>> >> > ----------------
>>>>> >> > Estimated Per-Host Mem: 22020096
>>>>> >> > Tables Missing Stats: default.dv_entersiteevent_prq_remote
>>>>> >> > Per Host Min Reservation: aopr-dhc001.lpdomain.com:22000(0)
>>>>> >> > aopr-dhc002.lpdomain.com:22000(0) aopr-dhc003.lpdomain.com:22000
>>>>> (0)
>>>>> >> > aopr-dhc004.lpdomain.com:22000(0) aopr-dhc005.lpdomain.com:22000
>>>>> (0)
>>>>> >> > Request Pool: root.impreports
>>>>> >> > Admission result: Admitted immediately
>>>>> >> > ExecSummary:
>>>>> >> > Operator #Hosts Avg Time Max Time #Rows Est. #Rows
>>>>> Peak Mem
>>>>> >> > Est. Peak Mem Detail
>>>>> >> >
>>>>> >> > ------------------------------------------------------------
>>>>> ------------------------------------------------------------------
>>>>> >> > 03:AGGREGATE 1 222.958us 222.958us 1 1
>>>>> 36.00 KB
>>>>> >> > 10.00 MB FINALIZE
>>>>> >> > 02:EXCHANGE 1 357.508ms 357.508ms 5 1
>>>>> 0
>>>>> >> > 0 UNPARTITIONED
>>>>> >> > 01:AGGREGATE 5 1.027ms 1.541ms 5 1
>>>>> 20.00 KB
>>>>> >> > 10.00 MB
>>>>> >> > 00:SCAN HDFS 5 282.797ms 333.715ms 2.29K -1
>>>>> 2.62 MB
>>>>> >> > 1.00 MB default.dv_entersiteevent_prq_remote
>>>>> >> > Errors:
>>>>> >> > Planner Timeline: 13.508ms
>>>>> >> > - Analysis finished: 3.303ms (3.303ms)
>>>>> >> > - Value transfer graph computed: 3.465ms (162.667us)
>>>>> >> > - Single node plan created: 10.352ms (6.886ms)
>>>>> >> > - Runtime filters computed: 10.677ms (325.364us)
>>>>> >> > - Distributed plan created: 10.733ms (55.854us)
>>>>> >> > - Lineage info computed: 10.898ms (164.937us)
>>>>> >> > - Planning finished: 13.508ms (2.610ms)
>>>>> >> > Query Timeline: 481.950ms
>>>>> >> > - Query submitted: 79.971us (79.971us)
>>>>> >> > - Planning finished: 33.947ms (33.867ms)
>>>>> >> > - Submit for admission: 40.067ms (6.120ms)
>>>>> >> > - Completed admission: 40.197ms (130.146us)
>>>>> >> > - Ready to start on 5 backends: 40.905ms (707.852us)
>>>>> >> > - All 5 execution backends (6 fragment instances) started:
>>>>> >> > 46.417ms
>>>>> >> > (5.512ms)
>>>>> >> > - Rows available: 424.548ms (378.131ms)
>>>>> >> > - First row fetched: 473.488ms (48.939ms)
>>>>> >> > - Unregister query: 477.951ms (4.462ms)
>>>>> >> > - ComputeScanRangeAssignmentTimer: 3.980ms
>>>>> >> > ImpalaServer:
>>>>> >> > - ClientFetchWaitTimer: 51.659ms
>>>>> >> > - RowMaterializationTimer: 1.726ms
>>>>> >> > Execution Profile c843dae48ce9b289:6ad3ffc000000000:(Total:
>>>>> 385.029ms,
>>>>> >> > non-child: 0.000ns, % non-child: 0.00%)
>>>>> >> > Number of filters: 0
>>>>> >> > Filter routing table:
>>>>> >> > ID Src. Node Tgt. Node(s) Target type Partition filter
>>>>> Pending
>>>>> >> > (Expected) First arrived Completed Enabled
>>>>> >> >
>>>>> >> > ------------------------------------------------------------
>>>>> -------------------------------------------------------
>>>>> >> >
>>>>> >> > Backend startup latencies: Count: 5, min / max: 3ms / 4ms,
>>>>> 25th
>>>>> >> > %-ile:
>>>>> >> > 3ms, 50th %-ile: 4ms, 75th %-ile: 4ms, 90th %-ile: 4ms, 95th
>>>>> %-ile: 4ms,
>>>>> >> > 99.9th %-ile: 4ms
>>>>> >> > Per Node Peak Memory Usage: aopr-dhc001.lpdomain.com:22000(2.68
>>>>> MB)
>>>>> >> > aopr-dhc002.lpdomain.com:22000(2.54 MB)
>>>>> >> > aopr-dhc003.lpdomain.com:22000(2.54
>>>>> >> > MB) aopr-dhc004.lpdomain.com:22000(2.41 MB)
>>>>> >> > aopr-dhc005.lpdomain.com:22000(2.56 MB)
>>>>> >> > - FiltersReceived: 0 (0)
>>>>> >> > - FinalizationTimer: 0.000ns
>>>>> >> > - NumBackends: 5 (5)
>>>>> >> > - NumFragmentInstances: 6 (6)
>>>>> >> > - NumFragments: 2 (2)
>>>>> >> > Averaged Fragment F01:(Total: 427.530ms, non-child: 49.428ms,
>>>>> %
>>>>> >> > non-child: 11.56%)
>>>>> >> > split sizes: min: 0, max: 0, avg: 0, stddev: 0
>>>>> >> > completion times: min:428.693ms max:428.693ms mean:
>>>>> 428.693ms
>>>>> >> > stddev:0.000ns
>>>>> >> > execution rates: min:0.00 /sec max:0.00 /sec mean:0.00
>>>>> /sec
>>>>> >> > stddev:0.00 /sec
>>>>> >> > num instances: 1
>>>>> >> > - AverageThreadTokens: 0.00
>>>>> >> > - BloomFilterBytes: 0
>>>>> >> > - PeakMemoryUsage: 44.02 KB (45072)
>>>>> >> > - PeakReservation: 0
>>>>> >> > - PeakUsedReservation: 0
>>>>> >> > - PerHostPeakMemUsage: 2.68 MB (2806113)
>>>>> >> > - RowsProduced: 1 (1)
>>>>> >> > - TotalNetworkReceiveTime: 357.481ms
>>>>> >> > - TotalNetworkSendTime: 0.000ns
>>>>> >> > - TotalStorageWaitTime: 0.000ns
>>>>> >> > - TotalThreadsInvoluntaryContextSwitches: 0 (0)
>>>>> >> > - TotalThreadsTotalWallClockTime: 406.708ms
>>>>> >> > - TotalThreadsSysTime: 0.000ns
>>>>> >> > - TotalThreadsUserTime: 0.000ns
>>>>> >> > - TotalThreadsVoluntaryContextSwitches: 7 (7)
>>>>> >> > Fragment Instance Lifecycle Timings:
>>>>> >> > - ExecTime: 49.055ms
>>>>> >> > - ExecTreeExecTime: 4.251us
>>>>> >> > - OpenTime: 357.663ms
>>>>> >> > - ExecTreeOpenTime: 357.649ms
>>>>> >> > - PrepareTime: 20.764ms
>>>>> >> > - ExecTreePrepareTime: 81.759us
>>>>> >> > PLAN_ROOT_SINK:
>>>>> >> > - PeakMemoryUsage: 0
>>>>> >> > CodeGen:(Total: 20.369ms, non-child: 20.369ms, % non-child:
>>>>> >> > 100.00%)
>>>>> >> > - CodegenTime: 0.000ns
>>>>> >> > - CompileTime: 0.000ns
>>>>> >> > - LoadTime: 0.000ns
>>>>> >> > - ModuleBitcodeSize: 1.86 MB (1950880)
>>>>> >> > - NumFunctions: 0 (0)
>>>>> >> > - NumInstructions: 0 (0)
>>>>> >> > - OptimizationTime: 0.000ns
>>>>> >> > - PeakMemoryUsage: 0
>>>>> >> > - PrepareTime: 19.685ms
>>>>> >> > AGGREGATION_NODE (id=3):(Total: 357.731ms, non-child:
>>>>> 222.958us, %
>>>>> >> > non-child: 0.06%)
>>>>> >> > - BuildTime: 7.368us
>>>>> >> > - GetResultsTime: 0.000ns
>>>>> >> > - HTResizeTime: 0.000ns
>>>>> >> > - HashBuckets: 0 (0)
>>>>> >> > - LargestPartitionPercent: 0 (0)
>>>>> >> > - MaxPartitionLevel: 0 (0)
>>>>> >> > - NumRepartitions: 0 (0)
>>>>> >> > - PartitionsCreated: 0 (0)
>>>>> >> > - PeakMemoryUsage: 36.00 KB (36864)
>>>>> >> > - RowsRepartitioned: 0 (0)
>>>>> >> > - RowsReturned: 1 (1)
>>>>> >> > - RowsReturnedRate: 2.00 /sec
>>>>> >> > - SpilledPartitions: 0 (0)
>>>>> >> > EXCHANGE_NODE (id=2):(Total: 357.508ms, non-child:
>>>>> 357.508ms, %
>>>>> >> > non-child: 100.00%)
>>>>> >> > - ConvertRowBatchTime: 8.881us
>>>>> >> > - PeakMemoryUsage: 0
>>>>> >> > - RowsReturned: 5 (5)
>>>>> >> > - RowsReturnedRate: 13.00 /sec
>>>>> >> > DataStreamReceiver:
>>>>> >> > - BytesReceived: 80.00 B (80)
>>>>> >> > - DeserializeRowBatchTimer: 24.165us
>>>>> >> > - FirstBatchArrivalWaitTime: 277.194ms
>>>>> >> > - PeakMemoryUsage: 4.02 KB (4112)
>>>>> >> > - SendersBlockedTimer: 0.000ns
>>>>> >> > - SendersBlockedTotalTimer(*): 0.000ns
>>>>> >> > Coordinator Fragment F01:
>>>>> >> > Instance c843dae48ce9b289:6ad3ffc000000000
>>>>> >> > (host=aopr-dhc001.lpdomain.com:22000):(Total: 427.530ms,
>>>>> non-child:
>>>>> >> > 49.428ms, % non-child: 11.56%)
>>>>> >> > - AverageThreadTokens: 0.00
>>>>> >> > - BloomFilterBytes: 0
>>>>> >> > - PeakMemoryUsage: 44.02 KB (45072)
>>>>> >> > - PeakReservation: 0
>>>>> >> > - PeakUsedReservation: 0
>>>>> >> > - PerHostPeakMemUsage: 2.68 MB (2806113)
>>>>> >> > - RowsProduced: 1 (1)
>>>>> >> > - TotalNetworkReceiveTime: 357.481ms
>>>>> >> > - TotalNetworkSendTime: 0.000ns
>>>>> >> > - TotalStorageWaitTime: 0.000ns
>>>>> >> > - TotalThreadsInvoluntaryContextSwitches: 0 (0)
>>>>> >> > - TotalThreadsTotalWallClockTime: 406.708ms
>>>>> >> > - TotalThreadsSysTime: 0.000ns
>>>>> >> > - TotalThreadsUserTime: 0.000ns
>>>>> >> > - TotalThreadsVoluntaryContextSwitches: 7 (7)
>>>>> >> > Fragment Instance Lifecycle Timings:
>>>>> >> > - ExecTime: 49.055ms
>>>>> >> > - ExecTreeExecTime: 4.251us
>>>>> >> > - OpenTime: 357.663ms
>>>>> >> > - ExecTreeOpenTime: 357.649ms
>>>>> >> > - PrepareTime: 20.764ms
>>>>> >> > - ExecTreePrepareTime: 81.759us
>>>>> >> > PLAN_ROOT_SINK:
>>>>> >> > - PeakMemoryUsage: 0
>>>>> >> > CodeGen:(Total: 20.369ms, non-child: 20.369ms, %
>>>>> non-child:
>>>>> >> > 100.00%)
>>>>> >> > - CodegenTime: 0.000ns
>>>>> >> > - CompileTime: 0.000ns
>>>>> >> > - LoadTime: 0.000ns
>>>>> >> > - ModuleBitcodeSize: 1.86 MB (1950880)
>>>>> >> > - NumFunctions: 0 (0)
>>>>> >> > - NumInstructions: 0 (0)
>>>>> >> > - OptimizationTime: 0.000ns
>>>>> >> > - PeakMemoryUsage: 0
>>>>> >> > - PrepareTime: 19.685ms
>>>>> >> > AGGREGATION_NODE (id=3):(Total: 357.731ms, non-child:
>>>>> 222.958us,
>>>>> >> > %
>>>>> >> > non-child: 0.06%)
>>>>> >> > - BuildTime: 7.368us
>>>>> >> > - GetResultsTime: 0.000ns
>>>>> >> > - HTResizeTime: 0.000ns
>>>>> >> > - HashBuckets: 0 (0)
>>>>> >> > - LargestPartitionPercent: 0 (0)
>>>>> >> > - MaxPartitionLevel: 0 (0)
>>>>> >> > - NumRepartitions: 0 (0)
>>>>> >> > - PartitionsCreated: 0 (0)
>>>>> >> > - PeakMemoryUsage: 36.00 KB (36864)
>>>>> >> > - RowsRepartitioned: 0 (0)
>>>>> >> > - RowsReturned: 1 (1)
>>>>> >> > - RowsReturnedRate: 2.00 /sec
>>>>> >> > - SpilledPartitions: 0 (0)
>>>>> >> > EXCHANGE_NODE (id=2):(Total: 357.508ms, non-child:
>>>>> 357.508ms, %
>>>>> >> > non-child: 100.00%)
>>>>> >> > - ConvertRowBatchTime: 8.881us
>>>>> >> > - PeakMemoryUsage: 0
>>>>> >> > - RowsReturned: 5 (5)
>>>>> >> > - RowsReturnedRate: 13.00 /sec
>>>>> >> > DataStreamReceiver:
>>>>> >> > - BytesReceived: 80.00 B (80)
>>>>> >> > - DeserializeRowBatchTimer: 24.165us
>>>>> >> > - FirstBatchArrivalWaitTime: 277.194ms
>>>>> >> > - PeakMemoryUsage: 4.02 KB (4112)
>>>>> >> > - SendersBlockedTimer: 0.000ns
>>>>> >> > - SendersBlockedTotalTimer(*): 0.000ns
>>>>> >> > Averaged Fragment F00:(Total: 327.360ms, non-child: 1.578ms, %
>>>>> >> > non-child: 0.48%)
>>>>> >> > split sizes: min: 283.31 MB, max: 284.33 MB, avg: 284.04
>>>>> MB,
>>>>> >> > stddev:
>>>>> >> > 382.00 KB
>>>>> >> > completion times: min:299.457ms max:378.950ms mean:
>>>>> 328.881ms
>>>>> >> > stddev:31.507ms
>>>>> >> > execution rates: min:750.18 MB/sec max:949.01 MB/sec
>>>>> mean:871.27
>>>>> >> > MB/sec stddev:79.74 MB/sec
>>>>> >> > num instances: 5
>>>>> >> > - AverageThreadTokens: 15.00
>>>>> >> > - BloomFilterBytes: 0
>>>>> >> > - PeakMemoryUsage: 2.54 MB (2666852)
>>>>> >> > - PeakReservation: 0
>>>>> >> > - PeakUsedReservation: 0
>>>>> >> > - PerHostPeakMemUsage: 2.55 MB (2669309)
>>>>> >> > - RowsProduced: 1 (1)
>>>>> >> > - TotalNetworkReceiveTime: 0.000ns
>>>>> >> > - TotalNetworkSendTime: 303.541us
>>>>> >> > - TotalStorageWaitTime: 4s821ms
>>>>> >> > - TotalThreadsInvoluntaryContextSwitches: 104 (104)
>>>>> >> > - TotalThreadsTotalWallClockTime: 6s835ms
>>>>> >> > - TotalThreadsSysTime: 31.784ms
>>>>> >> > - TotalThreadsUserTime: 682.084ms
>>>>> >> > - TotalThreadsVoluntaryContextSwitches: 2.12K (2121)
>>>>> >> > Fragment Instance Lifecycle Timings:
>>>>> >> > - ExecTime: 500.816us
>>>>> >> > - ExecTreeExecTime: 8.200us
>>>>> >> > - OpenTime: 302.848ms
>>>>> >> > - ExecTreeOpenTime: 281.309ms
>>>>> >> > - PrepareTime: 23.946ms
>>>>> >> > - ExecTreePrepareTime: 2.513ms
>>>>> >> > DataStreamSender (dst_id=2):(Total: 269.902us, non-child:
>>>>> >> > 269.902us, %
>>>>> >> > non-child: 100.00%)
>>>>> >> > - BytesSent: 16.00 B (16)
>>>>> >> > - NetworkThroughput(*): 373.13 KB/sec
>>>>> >> > - OverallThroughput: 63.53 KB/sec
>>>>> >> > - PeakMemoryUsage: 16.00 KB (16384)
>>>>> >> > - RowsReturned: 1 (1)
>>>>> >> > - SerializeBatchTime: 14.525us
>>>>> >> > - TransmitDataRPCTime: 43.122us
>>>>> >> > - UncompressedRowBatchSize: 16.00 B (16)
>>>>> >> > CodeGen:(Total: 41.687ms, non-child: 41.687ms, % non-child:
>>>>> >> > 100.00%)
>>>>> >> > - CodegenTime: 1.194ms
>>>>> >> > - CompileTime: 7.360ms
>>>>> >> > - LoadTime: 0.000ns
>>>>> >> > - ModuleBitcodeSize: 1.86 MB (1950880)
>>>>> >> > - NumFunctions: 15 (15)
>>>>> >> > - NumInstructions: 214 (214)
>>>>> >> > - OptimizationTime: 13.529ms
>>>>> >> > - PeakMemoryUsage: 107.00 KB (109568)
>>>>> >> > - PrepareTime: 20.084ms
>>>>> >> > AGGREGATION_NODE (id=1):(Total: 283.825ms, non-child:
>>>>> 1.027ms, %
>>>>> >> > non-child: 0.36%)
>>>>> >> > - BuildTime: 1.259ms
>>>>> >> > - GetResultsTime: 0.000ns
>>>>> >> > - HTResizeTime: 0.000ns
>>>>> >> > - HashBuckets: 0 (0)
>>>>> >> > - LargestPartitionPercent: 0 (0)
>>>>> >> > - MaxPartitionLevel: 0 (0)
>>>>> >> > - NumRepartitions: 0 (0)
>>>>> >> > - PartitionsCreated: 0 (0)
>>>>> >> > - PeakMemoryUsage: 20.00 KB (20480)
>>>>> >> > - RowsRepartitioned: 0 (0)
>>>>> >> > - RowsReturned: 1 (1)
>>>>> >> > - RowsReturnedRate: 2.00 /sec
>>>>> >> > - SpilledPartitions: 0 (0)
>>>>> >> > HDFS_SCAN_NODE (id=0):(Total: 282.797ms, non-child:
>>>>> 282.797ms, %
>>>>> >> > non-child: 100.00%)
>>>>> >> > - AverageHdfsReadThreadConcurrency: 0.00
>>>>> >> > - AverageScannerThreadConcurrency: 14.40
>>>>> >> > - BytesRead: 44.52 MB (46686028)
>>>>> >> > - BytesReadDataNodeCache: 0
>>>>> >> > - BytesReadLocal: 44.52 MB (46686028)
>>>>> >> > - BytesReadRemoteUnexpected: 0
>>>>> >> > - BytesReadShortCircuit: 44.52 MB (46686028)
>>>>> >> > - CachedFileHandlesHitCount: 0 (0)
>>>>> >> > - CachedFileHandlesMissCount: 458 (458)
>>>>> >> > - CollectionItemsRead: 0 (0)
>>>>> >> > - DecompressionTime: 0.000ns
>>>>> >> > - MaxCompressedTextFileLength: 0
>>>>> >> > - NumColumns: 0 (0)
>>>>> >> > - NumDictFilteredRowGroups: 0 (0)
>>>>> >> > - NumDisksAccessed: 12 (12)
>>>>> >> > - NumRowGroups: 458 (458)
>>>>> >> > - NumScannerThreadsStarted: 24 (24)
>>>>> >> > - NumScannersWithNoReads: 0 (0)
>>>>> >> > - NumStatsFilteredRowGroups: 0 (0)
>>>>> >> > - PeakMemoryUsage: 2.50 MB (2625539)
>>>>> >> > - PerReadThreadRawHdfsThroughput: 1.29 GB/sec
>>>>> >> > - RemoteScanRanges: 0 (0)
>>>>> >> > - RowBatchQueueGetWaitTime: 269.787ms
>>>>> >> > - RowBatchQueuePutWaitTime: 0.000ns
>>>>> >> > - RowsRead: 0 (0)
>>>>> >> > - RowsReturned: 458 (458)
>>>>> >> > - RowsReturnedRate: 1.64 K/sec
>>>>> >> > - ScanRangesComplete: 458 (458)
>>>>> >> > - ScannerThreadsInvoluntaryContextSwitches: 99 (99)
>>>>> >> > - ScannerThreadsTotalWallClockTime: 6s532ms
>>>>> >> > - MaterializeTupleTime(*): 70.664us
>>>>> >> > - ScannerThreadsSysTime: 26.184ms
>>>>> >> > - ScannerThreadsUserTime: 650.089ms
>>>>> >> > - ScannerThreadsVoluntaryContextSwitches: 1.67K (1674)
>>>>> >> > - TotalRawHdfsReadTime(*): 34.416ms
>>>>> >> > - TotalReadThroughput: 36.86 MB/sec
>>>>> >> > Fragment F00:
>>>>> >> > Instance c843dae48ce9b289:6ad3ffc000000004
>>>>> >> > (host=aopr-dhc004.lpdomain.com:22000):(Total: 377.384ms,
>>>>> non-child:
>>>>> >> > 1.619ms,
>>>>> >> > % non-child: 0.43%)
>>>>> >> > Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>>>>> >> > 0:36/21.95 MB 11:33
>>>>> <https://maps.google.com/?q=MB+11:33&entry=gmail&source=g>/19.87 MB
>>>>> 9:27 <https://maps.google.com/?q=MB+9:27&entry=gmail&source=g>/15.16 MB
>>>>> 5:44 <https://maps.google.com/?q=MB+5:44&entry=gmail&source=g>/28.07 MB
>>>>> 6:56 <https://maps.google.com/?q=MB+6:56&entry=gmail&source=g>/33.32
>>>>> MB
>>>>> >> > 1:38/22.24 MB 10:44
>>>>> <https://maps.google.com/?q=MB+10:44&entry=gmail&source=g>/27.90 MB
>>>>> 3:34 <https://maps.google.com/?q=MB+3:34&entry=gmail&source=g>/20.26 MB
>>>>> 2:45 <https://maps.google.com/?q=MB+2:45&entry=gmail&source=g>/28.21 MB
>>>>> 7:31 <https://maps.google.com/?q=MB+7:31&entry=gmail&source=g>/19.33
>>>>> MB
>>>>> >> > 4:42/26.73 MB 8:34
>>>>> <https://maps.google.com/?q=MB+8:34&entry=gmail&source=g>/21.25 MB
>>>>> >> > MemoryUsage(500.000ms): 1.91 MB
>>>>> >> > ThreadUsage(500.000ms): 25
>>>>> >> > - AverageThreadTokens: 25.00
>>>>> >> > - BloomFilterBytes: 0
>>>>> >> > - PeakMemoryUsage: 2.41 MB (2531689)
>>>>> >> > - PeakReservation: 0
>>>>> >> > - PeakUsedReservation: 0
>>>>> >> > - PerHostPeakMemUsage: 2.41 MB (2531689)
>>>>> >> > - RowsProduced: 1 (1)
>>>>> >> > - TotalNetworkReceiveTime: 0.000ns
>>>>> >> > - TotalNetworkSendTime: 382.629us
>>>>> >> > - TotalStorageWaitTime: 5s790ms
>>>>> >> > - TotalThreadsInvoluntaryContextSwitches: 115 (115)
>>>>> >> > - TotalThreadsTotalWallClockTime: 8s122ms
>>>>> >> > - TotalThreadsSysTime: 33.984ms
>>>>> >> > - TotalThreadsUserTime: 702.879ms
>>>>> >> > - TotalThreadsVoluntaryContextSwitches: 2.04K (2037)
>>>>> >> > Fragment Instance Lifecycle Timings:
>>>>> >> > - ExecTime: 604.005us
>>>>> >> > - ExecTreeExecTime: 8.338us
>>>>> >> > - OpenTime: 352.927ms
>>>>> >> > - ExecTreeOpenTime: 331.921ms
>>>>> >> > - PrepareTime: 23.783ms
>>>>> >> > - ExecTreePrepareTime: 2.559ms
>>>>> >> > DataStreamSender (dst_id=2):(Total: 331.818us, non-child:
>>>>> >> > 331.818us,
>>>>> >> > % non-child: 100.00%)
>>>>> >> > - BytesSent: 16.00 B (16)
>>>>> >> > - NetworkThroughput(*): 289.00 KB/sec
>>>>> >> > - OverallThroughput: 47.09 KB/sec
>>>>> >> > - PeakMemoryUsage: 16.00 KB (16384)
>>>>> >> > - RowsReturned: 1 (1)
>>>>> >> > - SerializeBatchTime: 14.466us
>>>>> >> > - TransmitDataRPCTime: 54.066us
>>>>> >> > - UncompressedRowBatchSize: 16.00 B (16)
>>>>> >> > CodeGen:(Total: 40.950ms, non-child: 40.950ms, %
>>>>> non-child:
>>>>> >> > 100.00%)
>>>>> >> > - CodegenTime: 1.122ms
>>>>> >> > - CompileTime: 7.318ms
>>>>> >> > - LoadTime: 0.000ns
>>>>> >> > - ModuleBitcodeSize: 1.86 MB (1950880)
>>>>> >> > - NumFunctions: 15 (15)
>>>>> >> > - NumInstructions: 214 (214)
>>>>> >> > - OptimizationTime: 13.106ms
>>>>> >> > - PeakMemoryUsage: 107.00 KB (109568)
>>>>> >> > - PrepareTime: 19.773ms
>>>>> >> > AGGREGATION_NODE (id=1):(Total: 334.482ms, non-child:
>>>>> 766.527us,
>>>>> >> > %
>>>>> >> > non-child: 0.23%)
>>>>> >> > ExecOption: Codegen Enabled
>>>>> >> > - BuildTime: 1.192ms
>>>>> >> > - GetResultsTime: 0.000ns
>>>>> >> > - HTResizeTime: 0.000ns
>>>>> >> > - HashBuckets: 0 (0)
>>>>> >> > - LargestPartitionPercent: 0 (0)
>>>>> >> > - MaxPartitionLevel: 0 (0)
>>>>> >> > - NumRepartitions: 0 (0)
>>>>> >> > - PartitionsCreated: 0 (0)
>>>>> >> > - PeakMemoryUsage: 20.00 KB (20480)
>>>>> >> > - RowsRepartitioned: 0 (0)
>>>>> >> > - RowsReturned: 1 (1)
>>>>> >> > - RowsReturnedRate: 2.00 /sec
>>>>> >> > - SpilledPartitions: 0 (0)
>>>>> >> > HDFS_SCAN_NODE (id=0):(Total: 333.715ms, non-child:
>>>>> 333.715ms, %
>>>>> >> > non-child: 100.00%)
>>>>> >> > Hdfs split stats (<volume id>:<# splits>/<split
>>>>> lengths>):
>>>>> >> > 0:36/21.95 MB 11:33
>>>>> <https://maps.google.com/?q=MB+11:33&entry=gmail&source=g>/19.87 MB
>>>>> 9:27 <https://maps.google.com/?q=MB+9:27&entry=gmail&source=g>/15.16 MB
>>>>> 5:44 <https://maps.google.com/?q=MB+5:44&entry=gmail&source=g>/28.07 MB
>>>>> 6:56 <https://maps.google.com/?q=MB+6:56&entry=gmail&source=g>/33.32
>>>>> MB
>>>>> >> > 1:38/22.24 MB 10:44
>>>>> <https://maps.google.com/?q=MB+10:44&entry=gmail&source=g>/27.90 MB
>>>>> 3:34 <https://maps.google.com/?q=MB+3:34&entry=gmail&source=g>/20.26 MB
>>>>> 2:45 <https://maps.google.com/?q=MB+2:45&entry=gmail&source=g>/28.21 MB
>>>>> 7:31 <https://maps.google.com/?q=MB+7:31&entry=gmail&source=g>/19.33
>>>>> MB
>>>>> >> > 4:42/26.73 MB 8:34
>>>>> <https://maps.google.com/?q=MB+8:34&entry=gmail&source=g>/21.25 MB
>>>>> >> > ExecOption: PARQUET Codegen Enabled, Codegen enabled:
>>>>> 464 out
>>>>> >> > of
>>>>> >> > 464
>>>>> >> > Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0%
>>>>> 3:0%
>>>>> >> > 4:0%
>>>>> >> > 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
>>>>> >> > File Formats: PARQUET/Unknown(Skipped):464
>>>>> >> > BytesRead(500.000ms): 12.27 MB
>>>>> >> > - FooterProcessingTime: (Avg: 13.555ms ; Min: 5.433ms
>>>>> ; Max:
>>>>> >> > 62.957ms ; Number of samples: 464)
>>>>> >> > - AverageHdfsReadThreadConcurrency: 0.00
>>>>> >> > - AverageScannerThreadConcurrency: 24.00
>>>>> >> > - BytesRead: 45.01 MB (47193988)
>>>>> >> > - BytesReadDataNodeCache: 0
>>>>> >> > - BytesReadLocal: 45.01 MB (47193988)
>>>>> >> > - BytesReadRemoteUnexpected: 0
>>>>> >> > - BytesReadShortCircuit: 45.01 MB (47193988)
>>>>> >> > - CachedFileHandlesHitCount: 0 (0)
>>>>> >> > - CachedFileHandlesMissCount: 464 (464)
>>>>> >> > - CollectionItemsRead: 0 (0)
>>>>> >> > - DecompressionTime: 0.000ns
>>>>> >> > - MaxCompressedTextFileLength: 0
>>>>> >> > - NumColumns: 0 (0)
>>>>> >> > - NumDictFilteredRowGroups: 0 (0)
>>>>> >> > - NumDisksAccessed: 12 (12)
>>>>> >> > - NumRowGroups: 464 (464)
>>>>> >> > - NumScannerThreadsStarted: 24 (24)
>>>>> >> > - NumScannersWithNoReads: 0 (0)
>>>>> >> > - NumStatsFilteredRowGroups: 0 (0)
>>>>> >> > - PeakMemoryUsage: 2.38 MB (2490376)
>>>>> >> > - PerReadThreadRawHdfsThroughput: 1016.44 MB/sec
>>>>> >> > - RemoteScanRanges: 0 (0)
>>>>> >> > - RowBatchQueueGetWaitTime: 320.860ms
>>>>> >> > - RowBatchQueuePutWaitTime: 0.000ns
>>>>> >> > - RowsRead: 0 (0)
>>>>> >> > - RowsReturned: 464 (464)
>>>>> >> > - RowsReturnedRate: 1.39 K/sec
>>>>> >> > - ScanRangesComplete: 464 (464)
>>>>> >> > - ScannerThreadsInvoluntaryContextSwitches: 108 (108)
>>>>> >> > - ScannerThreadsTotalWallClockTime: 7s769ms
>>>>> >> > - MaterializeTupleTime(*): 75.514us
>>>>> >> > - ScannerThreadsSysTime: 29.984ms
>>>>> >> > - ScannerThreadsUserTime: 669.884ms
>>>>> >> > - ScannerThreadsVoluntaryContextSwitches: 1.60K (1597)
>>>>> >> > - TotalRawHdfsReadTime(*): 44.279ms
>>>>> >> > - TotalReadThroughput: 24.53 MB/sec
>>>>> >> > Instance c843dae48ce9b289:6ad3ffc000000002
>>>>> >> > (host=aopr-dhc003.lpdomain.com:22000):(Total: 351.014ms,
>>>>> non-child:
>>>>> >> > 1.641ms,
>>>>> >> > % non-child: 0.47%)
>>>>> >> > Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>>>>> >> > 0:35/20.89 MB 7:44
>>>>> <https://maps.google.com/?q=MB+7:44&entry=gmail&source=g>/26.64 MB
>>>>> 11:27 <https://maps.google.com/?q=MB+11:27&entry=gmail&source=g>/17.21
>>>>> MB 9:32 <https://maps.google.com/?q=MB+9:32&entry=gmail&source=g>/18.61
>>>>> MB 5:25 <https://maps.google.com/?q=MB+5:25&entry=gmail&source=g>/14.56
>>>>> MB
>>>>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+2:38&entry=gmail&source=g>
>>>>> >> > 2:38
>>>>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+2:38&entry=gmail&source=g>/24.32
>>>>> MB 6:31 <https://maps.google.com/?q=MB+6:31&entry=gmail&source=g>/19.85
>>>>> MB 8:50 <https://maps.google.com/?q=MB+8:50&entry=gmail&source=g>/30.76
>>>>> MB 10:41 <https://maps.google.com/?q=MB+10:41&entry=gmail&source=g>/25.92
>>>>> MB 3:31 <https://maps.google.com/?q=MB+3:31&entry=gmail&source=g>/19.78
>>>>> MB
>>>>> >> > 4:60/37.59 MB 1:44/27.19 MB
>>>>> >> > MemoryUsage(500.000ms): 1.16 MB
>>>>> >> > ThreadUsage(500.000ms): 25
>>>>> >> > - AverageThreadTokens: 25.00
>>>>> >> > - BloomFilterBytes: 0
>>>>> >> > - PeakMemoryUsage: 2.54 MB (2662753)
>>>>> >> > - PeakReservation: 0
>>>>> >> > - PeakUsedReservation: 0
>>>>> >> > - PerHostPeakMemUsage: 2.54 MB (2662753)
>>>>> >> > - RowsProduced: 1 (1)
>>>>> >> > - TotalNetworkReceiveTime: 0.000ns
>>>>> >> > - TotalNetworkSendTime: 342.746us
>>>>> >> > - TotalStorageWaitTime: 4s796ms
>>>>> >> > - TotalThreadsInvoluntaryContextSwitches: 93 (93)
>>>>> >> > - TotalThreadsTotalWallClockTime: 7s286ms
>>>>> >> > - TotalThreadsSysTime: 27.988ms
>>>>> >> > - TotalThreadsUserTime: 671.887ms
>>>>> >> > - TotalThreadsVoluntaryContextSwitches: 2.08K (2079)
>>>>> >> > Fragment Instance Lifecycle Timings:
>>>>> >> > - ExecTime: 524.006us
>>>>> >> > - ExecTreeExecTime: 7.920us
>>>>> >> > - OpenTime: 327.802ms
>>>>> >> > - ExecTreeOpenTime: 307.154ms
>>>>> >> > - PrepareTime: 22.617ms
>>>>> >> > - ExecTreePrepareTime: 2.650ms
>>>>> >> > DataStreamSender (dst_id=2):(Total: 348.990us, non-child:
>>>>> >> > 348.990us,
>>>>> >> > % non-child: 100.00%)
>>>>> >> > - BytesSent: 16.00 B (16)
>>>>> >> > - NetworkThroughput(*): 354.47 KB/sec
>>>>> >> > - OverallThroughput: 44.77 KB/sec
>>>>> >> > - PeakMemoryUsage: 16.00 KB (16384)
>>>>> >> > - RowsReturned: 1 (1)
>>>>> >> > - SerializeBatchTime: 15.602us
>>>>> >> > - TransmitDataRPCTime: 44.080us
>>>>> >> > - UncompressedRowBatchSize: 16.00 B (16)
>>>>> >> > CodeGen:(Total: 39.215ms, non-child: 39.215ms, %
>>>>> non-child:
>>>>> >> > 100.00%)
>>>>> >> > - CodegenTime: 1.183ms
>>>>> >> > - CompileTime: 7.168ms
>>>>> >> > - LoadTime: 0.000ns
>>>>> >> > - ModuleBitcodeSize: 1.86 MB (1950880)
>>>>> >> > - NumFunctions: 15 (15)
>>>>> >> > - NumInstructions: 214 (214)
>>>>> >> > - OptimizationTime: 12.929ms
>>>>> >> > - PeakMemoryUsage: 107.00 KB (109568)
>>>>> >> > - PrepareTime: 18.453ms
>>>>> >> > AGGREGATION_NODE (id=1):(Total: 309.807ms, non-child:
>>>>> 948.677us,
>>>>> >> > %
>>>>> >> > non-child: 0.31%)
>>>>> >> > ExecOption: Codegen Enabled
>>>>> >> > - BuildTime: 1.552ms
>>>>> >> > - GetResultsTime: 0.000ns
>>>>> >> > - HTResizeTime: 0.000ns
>>>>> >> > - HashBuckets: 0 (0)
>>>>> >> > - LargestPartitionPercent: 0 (0)
>>>>> >> > - MaxPartitionLevel: 0 (0)
>>>>> >> > - NumRepartitions: 0 (0)
>>>>> >> > - PartitionsCreated: 0 (0)
>>>>> >> > - PeakMemoryUsage: 20.00 KB (20480)
>>>>> >> > - RowsRepartitioned: 0 (0)
>>>>> >> > - RowsReturned: 1 (1)
>>>>> >> > - RowsReturnedRate: 3.00 /sec
>>>>> >> > - SpilledPartitions: 0 (0)
>>>>> >> > HDFS_SCAN_NODE (id=0):(Total: 308.859ms, non-child:
>>>>> 308.859ms, %
>>>>> >> > non-child: 100.00%)
>>>>> >> > Hdfs split stats (<volume id>:<# splits>/<split
>>>>> lengths>):
>>>>> >> > 0:35/20.89 MB 7:44
>>>>> <https://maps.google.com/?q=MB+7:44&entry=gmail&source=g>/26.64 MB
>>>>> 11:27 <https://maps.google.com/?q=MB+11:27&entry=gmail&source=g>/17.21
>>>>> MB 9:32 <https://maps.google.com/?q=MB+9:32&entry=gmail&source=g>/18.61
>>>>> MB 5:25 <https://maps.google.com/?q=MB+5:25&entry=gmail&source=g>/14.56
>>>>> MB
>>>>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+2:38&entry=gmail&source=g>
>>>>> >> > 2:38
>>>>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+2:38&entry=gmail&source=g>/24.32
>>>>> MB 6:31 <https://maps.google.com/?q=MB+6:31&entry=gmail&source=g>/19.85
>>>>> MB 8:50 <https://maps.google.com/?q=MB+8:50&entry=gmail&source=g>/30.76
>>>>> MB 10:41 <https://maps.google.com/?q=MB+10:41&entry=gmail&source=g>/25.92
>>>>> MB 3:31 <https://maps.google.com/?q=MB+3:31&entry=gmail&source=g>/19.78
>>>>> MB
>>>>> >> > 4:60/37.59 MB 1:44/27.19 MB
>>>>> >> > ExecOption: PARQUET Codegen Enabled, Codegen enabled:
>>>>> 458 out
>>>>> >> > of
>>>>> >> > 458
>>>>> >> > Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0%
>>>>> 3:0%
>>>>> >> > 4:0%
>>>>> >> > 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
>>>>> >> > File Formats: PARQUET/Unknown(Skipped):458
>>>>> >> > BytesRead(500.000ms): 36.74 MB
>>>>> >> > - FooterProcessingTime: (Avg: 11.502ms ; Min: 3.668ms
>>>>> ; Max:
>>>>> >> > 23.564ms ; Number of samples: 458)
>>>>> >> > - AverageHdfsReadThreadConcurrency: 0.00
>>>>> >> > - AverageScannerThreadConcurrency: 24.00
>>>>> >> > - BytesRead: 44.51 MB (46668432)
>>>>> >> > - BytesReadDataNodeCache: 0
>>>>> >> > - BytesReadLocal: 44.51 MB (46668432)
>>>>> >> > - BytesReadRemoteUnexpected: 0
>>>>> >> > - BytesReadShortCircuit: 44.51 MB (46668432)
>>>>> >> > - CachedFileHandlesHitCount: 0 (0)
>>>>> >> > - CachedFileHandlesMissCount: 458 (458)
>>>>> >> > - CollectionItemsRead: 0 (0)
>>>>> >> > - DecompressionTime: 0.000ns
>>>>> >> > - MaxCompressedTextFileLength: 0
>>>>> >> > - NumColumns: 0 (0)
>>>>> >> > - NumDictFilteredRowGroups: 0 (0)
>>>>> >> > - NumDisksAccessed: 12 (12)
>>>>> >> > - NumRowGroups: 458 (458)
>>>>> >> > - NumScannerThreadsStarted: 24 (24)
>>>>> >> > - NumScannersWithNoReads: 0 (0)
>>>>> >> > - NumStatsFilteredRowGroups: 0 (0)
>>>>> >> > - PeakMemoryUsage: 2.50 MB (2621440)
>>>>> >> > - PerReadThreadRawHdfsThroughput: 1.26 GB/sec
>>>>> >> > - RemoteScanRanges: 0 (0)
>>>>> >> > - RowBatchQueueGetWaitTime: 295.595ms
>>>>> >> > - RowBatchQueuePutWaitTime: 0.000ns
>>>>> >> > - RowsRead: 0 (0)
>>>>> >> > - RowsReturned: 458 (458)
>>>>> >> > - RowsReturnedRate: 1.48 K/sec
>>>>> >> > - ScanRangesComplete: 458 (458)
>>>>> >> > - ScannerThreadsInvoluntaryContextSwitches: 91 (91)
>>>>> >> > - ScannerThreadsTotalWallClockTime: 6s958ms
>>>>> >> > - MaterializeTupleTime(*): 71.630us
>>>>> >> > - ScannerThreadsSysTime: 16.989ms
>>>>> >> > - ScannerThreadsUserTime: 645.891ms
>>>>> >> > - ScannerThreadsVoluntaryContextSwitches: 1.63K (1630)
>>>>> >> > - TotalRawHdfsReadTime(*): 34.377ms
>>>>> >> > - TotalReadThroughput: 73.49 MB/sec
>>>>> >> > Instance c843dae48ce9b289:6ad3ffc000000001
>>>>> >> > (host=aopr-dhc005.lpdomain.com:22000):(Total: 308.195ms,
>>>>> non-child:
>>>>> >> > 1.784ms,
>>>>> >> > % non-child: 0.58%)
>>>>> >> > Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>>>>> >> > 2:46/28.97 MB 8:39
>>>>> <https://maps.google.com/?q=MB+8:39&entry=gmail&source=g>/24.34 MB
>>>>> 10:33 <https://maps.google.com/?q=MB+10:33&entry=gmail&source=g>/21.03
>>>>> MB 3:43 <https://maps.google.com/?q=MB+3:43&entry=gmail&source=g>/26.50
>>>>> MB 5:32 <https://maps.google.com/?q=MB+5:32&entry=gmail&source=g>/20.27
>>>>> MB
>>>>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+9:32&entry=gmail&source=g>
>>>>> >> > 9:32
>>>>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+9:32&entry=gmail&source=g>/19.86
>>>>> MB 6:25 <https://maps.google.com/?q=MB+6:25&entry=gmail&source=g>/16.01
>>>>> MB 7:41 <https://maps.google.com/?q=MB+7:41&entry=gmail&source=g>/24.77
>>>>> MB 1:42/25.01 MB 0:55/32.51 MB
>>>>> >> > 4:40/24.29 MB 11:33
>>>>> <https://maps.google.com/?q=MB+11:33&entry=gmail&source=g>/20.77 MB
>>>>> >> > - AverageThreadTokens: 0.00
>>>>> >> > - BloomFilterBytes: 0
>>>>> >> > - PeakMemoryUsage: 2.56 MB (2683241)
>>>>> >> > - PeakReservation: 0
>>>>> >> > - PeakUsedReservation: 0
>>>>> >> > - PerHostPeakMemUsage: 2.56 MB (2683241)
>>>>> >> > - RowsProduced: 1 (1)
>>>>> >> > - TotalNetworkReceiveTime: 0.000ns
>>>>> >> > - TotalNetworkSendTime: 292.765us
>>>>> >> > - TotalStorageWaitTime: 4s482ms
>>>>> >> > - TotalThreadsInvoluntaryContextSwitches: 113 (113)
>>>>> >> > - TotalThreadsTotalWallClockTime: 6s310ms
>>>>> >> > - TotalThreadsSysTime: 35.983ms
>>>>> >> > - TotalThreadsUserTime: 682.884ms
>>>>> >> > - TotalThreadsVoluntaryContextSwitches: 2.14K (2143)
>>>>> >> > Fragment Instance Lifecycle Timings:
>>>>> >> > - ExecTime: 532.572us
>>>>> >> > - ExecTreeExecTime: 8.377us
>>>>> >> > - OpenTime: 280.241ms
>>>>> >> > - ExecTreeOpenTime: 258.652ms
>>>>> >> > - PrepareTime: 27.342ms
>>>>> >> > - ExecTreePrepareTime: 2.863ms
>>>>> >> > DataStreamSender (dst_id=2):(Total: 309.794us, non-child:
>>>>> >> > 309.794us,
>>>>> >> > % non-child: 100.00%)
>>>>> >> > - BytesSent: 16.00 B (16)
>>>>> >> > - NetworkThroughput(*): 356.91 KB/sec
>>>>> >> > - OverallThroughput: 50.44 KB/sec
>>>>> >> > - PeakMemoryUsage: 16.00 KB (16384)
>>>>> >> > - RowsReturned: 1 (1)
>>>>> >> > - SerializeBatchTime: 14.262us
>>>>> >> > - TransmitDataRPCTime: 43.778us
>>>>> >> > - UncompressedRowBatchSize: 16.00 B (16)
>>>>> >> > CodeGen:(Total: 44.584ms, non-child: 44.584ms, %
>>>>> non-child:
>>>>> >> > 100.00%)
>>>>> >> > - CodegenTime: 1.382ms
>>>>> >> > - CompileTime: 7.346ms
>>>>> >> > - LoadTime: 0.000ns
>>>>> >> > - ModuleBitcodeSize: 1.86 MB (1950880)
>>>>> >> > - NumFunctions: 15 (15)
>>>>> >> > - NumInstructions: 214 (214)
>>>>> >> > - OptimizationTime: 13.462ms
>>>>> >> > - PeakMemoryUsage: 107.00 KB (109568)
>>>>> >> > - PrepareTime: 23.006ms
>>>>> >> > AGGREGATION_NODE (id=1):(Total: 261.516ms, non-child:
>>>>> 836.656us,
>>>>> >> > %
>>>>> >> > non-child: 0.32%)
>>>>> >> > ExecOption: Codegen Enabled
>>>>> >> > - BuildTime: 1.280ms
>>>>> >> > - GetResultsTime: 0.000ns
>>>>> >> > - HTResizeTime: 0.000ns
>>>>> >> > - HashBuckets: 0 (0)
>>>>> >> > - LargestPartitionPercent: 0 (0)
>>>>> >> > - MaxPartitionLevel: 0 (0)
>>>>> >> > - NumRepartitions: 0 (0)
>>>>> >> > - PartitionsCreated: 0 (0)
>>>>> >> > - PeakMemoryUsage: 20.00 KB (20480)
>>>>> >> > - RowsRepartitioned: 0 (0)
>>>>> >> > - RowsReturned: 1 (1)
>>>>> >> > - RowsReturnedRate: 3.00 /sec
>>>>> >> > - SpilledPartitions: 0 (0)
>>>>> >> > HDFS_SCAN_NODE (id=0):(Total: 260.680ms, non-child:
>>>>> 260.680ms, %
>>>>> >> > non-child: 100.00%)
>>>>> >> > Hdfs split stats (<volume id>:<# splits>/<split
>>>>> lengths>):
>>>>> >> > 2:46/28.97 MB 8:39
>>>>> <https://maps.google.com/?q=MB+8:39&entry=gmail&source=g>/24.34 MB
>>>>> 10:33 <https://maps.google.com/?q=MB+10:33&entry=gmail&source=g>/21.03
>>>>> MB 3:43 <https://maps.google.com/?q=MB+3:43&entry=gmail&source=g>/26.50
>>>>> MB 5:32 <https://maps.google.com/?q=MB+5:32&entry=gmail&source=g>/20.27
>>>>> MB
>>>>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+9:32&entry=gmail&source=g>
>>>>> >> > 9:32
>>>>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+9:32&entry=gmail&source=g>/19.86
>>>>> MB 6:25 <https://maps.google.com/?q=MB+6:25&entry=gmail&source=g>/16.01
>>>>> MB 7:41 <https://maps.google.com/?q=MB+7:41&entry=gmail&source=g>/24.77
>>>>> MB 1:42/25.01 MB 0:55/32.51 MB
>>>>> >> > 4:40/24.29 MB 11:33
>>>>> <https://maps.google.com/?q=MB+11:33&entry=gmail&source=g>/20.77 MB
>>>>> >> > ExecOption: PARQUET Codegen Enabled, Codegen enabled:
>>>>> 461 out
>>>>> >> > of
>>>>> >> > 461
>>>>> >> > Hdfs Read Thread Concurrency Bucket: 0:0% 1:0% 2:0%
>>>>> 3:0% 4:0%
>>>>> >> > 5:0%
>>>>> >> > 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
>>>>> >> > File Formats: PARQUET/Unknown(Skipped):461
>>>>> >> > - FooterProcessingTime: (Avg: 10.793ms ; Min: 3.087ms
>>>>> ; Max:
>>>>> >> > 20.253ms ; Number of samples: 461)
>>>>> >> > - AverageHdfsReadThreadConcurrency: 0.00
>>>>> >> > - AverageScannerThreadConcurrency: 0.00
>>>>> >> > - BytesRead: 44.73 MB (46906407)
>>>>> >> > - BytesReadDataNodeCache: 0
>>>>> >> > - BytesReadLocal: 44.73 MB (46906407)
>>>>> >> > - BytesReadRemoteUnexpected: 0
>>>>> >> > - BytesReadShortCircuit: 44.73 MB (46906407)
>>>>> >> > - CachedFileHandlesHitCount: 0 (0)
>>>>> >> > - CachedFileHandlesMissCount: 461 (461)
>>>>> >> > - CollectionItemsRead: 0 (0)
>>>>> >> > - DecompressionTime: 0.000ns
>>>>> >> > - MaxCompressedTextFileLength: 0
>>>>> >> > - NumColumns: 0 (0)
>>>>> >> > - NumDictFilteredRowGroups: 0 (0)
>>>>> >> > - NumDisksAccessed: 12 (12)
>>>>> >> > - NumRowGroups: 461 (461)
>>>>> >> > - NumScannerThreadsStarted: 24 (24)
>>>>> >> > - NumScannersWithNoReads: 0 (0)
>>>>> >> > - NumStatsFilteredRowGroups: 0 (0)
>>>>> >> > - PeakMemoryUsage: 2.52 MB (2641928)
>>>>> >> > - PerReadThreadRawHdfsThroughput: 1.31 GB/sec
>>>>> >> > - RemoteScanRanges: 0 (0)
>>>>> >> > - RowBatchQueueGetWaitTime: 246.691ms
>>>>> >> > - RowBatchQueuePutWaitTime: 0.000ns
>>>>> >> > - RowsRead: 0 (0)
>>>>> >> > - RowsReturned: 461 (461)
>>>>> >> > - RowsReturnedRate: 1.77 K/sec
>>>>> >> > - ScanRangesComplete: 461 (461)
>>>>> >> > - ScannerThreadsInvoluntaryContextSwitches: 109 (109)
>>>>> >> > - ScannerThreadsTotalWallClockTime: 6s030ms
>>>>> >> > - MaterializeTupleTime(*): 68.735us
>>>>> >> > - ScannerThreadsSysTime: 33.983ms
>>>>> >> > - ScannerThreadsUserTime: 647.889ms
>>>>> >> > - ScannerThreadsVoluntaryContextSwitches: 1.70K (1696)
>>>>> >> > - TotalRawHdfsReadTime(*): 33.405ms
>>>>> >> > - TotalReadThroughput: 0.00 /sec
>>>>> >> > Instance c843dae48ce9b289:6ad3ffc000000003
>>>>> >> > (host=aopr-dhc002.lpdomain.com:22000):(Total: 302.116ms,
>>>>> non-child:
>>>>> >> > 1.604ms,
>>>>> >> > % non-child: 0.53%)
>>>>> >> > Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>>>>> >> > 0:27/16.21 MB 11:33
>>>>> <https://maps.google.com/?q=MB+11:33&entry=gmail&source=g>/21.23 MB
>>>>> 6:36 <https://maps.google.com/?q=MB+6:36&entry=gmail&source=g>/23.15 MB
>>>>> 10:41 <https://maps.google.com/?q=MB+10:41&entry=gmail&source=g>/25.09
>>>>> MB 1:41/24.13 MB
>>>>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+9:32&entry=gmail&source=g>
>>>>> >> > 9:32
>>>>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+9:32&entry=gmail&source=g>/18.79
>>>>> MB 5:41 <https://maps.google.com/?q=MB+5:41&entry=gmail&source=g>/25.31
>>>>> MB 2:42 <https://maps.google.com/?q=MB+2:42&entry=gmail&source=g>/26.49
>>>>> MB 7:36 <https://maps.google.com/?q=MB+7:36&entry=gmail&source=g>/22.87
>>>>> MB 3:51 <https://maps.google.com/?q=MB+3:51&entry=gmail&source=g>/31.00
>>>>> MB
>>>>> >> > 4:41/25.99 MB 8:38
>>>>> <https://maps.google.com/?q=MB+8:38&entry=gmail&source=g>/23.79 MB
>>>>> >> > MemoryUsage(500.000ms): 700.35 KB
>>>>> >> > ThreadUsage(500.000ms): 25
>>>>> >> > - AverageThreadTokens: 25.00
>>>>> >> > - BloomFilterBytes: 0
>>>>> >> > - PeakMemoryUsage: 2.54 MB (2662753)
>>>>> >> > - PeakReservation: 0
>>>>> >> > - PeakUsedReservation: 0
>>>>> >> > - PerHostPeakMemUsage: 2.54 MB (2662753)
>>>>> >> > - RowsProduced: 1 (1)
>>>>> >> > - TotalNetworkReceiveTime: 0.000ns
>>>>> >> > - TotalNetworkSendTime: 300.675us
>>>>> >> > - TotalStorageWaitTime: 4s661ms
>>>>> >> > - TotalThreadsInvoluntaryContextSwitches: 90 (90)
>>>>> >> > - TotalThreadsTotalWallClockTime: 6s256ms
>>>>> >> > - TotalThreadsSysTime: 30.983ms
>>>>> >> > - TotalThreadsUserTime: 709.880ms
>>>>> >> > - TotalThreadsVoluntaryContextSwitches: 2.17K (2175)
>>>>> >> > Fragment Instance Lifecycle Timings:
>>>>> >> > - ExecTime: 492.627us
>>>>> >> > - ExecTreeExecTime: 8.191us
>>>>> >> > - OpenTime: 277.922ms
>>>>> >> > - ExecTreeOpenTime: 255.395ms
>>>>> >> > - PrepareTime: 23.647ms
>>>>> >> > - ExecTreePrepareTime: 2.243ms
>>>>> >> > DataStreamSender (dst_id=2):(Total: 194.517us, non-child:
>>>>> >> > 194.517us,
>>>>> >> > % non-child: 100.00%)
>>>>> >> > - BytesSent: 16.00 B (16)
>>>>> >> > - NetworkThroughput(*): 371.86 KB/sec
>>>>> >> > - OverallThroughput: 80.33 KB/sec
>>>>> >> > - PeakMemoryUsage: 16.00 KB (16384)
>>>>> >> > - RowsReturned: 1 (1)
>>>>> >> > - SerializeBatchTime: 14.035us
>>>>> >> > - TransmitDataRPCTime: 42.019us
>>>>> >> > - UncompressedRowBatchSize: 16.00 B (16)
>>>>> >> > CodeGen:(Total: 42.676ms, non-child: 42.676ms, %
>>>>> non-child:
>>>>> >> > 100.00%)
>>>>> >> > - CodegenTime: 1.236ms
>>>>> >> > - CompileTime: 7.717ms
>>>>> >> > - LoadTime: 0.000ns
>>>>> >> > - ModuleBitcodeSize: 1.86 MB (1950880)
>>>>> >> > - NumFunctions: 15 (15)
>>>>> >> > - NumInstructions: 214 (214)
>>>>> >> > - OptimizationTime: 14.130ms
>>>>> >> > - PeakMemoryUsage: 107.00 KB (109568)
>>>>> >> > - PrepareTime: 20.109ms
>>>>> >> > AGGREGATION_NODE (id=1):(Total: 257.640ms, non-child:
>>>>> 1.541ms, %
>>>>> >> > non-child: 0.60%)
>>>>> >> > ExecOption: Codegen Enabled
>>>>> >> > - BuildTime: 1.252ms
>>>>> >> > - GetResultsTime: 0.000ns
>>>>> >> > - HTResizeTime: 0.000ns
>>>>> >> > - HashBuckets: 0 (0)
>>>>> >> > - LargestPartitionPercent: 0 (0)
>>>>> >> > - MaxPartitionLevel: 0 (0)
>>>>> >> > - NumRepartitions: 0 (0)
>>>>> >> > - PartitionsCreated: 0 (0)
>>>>> >> > - PeakMemoryUsage: 20.00 KB (20480)
>>>>> >> > - RowsRepartitioned: 0 (0)
>>>>> >> > - RowsReturned: 1 (1)
>>>>> >> > - RowsReturnedRate: 3.00 /sec
>>>>> >> > - SpilledPartitions: 0 (0)
>>>>> >> > HDFS_SCAN_NODE (id=0):(Total: 256.099ms, non-child:
>>>>> 256.099ms, %
>>>>> >> > non-child: 100.00%)
>>>>> >> > Hdfs split stats (<volume id>:<# splits>/<split
>>>>> lengths>):
>>>>> >> > 0:27/16.21 MB 11:33
>>>>> <https://maps.google.com/?q=MB+11:33&entry=gmail&source=g>/21.23 MB
>>>>> 6:36 <https://maps.google.com/?q=MB+6:36&entry=gmail&source=g>/23.15 MB
>>>>> 10:41 <https://maps.google.com/?q=MB+10:41&entry=gmail&source=g>/25.09
>>>>> MB 1:41/24.13 MB
>>>>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+9:32&entry=gmail&source=g>
>>>>> >> > 9:32
>>>>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+9:32&entry=gmail&source=g>/18.79
>>>>> MB 5:41 <https://maps.google.com/?q=MB+5:41&entry=gmail&source=g>/25.31
>>>>> MB 2:42 <https://maps.google.com/?q=MB+2:42&entry=gmail&source=g>/26.49
>>>>> MB 7:36 <https://maps.google.com/?q=MB+7:36&entry=gmail&source=g>/22.87
>>>>> MB 3:51 <https://maps.google.com/?q=MB+3:51&entry=gmail&source=g>/31.00
>>>>> MB
>>>>> >> > 4:41/25.99 MB 8:38
>>>>> <https://maps.google.com/?q=MB+8:38&entry=gmail&source=g>/23.79 MB
>>>>> >> > ExecOption: PARQUET Codegen Enabled, Codegen enabled:
>>>>> 459 out
>>>>> >> > of
>>>>> >> > 459
>>>>> >> > Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0%
>>>>> 3:0%
>>>>> >> > 4:0%
>>>>> >> > 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
>>>>> >> > File Formats: PARQUET/Unknown(Skipped):459
>>>>> >> > BytesRead(500.000ms): 43.15 MB
>>>>> >> > - FooterProcessingTime: (Avg: 11.257ms ; Min: 3.989ms
>>>>> ; Max:
>>>>> >> > 23.004ms ; Number of samples: 459)
>>>>> >> > - AverageHdfsReadThreadConcurrency: 0.00
>>>>> >> > - AverageScannerThreadConcurrency: 24.00
>>>>> >> > - BytesRead: 44.52 MB (46678031)
>>>>> >> > - BytesReadDataNodeCache: 0
>>>>> >> > - BytesReadLocal: 44.52 MB (46678031)
>>>>> >> > - BytesReadRemoteUnexpected: 0
>>>>> >> > - BytesReadShortCircuit: 44.52 MB (46678031)
>>>>> >> > - CachedFileHandlesHitCount: 0 (0)
>>>>> >> > - CachedFileHandlesMissCount: 459 (459)
>>>>> >> > - CollectionItemsRead: 0 (0)
>>>>> >> > - DecompressionTime: 0.000ns
>>>>> >> > - MaxCompressedTextFileLength: 0
>>>>> >> > - NumColumns: 0 (0)
>>>>> >> > - NumDictFilteredRowGroups: 0 (0)
>>>>> >> > - NumDisksAccessed: 12 (12)
>>>>> >> > - NumRowGroups: 459 (459)
>>>>> >> > - NumScannerThreadsStarted: 24 (24)
>>>>> >> > - NumScannersWithNoReads: 0 (0)
>>>>> >> > - NumStatsFilteredRowGroups: 0 (0)
>>>>> >> > - PeakMemoryUsage: 2.50 MB (2621440)
>>>>> >> > - PerReadThreadRawHdfsThroughput: 1.33 GB/sec
>>>>> >> > - RemoteScanRanges: 0 (0)
>>>>> >> > - RowBatchQueueGetWaitTime: 243.426ms
>>>>> >> > - RowBatchQueuePutWaitTime: 0.000ns
>>>>> >> > - RowsRead: 0 (0)
>>>>> >> > - RowsReturned: 459 (459)
>>>>> >> > - RowsReturnedRate: 1.79 K/sec
>>>>> >> > - ScanRangesComplete: 459 (459)
>>>>> >> > - ScannerThreadsInvoluntaryContextSwitches: 85 (85)
>>>>> >> > - ScannerThreadsTotalWallClockTime: 5s978ms
>>>>> >> > - MaterializeTupleTime(*): 67.997us
>>>>> >> > - ScannerThreadsSysTime: 22.984ms
>>>>> >> > - ScannerThreadsUserTime: 678.884ms
>>>>> >> > - ScannerThreadsVoluntaryContextSwitches: 1.72K (1724)
>>>>> >> > - TotalRawHdfsReadTime(*): 32.626ms
>>>>> >> > - TotalReadThroughput: 86.30 MB/sec
>>>>> >> > Instance c843dae48ce9b289:6ad3ffc000000005
>>>>> >> > (host=aopr-dhc001.lpdomain.com:22000):(Total: 298.093ms,
>>>>> non-child:
>>>>> >> > 1.242ms,
>>>>> >> > % non-child: 0.42%)
>>>>> >> > Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>>>>> >> > 9:55/35.38 MB 0:37/24.23 MB 5:37
>>>>> <https://maps.google.com/?q=MB+5:37&entry=gmail&source=g>/23.31 MB
>>>>> 11:22 <https://maps.google.com/?q=MB+11:22&entry=gmail&source=g>/13.79
>>>>> MB 8:30 <https://maps.google.com/?q=MB+8:30&entry=gmail&source=g>/18.53
>>>>> MB
>>>>> >> > 4:33/20.45 MB 1:38/24.06 MB 10:29
>>>>> <https://maps.google.com/?q=MB+10:29&entry=gmail&source=g>/18.40 MB
>>>>> 3:39 <https://maps.google.com/?q=MB+3:39&entry=gmail&source=g>/24.35 MB
>>>>> 2:41 <https://maps.google.com/?q=MB+2:41&entry=gmail&source=g>/25.45
>>>>> MB
>>>>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+6:42&entry=gmail&source=g>
>>>>> >> > 6:42
>>>>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+6:42&entry=gmail&source=g>/27.03
>>>>> MB 7:47 <https://maps.google.com/?q=MB+7:47&entry=gmail&source=g>/29.22
>>>>> MB
>>>>> >> > - AverageThreadTokens: 0.00
>>>>> >> > - BloomFilterBytes: 0
>>>>> >> > - PeakMemoryUsage: 2.66 MB (2793825)
>>>>> >> > - PeakReservation: 0
>>>>> >> > - PeakUsedReservation: 0
>>>>> >> > - PerHostPeakMemUsage: 2.68 MB (2806113)
>>>>> >> > - RowsProduced: 1 (1)
>>>>> >> > - TotalNetworkReceiveTime: 0.000ns
>>>>> >> > - TotalNetworkSendTime: 198.894us
>>>>> >> > - TotalStorageWaitTime: 4s374ms
>>>>> >> > - TotalThreadsInvoluntaryContextSwitches: 110 (110)
>>>>> >> > - TotalThreadsTotalWallClockTime: 6s201ms
>>>>> >> > - TotalThreadsSysTime: 29.983ms
>>>>> >> > - TotalThreadsUserTime: 642.892ms
>>>>> >> > - TotalThreadsVoluntaryContextSwitches: 2.17K (2174)
>>>>> >> > Fragment Instance Lifecycle Timings:
>>>>> >> > - ExecTime: 350.873us
>>>>> >> > - ExecTreeExecTime: 8.173us
>>>>> >> > - OpenTime: 275.347ms
>>>>> >> > - ExecTreeOpenTime: 253.425ms
>>>>> >> > - PrepareTime: 22.341ms
>>>>> >> > - ExecTreePrepareTime: 2.251ms
>>>>> >> > DataStreamSender (dst_id=2):(Total: 164.394us, non-child:
>>>>> >> > 164.394us,
>>>>> >> > % non-child: 100.00%)
>>>>> >> > - BytesSent: 16.00 B (16)
>>>>> >> > - NetworkThroughput(*): 493.42 KB/sec
>>>>> >> > - OverallThroughput: 95.05 KB/sec
>>>>> >> > - PeakMemoryUsage: 16.00 KB (16384)
>>>>> >> > - RowsReturned: 1 (1)
>>>>> >> > - SerializeBatchTime: 14.262us
>>>>> >> > - TransmitDataRPCTime: 31.667us
>>>>> >> > - UncompressedRowBatchSize: 16.00 B (16)
>>>>> >> > CodeGen:(Total: 41.008ms, non-child: 41.008ms, %
>>>>> non-child:
>>>>> >> > 100.00%)
>>>>> >> > - CodegenTime: 1.049ms
>>>>> >> > - CompileTime: 7.248ms
>>>>> >> > - LoadTime: 0.000ns
>>>>> >> > - ModuleBitcodeSize: 1.86 MB (1950880)
>>>>> >> > - NumFunctions: 15 (15)
>>>>> >> > - NumInstructions: 214 (214)
>>>>> >> > - OptimizationTime: 14.018ms
>>>>> >> > - PeakMemoryUsage: 107.00 KB (109568)
>>>>> >> > - PrepareTime: 19.080ms
>>>>> >> > AGGREGATION_NODE (id=1):(Total: 255.678ms, non-child:
>>>>> 1.046ms, %
>>>>> >> > non-child: 0.41%)
>>>>> >> > ExecOption: Codegen Enabled
>>>>> >> > - BuildTime: 1.015ms
>>>>> >> > - GetResultsTime: 0.000ns
>>>>> >> > - HTResizeTime: 0.000ns
>>>>> >> > - HashBuckets: 0 (0)
>>>>> >> > - LargestPartitionPercent: 0 (0)
>>>>> >> > - MaxPartitionLevel: 0 (0)
>>>>> >> > - NumRepartitions: 0 (0)
>>>>> >> > - PartitionsCreated: 0 (0)
>>>>> >> > - PeakMemoryUsage: 20.00 KB (20480)
>>>>> >> > - RowsRepartitioned: 0 (0)
>>>>> >> > - RowsReturned: 1 (1)
>>>>> >> > - RowsReturnedRate: 3.00 /sec
>>>>> >> > - SpilledPartitions: 0 (0)
>>>>> >> > HDFS_SCAN_NODE (id=0):(Total: 254.631ms, non-child:
>>>>> 254.631ms, %
>>>>> >> > non-child: 100.00%)
>>>>> >> > Hdfs split stats (<volume id>:<# splits>/<split
>>>>> lengths>):
>>>>> >> > 9:55/35.38 MB 0:37/24.23 MB 5:37
>>>>> <https://maps.google.com/?q=MB+5:37&entry=gmail&source=g>/23.31 MB
>>>>> 11:22 <https://maps.google.com/?q=MB+11:22&entry=gmail&source=g>/13.79
>>>>> MB 8:30 <https://maps.google.com/?q=MB+8:30&entry=gmail&source=g>/18.53
>>>>> MB
>>>>> >> > 4:33/20.45 MB 1:38/24.06 MB 10:29
>>>>> <https://maps.google.com/?q=MB+10:29&entry=gmail&source=g>/18.40 MB
>>>>> 3:39 <https://maps.google.com/?q=MB+3:39&entry=gmail&source=g>/24.35 MB
>>>>> 2:41 <https://maps.google.com/?q=MB+2:41&entry=gmail&source=g>/25.45
>>>>> MB
>>>>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+6:42&entry=gmail&source=g>
>>>>> >> > 6:42
>>>>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+6:42&entry=gmail&source=g>/27.03
>>>>> MB 7:47 <https://maps.google.com/?q=MB+7:47&entry=gmail&source=g>/29.22
>>>>> MB
>>>>> >> > ExecOption: PARQUET Codegen Enabled, Codegen enabled:
>>>>> 450 out
>>>>> >> > of
>>>>> >> > 450
>>>>> >> > Hdfs Read Thread Concurrency Bucket: 0:0% 1:0% 2:0%
>>>>> 3:0% 4:0%
>>>>> >> > 5:0%
>>>>> >> > 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
>>>>> >> > File Formats: PARQUET/Unknown(Skipped):450
>>>>> >> > - FooterProcessingTime: (Avg: 10.727ms ; Min: 3.813ms
>>>>> ; Max:
>>>>> >> > 23.755ms ; Number of samples: 450)
>>>>> >> > - AverageHdfsReadThreadConcurrency: 0.00
>>>>> >> > - AverageScannerThreadConcurrency: 0.00
>>>>> >> > - BytesRead: 43.85 MB (45983284)
>>>>> >> > - BytesReadDataNodeCache: 0
>>>>> >> > - BytesReadLocal: 43.85 MB (45983284)
>>>>> >> > - BytesReadRemoteUnexpected: 0
>>>>> >> > - BytesReadShortCircuit: 43.85 MB (45983284)
>>>>> >> > - CachedFileHandlesHitCount: 0 (0)
>>>>> >> > - CachedFileHandlesMissCount: 450 (450)
>>>>> >> > - CollectionItemsRead: 0 (0)
>>>>> >> > - DecompressionTime: 0.000ns
>>>>> >> > - MaxCompressedTextFileLength: 0
>>>>> >> > - NumColumns: 0 (0)
>>>>> >> > - NumDictFilteredRowGroups: 0 (0)
>>>>> >> > - NumDisksAccessed: 12 (12)
>>>>> >> > - NumRowGroups: 450 (450)
>>>>> >> > - NumScannerThreadsStarted: 24 (24)
>>>>> >> > - NumScannersWithNoReads: 0 (0)
>>>>> >> > - NumStatsFilteredRowGroups: 0 (0)
>>>>> >> > - PeakMemoryUsage: 2.62 MB (2752512)
>>>>> >> > - PerReadThreadRawHdfsThroughput: 1.56 GB/sec
>>>>> >> > - RemoteScanRanges: 0 (0)
>>>>> >> > - RowBatchQueueGetWaitTime: 242.362ms
>>>>> >> > - RowBatchQueuePutWaitTime: 0.000ns
>>>>> >> > - RowsRead: 0 (0)
>>>>> >> > - RowsReturned: 450 (450)
>>>>> >> > - RowsReturnedRate: 1.77 K/sec
>>>>> >> > - ScanRangesComplete: 450 (450)
>>>>> >> > - ScannerThreadsInvoluntaryContextSwitches: 106 (106)
>>>>> >> > - ScannerThreadsTotalWallClockTime: 5s926ms
>>>>> >> > - MaterializeTupleTime(*): 69.447us
>>>>> >> > - ScannerThreadsSysTime: 26.984ms
>>>>> >> > - ScannerThreadsUserTime: 607.897ms
>>>>> >> > - ScannerThreadsVoluntaryContextSwitches: 1.72K (1723)
>>>>> >> > - TotalRawHdfsReadTime(*): 27.393ms
>>>>> >> > - TotalReadThroughput: 0.00 /sec
>>>>> >> > ===================================================
>>>>> >> >
>>>>> >> > Coordinator log:
>>>>> >> >
>>>>> >> > 01: query (string) = "select count(*) from
>>>>> dv_entersiteevent_prq_remote
>>>>> >> > where year=2017 and month=10",
>>>>> >> > 03: configuration (list) = list<string>[1] {
>>>>> >> > [0] = "REQUEST_POOL=impreports",
>>>>> >> > },
>>>>> >> > 04: hadoop_user (string) = "fawzea",
>>>>> >> > }
>>>>> >> > I0208 01:07:06.321281 7154 impala-beeswax-server.cc:459]
>>>>> >> > TClientRequest.queryOptions: TQueryOptions {
>>>>> >> > 01: abort_on_error (bool) = false,
>>>>> >> > 02: max_errors (i32) = 100,
>>>>> >> > 03: disable_codegen (bool) = false,
>>>>> >> > 04: batch_size (i32) = 0,
>>>>> >> > 05: num_nodes (i32) = 0,
>>>>> >> > 06: max_scan_range_length (i64) = 0,
>>>>> >> > 07: num_scanner_threads (i32) = 0,
>>>>> >> > 08: max_io_buffers (i32) = 0,
>>>>> >> > 09: allow_unsupported_formats (bool) = false,
>>>>> >> > 10: default_order_by_limit (i64) = -1,
>>>>> >> > 11: debug_action (string) = "",
>>>>> >> > 12: mem_limit (i64) = 0,
>>>>> >> > 13: abort_on_default_limit_exceeded (bool) = false,
>>>>> >> > 15: hbase_caching (i32) = 0,
>>>>> >> > 16: hbase_cache_blocks (bool) = false,
>>>>> >> > 17: parquet_file_size (i64) = 0,
>>>>> >> > 18: explain_level (i32) = 1,
>>>>> >> > 19: sync_ddl (bool) = false,
>>>>> >> > 20: request_pool (string) = "impreports",
>>>>> >> > 23: disable_cached_reads (bool) = false,
>>>>> >> > 24: disable_outermost_topn (bool) = false,
>>>>> >> > 25: rm_initial_mem (i64) = 0,
>>>>> >> > 26: query_timeout_s (i32) = 0,
>>>>> >> > 28: appx_count_distinct (bool) = false,
>>>>> >> > 29: disable_unsafe_spills (bool) = false,
>>>>> >> > 31: exec_single_node_rows_threshold (i32) = 100,
>>>>> >> > 32: optimize_partition_key_scans (bool) = false,
>>>>> >> > 33: replica_preference (i32) = 0,
>>>>> >> > 34: schedule_random_replica (bool) = false,
>>>>> >> > 35: scan_node_codegen_threshold (i64) = 1800000,
>>>>> >> > 36: disable_streaming_preaggregations (bool) = false,
>>>>> >> > 37: runtime_filter_mode (i32) = 2,
>>>>> >> > 38: runtime_bloom_filter_size (i32) = 1048576,
>>>>> >> > 39: runtime_filter_wait_time_ms (i32) = 0,
>>>>> >> > 40: disable_row_runtime_filtering (bool) = false,
>>>>> >> > 41: max_num_runtime_filters (i32) = 10,
>>>>> >> > 42: parquet_annotate_strings_utf8 (bool) = false,
>>>>> >> > 43: parquet_fallback_schema_resolution (i32) = 0,
>>>>> >> > 45: s3_skip_insert_staging (bool) = true,
>>>>> >> > 46: runtime_filter_min_size (i32) = 1048576,
>>>>> >> > 47: runtime_filter_max_size (i32) = 16777216,
>>>>> >> > 48: prefetch_mode (i32) = 1,
>>>>> >> > 49: strict_mode (bool) = false,
>>>>> >> > 50: scratch_limit (i64) = -1,
>>>>> >> > 51: enable_expr_rewrites (bool) = true,
>>>>> >> > 52: decimal_v2 (bool) = false,
>>>>> >> > 53: parquet_dictionary_filtering (bool) = true,
>>>>> >> > 54: parquet_array_resolution (i32) = 2,
>>>>> >> > 55: parquet_read_statistics (bool) = true,
>>>>> >> > 56: default_join_distribution_mode (i32) = 0,
>>>>> >> > 57: disable_codegen_rows_threshold (i32) = 50000,
>>>>> >> > 58: default_spillable_buffer_size (i64) = 2097152,
>>>>> >> > 59: min_spillable_buffer_size (i64) = 65536,
>>>>> >> > 60: max_row_size (i64) = 524288,
>>>>> >> > }
>>>>> >> > I0208 01:07:06.324410 7154 Frontend.java:909] Compiling query:
>>>>> select
>>>>> >> > count(*) from dv_entersiteevent_prq_remote where year=2017 and
>>>>> month=10
>>>>> >> > I0208 01:07:06.327203 7154 Frontend.java:948] Compiled query.
>>>>> >> > I0208 01:07:06.359748 7154 admission-controller.cc:510] Schedule
>>>>> for
>>>>> >> > id=5f49e927591a9042:faf8f28600000000 in pool_name=root.impreports
>>>>> >> > cluster_mem_needed=105.00 MB PoolConfig: max_requests=-1
>>>>> max_queued=200
>>>>> >> > max_mem=-1.00 B
>>>>> >> > I0208 01:07:06.359776 7154 admission-controller.cc:515] Stats:
>>>>> >> > agg_num_running=0, agg_num_queued=0, agg_mem_reserved=0,
>>>>> >> > local_host(local_mem_admitted=0, num_admitted_running=0,
>>>>> num_queued=0,
>>>>> >> > backend_mem_reserved=0)
>>>>> >> > I0208 01:07:06.359784 7154 admission-controller.cc:531] Admitted
>>>>> query
>>>>> >> > id=5f49e927591a9042:faf8f28600000000
>>>>> >> > I0208 01:07:06.359807 7154 coordinator.cc:99] Exec()
>>>>> >> > query_id=5f49e927591a9042:faf8f28600000000 stmt=select count(*)
>>>>> from
>>>>> >> > dv_entersiteevent_prq_remote where year=2017 and month=10
>>>>> >> > I0208 01:07:06.360694 7154 coordinator.cc:357] starting
>>>>> execution on 5
>>>>> >> > backends for query 5f49e927591a9042:faf8f28600000000
>>>>> >> > I0208 01:07:06.363255 63583 impala-internal-service.cc:44]
>>>>> >> > ExecQueryFInstances(): query_id=5f49e927591a9042:faf8f28600000000
>>>>> >> > I0208 01:07:06.363282 63583 query-exec-mgr.cc:46]
>>>>> StartQueryFInstances()
>>>>> >> > query_id=5f49e927591a9042:faf8f28600000000
>>>>> >> > coord=aopr-dhc001.lpdomain.com:22000
>>>>> >> > I0208 01:07:06.363296 63583 query-state.cc:173] Buffer pool limit
>>>>> for
>>>>> >> > 5f49e927591a9042:faf8f28600000000: 10802010521
>>>>> >> > I0208 01:07:06.363364 63583 initial-reservations.cc:60]
>>>>> Successfully
>>>>> >> > claimed
>>>>> >> > initial reservations (0) for query 5f49e927591a9042:faf8f28600000
>>>>> 000
>>>>> >> > I0208 01:07:06.363544 7155 query-state.cc:286] StartFInstances():
>>>>> >> > query_id=5f49e927591a9042:faf8f28600000000 #instances=2
>>>>> >> > I0208 01:07:06.363844 7155 query-state.cc:299] descriptor table
>>>>> for
>>>>> >> > query=5f49e927591a9042:faf8f28600000000
>>>>> >> > tuples:
>>>>> >> > Tuple(id=2 size=0 slots=[] tuple_path=[])
>>>>> >> > Tuple(id=1 size=8 slots=[Slot(id=2 type=BIGINT col_path=[]
>>>>> offset=0
>>>>> >> > null=(offset=0 mask=0) slot_idx=0 field_idx=-1)] tuple_path=[])
>>>>> >> > Tuple(id=0 size=8 slots=[Slot(id=3 type=BIGINT col_path=[]
>>>>> offset=0
>>>>> >> > null=(offset=0 mask=0) slot_idx=0 field_idx=-1)] tuple_path=[])
>>>>> >> > I0208 01:07:06.364013 7156 query-state.cc:377] Executing
>>>>> instance.
>>>>> >> > instance_id=5f49e927591a9042:faf8f28600000000 fragment_idx=0
>>>>> >> > per_fragment_instance_idx=0 coord_state_idx=0 #in-flight=1
>>>>> >> > I0208 01:07:06.364135 7157 query-state.cc:377] Executing
>>>>> instance.
>>>>> >> > instance_id=5f49e927591a9042:faf8f28600000005 fragment_idx=1
>>>>> >> > per_fragment_instance_idx=4 coord_state_idx=0 #in-flight=2
>>>>> >> > I0208 01:07:06.364241 7157 hdfs-scan-node.cc:160] Max row batch
>>>>> queue
>>>>> >> > size
>>>>> >> > for scan node '0' in fragment instance
>>>>> >> > '5f49e927591a9042:faf8f28600000005':
>>>>> >> > 160
>>>>> >> > I0208 01:07:06.366775 7154 coordinator.cc:370] started execution
>>>>> on 5
>>>>> >> > backends for query 5f49e927591a9042:faf8f28600000000
>>>>> >> > I0208 01:07:06.385625 7155 query-exec-mgr.cc:149]
>>>>> ReleaseQueryState():
>>>>> >> > query_id=5f49e927591a9042:faf8f28600000000 refcnt=4
>>>>> >> > I0208 01:07:06.671193 7559 impala-server.cc:1796] Connection from
>>>>> >> > client
>>>>> >> > ::ffff:10.16.136.3:55598 closed, closing 1 associated session(s)
>>>>> >> > I0208 01:07:06.676138 544 data-stream-mgr.cc:228] Reduced
>>>>> stream ID
>>>>> >> > cache
>>>>> >> > from 2 items, to 0, eviction took: 0
>>>>> >> > I0208 01:07:06.677202 544 coordinator.cc:955] Backend completed:
>>>>> >> > host=aopr-dhc002.lpdomain.com:22000 remaining=4
>>>>> >> > I0208 01:07:06.677214 544 coordinator-backend-state.cc:229]
>>>>> >> > query_id=5f49e927591a9042:faf8f28600000000: first in-progress
>>>>> backend:
>>>>> >> > aopr-dhc001.lpdomain.com:22000
>>>>> >> > I0208 01:07:06.684221 7157 query-state.cc:384] Instance
>>>>> completed.
>>>>> >> > instance_id=5f49e927591a9042:faf8f28600000005 #in-flight=1
>>>>> status=OK
>>>>> >> > I0208 01:07:06.684237 7157 query-exec-mgr.cc:149]
>>>>> ReleaseQueryState():
>>>>> >> > query_id=5f49e927591a9042:faf8f28600000000 refcnt=3
>>>>> >> > I0208 01:07:06.700275 520 coordinator.cc:955] Backend completed:
>>>>> >> > host=aopr-dhc005.lpdomain.com:22000 remaining=3
>>>>> >> > I0208 01:07:06.700289 520 coordinator-backend-state.cc:229]
>>>>> >> > query_id=5f49e927591a9042:faf8f28600000000: first in-progress
>>>>> backend:
>>>>> >> > aopr-dhc001.lpdomain.com:22000
>>>>> >> > I0208 01:07:06.704620 65258 coordinator.cc:955] Backend completed:
>>>>> >> > host=aopr-dhc003.lpdomain.com:22000 remaining=2
>>>>> >> > I0208 01:07:06.704666 65258 coordinator-backend-state.cc:229]
>>>>> >> > query_id=5f49e927591a9042:faf8f28600000000: first in-progress
>>>>> backend:
>>>>> >> > aopr-dhc001.lpdomain.com:22000
>>>>> >> > I0208 01:07:06.706041 7156 data-stream-mgr.cc:238]
>>>>> DeregisterRecvr():
>>>>> >> > fragment_instance_id=5f49e927591a9042:faf8f28600000000, node=2
>>>>> >> > I0208 01:07:06.706070 7156 data-stream-recvr.cc:235] cancelled
>>>>> stream:
>>>>> >> > fragment_instance_id_=5f49e927591a9042:faf8f28600000000 node_id=2
>>>>> >> > I0208 01:07:06.706809 64916 coordinator.cc:955] Backend completed:
>>>>> >> > host=aopr-dhc004.lpdomain.com:22000 remaining=1
>>>>> >> > I0208 01:07:06.706821 64916 coordinator-backend-state.cc:229]
>>>>> >> > query_id=5f49e927591a9042:faf8f28600000000: first in-progress
>>>>> backend:
>>>>> >> > aopr-dhc001.lpdomain.com:22000
>>>>> >> > I0208 01:07:06.789568 7154 impala-beeswax-server.cc:193]
>>>>> >> > get_results_metadata(): query_id=5f49e927591a9042:faf8
>>>>> f28600000000
>>>>> >> > I0208 01:07:06.790439 7154 coordinator.cc:789] Coordinator
>>>>> waiting for
>>>>> >> > backends to finish, 1 remaining
>>>>> >> > I0208 01:07:06.791052 7154 coordinator.cc:794] All backends
>>>>> finished
>>>>> >> > successfully.
>>>>> >> > I0208 01:07:06.791086 7154 coordinator.cc:1090] Release admssion
>>>>> >> > control
>>>>> >> > resources for query 5f49e927591a9042:faf8f28600000000
>>>>> >> > I0208 01:07:06.791363 7156 query-state.cc:384] Instance
>>>>> completed.
>>>>> >> > instance_id=5f49e927591a9042:faf8f28600000000 #in-flight=0
>>>>> status=OK
>>>>> >> > I0208 01:07:06.791414 7156 query-exec-mgr.cc:149]
>>>>> ReleaseQueryState():
>>>>> >> > query_id=5f49e927591a9042:faf8f28600000000 refcnt=2
>>>>> >> > I0208 01:07:06.793586 7154 impala-beeswax-server.cc:235] close():
>>>>> >> > query_id=5f49e927591a9042:faf8f28600000000
>>>>> >> > I0208 01:07:06.793602 7154 impala-server.cc:992]
>>>>> UnregisterQuery():
>>>>> >> > query_id=5f49e927591a9042:faf8f28600000000
>>>>> >> > I0208 01:07:06.793614 7154 impala-server.cc:1075] Cancel():
>>>>> >> > query_id=5f49e927591a9042:faf8f28600000000
>>>>> >> > I0208 01:07:06.793624 7154 coordinator.cc:895] Cancel()
>>>>> >> > query_id=5f49e927591a9042:faf8f28600000000
>>>>> >> > I0208 01:07:06.793637 7154 coordinator.cc:905] CancelBackends()
>>>>> >> > query_id=5f49e927591a9042:faf8f28600000000, tried to cancel 0
>>>>> backends
>>>>> >> > I0208 01:07:06.802776 7154 query-exec-mgr.cc:149]
>>>>> ReleaseQueryState():
>>>>> >> > query_id=5f49e927591a9042:faf8f28600000000 refcnt=1
>>>>> >> > I0208 01:07:06.814214 7154 impala-server.cc:1796] Connection from
>>>>> >> > client
>>>>> >> > ::ffff:10.16.144.131:51012 closed, closing 1 associated
>>>>> session(s)
>>>>> >> > I0208 01:07:07.258891 7567 impala-server.cc:1796] Connection from
>>>>> >> > client
>>>>> >> > ::ffff:10.16.136.2:56375 closed, closing 1 associated session(s)
>>>>> >> > I0208 01:07:11.600131 7819 impala-server.cc:1796] Connection from
>>>>> >> > client
>>>>> >> > ::ffff:10.16.136.3:27460 closed, closing 1 associated session(s)
>>>>> >> > I0208 01:07:12.262739 7821 impala-server.cc:1796] Connection from
>>>>> >> > client
>>>>> >> > ::ffff:10.16.136.2:52672 closed, closing 1 associated session(s)
>>>>> >> > I0208 01:07:16.665451 7822 impala-server.cc:1796] Connection from
>>>>> >> > client
>>>>> >> > ::ffff:10.16.136.3:10693 closed, closing 1 associated session(s)
>>>>> >> > I0208 01:07:17.235321 7825 impala-server.cc:1796] Connection from
>>>>> >> > client
>>>>> >> > ::ffff:10.16.136.2:34752 closed, closing 1 associated session(s)
>>>>> >> > I0208 01:07:21.410611 5858 impala-beeswax-server.cc:193]
>>>>> >> > get_results_metadata(): query_id=1d4fbf0f2571f1e9:ea34
>>>>> 6f7900000000
>>>>> >> > I0208 01:07:21.414175 5858 impala-beeswax-server.cc:235] close():
>>>>> >> > query_id=1d4fbf0f2571f1e9:ea346f7900000000
>>>>> >> > I0208 01:07:21.414185 5858 impala-server.cc:992]
>>>>> UnregisterQuery():
>>>>> >> > query_id=1d4fbf0f2571f1e9:ea346f7900000000
>>>>> >> > I0208 01:07:21.414191 5858 impala-server.cc:1075] Cancel():
>>>>> >> > query_id=1d4fbf0f2571f1e9:ea346f7900000000
>>>>> >> > I0208 01:07:21.421505 5858 impala-server.cc:1796] Connection from
>>>>> >> > client
>>>>> >> > ::ffff:10.26.180.75:60072 closed, closing 1 associated session(s)
>>>>> >> > I0208 01:07:21.605690 7830 impala-server.cc:1796] Connection from
>>>>> >> > client
>>>>> >> > ::ffff:10.16.136.3:46660 closed, closing 1 associated session(s)
>>>>> >> >
>>>>> >> >
>>>>> >> > On Thu, Feb 8, 2018 at 6:20 AM, Matthew Jacobs <
>>>>> mjacobs@apache.org>
>>>>> >> > wrote:
>>>>> >> >>
>>>>> >> >> Thanks! I think that configuration looks OK.
>>>>> >> >>
>>>>> >> >> 1) What version of Impala is this?
>>>>> >> >>
>>>>> >> >> 2) Can you send the query profile and the Impala coordinator log
>>>>> for
>>>>> >> >> the query where it's not behaving as expected?
>>>>> >> >>
>>>>> >> >> > When i'm running the following with fawzea user:
>>>>> >> >> > set request_pool=impreports;
>>>>> >> >> > select count(*) from dv_entersiteevent_prq_remote where
>>>>> year=2017 and
>>>>> >> >> > month=10;
>>>>> >> >> > I can see that the query is running with impreports pool.
>>>>> >> >>
>>>>> >> >> On Wed, Feb 7, 2018 at 7:59 PM, Fawze Abujaber <
>>>>> fawzeaj@gmail.com>
>>>>> >> >> wrote:
>>>>> >> >> > Hi Matt,
>>>>> >> >> >
>>>>> >> >> > [root@aopr-dhc001 ~]# cat
>>>>> >> >> >
>>>>> >> >> >
>>>>> >> >> > /var/run/cloudera-scm-agent/process/4558-impala-IMPALAD/impa
>>>>> la-conf/fair-scheduler.xml
>>>>> >> >> > <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
>>>>> >> >> > <allocations>
>>>>> >> >> > <queue name="root">
>>>>> >> >> > <weight>1.0</weight>
>>>>> >> >> > <schedulingPolicy>fair</schedulingPolicy>
>>>>> >> >> > <queue name="impiaas">
>>>>> >> >> > <weight>1.0</weight>
>>>>> >> >> > <schedulingPolicy>fair</schedulingPolicy>
>>>>> >> >> > <aclSubmitApps>fawzea </aclSubmitApps>
>>>>> >> >> > </queue>
>>>>> >> >> > <queue name="impreports">
>>>>> >> >> > <weight>1.0</weight>
>>>>> >> >> > <schedulingPolicy>fair</schedulingPolicy>
>>>>> >> >> > <aclSubmitApps>nira </aclSubmitApps>
>>>>> >> >> > </queue>
>>>>> >> >> > </queue>
>>>>> >> >> > <defaultQueueSchedulingPolicy
>>>>> >fair</defaultQueueSchedulingPolicy>
>>>>> >> >> > <queuePlacementPolicy>
>>>>> >> >> > <rule name="specified" create="false"/>
>>>>> >> >> > <rule name="secondaryGroupExistingQueue"/>
>>>>> >> >> > <rule name="default"/>
>>>>> >> >> > </queuePlacementPolicy>
>>>>> >> >> > </allocations>
>>>>> >> >> >
>>>>> >> >> >
>>>>> >> >> > On Thu, Feb 8, 2018 at 4:47 AM, Matthew Jacobs <
>>>>> jacobs.mh@gmail.com>
>>>>> >> >> > wrote:
>>>>> >> >> >>
>>>>> >> >> >> Yes, I saw that, but I'd like to see the actual configuration
>>>>> that
>>>>> >> >> >> is
>>>>> >> >> >> passed to Impala because it may not be specified in the right
>>>>> way to
>>>>> >> >> >> accomplish what you're trying to do in "b".
>>>>> >> >> >>
>>>>> >> >> >> On Wed, Feb 7, 2018 at 4:32 PM, Fawze Abujaber <
>>>>> fawzeaj@gmail.com>
>>>>> >> >> >> wrote:
>>>>> >> >> >> > Hi Matt,
>>>>> >> >> >> >
>>>>> >> >> >> > I created the pools manually and it’s completely
>>>>> nfigurations as
>>>>> >> >> >> > shown
>>>>> >> >> >> > in b.
>>>>> >> >> >> >
>>>>> >> >> >> > On Wed, 7 Feb 2018 at 23:17 Matthew Jacobs <
>>>>> jacobs.mh@gmail.com>
>>>>> >> >> >> > wrote:
>>>>> >> >> >> >>
>>>>> >> >> >> >> How, more specifically, are you configuring these pools?
>>>>> Can you
>>>>> >> >> >> >> share
>>>>> >> >> >> >> the configuration files?
>>>>> >> >> >> >>
>>>>> >> >> >> >> On Wed, Feb 7, 2018 at 11:42 AM, Fawze Abujaber
>>>>> >> >> >> >> <fa...@gmail.com>
>>>>> >> >> >> >> wrote:
>>>>> >> >> >> >> > Hello All,
>>>>> >> >> >> >> >
>>>>> >> >> >> >> > Will appreciate any insight of what i'm missing in my
>>>>> >> >> >> >> > configuration
>>>>> >> >> >> >> > to
>>>>> >> >> >> >> > get
>>>>> >> >> >> >> > the impala submission control working as expected ....
>>>>> >> >> >> >> >
>>>>> >> >> >> >> > a- Here is my placement Placement Rule:
>>>>> >> >> >> >> > 1- Use the pool Specified at run time., only if the pool
>>>>> >> >> >> >> > exists.
>>>>> >> >> >> >> > 2- Use the pool root.[secondary group].
>>>>> >> >> >> >> > 3- Use the pool root.default.
>>>>> >> >> >> >> >
>>>>> >> >> >> >> > b- Here is the impala resource pools
>>>>> >> >> >> >> >
>>>>> >> >> >> >> > Name Max Memory Max Running Queries
>>>>> Max
>>>>> >> >> >> >> > Queued
>>>>> >> >> >> >> > Queries
>>>>> >> >> >> >> > Queue Timeout Default Query Memory Limit Access Control
>>>>> >> >> >> >> > root.impiaas Unlimited
>>>>> Unlimited
>>>>> >> >> >> >> > 200 1 minute(s) No
>>>>> Default
>>>>> >> >> >> >> > Custom
>>>>> >> >> >> >> > root.impreports Unlimited
>>>>> Unlimited
>>>>> >> >> >> >> > 200 1 minute(s) No
>>>>> Default
>>>>> >> >> >> >> > Custom
>>>>> >> >> >> >> >
>>>>> >> >> >> >> >
>>>>> >> >> >> >> > C- Here is my Access control for root.impiaas pool:
>>>>> >> >> >> >> >
>>>>> >> >> >> >> > Allow these users and groups to submit to this pool
>>>>> >> >> >> >> >
>>>>> >> >> >> >> > Users: fawzea
>>>>> >> >> >> >> >
>>>>> >> >> >> >> > The same for impreports the allowed user and groups to
>>>>> submit
>>>>> >> >> >> >> > to
>>>>> >> >> >> >> > pool
>>>>> >> >> >> >> > is
>>>>> >> >> >> >> > only nira.
>>>>> >> >> >> >> >
>>>>> >> >> >> >> >
>>>>> >> >> >> >> > LDAP groups for fawzea user:
>>>>> >> >> >> >> >
>>>>> >> >> >> >> > Member in groups:
>>>>> >> >> >> >> >
>>>>> >> >> >> >> > - dba
>>>>> >> >> >> >> > - impiaas
>>>>> >> >> >> >> > - mysql
>>>>> >> >> >> >> >
>>>>> >> >> >> >> > LDAP group for nira user
>>>>> >> >> >> >> > Member in groups:
>>>>> >> >> >> >> > - dba
>>>>> >> >> >> >> > - impreports
>>>>> >> >> >> >> > - mysql
>>>>> >> >> >> >> >
>>>>> >> >> >> >> >
>>>>> >> >> >> >> >
>>>>> >> >> >> >> > Impala Queries UI shows:
>>>>> >> >> >> >> >
>>>>> >> >> >> >> > 1- select count(*) from dv_entersiteevent_prq_remote
>>>>> where
>>>>> >> >> >> >> > year=2017
>>>>> >> >> >> >> >
>>>>> >> >> >> >> > User: fawzea
>>>>> >> >> >> >> > Pool: root.impiaas
>>>>> >> >> >> >> >
>>>>> >> >> >> >> > 2- select count(*) from dv_entersiteevent_prq_remote
>>>>> where
>>>>> >> >> >> >> > year=2017
>>>>> >> >> >> >> > and
>>>>> >> >> >> >> > month=10
>>>>> >> >> >> >> > User: nira
>>>>> >> >> >> >> > Pool: root.impreports
>>>>> >> >> >> >> >
>>>>> >> >> >> >> > As shown above that the queries submitted based on the
>>>>> >> >> >> >> > secondary
>>>>> >> >> >> >> > group
>>>>> >> >> >> >> > rule.
>>>>> >> >> >> >> >
>>>>> >> >> >> >> > When i'm running the following with fawzea user:
>>>>> >> >> >> >> >
>>>>> >> >> >> >> > set request_pool=impreports;
>>>>> >> >> >> >> > select count(*) from dv_entersiteevent_prq_remote where
>>>>> >> >> >> >> > year=2017
>>>>> >> >> >> >> > and
>>>>> >> >> >> >> > month=10;
>>>>> >> >> >> >> >
>>>>> >> >> >> >> > I can see that the query is running with impreports pool.
>>>>> >> >> >> >> >
>>>>> >> >> >> >> >
>>>>> >> >> >> >> > What i'm missing here in my configuration to get this
>>>>> works as
>>>>> >> >> >> >> > expected.
>>>>> >> >> >> >> >
>>>>> >> >> >
>>>>> >> >> >
>>>>> >> >
>>>>> >> >
>>>>> >
>>>>> >
>>>>>
>>>>
>>>>
>>>
>>
>
Re: impala submission control
Posted by Fawze Abujaber <fa...@gmail.com>.
Sorry, but i'm able to replicate the issue on impala
version: 2.10.0-cdh5.13.0
I will be happy if anyone who can try to replicate the issue on his test
cluster.
On Thu, Feb 8, 2018 at 8:25 PM, Matthew Jacobs <ja...@gmail.com> wrote:
> I just filed one: https://issues.apache.org/jira/browse/IMPALA-6493
>
> On Thu, Feb 8, 2018 at 10:12 AM, Matthew Jacobs <ja...@gmail.com>
> wrote:
>
>> As I said, I don't see anything wrong with the configuration so there
>> could be a bug in Impala 2.11, likely due to IMPALA-5920. Can you file a
>> JIRA?
>>
>> On Thu, Feb 8, 2018 at 9:42 AM, Fawze Abujaber <fa...@gmail.com> wrote:
>>
>>> This is not fitting my situation,
>>>
>>> I removed even the placement role of the secondary group, and other
>>> users can still submit to the pool that i restrict it to my user.
>>>
>>> Just thinking in loud and think the problem might be in using the
>>> placement rule:
>>>
>>> Use the pool Specified at run time., only if the pool exists.
>>> Edit
>>>
>>>
>>> as it it the first rule, it always successes.
>>>
>>>
>>> On Thu, Feb 8, 2018 at 7:12 PM, Matthew Jacobs <ja...@gmail.com>
>>> wrote:
>>>
>>>> The aclSubmitApps can specify users or groups, so if the user name is
>>>> also a group name, that can lead to confusion, e.g.
>>>> https://issues.apache.org/jira/browse/IMPALA-2578
>>>> I'm not saying that is necessarily the problem, but something to
>>>> check. If we still can't find anything configured incorrectly I'd
>>>> recommend that you file a JIRA.
>>>>
>>>> On Thu, Feb 8, 2018 at 9:05 AM, Fawze Abujaber <fa...@gmail.com>
>>>> wrote:
>>>> > Hi Matt,
>>>> >
>>>> > If this is the situation, how we can explain that when i submit the
>>>> query it
>>>> > runs on my secondary group pool and when nira do the same it runs
>>>> under her
>>>> > secondary group pool.
>>>> >
>>>> > More than this, the issue not related to LDAP group as i'm using the
>>>> > submission restriction to a user and not a group.
>>>> >
>>>> > On Thu, Feb 8, 2018 at 7:00 PM, Matthew Jacobs <ja...@gmail.com>
>>>> wrote:
>>>> >>
>>>> >> As far as I can tell from this information, your configuration looks
>>>> >> fine. The only other potential issue I can think of is that the
>>>> >> secondary groups placement rule isn't picking up your LDAP groups
>>>> >> properly. Does the ACL behavior work if you remove the secondary
>>>> >> groups rule?
>>>> >>
>>>> >> On Wed, Feb 7, 2018 at 10:09 PM, Fawze Abujaber <fa...@gmail.com>
>>>> wrote:
>>>> >> > Hi Matt,
>>>> >> >
>>>> >> > v2.11.0-cdh5.14.0
>>>> >> >
>>>> >> > Query Runtime Profile:
>>>> >> > Query (id=c843dae48ce9b289:6ad3ffc000000000):
>>>> >> > Summary:
>>>> >> > Session ID: ca420ac477772b56:51ae460ce9e44999
>>>> >> > Session Type: BEESWAX
>>>> >> > Start Time: 2018-02-08 01:02:01.023558000
>>>> >> > End Time: 2018-02-08 01:02:01.501506000
>>>> >> > Query Type: QUERY
>>>> >> > Query State: FINISHED
>>>> >> > Query Status: OK
>>>> >> > Impala Version: impalad version 2.11.0-cdh5.14.0 RELEASE (build
>>>> >> > d68206561bce6b26762d62c01a78e6cd27aa7690)
>>>> >> > User: fawzea
>>>> >> > Connected User: fawzea
>>>> >> > Delegated User:
>>>> >> > Network Address: ::ffff:10.16.144.131:50078
>>>> >> > Default Db: default
>>>> >> > Sql Statement: select count(*) from
>>>> dv_entersiteevent_prq_remote
>>>> >> > where
>>>> >> > year=2017 and month=10
>>>> >> > Coordinator: aopr-dhc001.lpdomain.com:22000
>>>> >> > Query Options (set by configuration): REQUEST_POOL=impreports
>>>> >> > Query Options (set by configuration and planner):
>>>> >> > REQUEST_POOL=impreports,MT_DOP=0
>>>> >> > Plan:
>>>> >> > ----------------
>>>> >> > Max Per-Host Resource Reservation: Memory=0B
>>>> >> > Per-Host Resource Estimates: Memory=21.00MB
>>>> >> > WARNING: The following tables are missing relevant table and/or
>>>> column
>>>> >> > statistics.
>>>> >> > default.dv_entersiteevent_prq_remote
>>>> >> >
>>>> >> > F01:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1
>>>> >> > | Per-Host Resources: mem-estimate=10.00MB mem-reservation=0B
>>>> >> > PLAN-ROOT SINK
>>>> >> > | mem-estimate=0B mem-reservation=0B
>>>> >> > |
>>>> >> > 03:AGGREGATE [FINALIZE]
>>>> >> > | output: count:merge(*)
>>>> >> > | mem-estimate=10.00MB mem-reservation=0B spill-buffer=2.00MB
>>>> >> > | tuple-ids=1 row-size=8B cardinality=1
>>>> >> > |
>>>> >> > 02:EXCHANGE [UNPARTITIONED]
>>>> >> > | mem-estimate=0B mem-reservation=0B
>>>> >> > | tuple-ids=1 row-size=8B cardinality=1
>>>> >> > |
>>>> >> > F00:PLAN FRAGMENT [RANDOM] hosts=5 instances=5
>>>> >> > Per-Host Resources: mem-estimate=11.00MB mem-reservation=0B
>>>> >> > 01:AGGREGATE
>>>> >> > | output:
>>>> >> > sum_init_zero(default.dv_entersiteevent_prq_remote.parquet-stats:
>>>> >> > num_rows)
>>>> >> > | mem-estimate=10.00MB mem-reservation=0B spill-buffer=2.00MB
>>>> >> > | tuple-ids=1 row-size=8B cardinality=1
>>>> >> > |
>>>> >> > 00:SCAN HDFS [default.dv_entersiteevent_prq_remote, RANDOM]
>>>> >> > partitions=31/441 files=2292 size=1.39GB
>>>> >> > stats-rows=unavailable extrapolated-rows=disabled
>>>> >> > table stats: rows=unavailable size=unavailable
>>>> >> > column stats: all
>>>> >> > mem-estimate=1.00MB mem-reservation=0B
>>>> >> > tuple-ids=0 row-size=8B cardinality=unavailable
>>>> >> > ----------------
>>>> >> > Estimated Per-Host Mem: 22020096
>>>> >> > Tables Missing Stats: default.dv_entersiteevent_prq_remote
>>>> >> > Per Host Min Reservation: aopr-dhc001.lpdomain.com:22000(0)
>>>> >> > aopr-dhc002.lpdomain.com:22000(0) aopr-dhc003.lpdomain.com:22000
>>>> (0)
>>>> >> > aopr-dhc004.lpdomain.com:22000(0) aopr-dhc005.lpdomain.com:22000
>>>> (0)
>>>> >> > Request Pool: root.impreports
>>>> >> > Admission result: Admitted immediately
>>>> >> > ExecSummary:
>>>> >> > Operator #Hosts Avg Time Max Time #Rows Est. #Rows
>>>> Peak Mem
>>>> >> > Est. Peak Mem Detail
>>>> >> >
>>>> >> > ------------------------------------------------------------
>>>> ------------------------------------------------------------------
>>>> >> > 03:AGGREGATE 1 222.958us 222.958us 1 1
>>>> 36.00 KB
>>>> >> > 10.00 MB FINALIZE
>>>> >> > 02:EXCHANGE 1 357.508ms 357.508ms 5 1
>>>> 0
>>>> >> > 0 UNPARTITIONED
>>>> >> > 01:AGGREGATE 5 1.027ms 1.541ms 5 1
>>>> 20.00 KB
>>>> >> > 10.00 MB
>>>> >> > 00:SCAN HDFS 5 282.797ms 333.715ms 2.29K -1
>>>> 2.62 MB
>>>> >> > 1.00 MB default.dv_entersiteevent_prq_remote
>>>> >> > Errors:
>>>> >> > Planner Timeline: 13.508ms
>>>> >> > - Analysis finished: 3.303ms (3.303ms)
>>>> >> > - Value transfer graph computed: 3.465ms (162.667us)
>>>> >> > - Single node plan created: 10.352ms (6.886ms)
>>>> >> > - Runtime filters computed: 10.677ms (325.364us)
>>>> >> > - Distributed plan created: 10.733ms (55.854us)
>>>> >> > - Lineage info computed: 10.898ms (164.937us)
>>>> >> > - Planning finished: 13.508ms (2.610ms)
>>>> >> > Query Timeline: 481.950ms
>>>> >> > - Query submitted: 79.971us (79.971us)
>>>> >> > - Planning finished: 33.947ms (33.867ms)
>>>> >> > - Submit for admission: 40.067ms (6.120ms)
>>>> >> > - Completed admission: 40.197ms (130.146us)
>>>> >> > - Ready to start on 5 backends: 40.905ms (707.852us)
>>>> >> > - All 5 execution backends (6 fragment instances) started:
>>>> >> > 46.417ms
>>>> >> > (5.512ms)
>>>> >> > - Rows available: 424.548ms (378.131ms)
>>>> >> > - First row fetched: 473.488ms (48.939ms)
>>>> >> > - Unregister query: 477.951ms (4.462ms)
>>>> >> > - ComputeScanRangeAssignmentTimer: 3.980ms
>>>> >> > ImpalaServer:
>>>> >> > - ClientFetchWaitTimer: 51.659ms
>>>> >> > - RowMaterializationTimer: 1.726ms
>>>> >> > Execution Profile c843dae48ce9b289:6ad3ffc000000000:(Total:
>>>> 385.029ms,
>>>> >> > non-child: 0.000ns, % non-child: 0.00%)
>>>> >> > Number of filters: 0
>>>> >> > Filter routing table:
>>>> >> > ID Src. Node Tgt. Node(s) Target type Partition filter
>>>> Pending
>>>> >> > (Expected) First arrived Completed Enabled
>>>> >> >
>>>> >> > ------------------------------------------------------------
>>>> -------------------------------------------------------
>>>> >> >
>>>> >> > Backend startup latencies: Count: 5, min / max: 3ms / 4ms, 25th
>>>> >> > %-ile:
>>>> >> > 3ms, 50th %-ile: 4ms, 75th %-ile: 4ms, 90th %-ile: 4ms, 95th
>>>> %-ile: 4ms,
>>>> >> > 99.9th %-ile: 4ms
>>>> >> > Per Node Peak Memory Usage: aopr-dhc001.lpdomain.com:22000(2.68
>>>> MB)
>>>> >> > aopr-dhc002.lpdomain.com:22000(2.54 MB)
>>>> >> > aopr-dhc003.lpdomain.com:22000(2.54
>>>> >> > MB) aopr-dhc004.lpdomain.com:22000(2.41 MB)
>>>> >> > aopr-dhc005.lpdomain.com:22000(2.56 MB)
>>>> >> > - FiltersReceived: 0 (0)
>>>> >> > - FinalizationTimer: 0.000ns
>>>> >> > - NumBackends: 5 (5)
>>>> >> > - NumFragmentInstances: 6 (6)
>>>> >> > - NumFragments: 2 (2)
>>>> >> > Averaged Fragment F01:(Total: 427.530ms, non-child: 49.428ms, %
>>>> >> > non-child: 11.56%)
>>>> >> > split sizes: min: 0, max: 0, avg: 0, stddev: 0
>>>> >> > completion times: min:428.693ms max:428.693ms mean:
>>>> 428.693ms
>>>> >> > stddev:0.000ns
>>>> >> > execution rates: min:0.00 /sec max:0.00 /sec mean:0.00 /sec
>>>> >> > stddev:0.00 /sec
>>>> >> > num instances: 1
>>>> >> > - AverageThreadTokens: 0.00
>>>> >> > - BloomFilterBytes: 0
>>>> >> > - PeakMemoryUsage: 44.02 KB (45072)
>>>> >> > - PeakReservation: 0
>>>> >> > - PeakUsedReservation: 0
>>>> >> > - PerHostPeakMemUsage: 2.68 MB (2806113)
>>>> >> > - RowsProduced: 1 (1)
>>>> >> > - TotalNetworkReceiveTime: 357.481ms
>>>> >> > - TotalNetworkSendTime: 0.000ns
>>>> >> > - TotalStorageWaitTime: 0.000ns
>>>> >> > - TotalThreadsInvoluntaryContextSwitches: 0 (0)
>>>> >> > - TotalThreadsTotalWallClockTime: 406.708ms
>>>> >> > - TotalThreadsSysTime: 0.000ns
>>>> >> > - TotalThreadsUserTime: 0.000ns
>>>> >> > - TotalThreadsVoluntaryContextSwitches: 7 (7)
>>>> >> > Fragment Instance Lifecycle Timings:
>>>> >> > - ExecTime: 49.055ms
>>>> >> > - ExecTreeExecTime: 4.251us
>>>> >> > - OpenTime: 357.663ms
>>>> >> > - ExecTreeOpenTime: 357.649ms
>>>> >> > - PrepareTime: 20.764ms
>>>> >> > - ExecTreePrepareTime: 81.759us
>>>> >> > PLAN_ROOT_SINK:
>>>> >> > - PeakMemoryUsage: 0
>>>> >> > CodeGen:(Total: 20.369ms, non-child: 20.369ms, % non-child:
>>>> >> > 100.00%)
>>>> >> > - CodegenTime: 0.000ns
>>>> >> > - CompileTime: 0.000ns
>>>> >> > - LoadTime: 0.000ns
>>>> >> > - ModuleBitcodeSize: 1.86 MB (1950880)
>>>> >> > - NumFunctions: 0 (0)
>>>> >> > - NumInstructions: 0 (0)
>>>> >> > - OptimizationTime: 0.000ns
>>>> >> > - PeakMemoryUsage: 0
>>>> >> > - PrepareTime: 19.685ms
>>>> >> > AGGREGATION_NODE (id=3):(Total: 357.731ms, non-child:
>>>> 222.958us, %
>>>> >> > non-child: 0.06%)
>>>> >> > - BuildTime: 7.368us
>>>> >> > - GetResultsTime: 0.000ns
>>>> >> > - HTResizeTime: 0.000ns
>>>> >> > - HashBuckets: 0 (0)
>>>> >> > - LargestPartitionPercent: 0 (0)
>>>> >> > - MaxPartitionLevel: 0 (0)
>>>> >> > - NumRepartitions: 0 (0)
>>>> >> > - PartitionsCreated: 0 (0)
>>>> >> > - PeakMemoryUsage: 36.00 KB (36864)
>>>> >> > - RowsRepartitioned: 0 (0)
>>>> >> > - RowsReturned: 1 (1)
>>>> >> > - RowsReturnedRate: 2.00 /sec
>>>> >> > - SpilledPartitions: 0 (0)
>>>> >> > EXCHANGE_NODE (id=2):(Total: 357.508ms, non-child:
>>>> 357.508ms, %
>>>> >> > non-child: 100.00%)
>>>> >> > - ConvertRowBatchTime: 8.881us
>>>> >> > - PeakMemoryUsage: 0
>>>> >> > - RowsReturned: 5 (5)
>>>> >> > - RowsReturnedRate: 13.00 /sec
>>>> >> > DataStreamReceiver:
>>>> >> > - BytesReceived: 80.00 B (80)
>>>> >> > - DeserializeRowBatchTimer: 24.165us
>>>> >> > - FirstBatchArrivalWaitTime: 277.194ms
>>>> >> > - PeakMemoryUsage: 4.02 KB (4112)
>>>> >> > - SendersBlockedTimer: 0.000ns
>>>> >> > - SendersBlockedTotalTimer(*): 0.000ns
>>>> >> > Coordinator Fragment F01:
>>>> >> > Instance c843dae48ce9b289:6ad3ffc000000000
>>>> >> > (host=aopr-dhc001.lpdomain.com:22000):(Total: 427.530ms,
>>>> non-child:
>>>> >> > 49.428ms, % non-child: 11.56%)
>>>> >> > - AverageThreadTokens: 0.00
>>>> >> > - BloomFilterBytes: 0
>>>> >> > - PeakMemoryUsage: 44.02 KB (45072)
>>>> >> > - PeakReservation: 0
>>>> >> > - PeakUsedReservation: 0
>>>> >> > - PerHostPeakMemUsage: 2.68 MB (2806113)
>>>> >> > - RowsProduced: 1 (1)
>>>> >> > - TotalNetworkReceiveTime: 357.481ms
>>>> >> > - TotalNetworkSendTime: 0.000ns
>>>> >> > - TotalStorageWaitTime: 0.000ns
>>>> >> > - TotalThreadsInvoluntaryContextSwitches: 0 (0)
>>>> >> > - TotalThreadsTotalWallClockTime: 406.708ms
>>>> >> > - TotalThreadsSysTime: 0.000ns
>>>> >> > - TotalThreadsUserTime: 0.000ns
>>>> >> > - TotalThreadsVoluntaryContextSwitches: 7 (7)
>>>> >> > Fragment Instance Lifecycle Timings:
>>>> >> > - ExecTime: 49.055ms
>>>> >> > - ExecTreeExecTime: 4.251us
>>>> >> > - OpenTime: 357.663ms
>>>> >> > - ExecTreeOpenTime: 357.649ms
>>>> >> > - PrepareTime: 20.764ms
>>>> >> > - ExecTreePrepareTime: 81.759us
>>>> >> > PLAN_ROOT_SINK:
>>>> >> > - PeakMemoryUsage: 0
>>>> >> > CodeGen:(Total: 20.369ms, non-child: 20.369ms, % non-child:
>>>> >> > 100.00%)
>>>> >> > - CodegenTime: 0.000ns
>>>> >> > - CompileTime: 0.000ns
>>>> >> > - LoadTime: 0.000ns
>>>> >> > - ModuleBitcodeSize: 1.86 MB (1950880)
>>>> >> > - NumFunctions: 0 (0)
>>>> >> > - NumInstructions: 0 (0)
>>>> >> > - OptimizationTime: 0.000ns
>>>> >> > - PeakMemoryUsage: 0
>>>> >> > - PrepareTime: 19.685ms
>>>> >> > AGGREGATION_NODE (id=3):(Total: 357.731ms, non-child:
>>>> 222.958us,
>>>> >> > %
>>>> >> > non-child: 0.06%)
>>>> >> > - BuildTime: 7.368us
>>>> >> > - GetResultsTime: 0.000ns
>>>> >> > - HTResizeTime: 0.000ns
>>>> >> > - HashBuckets: 0 (0)
>>>> >> > - LargestPartitionPercent: 0 (0)
>>>> >> > - MaxPartitionLevel: 0 (0)
>>>> >> > - NumRepartitions: 0 (0)
>>>> >> > - PartitionsCreated: 0 (0)
>>>> >> > - PeakMemoryUsage: 36.00 KB (36864)
>>>> >> > - RowsRepartitioned: 0 (0)
>>>> >> > - RowsReturned: 1 (1)
>>>> >> > - RowsReturnedRate: 2.00 /sec
>>>> >> > - SpilledPartitions: 0 (0)
>>>> >> > EXCHANGE_NODE (id=2):(Total: 357.508ms, non-child:
>>>> 357.508ms, %
>>>> >> > non-child: 100.00%)
>>>> >> > - ConvertRowBatchTime: 8.881us
>>>> >> > - PeakMemoryUsage: 0
>>>> >> > - RowsReturned: 5 (5)
>>>> >> > - RowsReturnedRate: 13.00 /sec
>>>> >> > DataStreamReceiver:
>>>> >> > - BytesReceived: 80.00 B (80)
>>>> >> > - DeserializeRowBatchTimer: 24.165us
>>>> >> > - FirstBatchArrivalWaitTime: 277.194ms
>>>> >> > - PeakMemoryUsage: 4.02 KB (4112)
>>>> >> > - SendersBlockedTimer: 0.000ns
>>>> >> > - SendersBlockedTotalTimer(*): 0.000ns
>>>> >> > Averaged Fragment F00:(Total: 327.360ms, non-child: 1.578ms, %
>>>> >> > non-child: 0.48%)
>>>> >> > split sizes: min: 283.31 MB, max: 284.33 MB, avg: 284.04 MB,
>>>> >> > stddev:
>>>> >> > 382.00 KB
>>>> >> > completion times: min:299.457ms max:378.950ms mean:
>>>> 328.881ms
>>>> >> > stddev:31.507ms
>>>> >> > execution rates: min:750.18 MB/sec max:949.01 MB/sec
>>>> mean:871.27
>>>> >> > MB/sec stddev:79.74 MB/sec
>>>> >> > num instances: 5
>>>> >> > - AverageThreadTokens: 15.00
>>>> >> > - BloomFilterBytes: 0
>>>> >> > - PeakMemoryUsage: 2.54 MB (2666852)
>>>> >> > - PeakReservation: 0
>>>> >> > - PeakUsedReservation: 0
>>>> >> > - PerHostPeakMemUsage: 2.55 MB (2669309)
>>>> >> > - RowsProduced: 1 (1)
>>>> >> > - TotalNetworkReceiveTime: 0.000ns
>>>> >> > - TotalNetworkSendTime: 303.541us
>>>> >> > - TotalStorageWaitTime: 4s821ms
>>>> >> > - TotalThreadsInvoluntaryContextSwitches: 104 (104)
>>>> >> > - TotalThreadsTotalWallClockTime: 6s835ms
>>>> >> > - TotalThreadsSysTime: 31.784ms
>>>> >> > - TotalThreadsUserTime: 682.084ms
>>>> >> > - TotalThreadsVoluntaryContextSwitches: 2.12K (2121)
>>>> >> > Fragment Instance Lifecycle Timings:
>>>> >> > - ExecTime: 500.816us
>>>> >> > - ExecTreeExecTime: 8.200us
>>>> >> > - OpenTime: 302.848ms
>>>> >> > - ExecTreeOpenTime: 281.309ms
>>>> >> > - PrepareTime: 23.946ms
>>>> >> > - ExecTreePrepareTime: 2.513ms
>>>> >> > DataStreamSender (dst_id=2):(Total: 269.902us, non-child:
>>>> >> > 269.902us, %
>>>> >> > non-child: 100.00%)
>>>> >> > - BytesSent: 16.00 B (16)
>>>> >> > - NetworkThroughput(*): 373.13 KB/sec
>>>> >> > - OverallThroughput: 63.53 KB/sec
>>>> >> > - PeakMemoryUsage: 16.00 KB (16384)
>>>> >> > - RowsReturned: 1 (1)
>>>> >> > - SerializeBatchTime: 14.525us
>>>> >> > - TransmitDataRPCTime: 43.122us
>>>> >> > - UncompressedRowBatchSize: 16.00 B (16)
>>>> >> > CodeGen:(Total: 41.687ms, non-child: 41.687ms, % non-child:
>>>> >> > 100.00%)
>>>> >> > - CodegenTime: 1.194ms
>>>> >> > - CompileTime: 7.360ms
>>>> >> > - LoadTime: 0.000ns
>>>> >> > - ModuleBitcodeSize: 1.86 MB (1950880)
>>>> >> > - NumFunctions: 15 (15)
>>>> >> > - NumInstructions: 214 (214)
>>>> >> > - OptimizationTime: 13.529ms
>>>> >> > - PeakMemoryUsage: 107.00 KB (109568)
>>>> >> > - PrepareTime: 20.084ms
>>>> >> > AGGREGATION_NODE (id=1):(Total: 283.825ms, non-child:
>>>> 1.027ms, %
>>>> >> > non-child: 0.36%)
>>>> >> > - BuildTime: 1.259ms
>>>> >> > - GetResultsTime: 0.000ns
>>>> >> > - HTResizeTime: 0.000ns
>>>> >> > - HashBuckets: 0 (0)
>>>> >> > - LargestPartitionPercent: 0 (0)
>>>> >> > - MaxPartitionLevel: 0 (0)
>>>> >> > - NumRepartitions: 0 (0)
>>>> >> > - PartitionsCreated: 0 (0)
>>>> >> > - PeakMemoryUsage: 20.00 KB (20480)
>>>> >> > - RowsRepartitioned: 0 (0)
>>>> >> > - RowsReturned: 1 (1)
>>>> >> > - RowsReturnedRate: 2.00 /sec
>>>> >> > - SpilledPartitions: 0 (0)
>>>> >> > HDFS_SCAN_NODE (id=0):(Total: 282.797ms, non-child:
>>>> 282.797ms, %
>>>> >> > non-child: 100.00%)
>>>> >> > - AverageHdfsReadThreadConcurrency: 0.00
>>>> >> > - AverageScannerThreadConcurrency: 14.40
>>>> >> > - BytesRead: 44.52 MB (46686028)
>>>> >> > - BytesReadDataNodeCache: 0
>>>> >> > - BytesReadLocal: 44.52 MB (46686028)
>>>> >> > - BytesReadRemoteUnexpected: 0
>>>> >> > - BytesReadShortCircuit: 44.52 MB (46686028)
>>>> >> > - CachedFileHandlesHitCount: 0 (0)
>>>> >> > - CachedFileHandlesMissCount: 458 (458)
>>>> >> > - CollectionItemsRead: 0 (0)
>>>> >> > - DecompressionTime: 0.000ns
>>>> >> > - MaxCompressedTextFileLength: 0
>>>> >> > - NumColumns: 0 (0)
>>>> >> > - NumDictFilteredRowGroups: 0 (0)
>>>> >> > - NumDisksAccessed: 12 (12)
>>>> >> > - NumRowGroups: 458 (458)
>>>> >> > - NumScannerThreadsStarted: 24 (24)
>>>> >> > - NumScannersWithNoReads: 0 (0)
>>>> >> > - NumStatsFilteredRowGroups: 0 (0)
>>>> >> > - PeakMemoryUsage: 2.50 MB (2625539)
>>>> >> > - PerReadThreadRawHdfsThroughput: 1.29 GB/sec
>>>> >> > - RemoteScanRanges: 0 (0)
>>>> >> > - RowBatchQueueGetWaitTime: 269.787ms
>>>> >> > - RowBatchQueuePutWaitTime: 0.000ns
>>>> >> > - RowsRead: 0 (0)
>>>> >> > - RowsReturned: 458 (458)
>>>> >> > - RowsReturnedRate: 1.64 K/sec
>>>> >> > - ScanRangesComplete: 458 (458)
>>>> >> > - ScannerThreadsInvoluntaryContextSwitches: 99 (99)
>>>> >> > - ScannerThreadsTotalWallClockTime: 6s532ms
>>>> >> > - MaterializeTupleTime(*): 70.664us
>>>> >> > - ScannerThreadsSysTime: 26.184ms
>>>> >> > - ScannerThreadsUserTime: 650.089ms
>>>> >> > - ScannerThreadsVoluntaryContextSwitches: 1.67K (1674)
>>>> >> > - TotalRawHdfsReadTime(*): 34.416ms
>>>> >> > - TotalReadThroughput: 36.86 MB/sec
>>>> >> > Fragment F00:
>>>> >> > Instance c843dae48ce9b289:6ad3ffc000000004
>>>> >> > (host=aopr-dhc004.lpdomain.com:22000):(Total: 377.384ms,
>>>> non-child:
>>>> >> > 1.619ms,
>>>> >> > % non-child: 0.43%)
>>>> >> > Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>>>> >> > 0:36/21.95 MB 11:33
>>>> <https://maps.google.com/?q=MB+11:33&entry=gmail&source=g>/19.87 MB
>>>> 9:27 <https://maps.google.com/?q=MB+9:27&entry=gmail&source=g>/15.16 MB
>>>> 5:44 <https://maps.google.com/?q=MB+5:44&entry=gmail&source=g>/28.07 MB
>>>> 6:56 <https://maps.google.com/?q=MB+6:56&entry=gmail&source=g>/33.32 MB
>>>> >> > 1:38/22.24 MB 10:44
>>>> <https://maps.google.com/?q=MB+10:44&entry=gmail&source=g>/27.90 MB
>>>> 3:34 <https://maps.google.com/?q=MB+3:34&entry=gmail&source=g>/20.26 MB
>>>> 2:45 <https://maps.google.com/?q=MB+2:45&entry=gmail&source=g>/28.21 MB
>>>> 7:31 <https://maps.google.com/?q=MB+7:31&entry=gmail&source=g>/19.33 MB
>>>> >> > 4:42/26.73 MB 8:34
>>>> <https://maps.google.com/?q=MB+8:34&entry=gmail&source=g>/21.25 MB
>>>> >> > MemoryUsage(500.000ms): 1.91 MB
>>>> >> > ThreadUsage(500.000ms): 25
>>>> >> > - AverageThreadTokens: 25.00
>>>> >> > - BloomFilterBytes: 0
>>>> >> > - PeakMemoryUsage: 2.41 MB (2531689)
>>>> >> > - PeakReservation: 0
>>>> >> > - PeakUsedReservation: 0
>>>> >> > - PerHostPeakMemUsage: 2.41 MB (2531689)
>>>> >> > - RowsProduced: 1 (1)
>>>> >> > - TotalNetworkReceiveTime: 0.000ns
>>>> >> > - TotalNetworkSendTime: 382.629us
>>>> >> > - TotalStorageWaitTime: 5s790ms
>>>> >> > - TotalThreadsInvoluntaryContextSwitches: 115 (115)
>>>> >> > - TotalThreadsTotalWallClockTime: 8s122ms
>>>> >> > - TotalThreadsSysTime: 33.984ms
>>>> >> > - TotalThreadsUserTime: 702.879ms
>>>> >> > - TotalThreadsVoluntaryContextSwitches: 2.04K (2037)
>>>> >> > Fragment Instance Lifecycle Timings:
>>>> >> > - ExecTime: 604.005us
>>>> >> > - ExecTreeExecTime: 8.338us
>>>> >> > - OpenTime: 352.927ms
>>>> >> > - ExecTreeOpenTime: 331.921ms
>>>> >> > - PrepareTime: 23.783ms
>>>> >> > - ExecTreePrepareTime: 2.559ms
>>>> >> > DataStreamSender (dst_id=2):(Total: 331.818us, non-child:
>>>> >> > 331.818us,
>>>> >> > % non-child: 100.00%)
>>>> >> > - BytesSent: 16.00 B (16)
>>>> >> > - NetworkThroughput(*): 289.00 KB/sec
>>>> >> > - OverallThroughput: 47.09 KB/sec
>>>> >> > - PeakMemoryUsage: 16.00 KB (16384)
>>>> >> > - RowsReturned: 1 (1)
>>>> >> > - SerializeBatchTime: 14.466us
>>>> >> > - TransmitDataRPCTime: 54.066us
>>>> >> > - UncompressedRowBatchSize: 16.00 B (16)
>>>> >> > CodeGen:(Total: 40.950ms, non-child: 40.950ms, % non-child:
>>>> >> > 100.00%)
>>>> >> > - CodegenTime: 1.122ms
>>>> >> > - CompileTime: 7.318ms
>>>> >> > - LoadTime: 0.000ns
>>>> >> > - ModuleBitcodeSize: 1.86 MB (1950880)
>>>> >> > - NumFunctions: 15 (15)
>>>> >> > - NumInstructions: 214 (214)
>>>> >> > - OptimizationTime: 13.106ms
>>>> >> > - PeakMemoryUsage: 107.00 KB (109568)
>>>> >> > - PrepareTime: 19.773ms
>>>> >> > AGGREGATION_NODE (id=1):(Total: 334.482ms, non-child:
>>>> 766.527us,
>>>> >> > %
>>>> >> > non-child: 0.23%)
>>>> >> > ExecOption: Codegen Enabled
>>>> >> > - BuildTime: 1.192ms
>>>> >> > - GetResultsTime: 0.000ns
>>>> >> > - HTResizeTime: 0.000ns
>>>> >> > - HashBuckets: 0 (0)
>>>> >> > - LargestPartitionPercent: 0 (0)
>>>> >> > - MaxPartitionLevel: 0 (0)
>>>> >> > - NumRepartitions: 0 (0)
>>>> >> > - PartitionsCreated: 0 (0)
>>>> >> > - PeakMemoryUsage: 20.00 KB (20480)
>>>> >> > - RowsRepartitioned: 0 (0)
>>>> >> > - RowsReturned: 1 (1)
>>>> >> > - RowsReturnedRate: 2.00 /sec
>>>> >> > - SpilledPartitions: 0 (0)
>>>> >> > HDFS_SCAN_NODE (id=0):(Total: 333.715ms, non-child:
>>>> 333.715ms, %
>>>> >> > non-child: 100.00%)
>>>> >> > Hdfs split stats (<volume id>:<# splits>/<split
>>>> lengths>):
>>>> >> > 0:36/21.95 MB 11:33
>>>> <https://maps.google.com/?q=MB+11:33&entry=gmail&source=g>/19.87 MB
>>>> 9:27 <https://maps.google.com/?q=MB+9:27&entry=gmail&source=g>/15.16 MB
>>>> 5:44 <https://maps.google.com/?q=MB+5:44&entry=gmail&source=g>/28.07 MB
>>>> 6:56 <https://maps.google.com/?q=MB+6:56&entry=gmail&source=g>/33.32 MB
>>>> >> > 1:38/22.24 MB 10:44
>>>> <https://maps.google.com/?q=MB+10:44&entry=gmail&source=g>/27.90 MB
>>>> 3:34 <https://maps.google.com/?q=MB+3:34&entry=gmail&source=g>/20.26 MB
>>>> 2:45 <https://maps.google.com/?q=MB+2:45&entry=gmail&source=g>/28.21 MB
>>>> 7:31 <https://maps.google.com/?q=MB+7:31&entry=gmail&source=g>/19.33 MB
>>>> >> > 4:42/26.73 MB 8:34
>>>> <https://maps.google.com/?q=MB+8:34&entry=gmail&source=g>/21.25 MB
>>>> >> > ExecOption: PARQUET Codegen Enabled, Codegen enabled:
>>>> 464 out
>>>> >> > of
>>>> >> > 464
>>>> >> > Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0%
>>>> 3:0%
>>>> >> > 4:0%
>>>> >> > 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
>>>> >> > File Formats: PARQUET/Unknown(Skipped):464
>>>> >> > BytesRead(500.000ms): 12.27 MB
>>>> >> > - FooterProcessingTime: (Avg: 13.555ms ; Min: 5.433ms ;
>>>> Max:
>>>> >> > 62.957ms ; Number of samples: 464)
>>>> >> > - AverageHdfsReadThreadConcurrency: 0.00
>>>> >> > - AverageScannerThreadConcurrency: 24.00
>>>> >> > - BytesRead: 45.01 MB (47193988)
>>>> >> > - BytesReadDataNodeCache: 0
>>>> >> > - BytesReadLocal: 45.01 MB (47193988)
>>>> >> > - BytesReadRemoteUnexpected: 0
>>>> >> > - BytesReadShortCircuit: 45.01 MB (47193988)
>>>> >> > - CachedFileHandlesHitCount: 0 (0)
>>>> >> > - CachedFileHandlesMissCount: 464 (464)
>>>> >> > - CollectionItemsRead: 0 (0)
>>>> >> > - DecompressionTime: 0.000ns
>>>> >> > - MaxCompressedTextFileLength: 0
>>>> >> > - NumColumns: 0 (0)
>>>> >> > - NumDictFilteredRowGroups: 0 (0)
>>>> >> > - NumDisksAccessed: 12 (12)
>>>> >> > - NumRowGroups: 464 (464)
>>>> >> > - NumScannerThreadsStarted: 24 (24)
>>>> >> > - NumScannersWithNoReads: 0 (0)
>>>> >> > - NumStatsFilteredRowGroups: 0 (0)
>>>> >> > - PeakMemoryUsage: 2.38 MB (2490376)
>>>> >> > - PerReadThreadRawHdfsThroughput: 1016.44 MB/sec
>>>> >> > - RemoteScanRanges: 0 (0)
>>>> >> > - RowBatchQueueGetWaitTime: 320.860ms
>>>> >> > - RowBatchQueuePutWaitTime: 0.000ns
>>>> >> > - RowsRead: 0 (0)
>>>> >> > - RowsReturned: 464 (464)
>>>> >> > - RowsReturnedRate: 1.39 K/sec
>>>> >> > - ScanRangesComplete: 464 (464)
>>>> >> > - ScannerThreadsInvoluntaryContextSwitches: 108 (108)
>>>> >> > - ScannerThreadsTotalWallClockTime: 7s769ms
>>>> >> > - MaterializeTupleTime(*): 75.514us
>>>> >> > - ScannerThreadsSysTime: 29.984ms
>>>> >> > - ScannerThreadsUserTime: 669.884ms
>>>> >> > - ScannerThreadsVoluntaryContextSwitches: 1.60K (1597)
>>>> >> > - TotalRawHdfsReadTime(*): 44.279ms
>>>> >> > - TotalReadThroughput: 24.53 MB/sec
>>>> >> > Instance c843dae48ce9b289:6ad3ffc000000002
>>>> >> > (host=aopr-dhc003.lpdomain.com:22000):(Total: 351.014ms,
>>>> non-child:
>>>> >> > 1.641ms,
>>>> >> > % non-child: 0.47%)
>>>> >> > Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>>>> >> > 0:35/20.89 MB 7:44
>>>> <https://maps.google.com/?q=MB+7:44&entry=gmail&source=g>/26.64 MB
>>>> 11:27 <https://maps.google.com/?q=MB+11:27&entry=gmail&source=g>/17.21 MB
>>>> 9:32 <https://maps.google.com/?q=MB+9:32&entry=gmail&source=g>/18.61 MB
>>>> 5:25 <https://maps.google.com/?q=MB+5:25&entry=gmail&source=g>/14.56 MB
>>>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+2:38&entry=gmail&source=g>
>>>> >> > 2:38
>>>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+2:38&entry=gmail&source=g>/24.32
>>>> MB 6:31 <https://maps.google.com/?q=MB+6:31&entry=gmail&source=g>/19.85
>>>> MB 8:50 <https://maps.google.com/?q=MB+8:50&entry=gmail&source=g>/30.76
>>>> MB 10:41 <https://maps.google.com/?q=MB+10:41&entry=gmail&source=g>/25.92
>>>> MB 3:31 <https://maps.google.com/?q=MB+3:31&entry=gmail&source=g>/19.78
>>>> MB
>>>> >> > 4:60/37.59 MB 1:44/27.19 MB
>>>> >> > MemoryUsage(500.000ms): 1.16 MB
>>>> >> > ThreadUsage(500.000ms): 25
>>>> >> > - AverageThreadTokens: 25.00
>>>> >> > - BloomFilterBytes: 0
>>>> >> > - PeakMemoryUsage: 2.54 MB (2662753)
>>>> >> > - PeakReservation: 0
>>>> >> > - PeakUsedReservation: 0
>>>> >> > - PerHostPeakMemUsage: 2.54 MB (2662753)
>>>> >> > - RowsProduced: 1 (1)
>>>> >> > - TotalNetworkReceiveTime: 0.000ns
>>>> >> > - TotalNetworkSendTime: 342.746us
>>>> >> > - TotalStorageWaitTime: 4s796ms
>>>> >> > - TotalThreadsInvoluntaryContextSwitches: 93 (93)
>>>> >> > - TotalThreadsTotalWallClockTime: 7s286ms
>>>> >> > - TotalThreadsSysTime: 27.988ms
>>>> >> > - TotalThreadsUserTime: 671.887ms
>>>> >> > - TotalThreadsVoluntaryContextSwitches: 2.08K (2079)
>>>> >> > Fragment Instance Lifecycle Timings:
>>>> >> > - ExecTime: 524.006us
>>>> >> > - ExecTreeExecTime: 7.920us
>>>> >> > - OpenTime: 327.802ms
>>>> >> > - ExecTreeOpenTime: 307.154ms
>>>> >> > - PrepareTime: 22.617ms
>>>> >> > - ExecTreePrepareTime: 2.650ms
>>>> >> > DataStreamSender (dst_id=2):(Total: 348.990us, non-child:
>>>> >> > 348.990us,
>>>> >> > % non-child: 100.00%)
>>>> >> > - BytesSent: 16.00 B (16)
>>>> >> > - NetworkThroughput(*): 354.47 KB/sec
>>>> >> > - OverallThroughput: 44.77 KB/sec
>>>> >> > - PeakMemoryUsage: 16.00 KB (16384)
>>>> >> > - RowsReturned: 1 (1)
>>>> >> > - SerializeBatchTime: 15.602us
>>>> >> > - TransmitDataRPCTime: 44.080us
>>>> >> > - UncompressedRowBatchSize: 16.00 B (16)
>>>> >> > CodeGen:(Total: 39.215ms, non-child: 39.215ms, % non-child:
>>>> >> > 100.00%)
>>>> >> > - CodegenTime: 1.183ms
>>>> >> > - CompileTime: 7.168ms
>>>> >> > - LoadTime: 0.000ns
>>>> >> > - ModuleBitcodeSize: 1.86 MB (1950880)
>>>> >> > - NumFunctions: 15 (15)
>>>> >> > - NumInstructions: 214 (214)
>>>> >> > - OptimizationTime: 12.929ms
>>>> >> > - PeakMemoryUsage: 107.00 KB (109568)
>>>> >> > - PrepareTime: 18.453ms
>>>> >> > AGGREGATION_NODE (id=1):(Total: 309.807ms, non-child:
>>>> 948.677us,
>>>> >> > %
>>>> >> > non-child: 0.31%)
>>>> >> > ExecOption: Codegen Enabled
>>>> >> > - BuildTime: 1.552ms
>>>> >> > - GetResultsTime: 0.000ns
>>>> >> > - HTResizeTime: 0.000ns
>>>> >> > - HashBuckets: 0 (0)
>>>> >> > - LargestPartitionPercent: 0 (0)
>>>> >> > - MaxPartitionLevel: 0 (0)
>>>> >> > - NumRepartitions: 0 (0)
>>>> >> > - PartitionsCreated: 0 (0)
>>>> >> > - PeakMemoryUsage: 20.00 KB (20480)
>>>> >> > - RowsRepartitioned: 0 (0)
>>>> >> > - RowsReturned: 1 (1)
>>>> >> > - RowsReturnedRate: 3.00 /sec
>>>> >> > - SpilledPartitions: 0 (0)
>>>> >> > HDFS_SCAN_NODE (id=0):(Total: 308.859ms, non-child:
>>>> 308.859ms, %
>>>> >> > non-child: 100.00%)
>>>> >> > Hdfs split stats (<volume id>:<# splits>/<split
>>>> lengths>):
>>>> >> > 0:35/20.89 MB 7:44
>>>> <https://maps.google.com/?q=MB+7:44&entry=gmail&source=g>/26.64 MB
>>>> 11:27 <https://maps.google.com/?q=MB+11:27&entry=gmail&source=g>/17.21 MB
>>>> 9:32 <https://maps.google.com/?q=MB+9:32&entry=gmail&source=g>/18.61 MB
>>>> 5:25 <https://maps.google.com/?q=MB+5:25&entry=gmail&source=g>/14.56 MB
>>>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+2:38&entry=gmail&source=g>
>>>> >> > 2:38
>>>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+2:38&entry=gmail&source=g>/24.32
>>>> MB 6:31 <https://maps.google.com/?q=MB+6:31&entry=gmail&source=g>/19.85
>>>> MB 8:50 <https://maps.google.com/?q=MB+8:50&entry=gmail&source=g>/30.76
>>>> MB 10:41 <https://maps.google.com/?q=MB+10:41&entry=gmail&source=g>/25.92
>>>> MB 3:31 <https://maps.google.com/?q=MB+3:31&entry=gmail&source=g>/19.78
>>>> MB
>>>> >> > 4:60/37.59 MB 1:44/27.19 MB
>>>> >> > ExecOption: PARQUET Codegen Enabled, Codegen enabled:
>>>> 458 out
>>>> >> > of
>>>> >> > 458
>>>> >> > Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0%
>>>> 3:0%
>>>> >> > 4:0%
>>>> >> > 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
>>>> >> > File Formats: PARQUET/Unknown(Skipped):458
>>>> >> > BytesRead(500.000ms): 36.74 MB
>>>> >> > - FooterProcessingTime: (Avg: 11.502ms ; Min: 3.668ms ;
>>>> Max:
>>>> >> > 23.564ms ; Number of samples: 458)
>>>> >> > - AverageHdfsReadThreadConcurrency: 0.00
>>>> >> > - AverageScannerThreadConcurrency: 24.00
>>>> >> > - BytesRead: 44.51 MB (46668432)
>>>> >> > - BytesReadDataNodeCache: 0
>>>> >> > - BytesReadLocal: 44.51 MB (46668432)
>>>> >> > - BytesReadRemoteUnexpected: 0
>>>> >> > - BytesReadShortCircuit: 44.51 MB (46668432)
>>>> >> > - CachedFileHandlesHitCount: 0 (0)
>>>> >> > - CachedFileHandlesMissCount: 458 (458)
>>>> >> > - CollectionItemsRead: 0 (0)
>>>> >> > - DecompressionTime: 0.000ns
>>>> >> > - MaxCompressedTextFileLength: 0
>>>> >> > - NumColumns: 0 (0)
>>>> >> > - NumDictFilteredRowGroups: 0 (0)
>>>> >> > - NumDisksAccessed: 12 (12)
>>>> >> > - NumRowGroups: 458 (458)
>>>> >> > - NumScannerThreadsStarted: 24 (24)
>>>> >> > - NumScannersWithNoReads: 0 (0)
>>>> >> > - NumStatsFilteredRowGroups: 0 (0)
>>>> >> > - PeakMemoryUsage: 2.50 MB (2621440)
>>>> >> > - PerReadThreadRawHdfsThroughput: 1.26 GB/sec
>>>> >> > - RemoteScanRanges: 0 (0)
>>>> >> > - RowBatchQueueGetWaitTime: 295.595ms
>>>> >> > - RowBatchQueuePutWaitTime: 0.000ns
>>>> >> > - RowsRead: 0 (0)
>>>> >> > - RowsReturned: 458 (458)
>>>> >> > - RowsReturnedRate: 1.48 K/sec
>>>> >> > - ScanRangesComplete: 458 (458)
>>>> >> > - ScannerThreadsInvoluntaryContextSwitches: 91 (91)
>>>> >> > - ScannerThreadsTotalWallClockTime: 6s958ms
>>>> >> > - MaterializeTupleTime(*): 71.630us
>>>> >> > - ScannerThreadsSysTime: 16.989ms
>>>> >> > - ScannerThreadsUserTime: 645.891ms
>>>> >> > - ScannerThreadsVoluntaryContextSwitches: 1.63K (1630)
>>>> >> > - TotalRawHdfsReadTime(*): 34.377ms
>>>> >> > - TotalReadThroughput: 73.49 MB/sec
>>>> >> > Instance c843dae48ce9b289:6ad3ffc000000001
>>>> >> > (host=aopr-dhc005.lpdomain.com:22000):(Total: 308.195ms,
>>>> non-child:
>>>> >> > 1.784ms,
>>>> >> > % non-child: 0.58%)
>>>> >> > Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>>>> >> > 2:46/28.97 MB 8:39
>>>> <https://maps.google.com/?q=MB+8:39&entry=gmail&source=g>/24.34 MB
>>>> 10:33 <https://maps.google.com/?q=MB+10:33&entry=gmail&source=g>/21.03 MB
>>>> 3:43 <https://maps.google.com/?q=MB+3:43&entry=gmail&source=g>/26.50 MB
>>>> 5:32 <https://maps.google.com/?q=MB+5:32&entry=gmail&source=g>/20.27 MB
>>>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+9:32&entry=gmail&source=g>
>>>> >> > 9:32
>>>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+9:32&entry=gmail&source=g>/19.86
>>>> MB 6:25 <https://maps.google.com/?q=MB+6:25&entry=gmail&source=g>/16.01
>>>> MB 7:41 <https://maps.google.com/?q=MB+7:41&entry=gmail&source=g>/24.77
>>>> MB 1:42/25.01 MB 0:55/32.51 MB
>>>> >> > 4:40/24.29 MB 11:33
>>>> <https://maps.google.com/?q=MB+11:33&entry=gmail&source=g>/20.77 MB
>>>> >> > - AverageThreadTokens: 0.00
>>>> >> > - BloomFilterBytes: 0
>>>> >> > - PeakMemoryUsage: 2.56 MB (2683241)
>>>> >> > - PeakReservation: 0
>>>> >> > - PeakUsedReservation: 0
>>>> >> > - PerHostPeakMemUsage: 2.56 MB (2683241)
>>>> >> > - RowsProduced: 1 (1)
>>>> >> > - TotalNetworkReceiveTime: 0.000ns
>>>> >> > - TotalNetworkSendTime: 292.765us
>>>> >> > - TotalStorageWaitTime: 4s482ms
>>>> >> > - TotalThreadsInvoluntaryContextSwitches: 113 (113)
>>>> >> > - TotalThreadsTotalWallClockTime: 6s310ms
>>>> >> > - TotalThreadsSysTime: 35.983ms
>>>> >> > - TotalThreadsUserTime: 682.884ms
>>>> >> > - TotalThreadsVoluntaryContextSwitches: 2.14K (2143)
>>>> >> > Fragment Instance Lifecycle Timings:
>>>> >> > - ExecTime: 532.572us
>>>> >> > - ExecTreeExecTime: 8.377us
>>>> >> > - OpenTime: 280.241ms
>>>> >> > - ExecTreeOpenTime: 258.652ms
>>>> >> > - PrepareTime: 27.342ms
>>>> >> > - ExecTreePrepareTime: 2.863ms
>>>> >> > DataStreamSender (dst_id=2):(Total: 309.794us, non-child:
>>>> >> > 309.794us,
>>>> >> > % non-child: 100.00%)
>>>> >> > - BytesSent: 16.00 B (16)
>>>> >> > - NetworkThroughput(*): 356.91 KB/sec
>>>> >> > - OverallThroughput: 50.44 KB/sec
>>>> >> > - PeakMemoryUsage: 16.00 KB (16384)
>>>> >> > - RowsReturned: 1 (1)
>>>> >> > - SerializeBatchTime: 14.262us
>>>> >> > - TransmitDataRPCTime: 43.778us
>>>> >> > - UncompressedRowBatchSize: 16.00 B (16)
>>>> >> > CodeGen:(Total: 44.584ms, non-child: 44.584ms, % non-child:
>>>> >> > 100.00%)
>>>> >> > - CodegenTime: 1.382ms
>>>> >> > - CompileTime: 7.346ms
>>>> >> > - LoadTime: 0.000ns
>>>> >> > - ModuleBitcodeSize: 1.86 MB (1950880)
>>>> >> > - NumFunctions: 15 (15)
>>>> >> > - NumInstructions: 214 (214)
>>>> >> > - OptimizationTime: 13.462ms
>>>> >> > - PeakMemoryUsage: 107.00 KB (109568)
>>>> >> > - PrepareTime: 23.006ms
>>>> >> > AGGREGATION_NODE (id=1):(Total: 261.516ms, non-child:
>>>> 836.656us,
>>>> >> > %
>>>> >> > non-child: 0.32%)
>>>> >> > ExecOption: Codegen Enabled
>>>> >> > - BuildTime: 1.280ms
>>>> >> > - GetResultsTime: 0.000ns
>>>> >> > - HTResizeTime: 0.000ns
>>>> >> > - HashBuckets: 0 (0)
>>>> >> > - LargestPartitionPercent: 0 (0)
>>>> >> > - MaxPartitionLevel: 0 (0)
>>>> >> > - NumRepartitions: 0 (0)
>>>> >> > - PartitionsCreated: 0 (0)
>>>> >> > - PeakMemoryUsage: 20.00 KB (20480)
>>>> >> > - RowsRepartitioned: 0 (0)
>>>> >> > - RowsReturned: 1 (1)
>>>> >> > - RowsReturnedRate: 3.00 /sec
>>>> >> > - SpilledPartitions: 0 (0)
>>>> >> > HDFS_SCAN_NODE (id=0):(Total: 260.680ms, non-child:
>>>> 260.680ms, %
>>>> >> > non-child: 100.00%)
>>>> >> > Hdfs split stats (<volume id>:<# splits>/<split
>>>> lengths>):
>>>> >> > 2:46/28.97 MB 8:39
>>>> <https://maps.google.com/?q=MB+8:39&entry=gmail&source=g>/24.34 MB
>>>> 10:33 <https://maps.google.com/?q=MB+10:33&entry=gmail&source=g>/21.03 MB
>>>> 3:43 <https://maps.google.com/?q=MB+3:43&entry=gmail&source=g>/26.50 MB
>>>> 5:32 <https://maps.google.com/?q=MB+5:32&entry=gmail&source=g>/20.27 MB
>>>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+9:32&entry=gmail&source=g>
>>>> >> > 9:32
>>>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+9:32&entry=gmail&source=g>/19.86
>>>> MB 6:25 <https://maps.google.com/?q=MB+6:25&entry=gmail&source=g>/16.01
>>>> MB 7:41 <https://maps.google.com/?q=MB+7:41&entry=gmail&source=g>/24.77
>>>> MB 1:42/25.01 MB 0:55/32.51 MB
>>>> >> > 4:40/24.29 MB 11:33
>>>> <https://maps.google.com/?q=MB+11:33&entry=gmail&source=g>/20.77 MB
>>>> >> > ExecOption: PARQUET Codegen Enabled, Codegen enabled:
>>>> 461 out
>>>> >> > of
>>>> >> > 461
>>>> >> > Hdfs Read Thread Concurrency Bucket: 0:0% 1:0% 2:0% 3:0%
>>>> 4:0%
>>>> >> > 5:0%
>>>> >> > 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
>>>> >> > File Formats: PARQUET/Unknown(Skipped):461
>>>> >> > - FooterProcessingTime: (Avg: 10.793ms ; Min: 3.087ms ;
>>>> Max:
>>>> >> > 20.253ms ; Number of samples: 461)
>>>> >> > - AverageHdfsReadThreadConcurrency: 0.00
>>>> >> > - AverageScannerThreadConcurrency: 0.00
>>>> >> > - BytesRead: 44.73 MB (46906407)
>>>> >> > - BytesReadDataNodeCache: 0
>>>> >> > - BytesReadLocal: 44.73 MB (46906407)
>>>> >> > - BytesReadRemoteUnexpected: 0
>>>> >> > - BytesReadShortCircuit: 44.73 MB (46906407)
>>>> >> > - CachedFileHandlesHitCount: 0 (0)
>>>> >> > - CachedFileHandlesMissCount: 461 (461)
>>>> >> > - CollectionItemsRead: 0 (0)
>>>> >> > - DecompressionTime: 0.000ns
>>>> >> > - MaxCompressedTextFileLength: 0
>>>> >> > - NumColumns: 0 (0)
>>>> >> > - NumDictFilteredRowGroups: 0 (0)
>>>> >> > - NumDisksAccessed: 12 (12)
>>>> >> > - NumRowGroups: 461 (461)
>>>> >> > - NumScannerThreadsStarted: 24 (24)
>>>> >> > - NumScannersWithNoReads: 0 (0)
>>>> >> > - NumStatsFilteredRowGroups: 0 (0)
>>>> >> > - PeakMemoryUsage: 2.52 MB (2641928)
>>>> >> > - PerReadThreadRawHdfsThroughput: 1.31 GB/sec
>>>> >> > - RemoteScanRanges: 0 (0)
>>>> >> > - RowBatchQueueGetWaitTime: 246.691ms
>>>> >> > - RowBatchQueuePutWaitTime: 0.000ns
>>>> >> > - RowsRead: 0 (0)
>>>> >> > - RowsReturned: 461 (461)
>>>> >> > - RowsReturnedRate: 1.77 K/sec
>>>> >> > - ScanRangesComplete: 461 (461)
>>>> >> > - ScannerThreadsInvoluntaryContextSwitches: 109 (109)
>>>> >> > - ScannerThreadsTotalWallClockTime: 6s030ms
>>>> >> > - MaterializeTupleTime(*): 68.735us
>>>> >> > - ScannerThreadsSysTime: 33.983ms
>>>> >> > - ScannerThreadsUserTime: 647.889ms
>>>> >> > - ScannerThreadsVoluntaryContextSwitches: 1.70K (1696)
>>>> >> > - TotalRawHdfsReadTime(*): 33.405ms
>>>> >> > - TotalReadThroughput: 0.00 /sec
>>>> >> > Instance c843dae48ce9b289:6ad3ffc000000003
>>>> >> > (host=aopr-dhc002.lpdomain.com:22000):(Total: 302.116ms,
>>>> non-child:
>>>> >> > 1.604ms,
>>>> >> > % non-child: 0.53%)
>>>> >> > Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>>>> >> > 0:27/16.21 MB 11:33
>>>> <https://maps.google.com/?q=MB+11:33&entry=gmail&source=g>/21.23 MB
>>>> 6:36 <https://maps.google.com/?q=MB+6:36&entry=gmail&source=g>/23.15 MB
>>>> 10:41 <https://maps.google.com/?q=MB+10:41&entry=gmail&source=g>/25.09
>>>> MB 1:41/24.13 MB
>>>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+9:32&entry=gmail&source=g>
>>>> >> > 9:32
>>>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+9:32&entry=gmail&source=g>/18.79
>>>> MB 5:41 <https://maps.google.com/?q=MB+5:41&entry=gmail&source=g>/25.31
>>>> MB 2:42 <https://maps.google.com/?q=MB+2:42&entry=gmail&source=g>/26.49
>>>> MB 7:36 <https://maps.google.com/?q=MB+7:36&entry=gmail&source=g>/22.87
>>>> MB 3:51 <https://maps.google.com/?q=MB+3:51&entry=gmail&source=g>/31.00
>>>> MB
>>>> >> > 4:41/25.99 MB 8:38
>>>> <https://maps.google.com/?q=MB+8:38&entry=gmail&source=g>/23.79 MB
>>>> >> > MemoryUsage(500.000ms): 700.35 KB
>>>> >> > ThreadUsage(500.000ms): 25
>>>> >> > - AverageThreadTokens: 25.00
>>>> >> > - BloomFilterBytes: 0
>>>> >> > - PeakMemoryUsage: 2.54 MB (2662753)
>>>> >> > - PeakReservation: 0
>>>> >> > - PeakUsedReservation: 0
>>>> >> > - PerHostPeakMemUsage: 2.54 MB (2662753)
>>>> >> > - RowsProduced: 1 (1)
>>>> >> > - TotalNetworkReceiveTime: 0.000ns
>>>> >> > - TotalNetworkSendTime: 300.675us
>>>> >> > - TotalStorageWaitTime: 4s661ms
>>>> >> > - TotalThreadsInvoluntaryContextSwitches: 90 (90)
>>>> >> > - TotalThreadsTotalWallClockTime: 6s256ms
>>>> >> > - TotalThreadsSysTime: 30.983ms
>>>> >> > - TotalThreadsUserTime: 709.880ms
>>>> >> > - TotalThreadsVoluntaryContextSwitches: 2.17K (2175)
>>>> >> > Fragment Instance Lifecycle Timings:
>>>> >> > - ExecTime: 492.627us
>>>> >> > - ExecTreeExecTime: 8.191us
>>>> >> > - OpenTime: 277.922ms
>>>> >> > - ExecTreeOpenTime: 255.395ms
>>>> >> > - PrepareTime: 23.647ms
>>>> >> > - ExecTreePrepareTime: 2.243ms
>>>> >> > DataStreamSender (dst_id=2):(Total: 194.517us, non-child:
>>>> >> > 194.517us,
>>>> >> > % non-child: 100.00%)
>>>> >> > - BytesSent: 16.00 B (16)
>>>> >> > - NetworkThroughput(*): 371.86 KB/sec
>>>> >> > - OverallThroughput: 80.33 KB/sec
>>>> >> > - PeakMemoryUsage: 16.00 KB (16384)
>>>> >> > - RowsReturned: 1 (1)
>>>> >> > - SerializeBatchTime: 14.035us
>>>> >> > - TransmitDataRPCTime: 42.019us
>>>> >> > - UncompressedRowBatchSize: 16.00 B (16)
>>>> >> > CodeGen:(Total: 42.676ms, non-child: 42.676ms, % non-child:
>>>> >> > 100.00%)
>>>> >> > - CodegenTime: 1.236ms
>>>> >> > - CompileTime: 7.717ms
>>>> >> > - LoadTime: 0.000ns
>>>> >> > - ModuleBitcodeSize: 1.86 MB (1950880)
>>>> >> > - NumFunctions: 15 (15)
>>>> >> > - NumInstructions: 214 (214)
>>>> >> > - OptimizationTime: 14.130ms
>>>> >> > - PeakMemoryUsage: 107.00 KB (109568)
>>>> >> > - PrepareTime: 20.109ms
>>>> >> > AGGREGATION_NODE (id=1):(Total: 257.640ms, non-child:
>>>> 1.541ms, %
>>>> >> > non-child: 0.60%)
>>>> >> > ExecOption: Codegen Enabled
>>>> >> > - BuildTime: 1.252ms
>>>> >> > - GetResultsTime: 0.000ns
>>>> >> > - HTResizeTime: 0.000ns
>>>> >> > - HashBuckets: 0 (0)
>>>> >> > - LargestPartitionPercent: 0 (0)
>>>> >> > - MaxPartitionLevel: 0 (0)
>>>> >> > - NumRepartitions: 0 (0)
>>>> >> > - PartitionsCreated: 0 (0)
>>>> >> > - PeakMemoryUsage: 20.00 KB (20480)
>>>> >> > - RowsRepartitioned: 0 (0)
>>>> >> > - RowsReturned: 1 (1)
>>>> >> > - RowsReturnedRate: 3.00 /sec
>>>> >> > - SpilledPartitions: 0 (0)
>>>> >> > HDFS_SCAN_NODE (id=0):(Total: 256.099ms, non-child:
>>>> 256.099ms, %
>>>> >> > non-child: 100.00%)
>>>> >> > Hdfs split stats (<volume id>:<# splits>/<split
>>>> lengths>):
>>>> >> > 0:27/16.21 MB 11:33
>>>> <https://maps.google.com/?q=MB+11:33&entry=gmail&source=g>/21.23 MB
>>>> 6:36 <https://maps.google.com/?q=MB+6:36&entry=gmail&source=g>/23.15 MB
>>>> 10:41 <https://maps.google.com/?q=MB+10:41&entry=gmail&source=g>/25.09
>>>> MB 1:41/24.13 MB
>>>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+9:32&entry=gmail&source=g>
>>>> >> > 9:32
>>>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+9:32&entry=gmail&source=g>/18.79
>>>> MB 5:41 <https://maps.google.com/?q=MB+5:41&entry=gmail&source=g>/25.31
>>>> MB 2:42 <https://maps.google.com/?q=MB+2:42&entry=gmail&source=g>/26.49
>>>> MB 7:36 <https://maps.google.com/?q=MB+7:36&entry=gmail&source=g>/22.87
>>>> MB 3:51 <https://maps.google.com/?q=MB+3:51&entry=gmail&source=g>/31.00
>>>> MB
>>>> >> > 4:41/25.99 MB 8:38
>>>> <https://maps.google.com/?q=MB+8:38&entry=gmail&source=g>/23.79 MB
>>>> >> > ExecOption: PARQUET Codegen Enabled, Codegen enabled:
>>>> 459 out
>>>> >> > of
>>>> >> > 459
>>>> >> > Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0%
>>>> 3:0%
>>>> >> > 4:0%
>>>> >> > 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
>>>> >> > File Formats: PARQUET/Unknown(Skipped):459
>>>> >> > BytesRead(500.000ms): 43.15 MB
>>>> >> > - FooterProcessingTime: (Avg: 11.257ms ; Min: 3.989ms ;
>>>> Max:
>>>> >> > 23.004ms ; Number of samples: 459)
>>>> >> > - AverageHdfsReadThreadConcurrency: 0.00
>>>> >> > - AverageScannerThreadConcurrency: 24.00
>>>> >> > - BytesRead: 44.52 MB (46678031)
>>>> >> > - BytesReadDataNodeCache: 0
>>>> >> > - BytesReadLocal: 44.52 MB (46678031)
>>>> >> > - BytesReadRemoteUnexpected: 0
>>>> >> > - BytesReadShortCircuit: 44.52 MB (46678031)
>>>> >> > - CachedFileHandlesHitCount: 0 (0)
>>>> >> > - CachedFileHandlesMissCount: 459 (459)
>>>> >> > - CollectionItemsRead: 0 (0)
>>>> >> > - DecompressionTime: 0.000ns
>>>> >> > - MaxCompressedTextFileLength: 0
>>>> >> > - NumColumns: 0 (0)
>>>> >> > - NumDictFilteredRowGroups: 0 (0)
>>>> >> > - NumDisksAccessed: 12 (12)
>>>> >> > - NumRowGroups: 459 (459)
>>>> >> > - NumScannerThreadsStarted: 24 (24)
>>>> >> > - NumScannersWithNoReads: 0 (0)
>>>> >> > - NumStatsFilteredRowGroups: 0 (0)
>>>> >> > - PeakMemoryUsage: 2.50 MB (2621440)
>>>> >> > - PerReadThreadRawHdfsThroughput: 1.33 GB/sec
>>>> >> > - RemoteScanRanges: 0 (0)
>>>> >> > - RowBatchQueueGetWaitTime: 243.426ms
>>>> >> > - RowBatchQueuePutWaitTime: 0.000ns
>>>> >> > - RowsRead: 0 (0)
>>>> >> > - RowsReturned: 459 (459)
>>>> >> > - RowsReturnedRate: 1.79 K/sec
>>>> >> > - ScanRangesComplete: 459 (459)
>>>> >> > - ScannerThreadsInvoluntaryContextSwitches: 85 (85)
>>>> >> > - ScannerThreadsTotalWallClockTime: 5s978ms
>>>> >> > - MaterializeTupleTime(*): 67.997us
>>>> >> > - ScannerThreadsSysTime: 22.984ms
>>>> >> > - ScannerThreadsUserTime: 678.884ms
>>>> >> > - ScannerThreadsVoluntaryContextSwitches: 1.72K (1724)
>>>> >> > - TotalRawHdfsReadTime(*): 32.626ms
>>>> >> > - TotalReadThroughput: 86.30 MB/sec
>>>> >> > Instance c843dae48ce9b289:6ad3ffc000000005
>>>> >> > (host=aopr-dhc001.lpdomain.com:22000):(Total: 298.093ms,
>>>> non-child:
>>>> >> > 1.242ms,
>>>> >> > % non-child: 0.42%)
>>>> >> > Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>>>> >> > 9:55/35.38 MB 0:37/24.23 MB 5:37
>>>> <https://maps.google.com/?q=MB+5:37&entry=gmail&source=g>/23.31 MB
>>>> 11:22 <https://maps.google.com/?q=MB+11:22&entry=gmail&source=g>/13.79 MB
>>>> 8:30 <https://maps.google.com/?q=MB+8:30&entry=gmail&source=g>/18.53 MB
>>>> >> > 4:33/20.45 MB 1:38/24.06 MB 10:29
>>>> <https://maps.google.com/?q=MB+10:29&entry=gmail&source=g>/18.40 MB
>>>> 3:39 <https://maps.google.com/?q=MB+3:39&entry=gmail&source=g>/24.35 MB
>>>> 2:41 <https://maps.google.com/?q=MB+2:41&entry=gmail&source=g>/25.45 MB
>>>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+6:42&entry=gmail&source=g>
>>>> >> > 6:42
>>>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+6:42&entry=gmail&source=g>/27.03
>>>> MB 7:47 <https://maps.google.com/?q=MB+7:47&entry=gmail&source=g>/29.22
>>>> MB
>>>> >> > - AverageThreadTokens: 0.00
>>>> >> > - BloomFilterBytes: 0
>>>> >> > - PeakMemoryUsage: 2.66 MB (2793825)
>>>> >> > - PeakReservation: 0
>>>> >> > - PeakUsedReservation: 0
>>>> >> > - PerHostPeakMemUsage: 2.68 MB (2806113)
>>>> >> > - RowsProduced: 1 (1)
>>>> >> > - TotalNetworkReceiveTime: 0.000ns
>>>> >> > - TotalNetworkSendTime: 198.894us
>>>> >> > - TotalStorageWaitTime: 4s374ms
>>>> >> > - TotalThreadsInvoluntaryContextSwitches: 110 (110)
>>>> >> > - TotalThreadsTotalWallClockTime: 6s201ms
>>>> >> > - TotalThreadsSysTime: 29.983ms
>>>> >> > - TotalThreadsUserTime: 642.892ms
>>>> >> > - TotalThreadsVoluntaryContextSwitches: 2.17K (2174)
>>>> >> > Fragment Instance Lifecycle Timings:
>>>> >> > - ExecTime: 350.873us
>>>> >> > - ExecTreeExecTime: 8.173us
>>>> >> > - OpenTime: 275.347ms
>>>> >> > - ExecTreeOpenTime: 253.425ms
>>>> >> > - PrepareTime: 22.341ms
>>>> >> > - ExecTreePrepareTime: 2.251ms
>>>> >> > DataStreamSender (dst_id=2):(Total: 164.394us, non-child:
>>>> >> > 164.394us,
>>>> >> > % non-child: 100.00%)
>>>> >> > - BytesSent: 16.00 B (16)
>>>> >> > - NetworkThroughput(*): 493.42 KB/sec
>>>> >> > - OverallThroughput: 95.05 KB/sec
>>>> >> > - PeakMemoryUsage: 16.00 KB (16384)
>>>> >> > - RowsReturned: 1 (1)
>>>> >> > - SerializeBatchTime: 14.262us
>>>> >> > - TransmitDataRPCTime: 31.667us
>>>> >> > - UncompressedRowBatchSize: 16.00 B (16)
>>>> >> > CodeGen:(Total: 41.008ms, non-child: 41.008ms, % non-child:
>>>> >> > 100.00%)
>>>> >> > - CodegenTime: 1.049ms
>>>> >> > - CompileTime: 7.248ms
>>>> >> > - LoadTime: 0.000ns
>>>> >> > - ModuleBitcodeSize: 1.86 MB (1950880)
>>>> >> > - NumFunctions: 15 (15)
>>>> >> > - NumInstructions: 214 (214)
>>>> >> > - OptimizationTime: 14.018ms
>>>> >> > - PeakMemoryUsage: 107.00 KB (109568)
>>>> >> > - PrepareTime: 19.080ms
>>>> >> > AGGREGATION_NODE (id=1):(Total: 255.678ms, non-child:
>>>> 1.046ms, %
>>>> >> > non-child: 0.41%)
>>>> >> > ExecOption: Codegen Enabled
>>>> >> > - BuildTime: 1.015ms
>>>> >> > - GetResultsTime: 0.000ns
>>>> >> > - HTResizeTime: 0.000ns
>>>> >> > - HashBuckets: 0 (0)
>>>> >> > - LargestPartitionPercent: 0 (0)
>>>> >> > - MaxPartitionLevel: 0 (0)
>>>> >> > - NumRepartitions: 0 (0)
>>>> >> > - PartitionsCreated: 0 (0)
>>>> >> > - PeakMemoryUsage: 20.00 KB (20480)
>>>> >> > - RowsRepartitioned: 0 (0)
>>>> >> > - RowsReturned: 1 (1)
>>>> >> > - RowsReturnedRate: 3.00 /sec
>>>> >> > - SpilledPartitions: 0 (0)
>>>> >> > HDFS_SCAN_NODE (id=0):(Total: 254.631ms, non-child:
>>>> 254.631ms, %
>>>> >> > non-child: 100.00%)
>>>> >> > Hdfs split stats (<volume id>:<# splits>/<split
>>>> lengths>):
>>>> >> > 9:55/35.38 MB 0:37/24.23 MB 5:37
>>>> <https://maps.google.com/?q=MB+5:37&entry=gmail&source=g>/23.31 MB
>>>> 11:22 <https://maps.google.com/?q=MB+11:22&entry=gmail&source=g>/13.79 MB
>>>> 8:30 <https://maps.google.com/?q=MB+8:30&entry=gmail&source=g>/18.53 MB
>>>> >> > 4:33/20.45 MB 1:38/24.06 MB 10:29
>>>> <https://maps.google.com/?q=MB+10:29&entry=gmail&source=g>/18.40 MB
>>>> 3:39 <https://maps.google.com/?q=MB+3:39&entry=gmail&source=g>/24.35 MB
>>>> 2:41 <https://maps.google.com/?q=MB+2:41&entry=gmail&source=g>/25.45 MB
>>>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+6:42&entry=gmail&source=g>
>>>> >> > 6:42
>>>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+6:42&entry=gmail&source=g>/27.03
>>>> MB 7:47 <https://maps.google.com/?q=MB+7:47&entry=gmail&source=g>/29.22
>>>> MB
>>>> >> > ExecOption: PARQUET Codegen Enabled, Codegen enabled:
>>>> 450 out
>>>> >> > of
>>>> >> > 450
>>>> >> > Hdfs Read Thread Concurrency Bucket: 0:0% 1:0% 2:0% 3:0%
>>>> 4:0%
>>>> >> > 5:0%
>>>> >> > 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
>>>> >> > File Formats: PARQUET/Unknown(Skipped):450
>>>> >> > - FooterProcessingTime: (Avg: 10.727ms ; Min: 3.813ms ;
>>>> Max:
>>>> >> > 23.755ms ; Number of samples: 450)
>>>> >> > - AverageHdfsReadThreadConcurrency: 0.00
>>>> >> > - AverageScannerThreadConcurrency: 0.00
>>>> >> > - BytesRead: 43.85 MB (45983284)
>>>> >> > - BytesReadDataNodeCache: 0
>>>> >> > - BytesReadLocal: 43.85 MB (45983284)
>>>> >> > - BytesReadRemoteUnexpected: 0
>>>> >> > - BytesReadShortCircuit: 43.85 MB (45983284)
>>>> >> > - CachedFileHandlesHitCount: 0 (0)
>>>> >> > - CachedFileHandlesMissCount: 450 (450)
>>>> >> > - CollectionItemsRead: 0 (0)
>>>> >> > - DecompressionTime: 0.000ns
>>>> >> > - MaxCompressedTextFileLength: 0
>>>> >> > - NumColumns: 0 (0)
>>>> >> > - NumDictFilteredRowGroups: 0 (0)
>>>> >> > - NumDisksAccessed: 12 (12)
>>>> >> > - NumRowGroups: 450 (450)
>>>> >> > - NumScannerThreadsStarted: 24 (24)
>>>> >> > - NumScannersWithNoReads: 0 (0)
>>>> >> > - NumStatsFilteredRowGroups: 0 (0)
>>>> >> > - PeakMemoryUsage: 2.62 MB (2752512)
>>>> >> > - PerReadThreadRawHdfsThroughput: 1.56 GB/sec
>>>> >> > - RemoteScanRanges: 0 (0)
>>>> >> > - RowBatchQueueGetWaitTime: 242.362ms
>>>> >> > - RowBatchQueuePutWaitTime: 0.000ns
>>>> >> > - RowsRead: 0 (0)
>>>> >> > - RowsReturned: 450 (450)
>>>> >> > - RowsReturnedRate: 1.77 K/sec
>>>> >> > - ScanRangesComplete: 450 (450)
>>>> >> > - ScannerThreadsInvoluntaryContextSwitches: 106 (106)
>>>> >> > - ScannerThreadsTotalWallClockTime: 5s926ms
>>>> >> > - MaterializeTupleTime(*): 69.447us
>>>> >> > - ScannerThreadsSysTime: 26.984ms
>>>> >> > - ScannerThreadsUserTime: 607.897ms
>>>> >> > - ScannerThreadsVoluntaryContextSwitches: 1.72K (1723)
>>>> >> > - TotalRawHdfsReadTime(*): 27.393ms
>>>> >> > - TotalReadThroughput: 0.00 /sec
>>>> >> > ===================================================
>>>> >> >
>>>> >> > Coordinator log:
>>>> >> >
>>>> >> > 01: query (string) = "select count(*) from
>>>> dv_entersiteevent_prq_remote
>>>> >> > where year=2017 and month=10",
>>>> >> > 03: configuration (list) = list<string>[1] {
>>>> >> > [0] = "REQUEST_POOL=impreports",
>>>> >> > },
>>>> >> > 04: hadoop_user (string) = "fawzea",
>>>> >> > }
>>>> >> > I0208 01:07:06.321281 7154 impala-beeswax-server.cc:459]
>>>> >> > TClientRequest.queryOptions: TQueryOptions {
>>>> >> > 01: abort_on_error (bool) = false,
>>>> >> > 02: max_errors (i32) = 100,
>>>> >> > 03: disable_codegen (bool) = false,
>>>> >> > 04: batch_size (i32) = 0,
>>>> >> > 05: num_nodes (i32) = 0,
>>>> >> > 06: max_scan_range_length (i64) = 0,
>>>> >> > 07: num_scanner_threads (i32) = 0,
>>>> >> > 08: max_io_buffers (i32) = 0,
>>>> >> > 09: allow_unsupported_formats (bool) = false,
>>>> >> > 10: default_order_by_limit (i64) = -1,
>>>> >> > 11: debug_action (string) = "",
>>>> >> > 12: mem_limit (i64) = 0,
>>>> >> > 13: abort_on_default_limit_exceeded (bool) = false,
>>>> >> > 15: hbase_caching (i32) = 0,
>>>> >> > 16: hbase_cache_blocks (bool) = false,
>>>> >> > 17: parquet_file_size (i64) = 0,
>>>> >> > 18: explain_level (i32) = 1,
>>>> >> > 19: sync_ddl (bool) = false,
>>>> >> > 20: request_pool (string) = "impreports",
>>>> >> > 23: disable_cached_reads (bool) = false,
>>>> >> > 24: disable_outermost_topn (bool) = false,
>>>> >> > 25: rm_initial_mem (i64) = 0,
>>>> >> > 26: query_timeout_s (i32) = 0,
>>>> >> > 28: appx_count_distinct (bool) = false,
>>>> >> > 29: disable_unsafe_spills (bool) = false,
>>>> >> > 31: exec_single_node_rows_threshold (i32) = 100,
>>>> >> > 32: optimize_partition_key_scans (bool) = false,
>>>> >> > 33: replica_preference (i32) = 0,
>>>> >> > 34: schedule_random_replica (bool) = false,
>>>> >> > 35: scan_node_codegen_threshold (i64) = 1800000,
>>>> >> > 36: disable_streaming_preaggregations (bool) = false,
>>>> >> > 37: runtime_filter_mode (i32) = 2,
>>>> >> > 38: runtime_bloom_filter_size (i32) = 1048576,
>>>> >> > 39: runtime_filter_wait_time_ms (i32) = 0,
>>>> >> > 40: disable_row_runtime_filtering (bool) = false,
>>>> >> > 41: max_num_runtime_filters (i32) = 10,
>>>> >> > 42: parquet_annotate_strings_utf8 (bool) = false,
>>>> >> > 43: parquet_fallback_schema_resolution (i32) = 0,
>>>> >> > 45: s3_skip_insert_staging (bool) = true,
>>>> >> > 46: runtime_filter_min_size (i32) = 1048576,
>>>> >> > 47: runtime_filter_max_size (i32) = 16777216,
>>>> >> > 48: prefetch_mode (i32) = 1,
>>>> >> > 49: strict_mode (bool) = false,
>>>> >> > 50: scratch_limit (i64) = -1,
>>>> >> > 51: enable_expr_rewrites (bool) = true,
>>>> >> > 52: decimal_v2 (bool) = false,
>>>> >> > 53: parquet_dictionary_filtering (bool) = true,
>>>> >> > 54: parquet_array_resolution (i32) = 2,
>>>> >> > 55: parquet_read_statistics (bool) = true,
>>>> >> > 56: default_join_distribution_mode (i32) = 0,
>>>> >> > 57: disable_codegen_rows_threshold (i32) = 50000,
>>>> >> > 58: default_spillable_buffer_size (i64) = 2097152,
>>>> >> > 59: min_spillable_buffer_size (i64) = 65536,
>>>> >> > 60: max_row_size (i64) = 524288,
>>>> >> > }
>>>> >> > I0208 01:07:06.324410 7154 Frontend.java:909] Compiling query:
>>>> select
>>>> >> > count(*) from dv_entersiteevent_prq_remote where year=2017 and
>>>> month=10
>>>> >> > I0208 01:07:06.327203 7154 Frontend.java:948] Compiled query.
>>>> >> > I0208 01:07:06.359748 7154 admission-controller.cc:510] Schedule
>>>> for
>>>> >> > id=5f49e927591a9042:faf8f28600000000 in pool_name=root.impreports
>>>> >> > cluster_mem_needed=105.00 MB PoolConfig: max_requests=-1
>>>> max_queued=200
>>>> >> > max_mem=-1.00 B
>>>> >> > I0208 01:07:06.359776 7154 admission-controller.cc:515] Stats:
>>>> >> > agg_num_running=0, agg_num_queued=0, agg_mem_reserved=0,
>>>> >> > local_host(local_mem_admitted=0, num_admitted_running=0,
>>>> num_queued=0,
>>>> >> > backend_mem_reserved=0)
>>>> >> > I0208 01:07:06.359784 7154 admission-controller.cc:531] Admitted
>>>> query
>>>> >> > id=5f49e927591a9042:faf8f28600000000
>>>> >> > I0208 01:07:06.359807 7154 coordinator.cc:99] Exec()
>>>> >> > query_id=5f49e927591a9042:faf8f28600000000 stmt=select count(*)
>>>> from
>>>> >> > dv_entersiteevent_prq_remote where year=2017 and month=10
>>>> >> > I0208 01:07:06.360694 7154 coordinator.cc:357] starting execution
>>>> on 5
>>>> >> > backends for query 5f49e927591a9042:faf8f28600000000
>>>> >> > I0208 01:07:06.363255 63583 impala-internal-service.cc:44]
>>>> >> > ExecQueryFInstances(): query_id=5f49e927591a9042:faf8f28600000000
>>>> >> > I0208 01:07:06.363282 63583 query-exec-mgr.cc:46]
>>>> StartQueryFInstances()
>>>> >> > query_id=5f49e927591a9042:faf8f28600000000
>>>> >> > coord=aopr-dhc001.lpdomain.com:22000
>>>> >> > I0208 01:07:06.363296 63583 query-state.cc:173] Buffer pool limit
>>>> for
>>>> >> > 5f49e927591a9042:faf8f28600000000: 10802010521
>>>> >> > I0208 01:07:06.363364 63583 initial-reservations.cc:60]
>>>> Successfully
>>>> >> > claimed
>>>> >> > initial reservations (0) for query 5f49e927591a9042:faf8f28600000
>>>> 000
>>>> >> > I0208 01:07:06.363544 7155 query-state.cc:286] StartFInstances():
>>>> >> > query_id=5f49e927591a9042:faf8f28600000000 #instances=2
>>>> >> > I0208 01:07:06.363844 7155 query-state.cc:299] descriptor table
>>>> for
>>>> >> > query=5f49e927591a9042:faf8f28600000000
>>>> >> > tuples:
>>>> >> > Tuple(id=2 size=0 slots=[] tuple_path=[])
>>>> >> > Tuple(id=1 size=8 slots=[Slot(id=2 type=BIGINT col_path=[] offset=0
>>>> >> > null=(offset=0 mask=0) slot_idx=0 field_idx=-1)] tuple_path=[])
>>>> >> > Tuple(id=0 size=8 slots=[Slot(id=3 type=BIGINT col_path=[] offset=0
>>>> >> > null=(offset=0 mask=0) slot_idx=0 field_idx=-1)] tuple_path=[])
>>>> >> > I0208 01:07:06.364013 7156 query-state.cc:377] Executing instance.
>>>> >> > instance_id=5f49e927591a9042:faf8f28600000000 fragment_idx=0
>>>> >> > per_fragment_instance_idx=0 coord_state_idx=0 #in-flight=1
>>>> >> > I0208 01:07:06.364135 7157 query-state.cc:377] Executing instance.
>>>> >> > instance_id=5f49e927591a9042:faf8f28600000005 fragment_idx=1
>>>> >> > per_fragment_instance_idx=4 coord_state_idx=0 #in-flight=2
>>>> >> > I0208 01:07:06.364241 7157 hdfs-scan-node.cc:160] Max row batch
>>>> queue
>>>> >> > size
>>>> >> > for scan node '0' in fragment instance
>>>> >> > '5f49e927591a9042:faf8f28600000005':
>>>> >> > 160
>>>> >> > I0208 01:07:06.366775 7154 coordinator.cc:370] started execution
>>>> on 5
>>>> >> > backends for query 5f49e927591a9042:faf8f28600000000
>>>> >> > I0208 01:07:06.385625 7155 query-exec-mgr.cc:149]
>>>> ReleaseQueryState():
>>>> >> > query_id=5f49e927591a9042:faf8f28600000000 refcnt=4
>>>> >> > I0208 01:07:06.671193 7559 impala-server.cc:1796] Connection from
>>>> >> > client
>>>> >> > ::ffff:10.16.136.3:55598 closed, closing 1 associated session(s)
>>>> >> > I0208 01:07:06.676138 544 data-stream-mgr.cc:228] Reduced stream
>>>> ID
>>>> >> > cache
>>>> >> > from 2 items, to 0, eviction took: 0
>>>> >> > I0208 01:07:06.677202 544 coordinator.cc:955] Backend completed:
>>>> >> > host=aopr-dhc002.lpdomain.com:22000 remaining=4
>>>> >> > I0208 01:07:06.677214 544 coordinator-backend-state.cc:229]
>>>> >> > query_id=5f49e927591a9042:faf8f28600000000: first in-progress
>>>> backend:
>>>> >> > aopr-dhc001.lpdomain.com:22000
>>>> >> > I0208 01:07:06.684221 7157 query-state.cc:384] Instance completed.
>>>> >> > instance_id=5f49e927591a9042:faf8f28600000005 #in-flight=1
>>>> status=OK
>>>> >> > I0208 01:07:06.684237 7157 query-exec-mgr.cc:149]
>>>> ReleaseQueryState():
>>>> >> > query_id=5f49e927591a9042:faf8f28600000000 refcnt=3
>>>> >> > I0208 01:07:06.700275 520 coordinator.cc:955] Backend completed:
>>>> >> > host=aopr-dhc005.lpdomain.com:22000 remaining=3
>>>> >> > I0208 01:07:06.700289 520 coordinator-backend-state.cc:229]
>>>> >> > query_id=5f49e927591a9042:faf8f28600000000: first in-progress
>>>> backend:
>>>> >> > aopr-dhc001.lpdomain.com:22000
>>>> >> > I0208 01:07:06.704620 65258 coordinator.cc:955] Backend completed:
>>>> >> > host=aopr-dhc003.lpdomain.com:22000 remaining=2
>>>> >> > I0208 01:07:06.704666 65258 coordinator-backend-state.cc:229]
>>>> >> > query_id=5f49e927591a9042:faf8f28600000000: first in-progress
>>>> backend:
>>>> >> > aopr-dhc001.lpdomain.com:22000
>>>> >> > I0208 01:07:06.706041 7156 data-stream-mgr.cc:238]
>>>> DeregisterRecvr():
>>>> >> > fragment_instance_id=5f49e927591a9042:faf8f28600000000, node=2
>>>> >> > I0208 01:07:06.706070 7156 data-stream-recvr.cc:235] cancelled
>>>> stream:
>>>> >> > fragment_instance_id_=5f49e927591a9042:faf8f28600000000 node_id=2
>>>> >> > I0208 01:07:06.706809 64916 coordinator.cc:955] Backend completed:
>>>> >> > host=aopr-dhc004.lpdomain.com:22000 remaining=1
>>>> >> > I0208 01:07:06.706821 64916 coordinator-backend-state.cc:229]
>>>> >> > query_id=5f49e927591a9042:faf8f28600000000: first in-progress
>>>> backend:
>>>> >> > aopr-dhc001.lpdomain.com:22000
>>>> >> > I0208 01:07:06.789568 7154 impala-beeswax-server.cc:193]
>>>> >> > get_results_metadata(): query_id=5f49e927591a9042:faf8f28600000000
>>>> >> > I0208 01:07:06.790439 7154 coordinator.cc:789] Coordinator
>>>> waiting for
>>>> >> > backends to finish, 1 remaining
>>>> >> > I0208 01:07:06.791052 7154 coordinator.cc:794] All backends
>>>> finished
>>>> >> > successfully.
>>>> >> > I0208 01:07:06.791086 7154 coordinator.cc:1090] Release admssion
>>>> >> > control
>>>> >> > resources for query 5f49e927591a9042:faf8f28600000000
>>>> >> > I0208 01:07:06.791363 7156 query-state.cc:384] Instance completed.
>>>> >> > instance_id=5f49e927591a9042:faf8f28600000000 #in-flight=0
>>>> status=OK
>>>> >> > I0208 01:07:06.791414 7156 query-exec-mgr.cc:149]
>>>> ReleaseQueryState():
>>>> >> > query_id=5f49e927591a9042:faf8f28600000000 refcnt=2
>>>> >> > I0208 01:07:06.793586 7154 impala-beeswax-server.cc:235] close():
>>>> >> > query_id=5f49e927591a9042:faf8f28600000000
>>>> >> > I0208 01:07:06.793602 7154 impala-server.cc:992]
>>>> UnregisterQuery():
>>>> >> > query_id=5f49e927591a9042:faf8f28600000000
>>>> >> > I0208 01:07:06.793614 7154 impala-server.cc:1075] Cancel():
>>>> >> > query_id=5f49e927591a9042:faf8f28600000000
>>>> >> > I0208 01:07:06.793624 7154 coordinator.cc:895] Cancel()
>>>> >> > query_id=5f49e927591a9042:faf8f28600000000
>>>> >> > I0208 01:07:06.793637 7154 coordinator.cc:905] CancelBackends()
>>>> >> > query_id=5f49e927591a9042:faf8f28600000000, tried to cancel 0
>>>> backends
>>>> >> > I0208 01:07:06.802776 7154 query-exec-mgr.cc:149]
>>>> ReleaseQueryState():
>>>> >> > query_id=5f49e927591a9042:faf8f28600000000 refcnt=1
>>>> >> > I0208 01:07:06.814214 7154 impala-server.cc:1796] Connection from
>>>> >> > client
>>>> >> > ::ffff:10.16.144.131:51012 closed, closing 1 associated session(s)
>>>> >> > I0208 01:07:07.258891 7567 impala-server.cc:1796] Connection from
>>>> >> > client
>>>> >> > ::ffff:10.16.136.2:56375 closed, closing 1 associated session(s)
>>>> >> > I0208 01:07:11.600131 7819 impala-server.cc:1796] Connection from
>>>> >> > client
>>>> >> > ::ffff:10.16.136.3:27460 closed, closing 1 associated session(s)
>>>> >> > I0208 01:07:12.262739 7821 impala-server.cc:1796] Connection from
>>>> >> > client
>>>> >> > ::ffff:10.16.136.2:52672 closed, closing 1 associated session(s)
>>>> >> > I0208 01:07:16.665451 7822 impala-server.cc:1796] Connection from
>>>> >> > client
>>>> >> > ::ffff:10.16.136.3:10693 closed, closing 1 associated session(s)
>>>> >> > I0208 01:07:17.235321 7825 impala-server.cc:1796] Connection from
>>>> >> > client
>>>> >> > ::ffff:10.16.136.2:34752 closed, closing 1 associated session(s)
>>>> >> > I0208 01:07:21.410611 5858 impala-beeswax-server.cc:193]
>>>> >> > get_results_metadata(): query_id=1d4fbf0f2571f1e9:ea346f7900000000
>>>> >> > I0208 01:07:21.414175 5858 impala-beeswax-server.cc:235] close():
>>>> >> > query_id=1d4fbf0f2571f1e9:ea346f7900000000
>>>> >> > I0208 01:07:21.414185 5858 impala-server.cc:992]
>>>> UnregisterQuery():
>>>> >> > query_id=1d4fbf0f2571f1e9:ea346f7900000000
>>>> >> > I0208 01:07:21.414191 5858 impala-server.cc:1075] Cancel():
>>>> >> > query_id=1d4fbf0f2571f1e9:ea346f7900000000
>>>> >> > I0208 01:07:21.421505 5858 impala-server.cc:1796] Connection from
>>>> >> > client
>>>> >> > ::ffff:10.26.180.75:60072 closed, closing 1 associated session(s)
>>>> >> > I0208 01:07:21.605690 7830 impala-server.cc:1796] Connection from
>>>> >> > client
>>>> >> > ::ffff:10.16.136.3:46660 closed, closing 1 associated session(s)
>>>> >> >
>>>> >> >
>>>> >> > On Thu, Feb 8, 2018 at 6:20 AM, Matthew Jacobs <mjacobs@apache.org
>>>> >
>>>> >> > wrote:
>>>> >> >>
>>>> >> >> Thanks! I think that configuration looks OK.
>>>> >> >>
>>>> >> >> 1) What version of Impala is this?
>>>> >> >>
>>>> >> >> 2) Can you send the query profile and the Impala coordinator log
>>>> for
>>>> >> >> the query where it's not behaving as expected?
>>>> >> >>
>>>> >> >> > When i'm running the following with fawzea user:
>>>> >> >> > set request_pool=impreports;
>>>> >> >> > select count(*) from dv_entersiteevent_prq_remote where
>>>> year=2017 and
>>>> >> >> > month=10;
>>>> >> >> > I can see that the query is running with impreports pool.
>>>> >> >>
>>>> >> >> On Wed, Feb 7, 2018 at 7:59 PM, Fawze Abujaber <fawzeaj@gmail.com
>>>> >
>>>> >> >> wrote:
>>>> >> >> > Hi Matt,
>>>> >> >> >
>>>> >> >> > [root@aopr-dhc001 ~]# cat
>>>> >> >> >
>>>> >> >> >
>>>> >> >> > /var/run/cloudera-scm-agent/process/4558-impala-IMPALAD/impa
>>>> la-conf/fair-scheduler.xml
>>>> >> >> > <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
>>>> >> >> > <allocations>
>>>> >> >> > <queue name="root">
>>>> >> >> > <weight>1.0</weight>
>>>> >> >> > <schedulingPolicy>fair</schedulingPolicy>
>>>> >> >> > <queue name="impiaas">
>>>> >> >> > <weight>1.0</weight>
>>>> >> >> > <schedulingPolicy>fair</schedulingPolicy>
>>>> >> >> > <aclSubmitApps>fawzea </aclSubmitApps>
>>>> >> >> > </queue>
>>>> >> >> > <queue name="impreports">
>>>> >> >> > <weight>1.0</weight>
>>>> >> >> > <schedulingPolicy>fair</schedulingPolicy>
>>>> >> >> > <aclSubmitApps>nira </aclSubmitApps>
>>>> >> >> > </queue>
>>>> >> >> > </queue>
>>>> >> >> > <defaultQueueSchedulingPolicy>fair</defaultQueueSchedulingP
>>>> olicy>
>>>> >> >> > <queuePlacementPolicy>
>>>> >> >> > <rule name="specified" create="false"/>
>>>> >> >> > <rule name="secondaryGroupExistingQueue"/>
>>>> >> >> > <rule name="default"/>
>>>> >> >> > </queuePlacementPolicy>
>>>> >> >> > </allocations>
>>>> >> >> >
>>>> >> >> >
>>>> >> >> > On Thu, Feb 8, 2018 at 4:47 AM, Matthew Jacobs <
>>>> jacobs.mh@gmail.com>
>>>> >> >> > wrote:
>>>> >> >> >>
>>>> >> >> >> Yes, I saw that, but I'd like to see the actual configuration
>>>> that
>>>> >> >> >> is
>>>> >> >> >> passed to Impala because it may not be specified in the right
>>>> way to
>>>> >> >> >> accomplish what you're trying to do in "b".
>>>> >> >> >>
>>>> >> >> >> On Wed, Feb 7, 2018 at 4:32 PM, Fawze Abujaber <
>>>> fawzeaj@gmail.com>
>>>> >> >> >> wrote:
>>>> >> >> >> > Hi Matt,
>>>> >> >> >> >
>>>> >> >> >> > I created the pools manually and it’s completely
>>>> nfigurations as
>>>> >> >> >> > shown
>>>> >> >> >> > in b.
>>>> >> >> >> >
>>>> >> >> >> > On Wed, 7 Feb 2018 at 23:17 Matthew Jacobs <
>>>> jacobs.mh@gmail.com>
>>>> >> >> >> > wrote:
>>>> >> >> >> >>
>>>> >> >> >> >> How, more specifically, are you configuring these pools?
>>>> Can you
>>>> >> >> >> >> share
>>>> >> >> >> >> the configuration files?
>>>> >> >> >> >>
>>>> >> >> >> >> On Wed, Feb 7, 2018 at 11:42 AM, Fawze Abujaber
>>>> >> >> >> >> <fa...@gmail.com>
>>>> >> >> >> >> wrote:
>>>> >> >> >> >> > Hello All,
>>>> >> >> >> >> >
>>>> >> >> >> >> > Will appreciate any insight of what i'm missing in my
>>>> >> >> >> >> > configuration
>>>> >> >> >> >> > to
>>>> >> >> >> >> > get
>>>> >> >> >> >> > the impala submission control working as expected ....
>>>> >> >> >> >> >
>>>> >> >> >> >> > a- Here is my placement Placement Rule:
>>>> >> >> >> >> > 1- Use the pool Specified at run time., only if the pool
>>>> >> >> >> >> > exists.
>>>> >> >> >> >> > 2- Use the pool root.[secondary group].
>>>> >> >> >> >> > 3- Use the pool root.default.
>>>> >> >> >> >> >
>>>> >> >> >> >> > b- Here is the impala resource pools
>>>> >> >> >> >> >
>>>> >> >> >> >> > Name Max Memory Max Running Queries Max
>>>> >> >> >> >> > Queued
>>>> >> >> >> >> > Queries
>>>> >> >> >> >> > Queue Timeout Default Query Memory Limit Access Control
>>>> >> >> >> >> > root.impiaas Unlimited
>>>> Unlimited
>>>> >> >> >> >> > 200 1 minute(s) No
>>>> Default
>>>> >> >> >> >> > Custom
>>>> >> >> >> >> > root.impreports Unlimited
>>>> Unlimited
>>>> >> >> >> >> > 200 1 minute(s) No
>>>> Default
>>>> >> >> >> >> > Custom
>>>> >> >> >> >> >
>>>> >> >> >> >> >
>>>> >> >> >> >> > C- Here is my Access control for root.impiaas pool:
>>>> >> >> >> >> >
>>>> >> >> >> >> > Allow these users and groups to submit to this pool
>>>> >> >> >> >> >
>>>> >> >> >> >> > Users: fawzea
>>>> >> >> >> >> >
>>>> >> >> >> >> > The same for impreports the allowed user and groups to
>>>> submit
>>>> >> >> >> >> > to
>>>> >> >> >> >> > pool
>>>> >> >> >> >> > is
>>>> >> >> >> >> > only nira.
>>>> >> >> >> >> >
>>>> >> >> >> >> >
>>>> >> >> >> >> > LDAP groups for fawzea user:
>>>> >> >> >> >> >
>>>> >> >> >> >> > Member in groups:
>>>> >> >> >> >> >
>>>> >> >> >> >> > - dba
>>>> >> >> >> >> > - impiaas
>>>> >> >> >> >> > - mysql
>>>> >> >> >> >> >
>>>> >> >> >> >> > LDAP group for nira user
>>>> >> >> >> >> > Member in groups:
>>>> >> >> >> >> > - dba
>>>> >> >> >> >> > - impreports
>>>> >> >> >> >> > - mysql
>>>> >> >> >> >> >
>>>> >> >> >> >> >
>>>> >> >> >> >> >
>>>> >> >> >> >> > Impala Queries UI shows:
>>>> >> >> >> >> >
>>>> >> >> >> >> > 1- select count(*) from dv_entersiteevent_prq_remote where
>>>> >> >> >> >> > year=2017
>>>> >> >> >> >> >
>>>> >> >> >> >> > User: fawzea
>>>> >> >> >> >> > Pool: root.impiaas
>>>> >> >> >> >> >
>>>> >> >> >> >> > 2- select count(*) from dv_entersiteevent_prq_remote where
>>>> >> >> >> >> > year=2017
>>>> >> >> >> >> > and
>>>> >> >> >> >> > month=10
>>>> >> >> >> >> > User: nira
>>>> >> >> >> >> > Pool: root.impreports
>>>> >> >> >> >> >
>>>> >> >> >> >> > As shown above that the queries submitted based on the
>>>> >> >> >> >> > secondary
>>>> >> >> >> >> > group
>>>> >> >> >> >> > rule.
>>>> >> >> >> >> >
>>>> >> >> >> >> > When i'm running the following with fawzea user:
>>>> >> >> >> >> >
>>>> >> >> >> >> > set request_pool=impreports;
>>>> >> >> >> >> > select count(*) from dv_entersiteevent_prq_remote where
>>>> >> >> >> >> > year=2017
>>>> >> >> >> >> > and
>>>> >> >> >> >> > month=10;
>>>> >> >> >> >> >
>>>> >> >> >> >> > I can see that the query is running with impreports pool.
>>>> >> >> >> >> >
>>>> >> >> >> >> >
>>>> >> >> >> >> > What i'm missing here in my configuration to get this
>>>> works as
>>>> >> >> >> >> > expected.
>>>> >> >> >> >> >
>>>> >> >> >
>>>> >> >> >
>>>> >> >
>>>> >> >
>>>> >
>>>> >
>>>>
>>>
>>>
>>
>
Re: impala submission control
Posted by Matthew Jacobs <ja...@gmail.com>.
I just filed one: https://issues.apache.org/jira/browse/IMPALA-6493
On Thu, Feb 8, 2018 at 10:12 AM, Matthew Jacobs <ja...@gmail.com> wrote:
> As I said, I don't see anything wrong with the configuration so there
> could be a bug in Impala 2.11, likely due to IMPALA-5920. Can you file a
> JIRA?
>
> On Thu, Feb 8, 2018 at 9:42 AM, Fawze Abujaber <fa...@gmail.com> wrote:
>
>> This is not fitting my situation,
>>
>> I removed even the placement role of the secondary group, and other users
>> can still submit to the pool that i restrict it to my user.
>>
>> Just thinking in loud and think the problem might be in using the
>> placement rule:
>>
>> Use the pool Specified at run time., only if the pool exists.
>> Edit
>>
>>
>> as it it the first rule, it always successes.
>>
>>
>> On Thu, Feb 8, 2018 at 7:12 PM, Matthew Jacobs <ja...@gmail.com>
>> wrote:
>>
>>> The aclSubmitApps can specify users or groups, so if the user name is
>>> also a group name, that can lead to confusion, e.g.
>>> https://issues.apache.org/jira/browse/IMPALA-2578
>>> I'm not saying that is necessarily the problem, but something to
>>> check. If we still can't find anything configured incorrectly I'd
>>> recommend that you file a JIRA.
>>>
>>> On Thu, Feb 8, 2018 at 9:05 AM, Fawze Abujaber <fa...@gmail.com>
>>> wrote:
>>> > Hi Matt,
>>> >
>>> > If this is the situation, how we can explain that when i submit the
>>> query it
>>> > runs on my secondary group pool and when nira do the same it runs
>>> under her
>>> > secondary group pool.
>>> >
>>> > More than this, the issue not related to LDAP group as i'm using the
>>> > submission restriction to a user and not a group.
>>> >
>>> > On Thu, Feb 8, 2018 at 7:00 PM, Matthew Jacobs <ja...@gmail.com>
>>> wrote:
>>> >>
>>> >> As far as I can tell from this information, your configuration looks
>>> >> fine. The only other potential issue I can think of is that the
>>> >> secondary groups placement rule isn't picking up your LDAP groups
>>> >> properly. Does the ACL behavior work if you remove the secondary
>>> >> groups rule?
>>> >>
>>> >> On Wed, Feb 7, 2018 at 10:09 PM, Fawze Abujaber <fa...@gmail.com>
>>> wrote:
>>> >> > Hi Matt,
>>> >> >
>>> >> > v2.11.0-cdh5.14.0
>>> >> >
>>> >> > Query Runtime Profile:
>>> >> > Query (id=c843dae48ce9b289:6ad3ffc000000000):
>>> >> > Summary:
>>> >> > Session ID: ca420ac477772b56:51ae460ce9e44999
>>> >> > Session Type: BEESWAX
>>> >> > Start Time: 2018-02-08 01:02:01.023558000
>>> >> > End Time: 2018-02-08 01:02:01.501506000
>>> >> > Query Type: QUERY
>>> >> > Query State: FINISHED
>>> >> > Query Status: OK
>>> >> > Impala Version: impalad version 2.11.0-cdh5.14.0 RELEASE (build
>>> >> > d68206561bce6b26762d62c01a78e6cd27aa7690)
>>> >> > User: fawzea
>>> >> > Connected User: fawzea
>>> >> > Delegated User:
>>> >> > Network Address: ::ffff:10.16.144.131:50078
>>> >> > Default Db: default
>>> >> > Sql Statement: select count(*) from dv_entersiteevent_prq_remote
>>> >> > where
>>> >> > year=2017 and month=10
>>> >> > Coordinator: aopr-dhc001.lpdomain.com:22000
>>> >> > Query Options (set by configuration): REQUEST_POOL=impreports
>>> >> > Query Options (set by configuration and planner):
>>> >> > REQUEST_POOL=impreports,MT_DOP=0
>>> >> > Plan:
>>> >> > ----------------
>>> >> > Max Per-Host Resource Reservation: Memory=0B
>>> >> > Per-Host Resource Estimates: Memory=21.00MB
>>> >> > WARNING: The following tables are missing relevant table and/or
>>> column
>>> >> > statistics.
>>> >> > default.dv_entersiteevent_prq_remote
>>> >> >
>>> >> > F01:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1
>>> >> > | Per-Host Resources: mem-estimate=10.00MB mem-reservation=0B
>>> >> > PLAN-ROOT SINK
>>> >> > | mem-estimate=0B mem-reservation=0B
>>> >> > |
>>> >> > 03:AGGREGATE [FINALIZE]
>>> >> > | output: count:merge(*)
>>> >> > | mem-estimate=10.00MB mem-reservation=0B spill-buffer=2.00MB
>>> >> > | tuple-ids=1 row-size=8B cardinality=1
>>> >> > |
>>> >> > 02:EXCHANGE [UNPARTITIONED]
>>> >> > | mem-estimate=0B mem-reservation=0B
>>> >> > | tuple-ids=1 row-size=8B cardinality=1
>>> >> > |
>>> >> > F00:PLAN FRAGMENT [RANDOM] hosts=5 instances=5
>>> >> > Per-Host Resources: mem-estimate=11.00MB mem-reservation=0B
>>> >> > 01:AGGREGATE
>>> >> > | output:
>>> >> > sum_init_zero(default.dv_entersiteevent_prq_remote.parquet-stats:
>>> >> > num_rows)
>>> >> > | mem-estimate=10.00MB mem-reservation=0B spill-buffer=2.00MB
>>> >> > | tuple-ids=1 row-size=8B cardinality=1
>>> >> > |
>>> >> > 00:SCAN HDFS [default.dv_entersiteevent_prq_remote, RANDOM]
>>> >> > partitions=31/441 files=2292 size=1.39GB
>>> >> > stats-rows=unavailable extrapolated-rows=disabled
>>> >> > table stats: rows=unavailable size=unavailable
>>> >> > column stats: all
>>> >> > mem-estimate=1.00MB mem-reservation=0B
>>> >> > tuple-ids=0 row-size=8B cardinality=unavailable
>>> >> > ----------------
>>> >> > Estimated Per-Host Mem: 22020096
>>> >> > Tables Missing Stats: default.dv_entersiteevent_prq_remote
>>> >> > Per Host Min Reservation: aopr-dhc001.lpdomain.com:22000(0)
>>> >> > aopr-dhc002.lpdomain.com:22000(0) aopr-dhc003.lpdomain.com:22000(0)
>>> >> > aopr-dhc004.lpdomain.com:22000(0) aopr-dhc005.lpdomain.com:22000(0)
>>> >> > Request Pool: root.impreports
>>> >> > Admission result: Admitted immediately
>>> >> > ExecSummary:
>>> >> > Operator #Hosts Avg Time Max Time #Rows Est. #Rows
>>> Peak Mem
>>> >> > Est. Peak Mem Detail
>>> >> >
>>> >> > ------------------------------------------------------------
>>> ------------------------------------------------------------------
>>> >> > 03:AGGREGATE 1 222.958us 222.958us 1 1
>>> 36.00 KB
>>> >> > 10.00 MB FINALIZE
>>> >> > 02:EXCHANGE 1 357.508ms 357.508ms 5 1
>>> 0
>>> >> > 0 UNPARTITIONED
>>> >> > 01:AGGREGATE 5 1.027ms 1.541ms 5 1
>>> 20.00 KB
>>> >> > 10.00 MB
>>> >> > 00:SCAN HDFS 5 282.797ms 333.715ms 2.29K -1
>>> 2.62 MB
>>> >> > 1.00 MB default.dv_entersiteevent_prq_remote
>>> >> > Errors:
>>> >> > Planner Timeline: 13.508ms
>>> >> > - Analysis finished: 3.303ms (3.303ms)
>>> >> > - Value transfer graph computed: 3.465ms (162.667us)
>>> >> > - Single node plan created: 10.352ms (6.886ms)
>>> >> > - Runtime filters computed: 10.677ms (325.364us)
>>> >> > - Distributed plan created: 10.733ms (55.854us)
>>> >> > - Lineage info computed: 10.898ms (164.937us)
>>> >> > - Planning finished: 13.508ms (2.610ms)
>>> >> > Query Timeline: 481.950ms
>>> >> > - Query submitted: 79.971us (79.971us)
>>> >> > - Planning finished: 33.947ms (33.867ms)
>>> >> > - Submit for admission: 40.067ms (6.120ms)
>>> >> > - Completed admission: 40.197ms (130.146us)
>>> >> > - Ready to start on 5 backends: 40.905ms (707.852us)
>>> >> > - All 5 execution backends (6 fragment instances) started:
>>> >> > 46.417ms
>>> >> > (5.512ms)
>>> >> > - Rows available: 424.548ms (378.131ms)
>>> >> > - First row fetched: 473.488ms (48.939ms)
>>> >> > - Unregister query: 477.951ms (4.462ms)
>>> >> > - ComputeScanRangeAssignmentTimer: 3.980ms
>>> >> > ImpalaServer:
>>> >> > - ClientFetchWaitTimer: 51.659ms
>>> >> > - RowMaterializationTimer: 1.726ms
>>> >> > Execution Profile c843dae48ce9b289:6ad3ffc000000000:(Total:
>>> 385.029ms,
>>> >> > non-child: 0.000ns, % non-child: 0.00%)
>>> >> > Number of filters: 0
>>> >> > Filter routing table:
>>> >> > ID Src. Node Tgt. Node(s) Target type Partition filter Pending
>>> >> > (Expected) First arrived Completed Enabled
>>> >> >
>>> >> > ------------------------------------------------------------
>>> -------------------------------------------------------
>>> >> >
>>> >> > Backend startup latencies: Count: 5, min / max: 3ms / 4ms, 25th
>>> >> > %-ile:
>>> >> > 3ms, 50th %-ile: 4ms, 75th %-ile: 4ms, 90th %-ile: 4ms, 95th %-ile:
>>> 4ms,
>>> >> > 99.9th %-ile: 4ms
>>> >> > Per Node Peak Memory Usage: aopr-dhc001.lpdomain.com:22000(2.68
>>> MB)
>>> >> > aopr-dhc002.lpdomain.com:22000(2.54 MB)
>>> >> > aopr-dhc003.lpdomain.com:22000(2.54
>>> >> > MB) aopr-dhc004.lpdomain.com:22000(2.41 MB)
>>> >> > aopr-dhc005.lpdomain.com:22000(2.56 MB)
>>> >> > - FiltersReceived: 0 (0)
>>> >> > - FinalizationTimer: 0.000ns
>>> >> > - NumBackends: 5 (5)
>>> >> > - NumFragmentInstances: 6 (6)
>>> >> > - NumFragments: 2 (2)
>>> >> > Averaged Fragment F01:(Total: 427.530ms, non-child: 49.428ms, %
>>> >> > non-child: 11.56%)
>>> >> > split sizes: min: 0, max: 0, avg: 0, stddev: 0
>>> >> > completion times: min:428.693ms max:428.693ms mean:
>>> 428.693ms
>>> >> > stddev:0.000ns
>>> >> > execution rates: min:0.00 /sec max:0.00 /sec mean:0.00 /sec
>>> >> > stddev:0.00 /sec
>>> >> > num instances: 1
>>> >> > - AverageThreadTokens: 0.00
>>> >> > - BloomFilterBytes: 0
>>> >> > - PeakMemoryUsage: 44.02 KB (45072)
>>> >> > - PeakReservation: 0
>>> >> > - PeakUsedReservation: 0
>>> >> > - PerHostPeakMemUsage: 2.68 MB (2806113)
>>> >> > - RowsProduced: 1 (1)
>>> >> > - TotalNetworkReceiveTime: 357.481ms
>>> >> > - TotalNetworkSendTime: 0.000ns
>>> >> > - TotalStorageWaitTime: 0.000ns
>>> >> > - TotalThreadsInvoluntaryContextSwitches: 0 (0)
>>> >> > - TotalThreadsTotalWallClockTime: 406.708ms
>>> >> > - TotalThreadsSysTime: 0.000ns
>>> >> > - TotalThreadsUserTime: 0.000ns
>>> >> > - TotalThreadsVoluntaryContextSwitches: 7 (7)
>>> >> > Fragment Instance Lifecycle Timings:
>>> >> > - ExecTime: 49.055ms
>>> >> > - ExecTreeExecTime: 4.251us
>>> >> > - OpenTime: 357.663ms
>>> >> > - ExecTreeOpenTime: 357.649ms
>>> >> > - PrepareTime: 20.764ms
>>> >> > - ExecTreePrepareTime: 81.759us
>>> >> > PLAN_ROOT_SINK:
>>> >> > - PeakMemoryUsage: 0
>>> >> > CodeGen:(Total: 20.369ms, non-child: 20.369ms, % non-child:
>>> >> > 100.00%)
>>> >> > - CodegenTime: 0.000ns
>>> >> > - CompileTime: 0.000ns
>>> >> > - LoadTime: 0.000ns
>>> >> > - ModuleBitcodeSize: 1.86 MB (1950880)
>>> >> > - NumFunctions: 0 (0)
>>> >> > - NumInstructions: 0 (0)
>>> >> > - OptimizationTime: 0.000ns
>>> >> > - PeakMemoryUsage: 0
>>> >> > - PrepareTime: 19.685ms
>>> >> > AGGREGATION_NODE (id=3):(Total: 357.731ms, non-child:
>>> 222.958us, %
>>> >> > non-child: 0.06%)
>>> >> > - BuildTime: 7.368us
>>> >> > - GetResultsTime: 0.000ns
>>> >> > - HTResizeTime: 0.000ns
>>> >> > - HashBuckets: 0 (0)
>>> >> > - LargestPartitionPercent: 0 (0)
>>> >> > - MaxPartitionLevel: 0 (0)
>>> >> > - NumRepartitions: 0 (0)
>>> >> > - PartitionsCreated: 0 (0)
>>> >> > - PeakMemoryUsage: 36.00 KB (36864)
>>> >> > - RowsRepartitioned: 0 (0)
>>> >> > - RowsReturned: 1 (1)
>>> >> > - RowsReturnedRate: 2.00 /sec
>>> >> > - SpilledPartitions: 0 (0)
>>> >> > EXCHANGE_NODE (id=2):(Total: 357.508ms, non-child: 357.508ms,
>>> %
>>> >> > non-child: 100.00%)
>>> >> > - ConvertRowBatchTime: 8.881us
>>> >> > - PeakMemoryUsage: 0
>>> >> > - RowsReturned: 5 (5)
>>> >> > - RowsReturnedRate: 13.00 /sec
>>> >> > DataStreamReceiver:
>>> >> > - BytesReceived: 80.00 B (80)
>>> >> > - DeserializeRowBatchTimer: 24.165us
>>> >> > - FirstBatchArrivalWaitTime: 277.194ms
>>> >> > - PeakMemoryUsage: 4.02 KB (4112)
>>> >> > - SendersBlockedTimer: 0.000ns
>>> >> > - SendersBlockedTotalTimer(*): 0.000ns
>>> >> > Coordinator Fragment F01:
>>> >> > Instance c843dae48ce9b289:6ad3ffc000000000
>>> >> > (host=aopr-dhc001.lpdomain.com:22000):(Total: 427.530ms, non-child:
>>> >> > 49.428ms, % non-child: 11.56%)
>>> >> > - AverageThreadTokens: 0.00
>>> >> > - BloomFilterBytes: 0
>>> >> > - PeakMemoryUsage: 44.02 KB (45072)
>>> >> > - PeakReservation: 0
>>> >> > - PeakUsedReservation: 0
>>> >> > - PerHostPeakMemUsage: 2.68 MB (2806113)
>>> >> > - RowsProduced: 1 (1)
>>> >> > - TotalNetworkReceiveTime: 357.481ms
>>> >> > - TotalNetworkSendTime: 0.000ns
>>> >> > - TotalStorageWaitTime: 0.000ns
>>> >> > - TotalThreadsInvoluntaryContextSwitches: 0 (0)
>>> >> > - TotalThreadsTotalWallClockTime: 406.708ms
>>> >> > - TotalThreadsSysTime: 0.000ns
>>> >> > - TotalThreadsUserTime: 0.000ns
>>> >> > - TotalThreadsVoluntaryContextSwitches: 7 (7)
>>> >> > Fragment Instance Lifecycle Timings:
>>> >> > - ExecTime: 49.055ms
>>> >> > - ExecTreeExecTime: 4.251us
>>> >> > - OpenTime: 357.663ms
>>> >> > - ExecTreeOpenTime: 357.649ms
>>> >> > - PrepareTime: 20.764ms
>>> >> > - ExecTreePrepareTime: 81.759us
>>> >> > PLAN_ROOT_SINK:
>>> >> > - PeakMemoryUsage: 0
>>> >> > CodeGen:(Total: 20.369ms, non-child: 20.369ms, % non-child:
>>> >> > 100.00%)
>>> >> > - CodegenTime: 0.000ns
>>> >> > - CompileTime: 0.000ns
>>> >> > - LoadTime: 0.000ns
>>> >> > - ModuleBitcodeSize: 1.86 MB (1950880)
>>> >> > - NumFunctions: 0 (0)
>>> >> > - NumInstructions: 0 (0)
>>> >> > - OptimizationTime: 0.000ns
>>> >> > - PeakMemoryUsage: 0
>>> >> > - PrepareTime: 19.685ms
>>> >> > AGGREGATION_NODE (id=3):(Total: 357.731ms, non-child:
>>> 222.958us,
>>> >> > %
>>> >> > non-child: 0.06%)
>>> >> > - BuildTime: 7.368us
>>> >> > - GetResultsTime: 0.000ns
>>> >> > - HTResizeTime: 0.000ns
>>> >> > - HashBuckets: 0 (0)
>>> >> > - LargestPartitionPercent: 0 (0)
>>> >> > - MaxPartitionLevel: 0 (0)
>>> >> > - NumRepartitions: 0 (0)
>>> >> > - PartitionsCreated: 0 (0)
>>> >> > - PeakMemoryUsage: 36.00 KB (36864)
>>> >> > - RowsRepartitioned: 0 (0)
>>> >> > - RowsReturned: 1 (1)
>>> >> > - RowsReturnedRate: 2.00 /sec
>>> >> > - SpilledPartitions: 0 (0)
>>> >> > EXCHANGE_NODE (id=2):(Total: 357.508ms, non-child:
>>> 357.508ms, %
>>> >> > non-child: 100.00%)
>>> >> > - ConvertRowBatchTime: 8.881us
>>> >> > - PeakMemoryUsage: 0
>>> >> > - RowsReturned: 5 (5)
>>> >> > - RowsReturnedRate: 13.00 /sec
>>> >> > DataStreamReceiver:
>>> >> > - BytesReceived: 80.00 B (80)
>>> >> > - DeserializeRowBatchTimer: 24.165us
>>> >> > - FirstBatchArrivalWaitTime: 277.194ms
>>> >> > - PeakMemoryUsage: 4.02 KB (4112)
>>> >> > - SendersBlockedTimer: 0.000ns
>>> >> > - SendersBlockedTotalTimer(*): 0.000ns
>>> >> > Averaged Fragment F00:(Total: 327.360ms, non-child: 1.578ms, %
>>> >> > non-child: 0.48%)
>>> >> > split sizes: min: 283.31 MB, max: 284.33 MB, avg: 284.04 MB,
>>> >> > stddev:
>>> >> > 382.00 KB
>>> >> > completion times: min:299.457ms max:378.950ms mean:
>>> 328.881ms
>>> >> > stddev:31.507ms
>>> >> > execution rates: min:750.18 MB/sec max:949.01 MB/sec
>>> mean:871.27
>>> >> > MB/sec stddev:79.74 MB/sec
>>> >> > num instances: 5
>>> >> > - AverageThreadTokens: 15.00
>>> >> > - BloomFilterBytes: 0
>>> >> > - PeakMemoryUsage: 2.54 MB (2666852)
>>> >> > - PeakReservation: 0
>>> >> > - PeakUsedReservation: 0
>>> >> > - PerHostPeakMemUsage: 2.55 MB (2669309)
>>> >> > - RowsProduced: 1 (1)
>>> >> > - TotalNetworkReceiveTime: 0.000ns
>>> >> > - TotalNetworkSendTime: 303.541us
>>> >> > - TotalStorageWaitTime: 4s821ms
>>> >> > - TotalThreadsInvoluntaryContextSwitches: 104 (104)
>>> >> > - TotalThreadsTotalWallClockTime: 6s835ms
>>> >> > - TotalThreadsSysTime: 31.784ms
>>> >> > - TotalThreadsUserTime: 682.084ms
>>> >> > - TotalThreadsVoluntaryContextSwitches: 2.12K (2121)
>>> >> > Fragment Instance Lifecycle Timings:
>>> >> > - ExecTime: 500.816us
>>> >> > - ExecTreeExecTime: 8.200us
>>> >> > - OpenTime: 302.848ms
>>> >> > - ExecTreeOpenTime: 281.309ms
>>> >> > - PrepareTime: 23.946ms
>>> >> > - ExecTreePrepareTime: 2.513ms
>>> >> > DataStreamSender (dst_id=2):(Total: 269.902us, non-child:
>>> >> > 269.902us, %
>>> >> > non-child: 100.00%)
>>> >> > - BytesSent: 16.00 B (16)
>>> >> > - NetworkThroughput(*): 373.13 KB/sec
>>> >> > - OverallThroughput: 63.53 KB/sec
>>> >> > - PeakMemoryUsage: 16.00 KB (16384)
>>> >> > - RowsReturned: 1 (1)
>>> >> > - SerializeBatchTime: 14.525us
>>> >> > - TransmitDataRPCTime: 43.122us
>>> >> > - UncompressedRowBatchSize: 16.00 B (16)
>>> >> > CodeGen:(Total: 41.687ms, non-child: 41.687ms, % non-child:
>>> >> > 100.00%)
>>> >> > - CodegenTime: 1.194ms
>>> >> > - CompileTime: 7.360ms
>>> >> > - LoadTime: 0.000ns
>>> >> > - ModuleBitcodeSize: 1.86 MB (1950880)
>>> >> > - NumFunctions: 15 (15)
>>> >> > - NumInstructions: 214 (214)
>>> >> > - OptimizationTime: 13.529ms
>>> >> > - PeakMemoryUsage: 107.00 KB (109568)
>>> >> > - PrepareTime: 20.084ms
>>> >> > AGGREGATION_NODE (id=1):(Total: 283.825ms, non-child:
>>> 1.027ms, %
>>> >> > non-child: 0.36%)
>>> >> > - BuildTime: 1.259ms
>>> >> > - GetResultsTime: 0.000ns
>>> >> > - HTResizeTime: 0.000ns
>>> >> > - HashBuckets: 0 (0)
>>> >> > - LargestPartitionPercent: 0 (0)
>>> >> > - MaxPartitionLevel: 0 (0)
>>> >> > - NumRepartitions: 0 (0)
>>> >> > - PartitionsCreated: 0 (0)
>>> >> > - PeakMemoryUsage: 20.00 KB (20480)
>>> >> > - RowsRepartitioned: 0 (0)
>>> >> > - RowsReturned: 1 (1)
>>> >> > - RowsReturnedRate: 2.00 /sec
>>> >> > - SpilledPartitions: 0 (0)
>>> >> > HDFS_SCAN_NODE (id=0):(Total: 282.797ms, non-child:
>>> 282.797ms, %
>>> >> > non-child: 100.00%)
>>> >> > - AverageHdfsReadThreadConcurrency: 0.00
>>> >> > - AverageScannerThreadConcurrency: 14.40
>>> >> > - BytesRead: 44.52 MB (46686028)
>>> >> > - BytesReadDataNodeCache: 0
>>> >> > - BytesReadLocal: 44.52 MB (46686028)
>>> >> > - BytesReadRemoteUnexpected: 0
>>> >> > - BytesReadShortCircuit: 44.52 MB (46686028)
>>> >> > - CachedFileHandlesHitCount: 0 (0)
>>> >> > - CachedFileHandlesMissCount: 458 (458)
>>> >> > - CollectionItemsRead: 0 (0)
>>> >> > - DecompressionTime: 0.000ns
>>> >> > - MaxCompressedTextFileLength: 0
>>> >> > - NumColumns: 0 (0)
>>> >> > - NumDictFilteredRowGroups: 0 (0)
>>> >> > - NumDisksAccessed: 12 (12)
>>> >> > - NumRowGroups: 458 (458)
>>> >> > - NumScannerThreadsStarted: 24 (24)
>>> >> > - NumScannersWithNoReads: 0 (0)
>>> >> > - NumStatsFilteredRowGroups: 0 (0)
>>> >> > - PeakMemoryUsage: 2.50 MB (2625539)
>>> >> > - PerReadThreadRawHdfsThroughput: 1.29 GB/sec
>>> >> > - RemoteScanRanges: 0 (0)
>>> >> > - RowBatchQueueGetWaitTime: 269.787ms
>>> >> > - RowBatchQueuePutWaitTime: 0.000ns
>>> >> > - RowsRead: 0 (0)
>>> >> > - RowsReturned: 458 (458)
>>> >> > - RowsReturnedRate: 1.64 K/sec
>>> >> > - ScanRangesComplete: 458 (458)
>>> >> > - ScannerThreadsInvoluntaryContextSwitches: 99 (99)
>>> >> > - ScannerThreadsTotalWallClockTime: 6s532ms
>>> >> > - MaterializeTupleTime(*): 70.664us
>>> >> > - ScannerThreadsSysTime: 26.184ms
>>> >> > - ScannerThreadsUserTime: 650.089ms
>>> >> > - ScannerThreadsVoluntaryContextSwitches: 1.67K (1674)
>>> >> > - TotalRawHdfsReadTime(*): 34.416ms
>>> >> > - TotalReadThroughput: 36.86 MB/sec
>>> >> > Fragment F00:
>>> >> > Instance c843dae48ce9b289:6ad3ffc000000004
>>> >> > (host=aopr-dhc004.lpdomain.com:22000):(Total: 377.384ms, non-child:
>>> >> > 1.619ms,
>>> >> > % non-child: 0.43%)
>>> >> > Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>>> >> > 0:36/21.95 MB 11:33
>>> <https://maps.google.com/?q=MB+11:33&entry=gmail&source=g>/19.87 MB 9:27
>>> <https://maps.google.com/?q=MB+9:27&entry=gmail&source=g>/15.16 MB 5:44
>>> <https://maps.google.com/?q=MB+5:44&entry=gmail&source=g>/28.07 MB 6:56
>>> <https://maps.google.com/?q=MB+6:56&entry=gmail&source=g>/33.32 MB
>>> >> > 1:38/22.24 MB 10:44
>>> <https://maps.google.com/?q=MB+10:44&entry=gmail&source=g>/27.90 MB 3:34
>>> <https://maps.google.com/?q=MB+3:34&entry=gmail&source=g>/20.26 MB 2:45
>>> <https://maps.google.com/?q=MB+2:45&entry=gmail&source=g>/28.21 MB 7:31
>>> <https://maps.google.com/?q=MB+7:31&entry=gmail&source=g>/19.33 MB
>>> >> > 4:42/26.73 MB 8:34
>>> <https://maps.google.com/?q=MB+8:34&entry=gmail&source=g>/21.25 MB
>>> >> > MemoryUsage(500.000ms): 1.91 MB
>>> >> > ThreadUsage(500.000ms): 25
>>> >> > - AverageThreadTokens: 25.00
>>> >> > - BloomFilterBytes: 0
>>> >> > - PeakMemoryUsage: 2.41 MB (2531689)
>>> >> > - PeakReservation: 0
>>> >> > - PeakUsedReservation: 0
>>> >> > - PerHostPeakMemUsage: 2.41 MB (2531689)
>>> >> > - RowsProduced: 1 (1)
>>> >> > - TotalNetworkReceiveTime: 0.000ns
>>> >> > - TotalNetworkSendTime: 382.629us
>>> >> > - TotalStorageWaitTime: 5s790ms
>>> >> > - TotalThreadsInvoluntaryContextSwitches: 115 (115)
>>> >> > - TotalThreadsTotalWallClockTime: 8s122ms
>>> >> > - TotalThreadsSysTime: 33.984ms
>>> >> > - TotalThreadsUserTime: 702.879ms
>>> >> > - TotalThreadsVoluntaryContextSwitches: 2.04K (2037)
>>> >> > Fragment Instance Lifecycle Timings:
>>> >> > - ExecTime: 604.005us
>>> >> > - ExecTreeExecTime: 8.338us
>>> >> > - OpenTime: 352.927ms
>>> >> > - ExecTreeOpenTime: 331.921ms
>>> >> > - PrepareTime: 23.783ms
>>> >> > - ExecTreePrepareTime: 2.559ms
>>> >> > DataStreamSender (dst_id=2):(Total: 331.818us, non-child:
>>> >> > 331.818us,
>>> >> > % non-child: 100.00%)
>>> >> > - BytesSent: 16.00 B (16)
>>> >> > - NetworkThroughput(*): 289.00 KB/sec
>>> >> > - OverallThroughput: 47.09 KB/sec
>>> >> > - PeakMemoryUsage: 16.00 KB (16384)
>>> >> > - RowsReturned: 1 (1)
>>> >> > - SerializeBatchTime: 14.466us
>>> >> > - TransmitDataRPCTime: 54.066us
>>> >> > - UncompressedRowBatchSize: 16.00 B (16)
>>> >> > CodeGen:(Total: 40.950ms, non-child: 40.950ms, % non-child:
>>> >> > 100.00%)
>>> >> > - CodegenTime: 1.122ms
>>> >> > - CompileTime: 7.318ms
>>> >> > - LoadTime: 0.000ns
>>> >> > - ModuleBitcodeSize: 1.86 MB (1950880)
>>> >> > - NumFunctions: 15 (15)
>>> >> > - NumInstructions: 214 (214)
>>> >> > - OptimizationTime: 13.106ms
>>> >> > - PeakMemoryUsage: 107.00 KB (109568)
>>> >> > - PrepareTime: 19.773ms
>>> >> > AGGREGATION_NODE (id=1):(Total: 334.482ms, non-child:
>>> 766.527us,
>>> >> > %
>>> >> > non-child: 0.23%)
>>> >> > ExecOption: Codegen Enabled
>>> >> > - BuildTime: 1.192ms
>>> >> > - GetResultsTime: 0.000ns
>>> >> > - HTResizeTime: 0.000ns
>>> >> > - HashBuckets: 0 (0)
>>> >> > - LargestPartitionPercent: 0 (0)
>>> >> > - MaxPartitionLevel: 0 (0)
>>> >> > - NumRepartitions: 0 (0)
>>> >> > - PartitionsCreated: 0 (0)
>>> >> > - PeakMemoryUsage: 20.00 KB (20480)
>>> >> > - RowsRepartitioned: 0 (0)
>>> >> > - RowsReturned: 1 (1)
>>> >> > - RowsReturnedRate: 2.00 /sec
>>> >> > - SpilledPartitions: 0 (0)
>>> >> > HDFS_SCAN_NODE (id=0):(Total: 333.715ms, non-child:
>>> 333.715ms, %
>>> >> > non-child: 100.00%)
>>> >> > Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>>> >> > 0:36/21.95 MB 11:33
>>> <https://maps.google.com/?q=MB+11:33&entry=gmail&source=g>/19.87 MB 9:27
>>> <https://maps.google.com/?q=MB+9:27&entry=gmail&source=g>/15.16 MB 5:44
>>> <https://maps.google.com/?q=MB+5:44&entry=gmail&source=g>/28.07 MB 6:56
>>> <https://maps.google.com/?q=MB+6:56&entry=gmail&source=g>/33.32 MB
>>> >> > 1:38/22.24 MB 10:44
>>> <https://maps.google.com/?q=MB+10:44&entry=gmail&source=g>/27.90 MB 3:34
>>> <https://maps.google.com/?q=MB+3:34&entry=gmail&source=g>/20.26 MB 2:45
>>> <https://maps.google.com/?q=MB+2:45&entry=gmail&source=g>/28.21 MB 7:31
>>> <https://maps.google.com/?q=MB+7:31&entry=gmail&source=g>/19.33 MB
>>> >> > 4:42/26.73 MB 8:34
>>> <https://maps.google.com/?q=MB+8:34&entry=gmail&source=g>/21.25 MB
>>> >> > ExecOption: PARQUET Codegen Enabled, Codegen enabled: 464
>>> out
>>> >> > of
>>> >> > 464
>>> >> > Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0%
>>> >> > 4:0%
>>> >> > 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
>>> >> > File Formats: PARQUET/Unknown(Skipped):464
>>> >> > BytesRead(500.000ms): 12.27 MB
>>> >> > - FooterProcessingTime: (Avg: 13.555ms ; Min: 5.433ms ;
>>> Max:
>>> >> > 62.957ms ; Number of samples: 464)
>>> >> > - AverageHdfsReadThreadConcurrency: 0.00
>>> >> > - AverageScannerThreadConcurrency: 24.00
>>> >> > - BytesRead: 45.01 MB (47193988)
>>> >> > - BytesReadDataNodeCache: 0
>>> >> > - BytesReadLocal: 45.01 MB (47193988)
>>> >> > - BytesReadRemoteUnexpected: 0
>>> >> > - BytesReadShortCircuit: 45.01 MB (47193988)
>>> >> > - CachedFileHandlesHitCount: 0 (0)
>>> >> > - CachedFileHandlesMissCount: 464 (464)
>>> >> > - CollectionItemsRead: 0 (0)
>>> >> > - DecompressionTime: 0.000ns
>>> >> > - MaxCompressedTextFileLength: 0
>>> >> > - NumColumns: 0 (0)
>>> >> > - NumDictFilteredRowGroups: 0 (0)
>>> >> > - NumDisksAccessed: 12 (12)
>>> >> > - NumRowGroups: 464 (464)
>>> >> > - NumScannerThreadsStarted: 24 (24)
>>> >> > - NumScannersWithNoReads: 0 (0)
>>> >> > - NumStatsFilteredRowGroups: 0 (0)
>>> >> > - PeakMemoryUsage: 2.38 MB (2490376)
>>> >> > - PerReadThreadRawHdfsThroughput: 1016.44 MB/sec
>>> >> > - RemoteScanRanges: 0 (0)
>>> >> > - RowBatchQueueGetWaitTime: 320.860ms
>>> >> > - RowBatchQueuePutWaitTime: 0.000ns
>>> >> > - RowsRead: 0 (0)
>>> >> > - RowsReturned: 464 (464)
>>> >> > - RowsReturnedRate: 1.39 K/sec
>>> >> > - ScanRangesComplete: 464 (464)
>>> >> > - ScannerThreadsInvoluntaryContextSwitches: 108 (108)
>>> >> > - ScannerThreadsTotalWallClockTime: 7s769ms
>>> >> > - MaterializeTupleTime(*): 75.514us
>>> >> > - ScannerThreadsSysTime: 29.984ms
>>> >> > - ScannerThreadsUserTime: 669.884ms
>>> >> > - ScannerThreadsVoluntaryContextSwitches: 1.60K (1597)
>>> >> > - TotalRawHdfsReadTime(*): 44.279ms
>>> >> > - TotalReadThroughput: 24.53 MB/sec
>>> >> > Instance c843dae48ce9b289:6ad3ffc000000002
>>> >> > (host=aopr-dhc003.lpdomain.com:22000):(Total: 351.014ms, non-child:
>>> >> > 1.641ms,
>>> >> > % non-child: 0.47%)
>>> >> > Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>>> >> > 0:35/20.89 MB 7:44
>>> <https://maps.google.com/?q=MB+7:44&entry=gmail&source=g>/26.64 MB 11:27
>>> <https://maps.google.com/?q=MB+11:27&entry=gmail&source=g>/17.21 MB 9:32
>>> <https://maps.google.com/?q=MB+9:32&entry=gmail&source=g>/18.61 MB 5:25
>>> <https://maps.google.com/?q=MB+5:25&entry=gmail&source=g>/14.56 MB
>>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+2:38&entry=gmail&source=g>
>>> >> > 2:38
>>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+2:38&entry=gmail&source=g>/24.32
>>> MB 6:31 <https://maps.google.com/?q=MB+6:31&entry=gmail&source=g>/19.85 MB
>>> 8:50 <https://maps.google.com/?q=MB+8:50&entry=gmail&source=g>/30.76 MB
>>> 10:41 <https://maps.google.com/?q=MB+10:41&entry=gmail&source=g>/25.92 MB
>>> 3:31 <https://maps.google.com/?q=MB+3:31&entry=gmail&source=g>/19.78 MB
>>> >> > 4:60/37.59 MB 1:44/27.19 MB
>>> >> > MemoryUsage(500.000ms): 1.16 MB
>>> >> > ThreadUsage(500.000ms): 25
>>> >> > - AverageThreadTokens: 25.00
>>> >> > - BloomFilterBytes: 0
>>> >> > - PeakMemoryUsage: 2.54 MB (2662753)
>>> >> > - PeakReservation: 0
>>> >> > - PeakUsedReservation: 0
>>> >> > - PerHostPeakMemUsage: 2.54 MB (2662753)
>>> >> > - RowsProduced: 1 (1)
>>> >> > - TotalNetworkReceiveTime: 0.000ns
>>> >> > - TotalNetworkSendTime: 342.746us
>>> >> > - TotalStorageWaitTime: 4s796ms
>>> >> > - TotalThreadsInvoluntaryContextSwitches: 93 (93)
>>> >> > - TotalThreadsTotalWallClockTime: 7s286ms
>>> >> > - TotalThreadsSysTime: 27.988ms
>>> >> > - TotalThreadsUserTime: 671.887ms
>>> >> > - TotalThreadsVoluntaryContextSwitches: 2.08K (2079)
>>> >> > Fragment Instance Lifecycle Timings:
>>> >> > - ExecTime: 524.006us
>>> >> > - ExecTreeExecTime: 7.920us
>>> >> > - OpenTime: 327.802ms
>>> >> > - ExecTreeOpenTime: 307.154ms
>>> >> > - PrepareTime: 22.617ms
>>> >> > - ExecTreePrepareTime: 2.650ms
>>> >> > DataStreamSender (dst_id=2):(Total: 348.990us, non-child:
>>> >> > 348.990us,
>>> >> > % non-child: 100.00%)
>>> >> > - BytesSent: 16.00 B (16)
>>> >> > - NetworkThroughput(*): 354.47 KB/sec
>>> >> > - OverallThroughput: 44.77 KB/sec
>>> >> > - PeakMemoryUsage: 16.00 KB (16384)
>>> >> > - RowsReturned: 1 (1)
>>> >> > - SerializeBatchTime: 15.602us
>>> >> > - TransmitDataRPCTime: 44.080us
>>> >> > - UncompressedRowBatchSize: 16.00 B (16)
>>> >> > CodeGen:(Total: 39.215ms, non-child: 39.215ms, % non-child:
>>> >> > 100.00%)
>>> >> > - CodegenTime: 1.183ms
>>> >> > - CompileTime: 7.168ms
>>> >> > - LoadTime: 0.000ns
>>> >> > - ModuleBitcodeSize: 1.86 MB (1950880)
>>> >> > - NumFunctions: 15 (15)
>>> >> > - NumInstructions: 214 (214)
>>> >> > - OptimizationTime: 12.929ms
>>> >> > - PeakMemoryUsage: 107.00 KB (109568)
>>> >> > - PrepareTime: 18.453ms
>>> >> > AGGREGATION_NODE (id=1):(Total: 309.807ms, non-child:
>>> 948.677us,
>>> >> > %
>>> >> > non-child: 0.31%)
>>> >> > ExecOption: Codegen Enabled
>>> >> > - BuildTime: 1.552ms
>>> >> > - GetResultsTime: 0.000ns
>>> >> > - HTResizeTime: 0.000ns
>>> >> > - HashBuckets: 0 (0)
>>> >> > - LargestPartitionPercent: 0 (0)
>>> >> > - MaxPartitionLevel: 0 (0)
>>> >> > - NumRepartitions: 0 (0)
>>> >> > - PartitionsCreated: 0 (0)
>>> >> > - PeakMemoryUsage: 20.00 KB (20480)
>>> >> > - RowsRepartitioned: 0 (0)
>>> >> > - RowsReturned: 1 (1)
>>> >> > - RowsReturnedRate: 3.00 /sec
>>> >> > - SpilledPartitions: 0 (0)
>>> >> > HDFS_SCAN_NODE (id=0):(Total: 308.859ms, non-child:
>>> 308.859ms, %
>>> >> > non-child: 100.00%)
>>> >> > Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>>> >> > 0:35/20.89 MB 7:44
>>> <https://maps.google.com/?q=MB+7:44&entry=gmail&source=g>/26.64 MB 11:27
>>> <https://maps.google.com/?q=MB+11:27&entry=gmail&source=g>/17.21 MB 9:32
>>> <https://maps.google.com/?q=MB+9:32&entry=gmail&source=g>/18.61 MB 5:25
>>> <https://maps.google.com/?q=MB+5:25&entry=gmail&source=g>/14.56 MB
>>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+2:38&entry=gmail&source=g>
>>> >> > 2:38
>>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+2:38&entry=gmail&source=g>/24.32
>>> MB 6:31 <https://maps.google.com/?q=MB+6:31&entry=gmail&source=g>/19.85 MB
>>> 8:50 <https://maps.google.com/?q=MB+8:50&entry=gmail&source=g>/30.76 MB
>>> 10:41 <https://maps.google.com/?q=MB+10:41&entry=gmail&source=g>/25.92 MB
>>> 3:31 <https://maps.google.com/?q=MB+3:31&entry=gmail&source=g>/19.78 MB
>>> >> > 4:60/37.59 MB 1:44/27.19 MB
>>> >> > ExecOption: PARQUET Codegen Enabled, Codegen enabled: 458
>>> out
>>> >> > of
>>> >> > 458
>>> >> > Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0%
>>> >> > 4:0%
>>> >> > 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
>>> >> > File Formats: PARQUET/Unknown(Skipped):458
>>> >> > BytesRead(500.000ms): 36.74 MB
>>> >> > - FooterProcessingTime: (Avg: 11.502ms ; Min: 3.668ms ;
>>> Max:
>>> >> > 23.564ms ; Number of samples: 458)
>>> >> > - AverageHdfsReadThreadConcurrency: 0.00
>>> >> > - AverageScannerThreadConcurrency: 24.00
>>> >> > - BytesRead: 44.51 MB (46668432)
>>> >> > - BytesReadDataNodeCache: 0
>>> >> > - BytesReadLocal: 44.51 MB (46668432)
>>> >> > - BytesReadRemoteUnexpected: 0
>>> >> > - BytesReadShortCircuit: 44.51 MB (46668432)
>>> >> > - CachedFileHandlesHitCount: 0 (0)
>>> >> > - CachedFileHandlesMissCount: 458 (458)
>>> >> > - CollectionItemsRead: 0 (0)
>>> >> > - DecompressionTime: 0.000ns
>>> >> > - MaxCompressedTextFileLength: 0
>>> >> > - NumColumns: 0 (0)
>>> >> > - NumDictFilteredRowGroups: 0 (0)
>>> >> > - NumDisksAccessed: 12 (12)
>>> >> > - NumRowGroups: 458 (458)
>>> >> > - NumScannerThreadsStarted: 24 (24)
>>> >> > - NumScannersWithNoReads: 0 (0)
>>> >> > - NumStatsFilteredRowGroups: 0 (0)
>>> >> > - PeakMemoryUsage: 2.50 MB (2621440)
>>> >> > - PerReadThreadRawHdfsThroughput: 1.26 GB/sec
>>> >> > - RemoteScanRanges: 0 (0)
>>> >> > - RowBatchQueueGetWaitTime: 295.595ms
>>> >> > - RowBatchQueuePutWaitTime: 0.000ns
>>> >> > - RowsRead: 0 (0)
>>> >> > - RowsReturned: 458 (458)
>>> >> > - RowsReturnedRate: 1.48 K/sec
>>> >> > - ScanRangesComplete: 458 (458)
>>> >> > - ScannerThreadsInvoluntaryContextSwitches: 91 (91)
>>> >> > - ScannerThreadsTotalWallClockTime: 6s958ms
>>> >> > - MaterializeTupleTime(*): 71.630us
>>> >> > - ScannerThreadsSysTime: 16.989ms
>>> >> > - ScannerThreadsUserTime: 645.891ms
>>> >> > - ScannerThreadsVoluntaryContextSwitches: 1.63K (1630)
>>> >> > - TotalRawHdfsReadTime(*): 34.377ms
>>> >> > - TotalReadThroughput: 73.49 MB/sec
>>> >> > Instance c843dae48ce9b289:6ad3ffc000000001
>>> >> > (host=aopr-dhc005.lpdomain.com:22000):(Total: 308.195ms, non-child:
>>> >> > 1.784ms,
>>> >> > % non-child: 0.58%)
>>> >> > Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>>> >> > 2:46/28.97 MB 8:39
>>> <https://maps.google.com/?q=MB+8:39&entry=gmail&source=g>/24.34 MB 10:33
>>> <https://maps.google.com/?q=MB+10:33&entry=gmail&source=g>/21.03 MB 3:43
>>> <https://maps.google.com/?q=MB+3:43&entry=gmail&source=g>/26.50 MB 5:32
>>> <https://maps.google.com/?q=MB+5:32&entry=gmail&source=g>/20.27 MB
>>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+9:32&entry=gmail&source=g>
>>> >> > 9:32
>>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+9:32&entry=gmail&source=g>/19.86
>>> MB 6:25 <https://maps.google.com/?q=MB+6:25&entry=gmail&source=g>/16.01 MB
>>> 7:41 <https://maps.google.com/?q=MB+7:41&entry=gmail&source=g>/24.77 MB
>>> 1:42/25.01 MB 0:55/32.51 MB
>>> >> > 4:40/24.29 MB 11:33
>>> <https://maps.google.com/?q=MB+11:33&entry=gmail&source=g>/20.77 MB
>>> >> > - AverageThreadTokens: 0.00
>>> >> > - BloomFilterBytes: 0
>>> >> > - PeakMemoryUsage: 2.56 MB (2683241)
>>> >> > - PeakReservation: 0
>>> >> > - PeakUsedReservation: 0
>>> >> > - PerHostPeakMemUsage: 2.56 MB (2683241)
>>> >> > - RowsProduced: 1 (1)
>>> >> > - TotalNetworkReceiveTime: 0.000ns
>>> >> > - TotalNetworkSendTime: 292.765us
>>> >> > - TotalStorageWaitTime: 4s482ms
>>> >> > - TotalThreadsInvoluntaryContextSwitches: 113 (113)
>>> >> > - TotalThreadsTotalWallClockTime: 6s310ms
>>> >> > - TotalThreadsSysTime: 35.983ms
>>> >> > - TotalThreadsUserTime: 682.884ms
>>> >> > - TotalThreadsVoluntaryContextSwitches: 2.14K (2143)
>>> >> > Fragment Instance Lifecycle Timings:
>>> >> > - ExecTime: 532.572us
>>> >> > - ExecTreeExecTime: 8.377us
>>> >> > - OpenTime: 280.241ms
>>> >> > - ExecTreeOpenTime: 258.652ms
>>> >> > - PrepareTime: 27.342ms
>>> >> > - ExecTreePrepareTime: 2.863ms
>>> >> > DataStreamSender (dst_id=2):(Total: 309.794us, non-child:
>>> >> > 309.794us,
>>> >> > % non-child: 100.00%)
>>> >> > - BytesSent: 16.00 B (16)
>>> >> > - NetworkThroughput(*): 356.91 KB/sec
>>> >> > - OverallThroughput: 50.44 KB/sec
>>> >> > - PeakMemoryUsage: 16.00 KB (16384)
>>> >> > - RowsReturned: 1 (1)
>>> >> > - SerializeBatchTime: 14.262us
>>> >> > - TransmitDataRPCTime: 43.778us
>>> >> > - UncompressedRowBatchSize: 16.00 B (16)
>>> >> > CodeGen:(Total: 44.584ms, non-child: 44.584ms, % non-child:
>>> >> > 100.00%)
>>> >> > - CodegenTime: 1.382ms
>>> >> > - CompileTime: 7.346ms
>>> >> > - LoadTime: 0.000ns
>>> >> > - ModuleBitcodeSize: 1.86 MB (1950880)
>>> >> > - NumFunctions: 15 (15)
>>> >> > - NumInstructions: 214 (214)
>>> >> > - OptimizationTime: 13.462ms
>>> >> > - PeakMemoryUsage: 107.00 KB (109568)
>>> >> > - PrepareTime: 23.006ms
>>> >> > AGGREGATION_NODE (id=1):(Total: 261.516ms, non-child:
>>> 836.656us,
>>> >> > %
>>> >> > non-child: 0.32%)
>>> >> > ExecOption: Codegen Enabled
>>> >> > - BuildTime: 1.280ms
>>> >> > - GetResultsTime: 0.000ns
>>> >> > - HTResizeTime: 0.000ns
>>> >> > - HashBuckets: 0 (0)
>>> >> > - LargestPartitionPercent: 0 (0)
>>> >> > - MaxPartitionLevel: 0 (0)
>>> >> > - NumRepartitions: 0 (0)
>>> >> > - PartitionsCreated: 0 (0)
>>> >> > - PeakMemoryUsage: 20.00 KB (20480)
>>> >> > - RowsRepartitioned: 0 (0)
>>> >> > - RowsReturned: 1 (1)
>>> >> > - RowsReturnedRate: 3.00 /sec
>>> >> > - SpilledPartitions: 0 (0)
>>> >> > HDFS_SCAN_NODE (id=0):(Total: 260.680ms, non-child:
>>> 260.680ms, %
>>> >> > non-child: 100.00%)
>>> >> > Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>>> >> > 2:46/28.97 MB 8:39
>>> <https://maps.google.com/?q=MB+8:39&entry=gmail&source=g>/24.34 MB 10:33
>>> <https://maps.google.com/?q=MB+10:33&entry=gmail&source=g>/21.03 MB 3:43
>>> <https://maps.google.com/?q=MB+3:43&entry=gmail&source=g>/26.50 MB 5:32
>>> <https://maps.google.com/?q=MB+5:32&entry=gmail&source=g>/20.27 MB
>>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+9:32&entry=gmail&source=g>
>>> >> > 9:32
>>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+9:32&entry=gmail&source=g>/19.86
>>> MB 6:25 <https://maps.google.com/?q=MB+6:25&entry=gmail&source=g>/16.01 MB
>>> 7:41 <https://maps.google.com/?q=MB+7:41&entry=gmail&source=g>/24.77 MB
>>> 1:42/25.01 MB 0:55/32.51 MB
>>> >> > 4:40/24.29 MB 11:33
>>> <https://maps.google.com/?q=MB+11:33&entry=gmail&source=g>/20.77 MB
>>> >> > ExecOption: PARQUET Codegen Enabled, Codegen enabled: 461
>>> out
>>> >> > of
>>> >> > 461
>>> >> > Hdfs Read Thread Concurrency Bucket: 0:0% 1:0% 2:0% 3:0%
>>> 4:0%
>>> >> > 5:0%
>>> >> > 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
>>> >> > File Formats: PARQUET/Unknown(Skipped):461
>>> >> > - FooterProcessingTime: (Avg: 10.793ms ; Min: 3.087ms ;
>>> Max:
>>> >> > 20.253ms ; Number of samples: 461)
>>> >> > - AverageHdfsReadThreadConcurrency: 0.00
>>> >> > - AverageScannerThreadConcurrency: 0.00
>>> >> > - BytesRead: 44.73 MB (46906407)
>>> >> > - BytesReadDataNodeCache: 0
>>> >> > - BytesReadLocal: 44.73 MB (46906407)
>>> >> > - BytesReadRemoteUnexpected: 0
>>> >> > - BytesReadShortCircuit: 44.73 MB (46906407)
>>> >> > - CachedFileHandlesHitCount: 0 (0)
>>> >> > - CachedFileHandlesMissCount: 461 (461)
>>> >> > - CollectionItemsRead: 0 (0)
>>> >> > - DecompressionTime: 0.000ns
>>> >> > - MaxCompressedTextFileLength: 0
>>> >> > - NumColumns: 0 (0)
>>> >> > - NumDictFilteredRowGroups: 0 (0)
>>> >> > - NumDisksAccessed: 12 (12)
>>> >> > - NumRowGroups: 461 (461)
>>> >> > - NumScannerThreadsStarted: 24 (24)
>>> >> > - NumScannersWithNoReads: 0 (0)
>>> >> > - NumStatsFilteredRowGroups: 0 (0)
>>> >> > - PeakMemoryUsage: 2.52 MB (2641928)
>>> >> > - PerReadThreadRawHdfsThroughput: 1.31 GB/sec
>>> >> > - RemoteScanRanges: 0 (0)
>>> >> > - RowBatchQueueGetWaitTime: 246.691ms
>>> >> > - RowBatchQueuePutWaitTime: 0.000ns
>>> >> > - RowsRead: 0 (0)
>>> >> > - RowsReturned: 461 (461)
>>> >> > - RowsReturnedRate: 1.77 K/sec
>>> >> > - ScanRangesComplete: 461 (461)
>>> >> > - ScannerThreadsInvoluntaryContextSwitches: 109 (109)
>>> >> > - ScannerThreadsTotalWallClockTime: 6s030ms
>>> >> > - MaterializeTupleTime(*): 68.735us
>>> >> > - ScannerThreadsSysTime: 33.983ms
>>> >> > - ScannerThreadsUserTime: 647.889ms
>>> >> > - ScannerThreadsVoluntaryContextSwitches: 1.70K (1696)
>>> >> > - TotalRawHdfsReadTime(*): 33.405ms
>>> >> > - TotalReadThroughput: 0.00 /sec
>>> >> > Instance c843dae48ce9b289:6ad3ffc000000003
>>> >> > (host=aopr-dhc002.lpdomain.com:22000):(Total: 302.116ms, non-child:
>>> >> > 1.604ms,
>>> >> > % non-child: 0.53%)
>>> >> > Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>>> >> > 0:27/16.21 MB 11:33
>>> <https://maps.google.com/?q=MB+11:33&entry=gmail&source=g>/21.23 MB 6:36
>>> <https://maps.google.com/?q=MB+6:36&entry=gmail&source=g>/23.15 MB 10:41
>>> <https://maps.google.com/?q=MB+10:41&entry=gmail&source=g>/25.09 MB
>>> 1:41/24.13 MB
>>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+9:32&entry=gmail&source=g>
>>> >> > 9:32
>>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+9:32&entry=gmail&source=g>/18.79
>>> MB 5:41 <https://maps.google.com/?q=MB+5:41&entry=gmail&source=g>/25.31 MB
>>> 2:42 <https://maps.google.com/?q=MB+2:42&entry=gmail&source=g>/26.49 MB
>>> 7:36 <https://maps.google.com/?q=MB+7:36&entry=gmail&source=g>/22.87 MB
>>> 3:51 <https://maps.google.com/?q=MB+3:51&entry=gmail&source=g>/31.00 MB
>>> >> > 4:41/25.99 MB 8:38
>>> <https://maps.google.com/?q=MB+8:38&entry=gmail&source=g>/23.79 MB
>>> >> > MemoryUsage(500.000ms): 700.35 KB
>>> >> > ThreadUsage(500.000ms): 25
>>> >> > - AverageThreadTokens: 25.00
>>> >> > - BloomFilterBytes: 0
>>> >> > - PeakMemoryUsage: 2.54 MB (2662753)
>>> >> > - PeakReservation: 0
>>> >> > - PeakUsedReservation: 0
>>> >> > - PerHostPeakMemUsage: 2.54 MB (2662753)
>>> >> > - RowsProduced: 1 (1)
>>> >> > - TotalNetworkReceiveTime: 0.000ns
>>> >> > - TotalNetworkSendTime: 300.675us
>>> >> > - TotalStorageWaitTime: 4s661ms
>>> >> > - TotalThreadsInvoluntaryContextSwitches: 90 (90)
>>> >> > - TotalThreadsTotalWallClockTime: 6s256ms
>>> >> > - TotalThreadsSysTime: 30.983ms
>>> >> > - TotalThreadsUserTime: 709.880ms
>>> >> > - TotalThreadsVoluntaryContextSwitches: 2.17K (2175)
>>> >> > Fragment Instance Lifecycle Timings:
>>> >> > - ExecTime: 492.627us
>>> >> > - ExecTreeExecTime: 8.191us
>>> >> > - OpenTime: 277.922ms
>>> >> > - ExecTreeOpenTime: 255.395ms
>>> >> > - PrepareTime: 23.647ms
>>> >> > - ExecTreePrepareTime: 2.243ms
>>> >> > DataStreamSender (dst_id=2):(Total: 194.517us, non-child:
>>> >> > 194.517us,
>>> >> > % non-child: 100.00%)
>>> >> > - BytesSent: 16.00 B (16)
>>> >> > - NetworkThroughput(*): 371.86 KB/sec
>>> >> > - OverallThroughput: 80.33 KB/sec
>>> >> > - PeakMemoryUsage: 16.00 KB (16384)
>>> >> > - RowsReturned: 1 (1)
>>> >> > - SerializeBatchTime: 14.035us
>>> >> > - TransmitDataRPCTime: 42.019us
>>> >> > - UncompressedRowBatchSize: 16.00 B (16)
>>> >> > CodeGen:(Total: 42.676ms, non-child: 42.676ms, % non-child:
>>> >> > 100.00%)
>>> >> > - CodegenTime: 1.236ms
>>> >> > - CompileTime: 7.717ms
>>> >> > - LoadTime: 0.000ns
>>> >> > - ModuleBitcodeSize: 1.86 MB (1950880)
>>> >> > - NumFunctions: 15 (15)
>>> >> > - NumInstructions: 214 (214)
>>> >> > - OptimizationTime: 14.130ms
>>> >> > - PeakMemoryUsage: 107.00 KB (109568)
>>> >> > - PrepareTime: 20.109ms
>>> >> > AGGREGATION_NODE (id=1):(Total: 257.640ms, non-child:
>>> 1.541ms, %
>>> >> > non-child: 0.60%)
>>> >> > ExecOption: Codegen Enabled
>>> >> > - BuildTime: 1.252ms
>>> >> > - GetResultsTime: 0.000ns
>>> >> > - HTResizeTime: 0.000ns
>>> >> > - HashBuckets: 0 (0)
>>> >> > - LargestPartitionPercent: 0 (0)
>>> >> > - MaxPartitionLevel: 0 (0)
>>> >> > - NumRepartitions: 0 (0)
>>> >> > - PartitionsCreated: 0 (0)
>>> >> > - PeakMemoryUsage: 20.00 KB (20480)
>>> >> > - RowsRepartitioned: 0 (0)
>>> >> > - RowsReturned: 1 (1)
>>> >> > - RowsReturnedRate: 3.00 /sec
>>> >> > - SpilledPartitions: 0 (0)
>>> >> > HDFS_SCAN_NODE (id=0):(Total: 256.099ms, non-child:
>>> 256.099ms, %
>>> >> > non-child: 100.00%)
>>> >> > Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>>> >> > 0:27/16.21 MB 11:33
>>> <https://maps.google.com/?q=MB+11:33&entry=gmail&source=g>/21.23 MB 6:36
>>> <https://maps.google.com/?q=MB+6:36&entry=gmail&source=g>/23.15 MB 10:41
>>> <https://maps.google.com/?q=MB+10:41&entry=gmail&source=g>/25.09 MB
>>> 1:41/24.13 MB
>>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+9:32&entry=gmail&source=g>
>>> >> > 9:32
>>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+9:32&entry=gmail&source=g>/18.79
>>> MB 5:41 <https://maps.google.com/?q=MB+5:41&entry=gmail&source=g>/25.31 MB
>>> 2:42 <https://maps.google.com/?q=MB+2:42&entry=gmail&source=g>/26.49 MB
>>> 7:36 <https://maps.google.com/?q=MB+7:36&entry=gmail&source=g>/22.87 MB
>>> 3:51 <https://maps.google.com/?q=MB+3:51&entry=gmail&source=g>/31.00 MB
>>> >> > 4:41/25.99 MB 8:38
>>> <https://maps.google.com/?q=MB+8:38&entry=gmail&source=g>/23.79 MB
>>> >> > ExecOption: PARQUET Codegen Enabled, Codegen enabled: 459
>>> out
>>> >> > of
>>> >> > 459
>>> >> > Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0%
>>> >> > 4:0%
>>> >> > 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
>>> >> > File Formats: PARQUET/Unknown(Skipped):459
>>> >> > BytesRead(500.000ms): 43.15 MB
>>> >> > - FooterProcessingTime: (Avg: 11.257ms ; Min: 3.989ms ;
>>> Max:
>>> >> > 23.004ms ; Number of samples: 459)
>>> >> > - AverageHdfsReadThreadConcurrency: 0.00
>>> >> > - AverageScannerThreadConcurrency: 24.00
>>> >> > - BytesRead: 44.52 MB (46678031)
>>> >> > - BytesReadDataNodeCache: 0
>>> >> > - BytesReadLocal: 44.52 MB (46678031)
>>> >> > - BytesReadRemoteUnexpected: 0
>>> >> > - BytesReadShortCircuit: 44.52 MB (46678031)
>>> >> > - CachedFileHandlesHitCount: 0 (0)
>>> >> > - CachedFileHandlesMissCount: 459 (459)
>>> >> > - CollectionItemsRead: 0 (0)
>>> >> > - DecompressionTime: 0.000ns
>>> >> > - MaxCompressedTextFileLength: 0
>>> >> > - NumColumns: 0 (0)
>>> >> > - NumDictFilteredRowGroups: 0 (0)
>>> >> > - NumDisksAccessed: 12 (12)
>>> >> > - NumRowGroups: 459 (459)
>>> >> > - NumScannerThreadsStarted: 24 (24)
>>> >> > - NumScannersWithNoReads: 0 (0)
>>> >> > - NumStatsFilteredRowGroups: 0 (0)
>>> >> > - PeakMemoryUsage: 2.50 MB (2621440)
>>> >> > - PerReadThreadRawHdfsThroughput: 1.33 GB/sec
>>> >> > - RemoteScanRanges: 0 (0)
>>> >> > - RowBatchQueueGetWaitTime: 243.426ms
>>> >> > - RowBatchQueuePutWaitTime: 0.000ns
>>> >> > - RowsRead: 0 (0)
>>> >> > - RowsReturned: 459 (459)
>>> >> > - RowsReturnedRate: 1.79 K/sec
>>> >> > - ScanRangesComplete: 459 (459)
>>> >> > - ScannerThreadsInvoluntaryContextSwitches: 85 (85)
>>> >> > - ScannerThreadsTotalWallClockTime: 5s978ms
>>> >> > - MaterializeTupleTime(*): 67.997us
>>> >> > - ScannerThreadsSysTime: 22.984ms
>>> >> > - ScannerThreadsUserTime: 678.884ms
>>> >> > - ScannerThreadsVoluntaryContextSwitches: 1.72K (1724)
>>> >> > - TotalRawHdfsReadTime(*): 32.626ms
>>> >> > - TotalReadThroughput: 86.30 MB/sec
>>> >> > Instance c843dae48ce9b289:6ad3ffc000000005
>>> >> > (host=aopr-dhc001.lpdomain.com:22000):(Total: 298.093ms, non-child:
>>> >> > 1.242ms,
>>> >> > % non-child: 0.42%)
>>> >> > Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>>> >> > 9:55/35.38 MB 0:37/24.23 MB 5:37
>>> <https://maps.google.com/?q=MB+5:37&entry=gmail&source=g>/23.31 MB 11:22
>>> <https://maps.google.com/?q=MB+11:22&entry=gmail&source=g>/13.79 MB 8:30
>>> <https://maps.google.com/?q=MB+8:30&entry=gmail&source=g>/18.53 MB
>>> >> > 4:33/20.45 MB 1:38/24.06 MB 10:29
>>> <https://maps.google.com/?q=MB+10:29&entry=gmail&source=g>/18.40 MB 3:39
>>> <https://maps.google.com/?q=MB+3:39&entry=gmail&source=g>/24.35 MB 2:41
>>> <https://maps.google.com/?q=MB+2:41&entry=gmail&source=g>/25.45 MB
>>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+6:42&entry=gmail&source=g>
>>> >> > 6:42
>>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+6:42&entry=gmail&source=g>/27.03
>>> MB 7:47 <https://maps.google.com/?q=MB+7:47&entry=gmail&source=g>/29.22
>>> MB
>>> >> > - AverageThreadTokens: 0.00
>>> >> > - BloomFilterBytes: 0
>>> >> > - PeakMemoryUsage: 2.66 MB (2793825)
>>> >> > - PeakReservation: 0
>>> >> > - PeakUsedReservation: 0
>>> >> > - PerHostPeakMemUsage: 2.68 MB (2806113)
>>> >> > - RowsProduced: 1 (1)
>>> >> > - TotalNetworkReceiveTime: 0.000ns
>>> >> > - TotalNetworkSendTime: 198.894us
>>> >> > - TotalStorageWaitTime: 4s374ms
>>> >> > - TotalThreadsInvoluntaryContextSwitches: 110 (110)
>>> >> > - TotalThreadsTotalWallClockTime: 6s201ms
>>> >> > - TotalThreadsSysTime: 29.983ms
>>> >> > - TotalThreadsUserTime: 642.892ms
>>> >> > - TotalThreadsVoluntaryContextSwitches: 2.17K (2174)
>>> >> > Fragment Instance Lifecycle Timings:
>>> >> > - ExecTime: 350.873us
>>> >> > - ExecTreeExecTime: 8.173us
>>> >> > - OpenTime: 275.347ms
>>> >> > - ExecTreeOpenTime: 253.425ms
>>> >> > - PrepareTime: 22.341ms
>>> >> > - ExecTreePrepareTime: 2.251ms
>>> >> > DataStreamSender (dst_id=2):(Total: 164.394us, non-child:
>>> >> > 164.394us,
>>> >> > % non-child: 100.00%)
>>> >> > - BytesSent: 16.00 B (16)
>>> >> > - NetworkThroughput(*): 493.42 KB/sec
>>> >> > - OverallThroughput: 95.05 KB/sec
>>> >> > - PeakMemoryUsage: 16.00 KB (16384)
>>> >> > - RowsReturned: 1 (1)
>>> >> > - SerializeBatchTime: 14.262us
>>> >> > - TransmitDataRPCTime: 31.667us
>>> >> > - UncompressedRowBatchSize: 16.00 B (16)
>>> >> > CodeGen:(Total: 41.008ms, non-child: 41.008ms, % non-child:
>>> >> > 100.00%)
>>> >> > - CodegenTime: 1.049ms
>>> >> > - CompileTime: 7.248ms
>>> >> > - LoadTime: 0.000ns
>>> >> > - ModuleBitcodeSize: 1.86 MB (1950880)
>>> >> > - NumFunctions: 15 (15)
>>> >> > - NumInstructions: 214 (214)
>>> >> > - OptimizationTime: 14.018ms
>>> >> > - PeakMemoryUsage: 107.00 KB (109568)
>>> >> > - PrepareTime: 19.080ms
>>> >> > AGGREGATION_NODE (id=1):(Total: 255.678ms, non-child:
>>> 1.046ms, %
>>> >> > non-child: 0.41%)
>>> >> > ExecOption: Codegen Enabled
>>> >> > - BuildTime: 1.015ms
>>> >> > - GetResultsTime: 0.000ns
>>> >> > - HTResizeTime: 0.000ns
>>> >> > - HashBuckets: 0 (0)
>>> >> > - LargestPartitionPercent: 0 (0)
>>> >> > - MaxPartitionLevel: 0 (0)
>>> >> > - NumRepartitions: 0 (0)
>>> >> > - PartitionsCreated: 0 (0)
>>> >> > - PeakMemoryUsage: 20.00 KB (20480)
>>> >> > - RowsRepartitioned: 0 (0)
>>> >> > - RowsReturned: 1 (1)
>>> >> > - RowsReturnedRate: 3.00 /sec
>>> >> > - SpilledPartitions: 0 (0)
>>> >> > HDFS_SCAN_NODE (id=0):(Total: 254.631ms, non-child:
>>> 254.631ms, %
>>> >> > non-child: 100.00%)
>>> >> > Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>>> >> > 9:55/35.38 MB 0:37/24.23 MB 5:37
>>> <https://maps.google.com/?q=MB+5:37&entry=gmail&source=g>/23.31 MB 11:22
>>> <https://maps.google.com/?q=MB+11:22&entry=gmail&source=g>/13.79 MB 8:30
>>> <https://maps.google.com/?q=MB+8:30&entry=gmail&source=g>/18.53 MB
>>> >> > 4:33/20.45 MB 1:38/24.06 MB 10:29
>>> <https://maps.google.com/?q=MB+10:29&entry=gmail&source=g>/18.40 MB 3:39
>>> <https://maps.google.com/?q=MB+3:39&entry=gmail&source=g>/24.35 MB 2:41
>>> <https://maps.google.com/?q=MB+2:41&entry=gmail&source=g>/25.45 MB
>>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+6:42&entry=gmail&source=g>
>>> >> > 6:42
>>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+6:42&entry=gmail&source=g>/27.03
>>> MB 7:47 <https://maps.google.com/?q=MB+7:47&entry=gmail&source=g>/29.22
>>> MB
>>> >> > ExecOption: PARQUET Codegen Enabled, Codegen enabled: 450
>>> out
>>> >> > of
>>> >> > 450
>>> >> > Hdfs Read Thread Concurrency Bucket: 0:0% 1:0% 2:0% 3:0%
>>> 4:0%
>>> >> > 5:0%
>>> >> > 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
>>> >> > File Formats: PARQUET/Unknown(Skipped):450
>>> >> > - FooterProcessingTime: (Avg: 10.727ms ; Min: 3.813ms ;
>>> Max:
>>> >> > 23.755ms ; Number of samples: 450)
>>> >> > - AverageHdfsReadThreadConcurrency: 0.00
>>> >> > - AverageScannerThreadConcurrency: 0.00
>>> >> > - BytesRead: 43.85 MB (45983284)
>>> >> > - BytesReadDataNodeCache: 0
>>> >> > - BytesReadLocal: 43.85 MB (45983284)
>>> >> > - BytesReadRemoteUnexpected: 0
>>> >> > - BytesReadShortCircuit: 43.85 MB (45983284)
>>> >> > - CachedFileHandlesHitCount: 0 (0)
>>> >> > - CachedFileHandlesMissCount: 450 (450)
>>> >> > - CollectionItemsRead: 0 (0)
>>> >> > - DecompressionTime: 0.000ns
>>> >> > - MaxCompressedTextFileLength: 0
>>> >> > - NumColumns: 0 (0)
>>> >> > - NumDictFilteredRowGroups: 0 (0)
>>> >> > - NumDisksAccessed: 12 (12)
>>> >> > - NumRowGroups: 450 (450)
>>> >> > - NumScannerThreadsStarted: 24 (24)
>>> >> > - NumScannersWithNoReads: 0 (0)
>>> >> > - NumStatsFilteredRowGroups: 0 (0)
>>> >> > - PeakMemoryUsage: 2.62 MB (2752512)
>>> >> > - PerReadThreadRawHdfsThroughput: 1.56 GB/sec
>>> >> > - RemoteScanRanges: 0 (0)
>>> >> > - RowBatchQueueGetWaitTime: 242.362ms
>>> >> > - RowBatchQueuePutWaitTime: 0.000ns
>>> >> > - RowsRead: 0 (0)
>>> >> > - RowsReturned: 450 (450)
>>> >> > - RowsReturnedRate: 1.77 K/sec
>>> >> > - ScanRangesComplete: 450 (450)
>>> >> > - ScannerThreadsInvoluntaryContextSwitches: 106 (106)
>>> >> > - ScannerThreadsTotalWallClockTime: 5s926ms
>>> >> > - MaterializeTupleTime(*): 69.447us
>>> >> > - ScannerThreadsSysTime: 26.984ms
>>> >> > - ScannerThreadsUserTime: 607.897ms
>>> >> > - ScannerThreadsVoluntaryContextSwitches: 1.72K (1723)
>>> >> > - TotalRawHdfsReadTime(*): 27.393ms
>>> >> > - TotalReadThroughput: 0.00 /sec
>>> >> > ===================================================
>>> >> >
>>> >> > Coordinator log:
>>> >> >
>>> >> > 01: query (string) = "select count(*) from
>>> dv_entersiteevent_prq_remote
>>> >> > where year=2017 and month=10",
>>> >> > 03: configuration (list) = list<string>[1] {
>>> >> > [0] = "REQUEST_POOL=impreports",
>>> >> > },
>>> >> > 04: hadoop_user (string) = "fawzea",
>>> >> > }
>>> >> > I0208 01:07:06.321281 7154 impala-beeswax-server.cc:459]
>>> >> > TClientRequest.queryOptions: TQueryOptions {
>>> >> > 01: abort_on_error (bool) = false,
>>> >> > 02: max_errors (i32) = 100,
>>> >> > 03: disable_codegen (bool) = false,
>>> >> > 04: batch_size (i32) = 0,
>>> >> > 05: num_nodes (i32) = 0,
>>> >> > 06: max_scan_range_length (i64) = 0,
>>> >> > 07: num_scanner_threads (i32) = 0,
>>> >> > 08: max_io_buffers (i32) = 0,
>>> >> > 09: allow_unsupported_formats (bool) = false,
>>> >> > 10: default_order_by_limit (i64) = -1,
>>> >> > 11: debug_action (string) = "",
>>> >> > 12: mem_limit (i64) = 0,
>>> >> > 13: abort_on_default_limit_exceeded (bool) = false,
>>> >> > 15: hbase_caching (i32) = 0,
>>> >> > 16: hbase_cache_blocks (bool) = false,
>>> >> > 17: parquet_file_size (i64) = 0,
>>> >> > 18: explain_level (i32) = 1,
>>> >> > 19: sync_ddl (bool) = false,
>>> >> > 20: request_pool (string) = "impreports",
>>> >> > 23: disable_cached_reads (bool) = false,
>>> >> > 24: disable_outermost_topn (bool) = false,
>>> >> > 25: rm_initial_mem (i64) = 0,
>>> >> > 26: query_timeout_s (i32) = 0,
>>> >> > 28: appx_count_distinct (bool) = false,
>>> >> > 29: disable_unsafe_spills (bool) = false,
>>> >> > 31: exec_single_node_rows_threshold (i32) = 100,
>>> >> > 32: optimize_partition_key_scans (bool) = false,
>>> >> > 33: replica_preference (i32) = 0,
>>> >> > 34: schedule_random_replica (bool) = false,
>>> >> > 35: scan_node_codegen_threshold (i64) = 1800000,
>>> >> > 36: disable_streaming_preaggregations (bool) = false,
>>> >> > 37: runtime_filter_mode (i32) = 2,
>>> >> > 38: runtime_bloom_filter_size (i32) = 1048576,
>>> >> > 39: runtime_filter_wait_time_ms (i32) = 0,
>>> >> > 40: disable_row_runtime_filtering (bool) = false,
>>> >> > 41: max_num_runtime_filters (i32) = 10,
>>> >> > 42: parquet_annotate_strings_utf8 (bool) = false,
>>> >> > 43: parquet_fallback_schema_resolution (i32) = 0,
>>> >> > 45: s3_skip_insert_staging (bool) = true,
>>> >> > 46: runtime_filter_min_size (i32) = 1048576,
>>> >> > 47: runtime_filter_max_size (i32) = 16777216,
>>> >> > 48: prefetch_mode (i32) = 1,
>>> >> > 49: strict_mode (bool) = false,
>>> >> > 50: scratch_limit (i64) = -1,
>>> >> > 51: enable_expr_rewrites (bool) = true,
>>> >> > 52: decimal_v2 (bool) = false,
>>> >> > 53: parquet_dictionary_filtering (bool) = true,
>>> >> > 54: parquet_array_resolution (i32) = 2,
>>> >> > 55: parquet_read_statistics (bool) = true,
>>> >> > 56: default_join_distribution_mode (i32) = 0,
>>> >> > 57: disable_codegen_rows_threshold (i32) = 50000,
>>> >> > 58: default_spillable_buffer_size (i64) = 2097152,
>>> >> > 59: min_spillable_buffer_size (i64) = 65536,
>>> >> > 60: max_row_size (i64) = 524288,
>>> >> > }
>>> >> > I0208 01:07:06.324410 7154 Frontend.java:909] Compiling query:
>>> select
>>> >> > count(*) from dv_entersiteevent_prq_remote where year=2017 and
>>> month=10
>>> >> > I0208 01:07:06.327203 7154 Frontend.java:948] Compiled query.
>>> >> > I0208 01:07:06.359748 7154 admission-controller.cc:510] Schedule
>>> for
>>> >> > id=5f49e927591a9042:faf8f28600000000 in pool_name=root.impreports
>>> >> > cluster_mem_needed=105.00 MB PoolConfig: max_requests=-1
>>> max_queued=200
>>> >> > max_mem=-1.00 B
>>> >> > I0208 01:07:06.359776 7154 admission-controller.cc:515] Stats:
>>> >> > agg_num_running=0, agg_num_queued=0, agg_mem_reserved=0,
>>> >> > local_host(local_mem_admitted=0, num_admitted_running=0,
>>> num_queued=0,
>>> >> > backend_mem_reserved=0)
>>> >> > I0208 01:07:06.359784 7154 admission-controller.cc:531] Admitted
>>> query
>>> >> > id=5f49e927591a9042:faf8f28600000000
>>> >> > I0208 01:07:06.359807 7154 coordinator.cc:99] Exec()
>>> >> > query_id=5f49e927591a9042:faf8f28600000000 stmt=select count(*)
>>> from
>>> >> > dv_entersiteevent_prq_remote where year=2017 and month=10
>>> >> > I0208 01:07:06.360694 7154 coordinator.cc:357] starting execution
>>> on 5
>>> >> > backends for query 5f49e927591a9042:faf8f28600000000
>>> >> > I0208 01:07:06.363255 63583 impala-internal-service.cc:44]
>>> >> > ExecQueryFInstances(): query_id=5f49e927591a9042:faf8f28600000000
>>> >> > I0208 01:07:06.363282 63583 query-exec-mgr.cc:46]
>>> StartQueryFInstances()
>>> >> > query_id=5f49e927591a9042:faf8f28600000000
>>> >> > coord=aopr-dhc001.lpdomain.com:22000
>>> >> > I0208 01:07:06.363296 63583 query-state.cc:173] Buffer pool limit
>>> for
>>> >> > 5f49e927591a9042:faf8f28600000000: 10802010521
>>> >> > I0208 01:07:06.363364 63583 initial-reservations.cc:60] Successfully
>>> >> > claimed
>>> >> > initial reservations (0) for query 5f49e927591a9042:faf8f28600000
>>> 000
>>> >> > I0208 01:07:06.363544 7155 query-state.cc:286] StartFInstances():
>>> >> > query_id=5f49e927591a9042:faf8f28600000000 #instances=2
>>> >> > I0208 01:07:06.363844 7155 query-state.cc:299] descriptor table for
>>> >> > query=5f49e927591a9042:faf8f28600000000
>>> >> > tuples:
>>> >> > Tuple(id=2 size=0 slots=[] tuple_path=[])
>>> >> > Tuple(id=1 size=8 slots=[Slot(id=2 type=BIGINT col_path=[] offset=0
>>> >> > null=(offset=0 mask=0) slot_idx=0 field_idx=-1)] tuple_path=[])
>>> >> > Tuple(id=0 size=8 slots=[Slot(id=3 type=BIGINT col_path=[] offset=0
>>> >> > null=(offset=0 mask=0) slot_idx=0 field_idx=-1)] tuple_path=[])
>>> >> > I0208 01:07:06.364013 7156 query-state.cc:377] Executing instance.
>>> >> > instance_id=5f49e927591a9042:faf8f28600000000 fragment_idx=0
>>> >> > per_fragment_instance_idx=0 coord_state_idx=0 #in-flight=1
>>> >> > I0208 01:07:06.364135 7157 query-state.cc:377] Executing instance.
>>> >> > instance_id=5f49e927591a9042:faf8f28600000005 fragment_idx=1
>>> >> > per_fragment_instance_idx=4 coord_state_idx=0 #in-flight=2
>>> >> > I0208 01:07:06.364241 7157 hdfs-scan-node.cc:160] Max row batch
>>> queue
>>> >> > size
>>> >> > for scan node '0' in fragment instance
>>> >> > '5f49e927591a9042:faf8f28600000005':
>>> >> > 160
>>> >> > I0208 01:07:06.366775 7154 coordinator.cc:370] started execution
>>> on 5
>>> >> > backends for query 5f49e927591a9042:faf8f28600000000
>>> >> > I0208 01:07:06.385625 7155 query-exec-mgr.cc:149]
>>> ReleaseQueryState():
>>> >> > query_id=5f49e927591a9042:faf8f28600000000 refcnt=4
>>> >> > I0208 01:07:06.671193 7559 impala-server.cc:1796] Connection from
>>> >> > client
>>> >> > ::ffff:10.16.136.3:55598 closed, closing 1 associated session(s)
>>> >> > I0208 01:07:06.676138 544 data-stream-mgr.cc:228] Reduced stream
>>> ID
>>> >> > cache
>>> >> > from 2 items, to 0, eviction took: 0
>>> >> > I0208 01:07:06.677202 544 coordinator.cc:955] Backend completed:
>>> >> > host=aopr-dhc002.lpdomain.com:22000 remaining=4
>>> >> > I0208 01:07:06.677214 544 coordinator-backend-state.cc:229]
>>> >> > query_id=5f49e927591a9042:faf8f28600000000: first in-progress
>>> backend:
>>> >> > aopr-dhc001.lpdomain.com:22000
>>> >> > I0208 01:07:06.684221 7157 query-state.cc:384] Instance completed.
>>> >> > instance_id=5f49e927591a9042:faf8f28600000005 #in-flight=1
>>> status=OK
>>> >> > I0208 01:07:06.684237 7157 query-exec-mgr.cc:149]
>>> ReleaseQueryState():
>>> >> > query_id=5f49e927591a9042:faf8f28600000000 refcnt=3
>>> >> > I0208 01:07:06.700275 520 coordinator.cc:955] Backend completed:
>>> >> > host=aopr-dhc005.lpdomain.com:22000 remaining=3
>>> >> > I0208 01:07:06.700289 520 coordinator-backend-state.cc:229]
>>> >> > query_id=5f49e927591a9042:faf8f28600000000: first in-progress
>>> backend:
>>> >> > aopr-dhc001.lpdomain.com:22000
>>> >> > I0208 01:07:06.704620 65258 coordinator.cc:955] Backend completed:
>>> >> > host=aopr-dhc003.lpdomain.com:22000 remaining=2
>>> >> > I0208 01:07:06.704666 65258 coordinator-backend-state.cc:229]
>>> >> > query_id=5f49e927591a9042:faf8f28600000000: first in-progress
>>> backend:
>>> >> > aopr-dhc001.lpdomain.com:22000
>>> >> > I0208 01:07:06.706041 7156 data-stream-mgr.cc:238]
>>> DeregisterRecvr():
>>> >> > fragment_instance_id=5f49e927591a9042:faf8f28600000000, node=2
>>> >> > I0208 01:07:06.706070 7156 data-stream-recvr.cc:235] cancelled
>>> stream:
>>> >> > fragment_instance_id_=5f49e927591a9042:faf8f28600000000 node_id=2
>>> >> > I0208 01:07:06.706809 64916 coordinator.cc:955] Backend completed:
>>> >> > host=aopr-dhc004.lpdomain.com:22000 remaining=1
>>> >> > I0208 01:07:06.706821 64916 coordinator-backend-state.cc:229]
>>> >> > query_id=5f49e927591a9042:faf8f28600000000: first in-progress
>>> backend:
>>> >> > aopr-dhc001.lpdomain.com:22000
>>> >> > I0208 01:07:06.789568 7154 impala-beeswax-server.cc:193]
>>> >> > get_results_metadata(): query_id=5f49e927591a9042:faf8f28600000000
>>> >> > I0208 01:07:06.790439 7154 coordinator.cc:789] Coordinator waiting
>>> for
>>> >> > backends to finish, 1 remaining
>>> >> > I0208 01:07:06.791052 7154 coordinator.cc:794] All backends
>>> finished
>>> >> > successfully.
>>> >> > I0208 01:07:06.791086 7154 coordinator.cc:1090] Release admssion
>>> >> > control
>>> >> > resources for query 5f49e927591a9042:faf8f28600000000
>>> >> > I0208 01:07:06.791363 7156 query-state.cc:384] Instance completed.
>>> >> > instance_id=5f49e927591a9042:faf8f28600000000 #in-flight=0
>>> status=OK
>>> >> > I0208 01:07:06.791414 7156 query-exec-mgr.cc:149]
>>> ReleaseQueryState():
>>> >> > query_id=5f49e927591a9042:faf8f28600000000 refcnt=2
>>> >> > I0208 01:07:06.793586 7154 impala-beeswax-server.cc:235] close():
>>> >> > query_id=5f49e927591a9042:faf8f28600000000
>>> >> > I0208 01:07:06.793602 7154 impala-server.cc:992] UnregisterQuery():
>>> >> > query_id=5f49e927591a9042:faf8f28600000000
>>> >> > I0208 01:07:06.793614 7154 impala-server.cc:1075] Cancel():
>>> >> > query_id=5f49e927591a9042:faf8f28600000000
>>> >> > I0208 01:07:06.793624 7154 coordinator.cc:895] Cancel()
>>> >> > query_id=5f49e927591a9042:faf8f28600000000
>>> >> > I0208 01:07:06.793637 7154 coordinator.cc:905] CancelBackends()
>>> >> > query_id=5f49e927591a9042:faf8f28600000000, tried to cancel 0
>>> backends
>>> >> > I0208 01:07:06.802776 7154 query-exec-mgr.cc:149]
>>> ReleaseQueryState():
>>> >> > query_id=5f49e927591a9042:faf8f28600000000 refcnt=1
>>> >> > I0208 01:07:06.814214 7154 impala-server.cc:1796] Connection from
>>> >> > client
>>> >> > ::ffff:10.16.144.131:51012 closed, closing 1 associated session(s)
>>> >> > I0208 01:07:07.258891 7567 impala-server.cc:1796] Connection from
>>> >> > client
>>> >> > ::ffff:10.16.136.2:56375 closed, closing 1 associated session(s)
>>> >> > I0208 01:07:11.600131 7819 impala-server.cc:1796] Connection from
>>> >> > client
>>> >> > ::ffff:10.16.136.3:27460 closed, closing 1 associated session(s)
>>> >> > I0208 01:07:12.262739 7821 impala-server.cc:1796] Connection from
>>> >> > client
>>> >> > ::ffff:10.16.136.2:52672 closed, closing 1 associated session(s)
>>> >> > I0208 01:07:16.665451 7822 impala-server.cc:1796] Connection from
>>> >> > client
>>> >> > ::ffff:10.16.136.3:10693 closed, closing 1 associated session(s)
>>> >> > I0208 01:07:17.235321 7825 impala-server.cc:1796] Connection from
>>> >> > client
>>> >> > ::ffff:10.16.136.2:34752 closed, closing 1 associated session(s)
>>> >> > I0208 01:07:21.410611 5858 impala-beeswax-server.cc:193]
>>> >> > get_results_metadata(): query_id=1d4fbf0f2571f1e9:ea346f7900000000
>>> >> > I0208 01:07:21.414175 5858 impala-beeswax-server.cc:235] close():
>>> >> > query_id=1d4fbf0f2571f1e9:ea346f7900000000
>>> >> > I0208 01:07:21.414185 5858 impala-server.cc:992] UnregisterQuery():
>>> >> > query_id=1d4fbf0f2571f1e9:ea346f7900000000
>>> >> > I0208 01:07:21.414191 5858 impala-server.cc:1075] Cancel():
>>> >> > query_id=1d4fbf0f2571f1e9:ea346f7900000000
>>> >> > I0208 01:07:21.421505 5858 impala-server.cc:1796] Connection from
>>> >> > client
>>> >> > ::ffff:10.26.180.75:60072 closed, closing 1 associated session(s)
>>> >> > I0208 01:07:21.605690 7830 impala-server.cc:1796] Connection from
>>> >> > client
>>> >> > ::ffff:10.16.136.3:46660 closed, closing 1 associated session(s)
>>> >> >
>>> >> >
>>> >> > On Thu, Feb 8, 2018 at 6:20 AM, Matthew Jacobs <mj...@apache.org>
>>> >> > wrote:
>>> >> >>
>>> >> >> Thanks! I think that configuration looks OK.
>>> >> >>
>>> >> >> 1) What version of Impala is this?
>>> >> >>
>>> >> >> 2) Can you send the query profile and the Impala coordinator log
>>> for
>>> >> >> the query where it's not behaving as expected?
>>> >> >>
>>> >> >> > When i'm running the following with fawzea user:
>>> >> >> > set request_pool=impreports;
>>> >> >> > select count(*) from dv_entersiteevent_prq_remote where
>>> year=2017 and
>>> >> >> > month=10;
>>> >> >> > I can see that the query is running with impreports pool.
>>> >> >>
>>> >> >> On Wed, Feb 7, 2018 at 7:59 PM, Fawze Abujaber <fa...@gmail.com>
>>> >> >> wrote:
>>> >> >> > Hi Matt,
>>> >> >> >
>>> >> >> > [root@aopr-dhc001 ~]# cat
>>> >> >> >
>>> >> >> >
>>> >> >> > /var/run/cloudera-scm-agent/process/4558-impala-IMPALAD/impa
>>> la-conf/fair-scheduler.xml
>>> >> >> > <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
>>> >> >> > <allocations>
>>> >> >> > <queue name="root">
>>> >> >> > <weight>1.0</weight>
>>> >> >> > <schedulingPolicy>fair</schedulingPolicy>
>>> >> >> > <queue name="impiaas">
>>> >> >> > <weight>1.0</weight>
>>> >> >> > <schedulingPolicy>fair</schedulingPolicy>
>>> >> >> > <aclSubmitApps>fawzea </aclSubmitApps>
>>> >> >> > </queue>
>>> >> >> > <queue name="impreports">
>>> >> >> > <weight>1.0</weight>
>>> >> >> > <schedulingPolicy>fair</schedulingPolicy>
>>> >> >> > <aclSubmitApps>nira </aclSubmitApps>
>>> >> >> > </queue>
>>> >> >> > </queue>
>>> >> >> > <defaultQueueSchedulingPolicy>fair</
>>> defaultQueueSchedulingPolicy>
>>> >> >> > <queuePlacementPolicy>
>>> >> >> > <rule name="specified" create="false"/>
>>> >> >> > <rule name="secondaryGroupExistingQueue"/>
>>> >> >> > <rule name="default"/>
>>> >> >> > </queuePlacementPolicy>
>>> >> >> > </allocations>
>>> >> >> >
>>> >> >> >
>>> >> >> > On Thu, Feb 8, 2018 at 4:47 AM, Matthew Jacobs <
>>> jacobs.mh@gmail.com>
>>> >> >> > wrote:
>>> >> >> >>
>>> >> >> >> Yes, I saw that, but I'd like to see the actual configuration
>>> that
>>> >> >> >> is
>>> >> >> >> passed to Impala because it may not be specified in the right
>>> way to
>>> >> >> >> accomplish what you're trying to do in "b".
>>> >> >> >>
>>> >> >> >> On Wed, Feb 7, 2018 at 4:32 PM, Fawze Abujaber <
>>> fawzeaj@gmail.com>
>>> >> >> >> wrote:
>>> >> >> >> > Hi Matt,
>>> >> >> >> >
>>> >> >> >> > I created the pools manually and it’s completely nfigurations
>>> as
>>> >> >> >> > shown
>>> >> >> >> > in b.
>>> >> >> >> >
>>> >> >> >> > On Wed, 7 Feb 2018 at 23:17 Matthew Jacobs <
>>> jacobs.mh@gmail.com>
>>> >> >> >> > wrote:
>>> >> >> >> >>
>>> >> >> >> >> How, more specifically, are you configuring these pools? Can
>>> you
>>> >> >> >> >> share
>>> >> >> >> >> the configuration files?
>>> >> >> >> >>
>>> >> >> >> >> On Wed, Feb 7, 2018 at 11:42 AM, Fawze Abujaber
>>> >> >> >> >> <fa...@gmail.com>
>>> >> >> >> >> wrote:
>>> >> >> >> >> > Hello All,
>>> >> >> >> >> >
>>> >> >> >> >> > Will appreciate any insight of what i'm missing in my
>>> >> >> >> >> > configuration
>>> >> >> >> >> > to
>>> >> >> >> >> > get
>>> >> >> >> >> > the impala submission control working as expected ....
>>> >> >> >> >> >
>>> >> >> >> >> > a- Here is my placement Placement Rule:
>>> >> >> >> >> > 1- Use the pool Specified at run time., only if the pool
>>> >> >> >> >> > exists.
>>> >> >> >> >> > 2- Use the pool root.[secondary group].
>>> >> >> >> >> > 3- Use the pool root.default.
>>> >> >> >> >> >
>>> >> >> >> >> > b- Here is the impala resource pools
>>> >> >> >> >> >
>>> >> >> >> >> > Name Max Memory Max Running Queries Max
>>> >> >> >> >> > Queued
>>> >> >> >> >> > Queries
>>> >> >> >> >> > Queue Timeout Default Query Memory Limit Access Control
>>> >> >> >> >> > root.impiaas Unlimited
>>> Unlimited
>>> >> >> >> >> > 200 1 minute(s) No
>>> Default
>>> >> >> >> >> > Custom
>>> >> >> >> >> > root.impreports Unlimited
>>> Unlimited
>>> >> >> >> >> > 200 1 minute(s) No
>>> Default
>>> >> >> >> >> > Custom
>>> >> >> >> >> >
>>> >> >> >> >> >
>>> >> >> >> >> > C- Here is my Access control for root.impiaas pool:
>>> >> >> >> >> >
>>> >> >> >> >> > Allow these users and groups to submit to this pool
>>> >> >> >> >> >
>>> >> >> >> >> > Users: fawzea
>>> >> >> >> >> >
>>> >> >> >> >> > The same for impreports the allowed user and groups to
>>> submit
>>> >> >> >> >> > to
>>> >> >> >> >> > pool
>>> >> >> >> >> > is
>>> >> >> >> >> > only nira.
>>> >> >> >> >> >
>>> >> >> >> >> >
>>> >> >> >> >> > LDAP groups for fawzea user:
>>> >> >> >> >> >
>>> >> >> >> >> > Member in groups:
>>> >> >> >> >> >
>>> >> >> >> >> > - dba
>>> >> >> >> >> > - impiaas
>>> >> >> >> >> > - mysql
>>> >> >> >> >> >
>>> >> >> >> >> > LDAP group for nira user
>>> >> >> >> >> > Member in groups:
>>> >> >> >> >> > - dba
>>> >> >> >> >> > - impreports
>>> >> >> >> >> > - mysql
>>> >> >> >> >> >
>>> >> >> >> >> >
>>> >> >> >> >> >
>>> >> >> >> >> > Impala Queries UI shows:
>>> >> >> >> >> >
>>> >> >> >> >> > 1- select count(*) from dv_entersiteevent_prq_remote where
>>> >> >> >> >> > year=2017
>>> >> >> >> >> >
>>> >> >> >> >> > User: fawzea
>>> >> >> >> >> > Pool: root.impiaas
>>> >> >> >> >> >
>>> >> >> >> >> > 2- select count(*) from dv_entersiteevent_prq_remote where
>>> >> >> >> >> > year=2017
>>> >> >> >> >> > and
>>> >> >> >> >> > month=10
>>> >> >> >> >> > User: nira
>>> >> >> >> >> > Pool: root.impreports
>>> >> >> >> >> >
>>> >> >> >> >> > As shown above that the queries submitted based on the
>>> >> >> >> >> > secondary
>>> >> >> >> >> > group
>>> >> >> >> >> > rule.
>>> >> >> >> >> >
>>> >> >> >> >> > When i'm running the following with fawzea user:
>>> >> >> >> >> >
>>> >> >> >> >> > set request_pool=impreports;
>>> >> >> >> >> > select count(*) from dv_entersiteevent_prq_remote where
>>> >> >> >> >> > year=2017
>>> >> >> >> >> > and
>>> >> >> >> >> > month=10;
>>> >> >> >> >> >
>>> >> >> >> >> > I can see that the query is running with impreports pool.
>>> >> >> >> >> >
>>> >> >> >> >> >
>>> >> >> >> >> > What i'm missing here in my configuration to get this
>>> works as
>>> >> >> >> >> > expected.
>>> >> >> >> >> >
>>> >> >> >
>>> >> >> >
>>> >> >
>>> >> >
>>> >
>>> >
>>>
>>
>>
>
Re: impala submission control
Posted by Matthew Jacobs <ja...@gmail.com>.
As I said, I don't see anything wrong with the configuration so there could
be a bug in Impala 2.11, likely due to IMPALA-5920. Can you file a JIRA?
On Thu, Feb 8, 2018 at 9:42 AM, Fawze Abujaber <fa...@gmail.com> wrote:
> This is not fitting my situation,
>
> I removed even the placement role of the secondary group, and other users
> can still submit to the pool that i restrict it to my user.
>
> Just thinking in loud and think the problem might be in using the
> placement rule:
>
> Use the pool Specified at run time., only if the pool exists.
> Edit
>
>
> as it it the first rule, it always successes.
>
>
> On Thu, Feb 8, 2018 at 7:12 PM, Matthew Jacobs <ja...@gmail.com>
> wrote:
>
>> The aclSubmitApps can specify users or groups, so if the user name is
>> also a group name, that can lead to confusion, e.g.
>> https://issues.apache.org/jira/browse/IMPALA-2578
>> I'm not saying that is necessarily the problem, but something to
>> check. If we still can't find anything configured incorrectly I'd
>> recommend that you file a JIRA.
>>
>> On Thu, Feb 8, 2018 at 9:05 AM, Fawze Abujaber <fa...@gmail.com> wrote:
>> > Hi Matt,
>> >
>> > If this is the situation, how we can explain that when i submit the
>> query it
>> > runs on my secondary group pool and when nira do the same it runs under
>> her
>> > secondary group pool.
>> >
>> > More than this, the issue not related to LDAP group as i'm using the
>> > submission restriction to a user and not a group.
>> >
>> > On Thu, Feb 8, 2018 at 7:00 PM, Matthew Jacobs <ja...@gmail.com>
>> wrote:
>> >>
>> >> As far as I can tell from this information, your configuration looks
>> >> fine. The only other potential issue I can think of is that the
>> >> secondary groups placement rule isn't picking up your LDAP groups
>> >> properly. Does the ACL behavior work if you remove the secondary
>> >> groups rule?
>> >>
>> >> On Wed, Feb 7, 2018 at 10:09 PM, Fawze Abujaber <fa...@gmail.com>
>> wrote:
>> >> > Hi Matt,
>> >> >
>> >> > v2.11.0-cdh5.14.0
>> >> >
>> >> > Query Runtime Profile:
>> >> > Query (id=c843dae48ce9b289:6ad3ffc000000000):
>> >> > Summary:
>> >> > Session ID: ca420ac477772b56:51ae460ce9e44999
>> >> > Session Type: BEESWAX
>> >> > Start Time: 2018-02-08 01:02:01.023558000
>> >> > End Time: 2018-02-08 01:02:01.501506000
>> >> > Query Type: QUERY
>> >> > Query State: FINISHED
>> >> > Query Status: OK
>> >> > Impala Version: impalad version 2.11.0-cdh5.14.0 RELEASE (build
>> >> > d68206561bce6b26762d62c01a78e6cd27aa7690)
>> >> > User: fawzea
>> >> > Connected User: fawzea
>> >> > Delegated User:
>> >> > Network Address: ::ffff:10.16.144.131:50078
>> >> > Default Db: default
>> >> > Sql Statement: select count(*) from dv_entersiteevent_prq_remote
>> >> > where
>> >> > year=2017 and month=10
>> >> > Coordinator: aopr-dhc001.lpdomain.com:22000
>> >> > Query Options (set by configuration): REQUEST_POOL=impreports
>> >> > Query Options (set by configuration and planner):
>> >> > REQUEST_POOL=impreports,MT_DOP=0
>> >> > Plan:
>> >> > ----------------
>> >> > Max Per-Host Resource Reservation: Memory=0B
>> >> > Per-Host Resource Estimates: Memory=21.00MB
>> >> > WARNING: The following tables are missing relevant table and/or
>> column
>> >> > statistics.
>> >> > default.dv_entersiteevent_prq_remote
>> >> >
>> >> > F01:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1
>> >> > | Per-Host Resources: mem-estimate=10.00MB mem-reservation=0B
>> >> > PLAN-ROOT SINK
>> >> > | mem-estimate=0B mem-reservation=0B
>> >> > |
>> >> > 03:AGGREGATE [FINALIZE]
>> >> > | output: count:merge(*)
>> >> > | mem-estimate=10.00MB mem-reservation=0B spill-buffer=2.00MB
>> >> > | tuple-ids=1 row-size=8B cardinality=1
>> >> > |
>> >> > 02:EXCHANGE [UNPARTITIONED]
>> >> > | mem-estimate=0B mem-reservation=0B
>> >> > | tuple-ids=1 row-size=8B cardinality=1
>> >> > |
>> >> > F00:PLAN FRAGMENT [RANDOM] hosts=5 instances=5
>> >> > Per-Host Resources: mem-estimate=11.00MB mem-reservation=0B
>> >> > 01:AGGREGATE
>> >> > | output:
>> >> > sum_init_zero(default.dv_entersiteevent_prq_remote.parquet-stats:
>> >> > num_rows)
>> >> > | mem-estimate=10.00MB mem-reservation=0B spill-buffer=2.00MB
>> >> > | tuple-ids=1 row-size=8B cardinality=1
>> >> > |
>> >> > 00:SCAN HDFS [default.dv_entersiteevent_prq_remote, RANDOM]
>> >> > partitions=31/441 files=2292 size=1.39GB
>> >> > stats-rows=unavailable extrapolated-rows=disabled
>> >> > table stats: rows=unavailable size=unavailable
>> >> > column stats: all
>> >> > mem-estimate=1.00MB mem-reservation=0B
>> >> > tuple-ids=0 row-size=8B cardinality=unavailable
>> >> > ----------------
>> >> > Estimated Per-Host Mem: 22020096
>> >> > Tables Missing Stats: default.dv_entersiteevent_prq_remote
>> >> > Per Host Min Reservation: aopr-dhc001.lpdomain.com:22000(0)
>> >> > aopr-dhc002.lpdomain.com:22000(0) aopr-dhc003.lpdomain.com:22000(0)
>> >> > aopr-dhc004.lpdomain.com:22000(0) aopr-dhc005.lpdomain.com:22000(0)
>> >> > Request Pool: root.impreports
>> >> > Admission result: Admitted immediately
>> >> > ExecSummary:
>> >> > Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak
>> Mem
>> >> > Est. Peak Mem Detail
>> >> >
>> >> > ------------------------------------------------------------
>> ------------------------------------------------------------------
>> >> > 03:AGGREGATE 1 222.958us 222.958us 1 1
>> 36.00 KB
>> >> > 10.00 MB FINALIZE
>> >> > 02:EXCHANGE 1 357.508ms 357.508ms 5 1
>> 0
>> >> > 0 UNPARTITIONED
>> >> > 01:AGGREGATE 5 1.027ms 1.541ms 5 1
>> 20.00 KB
>> >> > 10.00 MB
>> >> > 00:SCAN HDFS 5 282.797ms 333.715ms 2.29K -1
>> 2.62 MB
>> >> > 1.00 MB default.dv_entersiteevent_prq_remote
>> >> > Errors:
>> >> > Planner Timeline: 13.508ms
>> >> > - Analysis finished: 3.303ms (3.303ms)
>> >> > - Value transfer graph computed: 3.465ms (162.667us)
>> >> > - Single node plan created: 10.352ms (6.886ms)
>> >> > - Runtime filters computed: 10.677ms (325.364us)
>> >> > - Distributed plan created: 10.733ms (55.854us)
>> >> > - Lineage info computed: 10.898ms (164.937us)
>> >> > - Planning finished: 13.508ms (2.610ms)
>> >> > Query Timeline: 481.950ms
>> >> > - Query submitted: 79.971us (79.971us)
>> >> > - Planning finished: 33.947ms (33.867ms)
>> >> > - Submit for admission: 40.067ms (6.120ms)
>> >> > - Completed admission: 40.197ms (130.146us)
>> >> > - Ready to start on 5 backends: 40.905ms (707.852us)
>> >> > - All 5 execution backends (6 fragment instances) started:
>> >> > 46.417ms
>> >> > (5.512ms)
>> >> > - Rows available: 424.548ms (378.131ms)
>> >> > - First row fetched: 473.488ms (48.939ms)
>> >> > - Unregister query: 477.951ms (4.462ms)
>> >> > - ComputeScanRangeAssignmentTimer: 3.980ms
>> >> > ImpalaServer:
>> >> > - ClientFetchWaitTimer: 51.659ms
>> >> > - RowMaterializationTimer: 1.726ms
>> >> > Execution Profile c843dae48ce9b289:6ad3ffc000000000:(Total:
>> 385.029ms,
>> >> > non-child: 0.000ns, % non-child: 0.00%)
>> >> > Number of filters: 0
>> >> > Filter routing table:
>> >> > ID Src. Node Tgt. Node(s) Target type Partition filter Pending
>> >> > (Expected) First arrived Completed Enabled
>> >> >
>> >> > ------------------------------------------------------------
>> -------------------------------------------------------
>> >> >
>> >> > Backend startup latencies: Count: 5, min / max: 3ms / 4ms, 25th
>> >> > %-ile:
>> >> > 3ms, 50th %-ile: 4ms, 75th %-ile: 4ms, 90th %-ile: 4ms, 95th %-ile:
>> 4ms,
>> >> > 99.9th %-ile: 4ms
>> >> > Per Node Peak Memory Usage: aopr-dhc001.lpdomain.com:22000(2.68
>> MB)
>> >> > aopr-dhc002.lpdomain.com:22000(2.54 MB)
>> >> > aopr-dhc003.lpdomain.com:22000(2.54
>> >> > MB) aopr-dhc004.lpdomain.com:22000(2.41 MB)
>> >> > aopr-dhc005.lpdomain.com:22000(2.56 MB)
>> >> > - FiltersReceived: 0 (0)
>> >> > - FinalizationTimer: 0.000ns
>> >> > - NumBackends: 5 (5)
>> >> > - NumFragmentInstances: 6 (6)
>> >> > - NumFragments: 2 (2)
>> >> > Averaged Fragment F01:(Total: 427.530ms, non-child: 49.428ms, %
>> >> > non-child: 11.56%)
>> >> > split sizes: min: 0, max: 0, avg: 0, stddev: 0
>> >> > completion times: min:428.693ms max:428.693ms mean: 428.693ms
>> >> > stddev:0.000ns
>> >> > execution rates: min:0.00 /sec max:0.00 /sec mean:0.00 /sec
>> >> > stddev:0.00 /sec
>> >> > num instances: 1
>> >> > - AverageThreadTokens: 0.00
>> >> > - BloomFilterBytes: 0
>> >> > - PeakMemoryUsage: 44.02 KB (45072)
>> >> > - PeakReservation: 0
>> >> > - PeakUsedReservation: 0
>> >> > - PerHostPeakMemUsage: 2.68 MB (2806113)
>> >> > - RowsProduced: 1 (1)
>> >> > - TotalNetworkReceiveTime: 357.481ms
>> >> > - TotalNetworkSendTime: 0.000ns
>> >> > - TotalStorageWaitTime: 0.000ns
>> >> > - TotalThreadsInvoluntaryContextSwitches: 0 (0)
>> >> > - TotalThreadsTotalWallClockTime: 406.708ms
>> >> > - TotalThreadsSysTime: 0.000ns
>> >> > - TotalThreadsUserTime: 0.000ns
>> >> > - TotalThreadsVoluntaryContextSwitches: 7 (7)
>> >> > Fragment Instance Lifecycle Timings:
>> >> > - ExecTime: 49.055ms
>> >> > - ExecTreeExecTime: 4.251us
>> >> > - OpenTime: 357.663ms
>> >> > - ExecTreeOpenTime: 357.649ms
>> >> > - PrepareTime: 20.764ms
>> >> > - ExecTreePrepareTime: 81.759us
>> >> > PLAN_ROOT_SINK:
>> >> > - PeakMemoryUsage: 0
>> >> > CodeGen:(Total: 20.369ms, non-child: 20.369ms, % non-child:
>> >> > 100.00%)
>> >> > - CodegenTime: 0.000ns
>> >> > - CompileTime: 0.000ns
>> >> > - LoadTime: 0.000ns
>> >> > - ModuleBitcodeSize: 1.86 MB (1950880)
>> >> > - NumFunctions: 0 (0)
>> >> > - NumInstructions: 0 (0)
>> >> > - OptimizationTime: 0.000ns
>> >> > - PeakMemoryUsage: 0
>> >> > - PrepareTime: 19.685ms
>> >> > AGGREGATION_NODE (id=3):(Total: 357.731ms, non-child:
>> 222.958us, %
>> >> > non-child: 0.06%)
>> >> > - BuildTime: 7.368us
>> >> > - GetResultsTime: 0.000ns
>> >> > - HTResizeTime: 0.000ns
>> >> > - HashBuckets: 0 (0)
>> >> > - LargestPartitionPercent: 0 (0)
>> >> > - MaxPartitionLevel: 0 (0)
>> >> > - NumRepartitions: 0 (0)
>> >> > - PartitionsCreated: 0 (0)
>> >> > - PeakMemoryUsage: 36.00 KB (36864)
>> >> > - RowsRepartitioned: 0 (0)
>> >> > - RowsReturned: 1 (1)
>> >> > - RowsReturnedRate: 2.00 /sec
>> >> > - SpilledPartitions: 0 (0)
>> >> > EXCHANGE_NODE (id=2):(Total: 357.508ms, non-child: 357.508ms, %
>> >> > non-child: 100.00%)
>> >> > - ConvertRowBatchTime: 8.881us
>> >> > - PeakMemoryUsage: 0
>> >> > - RowsReturned: 5 (5)
>> >> > - RowsReturnedRate: 13.00 /sec
>> >> > DataStreamReceiver:
>> >> > - BytesReceived: 80.00 B (80)
>> >> > - DeserializeRowBatchTimer: 24.165us
>> >> > - FirstBatchArrivalWaitTime: 277.194ms
>> >> > - PeakMemoryUsage: 4.02 KB (4112)
>> >> > - SendersBlockedTimer: 0.000ns
>> >> > - SendersBlockedTotalTimer(*): 0.000ns
>> >> > Coordinator Fragment F01:
>> >> > Instance c843dae48ce9b289:6ad3ffc000000000
>> >> > (host=aopr-dhc001.lpdomain.com:22000):(Total: 427.530ms, non-child:
>> >> > 49.428ms, % non-child: 11.56%)
>> >> > - AverageThreadTokens: 0.00
>> >> > - BloomFilterBytes: 0
>> >> > - PeakMemoryUsage: 44.02 KB (45072)
>> >> > - PeakReservation: 0
>> >> > - PeakUsedReservation: 0
>> >> > - PerHostPeakMemUsage: 2.68 MB (2806113)
>> >> > - RowsProduced: 1 (1)
>> >> > - TotalNetworkReceiveTime: 357.481ms
>> >> > - TotalNetworkSendTime: 0.000ns
>> >> > - TotalStorageWaitTime: 0.000ns
>> >> > - TotalThreadsInvoluntaryContextSwitches: 0 (0)
>> >> > - TotalThreadsTotalWallClockTime: 406.708ms
>> >> > - TotalThreadsSysTime: 0.000ns
>> >> > - TotalThreadsUserTime: 0.000ns
>> >> > - TotalThreadsVoluntaryContextSwitches: 7 (7)
>> >> > Fragment Instance Lifecycle Timings:
>> >> > - ExecTime: 49.055ms
>> >> > - ExecTreeExecTime: 4.251us
>> >> > - OpenTime: 357.663ms
>> >> > - ExecTreeOpenTime: 357.649ms
>> >> > - PrepareTime: 20.764ms
>> >> > - ExecTreePrepareTime: 81.759us
>> >> > PLAN_ROOT_SINK:
>> >> > - PeakMemoryUsage: 0
>> >> > CodeGen:(Total: 20.369ms, non-child: 20.369ms, % non-child:
>> >> > 100.00%)
>> >> > - CodegenTime: 0.000ns
>> >> > - CompileTime: 0.000ns
>> >> > - LoadTime: 0.000ns
>> >> > - ModuleBitcodeSize: 1.86 MB (1950880)
>> >> > - NumFunctions: 0 (0)
>> >> > - NumInstructions: 0 (0)
>> >> > - OptimizationTime: 0.000ns
>> >> > - PeakMemoryUsage: 0
>> >> > - PrepareTime: 19.685ms
>> >> > AGGREGATION_NODE (id=3):(Total: 357.731ms, non-child:
>> 222.958us,
>> >> > %
>> >> > non-child: 0.06%)
>> >> > - BuildTime: 7.368us
>> >> > - GetResultsTime: 0.000ns
>> >> > - HTResizeTime: 0.000ns
>> >> > - HashBuckets: 0 (0)
>> >> > - LargestPartitionPercent: 0 (0)
>> >> > - MaxPartitionLevel: 0 (0)
>> >> > - NumRepartitions: 0 (0)
>> >> > - PartitionsCreated: 0 (0)
>> >> > - PeakMemoryUsage: 36.00 KB (36864)
>> >> > - RowsRepartitioned: 0 (0)
>> >> > - RowsReturned: 1 (1)
>> >> > - RowsReturnedRate: 2.00 /sec
>> >> > - SpilledPartitions: 0 (0)
>> >> > EXCHANGE_NODE (id=2):(Total: 357.508ms, non-child:
>> 357.508ms, %
>> >> > non-child: 100.00%)
>> >> > - ConvertRowBatchTime: 8.881us
>> >> > - PeakMemoryUsage: 0
>> >> > - RowsReturned: 5 (5)
>> >> > - RowsReturnedRate: 13.00 /sec
>> >> > DataStreamReceiver:
>> >> > - BytesReceived: 80.00 B (80)
>> >> > - DeserializeRowBatchTimer: 24.165us
>> >> > - FirstBatchArrivalWaitTime: 277.194ms
>> >> > - PeakMemoryUsage: 4.02 KB (4112)
>> >> > - SendersBlockedTimer: 0.000ns
>> >> > - SendersBlockedTotalTimer(*): 0.000ns
>> >> > Averaged Fragment F00:(Total: 327.360ms, non-child: 1.578ms, %
>> >> > non-child: 0.48%)
>> >> > split sizes: min: 283.31 MB, max: 284.33 MB, avg: 284.04 MB,
>> >> > stddev:
>> >> > 382.00 KB
>> >> > completion times: min:299.457ms max:378.950ms mean: 328.881ms
>> >> > stddev:31.507ms
>> >> > execution rates: min:750.18 MB/sec max:949.01 MB/sec
>> mean:871.27
>> >> > MB/sec stddev:79.74 MB/sec
>> >> > num instances: 5
>> >> > - AverageThreadTokens: 15.00
>> >> > - BloomFilterBytes: 0
>> >> > - PeakMemoryUsage: 2.54 MB (2666852)
>> >> > - PeakReservation: 0
>> >> > - PeakUsedReservation: 0
>> >> > - PerHostPeakMemUsage: 2.55 MB (2669309)
>> >> > - RowsProduced: 1 (1)
>> >> > - TotalNetworkReceiveTime: 0.000ns
>> >> > - TotalNetworkSendTime: 303.541us
>> >> > - TotalStorageWaitTime: 4s821ms
>> >> > - TotalThreadsInvoluntaryContextSwitches: 104 (104)
>> >> > - TotalThreadsTotalWallClockTime: 6s835ms
>> >> > - TotalThreadsSysTime: 31.784ms
>> >> > - TotalThreadsUserTime: 682.084ms
>> >> > - TotalThreadsVoluntaryContextSwitches: 2.12K (2121)
>> >> > Fragment Instance Lifecycle Timings:
>> >> > - ExecTime: 500.816us
>> >> > - ExecTreeExecTime: 8.200us
>> >> > - OpenTime: 302.848ms
>> >> > - ExecTreeOpenTime: 281.309ms
>> >> > - PrepareTime: 23.946ms
>> >> > - ExecTreePrepareTime: 2.513ms
>> >> > DataStreamSender (dst_id=2):(Total: 269.902us, non-child:
>> >> > 269.902us, %
>> >> > non-child: 100.00%)
>> >> > - BytesSent: 16.00 B (16)
>> >> > - NetworkThroughput(*): 373.13 KB/sec
>> >> > - OverallThroughput: 63.53 KB/sec
>> >> > - PeakMemoryUsage: 16.00 KB (16384)
>> >> > - RowsReturned: 1 (1)
>> >> > - SerializeBatchTime: 14.525us
>> >> > - TransmitDataRPCTime: 43.122us
>> >> > - UncompressedRowBatchSize: 16.00 B (16)
>> >> > CodeGen:(Total: 41.687ms, non-child: 41.687ms, % non-child:
>> >> > 100.00%)
>> >> > - CodegenTime: 1.194ms
>> >> > - CompileTime: 7.360ms
>> >> > - LoadTime: 0.000ns
>> >> > - ModuleBitcodeSize: 1.86 MB (1950880)
>> >> > - NumFunctions: 15 (15)
>> >> > - NumInstructions: 214 (214)
>> >> > - OptimizationTime: 13.529ms
>> >> > - PeakMemoryUsage: 107.00 KB (109568)
>> >> > - PrepareTime: 20.084ms
>> >> > AGGREGATION_NODE (id=1):(Total: 283.825ms, non-child: 1.027ms,
>> %
>> >> > non-child: 0.36%)
>> >> > - BuildTime: 1.259ms
>> >> > - GetResultsTime: 0.000ns
>> >> > - HTResizeTime: 0.000ns
>> >> > - HashBuckets: 0 (0)
>> >> > - LargestPartitionPercent: 0 (0)
>> >> > - MaxPartitionLevel: 0 (0)
>> >> > - NumRepartitions: 0 (0)
>> >> > - PartitionsCreated: 0 (0)
>> >> > - PeakMemoryUsage: 20.00 KB (20480)
>> >> > - RowsRepartitioned: 0 (0)
>> >> > - RowsReturned: 1 (1)
>> >> > - RowsReturnedRate: 2.00 /sec
>> >> > - SpilledPartitions: 0 (0)
>> >> > HDFS_SCAN_NODE (id=0):(Total: 282.797ms, non-child: 282.797ms,
>> %
>> >> > non-child: 100.00%)
>> >> > - AverageHdfsReadThreadConcurrency: 0.00
>> >> > - AverageScannerThreadConcurrency: 14.40
>> >> > - BytesRead: 44.52 MB (46686028)
>> >> > - BytesReadDataNodeCache: 0
>> >> > - BytesReadLocal: 44.52 MB (46686028)
>> >> > - BytesReadRemoteUnexpected: 0
>> >> > - BytesReadShortCircuit: 44.52 MB (46686028)
>> >> > - CachedFileHandlesHitCount: 0 (0)
>> >> > - CachedFileHandlesMissCount: 458 (458)
>> >> > - CollectionItemsRead: 0 (0)
>> >> > - DecompressionTime: 0.000ns
>> >> > - MaxCompressedTextFileLength: 0
>> >> > - NumColumns: 0 (0)
>> >> > - NumDictFilteredRowGroups: 0 (0)
>> >> > - NumDisksAccessed: 12 (12)
>> >> > - NumRowGroups: 458 (458)
>> >> > - NumScannerThreadsStarted: 24 (24)
>> >> > - NumScannersWithNoReads: 0 (0)
>> >> > - NumStatsFilteredRowGroups: 0 (0)
>> >> > - PeakMemoryUsage: 2.50 MB (2625539)
>> >> > - PerReadThreadRawHdfsThroughput: 1.29 GB/sec
>> >> > - RemoteScanRanges: 0 (0)
>> >> > - RowBatchQueueGetWaitTime: 269.787ms
>> >> > - RowBatchQueuePutWaitTime: 0.000ns
>> >> > - RowsRead: 0 (0)
>> >> > - RowsReturned: 458 (458)
>> >> > - RowsReturnedRate: 1.64 K/sec
>> >> > - ScanRangesComplete: 458 (458)
>> >> > - ScannerThreadsInvoluntaryContextSwitches: 99 (99)
>> >> > - ScannerThreadsTotalWallClockTime: 6s532ms
>> >> > - MaterializeTupleTime(*): 70.664us
>> >> > - ScannerThreadsSysTime: 26.184ms
>> >> > - ScannerThreadsUserTime: 650.089ms
>> >> > - ScannerThreadsVoluntaryContextSwitches: 1.67K (1674)
>> >> > - TotalRawHdfsReadTime(*): 34.416ms
>> >> > - TotalReadThroughput: 36.86 MB/sec
>> >> > Fragment F00:
>> >> > Instance c843dae48ce9b289:6ad3ffc000000004
>> >> > (host=aopr-dhc004.lpdomain.com:22000):(Total: 377.384ms, non-child:
>> >> > 1.619ms,
>> >> > % non-child: 0.43%)
>> >> > Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>> >> > 0:36/21.95 MB 11:33
>> <https://maps.google.com/?q=MB+11:33&entry=gmail&source=g>/19.87 MB 9:27
>> <https://maps.google.com/?q=MB+9:27&entry=gmail&source=g>/15.16 MB 5:44
>> <https://maps.google.com/?q=MB+5:44&entry=gmail&source=g>/28.07 MB 6:56
>> <https://maps.google.com/?q=MB+6:56&entry=gmail&source=g>/33.32 MB
>> >> > 1:38/22.24 MB 10:44
>> <https://maps.google.com/?q=MB+10:44&entry=gmail&source=g>/27.90 MB 3:34
>> <https://maps.google.com/?q=MB+3:34&entry=gmail&source=g>/20.26 MB 2:45
>> <https://maps.google.com/?q=MB+2:45&entry=gmail&source=g>/28.21 MB 7:31
>> <https://maps.google.com/?q=MB+7:31&entry=gmail&source=g>/19.33 MB
>> >> > 4:42/26.73 MB 8:34
>> <https://maps.google.com/?q=MB+8:34&entry=gmail&source=g>/21.25 MB
>> >> > MemoryUsage(500.000ms): 1.91 MB
>> >> > ThreadUsage(500.000ms): 25
>> >> > - AverageThreadTokens: 25.00
>> >> > - BloomFilterBytes: 0
>> >> > - PeakMemoryUsage: 2.41 MB (2531689)
>> >> > - PeakReservation: 0
>> >> > - PeakUsedReservation: 0
>> >> > - PerHostPeakMemUsage: 2.41 MB (2531689)
>> >> > - RowsProduced: 1 (1)
>> >> > - TotalNetworkReceiveTime: 0.000ns
>> >> > - TotalNetworkSendTime: 382.629us
>> >> > - TotalStorageWaitTime: 5s790ms
>> >> > - TotalThreadsInvoluntaryContextSwitches: 115 (115)
>> >> > - TotalThreadsTotalWallClockTime: 8s122ms
>> >> > - TotalThreadsSysTime: 33.984ms
>> >> > - TotalThreadsUserTime: 702.879ms
>> >> > - TotalThreadsVoluntaryContextSwitches: 2.04K (2037)
>> >> > Fragment Instance Lifecycle Timings:
>> >> > - ExecTime: 604.005us
>> >> > - ExecTreeExecTime: 8.338us
>> >> > - OpenTime: 352.927ms
>> >> > - ExecTreeOpenTime: 331.921ms
>> >> > - PrepareTime: 23.783ms
>> >> > - ExecTreePrepareTime: 2.559ms
>> >> > DataStreamSender (dst_id=2):(Total: 331.818us, non-child:
>> >> > 331.818us,
>> >> > % non-child: 100.00%)
>> >> > - BytesSent: 16.00 B (16)
>> >> > - NetworkThroughput(*): 289.00 KB/sec
>> >> > - OverallThroughput: 47.09 KB/sec
>> >> > - PeakMemoryUsage: 16.00 KB (16384)
>> >> > - RowsReturned: 1 (1)
>> >> > - SerializeBatchTime: 14.466us
>> >> > - TransmitDataRPCTime: 54.066us
>> >> > - UncompressedRowBatchSize: 16.00 B (16)
>> >> > CodeGen:(Total: 40.950ms, non-child: 40.950ms, % non-child:
>> >> > 100.00%)
>> >> > - CodegenTime: 1.122ms
>> >> > - CompileTime: 7.318ms
>> >> > - LoadTime: 0.000ns
>> >> > - ModuleBitcodeSize: 1.86 MB (1950880)
>> >> > - NumFunctions: 15 (15)
>> >> > - NumInstructions: 214 (214)
>> >> > - OptimizationTime: 13.106ms
>> >> > - PeakMemoryUsage: 107.00 KB (109568)
>> >> > - PrepareTime: 19.773ms
>> >> > AGGREGATION_NODE (id=1):(Total: 334.482ms, non-child:
>> 766.527us,
>> >> > %
>> >> > non-child: 0.23%)
>> >> > ExecOption: Codegen Enabled
>> >> > - BuildTime: 1.192ms
>> >> > - GetResultsTime: 0.000ns
>> >> > - HTResizeTime: 0.000ns
>> >> > - HashBuckets: 0 (0)
>> >> > - LargestPartitionPercent: 0 (0)
>> >> > - MaxPartitionLevel: 0 (0)
>> >> > - NumRepartitions: 0 (0)
>> >> > - PartitionsCreated: 0 (0)
>> >> > - PeakMemoryUsage: 20.00 KB (20480)
>> >> > - RowsRepartitioned: 0 (0)
>> >> > - RowsReturned: 1 (1)
>> >> > - RowsReturnedRate: 2.00 /sec
>> >> > - SpilledPartitions: 0 (0)
>> >> > HDFS_SCAN_NODE (id=0):(Total: 333.715ms, non-child:
>> 333.715ms, %
>> >> > non-child: 100.00%)
>> >> > Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>> >> > 0:36/21.95 MB 11:33
>> <https://maps.google.com/?q=MB+11:33&entry=gmail&source=g>/19.87 MB 9:27
>> <https://maps.google.com/?q=MB+9:27&entry=gmail&source=g>/15.16 MB 5:44
>> <https://maps.google.com/?q=MB+5:44&entry=gmail&source=g>/28.07 MB 6:56
>> <https://maps.google.com/?q=MB+6:56&entry=gmail&source=g>/33.32 MB
>> >> > 1:38/22.24 MB 10:44
>> <https://maps.google.com/?q=MB+10:44&entry=gmail&source=g>/27.90 MB 3:34
>> <https://maps.google.com/?q=MB+3:34&entry=gmail&source=g>/20.26 MB 2:45
>> <https://maps.google.com/?q=MB+2:45&entry=gmail&source=g>/28.21 MB 7:31
>> <https://maps.google.com/?q=MB+7:31&entry=gmail&source=g>/19.33 MB
>> >> > 4:42/26.73 MB 8:34
>> <https://maps.google.com/?q=MB+8:34&entry=gmail&source=g>/21.25 MB
>> >> > ExecOption: PARQUET Codegen Enabled, Codegen enabled: 464
>> out
>> >> > of
>> >> > 464
>> >> > Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0%
>> >> > 4:0%
>> >> > 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
>> >> > File Formats: PARQUET/Unknown(Skipped):464
>> >> > BytesRead(500.000ms): 12.27 MB
>> >> > - FooterProcessingTime: (Avg: 13.555ms ; Min: 5.433ms ;
>> Max:
>> >> > 62.957ms ; Number of samples: 464)
>> >> > - AverageHdfsReadThreadConcurrency: 0.00
>> >> > - AverageScannerThreadConcurrency: 24.00
>> >> > - BytesRead: 45.01 MB (47193988)
>> >> > - BytesReadDataNodeCache: 0
>> >> > - BytesReadLocal: 45.01 MB (47193988)
>> >> > - BytesReadRemoteUnexpected: 0
>> >> > - BytesReadShortCircuit: 45.01 MB (47193988)
>> >> > - CachedFileHandlesHitCount: 0 (0)
>> >> > - CachedFileHandlesMissCount: 464 (464)
>> >> > - CollectionItemsRead: 0 (0)
>> >> > - DecompressionTime: 0.000ns
>> >> > - MaxCompressedTextFileLength: 0
>> >> > - NumColumns: 0 (0)
>> >> > - NumDictFilteredRowGroups: 0 (0)
>> >> > - NumDisksAccessed: 12 (12)
>> >> > - NumRowGroups: 464 (464)
>> >> > - NumScannerThreadsStarted: 24 (24)
>> >> > - NumScannersWithNoReads: 0 (0)
>> >> > - NumStatsFilteredRowGroups: 0 (0)
>> >> > - PeakMemoryUsage: 2.38 MB (2490376)
>> >> > - PerReadThreadRawHdfsThroughput: 1016.44 MB/sec
>> >> > - RemoteScanRanges: 0 (0)
>> >> > - RowBatchQueueGetWaitTime: 320.860ms
>> >> > - RowBatchQueuePutWaitTime: 0.000ns
>> >> > - RowsRead: 0 (0)
>> >> > - RowsReturned: 464 (464)
>> >> > - RowsReturnedRate: 1.39 K/sec
>> >> > - ScanRangesComplete: 464 (464)
>> >> > - ScannerThreadsInvoluntaryContextSwitches: 108 (108)
>> >> > - ScannerThreadsTotalWallClockTime: 7s769ms
>> >> > - MaterializeTupleTime(*): 75.514us
>> >> > - ScannerThreadsSysTime: 29.984ms
>> >> > - ScannerThreadsUserTime: 669.884ms
>> >> > - ScannerThreadsVoluntaryContextSwitches: 1.60K (1597)
>> >> > - TotalRawHdfsReadTime(*): 44.279ms
>> >> > - TotalReadThroughput: 24.53 MB/sec
>> >> > Instance c843dae48ce9b289:6ad3ffc000000002
>> >> > (host=aopr-dhc003.lpdomain.com:22000):(Total: 351.014ms, non-child:
>> >> > 1.641ms,
>> >> > % non-child: 0.47%)
>> >> > Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>> >> > 0:35/20.89 MB 7:44
>> <https://maps.google.com/?q=MB+7:44&entry=gmail&source=g>/26.64 MB 11:27
>> <https://maps.google.com/?q=MB+11:27&entry=gmail&source=g>/17.21 MB 9:32
>> <https://maps.google.com/?q=MB+9:32&entry=gmail&source=g>/18.61 MB 5:25
>> <https://maps.google.com/?q=MB+5:25&entry=gmail&source=g>/14.56 MB
>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+2:38&entry=gmail&source=g>
>> >> > 2:38
>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+2:38&entry=gmail&source=g>/24.32
>> MB 6:31 <https://maps.google.com/?q=MB+6:31&entry=gmail&source=g>/19.85 MB
>> 8:50 <https://maps.google.com/?q=MB+8:50&entry=gmail&source=g>/30.76 MB
>> 10:41 <https://maps.google.com/?q=MB+10:41&entry=gmail&source=g>/25.92 MB
>> 3:31 <https://maps.google.com/?q=MB+3:31&entry=gmail&source=g>/19.78 MB
>> >> > 4:60/37.59 MB 1:44/27.19 MB
>> >> > MemoryUsage(500.000ms): 1.16 MB
>> >> > ThreadUsage(500.000ms): 25
>> >> > - AverageThreadTokens: 25.00
>> >> > - BloomFilterBytes: 0
>> >> > - PeakMemoryUsage: 2.54 MB (2662753)
>> >> > - PeakReservation: 0
>> >> > - PeakUsedReservation: 0
>> >> > - PerHostPeakMemUsage: 2.54 MB (2662753)
>> >> > - RowsProduced: 1 (1)
>> >> > - TotalNetworkReceiveTime: 0.000ns
>> >> > - TotalNetworkSendTime: 342.746us
>> >> > - TotalStorageWaitTime: 4s796ms
>> >> > - TotalThreadsInvoluntaryContextSwitches: 93 (93)
>> >> > - TotalThreadsTotalWallClockTime: 7s286ms
>> >> > - TotalThreadsSysTime: 27.988ms
>> >> > - TotalThreadsUserTime: 671.887ms
>> >> > - TotalThreadsVoluntaryContextSwitches: 2.08K (2079)
>> >> > Fragment Instance Lifecycle Timings:
>> >> > - ExecTime: 524.006us
>> >> > - ExecTreeExecTime: 7.920us
>> >> > - OpenTime: 327.802ms
>> >> > - ExecTreeOpenTime: 307.154ms
>> >> > - PrepareTime: 22.617ms
>> >> > - ExecTreePrepareTime: 2.650ms
>> >> > DataStreamSender (dst_id=2):(Total: 348.990us, non-child:
>> >> > 348.990us,
>> >> > % non-child: 100.00%)
>> >> > - BytesSent: 16.00 B (16)
>> >> > - NetworkThroughput(*): 354.47 KB/sec
>> >> > - OverallThroughput: 44.77 KB/sec
>> >> > - PeakMemoryUsage: 16.00 KB (16384)
>> >> > - RowsReturned: 1 (1)
>> >> > - SerializeBatchTime: 15.602us
>> >> > - TransmitDataRPCTime: 44.080us
>> >> > - UncompressedRowBatchSize: 16.00 B (16)
>> >> > CodeGen:(Total: 39.215ms, non-child: 39.215ms, % non-child:
>> >> > 100.00%)
>> >> > - CodegenTime: 1.183ms
>> >> > - CompileTime: 7.168ms
>> >> > - LoadTime: 0.000ns
>> >> > - ModuleBitcodeSize: 1.86 MB (1950880)
>> >> > - NumFunctions: 15 (15)
>> >> > - NumInstructions: 214 (214)
>> >> > - OptimizationTime: 12.929ms
>> >> > - PeakMemoryUsage: 107.00 KB (109568)
>> >> > - PrepareTime: 18.453ms
>> >> > AGGREGATION_NODE (id=1):(Total: 309.807ms, non-child:
>> 948.677us,
>> >> > %
>> >> > non-child: 0.31%)
>> >> > ExecOption: Codegen Enabled
>> >> > - BuildTime: 1.552ms
>> >> > - GetResultsTime: 0.000ns
>> >> > - HTResizeTime: 0.000ns
>> >> > - HashBuckets: 0 (0)
>> >> > - LargestPartitionPercent: 0 (0)
>> >> > - MaxPartitionLevel: 0 (0)
>> >> > - NumRepartitions: 0 (0)
>> >> > - PartitionsCreated: 0 (0)
>> >> > - PeakMemoryUsage: 20.00 KB (20480)
>> >> > - RowsRepartitioned: 0 (0)
>> >> > - RowsReturned: 1 (1)
>> >> > - RowsReturnedRate: 3.00 /sec
>> >> > - SpilledPartitions: 0 (0)
>> >> > HDFS_SCAN_NODE (id=0):(Total: 308.859ms, non-child:
>> 308.859ms, %
>> >> > non-child: 100.00%)
>> >> > Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>> >> > 0:35/20.89 MB 7:44
>> <https://maps.google.com/?q=MB+7:44&entry=gmail&source=g>/26.64 MB 11:27
>> <https://maps.google.com/?q=MB+11:27&entry=gmail&source=g>/17.21 MB 9:32
>> <https://maps.google.com/?q=MB+9:32&entry=gmail&source=g>/18.61 MB 5:25
>> <https://maps.google.com/?q=MB+5:25&entry=gmail&source=g>/14.56 MB
>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+2:38&entry=gmail&source=g>
>> >> > 2:38
>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+2:38&entry=gmail&source=g>/24.32
>> MB 6:31 <https://maps.google.com/?q=MB+6:31&entry=gmail&source=g>/19.85 MB
>> 8:50 <https://maps.google.com/?q=MB+8:50&entry=gmail&source=g>/30.76 MB
>> 10:41 <https://maps.google.com/?q=MB+10:41&entry=gmail&source=g>/25.92 MB
>> 3:31 <https://maps.google.com/?q=MB+3:31&entry=gmail&source=g>/19.78 MB
>> >> > 4:60/37.59 MB 1:44/27.19 MB
>> >> > ExecOption: PARQUET Codegen Enabled, Codegen enabled: 458
>> out
>> >> > of
>> >> > 458
>> >> > Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0%
>> >> > 4:0%
>> >> > 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
>> >> > File Formats: PARQUET/Unknown(Skipped):458
>> >> > BytesRead(500.000ms): 36.74 MB
>> >> > - FooterProcessingTime: (Avg: 11.502ms ; Min: 3.668ms ;
>> Max:
>> >> > 23.564ms ; Number of samples: 458)
>> >> > - AverageHdfsReadThreadConcurrency: 0.00
>> >> > - AverageScannerThreadConcurrency: 24.00
>> >> > - BytesRead: 44.51 MB (46668432)
>> >> > - BytesReadDataNodeCache: 0
>> >> > - BytesReadLocal: 44.51 MB (46668432)
>> >> > - BytesReadRemoteUnexpected: 0
>> >> > - BytesReadShortCircuit: 44.51 MB (46668432)
>> >> > - CachedFileHandlesHitCount: 0 (0)
>> >> > - CachedFileHandlesMissCount: 458 (458)
>> >> > - CollectionItemsRead: 0 (0)
>> >> > - DecompressionTime: 0.000ns
>> >> > - MaxCompressedTextFileLength: 0
>> >> > - NumColumns: 0 (0)
>> >> > - NumDictFilteredRowGroups: 0 (0)
>> >> > - NumDisksAccessed: 12 (12)
>> >> > - NumRowGroups: 458 (458)
>> >> > - NumScannerThreadsStarted: 24 (24)
>> >> > - NumScannersWithNoReads: 0 (0)
>> >> > - NumStatsFilteredRowGroups: 0 (0)
>> >> > - PeakMemoryUsage: 2.50 MB (2621440)
>> >> > - PerReadThreadRawHdfsThroughput: 1.26 GB/sec
>> >> > - RemoteScanRanges: 0 (0)
>> >> > - RowBatchQueueGetWaitTime: 295.595ms
>> >> > - RowBatchQueuePutWaitTime: 0.000ns
>> >> > - RowsRead: 0 (0)
>> >> > - RowsReturned: 458 (458)
>> >> > - RowsReturnedRate: 1.48 K/sec
>> >> > - ScanRangesComplete: 458 (458)
>> >> > - ScannerThreadsInvoluntaryContextSwitches: 91 (91)
>> >> > - ScannerThreadsTotalWallClockTime: 6s958ms
>> >> > - MaterializeTupleTime(*): 71.630us
>> >> > - ScannerThreadsSysTime: 16.989ms
>> >> > - ScannerThreadsUserTime: 645.891ms
>> >> > - ScannerThreadsVoluntaryContextSwitches: 1.63K (1630)
>> >> > - TotalRawHdfsReadTime(*): 34.377ms
>> >> > - TotalReadThroughput: 73.49 MB/sec
>> >> > Instance c843dae48ce9b289:6ad3ffc000000001
>> >> > (host=aopr-dhc005.lpdomain.com:22000):(Total: 308.195ms, non-child:
>> >> > 1.784ms,
>> >> > % non-child: 0.58%)
>> >> > Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>> >> > 2:46/28.97 MB 8:39
>> <https://maps.google.com/?q=MB+8:39&entry=gmail&source=g>/24.34 MB 10:33
>> <https://maps.google.com/?q=MB+10:33&entry=gmail&source=g>/21.03 MB 3:43
>> <https://maps.google.com/?q=MB+3:43&entry=gmail&source=g>/26.50 MB 5:32
>> <https://maps.google.com/?q=MB+5:32&entry=gmail&source=g>/20.27 MB
>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+9:32&entry=gmail&source=g>
>> >> > 9:32
>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+9:32&entry=gmail&source=g>/19.86
>> MB 6:25 <https://maps.google.com/?q=MB+6:25&entry=gmail&source=g>/16.01 MB
>> 7:41 <https://maps.google.com/?q=MB+7:41&entry=gmail&source=g>/24.77 MB
>> 1:42/25.01 MB 0:55/32.51 MB
>> >> > 4:40/24.29 MB 11:33
>> <https://maps.google.com/?q=MB+11:33&entry=gmail&source=g>/20.77 MB
>> >> > - AverageThreadTokens: 0.00
>> >> > - BloomFilterBytes: 0
>> >> > - PeakMemoryUsage: 2.56 MB (2683241)
>> >> > - PeakReservation: 0
>> >> > - PeakUsedReservation: 0
>> >> > - PerHostPeakMemUsage: 2.56 MB (2683241)
>> >> > - RowsProduced: 1 (1)
>> >> > - TotalNetworkReceiveTime: 0.000ns
>> >> > - TotalNetworkSendTime: 292.765us
>> >> > - TotalStorageWaitTime: 4s482ms
>> >> > - TotalThreadsInvoluntaryContextSwitches: 113 (113)
>> >> > - TotalThreadsTotalWallClockTime: 6s310ms
>> >> > - TotalThreadsSysTime: 35.983ms
>> >> > - TotalThreadsUserTime: 682.884ms
>> >> > - TotalThreadsVoluntaryContextSwitches: 2.14K (2143)
>> >> > Fragment Instance Lifecycle Timings:
>> >> > - ExecTime: 532.572us
>> >> > - ExecTreeExecTime: 8.377us
>> >> > - OpenTime: 280.241ms
>> >> > - ExecTreeOpenTime: 258.652ms
>> >> > - PrepareTime: 27.342ms
>> >> > - ExecTreePrepareTime: 2.863ms
>> >> > DataStreamSender (dst_id=2):(Total: 309.794us, non-child:
>> >> > 309.794us,
>> >> > % non-child: 100.00%)
>> >> > - BytesSent: 16.00 B (16)
>> >> > - NetworkThroughput(*): 356.91 KB/sec
>> >> > - OverallThroughput: 50.44 KB/sec
>> >> > - PeakMemoryUsage: 16.00 KB (16384)
>> >> > - RowsReturned: 1 (1)
>> >> > - SerializeBatchTime: 14.262us
>> >> > - TransmitDataRPCTime: 43.778us
>> >> > - UncompressedRowBatchSize: 16.00 B (16)
>> >> > CodeGen:(Total: 44.584ms, non-child: 44.584ms, % non-child:
>> >> > 100.00%)
>> >> > - CodegenTime: 1.382ms
>> >> > - CompileTime: 7.346ms
>> >> > - LoadTime: 0.000ns
>> >> > - ModuleBitcodeSize: 1.86 MB (1950880)
>> >> > - NumFunctions: 15 (15)
>> >> > - NumInstructions: 214 (214)
>> >> > - OptimizationTime: 13.462ms
>> >> > - PeakMemoryUsage: 107.00 KB (109568)
>> >> > - PrepareTime: 23.006ms
>> >> > AGGREGATION_NODE (id=1):(Total: 261.516ms, non-child:
>> 836.656us,
>> >> > %
>> >> > non-child: 0.32%)
>> >> > ExecOption: Codegen Enabled
>> >> > - BuildTime: 1.280ms
>> >> > - GetResultsTime: 0.000ns
>> >> > - HTResizeTime: 0.000ns
>> >> > - HashBuckets: 0 (0)
>> >> > - LargestPartitionPercent: 0 (0)
>> >> > - MaxPartitionLevel: 0 (0)
>> >> > - NumRepartitions: 0 (0)
>> >> > - PartitionsCreated: 0 (0)
>> >> > - PeakMemoryUsage: 20.00 KB (20480)
>> >> > - RowsRepartitioned: 0 (0)
>> >> > - RowsReturned: 1 (1)
>> >> > - RowsReturnedRate: 3.00 /sec
>> >> > - SpilledPartitions: 0 (0)
>> >> > HDFS_SCAN_NODE (id=0):(Total: 260.680ms, non-child:
>> 260.680ms, %
>> >> > non-child: 100.00%)
>> >> > Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>> >> > 2:46/28.97 MB 8:39
>> <https://maps.google.com/?q=MB+8:39&entry=gmail&source=g>/24.34 MB 10:33
>> <https://maps.google.com/?q=MB+10:33&entry=gmail&source=g>/21.03 MB 3:43
>> <https://maps.google.com/?q=MB+3:43&entry=gmail&source=g>/26.50 MB 5:32
>> <https://maps.google.com/?q=MB+5:32&entry=gmail&source=g>/20.27 MB
>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+9:32&entry=gmail&source=g>
>> >> > 9:32
>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+9:32&entry=gmail&source=g>/19.86
>> MB 6:25 <https://maps.google.com/?q=MB+6:25&entry=gmail&source=g>/16.01 MB
>> 7:41 <https://maps.google.com/?q=MB+7:41&entry=gmail&source=g>/24.77 MB
>> 1:42/25.01 MB 0:55/32.51 MB
>> >> > 4:40/24.29 MB 11:33
>> <https://maps.google.com/?q=MB+11:33&entry=gmail&source=g>/20.77 MB
>> >> > ExecOption: PARQUET Codegen Enabled, Codegen enabled: 461
>> out
>> >> > of
>> >> > 461
>> >> > Hdfs Read Thread Concurrency Bucket: 0:0% 1:0% 2:0% 3:0%
>> 4:0%
>> >> > 5:0%
>> >> > 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
>> >> > File Formats: PARQUET/Unknown(Skipped):461
>> >> > - FooterProcessingTime: (Avg: 10.793ms ; Min: 3.087ms ;
>> Max:
>> >> > 20.253ms ; Number of samples: 461)
>> >> > - AverageHdfsReadThreadConcurrency: 0.00
>> >> > - AverageScannerThreadConcurrency: 0.00
>> >> > - BytesRead: 44.73 MB (46906407)
>> >> > - BytesReadDataNodeCache: 0
>> >> > - BytesReadLocal: 44.73 MB (46906407)
>> >> > - BytesReadRemoteUnexpected: 0
>> >> > - BytesReadShortCircuit: 44.73 MB (46906407)
>> >> > - CachedFileHandlesHitCount: 0 (0)
>> >> > - CachedFileHandlesMissCount: 461 (461)
>> >> > - CollectionItemsRead: 0 (0)
>> >> > - DecompressionTime: 0.000ns
>> >> > - MaxCompressedTextFileLength: 0
>> >> > - NumColumns: 0 (0)
>> >> > - NumDictFilteredRowGroups: 0 (0)
>> >> > - NumDisksAccessed: 12 (12)
>> >> > - NumRowGroups: 461 (461)
>> >> > - NumScannerThreadsStarted: 24 (24)
>> >> > - NumScannersWithNoReads: 0 (0)
>> >> > - NumStatsFilteredRowGroups: 0 (0)
>> >> > - PeakMemoryUsage: 2.52 MB (2641928)
>> >> > - PerReadThreadRawHdfsThroughput: 1.31 GB/sec
>> >> > - RemoteScanRanges: 0 (0)
>> >> > - RowBatchQueueGetWaitTime: 246.691ms
>> >> > - RowBatchQueuePutWaitTime: 0.000ns
>> >> > - RowsRead: 0 (0)
>> >> > - RowsReturned: 461 (461)
>> >> > - RowsReturnedRate: 1.77 K/sec
>> >> > - ScanRangesComplete: 461 (461)
>> >> > - ScannerThreadsInvoluntaryContextSwitches: 109 (109)
>> >> > - ScannerThreadsTotalWallClockTime: 6s030ms
>> >> > - MaterializeTupleTime(*): 68.735us
>> >> > - ScannerThreadsSysTime: 33.983ms
>> >> > - ScannerThreadsUserTime: 647.889ms
>> >> > - ScannerThreadsVoluntaryContextSwitches: 1.70K (1696)
>> >> > - TotalRawHdfsReadTime(*): 33.405ms
>> >> > - TotalReadThroughput: 0.00 /sec
>> >> > Instance c843dae48ce9b289:6ad3ffc000000003
>> >> > (host=aopr-dhc002.lpdomain.com:22000):(Total: 302.116ms, non-child:
>> >> > 1.604ms,
>> >> > % non-child: 0.53%)
>> >> > Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>> >> > 0:27/16.21 MB 11:33
>> <https://maps.google.com/?q=MB+11:33&entry=gmail&source=g>/21.23 MB 6:36
>> <https://maps.google.com/?q=MB+6:36&entry=gmail&source=g>/23.15 MB 10:41
>> <https://maps.google.com/?q=MB+10:41&entry=gmail&source=g>/25.09 MB
>> 1:41/24.13 MB
>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+9:32&entry=gmail&source=g>
>> >> > 9:32
>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+9:32&entry=gmail&source=g>/18.79
>> MB 5:41 <https://maps.google.com/?q=MB+5:41&entry=gmail&source=g>/25.31 MB
>> 2:42 <https://maps.google.com/?q=MB+2:42&entry=gmail&source=g>/26.49 MB
>> 7:36 <https://maps.google.com/?q=MB+7:36&entry=gmail&source=g>/22.87 MB
>> 3:51 <https://maps.google.com/?q=MB+3:51&entry=gmail&source=g>/31.00 MB
>> >> > 4:41/25.99 MB 8:38
>> <https://maps.google.com/?q=MB+8:38&entry=gmail&source=g>/23.79 MB
>> >> > MemoryUsage(500.000ms): 700.35 KB
>> >> > ThreadUsage(500.000ms): 25
>> >> > - AverageThreadTokens: 25.00
>> >> > - BloomFilterBytes: 0
>> >> > - PeakMemoryUsage: 2.54 MB (2662753)
>> >> > - PeakReservation: 0
>> >> > - PeakUsedReservation: 0
>> >> > - PerHostPeakMemUsage: 2.54 MB (2662753)
>> >> > - RowsProduced: 1 (1)
>> >> > - TotalNetworkReceiveTime: 0.000ns
>> >> > - TotalNetworkSendTime: 300.675us
>> >> > - TotalStorageWaitTime: 4s661ms
>> >> > - TotalThreadsInvoluntaryContextSwitches: 90 (90)
>> >> > - TotalThreadsTotalWallClockTime: 6s256ms
>> >> > - TotalThreadsSysTime: 30.983ms
>> >> > - TotalThreadsUserTime: 709.880ms
>> >> > - TotalThreadsVoluntaryContextSwitches: 2.17K (2175)
>> >> > Fragment Instance Lifecycle Timings:
>> >> > - ExecTime: 492.627us
>> >> > - ExecTreeExecTime: 8.191us
>> >> > - OpenTime: 277.922ms
>> >> > - ExecTreeOpenTime: 255.395ms
>> >> > - PrepareTime: 23.647ms
>> >> > - ExecTreePrepareTime: 2.243ms
>> >> > DataStreamSender (dst_id=2):(Total: 194.517us, non-child:
>> >> > 194.517us,
>> >> > % non-child: 100.00%)
>> >> > - BytesSent: 16.00 B (16)
>> >> > - NetworkThroughput(*): 371.86 KB/sec
>> >> > - OverallThroughput: 80.33 KB/sec
>> >> > - PeakMemoryUsage: 16.00 KB (16384)
>> >> > - RowsReturned: 1 (1)
>> >> > - SerializeBatchTime: 14.035us
>> >> > - TransmitDataRPCTime: 42.019us
>> >> > - UncompressedRowBatchSize: 16.00 B (16)
>> >> > CodeGen:(Total: 42.676ms, non-child: 42.676ms, % non-child:
>> >> > 100.00%)
>> >> > - CodegenTime: 1.236ms
>> >> > - CompileTime: 7.717ms
>> >> > - LoadTime: 0.000ns
>> >> > - ModuleBitcodeSize: 1.86 MB (1950880)
>> >> > - NumFunctions: 15 (15)
>> >> > - NumInstructions: 214 (214)
>> >> > - OptimizationTime: 14.130ms
>> >> > - PeakMemoryUsage: 107.00 KB (109568)
>> >> > - PrepareTime: 20.109ms
>> >> > AGGREGATION_NODE (id=1):(Total: 257.640ms, non-child:
>> 1.541ms, %
>> >> > non-child: 0.60%)
>> >> > ExecOption: Codegen Enabled
>> >> > - BuildTime: 1.252ms
>> >> > - GetResultsTime: 0.000ns
>> >> > - HTResizeTime: 0.000ns
>> >> > - HashBuckets: 0 (0)
>> >> > - LargestPartitionPercent: 0 (0)
>> >> > - MaxPartitionLevel: 0 (0)
>> >> > - NumRepartitions: 0 (0)
>> >> > - PartitionsCreated: 0 (0)
>> >> > - PeakMemoryUsage: 20.00 KB (20480)
>> >> > - RowsRepartitioned: 0 (0)
>> >> > - RowsReturned: 1 (1)
>> >> > - RowsReturnedRate: 3.00 /sec
>> >> > - SpilledPartitions: 0 (0)
>> >> > HDFS_SCAN_NODE (id=0):(Total: 256.099ms, non-child:
>> 256.099ms, %
>> >> > non-child: 100.00%)
>> >> > Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>> >> > 0:27/16.21 MB 11:33
>> <https://maps.google.com/?q=MB+11:33&entry=gmail&source=g>/21.23 MB 6:36
>> <https://maps.google.com/?q=MB+6:36&entry=gmail&source=g>/23.15 MB 10:41
>> <https://maps.google.com/?q=MB+10:41&entry=gmail&source=g>/25.09 MB
>> 1:41/24.13 MB
>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+9:32&entry=gmail&source=g>
>> >> > 9:32
>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+9:32&entry=gmail&source=g>/18.79
>> MB 5:41 <https://maps.google.com/?q=MB+5:41&entry=gmail&source=g>/25.31 MB
>> 2:42 <https://maps.google.com/?q=MB+2:42&entry=gmail&source=g>/26.49 MB
>> 7:36 <https://maps.google.com/?q=MB+7:36&entry=gmail&source=g>/22.87 MB
>> 3:51 <https://maps.google.com/?q=MB+3:51&entry=gmail&source=g>/31.00 MB
>> >> > 4:41/25.99 MB 8:38
>> <https://maps.google.com/?q=MB+8:38&entry=gmail&source=g>/23.79 MB
>> >> > ExecOption: PARQUET Codegen Enabled, Codegen enabled: 459
>> out
>> >> > of
>> >> > 459
>> >> > Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0%
>> >> > 4:0%
>> >> > 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
>> >> > File Formats: PARQUET/Unknown(Skipped):459
>> >> > BytesRead(500.000ms): 43.15 MB
>> >> > - FooterProcessingTime: (Avg: 11.257ms ; Min: 3.989ms ;
>> Max:
>> >> > 23.004ms ; Number of samples: 459)
>> >> > - AverageHdfsReadThreadConcurrency: 0.00
>> >> > - AverageScannerThreadConcurrency: 24.00
>> >> > - BytesRead: 44.52 MB (46678031)
>> >> > - BytesReadDataNodeCache: 0
>> >> > - BytesReadLocal: 44.52 MB (46678031)
>> >> > - BytesReadRemoteUnexpected: 0
>> >> > - BytesReadShortCircuit: 44.52 MB (46678031)
>> >> > - CachedFileHandlesHitCount: 0 (0)
>> >> > - CachedFileHandlesMissCount: 459 (459)
>> >> > - CollectionItemsRead: 0 (0)
>> >> > - DecompressionTime: 0.000ns
>> >> > - MaxCompressedTextFileLength: 0
>> >> > - NumColumns: 0 (0)
>> >> > - NumDictFilteredRowGroups: 0 (0)
>> >> > - NumDisksAccessed: 12 (12)
>> >> > - NumRowGroups: 459 (459)
>> >> > - NumScannerThreadsStarted: 24 (24)
>> >> > - NumScannersWithNoReads: 0 (0)
>> >> > - NumStatsFilteredRowGroups: 0 (0)
>> >> > - PeakMemoryUsage: 2.50 MB (2621440)
>> >> > - PerReadThreadRawHdfsThroughput: 1.33 GB/sec
>> >> > - RemoteScanRanges: 0 (0)
>> >> > - RowBatchQueueGetWaitTime: 243.426ms
>> >> > - RowBatchQueuePutWaitTime: 0.000ns
>> >> > - RowsRead: 0 (0)
>> >> > - RowsReturned: 459 (459)
>> >> > - RowsReturnedRate: 1.79 K/sec
>> >> > - ScanRangesComplete: 459 (459)
>> >> > - ScannerThreadsInvoluntaryContextSwitches: 85 (85)
>> >> > - ScannerThreadsTotalWallClockTime: 5s978ms
>> >> > - MaterializeTupleTime(*): 67.997us
>> >> > - ScannerThreadsSysTime: 22.984ms
>> >> > - ScannerThreadsUserTime: 678.884ms
>> >> > - ScannerThreadsVoluntaryContextSwitches: 1.72K (1724)
>> >> > - TotalRawHdfsReadTime(*): 32.626ms
>> >> > - TotalReadThroughput: 86.30 MB/sec
>> >> > Instance c843dae48ce9b289:6ad3ffc000000005
>> >> > (host=aopr-dhc001.lpdomain.com:22000):(Total: 298.093ms, non-child:
>> >> > 1.242ms,
>> >> > % non-child: 0.42%)
>> >> > Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>> >> > 9:55/35.38 MB 0:37/24.23 MB 5:37
>> <https://maps.google.com/?q=MB+5:37&entry=gmail&source=g>/23.31 MB 11:22
>> <https://maps.google.com/?q=MB+11:22&entry=gmail&source=g>/13.79 MB 8:30
>> <https://maps.google.com/?q=MB+8:30&entry=gmail&source=g>/18.53 MB
>> >> > 4:33/20.45 MB 1:38/24.06 MB 10:29
>> <https://maps.google.com/?q=MB+10:29&entry=gmail&source=g>/18.40 MB 3:39
>> <https://maps.google.com/?q=MB+3:39&entry=gmail&source=g>/24.35 MB 2:41
>> <https://maps.google.com/?q=MB+2:41&entry=gmail&source=g>/25.45 MB
>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+6:42&entry=gmail&source=g>
>> >> > 6:42
>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+6:42&entry=gmail&source=g>/27.03
>> MB 7:47 <https://maps.google.com/?q=MB+7:47&entry=gmail&source=g>/29.22
>> MB
>> >> > - AverageThreadTokens: 0.00
>> >> > - BloomFilterBytes: 0
>> >> > - PeakMemoryUsage: 2.66 MB (2793825)
>> >> > - PeakReservation: 0
>> >> > - PeakUsedReservation: 0
>> >> > - PerHostPeakMemUsage: 2.68 MB (2806113)
>> >> > - RowsProduced: 1 (1)
>> >> > - TotalNetworkReceiveTime: 0.000ns
>> >> > - TotalNetworkSendTime: 198.894us
>> >> > - TotalStorageWaitTime: 4s374ms
>> >> > - TotalThreadsInvoluntaryContextSwitches: 110 (110)
>> >> > - TotalThreadsTotalWallClockTime: 6s201ms
>> >> > - TotalThreadsSysTime: 29.983ms
>> >> > - TotalThreadsUserTime: 642.892ms
>> >> > - TotalThreadsVoluntaryContextSwitches: 2.17K (2174)
>> >> > Fragment Instance Lifecycle Timings:
>> >> > - ExecTime: 350.873us
>> >> > - ExecTreeExecTime: 8.173us
>> >> > - OpenTime: 275.347ms
>> >> > - ExecTreeOpenTime: 253.425ms
>> >> > - PrepareTime: 22.341ms
>> >> > - ExecTreePrepareTime: 2.251ms
>> >> > DataStreamSender (dst_id=2):(Total: 164.394us, non-child:
>> >> > 164.394us,
>> >> > % non-child: 100.00%)
>> >> > - BytesSent: 16.00 B (16)
>> >> > - NetworkThroughput(*): 493.42 KB/sec
>> >> > - OverallThroughput: 95.05 KB/sec
>> >> > - PeakMemoryUsage: 16.00 KB (16384)
>> >> > - RowsReturned: 1 (1)
>> >> > - SerializeBatchTime: 14.262us
>> >> > - TransmitDataRPCTime: 31.667us
>> >> > - UncompressedRowBatchSize: 16.00 B (16)
>> >> > CodeGen:(Total: 41.008ms, non-child: 41.008ms, % non-child:
>> >> > 100.00%)
>> >> > - CodegenTime: 1.049ms
>> >> > - CompileTime: 7.248ms
>> >> > - LoadTime: 0.000ns
>> >> > - ModuleBitcodeSize: 1.86 MB (1950880)
>> >> > - NumFunctions: 15 (15)
>> >> > - NumInstructions: 214 (214)
>> >> > - OptimizationTime: 14.018ms
>> >> > - PeakMemoryUsage: 107.00 KB (109568)
>> >> > - PrepareTime: 19.080ms
>> >> > AGGREGATION_NODE (id=1):(Total: 255.678ms, non-child:
>> 1.046ms, %
>> >> > non-child: 0.41%)
>> >> > ExecOption: Codegen Enabled
>> >> > - BuildTime: 1.015ms
>> >> > - GetResultsTime: 0.000ns
>> >> > - HTResizeTime: 0.000ns
>> >> > - HashBuckets: 0 (0)
>> >> > - LargestPartitionPercent: 0 (0)
>> >> > - MaxPartitionLevel: 0 (0)
>> >> > - NumRepartitions: 0 (0)
>> >> > - PartitionsCreated: 0 (0)
>> >> > - PeakMemoryUsage: 20.00 KB (20480)
>> >> > - RowsRepartitioned: 0 (0)
>> >> > - RowsReturned: 1 (1)
>> >> > - RowsReturnedRate: 3.00 /sec
>> >> > - SpilledPartitions: 0 (0)
>> >> > HDFS_SCAN_NODE (id=0):(Total: 254.631ms, non-child:
>> 254.631ms, %
>> >> > non-child: 100.00%)
>> >> > Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>> >> > 9:55/35.38 MB 0:37/24.23 MB 5:37
>> <https://maps.google.com/?q=MB+5:37&entry=gmail&source=g>/23.31 MB 11:22
>> <https://maps.google.com/?q=MB+11:22&entry=gmail&source=g>/13.79 MB 8:30
>> <https://maps.google.com/?q=MB+8:30&entry=gmail&source=g>/18.53 MB
>> >> > 4:33/20.45 MB 1:38/24.06 MB 10:29
>> <https://maps.google.com/?q=MB+10:29&entry=gmail&source=g>/18.40 MB 3:39
>> <https://maps.google.com/?q=MB+3:39&entry=gmail&source=g>/24.35 MB 2:41
>> <https://maps.google.com/?q=MB+2:41&entry=gmail&source=g>/25.45 MB
>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+6:42&entry=gmail&source=g>
>> >> > 6:42
>> <https://maps.google.com/?q=MB+%0D+%3E%3E+%3E+6:42&entry=gmail&source=g>/27.03
>> MB 7:47 <https://maps.google.com/?q=MB+7:47&entry=gmail&source=g>/29.22
>> MB
>> >> > ExecOption: PARQUET Codegen Enabled, Codegen enabled: 450
>> out
>> >> > of
>> >> > 450
>> >> > Hdfs Read Thread Concurrency Bucket: 0:0% 1:0% 2:0% 3:0%
>> 4:0%
>> >> > 5:0%
>> >> > 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
>> >> > File Formats: PARQUET/Unknown(Skipped):450
>> >> > - FooterProcessingTime: (Avg: 10.727ms ; Min: 3.813ms ;
>> Max:
>> >> > 23.755ms ; Number of samples: 450)
>> >> > - AverageHdfsReadThreadConcurrency: 0.00
>> >> > - AverageScannerThreadConcurrency: 0.00
>> >> > - BytesRead: 43.85 MB (45983284)
>> >> > - BytesReadDataNodeCache: 0
>> >> > - BytesReadLocal: 43.85 MB (45983284)
>> >> > - BytesReadRemoteUnexpected: 0
>> >> > - BytesReadShortCircuit: 43.85 MB (45983284)
>> >> > - CachedFileHandlesHitCount: 0 (0)
>> >> > - CachedFileHandlesMissCount: 450 (450)
>> >> > - CollectionItemsRead: 0 (0)
>> >> > - DecompressionTime: 0.000ns
>> >> > - MaxCompressedTextFileLength: 0
>> >> > - NumColumns: 0 (0)
>> >> > - NumDictFilteredRowGroups: 0 (0)
>> >> > - NumDisksAccessed: 12 (12)
>> >> > - NumRowGroups: 450 (450)
>> >> > - NumScannerThreadsStarted: 24 (24)
>> >> > - NumScannersWithNoReads: 0 (0)
>> >> > - NumStatsFilteredRowGroups: 0 (0)
>> >> > - PeakMemoryUsage: 2.62 MB (2752512)
>> >> > - PerReadThreadRawHdfsThroughput: 1.56 GB/sec
>> >> > - RemoteScanRanges: 0 (0)
>> >> > - RowBatchQueueGetWaitTime: 242.362ms
>> >> > - RowBatchQueuePutWaitTime: 0.000ns
>> >> > - RowsRead: 0 (0)
>> >> > - RowsReturned: 450 (450)
>> >> > - RowsReturnedRate: 1.77 K/sec
>> >> > - ScanRangesComplete: 450 (450)
>> >> > - ScannerThreadsInvoluntaryContextSwitches: 106 (106)
>> >> > - ScannerThreadsTotalWallClockTime: 5s926ms
>> >> > - MaterializeTupleTime(*): 69.447us
>> >> > - ScannerThreadsSysTime: 26.984ms
>> >> > - ScannerThreadsUserTime: 607.897ms
>> >> > - ScannerThreadsVoluntaryContextSwitches: 1.72K (1723)
>> >> > - TotalRawHdfsReadTime(*): 27.393ms
>> >> > - TotalReadThroughput: 0.00 /sec
>> >> > ===================================================
>> >> >
>> >> > Coordinator log:
>> >> >
>> >> > 01: query (string) = "select count(*) from
>> dv_entersiteevent_prq_remote
>> >> > where year=2017 and month=10",
>> >> > 03: configuration (list) = list<string>[1] {
>> >> > [0] = "REQUEST_POOL=impreports",
>> >> > },
>> >> > 04: hadoop_user (string) = "fawzea",
>> >> > }
>> >> > I0208 01:07:06.321281 7154 impala-beeswax-server.cc:459]
>> >> > TClientRequest.queryOptions: TQueryOptions {
>> >> > 01: abort_on_error (bool) = false,
>> >> > 02: max_errors (i32) = 100,
>> >> > 03: disable_codegen (bool) = false,
>> >> > 04: batch_size (i32) = 0,
>> >> > 05: num_nodes (i32) = 0,
>> >> > 06: max_scan_range_length (i64) = 0,
>> >> > 07: num_scanner_threads (i32) = 0,
>> >> > 08: max_io_buffers (i32) = 0,
>> >> > 09: allow_unsupported_formats (bool) = false,
>> >> > 10: default_order_by_limit (i64) = -1,
>> >> > 11: debug_action (string) = "",
>> >> > 12: mem_limit (i64) = 0,
>> >> > 13: abort_on_default_limit_exceeded (bool) = false,
>> >> > 15: hbase_caching (i32) = 0,
>> >> > 16: hbase_cache_blocks (bool) = false,
>> >> > 17: parquet_file_size (i64) = 0,
>> >> > 18: explain_level (i32) = 1,
>> >> > 19: sync_ddl (bool) = false,
>> >> > 20: request_pool (string) = "impreports",
>> >> > 23: disable_cached_reads (bool) = false,
>> >> > 24: disable_outermost_topn (bool) = false,
>> >> > 25: rm_initial_mem (i64) = 0,
>> >> > 26: query_timeout_s (i32) = 0,
>> >> > 28: appx_count_distinct (bool) = false,
>> >> > 29: disable_unsafe_spills (bool) = false,
>> >> > 31: exec_single_node_rows_threshold (i32) = 100,
>> >> > 32: optimize_partition_key_scans (bool) = false,
>> >> > 33: replica_preference (i32) = 0,
>> >> > 34: schedule_random_replica (bool) = false,
>> >> > 35: scan_node_codegen_threshold (i64) = 1800000,
>> >> > 36: disable_streaming_preaggregations (bool) = false,
>> >> > 37: runtime_filter_mode (i32) = 2,
>> >> > 38: runtime_bloom_filter_size (i32) = 1048576,
>> >> > 39: runtime_filter_wait_time_ms (i32) = 0,
>> >> > 40: disable_row_runtime_filtering (bool) = false,
>> >> > 41: max_num_runtime_filters (i32) = 10,
>> >> > 42: parquet_annotate_strings_utf8 (bool) = false,
>> >> > 43: parquet_fallback_schema_resolution (i32) = 0,
>> >> > 45: s3_skip_insert_staging (bool) = true,
>> >> > 46: runtime_filter_min_size (i32) = 1048576,
>> >> > 47: runtime_filter_max_size (i32) = 16777216,
>> >> > 48: prefetch_mode (i32) = 1,
>> >> > 49: strict_mode (bool) = false,
>> >> > 50: scratch_limit (i64) = -1,
>> >> > 51: enable_expr_rewrites (bool) = true,
>> >> > 52: decimal_v2 (bool) = false,
>> >> > 53: parquet_dictionary_filtering (bool) = true,
>> >> > 54: parquet_array_resolution (i32) = 2,
>> >> > 55: parquet_read_statistics (bool) = true,
>> >> > 56: default_join_distribution_mode (i32) = 0,
>> >> > 57: disable_codegen_rows_threshold (i32) = 50000,
>> >> > 58: default_spillable_buffer_size (i64) = 2097152,
>> >> > 59: min_spillable_buffer_size (i64) = 65536,
>> >> > 60: max_row_size (i64) = 524288,
>> >> > }
>> >> > I0208 01:07:06.324410 7154 Frontend.java:909] Compiling query:
>> select
>> >> > count(*) from dv_entersiteevent_prq_remote where year=2017 and
>> month=10
>> >> > I0208 01:07:06.327203 7154 Frontend.java:948] Compiled query.
>> >> > I0208 01:07:06.359748 7154 admission-controller.cc:510] Schedule for
>> >> > id=5f49e927591a9042:faf8f28600000000 in pool_name=root.impreports
>> >> > cluster_mem_needed=105.00 MB PoolConfig: max_requests=-1
>> max_queued=200
>> >> > max_mem=-1.00 B
>> >> > I0208 01:07:06.359776 7154 admission-controller.cc:515] Stats:
>> >> > agg_num_running=0, agg_num_queued=0, agg_mem_reserved=0,
>> >> > local_host(local_mem_admitted=0, num_admitted_running=0,
>> num_queued=0,
>> >> > backend_mem_reserved=0)
>> >> > I0208 01:07:06.359784 7154 admission-controller.cc:531] Admitted
>> query
>> >> > id=5f49e927591a9042:faf8f28600000000
>> >> > I0208 01:07:06.359807 7154 coordinator.cc:99] Exec()
>> >> > query_id=5f49e927591a9042:faf8f28600000000 stmt=select count(*) from
>> >> > dv_entersiteevent_prq_remote where year=2017 and month=10
>> >> > I0208 01:07:06.360694 7154 coordinator.cc:357] starting execution
>> on 5
>> >> > backends for query 5f49e927591a9042:faf8f28600000000
>> >> > I0208 01:07:06.363255 63583 impala-internal-service.cc:44]
>> >> > ExecQueryFInstances(): query_id=5f49e927591a9042:faf8f28600000000
>> >> > I0208 01:07:06.363282 63583 query-exec-mgr.cc:46]
>> StartQueryFInstances()
>> >> > query_id=5f49e927591a9042:faf8f28600000000
>> >> > coord=aopr-dhc001.lpdomain.com:22000
>> >> > I0208 01:07:06.363296 63583 query-state.cc:173] Buffer pool limit for
>> >> > 5f49e927591a9042:faf8f28600000000: 10802010521
>> >> > I0208 01:07:06.363364 63583 initial-reservations.cc:60] Successfully
>> >> > claimed
>> >> > initial reservations (0) for query 5f49e927591a9042:faf8f28600000000
>> >> > I0208 01:07:06.363544 7155 query-state.cc:286] StartFInstances():
>> >> > query_id=5f49e927591a9042:faf8f28600000000 #instances=2
>> >> > I0208 01:07:06.363844 7155 query-state.cc:299] descriptor table for
>> >> > query=5f49e927591a9042:faf8f28600000000
>> >> > tuples:
>> >> > Tuple(id=2 size=0 slots=[] tuple_path=[])
>> >> > Tuple(id=1 size=8 slots=[Slot(id=2 type=BIGINT col_path=[] offset=0
>> >> > null=(offset=0 mask=0) slot_idx=0 field_idx=-1)] tuple_path=[])
>> >> > Tuple(id=0 size=8 slots=[Slot(id=3 type=BIGINT col_path=[] offset=0
>> >> > null=(offset=0 mask=0) slot_idx=0 field_idx=-1)] tuple_path=[])
>> >> > I0208 01:07:06.364013 7156 query-state.cc:377] Executing instance.
>> >> > instance_id=5f49e927591a9042:faf8f28600000000 fragment_idx=0
>> >> > per_fragment_instance_idx=0 coord_state_idx=0 #in-flight=1
>> >> > I0208 01:07:06.364135 7157 query-state.cc:377] Executing instance.
>> >> > instance_id=5f49e927591a9042:faf8f28600000005 fragment_idx=1
>> >> > per_fragment_instance_idx=4 coord_state_idx=0 #in-flight=2
>> >> > I0208 01:07:06.364241 7157 hdfs-scan-node.cc:160] Max row batch
>> queue
>> >> > size
>> >> > for scan node '0' in fragment instance
>> >> > '5f49e927591a9042:faf8f28600000005':
>> >> > 160
>> >> > I0208 01:07:06.366775 7154 coordinator.cc:370] started execution on
>> 5
>> >> > backends for query 5f49e927591a9042:faf8f28600000000
>> >> > I0208 01:07:06.385625 7155 query-exec-mgr.cc:149]
>> ReleaseQueryState():
>> >> > query_id=5f49e927591a9042:faf8f28600000000 refcnt=4
>> >> > I0208 01:07:06.671193 7559 impala-server.cc:1796] Connection from
>> >> > client
>> >> > ::ffff:10.16.136.3:55598 closed, closing 1 associated session(s)
>> >> > I0208 01:07:06.676138 544 data-stream-mgr.cc:228] Reduced stream ID
>> >> > cache
>> >> > from 2 items, to 0, eviction took: 0
>> >> > I0208 01:07:06.677202 544 coordinator.cc:955] Backend completed:
>> >> > host=aopr-dhc002.lpdomain.com:22000 remaining=4
>> >> > I0208 01:07:06.677214 544 coordinator-backend-state.cc:229]
>> >> > query_id=5f49e927591a9042:faf8f28600000000: first in-progress
>> backend:
>> >> > aopr-dhc001.lpdomain.com:22000
>> >> > I0208 01:07:06.684221 7157 query-state.cc:384] Instance completed.
>> >> > instance_id=5f49e927591a9042:faf8f28600000005 #in-flight=1 status=OK
>> >> > I0208 01:07:06.684237 7157 query-exec-mgr.cc:149]
>> ReleaseQueryState():
>> >> > query_id=5f49e927591a9042:faf8f28600000000 refcnt=3
>> >> > I0208 01:07:06.700275 520 coordinator.cc:955] Backend completed:
>> >> > host=aopr-dhc005.lpdomain.com:22000 remaining=3
>> >> > I0208 01:07:06.700289 520 coordinator-backend-state.cc:229]
>> >> > query_id=5f49e927591a9042:faf8f28600000000: first in-progress
>> backend:
>> >> > aopr-dhc001.lpdomain.com:22000
>> >> > I0208 01:07:06.704620 65258 coordinator.cc:955] Backend completed:
>> >> > host=aopr-dhc003.lpdomain.com:22000 remaining=2
>> >> > I0208 01:07:06.704666 65258 coordinator-backend-state.cc:229]
>> >> > query_id=5f49e927591a9042:faf8f28600000000: first in-progress
>> backend:
>> >> > aopr-dhc001.lpdomain.com:22000
>> >> > I0208 01:07:06.706041 7156 data-stream-mgr.cc:238]
>> DeregisterRecvr():
>> >> > fragment_instance_id=5f49e927591a9042:faf8f28600000000, node=2
>> >> > I0208 01:07:06.706070 7156 data-stream-recvr.cc:235] cancelled
>> stream:
>> >> > fragment_instance_id_=5f49e927591a9042:faf8f28600000000 node_id=2
>> >> > I0208 01:07:06.706809 64916 coordinator.cc:955] Backend completed:
>> >> > host=aopr-dhc004.lpdomain.com:22000 remaining=1
>> >> > I0208 01:07:06.706821 64916 coordinator-backend-state.cc:229]
>> >> > query_id=5f49e927591a9042:faf8f28600000000: first in-progress
>> backend:
>> >> > aopr-dhc001.lpdomain.com:22000
>> >> > I0208 01:07:06.789568 7154 impala-beeswax-server.cc:193]
>> >> > get_results_metadata(): query_id=5f49e927591a9042:faf8f28600000000
>> >> > I0208 01:07:06.790439 7154 coordinator.cc:789] Coordinator waiting
>> for
>> >> > backends to finish, 1 remaining
>> >> > I0208 01:07:06.791052 7154 coordinator.cc:794] All backends finished
>> >> > successfully.
>> >> > I0208 01:07:06.791086 7154 coordinator.cc:1090] Release admssion
>> >> > control
>> >> > resources for query 5f49e927591a9042:faf8f28600000000
>> >> > I0208 01:07:06.791363 7156 query-state.cc:384] Instance completed.
>> >> > instance_id=5f49e927591a9042:faf8f28600000000 #in-flight=0 status=OK
>> >> > I0208 01:07:06.791414 7156 query-exec-mgr.cc:149]
>> ReleaseQueryState():
>> >> > query_id=5f49e927591a9042:faf8f28600000000 refcnt=2
>> >> > I0208 01:07:06.793586 7154 impala-beeswax-server.cc:235] close():
>> >> > query_id=5f49e927591a9042:faf8f28600000000
>> >> > I0208 01:07:06.793602 7154 impala-server.cc:992] UnregisterQuery():
>> >> > query_id=5f49e927591a9042:faf8f28600000000
>> >> > I0208 01:07:06.793614 7154 impala-server.cc:1075] Cancel():
>> >> > query_id=5f49e927591a9042:faf8f28600000000
>> >> > I0208 01:07:06.793624 7154 coordinator.cc:895] Cancel()
>> >> > query_id=5f49e927591a9042:faf8f28600000000
>> >> > I0208 01:07:06.793637 7154 coordinator.cc:905] CancelBackends()
>> >> > query_id=5f49e927591a9042:faf8f28600000000, tried to cancel 0
>> backends
>> >> > I0208 01:07:06.802776 7154 query-exec-mgr.cc:149]
>> ReleaseQueryState():
>> >> > query_id=5f49e927591a9042:faf8f28600000000 refcnt=1
>> >> > I0208 01:07:06.814214 7154 impala-server.cc:1796] Connection from
>> >> > client
>> >> > ::ffff:10.16.144.131:51012 closed, closing 1 associated session(s)
>> >> > I0208 01:07:07.258891 7567 impala-server.cc:1796] Connection from
>> >> > client
>> >> > ::ffff:10.16.136.2:56375 closed, closing 1 associated session(s)
>> >> > I0208 01:07:11.600131 7819 impala-server.cc:1796] Connection from
>> >> > client
>> >> > ::ffff:10.16.136.3:27460 closed, closing 1 associated session(s)
>> >> > I0208 01:07:12.262739 7821 impala-server.cc:1796] Connection from
>> >> > client
>> >> > ::ffff:10.16.136.2:52672 closed, closing 1 associated session(s)
>> >> > I0208 01:07:16.665451 7822 impala-server.cc:1796] Connection from
>> >> > client
>> >> > ::ffff:10.16.136.3:10693 closed, closing 1 associated session(s)
>> >> > I0208 01:07:17.235321 7825 impala-server.cc:1796] Connection from
>> >> > client
>> >> > ::ffff:10.16.136.2:34752 closed, closing 1 associated session(s)
>> >> > I0208 01:07:21.410611 5858 impala-beeswax-server.cc:193]
>> >> > get_results_metadata(): query_id=1d4fbf0f2571f1e9:ea346f7900000000
>> >> > I0208 01:07:21.414175 5858 impala-beeswax-server.cc:235] close():
>> >> > query_id=1d4fbf0f2571f1e9:ea346f7900000000
>> >> > I0208 01:07:21.414185 5858 impala-server.cc:992] UnregisterQuery():
>> >> > query_id=1d4fbf0f2571f1e9:ea346f7900000000
>> >> > I0208 01:07:21.414191 5858 impala-server.cc:1075] Cancel():
>> >> > query_id=1d4fbf0f2571f1e9:ea346f7900000000
>> >> > I0208 01:07:21.421505 5858 impala-server.cc:1796] Connection from
>> >> > client
>> >> > ::ffff:10.26.180.75:60072 closed, closing 1 associated session(s)
>> >> > I0208 01:07:21.605690 7830 impala-server.cc:1796] Connection from
>> >> > client
>> >> > ::ffff:10.16.136.3:46660 closed, closing 1 associated session(s)
>> >> >
>> >> >
>> >> > On Thu, Feb 8, 2018 at 6:20 AM, Matthew Jacobs <mj...@apache.org>
>> >> > wrote:
>> >> >>
>> >> >> Thanks! I think that configuration looks OK.
>> >> >>
>> >> >> 1) What version of Impala is this?
>> >> >>
>> >> >> 2) Can you send the query profile and the Impala coordinator log for
>> >> >> the query where it's not behaving as expected?
>> >> >>
>> >> >> > When i'm running the following with fawzea user:
>> >> >> > set request_pool=impreports;
>> >> >> > select count(*) from dv_entersiteevent_prq_remote where year=2017
>> and
>> >> >> > month=10;
>> >> >> > I can see that the query is running with impreports pool.
>> >> >>
>> >> >> On Wed, Feb 7, 2018 at 7:59 PM, Fawze Abujaber <fa...@gmail.com>
>> >> >> wrote:
>> >> >> > Hi Matt,
>> >> >> >
>> >> >> > [root@aopr-dhc001 ~]# cat
>> >> >> >
>> >> >> >
>> >> >> > /var/run/cloudera-scm-agent/process/4558-impala-IMPALAD/impa
>> la-conf/fair-scheduler.xml
>> >> >> > <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
>> >> >> > <allocations>
>> >> >> > <queue name="root">
>> >> >> > <weight>1.0</weight>
>> >> >> > <schedulingPolicy>fair</schedulingPolicy>
>> >> >> > <queue name="impiaas">
>> >> >> > <weight>1.0</weight>
>> >> >> > <schedulingPolicy>fair</schedulingPolicy>
>> >> >> > <aclSubmitApps>fawzea </aclSubmitApps>
>> >> >> > </queue>
>> >> >> > <queue name="impreports">
>> >> >> > <weight>1.0</weight>
>> >> >> > <schedulingPolicy>fair</schedulingPolicy>
>> >> >> > <aclSubmitApps>nira </aclSubmitApps>
>> >> >> > </queue>
>> >> >> > </queue>
>> >> >> > <defaultQueueSchedulingPolicy>fair</defaultQueueSchedulingPol
>> icy>
>> >> >> > <queuePlacementPolicy>
>> >> >> > <rule name="specified" create="false"/>
>> >> >> > <rule name="secondaryGroupExistingQueue"/>
>> >> >> > <rule name="default"/>
>> >> >> > </queuePlacementPolicy>
>> >> >> > </allocations>
>> >> >> >
>> >> >> >
>> >> >> > On Thu, Feb 8, 2018 at 4:47 AM, Matthew Jacobs <
>> jacobs.mh@gmail.com>
>> >> >> > wrote:
>> >> >> >>
>> >> >> >> Yes, I saw that, but I'd like to see the actual configuration
>> that
>> >> >> >> is
>> >> >> >> passed to Impala because it may not be specified in the right
>> way to
>> >> >> >> accomplish what you're trying to do in "b".
>> >> >> >>
>> >> >> >> On Wed, Feb 7, 2018 at 4:32 PM, Fawze Abujaber <
>> fawzeaj@gmail.com>
>> >> >> >> wrote:
>> >> >> >> > Hi Matt,
>> >> >> >> >
>> >> >> >> > I created the pools manually and it’s completely nfigurations
>> as
>> >> >> >> > shown
>> >> >> >> > in b.
>> >> >> >> >
>> >> >> >> > On Wed, 7 Feb 2018 at 23:17 Matthew Jacobs <
>> jacobs.mh@gmail.com>
>> >> >> >> > wrote:
>> >> >> >> >>
>> >> >> >> >> How, more specifically, are you configuring these pools? Can
>> you
>> >> >> >> >> share
>> >> >> >> >> the configuration files?
>> >> >> >> >>
>> >> >> >> >> On Wed, Feb 7, 2018 at 11:42 AM, Fawze Abujaber
>> >> >> >> >> <fa...@gmail.com>
>> >> >> >> >> wrote:
>> >> >> >> >> > Hello All,
>> >> >> >> >> >
>> >> >> >> >> > Will appreciate any insight of what i'm missing in my
>> >> >> >> >> > configuration
>> >> >> >> >> > to
>> >> >> >> >> > get
>> >> >> >> >> > the impala submission control working as expected ....
>> >> >> >> >> >
>> >> >> >> >> > a- Here is my placement Placement Rule:
>> >> >> >> >> > 1- Use the pool Specified at run time., only if the pool
>> >> >> >> >> > exists.
>> >> >> >> >> > 2- Use the pool root.[secondary group].
>> >> >> >> >> > 3- Use the pool root.default.
>> >> >> >> >> >
>> >> >> >> >> > b- Here is the impala resource pools
>> >> >> >> >> >
>> >> >> >> >> > Name Max Memory Max Running Queries Max
>> >> >> >> >> > Queued
>> >> >> >> >> > Queries
>> >> >> >> >> > Queue Timeout Default Query Memory Limit Access Control
>> >> >> >> >> > root.impiaas Unlimited
>> Unlimited
>> >> >> >> >> > 200 1 minute(s) No
>> Default
>> >> >> >> >> > Custom
>> >> >> >> >> > root.impreports Unlimited Unlimited
>> >> >> >> >> > 200 1 minute(s) No
>> Default
>> >> >> >> >> > Custom
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> > C- Here is my Access control for root.impiaas pool:
>> >> >> >> >> >
>> >> >> >> >> > Allow these users and groups to submit to this pool
>> >> >> >> >> >
>> >> >> >> >> > Users: fawzea
>> >> >> >> >> >
>> >> >> >> >> > The same for impreports the allowed user and groups to
>> submit
>> >> >> >> >> > to
>> >> >> >> >> > pool
>> >> >> >> >> > is
>> >> >> >> >> > only nira.
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> > LDAP groups for fawzea user:
>> >> >> >> >> >
>> >> >> >> >> > Member in groups:
>> >> >> >> >> >
>> >> >> >> >> > - dba
>> >> >> >> >> > - impiaas
>> >> >> >> >> > - mysql
>> >> >> >> >> >
>> >> >> >> >> > LDAP group for nira user
>> >> >> >> >> > Member in groups:
>> >> >> >> >> > - dba
>> >> >> >> >> > - impreports
>> >> >> >> >> > - mysql
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> > Impala Queries UI shows:
>> >> >> >> >> >
>> >> >> >> >> > 1- select count(*) from dv_entersiteevent_prq_remote where
>> >> >> >> >> > year=2017
>> >> >> >> >> >
>> >> >> >> >> > User: fawzea
>> >> >> >> >> > Pool: root.impiaas
>> >> >> >> >> >
>> >> >> >> >> > 2- select count(*) from dv_entersiteevent_prq_remote where
>> >> >> >> >> > year=2017
>> >> >> >> >> > and
>> >> >> >> >> > month=10
>> >> >> >> >> > User: nira
>> >> >> >> >> > Pool: root.impreports
>> >> >> >> >> >
>> >> >> >> >> > As shown above that the queries submitted based on the
>> >> >> >> >> > secondary
>> >> >> >> >> > group
>> >> >> >> >> > rule.
>> >> >> >> >> >
>> >> >> >> >> > When i'm running the following with fawzea user:
>> >> >> >> >> >
>> >> >> >> >> > set request_pool=impreports;
>> >> >> >> >> > select count(*) from dv_entersiteevent_prq_remote where
>> >> >> >> >> > year=2017
>> >> >> >> >> > and
>> >> >> >> >> > month=10;
>> >> >> >> >> >
>> >> >> >> >> > I can see that the query is running with impreports pool.
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> > What i'm missing here in my configuration to get this works
>> as
>> >> >> >> >> > expected.
>> >> >> >> >> >
>> >> >> >
>> >> >> >
>> >> >
>> >> >
>> >
>> >
>>
>
>
Re: impala submission control
Posted by Fawze Abujaber <fa...@gmail.com>.
This is not fitting my situation,
I removed even the placement role of the secondary group, and other users
can still submit to the pool that i restrict it to my user.
Just thinking in loud and think the problem might be in using the placement
rule:
Use the pool Specified at run time., only if the pool exists.
Edit
as it it the first rule, it always successes.
On Thu, Feb 8, 2018 at 7:12 PM, Matthew Jacobs <ja...@gmail.com> wrote:
> The aclSubmitApps can specify users or groups, so if the user name is
> also a group name, that can lead to confusion, e.g.
> https://issues.apache.org/jira/browse/IMPALA-2578
> I'm not saying that is necessarily the problem, but something to
> check. If we still can't find anything configured incorrectly I'd
> recommend that you file a JIRA.
>
> On Thu, Feb 8, 2018 at 9:05 AM, Fawze Abujaber <fa...@gmail.com> wrote:
> > Hi Matt,
> >
> > If this is the situation, how we can explain that when i submit the
> query it
> > runs on my secondary group pool and when nira do the same it runs under
> her
> > secondary group pool.
> >
> > More than this, the issue not related to LDAP group as i'm using the
> > submission restriction to a user and not a group.
> >
> > On Thu, Feb 8, 2018 at 7:00 PM, Matthew Jacobs <ja...@gmail.com>
> wrote:
> >>
> >> As far as I can tell from this information, your configuration looks
> >> fine. The only other potential issue I can think of is that the
> >> secondary groups placement rule isn't picking up your LDAP groups
> >> properly. Does the ACL behavior work if you remove the secondary
> >> groups rule?
> >>
> >> On Wed, Feb 7, 2018 at 10:09 PM, Fawze Abujaber <fa...@gmail.com>
> wrote:
> >> > Hi Matt,
> >> >
> >> > v2.11.0-cdh5.14.0
> >> >
> >> > Query Runtime Profile:
> >> > Query (id=c843dae48ce9b289:6ad3ffc000000000):
> >> > Summary:
> >> > Session ID: ca420ac477772b56:51ae460ce9e44999
> >> > Session Type: BEESWAX
> >> > Start Time: 2018-02-08 01:02:01.023558000
> >> > End Time: 2018-02-08 01:02:01.501506000
> >> > Query Type: QUERY
> >> > Query State: FINISHED
> >> > Query Status: OK
> >> > Impala Version: impalad version 2.11.0-cdh5.14.0 RELEASE (build
> >> > d68206561bce6b26762d62c01a78e6cd27aa7690)
> >> > User: fawzea
> >> > Connected User: fawzea
> >> > Delegated User:
> >> > Network Address: ::ffff:10.16.144.131:50078
> >> > Default Db: default
> >> > Sql Statement: select count(*) from dv_entersiteevent_prq_remote
> >> > where
> >> > year=2017 and month=10
> >> > Coordinator: aopr-dhc001.lpdomain.com:22000
> >> > Query Options (set by configuration): REQUEST_POOL=impreports
> >> > Query Options (set by configuration and planner):
> >> > REQUEST_POOL=impreports,MT_DOP=0
> >> > Plan:
> >> > ----------------
> >> > Max Per-Host Resource Reservation: Memory=0B
> >> > Per-Host Resource Estimates: Memory=21.00MB
> >> > WARNING: The following tables are missing relevant table and/or column
> >> > statistics.
> >> > default.dv_entersiteevent_prq_remote
> >> >
> >> > F01:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1
> >> > | Per-Host Resources: mem-estimate=10.00MB mem-reservation=0B
> >> > PLAN-ROOT SINK
> >> > | mem-estimate=0B mem-reservation=0B
> >> > |
> >> > 03:AGGREGATE [FINALIZE]
> >> > | output: count:merge(*)
> >> > | mem-estimate=10.00MB mem-reservation=0B spill-buffer=2.00MB
> >> > | tuple-ids=1 row-size=8B cardinality=1
> >> > |
> >> > 02:EXCHANGE [UNPARTITIONED]
> >> > | mem-estimate=0B mem-reservation=0B
> >> > | tuple-ids=1 row-size=8B cardinality=1
> >> > |
> >> > F00:PLAN FRAGMENT [RANDOM] hosts=5 instances=5
> >> > Per-Host Resources: mem-estimate=11.00MB mem-reservation=0B
> >> > 01:AGGREGATE
> >> > | output:
> >> > sum_init_zero(default.dv_entersiteevent_prq_remote.parquet-stats:
> >> > num_rows)
> >> > | mem-estimate=10.00MB mem-reservation=0B spill-buffer=2.00MB
> >> > | tuple-ids=1 row-size=8B cardinality=1
> >> > |
> >> > 00:SCAN HDFS [default.dv_entersiteevent_prq_remote, RANDOM]
> >> > partitions=31/441 files=2292 size=1.39GB
> >> > stats-rows=unavailable extrapolated-rows=disabled
> >> > table stats: rows=unavailable size=unavailable
> >> > column stats: all
> >> > mem-estimate=1.00MB mem-reservation=0B
> >> > tuple-ids=0 row-size=8B cardinality=unavailable
> >> > ----------------
> >> > Estimated Per-Host Mem: 22020096
> >> > Tables Missing Stats: default.dv_entersiteevent_prq_remote
> >> > Per Host Min Reservation: aopr-dhc001.lpdomain.com:22000(0)
> >> > aopr-dhc002.lpdomain.com:22000(0) aopr-dhc003.lpdomain.com:22000(0)
> >> > aopr-dhc004.lpdomain.com:22000(0) aopr-dhc005.lpdomain.com:22000(0)
> >> > Request Pool: root.impreports
> >> > Admission result: Admitted immediately
> >> > ExecSummary:
> >> > Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak
> Mem
> >> > Est. Peak Mem Detail
> >> >
> >> > ------------------------------------------------------------
> ------------------------------------------------------------------
> >> > 03:AGGREGATE 1 222.958us 222.958us 1 1 36.00
> KB
> >> > 10.00 MB FINALIZE
> >> > 02:EXCHANGE 1 357.508ms 357.508ms 5 1
> 0
> >> > 0 UNPARTITIONED
> >> > 01:AGGREGATE 5 1.027ms 1.541ms 5 1 20.00
> KB
> >> > 10.00 MB
> >> > 00:SCAN HDFS 5 282.797ms 333.715ms 2.29K -1 2.62
> MB
> >> > 1.00 MB default.dv_entersiteevent_prq_remote
> >> > Errors:
> >> > Planner Timeline: 13.508ms
> >> > - Analysis finished: 3.303ms (3.303ms)
> >> > - Value transfer graph computed: 3.465ms (162.667us)
> >> > - Single node plan created: 10.352ms (6.886ms)
> >> > - Runtime filters computed: 10.677ms (325.364us)
> >> > - Distributed plan created: 10.733ms (55.854us)
> >> > - Lineage info computed: 10.898ms (164.937us)
> >> > - Planning finished: 13.508ms (2.610ms)
> >> > Query Timeline: 481.950ms
> >> > - Query submitted: 79.971us (79.971us)
> >> > - Planning finished: 33.947ms (33.867ms)
> >> > - Submit for admission: 40.067ms (6.120ms)
> >> > - Completed admission: 40.197ms (130.146us)
> >> > - Ready to start on 5 backends: 40.905ms (707.852us)
> >> > - All 5 execution backends (6 fragment instances) started:
> >> > 46.417ms
> >> > (5.512ms)
> >> > - Rows available: 424.548ms (378.131ms)
> >> > - First row fetched: 473.488ms (48.939ms)
> >> > - Unregister query: 477.951ms (4.462ms)
> >> > - ComputeScanRangeAssignmentTimer: 3.980ms
> >> > ImpalaServer:
> >> > - ClientFetchWaitTimer: 51.659ms
> >> > - RowMaterializationTimer: 1.726ms
> >> > Execution Profile c843dae48ce9b289:6ad3ffc000000000:(Total:
> 385.029ms,
> >> > non-child: 0.000ns, % non-child: 0.00%)
> >> > Number of filters: 0
> >> > Filter routing table:
> >> > ID Src. Node Tgt. Node(s) Target type Partition filter Pending
> >> > (Expected) First arrived Completed Enabled
> >> >
> >> > ------------------------------------------------------------
> -------------------------------------------------------
> >> >
> >> > Backend startup latencies: Count: 5, min / max: 3ms / 4ms, 25th
> >> > %-ile:
> >> > 3ms, 50th %-ile: 4ms, 75th %-ile: 4ms, 90th %-ile: 4ms, 95th %-ile:
> 4ms,
> >> > 99.9th %-ile: 4ms
> >> > Per Node Peak Memory Usage: aopr-dhc001.lpdomain.com:22000(2.68
> MB)
> >> > aopr-dhc002.lpdomain.com:22000(2.54 MB)
> >> > aopr-dhc003.lpdomain.com:22000(2.54
> >> > MB) aopr-dhc004.lpdomain.com:22000(2.41 MB)
> >> > aopr-dhc005.lpdomain.com:22000(2.56 MB)
> >> > - FiltersReceived: 0 (0)
> >> > - FinalizationTimer: 0.000ns
> >> > - NumBackends: 5 (5)
> >> > - NumFragmentInstances: 6 (6)
> >> > - NumFragments: 2 (2)
> >> > Averaged Fragment F01:(Total: 427.530ms, non-child: 49.428ms, %
> >> > non-child: 11.56%)
> >> > split sizes: min: 0, max: 0, avg: 0, stddev: 0
> >> > completion times: min:428.693ms max:428.693ms mean: 428.693ms
> >> > stddev:0.000ns
> >> > execution rates: min:0.00 /sec max:0.00 /sec mean:0.00 /sec
> >> > stddev:0.00 /sec
> >> > num instances: 1
> >> > - AverageThreadTokens: 0.00
> >> > - BloomFilterBytes: 0
> >> > - PeakMemoryUsage: 44.02 KB (45072)
> >> > - PeakReservation: 0
> >> > - PeakUsedReservation: 0
> >> > - PerHostPeakMemUsage: 2.68 MB (2806113)
> >> > - RowsProduced: 1 (1)
> >> > - TotalNetworkReceiveTime: 357.481ms
> >> > - TotalNetworkSendTime: 0.000ns
> >> > - TotalStorageWaitTime: 0.000ns
> >> > - TotalThreadsInvoluntaryContextSwitches: 0 (0)
> >> > - TotalThreadsTotalWallClockTime: 406.708ms
> >> > - TotalThreadsSysTime: 0.000ns
> >> > - TotalThreadsUserTime: 0.000ns
> >> > - TotalThreadsVoluntaryContextSwitches: 7 (7)
> >> > Fragment Instance Lifecycle Timings:
> >> > - ExecTime: 49.055ms
> >> > - ExecTreeExecTime: 4.251us
> >> > - OpenTime: 357.663ms
> >> > - ExecTreeOpenTime: 357.649ms
> >> > - PrepareTime: 20.764ms
> >> > - ExecTreePrepareTime: 81.759us
> >> > PLAN_ROOT_SINK:
> >> > - PeakMemoryUsage: 0
> >> > CodeGen:(Total: 20.369ms, non-child: 20.369ms, % non-child:
> >> > 100.00%)
> >> > - CodegenTime: 0.000ns
> >> > - CompileTime: 0.000ns
> >> > - LoadTime: 0.000ns
> >> > - ModuleBitcodeSize: 1.86 MB (1950880)
> >> > - NumFunctions: 0 (0)
> >> > - NumInstructions: 0 (0)
> >> > - OptimizationTime: 0.000ns
> >> > - PeakMemoryUsage: 0
> >> > - PrepareTime: 19.685ms
> >> > AGGREGATION_NODE (id=3):(Total: 357.731ms, non-child:
> 222.958us, %
> >> > non-child: 0.06%)
> >> > - BuildTime: 7.368us
> >> > - GetResultsTime: 0.000ns
> >> > - HTResizeTime: 0.000ns
> >> > - HashBuckets: 0 (0)
> >> > - LargestPartitionPercent: 0 (0)
> >> > - MaxPartitionLevel: 0 (0)
> >> > - NumRepartitions: 0 (0)
> >> > - PartitionsCreated: 0 (0)
> >> > - PeakMemoryUsage: 36.00 KB (36864)
> >> > - RowsRepartitioned: 0 (0)
> >> > - RowsReturned: 1 (1)
> >> > - RowsReturnedRate: 2.00 /sec
> >> > - SpilledPartitions: 0 (0)
> >> > EXCHANGE_NODE (id=2):(Total: 357.508ms, non-child: 357.508ms, %
> >> > non-child: 100.00%)
> >> > - ConvertRowBatchTime: 8.881us
> >> > - PeakMemoryUsage: 0
> >> > - RowsReturned: 5 (5)
> >> > - RowsReturnedRate: 13.00 /sec
> >> > DataStreamReceiver:
> >> > - BytesReceived: 80.00 B (80)
> >> > - DeserializeRowBatchTimer: 24.165us
> >> > - FirstBatchArrivalWaitTime: 277.194ms
> >> > - PeakMemoryUsage: 4.02 KB (4112)
> >> > - SendersBlockedTimer: 0.000ns
> >> > - SendersBlockedTotalTimer(*): 0.000ns
> >> > Coordinator Fragment F01:
> >> > Instance c843dae48ce9b289:6ad3ffc000000000
> >> > (host=aopr-dhc001.lpdomain.com:22000):(Total: 427.530ms, non-child:
> >> > 49.428ms, % non-child: 11.56%)
> >> > - AverageThreadTokens: 0.00
> >> > - BloomFilterBytes: 0
> >> > - PeakMemoryUsage: 44.02 KB (45072)
> >> > - PeakReservation: 0
> >> > - PeakUsedReservation: 0
> >> > - PerHostPeakMemUsage: 2.68 MB (2806113)
> >> > - RowsProduced: 1 (1)
> >> > - TotalNetworkReceiveTime: 357.481ms
> >> > - TotalNetworkSendTime: 0.000ns
> >> > - TotalStorageWaitTime: 0.000ns
> >> > - TotalThreadsInvoluntaryContextSwitches: 0 (0)
> >> > - TotalThreadsTotalWallClockTime: 406.708ms
> >> > - TotalThreadsSysTime: 0.000ns
> >> > - TotalThreadsUserTime: 0.000ns
> >> > - TotalThreadsVoluntaryContextSwitches: 7 (7)
> >> > Fragment Instance Lifecycle Timings:
> >> > - ExecTime: 49.055ms
> >> > - ExecTreeExecTime: 4.251us
> >> > - OpenTime: 357.663ms
> >> > - ExecTreeOpenTime: 357.649ms
> >> > - PrepareTime: 20.764ms
> >> > - ExecTreePrepareTime: 81.759us
> >> > PLAN_ROOT_SINK:
> >> > - PeakMemoryUsage: 0
> >> > CodeGen:(Total: 20.369ms, non-child: 20.369ms, % non-child:
> >> > 100.00%)
> >> > - CodegenTime: 0.000ns
> >> > - CompileTime: 0.000ns
> >> > - LoadTime: 0.000ns
> >> > - ModuleBitcodeSize: 1.86 MB (1950880)
> >> > - NumFunctions: 0 (0)
> >> > - NumInstructions: 0 (0)
> >> > - OptimizationTime: 0.000ns
> >> > - PeakMemoryUsage: 0
> >> > - PrepareTime: 19.685ms
> >> > AGGREGATION_NODE (id=3):(Total: 357.731ms, non-child:
> 222.958us,
> >> > %
> >> > non-child: 0.06%)
> >> > - BuildTime: 7.368us
> >> > - GetResultsTime: 0.000ns
> >> > - HTResizeTime: 0.000ns
> >> > - HashBuckets: 0 (0)
> >> > - LargestPartitionPercent: 0 (0)
> >> > - MaxPartitionLevel: 0 (0)
> >> > - NumRepartitions: 0 (0)
> >> > - PartitionsCreated: 0 (0)
> >> > - PeakMemoryUsage: 36.00 KB (36864)
> >> > - RowsRepartitioned: 0 (0)
> >> > - RowsReturned: 1 (1)
> >> > - RowsReturnedRate: 2.00 /sec
> >> > - SpilledPartitions: 0 (0)
> >> > EXCHANGE_NODE (id=2):(Total: 357.508ms, non-child: 357.508ms,
> %
> >> > non-child: 100.00%)
> >> > - ConvertRowBatchTime: 8.881us
> >> > - PeakMemoryUsage: 0
> >> > - RowsReturned: 5 (5)
> >> > - RowsReturnedRate: 13.00 /sec
> >> > DataStreamReceiver:
> >> > - BytesReceived: 80.00 B (80)
> >> > - DeserializeRowBatchTimer: 24.165us
> >> > - FirstBatchArrivalWaitTime: 277.194ms
> >> > - PeakMemoryUsage: 4.02 KB (4112)
> >> > - SendersBlockedTimer: 0.000ns
> >> > - SendersBlockedTotalTimer(*): 0.000ns
> >> > Averaged Fragment F00:(Total: 327.360ms, non-child: 1.578ms, %
> >> > non-child: 0.48%)
> >> > split sizes: min: 283.31 MB, max: 284.33 MB, avg: 284.04 MB,
> >> > stddev:
> >> > 382.00 KB
> >> > completion times: min:299.457ms max:378.950ms mean: 328.881ms
> >> > stddev:31.507ms
> >> > execution rates: min:750.18 MB/sec max:949.01 MB/sec
> mean:871.27
> >> > MB/sec stddev:79.74 MB/sec
> >> > num instances: 5
> >> > - AverageThreadTokens: 15.00
> >> > - BloomFilterBytes: 0
> >> > - PeakMemoryUsage: 2.54 MB (2666852)
> >> > - PeakReservation: 0
> >> > - PeakUsedReservation: 0
> >> > - PerHostPeakMemUsage: 2.55 MB (2669309)
> >> > - RowsProduced: 1 (1)
> >> > - TotalNetworkReceiveTime: 0.000ns
> >> > - TotalNetworkSendTime: 303.541us
> >> > - TotalStorageWaitTime: 4s821ms
> >> > - TotalThreadsInvoluntaryContextSwitches: 104 (104)
> >> > - TotalThreadsTotalWallClockTime: 6s835ms
> >> > - TotalThreadsSysTime: 31.784ms
> >> > - TotalThreadsUserTime: 682.084ms
> >> > - TotalThreadsVoluntaryContextSwitches: 2.12K (2121)
> >> > Fragment Instance Lifecycle Timings:
> >> > - ExecTime: 500.816us
> >> > - ExecTreeExecTime: 8.200us
> >> > - OpenTime: 302.848ms
> >> > - ExecTreeOpenTime: 281.309ms
> >> > - PrepareTime: 23.946ms
> >> > - ExecTreePrepareTime: 2.513ms
> >> > DataStreamSender (dst_id=2):(Total: 269.902us, non-child:
> >> > 269.902us, %
> >> > non-child: 100.00%)
> >> > - BytesSent: 16.00 B (16)
> >> > - NetworkThroughput(*): 373.13 KB/sec
> >> > - OverallThroughput: 63.53 KB/sec
> >> > - PeakMemoryUsage: 16.00 KB (16384)
> >> > - RowsReturned: 1 (1)
> >> > - SerializeBatchTime: 14.525us
> >> > - TransmitDataRPCTime: 43.122us
> >> > - UncompressedRowBatchSize: 16.00 B (16)
> >> > CodeGen:(Total: 41.687ms, non-child: 41.687ms, % non-child:
> >> > 100.00%)
> >> > - CodegenTime: 1.194ms
> >> > - CompileTime: 7.360ms
> >> > - LoadTime: 0.000ns
> >> > - ModuleBitcodeSize: 1.86 MB (1950880)
> >> > - NumFunctions: 15 (15)
> >> > - NumInstructions: 214 (214)
> >> > - OptimizationTime: 13.529ms
> >> > - PeakMemoryUsage: 107.00 KB (109568)
> >> > - PrepareTime: 20.084ms
> >> > AGGREGATION_NODE (id=1):(Total: 283.825ms, non-child: 1.027ms, %
> >> > non-child: 0.36%)
> >> > - BuildTime: 1.259ms
> >> > - GetResultsTime: 0.000ns
> >> > - HTResizeTime: 0.000ns
> >> > - HashBuckets: 0 (0)
> >> > - LargestPartitionPercent: 0 (0)
> >> > - MaxPartitionLevel: 0 (0)
> >> > - NumRepartitions: 0 (0)
> >> > - PartitionsCreated: 0 (0)
> >> > - PeakMemoryUsage: 20.00 KB (20480)
> >> > - RowsRepartitioned: 0 (0)
> >> > - RowsReturned: 1 (1)
> >> > - RowsReturnedRate: 2.00 /sec
> >> > - SpilledPartitions: 0 (0)
> >> > HDFS_SCAN_NODE (id=0):(Total: 282.797ms, non-child: 282.797ms, %
> >> > non-child: 100.00%)
> >> > - AverageHdfsReadThreadConcurrency: 0.00
> >> > - AverageScannerThreadConcurrency: 14.40
> >> > - BytesRead: 44.52 MB (46686028)
> >> > - BytesReadDataNodeCache: 0
> >> > - BytesReadLocal: 44.52 MB (46686028)
> >> > - BytesReadRemoteUnexpected: 0
> >> > - BytesReadShortCircuit: 44.52 MB (46686028)
> >> > - CachedFileHandlesHitCount: 0 (0)
> >> > - CachedFileHandlesMissCount: 458 (458)
> >> > - CollectionItemsRead: 0 (0)
> >> > - DecompressionTime: 0.000ns
> >> > - MaxCompressedTextFileLength: 0
> >> > - NumColumns: 0 (0)
> >> > - NumDictFilteredRowGroups: 0 (0)
> >> > - NumDisksAccessed: 12 (12)
> >> > - NumRowGroups: 458 (458)
> >> > - NumScannerThreadsStarted: 24 (24)
> >> > - NumScannersWithNoReads: 0 (0)
> >> > - NumStatsFilteredRowGroups: 0 (0)
> >> > - PeakMemoryUsage: 2.50 MB (2625539)
> >> > - PerReadThreadRawHdfsThroughput: 1.29 GB/sec
> >> > - RemoteScanRanges: 0 (0)
> >> > - RowBatchQueueGetWaitTime: 269.787ms
> >> > - RowBatchQueuePutWaitTime: 0.000ns
> >> > - RowsRead: 0 (0)
> >> > - RowsReturned: 458 (458)
> >> > - RowsReturnedRate: 1.64 K/sec
> >> > - ScanRangesComplete: 458 (458)
> >> > - ScannerThreadsInvoluntaryContextSwitches: 99 (99)
> >> > - ScannerThreadsTotalWallClockTime: 6s532ms
> >> > - MaterializeTupleTime(*): 70.664us
> >> > - ScannerThreadsSysTime: 26.184ms
> >> > - ScannerThreadsUserTime: 650.089ms
> >> > - ScannerThreadsVoluntaryContextSwitches: 1.67K (1674)
> >> > - TotalRawHdfsReadTime(*): 34.416ms
> >> > - TotalReadThroughput: 36.86 MB/sec
> >> > Fragment F00:
> >> > Instance c843dae48ce9b289:6ad3ffc000000004
> >> > (host=aopr-dhc004.lpdomain.com:22000):(Total: 377.384ms, non-child:
> >> > 1.619ms,
> >> > % non-child: 0.43%)
> >> > Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> >> > 0:36/21.95 MB 11:33/19.87 MB 9:27/15.16 MB 5:44/28.07 MB 6:56/33.32 MB
> >> > 1:38/22.24 MB 10:44/27.90 MB 3:34/20.26 MB 2:45/28.21 MB 7:31/19.33 MB
> >> > 4:42/26.73 MB 8:34/21.25 MB
> >> > MemoryUsage(500.000ms): 1.91 MB
> >> > ThreadUsage(500.000ms): 25
> >> > - AverageThreadTokens: 25.00
> >> > - BloomFilterBytes: 0
> >> > - PeakMemoryUsage: 2.41 MB (2531689)
> >> > - PeakReservation: 0
> >> > - PeakUsedReservation: 0
> >> > - PerHostPeakMemUsage: 2.41 MB (2531689)
> >> > - RowsProduced: 1 (1)
> >> > - TotalNetworkReceiveTime: 0.000ns
> >> > - TotalNetworkSendTime: 382.629us
> >> > - TotalStorageWaitTime: 5s790ms
> >> > - TotalThreadsInvoluntaryContextSwitches: 115 (115)
> >> > - TotalThreadsTotalWallClockTime: 8s122ms
> >> > - TotalThreadsSysTime: 33.984ms
> >> > - TotalThreadsUserTime: 702.879ms
> >> > - TotalThreadsVoluntaryContextSwitches: 2.04K (2037)
> >> > Fragment Instance Lifecycle Timings:
> >> > - ExecTime: 604.005us
> >> > - ExecTreeExecTime: 8.338us
> >> > - OpenTime: 352.927ms
> >> > - ExecTreeOpenTime: 331.921ms
> >> > - PrepareTime: 23.783ms
> >> > - ExecTreePrepareTime: 2.559ms
> >> > DataStreamSender (dst_id=2):(Total: 331.818us, non-child:
> >> > 331.818us,
> >> > % non-child: 100.00%)
> >> > - BytesSent: 16.00 B (16)
> >> > - NetworkThroughput(*): 289.00 KB/sec
> >> > - OverallThroughput: 47.09 KB/sec
> >> > - PeakMemoryUsage: 16.00 KB (16384)
> >> > - RowsReturned: 1 (1)
> >> > - SerializeBatchTime: 14.466us
> >> > - TransmitDataRPCTime: 54.066us
> >> > - UncompressedRowBatchSize: 16.00 B (16)
> >> > CodeGen:(Total: 40.950ms, non-child: 40.950ms, % non-child:
> >> > 100.00%)
> >> > - CodegenTime: 1.122ms
> >> > - CompileTime: 7.318ms
> >> > - LoadTime: 0.000ns
> >> > - ModuleBitcodeSize: 1.86 MB (1950880)
> >> > - NumFunctions: 15 (15)
> >> > - NumInstructions: 214 (214)
> >> > - OptimizationTime: 13.106ms
> >> > - PeakMemoryUsage: 107.00 KB (109568)
> >> > - PrepareTime: 19.773ms
> >> > AGGREGATION_NODE (id=1):(Total: 334.482ms, non-child:
> 766.527us,
> >> > %
> >> > non-child: 0.23%)
> >> > ExecOption: Codegen Enabled
> >> > - BuildTime: 1.192ms
> >> > - GetResultsTime: 0.000ns
> >> > - HTResizeTime: 0.000ns
> >> > - HashBuckets: 0 (0)
> >> > - LargestPartitionPercent: 0 (0)
> >> > - MaxPartitionLevel: 0 (0)
> >> > - NumRepartitions: 0 (0)
> >> > - PartitionsCreated: 0 (0)
> >> > - PeakMemoryUsage: 20.00 KB (20480)
> >> > - RowsRepartitioned: 0 (0)
> >> > - RowsReturned: 1 (1)
> >> > - RowsReturnedRate: 2.00 /sec
> >> > - SpilledPartitions: 0 (0)
> >> > HDFS_SCAN_NODE (id=0):(Total: 333.715ms, non-child:
> 333.715ms, %
> >> > non-child: 100.00%)
> >> > Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> >> > 0:36/21.95 MB 11:33/19.87 MB 9:27/15.16 MB 5:44/28.07 MB 6:56/33.32 MB
> >> > 1:38/22.24 MB 10:44/27.90 MB 3:34/20.26 MB 2:45/28.21 MB 7:31/19.33 MB
> >> > 4:42/26.73 MB 8:34/21.25 MB
> >> > ExecOption: PARQUET Codegen Enabled, Codegen enabled: 464
> out
> >> > of
> >> > 464
> >> > Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0%
> >> > 4:0%
> >> > 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
> >> > File Formats: PARQUET/Unknown(Skipped):464
> >> > BytesRead(500.000ms): 12.27 MB
> >> > - FooterProcessingTime: (Avg: 13.555ms ; Min: 5.433ms ;
> Max:
> >> > 62.957ms ; Number of samples: 464)
> >> > - AverageHdfsReadThreadConcurrency: 0.00
> >> > - AverageScannerThreadConcurrency: 24.00
> >> > - BytesRead: 45.01 MB (47193988)
> >> > - BytesReadDataNodeCache: 0
> >> > - BytesReadLocal: 45.01 MB (47193988)
> >> > - BytesReadRemoteUnexpected: 0
> >> > - BytesReadShortCircuit: 45.01 MB (47193988)
> >> > - CachedFileHandlesHitCount: 0 (0)
> >> > - CachedFileHandlesMissCount: 464 (464)
> >> > - CollectionItemsRead: 0 (0)
> >> > - DecompressionTime: 0.000ns
> >> > - MaxCompressedTextFileLength: 0
> >> > - NumColumns: 0 (0)
> >> > - NumDictFilteredRowGroups: 0 (0)
> >> > - NumDisksAccessed: 12 (12)
> >> > - NumRowGroups: 464 (464)
> >> > - NumScannerThreadsStarted: 24 (24)
> >> > - NumScannersWithNoReads: 0 (0)
> >> > - NumStatsFilteredRowGroups: 0 (0)
> >> > - PeakMemoryUsage: 2.38 MB (2490376)
> >> > - PerReadThreadRawHdfsThroughput: 1016.44 MB/sec
> >> > - RemoteScanRanges: 0 (0)
> >> > - RowBatchQueueGetWaitTime: 320.860ms
> >> > - RowBatchQueuePutWaitTime: 0.000ns
> >> > - RowsRead: 0 (0)
> >> > - RowsReturned: 464 (464)
> >> > - RowsReturnedRate: 1.39 K/sec
> >> > - ScanRangesComplete: 464 (464)
> >> > - ScannerThreadsInvoluntaryContextSwitches: 108 (108)
> >> > - ScannerThreadsTotalWallClockTime: 7s769ms
> >> > - MaterializeTupleTime(*): 75.514us
> >> > - ScannerThreadsSysTime: 29.984ms
> >> > - ScannerThreadsUserTime: 669.884ms
> >> > - ScannerThreadsVoluntaryContextSwitches: 1.60K (1597)
> >> > - TotalRawHdfsReadTime(*): 44.279ms
> >> > - TotalReadThroughput: 24.53 MB/sec
> >> > Instance c843dae48ce9b289:6ad3ffc000000002
> >> > (host=aopr-dhc003.lpdomain.com:22000):(Total: 351.014ms, non-child:
> >> > 1.641ms,
> >> > % non-child: 0.47%)
> >> > Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> >> > 0:35/20.89 MB 7:44/26.64 MB 11:27/17.21 MB 9:32/18.61 MB 5:25/14.56 MB
> >> > 2:38/24.32 MB 6:31/19.85 MB 8:50/30.76 MB 10:41/25.92 MB 3:31/19.78 MB
> >> > 4:60/37.59 MB 1:44/27.19 MB
> >> > MemoryUsage(500.000ms): 1.16 MB
> >> > ThreadUsage(500.000ms): 25
> >> > - AverageThreadTokens: 25.00
> >> > - BloomFilterBytes: 0
> >> > - PeakMemoryUsage: 2.54 MB (2662753)
> >> > - PeakReservation: 0
> >> > - PeakUsedReservation: 0
> >> > - PerHostPeakMemUsage: 2.54 MB (2662753)
> >> > - RowsProduced: 1 (1)
> >> > - TotalNetworkReceiveTime: 0.000ns
> >> > - TotalNetworkSendTime: 342.746us
> >> > - TotalStorageWaitTime: 4s796ms
> >> > - TotalThreadsInvoluntaryContextSwitches: 93 (93)
> >> > - TotalThreadsTotalWallClockTime: 7s286ms
> >> > - TotalThreadsSysTime: 27.988ms
> >> > - TotalThreadsUserTime: 671.887ms
> >> > - TotalThreadsVoluntaryContextSwitches: 2.08K (2079)
> >> > Fragment Instance Lifecycle Timings:
> >> > - ExecTime: 524.006us
> >> > - ExecTreeExecTime: 7.920us
> >> > - OpenTime: 327.802ms
> >> > - ExecTreeOpenTime: 307.154ms
> >> > - PrepareTime: 22.617ms
> >> > - ExecTreePrepareTime: 2.650ms
> >> > DataStreamSender (dst_id=2):(Total: 348.990us, non-child:
> >> > 348.990us,
> >> > % non-child: 100.00%)
> >> > - BytesSent: 16.00 B (16)
> >> > - NetworkThroughput(*): 354.47 KB/sec
> >> > - OverallThroughput: 44.77 KB/sec
> >> > - PeakMemoryUsage: 16.00 KB (16384)
> >> > - RowsReturned: 1 (1)
> >> > - SerializeBatchTime: 15.602us
> >> > - TransmitDataRPCTime: 44.080us
> >> > - UncompressedRowBatchSize: 16.00 B (16)
> >> > CodeGen:(Total: 39.215ms, non-child: 39.215ms, % non-child:
> >> > 100.00%)
> >> > - CodegenTime: 1.183ms
> >> > - CompileTime: 7.168ms
> >> > - LoadTime: 0.000ns
> >> > - ModuleBitcodeSize: 1.86 MB (1950880)
> >> > - NumFunctions: 15 (15)
> >> > - NumInstructions: 214 (214)
> >> > - OptimizationTime: 12.929ms
> >> > - PeakMemoryUsage: 107.00 KB (109568)
> >> > - PrepareTime: 18.453ms
> >> > AGGREGATION_NODE (id=1):(Total: 309.807ms, non-child:
> 948.677us,
> >> > %
> >> > non-child: 0.31%)
> >> > ExecOption: Codegen Enabled
> >> > - BuildTime: 1.552ms
> >> > - GetResultsTime: 0.000ns
> >> > - HTResizeTime: 0.000ns
> >> > - HashBuckets: 0 (0)
> >> > - LargestPartitionPercent: 0 (0)
> >> > - MaxPartitionLevel: 0 (0)
> >> > - NumRepartitions: 0 (0)
> >> > - PartitionsCreated: 0 (0)
> >> > - PeakMemoryUsage: 20.00 KB (20480)
> >> > - RowsRepartitioned: 0 (0)
> >> > - RowsReturned: 1 (1)
> >> > - RowsReturnedRate: 3.00 /sec
> >> > - SpilledPartitions: 0 (0)
> >> > HDFS_SCAN_NODE (id=0):(Total: 308.859ms, non-child:
> 308.859ms, %
> >> > non-child: 100.00%)
> >> > Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> >> > 0:35/20.89 MB 7:44/26.64 MB 11:27/17.21 MB 9:32/18.61 MB 5:25/14.56 MB
> >> > 2:38/24.32 MB 6:31/19.85 MB 8:50/30.76 MB 10:41/25.92 MB 3:31/19.78 MB
> >> > 4:60/37.59 MB 1:44/27.19 MB
> >> > ExecOption: PARQUET Codegen Enabled, Codegen enabled: 458
> out
> >> > of
> >> > 458
> >> > Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0%
> >> > 4:0%
> >> > 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
> >> > File Formats: PARQUET/Unknown(Skipped):458
> >> > BytesRead(500.000ms): 36.74 MB
> >> > - FooterProcessingTime: (Avg: 11.502ms ; Min: 3.668ms ;
> Max:
> >> > 23.564ms ; Number of samples: 458)
> >> > - AverageHdfsReadThreadConcurrency: 0.00
> >> > - AverageScannerThreadConcurrency: 24.00
> >> > - BytesRead: 44.51 MB (46668432)
> >> > - BytesReadDataNodeCache: 0
> >> > - BytesReadLocal: 44.51 MB (46668432)
> >> > - BytesReadRemoteUnexpected: 0
> >> > - BytesReadShortCircuit: 44.51 MB (46668432)
> >> > - CachedFileHandlesHitCount: 0 (0)
> >> > - CachedFileHandlesMissCount: 458 (458)
> >> > - CollectionItemsRead: 0 (0)
> >> > - DecompressionTime: 0.000ns
> >> > - MaxCompressedTextFileLength: 0
> >> > - NumColumns: 0 (0)
> >> > - NumDictFilteredRowGroups: 0 (0)
> >> > - NumDisksAccessed: 12 (12)
> >> > - NumRowGroups: 458 (458)
> >> > - NumScannerThreadsStarted: 24 (24)
> >> > - NumScannersWithNoReads: 0 (0)
> >> > - NumStatsFilteredRowGroups: 0 (0)
> >> > - PeakMemoryUsage: 2.50 MB (2621440)
> >> > - PerReadThreadRawHdfsThroughput: 1.26 GB/sec
> >> > - RemoteScanRanges: 0 (0)
> >> > - RowBatchQueueGetWaitTime: 295.595ms
> >> > - RowBatchQueuePutWaitTime: 0.000ns
> >> > - RowsRead: 0 (0)
> >> > - RowsReturned: 458 (458)
> >> > - RowsReturnedRate: 1.48 K/sec
> >> > - ScanRangesComplete: 458 (458)
> >> > - ScannerThreadsInvoluntaryContextSwitches: 91 (91)
> >> > - ScannerThreadsTotalWallClockTime: 6s958ms
> >> > - MaterializeTupleTime(*): 71.630us
> >> > - ScannerThreadsSysTime: 16.989ms
> >> > - ScannerThreadsUserTime: 645.891ms
> >> > - ScannerThreadsVoluntaryContextSwitches: 1.63K (1630)
> >> > - TotalRawHdfsReadTime(*): 34.377ms
> >> > - TotalReadThroughput: 73.49 MB/sec
> >> > Instance c843dae48ce9b289:6ad3ffc000000001
> >> > (host=aopr-dhc005.lpdomain.com:22000):(Total: 308.195ms, non-child:
> >> > 1.784ms,
> >> > % non-child: 0.58%)
> >> > Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> >> > 2:46/28.97 MB 8:39/24.34 MB 10:33/21.03 MB 3:43/26.50 MB 5:32/20.27 MB
> >> > 9:32/19.86 MB 6:25/16.01 MB 7:41/24.77 MB 1:42/25.01 MB 0:55/32.51 MB
> >> > 4:40/24.29 MB 11:33/20.77 MB
> >> > - AverageThreadTokens: 0.00
> >> > - BloomFilterBytes: 0
> >> > - PeakMemoryUsage: 2.56 MB (2683241)
> >> > - PeakReservation: 0
> >> > - PeakUsedReservation: 0
> >> > - PerHostPeakMemUsage: 2.56 MB (2683241)
> >> > - RowsProduced: 1 (1)
> >> > - TotalNetworkReceiveTime: 0.000ns
> >> > - TotalNetworkSendTime: 292.765us
> >> > - TotalStorageWaitTime: 4s482ms
> >> > - TotalThreadsInvoluntaryContextSwitches: 113 (113)
> >> > - TotalThreadsTotalWallClockTime: 6s310ms
> >> > - TotalThreadsSysTime: 35.983ms
> >> > - TotalThreadsUserTime: 682.884ms
> >> > - TotalThreadsVoluntaryContextSwitches: 2.14K (2143)
> >> > Fragment Instance Lifecycle Timings:
> >> > - ExecTime: 532.572us
> >> > - ExecTreeExecTime: 8.377us
> >> > - OpenTime: 280.241ms
> >> > - ExecTreeOpenTime: 258.652ms
> >> > - PrepareTime: 27.342ms
> >> > - ExecTreePrepareTime: 2.863ms
> >> > DataStreamSender (dst_id=2):(Total: 309.794us, non-child:
> >> > 309.794us,
> >> > % non-child: 100.00%)
> >> > - BytesSent: 16.00 B (16)
> >> > - NetworkThroughput(*): 356.91 KB/sec
> >> > - OverallThroughput: 50.44 KB/sec
> >> > - PeakMemoryUsage: 16.00 KB (16384)
> >> > - RowsReturned: 1 (1)
> >> > - SerializeBatchTime: 14.262us
> >> > - TransmitDataRPCTime: 43.778us
> >> > - UncompressedRowBatchSize: 16.00 B (16)
> >> > CodeGen:(Total: 44.584ms, non-child: 44.584ms, % non-child:
> >> > 100.00%)
> >> > - CodegenTime: 1.382ms
> >> > - CompileTime: 7.346ms
> >> > - LoadTime: 0.000ns
> >> > - ModuleBitcodeSize: 1.86 MB (1950880)
> >> > - NumFunctions: 15 (15)
> >> > - NumInstructions: 214 (214)
> >> > - OptimizationTime: 13.462ms
> >> > - PeakMemoryUsage: 107.00 KB (109568)
> >> > - PrepareTime: 23.006ms
> >> > AGGREGATION_NODE (id=1):(Total: 261.516ms, non-child:
> 836.656us,
> >> > %
> >> > non-child: 0.32%)
> >> > ExecOption: Codegen Enabled
> >> > - BuildTime: 1.280ms
> >> > - GetResultsTime: 0.000ns
> >> > - HTResizeTime: 0.000ns
> >> > - HashBuckets: 0 (0)
> >> > - LargestPartitionPercent: 0 (0)
> >> > - MaxPartitionLevel: 0 (0)
> >> > - NumRepartitions: 0 (0)
> >> > - PartitionsCreated: 0 (0)
> >> > - PeakMemoryUsage: 20.00 KB (20480)
> >> > - RowsRepartitioned: 0 (0)
> >> > - RowsReturned: 1 (1)
> >> > - RowsReturnedRate: 3.00 /sec
> >> > - SpilledPartitions: 0 (0)
> >> > HDFS_SCAN_NODE (id=0):(Total: 260.680ms, non-child:
> 260.680ms, %
> >> > non-child: 100.00%)
> >> > Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> >> > 2:46/28.97 MB 8:39/24.34 MB 10:33/21.03 MB 3:43/26.50 MB 5:32/20.27 MB
> >> > 9:32/19.86 MB 6:25/16.01 MB 7:41/24.77 MB 1:42/25.01 MB 0:55/32.51 MB
> >> > 4:40/24.29 MB 11:33/20.77 MB
> >> > ExecOption: PARQUET Codegen Enabled, Codegen enabled: 461
> out
> >> > of
> >> > 461
> >> > Hdfs Read Thread Concurrency Bucket: 0:0% 1:0% 2:0% 3:0%
> 4:0%
> >> > 5:0%
> >> > 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
> >> > File Formats: PARQUET/Unknown(Skipped):461
> >> > - FooterProcessingTime: (Avg: 10.793ms ; Min: 3.087ms ;
> Max:
> >> > 20.253ms ; Number of samples: 461)
> >> > - AverageHdfsReadThreadConcurrency: 0.00
> >> > - AverageScannerThreadConcurrency: 0.00
> >> > - BytesRead: 44.73 MB (46906407)
> >> > - BytesReadDataNodeCache: 0
> >> > - BytesReadLocal: 44.73 MB (46906407)
> >> > - BytesReadRemoteUnexpected: 0
> >> > - BytesReadShortCircuit: 44.73 MB (46906407)
> >> > - CachedFileHandlesHitCount: 0 (0)
> >> > - CachedFileHandlesMissCount: 461 (461)
> >> > - CollectionItemsRead: 0 (0)
> >> > - DecompressionTime: 0.000ns
> >> > - MaxCompressedTextFileLength: 0
> >> > - NumColumns: 0 (0)
> >> > - NumDictFilteredRowGroups: 0 (0)
> >> > - NumDisksAccessed: 12 (12)
> >> > - NumRowGroups: 461 (461)
> >> > - NumScannerThreadsStarted: 24 (24)
> >> > - NumScannersWithNoReads: 0 (0)
> >> > - NumStatsFilteredRowGroups: 0 (0)
> >> > - PeakMemoryUsage: 2.52 MB (2641928)
> >> > - PerReadThreadRawHdfsThroughput: 1.31 GB/sec
> >> > - RemoteScanRanges: 0 (0)
> >> > - RowBatchQueueGetWaitTime: 246.691ms
> >> > - RowBatchQueuePutWaitTime: 0.000ns
> >> > - RowsRead: 0 (0)
> >> > - RowsReturned: 461 (461)
> >> > - RowsReturnedRate: 1.77 K/sec
> >> > - ScanRangesComplete: 461 (461)
> >> > - ScannerThreadsInvoluntaryContextSwitches: 109 (109)
> >> > - ScannerThreadsTotalWallClockTime: 6s030ms
> >> > - MaterializeTupleTime(*): 68.735us
> >> > - ScannerThreadsSysTime: 33.983ms
> >> > - ScannerThreadsUserTime: 647.889ms
> >> > - ScannerThreadsVoluntaryContextSwitches: 1.70K (1696)
> >> > - TotalRawHdfsReadTime(*): 33.405ms
> >> > - TotalReadThroughput: 0.00 /sec
> >> > Instance c843dae48ce9b289:6ad3ffc000000003
> >> > (host=aopr-dhc002.lpdomain.com:22000):(Total: 302.116ms, non-child:
> >> > 1.604ms,
> >> > % non-child: 0.53%)
> >> > Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> >> > 0:27/16.21 MB 11:33/21.23 MB 6:36/23.15 MB 10:41/25.09 MB 1:41/24.13
> MB
> >> > 9:32/18.79 MB 5:41/25.31 MB 2:42/26.49 MB 7:36/22.87 MB 3:51/31.00 MB
> >> > 4:41/25.99 MB 8:38/23.79 MB
> >> > MemoryUsage(500.000ms): 700.35 KB
> >> > ThreadUsage(500.000ms): 25
> >> > - AverageThreadTokens: 25.00
> >> > - BloomFilterBytes: 0
> >> > - PeakMemoryUsage: 2.54 MB (2662753)
> >> > - PeakReservation: 0
> >> > - PeakUsedReservation: 0
> >> > - PerHostPeakMemUsage: 2.54 MB (2662753)
> >> > - RowsProduced: 1 (1)
> >> > - TotalNetworkReceiveTime: 0.000ns
> >> > - TotalNetworkSendTime: 300.675us
> >> > - TotalStorageWaitTime: 4s661ms
> >> > - TotalThreadsInvoluntaryContextSwitches: 90 (90)
> >> > - TotalThreadsTotalWallClockTime: 6s256ms
> >> > - TotalThreadsSysTime: 30.983ms
> >> > - TotalThreadsUserTime: 709.880ms
> >> > - TotalThreadsVoluntaryContextSwitches: 2.17K (2175)
> >> > Fragment Instance Lifecycle Timings:
> >> > - ExecTime: 492.627us
> >> > - ExecTreeExecTime: 8.191us
> >> > - OpenTime: 277.922ms
> >> > - ExecTreeOpenTime: 255.395ms
> >> > - PrepareTime: 23.647ms
> >> > - ExecTreePrepareTime: 2.243ms
> >> > DataStreamSender (dst_id=2):(Total: 194.517us, non-child:
> >> > 194.517us,
> >> > % non-child: 100.00%)
> >> > - BytesSent: 16.00 B (16)
> >> > - NetworkThroughput(*): 371.86 KB/sec
> >> > - OverallThroughput: 80.33 KB/sec
> >> > - PeakMemoryUsage: 16.00 KB (16384)
> >> > - RowsReturned: 1 (1)
> >> > - SerializeBatchTime: 14.035us
> >> > - TransmitDataRPCTime: 42.019us
> >> > - UncompressedRowBatchSize: 16.00 B (16)
> >> > CodeGen:(Total: 42.676ms, non-child: 42.676ms, % non-child:
> >> > 100.00%)
> >> > - CodegenTime: 1.236ms
> >> > - CompileTime: 7.717ms
> >> > - LoadTime: 0.000ns
> >> > - ModuleBitcodeSize: 1.86 MB (1950880)
> >> > - NumFunctions: 15 (15)
> >> > - NumInstructions: 214 (214)
> >> > - OptimizationTime: 14.130ms
> >> > - PeakMemoryUsage: 107.00 KB (109568)
> >> > - PrepareTime: 20.109ms
> >> > AGGREGATION_NODE (id=1):(Total: 257.640ms, non-child:
> 1.541ms, %
> >> > non-child: 0.60%)
> >> > ExecOption: Codegen Enabled
> >> > - BuildTime: 1.252ms
> >> > - GetResultsTime: 0.000ns
> >> > - HTResizeTime: 0.000ns
> >> > - HashBuckets: 0 (0)
> >> > - LargestPartitionPercent: 0 (0)
> >> > - MaxPartitionLevel: 0 (0)
> >> > - NumRepartitions: 0 (0)
> >> > - PartitionsCreated: 0 (0)
> >> > - PeakMemoryUsage: 20.00 KB (20480)
> >> > - RowsRepartitioned: 0 (0)
> >> > - RowsReturned: 1 (1)
> >> > - RowsReturnedRate: 3.00 /sec
> >> > - SpilledPartitions: 0 (0)
> >> > HDFS_SCAN_NODE (id=0):(Total: 256.099ms, non-child:
> 256.099ms, %
> >> > non-child: 100.00%)
> >> > Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> >> > 0:27/16.21 MB 11:33/21.23 MB 6:36/23.15 MB 10:41/25.09 MB 1:41/24.13
> MB
> >> > 9:32/18.79 MB 5:41/25.31 MB 2:42/26.49 MB 7:36/22.87 MB 3:51/31.00 MB
> >> > 4:41/25.99 MB 8:38/23.79 MB
> >> > ExecOption: PARQUET Codegen Enabled, Codegen enabled: 459
> out
> >> > of
> >> > 459
> >> > Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0%
> >> > 4:0%
> >> > 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
> >> > File Formats: PARQUET/Unknown(Skipped):459
> >> > BytesRead(500.000ms): 43.15 MB
> >> > - FooterProcessingTime: (Avg: 11.257ms ; Min: 3.989ms ;
> Max:
> >> > 23.004ms ; Number of samples: 459)
> >> > - AverageHdfsReadThreadConcurrency: 0.00
> >> > - AverageScannerThreadConcurrency: 24.00
> >> > - BytesRead: 44.52 MB (46678031)
> >> > - BytesReadDataNodeCache: 0
> >> > - BytesReadLocal: 44.52 MB (46678031)
> >> > - BytesReadRemoteUnexpected: 0
> >> > - BytesReadShortCircuit: 44.52 MB (46678031)
> >> > - CachedFileHandlesHitCount: 0 (0)
> >> > - CachedFileHandlesMissCount: 459 (459)
> >> > - CollectionItemsRead: 0 (0)
> >> > - DecompressionTime: 0.000ns
> >> > - MaxCompressedTextFileLength: 0
> >> > - NumColumns: 0 (0)
> >> > - NumDictFilteredRowGroups: 0 (0)
> >> > - NumDisksAccessed: 12 (12)
> >> > - NumRowGroups: 459 (459)
> >> > - NumScannerThreadsStarted: 24 (24)
> >> > - NumScannersWithNoReads: 0 (0)
> >> > - NumStatsFilteredRowGroups: 0 (0)
> >> > - PeakMemoryUsage: 2.50 MB (2621440)
> >> > - PerReadThreadRawHdfsThroughput: 1.33 GB/sec
> >> > - RemoteScanRanges: 0 (0)
> >> > - RowBatchQueueGetWaitTime: 243.426ms
> >> > - RowBatchQueuePutWaitTime: 0.000ns
> >> > - RowsRead: 0 (0)
> >> > - RowsReturned: 459 (459)
> >> > - RowsReturnedRate: 1.79 K/sec
> >> > - ScanRangesComplete: 459 (459)
> >> > - ScannerThreadsInvoluntaryContextSwitches: 85 (85)
> >> > - ScannerThreadsTotalWallClockTime: 5s978ms
> >> > - MaterializeTupleTime(*): 67.997us
> >> > - ScannerThreadsSysTime: 22.984ms
> >> > - ScannerThreadsUserTime: 678.884ms
> >> > - ScannerThreadsVoluntaryContextSwitches: 1.72K (1724)
> >> > - TotalRawHdfsReadTime(*): 32.626ms
> >> > - TotalReadThroughput: 86.30 MB/sec
> >> > Instance c843dae48ce9b289:6ad3ffc000000005
> >> > (host=aopr-dhc001.lpdomain.com:22000):(Total: 298.093ms, non-child:
> >> > 1.242ms,
> >> > % non-child: 0.42%)
> >> > Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> >> > 9:55/35.38 MB 0:37/24.23 MB 5:37/23.31 MB 11:22/13.79 MB 8:30/18.53 MB
> >> > 4:33/20.45 MB 1:38/24.06 MB 10:29/18.40 MB 3:39/24.35 MB 2:41/25.45 MB
> >> > 6:42/27.03 MB 7:47/29.22 MB
> >> > - AverageThreadTokens: 0.00
> >> > - BloomFilterBytes: 0
> >> > - PeakMemoryUsage: 2.66 MB (2793825)
> >> > - PeakReservation: 0
> >> > - PeakUsedReservation: 0
> >> > - PerHostPeakMemUsage: 2.68 MB (2806113)
> >> > - RowsProduced: 1 (1)
> >> > - TotalNetworkReceiveTime: 0.000ns
> >> > - TotalNetworkSendTime: 198.894us
> >> > - TotalStorageWaitTime: 4s374ms
> >> > - TotalThreadsInvoluntaryContextSwitches: 110 (110)
> >> > - TotalThreadsTotalWallClockTime: 6s201ms
> >> > - TotalThreadsSysTime: 29.983ms
> >> > - TotalThreadsUserTime: 642.892ms
> >> > - TotalThreadsVoluntaryContextSwitches: 2.17K (2174)
> >> > Fragment Instance Lifecycle Timings:
> >> > - ExecTime: 350.873us
> >> > - ExecTreeExecTime: 8.173us
> >> > - OpenTime: 275.347ms
> >> > - ExecTreeOpenTime: 253.425ms
> >> > - PrepareTime: 22.341ms
> >> > - ExecTreePrepareTime: 2.251ms
> >> > DataStreamSender (dst_id=2):(Total: 164.394us, non-child:
> >> > 164.394us,
> >> > % non-child: 100.00%)
> >> > - BytesSent: 16.00 B (16)
> >> > - NetworkThroughput(*): 493.42 KB/sec
> >> > - OverallThroughput: 95.05 KB/sec
> >> > - PeakMemoryUsage: 16.00 KB (16384)
> >> > - RowsReturned: 1 (1)
> >> > - SerializeBatchTime: 14.262us
> >> > - TransmitDataRPCTime: 31.667us
> >> > - UncompressedRowBatchSize: 16.00 B (16)
> >> > CodeGen:(Total: 41.008ms, non-child: 41.008ms, % non-child:
> >> > 100.00%)
> >> > - CodegenTime: 1.049ms
> >> > - CompileTime: 7.248ms
> >> > - LoadTime: 0.000ns
> >> > - ModuleBitcodeSize: 1.86 MB (1950880)
> >> > - NumFunctions: 15 (15)
> >> > - NumInstructions: 214 (214)
> >> > - OptimizationTime: 14.018ms
> >> > - PeakMemoryUsage: 107.00 KB (109568)
> >> > - PrepareTime: 19.080ms
> >> > AGGREGATION_NODE (id=1):(Total: 255.678ms, non-child:
> 1.046ms, %
> >> > non-child: 0.41%)
> >> > ExecOption: Codegen Enabled
> >> > - BuildTime: 1.015ms
> >> > - GetResultsTime: 0.000ns
> >> > - HTResizeTime: 0.000ns
> >> > - HashBuckets: 0 (0)
> >> > - LargestPartitionPercent: 0 (0)
> >> > - MaxPartitionLevel: 0 (0)
> >> > - NumRepartitions: 0 (0)
> >> > - PartitionsCreated: 0 (0)
> >> > - PeakMemoryUsage: 20.00 KB (20480)
> >> > - RowsRepartitioned: 0 (0)
> >> > - RowsReturned: 1 (1)
> >> > - RowsReturnedRate: 3.00 /sec
> >> > - SpilledPartitions: 0 (0)
> >> > HDFS_SCAN_NODE (id=0):(Total: 254.631ms, non-child:
> 254.631ms, %
> >> > non-child: 100.00%)
> >> > Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> >> > 9:55/35.38 MB 0:37/24.23 MB 5:37/23.31 MB 11:22/13.79 MB 8:30/18.53 MB
> >> > 4:33/20.45 MB 1:38/24.06 MB 10:29/18.40 MB 3:39/24.35 MB 2:41/25.45 MB
> >> > 6:42/27.03 MB 7:47/29.22 MB
> >> > ExecOption: PARQUET Codegen Enabled, Codegen enabled: 450
> out
> >> > of
> >> > 450
> >> > Hdfs Read Thread Concurrency Bucket: 0:0% 1:0% 2:0% 3:0%
> 4:0%
> >> > 5:0%
> >> > 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
> >> > File Formats: PARQUET/Unknown(Skipped):450
> >> > - FooterProcessingTime: (Avg: 10.727ms ; Min: 3.813ms ;
> Max:
> >> > 23.755ms ; Number of samples: 450)
> >> > - AverageHdfsReadThreadConcurrency: 0.00
> >> > - AverageScannerThreadConcurrency: 0.00
> >> > - BytesRead: 43.85 MB (45983284)
> >> > - BytesReadDataNodeCache: 0
> >> > - BytesReadLocal: 43.85 MB (45983284)
> >> > - BytesReadRemoteUnexpected: 0
> >> > - BytesReadShortCircuit: 43.85 MB (45983284)
> >> > - CachedFileHandlesHitCount: 0 (0)
> >> > - CachedFileHandlesMissCount: 450 (450)
> >> > - CollectionItemsRead: 0 (0)
> >> > - DecompressionTime: 0.000ns
> >> > - MaxCompressedTextFileLength: 0
> >> > - NumColumns: 0 (0)
> >> > - NumDictFilteredRowGroups: 0 (0)
> >> > - NumDisksAccessed: 12 (12)
> >> > - NumRowGroups: 450 (450)
> >> > - NumScannerThreadsStarted: 24 (24)
> >> > - NumScannersWithNoReads: 0 (0)
> >> > - NumStatsFilteredRowGroups: 0 (0)
> >> > - PeakMemoryUsage: 2.62 MB (2752512)
> >> > - PerReadThreadRawHdfsThroughput: 1.56 GB/sec
> >> > - RemoteScanRanges: 0 (0)
> >> > - RowBatchQueueGetWaitTime: 242.362ms
> >> > - RowBatchQueuePutWaitTime: 0.000ns
> >> > - RowsRead: 0 (0)
> >> > - RowsReturned: 450 (450)
> >> > - RowsReturnedRate: 1.77 K/sec
> >> > - ScanRangesComplete: 450 (450)
> >> > - ScannerThreadsInvoluntaryContextSwitches: 106 (106)
> >> > - ScannerThreadsTotalWallClockTime: 5s926ms
> >> > - MaterializeTupleTime(*): 69.447us
> >> > - ScannerThreadsSysTime: 26.984ms
> >> > - ScannerThreadsUserTime: 607.897ms
> >> > - ScannerThreadsVoluntaryContextSwitches: 1.72K (1723)
> >> > - TotalRawHdfsReadTime(*): 27.393ms
> >> > - TotalReadThroughput: 0.00 /sec
> >> > ===================================================
> >> >
> >> > Coordinator log:
> >> >
> >> > 01: query (string) = "select count(*) from
> dv_entersiteevent_prq_remote
> >> > where year=2017 and month=10",
> >> > 03: configuration (list) = list<string>[1] {
> >> > [0] = "REQUEST_POOL=impreports",
> >> > },
> >> > 04: hadoop_user (string) = "fawzea",
> >> > }
> >> > I0208 01:07:06.321281 7154 impala-beeswax-server.cc:459]
> >> > TClientRequest.queryOptions: TQueryOptions {
> >> > 01: abort_on_error (bool) = false,
> >> > 02: max_errors (i32) = 100,
> >> > 03: disable_codegen (bool) = false,
> >> > 04: batch_size (i32) = 0,
> >> > 05: num_nodes (i32) = 0,
> >> > 06: max_scan_range_length (i64) = 0,
> >> > 07: num_scanner_threads (i32) = 0,
> >> > 08: max_io_buffers (i32) = 0,
> >> > 09: allow_unsupported_formats (bool) = false,
> >> > 10: default_order_by_limit (i64) = -1,
> >> > 11: debug_action (string) = "",
> >> > 12: mem_limit (i64) = 0,
> >> > 13: abort_on_default_limit_exceeded (bool) = false,
> >> > 15: hbase_caching (i32) = 0,
> >> > 16: hbase_cache_blocks (bool) = false,
> >> > 17: parquet_file_size (i64) = 0,
> >> > 18: explain_level (i32) = 1,
> >> > 19: sync_ddl (bool) = false,
> >> > 20: request_pool (string) = "impreports",
> >> > 23: disable_cached_reads (bool) = false,
> >> > 24: disable_outermost_topn (bool) = false,
> >> > 25: rm_initial_mem (i64) = 0,
> >> > 26: query_timeout_s (i32) = 0,
> >> > 28: appx_count_distinct (bool) = false,
> >> > 29: disable_unsafe_spills (bool) = false,
> >> > 31: exec_single_node_rows_threshold (i32) = 100,
> >> > 32: optimize_partition_key_scans (bool) = false,
> >> > 33: replica_preference (i32) = 0,
> >> > 34: schedule_random_replica (bool) = false,
> >> > 35: scan_node_codegen_threshold (i64) = 1800000,
> >> > 36: disable_streaming_preaggregations (bool) = false,
> >> > 37: runtime_filter_mode (i32) = 2,
> >> > 38: runtime_bloom_filter_size (i32) = 1048576,
> >> > 39: runtime_filter_wait_time_ms (i32) = 0,
> >> > 40: disable_row_runtime_filtering (bool) = false,
> >> > 41: max_num_runtime_filters (i32) = 10,
> >> > 42: parquet_annotate_strings_utf8 (bool) = false,
> >> > 43: parquet_fallback_schema_resolution (i32) = 0,
> >> > 45: s3_skip_insert_staging (bool) = true,
> >> > 46: runtime_filter_min_size (i32) = 1048576,
> >> > 47: runtime_filter_max_size (i32) = 16777216,
> >> > 48: prefetch_mode (i32) = 1,
> >> > 49: strict_mode (bool) = false,
> >> > 50: scratch_limit (i64) = -1,
> >> > 51: enable_expr_rewrites (bool) = true,
> >> > 52: decimal_v2 (bool) = false,
> >> > 53: parquet_dictionary_filtering (bool) = true,
> >> > 54: parquet_array_resolution (i32) = 2,
> >> > 55: parquet_read_statistics (bool) = true,
> >> > 56: default_join_distribution_mode (i32) = 0,
> >> > 57: disable_codegen_rows_threshold (i32) = 50000,
> >> > 58: default_spillable_buffer_size (i64) = 2097152,
> >> > 59: min_spillable_buffer_size (i64) = 65536,
> >> > 60: max_row_size (i64) = 524288,
> >> > }
> >> > I0208 01:07:06.324410 7154 Frontend.java:909] Compiling query: select
> >> > count(*) from dv_entersiteevent_prq_remote where year=2017 and
> month=10
> >> > I0208 01:07:06.327203 7154 Frontend.java:948] Compiled query.
> >> > I0208 01:07:06.359748 7154 admission-controller.cc:510] Schedule for
> >> > id=5f49e927591a9042:faf8f28600000000 in pool_name=root.impreports
> >> > cluster_mem_needed=105.00 MB PoolConfig: max_requests=-1
> max_queued=200
> >> > max_mem=-1.00 B
> >> > I0208 01:07:06.359776 7154 admission-controller.cc:515] Stats:
> >> > agg_num_running=0, agg_num_queued=0, agg_mem_reserved=0,
> >> > local_host(local_mem_admitted=0, num_admitted_running=0,
> num_queued=0,
> >> > backend_mem_reserved=0)
> >> > I0208 01:07:06.359784 7154 admission-controller.cc:531] Admitted
> query
> >> > id=5f49e927591a9042:faf8f28600000000
> >> > I0208 01:07:06.359807 7154 coordinator.cc:99] Exec()
> >> > query_id=5f49e927591a9042:faf8f28600000000 stmt=select count(*) from
> >> > dv_entersiteevent_prq_remote where year=2017 and month=10
> >> > I0208 01:07:06.360694 7154 coordinator.cc:357] starting execution on
> 5
> >> > backends for query 5f49e927591a9042:faf8f28600000000
> >> > I0208 01:07:06.363255 63583 impala-internal-service.cc:44]
> >> > ExecQueryFInstances(): query_id=5f49e927591a9042:faf8f28600000000
> >> > I0208 01:07:06.363282 63583 query-exec-mgr.cc:46]
> StartQueryFInstances()
> >> > query_id=5f49e927591a9042:faf8f28600000000
> >> > coord=aopr-dhc001.lpdomain.com:22000
> >> > I0208 01:07:06.363296 63583 query-state.cc:173] Buffer pool limit for
> >> > 5f49e927591a9042:faf8f28600000000: 10802010521
> >> > I0208 01:07:06.363364 63583 initial-reservations.cc:60] Successfully
> >> > claimed
> >> > initial reservations (0) for query 5f49e927591a9042:faf8f28600000000
> >> > I0208 01:07:06.363544 7155 query-state.cc:286] StartFInstances():
> >> > query_id=5f49e927591a9042:faf8f28600000000 #instances=2
> >> > I0208 01:07:06.363844 7155 query-state.cc:299] descriptor table for
> >> > query=5f49e927591a9042:faf8f28600000000
> >> > tuples:
> >> > Tuple(id=2 size=0 slots=[] tuple_path=[])
> >> > Tuple(id=1 size=8 slots=[Slot(id=2 type=BIGINT col_path=[] offset=0
> >> > null=(offset=0 mask=0) slot_idx=0 field_idx=-1)] tuple_path=[])
> >> > Tuple(id=0 size=8 slots=[Slot(id=3 type=BIGINT col_path=[] offset=0
> >> > null=(offset=0 mask=0) slot_idx=0 field_idx=-1)] tuple_path=[])
> >> > I0208 01:07:06.364013 7156 query-state.cc:377] Executing instance.
> >> > instance_id=5f49e927591a9042:faf8f28600000000 fragment_idx=0
> >> > per_fragment_instance_idx=0 coord_state_idx=0 #in-flight=1
> >> > I0208 01:07:06.364135 7157 query-state.cc:377] Executing instance.
> >> > instance_id=5f49e927591a9042:faf8f28600000005 fragment_idx=1
> >> > per_fragment_instance_idx=4 coord_state_idx=0 #in-flight=2
> >> > I0208 01:07:06.364241 7157 hdfs-scan-node.cc:160] Max row batch queue
> >> > size
> >> > for scan node '0' in fragment instance
> >> > '5f49e927591a9042:faf8f28600000005':
> >> > 160
> >> > I0208 01:07:06.366775 7154 coordinator.cc:370] started execution on 5
> >> > backends for query 5f49e927591a9042:faf8f28600000000
> >> > I0208 01:07:06.385625 7155 query-exec-mgr.cc:149]
> ReleaseQueryState():
> >> > query_id=5f49e927591a9042:faf8f28600000000 refcnt=4
> >> > I0208 01:07:06.671193 7559 impala-server.cc:1796] Connection from
> >> > client
> >> > ::ffff:10.16.136.3:55598 closed, closing 1 associated session(s)
> >> > I0208 01:07:06.676138 544 data-stream-mgr.cc:228] Reduced stream ID
> >> > cache
> >> > from 2 items, to 0, eviction took: 0
> >> > I0208 01:07:06.677202 544 coordinator.cc:955] Backend completed:
> >> > host=aopr-dhc002.lpdomain.com:22000 remaining=4
> >> > I0208 01:07:06.677214 544 coordinator-backend-state.cc:229]
> >> > query_id=5f49e927591a9042:faf8f28600000000: first in-progress
> backend:
> >> > aopr-dhc001.lpdomain.com:22000
> >> > I0208 01:07:06.684221 7157 query-state.cc:384] Instance completed.
> >> > instance_id=5f49e927591a9042:faf8f28600000005 #in-flight=1 status=OK
> >> > I0208 01:07:06.684237 7157 query-exec-mgr.cc:149]
> ReleaseQueryState():
> >> > query_id=5f49e927591a9042:faf8f28600000000 refcnt=3
> >> > I0208 01:07:06.700275 520 coordinator.cc:955] Backend completed:
> >> > host=aopr-dhc005.lpdomain.com:22000 remaining=3
> >> > I0208 01:07:06.700289 520 coordinator-backend-state.cc:229]
> >> > query_id=5f49e927591a9042:faf8f28600000000: first in-progress
> backend:
> >> > aopr-dhc001.lpdomain.com:22000
> >> > I0208 01:07:06.704620 65258 coordinator.cc:955] Backend completed:
> >> > host=aopr-dhc003.lpdomain.com:22000 remaining=2
> >> > I0208 01:07:06.704666 65258 coordinator-backend-state.cc:229]
> >> > query_id=5f49e927591a9042:faf8f28600000000: first in-progress
> backend:
> >> > aopr-dhc001.lpdomain.com:22000
> >> > I0208 01:07:06.706041 7156 data-stream-mgr.cc:238] DeregisterRecvr():
> >> > fragment_instance_id=5f49e927591a9042:faf8f28600000000, node=2
> >> > I0208 01:07:06.706070 7156 data-stream-recvr.cc:235] cancelled
> stream:
> >> > fragment_instance_id_=5f49e927591a9042:faf8f28600000000 node_id=2
> >> > I0208 01:07:06.706809 64916 coordinator.cc:955] Backend completed:
> >> > host=aopr-dhc004.lpdomain.com:22000 remaining=1
> >> > I0208 01:07:06.706821 64916 coordinator-backend-state.cc:229]
> >> > query_id=5f49e927591a9042:faf8f28600000000: first in-progress
> backend:
> >> > aopr-dhc001.lpdomain.com:22000
> >> > I0208 01:07:06.789568 7154 impala-beeswax-server.cc:193]
> >> > get_results_metadata(): query_id=5f49e927591a9042:faf8f28600000000
> >> > I0208 01:07:06.790439 7154 coordinator.cc:789] Coordinator waiting
> for
> >> > backends to finish, 1 remaining
> >> > I0208 01:07:06.791052 7154 coordinator.cc:794] All backends finished
> >> > successfully.
> >> > I0208 01:07:06.791086 7154 coordinator.cc:1090] Release admssion
> >> > control
> >> > resources for query 5f49e927591a9042:faf8f28600000000
> >> > I0208 01:07:06.791363 7156 query-state.cc:384] Instance completed.
> >> > instance_id=5f49e927591a9042:faf8f28600000000 #in-flight=0 status=OK
> >> > I0208 01:07:06.791414 7156 query-exec-mgr.cc:149]
> ReleaseQueryState():
> >> > query_id=5f49e927591a9042:faf8f28600000000 refcnt=2
> >> > I0208 01:07:06.793586 7154 impala-beeswax-server.cc:235] close():
> >> > query_id=5f49e927591a9042:faf8f28600000000
> >> > I0208 01:07:06.793602 7154 impala-server.cc:992] UnregisterQuery():
> >> > query_id=5f49e927591a9042:faf8f28600000000
> >> > I0208 01:07:06.793614 7154 impala-server.cc:1075] Cancel():
> >> > query_id=5f49e927591a9042:faf8f28600000000
> >> > I0208 01:07:06.793624 7154 coordinator.cc:895] Cancel()
> >> > query_id=5f49e927591a9042:faf8f28600000000
> >> > I0208 01:07:06.793637 7154 coordinator.cc:905] CancelBackends()
> >> > query_id=5f49e927591a9042:faf8f28600000000, tried to cancel 0
> backends
> >> > I0208 01:07:06.802776 7154 query-exec-mgr.cc:149]
> ReleaseQueryState():
> >> > query_id=5f49e927591a9042:faf8f28600000000 refcnt=1
> >> > I0208 01:07:06.814214 7154 impala-server.cc:1796] Connection from
> >> > client
> >> > ::ffff:10.16.144.131:51012 closed, closing 1 associated session(s)
> >> > I0208 01:07:07.258891 7567 impala-server.cc:1796] Connection from
> >> > client
> >> > ::ffff:10.16.136.2:56375 closed, closing 1 associated session(s)
> >> > I0208 01:07:11.600131 7819 impala-server.cc:1796] Connection from
> >> > client
> >> > ::ffff:10.16.136.3:27460 closed, closing 1 associated session(s)
> >> > I0208 01:07:12.262739 7821 impala-server.cc:1796] Connection from
> >> > client
> >> > ::ffff:10.16.136.2:52672 closed, closing 1 associated session(s)
> >> > I0208 01:07:16.665451 7822 impala-server.cc:1796] Connection from
> >> > client
> >> > ::ffff:10.16.136.3:10693 closed, closing 1 associated session(s)
> >> > I0208 01:07:17.235321 7825 impala-server.cc:1796] Connection from
> >> > client
> >> > ::ffff:10.16.136.2:34752 closed, closing 1 associated session(s)
> >> > I0208 01:07:21.410611 5858 impala-beeswax-server.cc:193]
> >> > get_results_metadata(): query_id=1d4fbf0f2571f1e9:ea346f7900000000
> >> > I0208 01:07:21.414175 5858 impala-beeswax-server.cc:235] close():
> >> > query_id=1d4fbf0f2571f1e9:ea346f7900000000
> >> > I0208 01:07:21.414185 5858 impala-server.cc:992] UnregisterQuery():
> >> > query_id=1d4fbf0f2571f1e9:ea346f7900000000
> >> > I0208 01:07:21.414191 5858 impala-server.cc:1075] Cancel():
> >> > query_id=1d4fbf0f2571f1e9:ea346f7900000000
> >> > I0208 01:07:21.421505 5858 impala-server.cc:1796] Connection from
> >> > client
> >> > ::ffff:10.26.180.75:60072 closed, closing 1 associated session(s)
> >> > I0208 01:07:21.605690 7830 impala-server.cc:1796] Connection from
> >> > client
> >> > ::ffff:10.16.136.3:46660 closed, closing 1 associated session(s)
> >> >
> >> >
> >> > On Thu, Feb 8, 2018 at 6:20 AM, Matthew Jacobs <mj...@apache.org>
> >> > wrote:
> >> >>
> >> >> Thanks! I think that configuration looks OK.
> >> >>
> >> >> 1) What version of Impala is this?
> >> >>
> >> >> 2) Can you send the query profile and the Impala coordinator log for
> >> >> the query where it's not behaving as expected?
> >> >>
> >> >> > When i'm running the following with fawzea user:
> >> >> > set request_pool=impreports;
> >> >> > select count(*) from dv_entersiteevent_prq_remote where year=2017
> and
> >> >> > month=10;
> >> >> > I can see that the query is running with impreports pool.
> >> >>
> >> >> On Wed, Feb 7, 2018 at 7:59 PM, Fawze Abujaber <fa...@gmail.com>
> >> >> wrote:
> >> >> > Hi Matt,
> >> >> >
> >> >> > [root@aopr-dhc001 ~]# cat
> >> >> >
> >> >> >
> >> >> > /var/run/cloudera-scm-agent/process/4558-impala-IMPALAD/
> impala-conf/fair-scheduler.xml
> >> >> > <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
> >> >> > <allocations>
> >> >> > <queue name="root">
> >> >> > <weight>1.0</weight>
> >> >> > <schedulingPolicy>fair</schedulingPolicy>
> >> >> > <queue name="impiaas">
> >> >> > <weight>1.0</weight>
> >> >> > <schedulingPolicy>fair</schedulingPolicy>
> >> >> > <aclSubmitApps>fawzea </aclSubmitApps>
> >> >> > </queue>
> >> >> > <queue name="impreports">
> >> >> > <weight>1.0</weight>
> >> >> > <schedulingPolicy>fair</schedulingPolicy>
> >> >> > <aclSubmitApps>nira </aclSubmitApps>
> >> >> > </queue>
> >> >> > </queue>
> >> >> > <defaultQueueSchedulingPolicy>fair</
> defaultQueueSchedulingPolicy>
> >> >> > <queuePlacementPolicy>
> >> >> > <rule name="specified" create="false"/>
> >> >> > <rule name="secondaryGroupExistingQueue"/>
> >> >> > <rule name="default"/>
> >> >> > </queuePlacementPolicy>
> >> >> > </allocations>
> >> >> >
> >> >> >
> >> >> > On Thu, Feb 8, 2018 at 4:47 AM, Matthew Jacobs <
> jacobs.mh@gmail.com>
> >> >> > wrote:
> >> >> >>
> >> >> >> Yes, I saw that, but I'd like to see the actual configuration that
> >> >> >> is
> >> >> >> passed to Impala because it may not be specified in the right way
> to
> >> >> >> accomplish what you're trying to do in "b".
> >> >> >>
> >> >> >> On Wed, Feb 7, 2018 at 4:32 PM, Fawze Abujaber <fawzeaj@gmail.com
> >
> >> >> >> wrote:
> >> >> >> > Hi Matt,
> >> >> >> >
> >> >> >> > I created the pools manually and it’s completely nfigurations as
> >> >> >> > shown
> >> >> >> > in b.
> >> >> >> >
> >> >> >> > On Wed, 7 Feb 2018 at 23:17 Matthew Jacobs <jacobs.mh@gmail.com
> >
> >> >> >> > wrote:
> >> >> >> >>
> >> >> >> >> How, more specifically, are you configuring these pools? Can
> you
> >> >> >> >> share
> >> >> >> >> the configuration files?
> >> >> >> >>
> >> >> >> >> On Wed, Feb 7, 2018 at 11:42 AM, Fawze Abujaber
> >> >> >> >> <fa...@gmail.com>
> >> >> >> >> wrote:
> >> >> >> >> > Hello All,
> >> >> >> >> >
> >> >> >> >> > Will appreciate any insight of what i'm missing in my
> >> >> >> >> > configuration
> >> >> >> >> > to
> >> >> >> >> > get
> >> >> >> >> > the impala submission control working as expected ....
> >> >> >> >> >
> >> >> >> >> > a- Here is my placement Placement Rule:
> >> >> >> >> > 1- Use the pool Specified at run time., only if the pool
> >> >> >> >> > exists.
> >> >> >> >> > 2- Use the pool root.[secondary group].
> >> >> >> >> > 3- Use the pool root.default.
> >> >> >> >> >
> >> >> >> >> > b- Here is the impala resource pools
> >> >> >> >> >
> >> >> >> >> > Name Max Memory Max Running Queries Max
> >> >> >> >> > Queued
> >> >> >> >> > Queries
> >> >> >> >> > Queue Timeout Default Query Memory Limit Access Control
> >> >> >> >> > root.impiaas Unlimited Unlimited
> >> >> >> >> > 200 1 minute(s) No
> Default
> >> >> >> >> > Custom
> >> >> >> >> > root.impreports Unlimited Unlimited
> >> >> >> >> > 200 1 minute(s) No
> Default
> >> >> >> >> > Custom
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > C- Here is my Access control for root.impiaas pool:
> >> >> >> >> >
> >> >> >> >> > Allow these users and groups to submit to this pool
> >> >> >> >> >
> >> >> >> >> > Users: fawzea
> >> >> >> >> >
> >> >> >> >> > The same for impreports the allowed user and groups to submit
> >> >> >> >> > to
> >> >> >> >> > pool
> >> >> >> >> > is
> >> >> >> >> > only nira.
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > LDAP groups for fawzea user:
> >> >> >> >> >
> >> >> >> >> > Member in groups:
> >> >> >> >> >
> >> >> >> >> > - dba
> >> >> >> >> > - impiaas
> >> >> >> >> > - mysql
> >> >> >> >> >
> >> >> >> >> > LDAP group for nira user
> >> >> >> >> > Member in groups:
> >> >> >> >> > - dba
> >> >> >> >> > - impreports
> >> >> >> >> > - mysql
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > Impala Queries UI shows:
> >> >> >> >> >
> >> >> >> >> > 1- select count(*) from dv_entersiteevent_prq_remote where
> >> >> >> >> > year=2017
> >> >> >> >> >
> >> >> >> >> > User: fawzea
> >> >> >> >> > Pool: root.impiaas
> >> >> >> >> >
> >> >> >> >> > 2- select count(*) from dv_entersiteevent_prq_remote where
> >> >> >> >> > year=2017
> >> >> >> >> > and
> >> >> >> >> > month=10
> >> >> >> >> > User: nira
> >> >> >> >> > Pool: root.impreports
> >> >> >> >> >
> >> >> >> >> > As shown above that the queries submitted based on the
> >> >> >> >> > secondary
> >> >> >> >> > group
> >> >> >> >> > rule.
> >> >> >> >> >
> >> >> >> >> > When i'm running the following with fawzea user:
> >> >> >> >> >
> >> >> >> >> > set request_pool=impreports;
> >> >> >> >> > select count(*) from dv_entersiteevent_prq_remote where
> >> >> >> >> > year=2017
> >> >> >> >> > and
> >> >> >> >> > month=10;
> >> >> >> >> >
> >> >> >> >> > I can see that the query is running with impreports pool.
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > What i'm missing here in my configuration to get this works
> as
> >> >> >> >> > expected.
> >> >> >> >> >
> >> >> >
> >> >> >
> >> >
> >> >
> >
> >
>
Re: impala submission control
Posted by Matthew Jacobs <ja...@gmail.com>.
The aclSubmitApps can specify users or groups, so if the user name is
also a group name, that can lead to confusion, e.g.
https://issues.apache.org/jira/browse/IMPALA-2578
I'm not saying that is necessarily the problem, but something to
check. If we still can't find anything configured incorrectly I'd
recommend that you file a JIRA.
On Thu, Feb 8, 2018 at 9:05 AM, Fawze Abujaber <fa...@gmail.com> wrote:
> Hi Matt,
>
> If this is the situation, how we can explain that when i submit the query it
> runs on my secondary group pool and when nira do the same it runs under her
> secondary group pool.
>
> More than this, the issue not related to LDAP group as i'm using the
> submission restriction to a user and not a group.
>
> On Thu, Feb 8, 2018 at 7:00 PM, Matthew Jacobs <ja...@gmail.com> wrote:
>>
>> As far as I can tell from this information, your configuration looks
>> fine. The only other potential issue I can think of is that the
>> secondary groups placement rule isn't picking up your LDAP groups
>> properly. Does the ACL behavior work if you remove the secondary
>> groups rule?
>>
>> On Wed, Feb 7, 2018 at 10:09 PM, Fawze Abujaber <fa...@gmail.com> wrote:
>> > Hi Matt,
>> >
>> > v2.11.0-cdh5.14.0
>> >
>> > Query Runtime Profile:
>> > Query (id=c843dae48ce9b289:6ad3ffc000000000):
>> > Summary:
>> > Session ID: ca420ac477772b56:51ae460ce9e44999
>> > Session Type: BEESWAX
>> > Start Time: 2018-02-08 01:02:01.023558000
>> > End Time: 2018-02-08 01:02:01.501506000
>> > Query Type: QUERY
>> > Query State: FINISHED
>> > Query Status: OK
>> > Impala Version: impalad version 2.11.0-cdh5.14.0 RELEASE (build
>> > d68206561bce6b26762d62c01a78e6cd27aa7690)
>> > User: fawzea
>> > Connected User: fawzea
>> > Delegated User:
>> > Network Address: ::ffff:10.16.144.131:50078
>> > Default Db: default
>> > Sql Statement: select count(*) from dv_entersiteevent_prq_remote
>> > where
>> > year=2017 and month=10
>> > Coordinator: aopr-dhc001.lpdomain.com:22000
>> > Query Options (set by configuration): REQUEST_POOL=impreports
>> > Query Options (set by configuration and planner):
>> > REQUEST_POOL=impreports,MT_DOP=0
>> > Plan:
>> > ----------------
>> > Max Per-Host Resource Reservation: Memory=0B
>> > Per-Host Resource Estimates: Memory=21.00MB
>> > WARNING: The following tables are missing relevant table and/or column
>> > statistics.
>> > default.dv_entersiteevent_prq_remote
>> >
>> > F01:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1
>> > | Per-Host Resources: mem-estimate=10.00MB mem-reservation=0B
>> > PLAN-ROOT SINK
>> > | mem-estimate=0B mem-reservation=0B
>> > |
>> > 03:AGGREGATE [FINALIZE]
>> > | output: count:merge(*)
>> > | mem-estimate=10.00MB mem-reservation=0B spill-buffer=2.00MB
>> > | tuple-ids=1 row-size=8B cardinality=1
>> > |
>> > 02:EXCHANGE [UNPARTITIONED]
>> > | mem-estimate=0B mem-reservation=0B
>> > | tuple-ids=1 row-size=8B cardinality=1
>> > |
>> > F00:PLAN FRAGMENT [RANDOM] hosts=5 instances=5
>> > Per-Host Resources: mem-estimate=11.00MB mem-reservation=0B
>> > 01:AGGREGATE
>> > | output:
>> > sum_init_zero(default.dv_entersiteevent_prq_remote.parquet-stats:
>> > num_rows)
>> > | mem-estimate=10.00MB mem-reservation=0B spill-buffer=2.00MB
>> > | tuple-ids=1 row-size=8B cardinality=1
>> > |
>> > 00:SCAN HDFS [default.dv_entersiteevent_prq_remote, RANDOM]
>> > partitions=31/441 files=2292 size=1.39GB
>> > stats-rows=unavailable extrapolated-rows=disabled
>> > table stats: rows=unavailable size=unavailable
>> > column stats: all
>> > mem-estimate=1.00MB mem-reservation=0B
>> > tuple-ids=0 row-size=8B cardinality=unavailable
>> > ----------------
>> > Estimated Per-Host Mem: 22020096
>> > Tables Missing Stats: default.dv_entersiteevent_prq_remote
>> > Per Host Min Reservation: aopr-dhc001.lpdomain.com:22000(0)
>> > aopr-dhc002.lpdomain.com:22000(0) aopr-dhc003.lpdomain.com:22000(0)
>> > aopr-dhc004.lpdomain.com:22000(0) aopr-dhc005.lpdomain.com:22000(0)
>> > Request Pool: root.impreports
>> > Admission result: Admitted immediately
>> > ExecSummary:
>> > Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem
>> > Est. Peak Mem Detail
>> >
>> > ------------------------------------------------------------------------------------------------------------------------------
>> > 03:AGGREGATE 1 222.958us 222.958us 1 1 36.00 KB
>> > 10.00 MB FINALIZE
>> > 02:EXCHANGE 1 357.508ms 357.508ms 5 1 0
>> > 0 UNPARTITIONED
>> > 01:AGGREGATE 5 1.027ms 1.541ms 5 1 20.00 KB
>> > 10.00 MB
>> > 00:SCAN HDFS 5 282.797ms 333.715ms 2.29K -1 2.62 MB
>> > 1.00 MB default.dv_entersiteevent_prq_remote
>> > Errors:
>> > Planner Timeline: 13.508ms
>> > - Analysis finished: 3.303ms (3.303ms)
>> > - Value transfer graph computed: 3.465ms (162.667us)
>> > - Single node plan created: 10.352ms (6.886ms)
>> > - Runtime filters computed: 10.677ms (325.364us)
>> > - Distributed plan created: 10.733ms (55.854us)
>> > - Lineage info computed: 10.898ms (164.937us)
>> > - Planning finished: 13.508ms (2.610ms)
>> > Query Timeline: 481.950ms
>> > - Query submitted: 79.971us (79.971us)
>> > - Planning finished: 33.947ms (33.867ms)
>> > - Submit for admission: 40.067ms (6.120ms)
>> > - Completed admission: 40.197ms (130.146us)
>> > - Ready to start on 5 backends: 40.905ms (707.852us)
>> > - All 5 execution backends (6 fragment instances) started:
>> > 46.417ms
>> > (5.512ms)
>> > - Rows available: 424.548ms (378.131ms)
>> > - First row fetched: 473.488ms (48.939ms)
>> > - Unregister query: 477.951ms (4.462ms)
>> > - ComputeScanRangeAssignmentTimer: 3.980ms
>> > ImpalaServer:
>> > - ClientFetchWaitTimer: 51.659ms
>> > - RowMaterializationTimer: 1.726ms
>> > Execution Profile c843dae48ce9b289:6ad3ffc000000000:(Total: 385.029ms,
>> > non-child: 0.000ns, % non-child: 0.00%)
>> > Number of filters: 0
>> > Filter routing table:
>> > ID Src. Node Tgt. Node(s) Target type Partition filter Pending
>> > (Expected) First arrived Completed Enabled
>> >
>> > -------------------------------------------------------------------------------------------------------------------
>> >
>> > Backend startup latencies: Count: 5, min / max: 3ms / 4ms, 25th
>> > %-ile:
>> > 3ms, 50th %-ile: 4ms, 75th %-ile: 4ms, 90th %-ile: 4ms, 95th %-ile: 4ms,
>> > 99.9th %-ile: 4ms
>> > Per Node Peak Memory Usage: aopr-dhc001.lpdomain.com:22000(2.68 MB)
>> > aopr-dhc002.lpdomain.com:22000(2.54 MB)
>> > aopr-dhc003.lpdomain.com:22000(2.54
>> > MB) aopr-dhc004.lpdomain.com:22000(2.41 MB)
>> > aopr-dhc005.lpdomain.com:22000(2.56 MB)
>> > - FiltersReceived: 0 (0)
>> > - FinalizationTimer: 0.000ns
>> > - NumBackends: 5 (5)
>> > - NumFragmentInstances: 6 (6)
>> > - NumFragments: 2 (2)
>> > Averaged Fragment F01:(Total: 427.530ms, non-child: 49.428ms, %
>> > non-child: 11.56%)
>> > split sizes: min: 0, max: 0, avg: 0, stddev: 0
>> > completion times: min:428.693ms max:428.693ms mean: 428.693ms
>> > stddev:0.000ns
>> > execution rates: min:0.00 /sec max:0.00 /sec mean:0.00 /sec
>> > stddev:0.00 /sec
>> > num instances: 1
>> > - AverageThreadTokens: 0.00
>> > - BloomFilterBytes: 0
>> > - PeakMemoryUsage: 44.02 KB (45072)
>> > - PeakReservation: 0
>> > - PeakUsedReservation: 0
>> > - PerHostPeakMemUsage: 2.68 MB (2806113)
>> > - RowsProduced: 1 (1)
>> > - TotalNetworkReceiveTime: 357.481ms
>> > - TotalNetworkSendTime: 0.000ns
>> > - TotalStorageWaitTime: 0.000ns
>> > - TotalThreadsInvoluntaryContextSwitches: 0 (0)
>> > - TotalThreadsTotalWallClockTime: 406.708ms
>> > - TotalThreadsSysTime: 0.000ns
>> > - TotalThreadsUserTime: 0.000ns
>> > - TotalThreadsVoluntaryContextSwitches: 7 (7)
>> > Fragment Instance Lifecycle Timings:
>> > - ExecTime: 49.055ms
>> > - ExecTreeExecTime: 4.251us
>> > - OpenTime: 357.663ms
>> > - ExecTreeOpenTime: 357.649ms
>> > - PrepareTime: 20.764ms
>> > - ExecTreePrepareTime: 81.759us
>> > PLAN_ROOT_SINK:
>> > - PeakMemoryUsage: 0
>> > CodeGen:(Total: 20.369ms, non-child: 20.369ms, % non-child:
>> > 100.00%)
>> > - CodegenTime: 0.000ns
>> > - CompileTime: 0.000ns
>> > - LoadTime: 0.000ns
>> > - ModuleBitcodeSize: 1.86 MB (1950880)
>> > - NumFunctions: 0 (0)
>> > - NumInstructions: 0 (0)
>> > - OptimizationTime: 0.000ns
>> > - PeakMemoryUsage: 0
>> > - PrepareTime: 19.685ms
>> > AGGREGATION_NODE (id=3):(Total: 357.731ms, non-child: 222.958us, %
>> > non-child: 0.06%)
>> > - BuildTime: 7.368us
>> > - GetResultsTime: 0.000ns
>> > - HTResizeTime: 0.000ns
>> > - HashBuckets: 0 (0)
>> > - LargestPartitionPercent: 0 (0)
>> > - MaxPartitionLevel: 0 (0)
>> > - NumRepartitions: 0 (0)
>> > - PartitionsCreated: 0 (0)
>> > - PeakMemoryUsage: 36.00 KB (36864)
>> > - RowsRepartitioned: 0 (0)
>> > - RowsReturned: 1 (1)
>> > - RowsReturnedRate: 2.00 /sec
>> > - SpilledPartitions: 0 (0)
>> > EXCHANGE_NODE (id=2):(Total: 357.508ms, non-child: 357.508ms, %
>> > non-child: 100.00%)
>> > - ConvertRowBatchTime: 8.881us
>> > - PeakMemoryUsage: 0
>> > - RowsReturned: 5 (5)
>> > - RowsReturnedRate: 13.00 /sec
>> > DataStreamReceiver:
>> > - BytesReceived: 80.00 B (80)
>> > - DeserializeRowBatchTimer: 24.165us
>> > - FirstBatchArrivalWaitTime: 277.194ms
>> > - PeakMemoryUsage: 4.02 KB (4112)
>> > - SendersBlockedTimer: 0.000ns
>> > - SendersBlockedTotalTimer(*): 0.000ns
>> > Coordinator Fragment F01:
>> > Instance c843dae48ce9b289:6ad3ffc000000000
>> > (host=aopr-dhc001.lpdomain.com:22000):(Total: 427.530ms, non-child:
>> > 49.428ms, % non-child: 11.56%)
>> > - AverageThreadTokens: 0.00
>> > - BloomFilterBytes: 0
>> > - PeakMemoryUsage: 44.02 KB (45072)
>> > - PeakReservation: 0
>> > - PeakUsedReservation: 0
>> > - PerHostPeakMemUsage: 2.68 MB (2806113)
>> > - RowsProduced: 1 (1)
>> > - TotalNetworkReceiveTime: 357.481ms
>> > - TotalNetworkSendTime: 0.000ns
>> > - TotalStorageWaitTime: 0.000ns
>> > - TotalThreadsInvoluntaryContextSwitches: 0 (0)
>> > - TotalThreadsTotalWallClockTime: 406.708ms
>> > - TotalThreadsSysTime: 0.000ns
>> > - TotalThreadsUserTime: 0.000ns
>> > - TotalThreadsVoluntaryContextSwitches: 7 (7)
>> > Fragment Instance Lifecycle Timings:
>> > - ExecTime: 49.055ms
>> > - ExecTreeExecTime: 4.251us
>> > - OpenTime: 357.663ms
>> > - ExecTreeOpenTime: 357.649ms
>> > - PrepareTime: 20.764ms
>> > - ExecTreePrepareTime: 81.759us
>> > PLAN_ROOT_SINK:
>> > - PeakMemoryUsage: 0
>> > CodeGen:(Total: 20.369ms, non-child: 20.369ms, % non-child:
>> > 100.00%)
>> > - CodegenTime: 0.000ns
>> > - CompileTime: 0.000ns
>> > - LoadTime: 0.000ns
>> > - ModuleBitcodeSize: 1.86 MB (1950880)
>> > - NumFunctions: 0 (0)
>> > - NumInstructions: 0 (0)
>> > - OptimizationTime: 0.000ns
>> > - PeakMemoryUsage: 0
>> > - PrepareTime: 19.685ms
>> > AGGREGATION_NODE (id=3):(Total: 357.731ms, non-child: 222.958us,
>> > %
>> > non-child: 0.06%)
>> > - BuildTime: 7.368us
>> > - GetResultsTime: 0.000ns
>> > - HTResizeTime: 0.000ns
>> > - HashBuckets: 0 (0)
>> > - LargestPartitionPercent: 0 (0)
>> > - MaxPartitionLevel: 0 (0)
>> > - NumRepartitions: 0 (0)
>> > - PartitionsCreated: 0 (0)
>> > - PeakMemoryUsage: 36.00 KB (36864)
>> > - RowsRepartitioned: 0 (0)
>> > - RowsReturned: 1 (1)
>> > - RowsReturnedRate: 2.00 /sec
>> > - SpilledPartitions: 0 (0)
>> > EXCHANGE_NODE (id=2):(Total: 357.508ms, non-child: 357.508ms, %
>> > non-child: 100.00%)
>> > - ConvertRowBatchTime: 8.881us
>> > - PeakMemoryUsage: 0
>> > - RowsReturned: 5 (5)
>> > - RowsReturnedRate: 13.00 /sec
>> > DataStreamReceiver:
>> > - BytesReceived: 80.00 B (80)
>> > - DeserializeRowBatchTimer: 24.165us
>> > - FirstBatchArrivalWaitTime: 277.194ms
>> > - PeakMemoryUsage: 4.02 KB (4112)
>> > - SendersBlockedTimer: 0.000ns
>> > - SendersBlockedTotalTimer(*): 0.000ns
>> > Averaged Fragment F00:(Total: 327.360ms, non-child: 1.578ms, %
>> > non-child: 0.48%)
>> > split sizes: min: 283.31 MB, max: 284.33 MB, avg: 284.04 MB,
>> > stddev:
>> > 382.00 KB
>> > completion times: min:299.457ms max:378.950ms mean: 328.881ms
>> > stddev:31.507ms
>> > execution rates: min:750.18 MB/sec max:949.01 MB/sec mean:871.27
>> > MB/sec stddev:79.74 MB/sec
>> > num instances: 5
>> > - AverageThreadTokens: 15.00
>> > - BloomFilterBytes: 0
>> > - PeakMemoryUsage: 2.54 MB (2666852)
>> > - PeakReservation: 0
>> > - PeakUsedReservation: 0
>> > - PerHostPeakMemUsage: 2.55 MB (2669309)
>> > - RowsProduced: 1 (1)
>> > - TotalNetworkReceiveTime: 0.000ns
>> > - TotalNetworkSendTime: 303.541us
>> > - TotalStorageWaitTime: 4s821ms
>> > - TotalThreadsInvoluntaryContextSwitches: 104 (104)
>> > - TotalThreadsTotalWallClockTime: 6s835ms
>> > - TotalThreadsSysTime: 31.784ms
>> > - TotalThreadsUserTime: 682.084ms
>> > - TotalThreadsVoluntaryContextSwitches: 2.12K (2121)
>> > Fragment Instance Lifecycle Timings:
>> > - ExecTime: 500.816us
>> > - ExecTreeExecTime: 8.200us
>> > - OpenTime: 302.848ms
>> > - ExecTreeOpenTime: 281.309ms
>> > - PrepareTime: 23.946ms
>> > - ExecTreePrepareTime: 2.513ms
>> > DataStreamSender (dst_id=2):(Total: 269.902us, non-child:
>> > 269.902us, %
>> > non-child: 100.00%)
>> > - BytesSent: 16.00 B (16)
>> > - NetworkThroughput(*): 373.13 KB/sec
>> > - OverallThroughput: 63.53 KB/sec
>> > - PeakMemoryUsage: 16.00 KB (16384)
>> > - RowsReturned: 1 (1)
>> > - SerializeBatchTime: 14.525us
>> > - TransmitDataRPCTime: 43.122us
>> > - UncompressedRowBatchSize: 16.00 B (16)
>> > CodeGen:(Total: 41.687ms, non-child: 41.687ms, % non-child:
>> > 100.00%)
>> > - CodegenTime: 1.194ms
>> > - CompileTime: 7.360ms
>> > - LoadTime: 0.000ns
>> > - ModuleBitcodeSize: 1.86 MB (1950880)
>> > - NumFunctions: 15 (15)
>> > - NumInstructions: 214 (214)
>> > - OptimizationTime: 13.529ms
>> > - PeakMemoryUsage: 107.00 KB (109568)
>> > - PrepareTime: 20.084ms
>> > AGGREGATION_NODE (id=1):(Total: 283.825ms, non-child: 1.027ms, %
>> > non-child: 0.36%)
>> > - BuildTime: 1.259ms
>> > - GetResultsTime: 0.000ns
>> > - HTResizeTime: 0.000ns
>> > - HashBuckets: 0 (0)
>> > - LargestPartitionPercent: 0 (0)
>> > - MaxPartitionLevel: 0 (0)
>> > - NumRepartitions: 0 (0)
>> > - PartitionsCreated: 0 (0)
>> > - PeakMemoryUsage: 20.00 KB (20480)
>> > - RowsRepartitioned: 0 (0)
>> > - RowsReturned: 1 (1)
>> > - RowsReturnedRate: 2.00 /sec
>> > - SpilledPartitions: 0 (0)
>> > HDFS_SCAN_NODE (id=0):(Total: 282.797ms, non-child: 282.797ms, %
>> > non-child: 100.00%)
>> > - AverageHdfsReadThreadConcurrency: 0.00
>> > - AverageScannerThreadConcurrency: 14.40
>> > - BytesRead: 44.52 MB (46686028)
>> > - BytesReadDataNodeCache: 0
>> > - BytesReadLocal: 44.52 MB (46686028)
>> > - BytesReadRemoteUnexpected: 0
>> > - BytesReadShortCircuit: 44.52 MB (46686028)
>> > - CachedFileHandlesHitCount: 0 (0)
>> > - CachedFileHandlesMissCount: 458 (458)
>> > - CollectionItemsRead: 0 (0)
>> > - DecompressionTime: 0.000ns
>> > - MaxCompressedTextFileLength: 0
>> > - NumColumns: 0 (0)
>> > - NumDictFilteredRowGroups: 0 (0)
>> > - NumDisksAccessed: 12 (12)
>> > - NumRowGroups: 458 (458)
>> > - NumScannerThreadsStarted: 24 (24)
>> > - NumScannersWithNoReads: 0 (0)
>> > - NumStatsFilteredRowGroups: 0 (0)
>> > - PeakMemoryUsage: 2.50 MB (2625539)
>> > - PerReadThreadRawHdfsThroughput: 1.29 GB/sec
>> > - RemoteScanRanges: 0 (0)
>> > - RowBatchQueueGetWaitTime: 269.787ms
>> > - RowBatchQueuePutWaitTime: 0.000ns
>> > - RowsRead: 0 (0)
>> > - RowsReturned: 458 (458)
>> > - RowsReturnedRate: 1.64 K/sec
>> > - ScanRangesComplete: 458 (458)
>> > - ScannerThreadsInvoluntaryContextSwitches: 99 (99)
>> > - ScannerThreadsTotalWallClockTime: 6s532ms
>> > - MaterializeTupleTime(*): 70.664us
>> > - ScannerThreadsSysTime: 26.184ms
>> > - ScannerThreadsUserTime: 650.089ms
>> > - ScannerThreadsVoluntaryContextSwitches: 1.67K (1674)
>> > - TotalRawHdfsReadTime(*): 34.416ms
>> > - TotalReadThroughput: 36.86 MB/sec
>> > Fragment F00:
>> > Instance c843dae48ce9b289:6ad3ffc000000004
>> > (host=aopr-dhc004.lpdomain.com:22000):(Total: 377.384ms, non-child:
>> > 1.619ms,
>> > % non-child: 0.43%)
>> > Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>> > 0:36/21.95 MB 11:33/19.87 MB 9:27/15.16 MB 5:44/28.07 MB 6:56/33.32 MB
>> > 1:38/22.24 MB 10:44/27.90 MB 3:34/20.26 MB 2:45/28.21 MB 7:31/19.33 MB
>> > 4:42/26.73 MB 8:34/21.25 MB
>> > MemoryUsage(500.000ms): 1.91 MB
>> > ThreadUsage(500.000ms): 25
>> > - AverageThreadTokens: 25.00
>> > - BloomFilterBytes: 0
>> > - PeakMemoryUsage: 2.41 MB (2531689)
>> > - PeakReservation: 0
>> > - PeakUsedReservation: 0
>> > - PerHostPeakMemUsage: 2.41 MB (2531689)
>> > - RowsProduced: 1 (1)
>> > - TotalNetworkReceiveTime: 0.000ns
>> > - TotalNetworkSendTime: 382.629us
>> > - TotalStorageWaitTime: 5s790ms
>> > - TotalThreadsInvoluntaryContextSwitches: 115 (115)
>> > - TotalThreadsTotalWallClockTime: 8s122ms
>> > - TotalThreadsSysTime: 33.984ms
>> > - TotalThreadsUserTime: 702.879ms
>> > - TotalThreadsVoluntaryContextSwitches: 2.04K (2037)
>> > Fragment Instance Lifecycle Timings:
>> > - ExecTime: 604.005us
>> > - ExecTreeExecTime: 8.338us
>> > - OpenTime: 352.927ms
>> > - ExecTreeOpenTime: 331.921ms
>> > - PrepareTime: 23.783ms
>> > - ExecTreePrepareTime: 2.559ms
>> > DataStreamSender (dst_id=2):(Total: 331.818us, non-child:
>> > 331.818us,
>> > % non-child: 100.00%)
>> > - BytesSent: 16.00 B (16)
>> > - NetworkThroughput(*): 289.00 KB/sec
>> > - OverallThroughput: 47.09 KB/sec
>> > - PeakMemoryUsage: 16.00 KB (16384)
>> > - RowsReturned: 1 (1)
>> > - SerializeBatchTime: 14.466us
>> > - TransmitDataRPCTime: 54.066us
>> > - UncompressedRowBatchSize: 16.00 B (16)
>> > CodeGen:(Total: 40.950ms, non-child: 40.950ms, % non-child:
>> > 100.00%)
>> > - CodegenTime: 1.122ms
>> > - CompileTime: 7.318ms
>> > - LoadTime: 0.000ns
>> > - ModuleBitcodeSize: 1.86 MB (1950880)
>> > - NumFunctions: 15 (15)
>> > - NumInstructions: 214 (214)
>> > - OptimizationTime: 13.106ms
>> > - PeakMemoryUsage: 107.00 KB (109568)
>> > - PrepareTime: 19.773ms
>> > AGGREGATION_NODE (id=1):(Total: 334.482ms, non-child: 766.527us,
>> > %
>> > non-child: 0.23%)
>> > ExecOption: Codegen Enabled
>> > - BuildTime: 1.192ms
>> > - GetResultsTime: 0.000ns
>> > - HTResizeTime: 0.000ns
>> > - HashBuckets: 0 (0)
>> > - LargestPartitionPercent: 0 (0)
>> > - MaxPartitionLevel: 0 (0)
>> > - NumRepartitions: 0 (0)
>> > - PartitionsCreated: 0 (0)
>> > - PeakMemoryUsage: 20.00 KB (20480)
>> > - RowsRepartitioned: 0 (0)
>> > - RowsReturned: 1 (1)
>> > - RowsReturnedRate: 2.00 /sec
>> > - SpilledPartitions: 0 (0)
>> > HDFS_SCAN_NODE (id=0):(Total: 333.715ms, non-child: 333.715ms, %
>> > non-child: 100.00%)
>> > Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>> > 0:36/21.95 MB 11:33/19.87 MB 9:27/15.16 MB 5:44/28.07 MB 6:56/33.32 MB
>> > 1:38/22.24 MB 10:44/27.90 MB 3:34/20.26 MB 2:45/28.21 MB 7:31/19.33 MB
>> > 4:42/26.73 MB 8:34/21.25 MB
>> > ExecOption: PARQUET Codegen Enabled, Codegen enabled: 464 out
>> > of
>> > 464
>> > Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0%
>> > 4:0%
>> > 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
>> > File Formats: PARQUET/Unknown(Skipped):464
>> > BytesRead(500.000ms): 12.27 MB
>> > - FooterProcessingTime: (Avg: 13.555ms ; Min: 5.433ms ; Max:
>> > 62.957ms ; Number of samples: 464)
>> > - AverageHdfsReadThreadConcurrency: 0.00
>> > - AverageScannerThreadConcurrency: 24.00
>> > - BytesRead: 45.01 MB (47193988)
>> > - BytesReadDataNodeCache: 0
>> > - BytesReadLocal: 45.01 MB (47193988)
>> > - BytesReadRemoteUnexpected: 0
>> > - BytesReadShortCircuit: 45.01 MB (47193988)
>> > - CachedFileHandlesHitCount: 0 (0)
>> > - CachedFileHandlesMissCount: 464 (464)
>> > - CollectionItemsRead: 0 (0)
>> > - DecompressionTime: 0.000ns
>> > - MaxCompressedTextFileLength: 0
>> > - NumColumns: 0 (0)
>> > - NumDictFilteredRowGroups: 0 (0)
>> > - NumDisksAccessed: 12 (12)
>> > - NumRowGroups: 464 (464)
>> > - NumScannerThreadsStarted: 24 (24)
>> > - NumScannersWithNoReads: 0 (0)
>> > - NumStatsFilteredRowGroups: 0 (0)
>> > - PeakMemoryUsage: 2.38 MB (2490376)
>> > - PerReadThreadRawHdfsThroughput: 1016.44 MB/sec
>> > - RemoteScanRanges: 0 (0)
>> > - RowBatchQueueGetWaitTime: 320.860ms
>> > - RowBatchQueuePutWaitTime: 0.000ns
>> > - RowsRead: 0 (0)
>> > - RowsReturned: 464 (464)
>> > - RowsReturnedRate: 1.39 K/sec
>> > - ScanRangesComplete: 464 (464)
>> > - ScannerThreadsInvoluntaryContextSwitches: 108 (108)
>> > - ScannerThreadsTotalWallClockTime: 7s769ms
>> > - MaterializeTupleTime(*): 75.514us
>> > - ScannerThreadsSysTime: 29.984ms
>> > - ScannerThreadsUserTime: 669.884ms
>> > - ScannerThreadsVoluntaryContextSwitches: 1.60K (1597)
>> > - TotalRawHdfsReadTime(*): 44.279ms
>> > - TotalReadThroughput: 24.53 MB/sec
>> > Instance c843dae48ce9b289:6ad3ffc000000002
>> > (host=aopr-dhc003.lpdomain.com:22000):(Total: 351.014ms, non-child:
>> > 1.641ms,
>> > % non-child: 0.47%)
>> > Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>> > 0:35/20.89 MB 7:44/26.64 MB 11:27/17.21 MB 9:32/18.61 MB 5:25/14.56 MB
>> > 2:38/24.32 MB 6:31/19.85 MB 8:50/30.76 MB 10:41/25.92 MB 3:31/19.78 MB
>> > 4:60/37.59 MB 1:44/27.19 MB
>> > MemoryUsage(500.000ms): 1.16 MB
>> > ThreadUsage(500.000ms): 25
>> > - AverageThreadTokens: 25.00
>> > - BloomFilterBytes: 0
>> > - PeakMemoryUsage: 2.54 MB (2662753)
>> > - PeakReservation: 0
>> > - PeakUsedReservation: 0
>> > - PerHostPeakMemUsage: 2.54 MB (2662753)
>> > - RowsProduced: 1 (1)
>> > - TotalNetworkReceiveTime: 0.000ns
>> > - TotalNetworkSendTime: 342.746us
>> > - TotalStorageWaitTime: 4s796ms
>> > - TotalThreadsInvoluntaryContextSwitches: 93 (93)
>> > - TotalThreadsTotalWallClockTime: 7s286ms
>> > - TotalThreadsSysTime: 27.988ms
>> > - TotalThreadsUserTime: 671.887ms
>> > - TotalThreadsVoluntaryContextSwitches: 2.08K (2079)
>> > Fragment Instance Lifecycle Timings:
>> > - ExecTime: 524.006us
>> > - ExecTreeExecTime: 7.920us
>> > - OpenTime: 327.802ms
>> > - ExecTreeOpenTime: 307.154ms
>> > - PrepareTime: 22.617ms
>> > - ExecTreePrepareTime: 2.650ms
>> > DataStreamSender (dst_id=2):(Total: 348.990us, non-child:
>> > 348.990us,
>> > % non-child: 100.00%)
>> > - BytesSent: 16.00 B (16)
>> > - NetworkThroughput(*): 354.47 KB/sec
>> > - OverallThroughput: 44.77 KB/sec
>> > - PeakMemoryUsage: 16.00 KB (16384)
>> > - RowsReturned: 1 (1)
>> > - SerializeBatchTime: 15.602us
>> > - TransmitDataRPCTime: 44.080us
>> > - UncompressedRowBatchSize: 16.00 B (16)
>> > CodeGen:(Total: 39.215ms, non-child: 39.215ms, % non-child:
>> > 100.00%)
>> > - CodegenTime: 1.183ms
>> > - CompileTime: 7.168ms
>> > - LoadTime: 0.000ns
>> > - ModuleBitcodeSize: 1.86 MB (1950880)
>> > - NumFunctions: 15 (15)
>> > - NumInstructions: 214 (214)
>> > - OptimizationTime: 12.929ms
>> > - PeakMemoryUsage: 107.00 KB (109568)
>> > - PrepareTime: 18.453ms
>> > AGGREGATION_NODE (id=1):(Total: 309.807ms, non-child: 948.677us,
>> > %
>> > non-child: 0.31%)
>> > ExecOption: Codegen Enabled
>> > - BuildTime: 1.552ms
>> > - GetResultsTime: 0.000ns
>> > - HTResizeTime: 0.000ns
>> > - HashBuckets: 0 (0)
>> > - LargestPartitionPercent: 0 (0)
>> > - MaxPartitionLevel: 0 (0)
>> > - NumRepartitions: 0 (0)
>> > - PartitionsCreated: 0 (0)
>> > - PeakMemoryUsage: 20.00 KB (20480)
>> > - RowsRepartitioned: 0 (0)
>> > - RowsReturned: 1 (1)
>> > - RowsReturnedRate: 3.00 /sec
>> > - SpilledPartitions: 0 (0)
>> > HDFS_SCAN_NODE (id=0):(Total: 308.859ms, non-child: 308.859ms, %
>> > non-child: 100.00%)
>> > Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>> > 0:35/20.89 MB 7:44/26.64 MB 11:27/17.21 MB 9:32/18.61 MB 5:25/14.56 MB
>> > 2:38/24.32 MB 6:31/19.85 MB 8:50/30.76 MB 10:41/25.92 MB 3:31/19.78 MB
>> > 4:60/37.59 MB 1:44/27.19 MB
>> > ExecOption: PARQUET Codegen Enabled, Codegen enabled: 458 out
>> > of
>> > 458
>> > Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0%
>> > 4:0%
>> > 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
>> > File Formats: PARQUET/Unknown(Skipped):458
>> > BytesRead(500.000ms): 36.74 MB
>> > - FooterProcessingTime: (Avg: 11.502ms ; Min: 3.668ms ; Max:
>> > 23.564ms ; Number of samples: 458)
>> > - AverageHdfsReadThreadConcurrency: 0.00
>> > - AverageScannerThreadConcurrency: 24.00
>> > - BytesRead: 44.51 MB (46668432)
>> > - BytesReadDataNodeCache: 0
>> > - BytesReadLocal: 44.51 MB (46668432)
>> > - BytesReadRemoteUnexpected: 0
>> > - BytesReadShortCircuit: 44.51 MB (46668432)
>> > - CachedFileHandlesHitCount: 0 (0)
>> > - CachedFileHandlesMissCount: 458 (458)
>> > - CollectionItemsRead: 0 (0)
>> > - DecompressionTime: 0.000ns
>> > - MaxCompressedTextFileLength: 0
>> > - NumColumns: 0 (0)
>> > - NumDictFilteredRowGroups: 0 (0)
>> > - NumDisksAccessed: 12 (12)
>> > - NumRowGroups: 458 (458)
>> > - NumScannerThreadsStarted: 24 (24)
>> > - NumScannersWithNoReads: 0 (0)
>> > - NumStatsFilteredRowGroups: 0 (0)
>> > - PeakMemoryUsage: 2.50 MB (2621440)
>> > - PerReadThreadRawHdfsThroughput: 1.26 GB/sec
>> > - RemoteScanRanges: 0 (0)
>> > - RowBatchQueueGetWaitTime: 295.595ms
>> > - RowBatchQueuePutWaitTime: 0.000ns
>> > - RowsRead: 0 (0)
>> > - RowsReturned: 458 (458)
>> > - RowsReturnedRate: 1.48 K/sec
>> > - ScanRangesComplete: 458 (458)
>> > - ScannerThreadsInvoluntaryContextSwitches: 91 (91)
>> > - ScannerThreadsTotalWallClockTime: 6s958ms
>> > - MaterializeTupleTime(*): 71.630us
>> > - ScannerThreadsSysTime: 16.989ms
>> > - ScannerThreadsUserTime: 645.891ms
>> > - ScannerThreadsVoluntaryContextSwitches: 1.63K (1630)
>> > - TotalRawHdfsReadTime(*): 34.377ms
>> > - TotalReadThroughput: 73.49 MB/sec
>> > Instance c843dae48ce9b289:6ad3ffc000000001
>> > (host=aopr-dhc005.lpdomain.com:22000):(Total: 308.195ms, non-child:
>> > 1.784ms,
>> > % non-child: 0.58%)
>> > Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>> > 2:46/28.97 MB 8:39/24.34 MB 10:33/21.03 MB 3:43/26.50 MB 5:32/20.27 MB
>> > 9:32/19.86 MB 6:25/16.01 MB 7:41/24.77 MB 1:42/25.01 MB 0:55/32.51 MB
>> > 4:40/24.29 MB 11:33/20.77 MB
>> > - AverageThreadTokens: 0.00
>> > - BloomFilterBytes: 0
>> > - PeakMemoryUsage: 2.56 MB (2683241)
>> > - PeakReservation: 0
>> > - PeakUsedReservation: 0
>> > - PerHostPeakMemUsage: 2.56 MB (2683241)
>> > - RowsProduced: 1 (1)
>> > - TotalNetworkReceiveTime: 0.000ns
>> > - TotalNetworkSendTime: 292.765us
>> > - TotalStorageWaitTime: 4s482ms
>> > - TotalThreadsInvoluntaryContextSwitches: 113 (113)
>> > - TotalThreadsTotalWallClockTime: 6s310ms
>> > - TotalThreadsSysTime: 35.983ms
>> > - TotalThreadsUserTime: 682.884ms
>> > - TotalThreadsVoluntaryContextSwitches: 2.14K (2143)
>> > Fragment Instance Lifecycle Timings:
>> > - ExecTime: 532.572us
>> > - ExecTreeExecTime: 8.377us
>> > - OpenTime: 280.241ms
>> > - ExecTreeOpenTime: 258.652ms
>> > - PrepareTime: 27.342ms
>> > - ExecTreePrepareTime: 2.863ms
>> > DataStreamSender (dst_id=2):(Total: 309.794us, non-child:
>> > 309.794us,
>> > % non-child: 100.00%)
>> > - BytesSent: 16.00 B (16)
>> > - NetworkThroughput(*): 356.91 KB/sec
>> > - OverallThroughput: 50.44 KB/sec
>> > - PeakMemoryUsage: 16.00 KB (16384)
>> > - RowsReturned: 1 (1)
>> > - SerializeBatchTime: 14.262us
>> > - TransmitDataRPCTime: 43.778us
>> > - UncompressedRowBatchSize: 16.00 B (16)
>> > CodeGen:(Total: 44.584ms, non-child: 44.584ms, % non-child:
>> > 100.00%)
>> > - CodegenTime: 1.382ms
>> > - CompileTime: 7.346ms
>> > - LoadTime: 0.000ns
>> > - ModuleBitcodeSize: 1.86 MB (1950880)
>> > - NumFunctions: 15 (15)
>> > - NumInstructions: 214 (214)
>> > - OptimizationTime: 13.462ms
>> > - PeakMemoryUsage: 107.00 KB (109568)
>> > - PrepareTime: 23.006ms
>> > AGGREGATION_NODE (id=1):(Total: 261.516ms, non-child: 836.656us,
>> > %
>> > non-child: 0.32%)
>> > ExecOption: Codegen Enabled
>> > - BuildTime: 1.280ms
>> > - GetResultsTime: 0.000ns
>> > - HTResizeTime: 0.000ns
>> > - HashBuckets: 0 (0)
>> > - LargestPartitionPercent: 0 (0)
>> > - MaxPartitionLevel: 0 (0)
>> > - NumRepartitions: 0 (0)
>> > - PartitionsCreated: 0 (0)
>> > - PeakMemoryUsage: 20.00 KB (20480)
>> > - RowsRepartitioned: 0 (0)
>> > - RowsReturned: 1 (1)
>> > - RowsReturnedRate: 3.00 /sec
>> > - SpilledPartitions: 0 (0)
>> > HDFS_SCAN_NODE (id=0):(Total: 260.680ms, non-child: 260.680ms, %
>> > non-child: 100.00%)
>> > Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>> > 2:46/28.97 MB 8:39/24.34 MB 10:33/21.03 MB 3:43/26.50 MB 5:32/20.27 MB
>> > 9:32/19.86 MB 6:25/16.01 MB 7:41/24.77 MB 1:42/25.01 MB 0:55/32.51 MB
>> > 4:40/24.29 MB 11:33/20.77 MB
>> > ExecOption: PARQUET Codegen Enabled, Codegen enabled: 461 out
>> > of
>> > 461
>> > Hdfs Read Thread Concurrency Bucket: 0:0% 1:0% 2:0% 3:0% 4:0%
>> > 5:0%
>> > 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
>> > File Formats: PARQUET/Unknown(Skipped):461
>> > - FooterProcessingTime: (Avg: 10.793ms ; Min: 3.087ms ; Max:
>> > 20.253ms ; Number of samples: 461)
>> > - AverageHdfsReadThreadConcurrency: 0.00
>> > - AverageScannerThreadConcurrency: 0.00
>> > - BytesRead: 44.73 MB (46906407)
>> > - BytesReadDataNodeCache: 0
>> > - BytesReadLocal: 44.73 MB (46906407)
>> > - BytesReadRemoteUnexpected: 0
>> > - BytesReadShortCircuit: 44.73 MB (46906407)
>> > - CachedFileHandlesHitCount: 0 (0)
>> > - CachedFileHandlesMissCount: 461 (461)
>> > - CollectionItemsRead: 0 (0)
>> > - DecompressionTime: 0.000ns
>> > - MaxCompressedTextFileLength: 0
>> > - NumColumns: 0 (0)
>> > - NumDictFilteredRowGroups: 0 (0)
>> > - NumDisksAccessed: 12 (12)
>> > - NumRowGroups: 461 (461)
>> > - NumScannerThreadsStarted: 24 (24)
>> > - NumScannersWithNoReads: 0 (0)
>> > - NumStatsFilteredRowGroups: 0 (0)
>> > - PeakMemoryUsage: 2.52 MB (2641928)
>> > - PerReadThreadRawHdfsThroughput: 1.31 GB/sec
>> > - RemoteScanRanges: 0 (0)
>> > - RowBatchQueueGetWaitTime: 246.691ms
>> > - RowBatchQueuePutWaitTime: 0.000ns
>> > - RowsRead: 0 (0)
>> > - RowsReturned: 461 (461)
>> > - RowsReturnedRate: 1.77 K/sec
>> > - ScanRangesComplete: 461 (461)
>> > - ScannerThreadsInvoluntaryContextSwitches: 109 (109)
>> > - ScannerThreadsTotalWallClockTime: 6s030ms
>> > - MaterializeTupleTime(*): 68.735us
>> > - ScannerThreadsSysTime: 33.983ms
>> > - ScannerThreadsUserTime: 647.889ms
>> > - ScannerThreadsVoluntaryContextSwitches: 1.70K (1696)
>> > - TotalRawHdfsReadTime(*): 33.405ms
>> > - TotalReadThroughput: 0.00 /sec
>> > Instance c843dae48ce9b289:6ad3ffc000000003
>> > (host=aopr-dhc002.lpdomain.com:22000):(Total: 302.116ms, non-child:
>> > 1.604ms,
>> > % non-child: 0.53%)
>> > Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>> > 0:27/16.21 MB 11:33/21.23 MB 6:36/23.15 MB 10:41/25.09 MB 1:41/24.13 MB
>> > 9:32/18.79 MB 5:41/25.31 MB 2:42/26.49 MB 7:36/22.87 MB 3:51/31.00 MB
>> > 4:41/25.99 MB 8:38/23.79 MB
>> > MemoryUsage(500.000ms): 700.35 KB
>> > ThreadUsage(500.000ms): 25
>> > - AverageThreadTokens: 25.00
>> > - BloomFilterBytes: 0
>> > - PeakMemoryUsage: 2.54 MB (2662753)
>> > - PeakReservation: 0
>> > - PeakUsedReservation: 0
>> > - PerHostPeakMemUsage: 2.54 MB (2662753)
>> > - RowsProduced: 1 (1)
>> > - TotalNetworkReceiveTime: 0.000ns
>> > - TotalNetworkSendTime: 300.675us
>> > - TotalStorageWaitTime: 4s661ms
>> > - TotalThreadsInvoluntaryContextSwitches: 90 (90)
>> > - TotalThreadsTotalWallClockTime: 6s256ms
>> > - TotalThreadsSysTime: 30.983ms
>> > - TotalThreadsUserTime: 709.880ms
>> > - TotalThreadsVoluntaryContextSwitches: 2.17K (2175)
>> > Fragment Instance Lifecycle Timings:
>> > - ExecTime: 492.627us
>> > - ExecTreeExecTime: 8.191us
>> > - OpenTime: 277.922ms
>> > - ExecTreeOpenTime: 255.395ms
>> > - PrepareTime: 23.647ms
>> > - ExecTreePrepareTime: 2.243ms
>> > DataStreamSender (dst_id=2):(Total: 194.517us, non-child:
>> > 194.517us,
>> > % non-child: 100.00%)
>> > - BytesSent: 16.00 B (16)
>> > - NetworkThroughput(*): 371.86 KB/sec
>> > - OverallThroughput: 80.33 KB/sec
>> > - PeakMemoryUsage: 16.00 KB (16384)
>> > - RowsReturned: 1 (1)
>> > - SerializeBatchTime: 14.035us
>> > - TransmitDataRPCTime: 42.019us
>> > - UncompressedRowBatchSize: 16.00 B (16)
>> > CodeGen:(Total: 42.676ms, non-child: 42.676ms, % non-child:
>> > 100.00%)
>> > - CodegenTime: 1.236ms
>> > - CompileTime: 7.717ms
>> > - LoadTime: 0.000ns
>> > - ModuleBitcodeSize: 1.86 MB (1950880)
>> > - NumFunctions: 15 (15)
>> > - NumInstructions: 214 (214)
>> > - OptimizationTime: 14.130ms
>> > - PeakMemoryUsage: 107.00 KB (109568)
>> > - PrepareTime: 20.109ms
>> > AGGREGATION_NODE (id=1):(Total: 257.640ms, non-child: 1.541ms, %
>> > non-child: 0.60%)
>> > ExecOption: Codegen Enabled
>> > - BuildTime: 1.252ms
>> > - GetResultsTime: 0.000ns
>> > - HTResizeTime: 0.000ns
>> > - HashBuckets: 0 (0)
>> > - LargestPartitionPercent: 0 (0)
>> > - MaxPartitionLevel: 0 (0)
>> > - NumRepartitions: 0 (0)
>> > - PartitionsCreated: 0 (0)
>> > - PeakMemoryUsage: 20.00 KB (20480)
>> > - RowsRepartitioned: 0 (0)
>> > - RowsReturned: 1 (1)
>> > - RowsReturnedRate: 3.00 /sec
>> > - SpilledPartitions: 0 (0)
>> > HDFS_SCAN_NODE (id=0):(Total: 256.099ms, non-child: 256.099ms, %
>> > non-child: 100.00%)
>> > Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>> > 0:27/16.21 MB 11:33/21.23 MB 6:36/23.15 MB 10:41/25.09 MB 1:41/24.13 MB
>> > 9:32/18.79 MB 5:41/25.31 MB 2:42/26.49 MB 7:36/22.87 MB 3:51/31.00 MB
>> > 4:41/25.99 MB 8:38/23.79 MB
>> > ExecOption: PARQUET Codegen Enabled, Codegen enabled: 459 out
>> > of
>> > 459
>> > Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0%
>> > 4:0%
>> > 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
>> > File Formats: PARQUET/Unknown(Skipped):459
>> > BytesRead(500.000ms): 43.15 MB
>> > - FooterProcessingTime: (Avg: 11.257ms ; Min: 3.989ms ; Max:
>> > 23.004ms ; Number of samples: 459)
>> > - AverageHdfsReadThreadConcurrency: 0.00
>> > - AverageScannerThreadConcurrency: 24.00
>> > - BytesRead: 44.52 MB (46678031)
>> > - BytesReadDataNodeCache: 0
>> > - BytesReadLocal: 44.52 MB (46678031)
>> > - BytesReadRemoteUnexpected: 0
>> > - BytesReadShortCircuit: 44.52 MB (46678031)
>> > - CachedFileHandlesHitCount: 0 (0)
>> > - CachedFileHandlesMissCount: 459 (459)
>> > - CollectionItemsRead: 0 (0)
>> > - DecompressionTime: 0.000ns
>> > - MaxCompressedTextFileLength: 0
>> > - NumColumns: 0 (0)
>> > - NumDictFilteredRowGroups: 0 (0)
>> > - NumDisksAccessed: 12 (12)
>> > - NumRowGroups: 459 (459)
>> > - NumScannerThreadsStarted: 24 (24)
>> > - NumScannersWithNoReads: 0 (0)
>> > - NumStatsFilteredRowGroups: 0 (0)
>> > - PeakMemoryUsage: 2.50 MB (2621440)
>> > - PerReadThreadRawHdfsThroughput: 1.33 GB/sec
>> > - RemoteScanRanges: 0 (0)
>> > - RowBatchQueueGetWaitTime: 243.426ms
>> > - RowBatchQueuePutWaitTime: 0.000ns
>> > - RowsRead: 0 (0)
>> > - RowsReturned: 459 (459)
>> > - RowsReturnedRate: 1.79 K/sec
>> > - ScanRangesComplete: 459 (459)
>> > - ScannerThreadsInvoluntaryContextSwitches: 85 (85)
>> > - ScannerThreadsTotalWallClockTime: 5s978ms
>> > - MaterializeTupleTime(*): 67.997us
>> > - ScannerThreadsSysTime: 22.984ms
>> > - ScannerThreadsUserTime: 678.884ms
>> > - ScannerThreadsVoluntaryContextSwitches: 1.72K (1724)
>> > - TotalRawHdfsReadTime(*): 32.626ms
>> > - TotalReadThroughput: 86.30 MB/sec
>> > Instance c843dae48ce9b289:6ad3ffc000000005
>> > (host=aopr-dhc001.lpdomain.com:22000):(Total: 298.093ms, non-child:
>> > 1.242ms,
>> > % non-child: 0.42%)
>> > Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>> > 9:55/35.38 MB 0:37/24.23 MB 5:37/23.31 MB 11:22/13.79 MB 8:30/18.53 MB
>> > 4:33/20.45 MB 1:38/24.06 MB 10:29/18.40 MB 3:39/24.35 MB 2:41/25.45 MB
>> > 6:42/27.03 MB 7:47/29.22 MB
>> > - AverageThreadTokens: 0.00
>> > - BloomFilterBytes: 0
>> > - PeakMemoryUsage: 2.66 MB (2793825)
>> > - PeakReservation: 0
>> > - PeakUsedReservation: 0
>> > - PerHostPeakMemUsage: 2.68 MB (2806113)
>> > - RowsProduced: 1 (1)
>> > - TotalNetworkReceiveTime: 0.000ns
>> > - TotalNetworkSendTime: 198.894us
>> > - TotalStorageWaitTime: 4s374ms
>> > - TotalThreadsInvoluntaryContextSwitches: 110 (110)
>> > - TotalThreadsTotalWallClockTime: 6s201ms
>> > - TotalThreadsSysTime: 29.983ms
>> > - TotalThreadsUserTime: 642.892ms
>> > - TotalThreadsVoluntaryContextSwitches: 2.17K (2174)
>> > Fragment Instance Lifecycle Timings:
>> > - ExecTime: 350.873us
>> > - ExecTreeExecTime: 8.173us
>> > - OpenTime: 275.347ms
>> > - ExecTreeOpenTime: 253.425ms
>> > - PrepareTime: 22.341ms
>> > - ExecTreePrepareTime: 2.251ms
>> > DataStreamSender (dst_id=2):(Total: 164.394us, non-child:
>> > 164.394us,
>> > % non-child: 100.00%)
>> > - BytesSent: 16.00 B (16)
>> > - NetworkThroughput(*): 493.42 KB/sec
>> > - OverallThroughput: 95.05 KB/sec
>> > - PeakMemoryUsage: 16.00 KB (16384)
>> > - RowsReturned: 1 (1)
>> > - SerializeBatchTime: 14.262us
>> > - TransmitDataRPCTime: 31.667us
>> > - UncompressedRowBatchSize: 16.00 B (16)
>> > CodeGen:(Total: 41.008ms, non-child: 41.008ms, % non-child:
>> > 100.00%)
>> > - CodegenTime: 1.049ms
>> > - CompileTime: 7.248ms
>> > - LoadTime: 0.000ns
>> > - ModuleBitcodeSize: 1.86 MB (1950880)
>> > - NumFunctions: 15 (15)
>> > - NumInstructions: 214 (214)
>> > - OptimizationTime: 14.018ms
>> > - PeakMemoryUsage: 107.00 KB (109568)
>> > - PrepareTime: 19.080ms
>> > AGGREGATION_NODE (id=1):(Total: 255.678ms, non-child: 1.046ms, %
>> > non-child: 0.41%)
>> > ExecOption: Codegen Enabled
>> > - BuildTime: 1.015ms
>> > - GetResultsTime: 0.000ns
>> > - HTResizeTime: 0.000ns
>> > - HashBuckets: 0 (0)
>> > - LargestPartitionPercent: 0 (0)
>> > - MaxPartitionLevel: 0 (0)
>> > - NumRepartitions: 0 (0)
>> > - PartitionsCreated: 0 (0)
>> > - PeakMemoryUsage: 20.00 KB (20480)
>> > - RowsRepartitioned: 0 (0)
>> > - RowsReturned: 1 (1)
>> > - RowsReturnedRate: 3.00 /sec
>> > - SpilledPartitions: 0 (0)
>> > HDFS_SCAN_NODE (id=0):(Total: 254.631ms, non-child: 254.631ms, %
>> > non-child: 100.00%)
>> > Hdfs split stats (<volume id>:<# splits>/<split lengths>):
>> > 9:55/35.38 MB 0:37/24.23 MB 5:37/23.31 MB 11:22/13.79 MB 8:30/18.53 MB
>> > 4:33/20.45 MB 1:38/24.06 MB 10:29/18.40 MB 3:39/24.35 MB 2:41/25.45 MB
>> > 6:42/27.03 MB 7:47/29.22 MB
>> > ExecOption: PARQUET Codegen Enabled, Codegen enabled: 450 out
>> > of
>> > 450
>> > Hdfs Read Thread Concurrency Bucket: 0:0% 1:0% 2:0% 3:0% 4:0%
>> > 5:0%
>> > 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
>> > File Formats: PARQUET/Unknown(Skipped):450
>> > - FooterProcessingTime: (Avg: 10.727ms ; Min: 3.813ms ; Max:
>> > 23.755ms ; Number of samples: 450)
>> > - AverageHdfsReadThreadConcurrency: 0.00
>> > - AverageScannerThreadConcurrency: 0.00
>> > - BytesRead: 43.85 MB (45983284)
>> > - BytesReadDataNodeCache: 0
>> > - BytesReadLocal: 43.85 MB (45983284)
>> > - BytesReadRemoteUnexpected: 0
>> > - BytesReadShortCircuit: 43.85 MB (45983284)
>> > - CachedFileHandlesHitCount: 0 (0)
>> > - CachedFileHandlesMissCount: 450 (450)
>> > - CollectionItemsRead: 0 (0)
>> > - DecompressionTime: 0.000ns
>> > - MaxCompressedTextFileLength: 0
>> > - NumColumns: 0 (0)
>> > - NumDictFilteredRowGroups: 0 (0)
>> > - NumDisksAccessed: 12 (12)
>> > - NumRowGroups: 450 (450)
>> > - NumScannerThreadsStarted: 24 (24)
>> > - NumScannersWithNoReads: 0 (0)
>> > - NumStatsFilteredRowGroups: 0 (0)
>> > - PeakMemoryUsage: 2.62 MB (2752512)
>> > - PerReadThreadRawHdfsThroughput: 1.56 GB/sec
>> > - RemoteScanRanges: 0 (0)
>> > - RowBatchQueueGetWaitTime: 242.362ms
>> > - RowBatchQueuePutWaitTime: 0.000ns
>> > - RowsRead: 0 (0)
>> > - RowsReturned: 450 (450)
>> > - RowsReturnedRate: 1.77 K/sec
>> > - ScanRangesComplete: 450 (450)
>> > - ScannerThreadsInvoluntaryContextSwitches: 106 (106)
>> > - ScannerThreadsTotalWallClockTime: 5s926ms
>> > - MaterializeTupleTime(*): 69.447us
>> > - ScannerThreadsSysTime: 26.984ms
>> > - ScannerThreadsUserTime: 607.897ms
>> > - ScannerThreadsVoluntaryContextSwitches: 1.72K (1723)
>> > - TotalRawHdfsReadTime(*): 27.393ms
>> > - TotalReadThroughput: 0.00 /sec
>> > ===================================================
>> >
>> > Coordinator log:
>> >
>> > 01: query (string) = "select count(*) from dv_entersiteevent_prq_remote
>> > where year=2017 and month=10",
>> > 03: configuration (list) = list<string>[1] {
>> > [0] = "REQUEST_POOL=impreports",
>> > },
>> > 04: hadoop_user (string) = "fawzea",
>> > }
>> > I0208 01:07:06.321281 7154 impala-beeswax-server.cc:459]
>> > TClientRequest.queryOptions: TQueryOptions {
>> > 01: abort_on_error (bool) = false,
>> > 02: max_errors (i32) = 100,
>> > 03: disable_codegen (bool) = false,
>> > 04: batch_size (i32) = 0,
>> > 05: num_nodes (i32) = 0,
>> > 06: max_scan_range_length (i64) = 0,
>> > 07: num_scanner_threads (i32) = 0,
>> > 08: max_io_buffers (i32) = 0,
>> > 09: allow_unsupported_formats (bool) = false,
>> > 10: default_order_by_limit (i64) = -1,
>> > 11: debug_action (string) = "",
>> > 12: mem_limit (i64) = 0,
>> > 13: abort_on_default_limit_exceeded (bool) = false,
>> > 15: hbase_caching (i32) = 0,
>> > 16: hbase_cache_blocks (bool) = false,
>> > 17: parquet_file_size (i64) = 0,
>> > 18: explain_level (i32) = 1,
>> > 19: sync_ddl (bool) = false,
>> > 20: request_pool (string) = "impreports",
>> > 23: disable_cached_reads (bool) = false,
>> > 24: disable_outermost_topn (bool) = false,
>> > 25: rm_initial_mem (i64) = 0,
>> > 26: query_timeout_s (i32) = 0,
>> > 28: appx_count_distinct (bool) = false,
>> > 29: disable_unsafe_spills (bool) = false,
>> > 31: exec_single_node_rows_threshold (i32) = 100,
>> > 32: optimize_partition_key_scans (bool) = false,
>> > 33: replica_preference (i32) = 0,
>> > 34: schedule_random_replica (bool) = false,
>> > 35: scan_node_codegen_threshold (i64) = 1800000,
>> > 36: disable_streaming_preaggregations (bool) = false,
>> > 37: runtime_filter_mode (i32) = 2,
>> > 38: runtime_bloom_filter_size (i32) = 1048576,
>> > 39: runtime_filter_wait_time_ms (i32) = 0,
>> > 40: disable_row_runtime_filtering (bool) = false,
>> > 41: max_num_runtime_filters (i32) = 10,
>> > 42: parquet_annotate_strings_utf8 (bool) = false,
>> > 43: parquet_fallback_schema_resolution (i32) = 0,
>> > 45: s3_skip_insert_staging (bool) = true,
>> > 46: runtime_filter_min_size (i32) = 1048576,
>> > 47: runtime_filter_max_size (i32) = 16777216,
>> > 48: prefetch_mode (i32) = 1,
>> > 49: strict_mode (bool) = false,
>> > 50: scratch_limit (i64) = -1,
>> > 51: enable_expr_rewrites (bool) = true,
>> > 52: decimal_v2 (bool) = false,
>> > 53: parquet_dictionary_filtering (bool) = true,
>> > 54: parquet_array_resolution (i32) = 2,
>> > 55: parquet_read_statistics (bool) = true,
>> > 56: default_join_distribution_mode (i32) = 0,
>> > 57: disable_codegen_rows_threshold (i32) = 50000,
>> > 58: default_spillable_buffer_size (i64) = 2097152,
>> > 59: min_spillable_buffer_size (i64) = 65536,
>> > 60: max_row_size (i64) = 524288,
>> > }
>> > I0208 01:07:06.324410 7154 Frontend.java:909] Compiling query: select
>> > count(*) from dv_entersiteevent_prq_remote where year=2017 and month=10
>> > I0208 01:07:06.327203 7154 Frontend.java:948] Compiled query.
>> > I0208 01:07:06.359748 7154 admission-controller.cc:510] Schedule for
>> > id=5f49e927591a9042:faf8f28600000000 in pool_name=root.impreports
>> > cluster_mem_needed=105.00 MB PoolConfig: max_requests=-1 max_queued=200
>> > max_mem=-1.00 B
>> > I0208 01:07:06.359776 7154 admission-controller.cc:515] Stats:
>> > agg_num_running=0, agg_num_queued=0, agg_mem_reserved=0,
>> > local_host(local_mem_admitted=0, num_admitted_running=0, num_queued=0,
>> > backend_mem_reserved=0)
>> > I0208 01:07:06.359784 7154 admission-controller.cc:531] Admitted query
>> > id=5f49e927591a9042:faf8f28600000000
>> > I0208 01:07:06.359807 7154 coordinator.cc:99] Exec()
>> > query_id=5f49e927591a9042:faf8f28600000000 stmt=select count(*) from
>> > dv_entersiteevent_prq_remote where year=2017 and month=10
>> > I0208 01:07:06.360694 7154 coordinator.cc:357] starting execution on 5
>> > backends for query 5f49e927591a9042:faf8f28600000000
>> > I0208 01:07:06.363255 63583 impala-internal-service.cc:44]
>> > ExecQueryFInstances(): query_id=5f49e927591a9042:faf8f28600000000
>> > I0208 01:07:06.363282 63583 query-exec-mgr.cc:46] StartQueryFInstances()
>> > query_id=5f49e927591a9042:faf8f28600000000
>> > coord=aopr-dhc001.lpdomain.com:22000
>> > I0208 01:07:06.363296 63583 query-state.cc:173] Buffer pool limit for
>> > 5f49e927591a9042:faf8f28600000000: 10802010521
>> > I0208 01:07:06.363364 63583 initial-reservations.cc:60] Successfully
>> > claimed
>> > initial reservations (0) for query 5f49e927591a9042:faf8f28600000000
>> > I0208 01:07:06.363544 7155 query-state.cc:286] StartFInstances():
>> > query_id=5f49e927591a9042:faf8f28600000000 #instances=2
>> > I0208 01:07:06.363844 7155 query-state.cc:299] descriptor table for
>> > query=5f49e927591a9042:faf8f28600000000
>> > tuples:
>> > Tuple(id=2 size=0 slots=[] tuple_path=[])
>> > Tuple(id=1 size=8 slots=[Slot(id=2 type=BIGINT col_path=[] offset=0
>> > null=(offset=0 mask=0) slot_idx=0 field_idx=-1)] tuple_path=[])
>> > Tuple(id=0 size=8 slots=[Slot(id=3 type=BIGINT col_path=[] offset=0
>> > null=(offset=0 mask=0) slot_idx=0 field_idx=-1)] tuple_path=[])
>> > I0208 01:07:06.364013 7156 query-state.cc:377] Executing instance.
>> > instance_id=5f49e927591a9042:faf8f28600000000 fragment_idx=0
>> > per_fragment_instance_idx=0 coord_state_idx=0 #in-flight=1
>> > I0208 01:07:06.364135 7157 query-state.cc:377] Executing instance.
>> > instance_id=5f49e927591a9042:faf8f28600000005 fragment_idx=1
>> > per_fragment_instance_idx=4 coord_state_idx=0 #in-flight=2
>> > I0208 01:07:06.364241 7157 hdfs-scan-node.cc:160] Max row batch queue
>> > size
>> > for scan node '0' in fragment instance
>> > '5f49e927591a9042:faf8f28600000005':
>> > 160
>> > I0208 01:07:06.366775 7154 coordinator.cc:370] started execution on 5
>> > backends for query 5f49e927591a9042:faf8f28600000000
>> > I0208 01:07:06.385625 7155 query-exec-mgr.cc:149] ReleaseQueryState():
>> > query_id=5f49e927591a9042:faf8f28600000000 refcnt=4
>> > I0208 01:07:06.671193 7559 impala-server.cc:1796] Connection from
>> > client
>> > ::ffff:10.16.136.3:55598 closed, closing 1 associated session(s)
>> > I0208 01:07:06.676138 544 data-stream-mgr.cc:228] Reduced stream ID
>> > cache
>> > from 2 items, to 0, eviction took: 0
>> > I0208 01:07:06.677202 544 coordinator.cc:955] Backend completed:
>> > host=aopr-dhc002.lpdomain.com:22000 remaining=4
>> > I0208 01:07:06.677214 544 coordinator-backend-state.cc:229]
>> > query_id=5f49e927591a9042:faf8f28600000000: first in-progress backend:
>> > aopr-dhc001.lpdomain.com:22000
>> > I0208 01:07:06.684221 7157 query-state.cc:384] Instance completed.
>> > instance_id=5f49e927591a9042:faf8f28600000005 #in-flight=1 status=OK
>> > I0208 01:07:06.684237 7157 query-exec-mgr.cc:149] ReleaseQueryState():
>> > query_id=5f49e927591a9042:faf8f28600000000 refcnt=3
>> > I0208 01:07:06.700275 520 coordinator.cc:955] Backend completed:
>> > host=aopr-dhc005.lpdomain.com:22000 remaining=3
>> > I0208 01:07:06.700289 520 coordinator-backend-state.cc:229]
>> > query_id=5f49e927591a9042:faf8f28600000000: first in-progress backend:
>> > aopr-dhc001.lpdomain.com:22000
>> > I0208 01:07:06.704620 65258 coordinator.cc:955] Backend completed:
>> > host=aopr-dhc003.lpdomain.com:22000 remaining=2
>> > I0208 01:07:06.704666 65258 coordinator-backend-state.cc:229]
>> > query_id=5f49e927591a9042:faf8f28600000000: first in-progress backend:
>> > aopr-dhc001.lpdomain.com:22000
>> > I0208 01:07:06.706041 7156 data-stream-mgr.cc:238] DeregisterRecvr():
>> > fragment_instance_id=5f49e927591a9042:faf8f28600000000, node=2
>> > I0208 01:07:06.706070 7156 data-stream-recvr.cc:235] cancelled stream:
>> > fragment_instance_id_=5f49e927591a9042:faf8f28600000000 node_id=2
>> > I0208 01:07:06.706809 64916 coordinator.cc:955] Backend completed:
>> > host=aopr-dhc004.lpdomain.com:22000 remaining=1
>> > I0208 01:07:06.706821 64916 coordinator-backend-state.cc:229]
>> > query_id=5f49e927591a9042:faf8f28600000000: first in-progress backend:
>> > aopr-dhc001.lpdomain.com:22000
>> > I0208 01:07:06.789568 7154 impala-beeswax-server.cc:193]
>> > get_results_metadata(): query_id=5f49e927591a9042:faf8f28600000000
>> > I0208 01:07:06.790439 7154 coordinator.cc:789] Coordinator waiting for
>> > backends to finish, 1 remaining
>> > I0208 01:07:06.791052 7154 coordinator.cc:794] All backends finished
>> > successfully.
>> > I0208 01:07:06.791086 7154 coordinator.cc:1090] Release admssion
>> > control
>> > resources for query 5f49e927591a9042:faf8f28600000000
>> > I0208 01:07:06.791363 7156 query-state.cc:384] Instance completed.
>> > instance_id=5f49e927591a9042:faf8f28600000000 #in-flight=0 status=OK
>> > I0208 01:07:06.791414 7156 query-exec-mgr.cc:149] ReleaseQueryState():
>> > query_id=5f49e927591a9042:faf8f28600000000 refcnt=2
>> > I0208 01:07:06.793586 7154 impala-beeswax-server.cc:235] close():
>> > query_id=5f49e927591a9042:faf8f28600000000
>> > I0208 01:07:06.793602 7154 impala-server.cc:992] UnregisterQuery():
>> > query_id=5f49e927591a9042:faf8f28600000000
>> > I0208 01:07:06.793614 7154 impala-server.cc:1075] Cancel():
>> > query_id=5f49e927591a9042:faf8f28600000000
>> > I0208 01:07:06.793624 7154 coordinator.cc:895] Cancel()
>> > query_id=5f49e927591a9042:faf8f28600000000
>> > I0208 01:07:06.793637 7154 coordinator.cc:905] CancelBackends()
>> > query_id=5f49e927591a9042:faf8f28600000000, tried to cancel 0 backends
>> > I0208 01:07:06.802776 7154 query-exec-mgr.cc:149] ReleaseQueryState():
>> > query_id=5f49e927591a9042:faf8f28600000000 refcnt=1
>> > I0208 01:07:06.814214 7154 impala-server.cc:1796] Connection from
>> > client
>> > ::ffff:10.16.144.131:51012 closed, closing 1 associated session(s)
>> > I0208 01:07:07.258891 7567 impala-server.cc:1796] Connection from
>> > client
>> > ::ffff:10.16.136.2:56375 closed, closing 1 associated session(s)
>> > I0208 01:07:11.600131 7819 impala-server.cc:1796] Connection from
>> > client
>> > ::ffff:10.16.136.3:27460 closed, closing 1 associated session(s)
>> > I0208 01:07:12.262739 7821 impala-server.cc:1796] Connection from
>> > client
>> > ::ffff:10.16.136.2:52672 closed, closing 1 associated session(s)
>> > I0208 01:07:16.665451 7822 impala-server.cc:1796] Connection from
>> > client
>> > ::ffff:10.16.136.3:10693 closed, closing 1 associated session(s)
>> > I0208 01:07:17.235321 7825 impala-server.cc:1796] Connection from
>> > client
>> > ::ffff:10.16.136.2:34752 closed, closing 1 associated session(s)
>> > I0208 01:07:21.410611 5858 impala-beeswax-server.cc:193]
>> > get_results_metadata(): query_id=1d4fbf0f2571f1e9:ea346f7900000000
>> > I0208 01:07:21.414175 5858 impala-beeswax-server.cc:235] close():
>> > query_id=1d4fbf0f2571f1e9:ea346f7900000000
>> > I0208 01:07:21.414185 5858 impala-server.cc:992] UnregisterQuery():
>> > query_id=1d4fbf0f2571f1e9:ea346f7900000000
>> > I0208 01:07:21.414191 5858 impala-server.cc:1075] Cancel():
>> > query_id=1d4fbf0f2571f1e9:ea346f7900000000
>> > I0208 01:07:21.421505 5858 impala-server.cc:1796] Connection from
>> > client
>> > ::ffff:10.26.180.75:60072 closed, closing 1 associated session(s)
>> > I0208 01:07:21.605690 7830 impala-server.cc:1796] Connection from
>> > client
>> > ::ffff:10.16.136.3:46660 closed, closing 1 associated session(s)
>> >
>> >
>> > On Thu, Feb 8, 2018 at 6:20 AM, Matthew Jacobs <mj...@apache.org>
>> > wrote:
>> >>
>> >> Thanks! I think that configuration looks OK.
>> >>
>> >> 1) What version of Impala is this?
>> >>
>> >> 2) Can you send the query profile and the Impala coordinator log for
>> >> the query where it's not behaving as expected?
>> >>
>> >> > When i'm running the following with fawzea user:
>> >> > set request_pool=impreports;
>> >> > select count(*) from dv_entersiteevent_prq_remote where year=2017 and
>> >> > month=10;
>> >> > I can see that the query is running with impreports pool.
>> >>
>> >> On Wed, Feb 7, 2018 at 7:59 PM, Fawze Abujaber <fa...@gmail.com>
>> >> wrote:
>> >> > Hi Matt,
>> >> >
>> >> > [root@aopr-dhc001 ~]# cat
>> >> >
>> >> >
>> >> > /var/run/cloudera-scm-agent/process/4558-impala-IMPALAD/impala-conf/fair-scheduler.xml
>> >> > <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
>> >> > <allocations>
>> >> > <queue name="root">
>> >> > <weight>1.0</weight>
>> >> > <schedulingPolicy>fair</schedulingPolicy>
>> >> > <queue name="impiaas">
>> >> > <weight>1.0</weight>
>> >> > <schedulingPolicy>fair</schedulingPolicy>
>> >> > <aclSubmitApps>fawzea </aclSubmitApps>
>> >> > </queue>
>> >> > <queue name="impreports">
>> >> > <weight>1.0</weight>
>> >> > <schedulingPolicy>fair</schedulingPolicy>
>> >> > <aclSubmitApps>nira </aclSubmitApps>
>> >> > </queue>
>> >> > </queue>
>> >> > <defaultQueueSchedulingPolicy>fair</defaultQueueSchedulingPolicy>
>> >> > <queuePlacementPolicy>
>> >> > <rule name="specified" create="false"/>
>> >> > <rule name="secondaryGroupExistingQueue"/>
>> >> > <rule name="default"/>
>> >> > </queuePlacementPolicy>
>> >> > </allocations>
>> >> >
>> >> >
>> >> > On Thu, Feb 8, 2018 at 4:47 AM, Matthew Jacobs <ja...@gmail.com>
>> >> > wrote:
>> >> >>
>> >> >> Yes, I saw that, but I'd like to see the actual configuration that
>> >> >> is
>> >> >> passed to Impala because it may not be specified in the right way to
>> >> >> accomplish what you're trying to do in "b".
>> >> >>
>> >> >> On Wed, Feb 7, 2018 at 4:32 PM, Fawze Abujaber <fa...@gmail.com>
>> >> >> wrote:
>> >> >> > Hi Matt,
>> >> >> >
>> >> >> > I created the pools manually and it’s completely nfigurations as
>> >> >> > shown
>> >> >> > in b.
>> >> >> >
>> >> >> > On Wed, 7 Feb 2018 at 23:17 Matthew Jacobs <ja...@gmail.com>
>> >> >> > wrote:
>> >> >> >>
>> >> >> >> How, more specifically, are you configuring these pools? Can you
>> >> >> >> share
>> >> >> >> the configuration files?
>> >> >> >>
>> >> >> >> On Wed, Feb 7, 2018 at 11:42 AM, Fawze Abujaber
>> >> >> >> <fa...@gmail.com>
>> >> >> >> wrote:
>> >> >> >> > Hello All,
>> >> >> >> >
>> >> >> >> > Will appreciate any insight of what i'm missing in my
>> >> >> >> > configuration
>> >> >> >> > to
>> >> >> >> > get
>> >> >> >> > the impala submission control working as expected ....
>> >> >> >> >
>> >> >> >> > a- Here is my placement Placement Rule:
>> >> >> >> > 1- Use the pool Specified at run time., only if the pool
>> >> >> >> > exists.
>> >> >> >> > 2- Use the pool root.[secondary group].
>> >> >> >> > 3- Use the pool root.default.
>> >> >> >> >
>> >> >> >> > b- Here is the impala resource pools
>> >> >> >> >
>> >> >> >> > Name Max Memory Max Running Queries Max
>> >> >> >> > Queued
>> >> >> >> > Queries
>> >> >> >> > Queue Timeout Default Query Memory Limit Access Control
>> >> >> >> > root.impiaas Unlimited Unlimited
>> >> >> >> > 200 1 minute(s) No Default
>> >> >> >> > Custom
>> >> >> >> > root.impreports Unlimited Unlimited
>> >> >> >> > 200 1 minute(s) No Default
>> >> >> >> > Custom
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > C- Here is my Access control for root.impiaas pool:
>> >> >> >> >
>> >> >> >> > Allow these users and groups to submit to this pool
>> >> >> >> >
>> >> >> >> > Users: fawzea
>> >> >> >> >
>> >> >> >> > The same for impreports the allowed user and groups to submit
>> >> >> >> > to
>> >> >> >> > pool
>> >> >> >> > is
>> >> >> >> > only nira.
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > LDAP groups for fawzea user:
>> >> >> >> >
>> >> >> >> > Member in groups:
>> >> >> >> >
>> >> >> >> > - dba
>> >> >> >> > - impiaas
>> >> >> >> > - mysql
>> >> >> >> >
>> >> >> >> > LDAP group for nira user
>> >> >> >> > Member in groups:
>> >> >> >> > - dba
>> >> >> >> > - impreports
>> >> >> >> > - mysql
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > Impala Queries UI shows:
>> >> >> >> >
>> >> >> >> > 1- select count(*) from dv_entersiteevent_prq_remote where
>> >> >> >> > year=2017
>> >> >> >> >
>> >> >> >> > User: fawzea
>> >> >> >> > Pool: root.impiaas
>> >> >> >> >
>> >> >> >> > 2- select count(*) from dv_entersiteevent_prq_remote where
>> >> >> >> > year=2017
>> >> >> >> > and
>> >> >> >> > month=10
>> >> >> >> > User: nira
>> >> >> >> > Pool: root.impreports
>> >> >> >> >
>> >> >> >> > As shown above that the queries submitted based on the
>> >> >> >> > secondary
>> >> >> >> > group
>> >> >> >> > rule.
>> >> >> >> >
>> >> >> >> > When i'm running the following with fawzea user:
>> >> >> >> >
>> >> >> >> > set request_pool=impreports;
>> >> >> >> > select count(*) from dv_entersiteevent_prq_remote where
>> >> >> >> > year=2017
>> >> >> >> > and
>> >> >> >> > month=10;
>> >> >> >> >
>> >> >> >> > I can see that the query is running with impreports pool.
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > What i'm missing here in my configuration to get this works as
>> >> >> >> > expected.
>> >> >> >> >
>> >> >
>> >> >
>> >
>> >
>
>
Re: impala submission control
Posted by Fawze Abujaber <fa...@gmail.com>.
Hi Matt,
If this is the situation, how we can explain that when i submit the query
it runs on my secondary group pool and when nira do the same it runs under
her secondary group pool.
More than this, the issue not related to LDAP group as i'm using the
submission restriction to a user and not a group.
On Thu, Feb 8, 2018 at 7:00 PM, Matthew Jacobs <ja...@gmail.com> wrote:
> As far as I can tell from this information, your configuration looks
> fine. The only other potential issue I can think of is that the
> secondary groups placement rule isn't picking up your LDAP groups
> properly. Does the ACL behavior work if you remove the secondary
> groups rule?
>
> On Wed, Feb 7, 2018 at 10:09 PM, Fawze Abujaber <fa...@gmail.com> wrote:
> > Hi Matt,
> >
> > v2.11.0-cdh5.14.0
> >
> > Query Runtime Profile:
> > Query (id=c843dae48ce9b289:6ad3ffc000000000):
> > Summary:
> > Session ID: ca420ac477772b56:51ae460ce9e44999
> > Session Type: BEESWAX
> > Start Time: 2018-02-08 01:02:01.023558000
> > End Time: 2018-02-08 01:02:01.501506000
> > Query Type: QUERY
> > Query State: FINISHED
> > Query Status: OK
> > Impala Version: impalad version 2.11.0-cdh5.14.0 RELEASE (build
> > d68206561bce6b26762d62c01a78e6cd27aa7690)
> > User: fawzea
> > Connected User: fawzea
> > Delegated User:
> > Network Address: ::ffff:10.16.144.131:50078
> > Default Db: default
> > Sql Statement: select count(*) from dv_entersiteevent_prq_remote
> where
> > year=2017 and month=10
> > Coordinator: aopr-dhc001.lpdomain.com:22000
> > Query Options (set by configuration): REQUEST_POOL=impreports
> > Query Options (set by configuration and planner):
> > REQUEST_POOL=impreports,MT_DOP=0
> > Plan:
> > ----------------
> > Max Per-Host Resource Reservation: Memory=0B
> > Per-Host Resource Estimates: Memory=21.00MB
> > WARNING: The following tables are missing relevant table and/or column
> > statistics.
> > default.dv_entersiteevent_prq_remote
> >
> > F01:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1
> > | Per-Host Resources: mem-estimate=10.00MB mem-reservation=0B
> > PLAN-ROOT SINK
> > | mem-estimate=0B mem-reservation=0B
> > |
> > 03:AGGREGATE [FINALIZE]
> > | output: count:merge(*)
> > | mem-estimate=10.00MB mem-reservation=0B spill-buffer=2.00MB
> > | tuple-ids=1 row-size=8B cardinality=1
> > |
> > 02:EXCHANGE [UNPARTITIONED]
> > | mem-estimate=0B mem-reservation=0B
> > | tuple-ids=1 row-size=8B cardinality=1
> > |
> > F00:PLAN FRAGMENT [RANDOM] hosts=5 instances=5
> > Per-Host Resources: mem-estimate=11.00MB mem-reservation=0B
> > 01:AGGREGATE
> > | output: sum_init_zero(default.dv_entersiteevent_prq_remote.
> parquet-stats:
> > num_rows)
> > | mem-estimate=10.00MB mem-reservation=0B spill-buffer=2.00MB
> > | tuple-ids=1 row-size=8B cardinality=1
> > |
> > 00:SCAN HDFS [default.dv_entersiteevent_prq_remote, RANDOM]
> > partitions=31/441 files=2292 size=1.39GB
> > stats-rows=unavailable extrapolated-rows=disabled
> > table stats: rows=unavailable size=unavailable
> > column stats: all
> > mem-estimate=1.00MB mem-reservation=0B
> > tuple-ids=0 row-size=8B cardinality=unavailable
> > ----------------
> > Estimated Per-Host Mem: 22020096
> > Tables Missing Stats: default.dv_entersiteevent_prq_remote
> > Per Host Min Reservation: aopr-dhc001.lpdomain.com:22000(0)
> > aopr-dhc002.lpdomain.com:22000(0) aopr-dhc003.lpdomain.com:22000(0)
> > aopr-dhc004.lpdomain.com:22000(0) aopr-dhc005.lpdomain.com:22000(0)
> > Request Pool: root.impreports
> > Admission result: Admitted immediately
> > ExecSummary:
> > Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem
> > Est. Peak Mem Detail
> > ------------------------------------------------------------
> ------------------------------------------------------------------
> > 03:AGGREGATE 1 222.958us 222.958us 1 1 36.00 KB
> > 10.00 MB FINALIZE
> > 02:EXCHANGE 1 357.508ms 357.508ms 5 1 0
> > 0 UNPARTITIONED
> > 01:AGGREGATE 5 1.027ms 1.541ms 5 1 20.00 KB
> > 10.00 MB
> > 00:SCAN HDFS 5 282.797ms 333.715ms 2.29K -1 2.62 MB
> > 1.00 MB default.dv_entersiteevent_prq_remote
> > Errors:
> > Planner Timeline: 13.508ms
> > - Analysis finished: 3.303ms (3.303ms)
> > - Value transfer graph computed: 3.465ms (162.667us)
> > - Single node plan created: 10.352ms (6.886ms)
> > - Runtime filters computed: 10.677ms (325.364us)
> > - Distributed plan created: 10.733ms (55.854us)
> > - Lineage info computed: 10.898ms (164.937us)
> > - Planning finished: 13.508ms (2.610ms)
> > Query Timeline: 481.950ms
> > - Query submitted: 79.971us (79.971us)
> > - Planning finished: 33.947ms (33.867ms)
> > - Submit for admission: 40.067ms (6.120ms)
> > - Completed admission: 40.197ms (130.146us)
> > - Ready to start on 5 backends: 40.905ms (707.852us)
> > - All 5 execution backends (6 fragment instances) started:
> 46.417ms
> > (5.512ms)
> > - Rows available: 424.548ms (378.131ms)
> > - First row fetched: 473.488ms (48.939ms)
> > - Unregister query: 477.951ms (4.462ms)
> > - ComputeScanRangeAssignmentTimer: 3.980ms
> > ImpalaServer:
> > - ClientFetchWaitTimer: 51.659ms
> > - RowMaterializationTimer: 1.726ms
> > Execution Profile c843dae48ce9b289:6ad3ffc000000000:(Total: 385.029ms,
> > non-child: 0.000ns, % non-child: 0.00%)
> > Number of filters: 0
> > Filter routing table:
> > ID Src. Node Tgt. Node(s) Target type Partition filter Pending
> > (Expected) First arrived Completed Enabled
> > ------------------------------------------------------------
> -------------------------------------------------------
> >
> > Backend startup latencies: Count: 5, min / max: 3ms / 4ms, 25th
> %-ile:
> > 3ms, 50th %-ile: 4ms, 75th %-ile: 4ms, 90th %-ile: 4ms, 95th %-ile: 4ms,
> > 99.9th %-ile: 4ms
> > Per Node Peak Memory Usage: aopr-dhc001.lpdomain.com:22000(2.68 MB)
> > aopr-dhc002.lpdomain.com:22000(2.54 MB) aopr-dhc003.lpdomain.com:22000
> (2.54
> > MB) aopr-dhc004.lpdomain.com:22000(2.41 MB)
> > aopr-dhc005.lpdomain.com:22000(2.56 MB)
> > - FiltersReceived: 0 (0)
> > - FinalizationTimer: 0.000ns
> > - NumBackends: 5 (5)
> > - NumFragmentInstances: 6 (6)
> > - NumFragments: 2 (2)
> > Averaged Fragment F01:(Total: 427.530ms, non-child: 49.428ms, %
> > non-child: 11.56%)
> > split sizes: min: 0, max: 0, avg: 0, stddev: 0
> > completion times: min:428.693ms max:428.693ms mean: 428.693ms
> > stddev:0.000ns
> > execution rates: min:0.00 /sec max:0.00 /sec mean:0.00 /sec
> > stddev:0.00 /sec
> > num instances: 1
> > - AverageThreadTokens: 0.00
> > - BloomFilterBytes: 0
> > - PeakMemoryUsage: 44.02 KB (45072)
> > - PeakReservation: 0
> > - PeakUsedReservation: 0
> > - PerHostPeakMemUsage: 2.68 MB (2806113)
> > - RowsProduced: 1 (1)
> > - TotalNetworkReceiveTime: 357.481ms
> > - TotalNetworkSendTime: 0.000ns
> > - TotalStorageWaitTime: 0.000ns
> > - TotalThreadsInvoluntaryContextSwitches: 0 (0)
> > - TotalThreadsTotalWallClockTime: 406.708ms
> > - TotalThreadsSysTime: 0.000ns
> > - TotalThreadsUserTime: 0.000ns
> > - TotalThreadsVoluntaryContextSwitches: 7 (7)
> > Fragment Instance Lifecycle Timings:
> > - ExecTime: 49.055ms
> > - ExecTreeExecTime: 4.251us
> > - OpenTime: 357.663ms
> > - ExecTreeOpenTime: 357.649ms
> > - PrepareTime: 20.764ms
> > - ExecTreePrepareTime: 81.759us
> > PLAN_ROOT_SINK:
> > - PeakMemoryUsage: 0
> > CodeGen:(Total: 20.369ms, non-child: 20.369ms, % non-child:
> 100.00%)
> > - CodegenTime: 0.000ns
> > - CompileTime: 0.000ns
> > - LoadTime: 0.000ns
> > - ModuleBitcodeSize: 1.86 MB (1950880)
> > - NumFunctions: 0 (0)
> > - NumInstructions: 0 (0)
> > - OptimizationTime: 0.000ns
> > - PeakMemoryUsage: 0
> > - PrepareTime: 19.685ms
> > AGGREGATION_NODE (id=3):(Total: 357.731ms, non-child: 222.958us, %
> > non-child: 0.06%)
> > - BuildTime: 7.368us
> > - GetResultsTime: 0.000ns
> > - HTResizeTime: 0.000ns
> > - HashBuckets: 0 (0)
> > - LargestPartitionPercent: 0 (0)
> > - MaxPartitionLevel: 0 (0)
> > - NumRepartitions: 0 (0)
> > - PartitionsCreated: 0 (0)
> > - PeakMemoryUsage: 36.00 KB (36864)
> > - RowsRepartitioned: 0 (0)
> > - RowsReturned: 1 (1)
> > - RowsReturnedRate: 2.00 /sec
> > - SpilledPartitions: 0 (0)
> > EXCHANGE_NODE (id=2):(Total: 357.508ms, non-child: 357.508ms, %
> > non-child: 100.00%)
> > - ConvertRowBatchTime: 8.881us
> > - PeakMemoryUsage: 0
> > - RowsReturned: 5 (5)
> > - RowsReturnedRate: 13.00 /sec
> > DataStreamReceiver:
> > - BytesReceived: 80.00 B (80)
> > - DeserializeRowBatchTimer: 24.165us
> > - FirstBatchArrivalWaitTime: 277.194ms
> > - PeakMemoryUsage: 4.02 KB (4112)
> > - SendersBlockedTimer: 0.000ns
> > - SendersBlockedTotalTimer(*): 0.000ns
> > Coordinator Fragment F01:
> > Instance c843dae48ce9b289:6ad3ffc000000000
> > (host=aopr-dhc001.lpdomain.com:22000):(Total: 427.530ms, non-child:
> > 49.428ms, % non-child: 11.56%)
> > - AverageThreadTokens: 0.00
> > - BloomFilterBytes: 0
> > - PeakMemoryUsage: 44.02 KB (45072)
> > - PeakReservation: 0
> > - PeakUsedReservation: 0
> > - PerHostPeakMemUsage: 2.68 MB (2806113)
> > - RowsProduced: 1 (1)
> > - TotalNetworkReceiveTime: 357.481ms
> > - TotalNetworkSendTime: 0.000ns
> > - TotalStorageWaitTime: 0.000ns
> > - TotalThreadsInvoluntaryContextSwitches: 0 (0)
> > - TotalThreadsTotalWallClockTime: 406.708ms
> > - TotalThreadsSysTime: 0.000ns
> > - TotalThreadsUserTime: 0.000ns
> > - TotalThreadsVoluntaryContextSwitches: 7 (7)
> > Fragment Instance Lifecycle Timings:
> > - ExecTime: 49.055ms
> > - ExecTreeExecTime: 4.251us
> > - OpenTime: 357.663ms
> > - ExecTreeOpenTime: 357.649ms
> > - PrepareTime: 20.764ms
> > - ExecTreePrepareTime: 81.759us
> > PLAN_ROOT_SINK:
> > - PeakMemoryUsage: 0
> > CodeGen:(Total: 20.369ms, non-child: 20.369ms, % non-child:
> 100.00%)
> > - CodegenTime: 0.000ns
> > - CompileTime: 0.000ns
> > - LoadTime: 0.000ns
> > - ModuleBitcodeSize: 1.86 MB (1950880)
> > - NumFunctions: 0 (0)
> > - NumInstructions: 0 (0)
> > - OptimizationTime: 0.000ns
> > - PeakMemoryUsage: 0
> > - PrepareTime: 19.685ms
> > AGGREGATION_NODE (id=3):(Total: 357.731ms, non-child: 222.958us,
> %
> > non-child: 0.06%)
> > - BuildTime: 7.368us
> > - GetResultsTime: 0.000ns
> > - HTResizeTime: 0.000ns
> > - HashBuckets: 0 (0)
> > - LargestPartitionPercent: 0 (0)
> > - MaxPartitionLevel: 0 (0)
> > - NumRepartitions: 0 (0)
> > - PartitionsCreated: 0 (0)
> > - PeakMemoryUsage: 36.00 KB (36864)
> > - RowsRepartitioned: 0 (0)
> > - RowsReturned: 1 (1)
> > - RowsReturnedRate: 2.00 /sec
> > - SpilledPartitions: 0 (0)
> > EXCHANGE_NODE (id=2):(Total: 357.508ms, non-child: 357.508ms, %
> > non-child: 100.00%)
> > - ConvertRowBatchTime: 8.881us
> > - PeakMemoryUsage: 0
> > - RowsReturned: 5 (5)
> > - RowsReturnedRate: 13.00 /sec
> > DataStreamReceiver:
> > - BytesReceived: 80.00 B (80)
> > - DeserializeRowBatchTimer: 24.165us
> > - FirstBatchArrivalWaitTime: 277.194ms
> > - PeakMemoryUsage: 4.02 KB (4112)
> > - SendersBlockedTimer: 0.000ns
> > - SendersBlockedTotalTimer(*): 0.000ns
> > Averaged Fragment F00:(Total: 327.360ms, non-child: 1.578ms, %
> > non-child: 0.48%)
> > split sizes: min: 283.31 MB, max: 284.33 MB, avg: 284.04 MB,
> stddev:
> > 382.00 KB
> > completion times: min:299.457ms max:378.950ms mean: 328.881ms
> > stddev:31.507ms
> > execution rates: min:750.18 MB/sec max:949.01 MB/sec mean:871.27
> > MB/sec stddev:79.74 MB/sec
> > num instances: 5
> > - AverageThreadTokens: 15.00
> > - BloomFilterBytes: 0
> > - PeakMemoryUsage: 2.54 MB (2666852)
> > - PeakReservation: 0
> > - PeakUsedReservation: 0
> > - PerHostPeakMemUsage: 2.55 MB (2669309)
> > - RowsProduced: 1 (1)
> > - TotalNetworkReceiveTime: 0.000ns
> > - TotalNetworkSendTime: 303.541us
> > - TotalStorageWaitTime: 4s821ms
> > - TotalThreadsInvoluntaryContextSwitches: 104 (104)
> > - TotalThreadsTotalWallClockTime: 6s835ms
> > - TotalThreadsSysTime: 31.784ms
> > - TotalThreadsUserTime: 682.084ms
> > - TotalThreadsVoluntaryContextSwitches: 2.12K (2121)
> > Fragment Instance Lifecycle Timings:
> > - ExecTime: 500.816us
> > - ExecTreeExecTime: 8.200us
> > - OpenTime: 302.848ms
> > - ExecTreeOpenTime: 281.309ms
> > - PrepareTime: 23.946ms
> > - ExecTreePrepareTime: 2.513ms
> > DataStreamSender (dst_id=2):(Total: 269.902us, non-child:
> 269.902us, %
> > non-child: 100.00%)
> > - BytesSent: 16.00 B (16)
> > - NetworkThroughput(*): 373.13 KB/sec
> > - OverallThroughput: 63.53 KB/sec
> > - PeakMemoryUsage: 16.00 KB (16384)
> > - RowsReturned: 1 (1)
> > - SerializeBatchTime: 14.525us
> > - TransmitDataRPCTime: 43.122us
> > - UncompressedRowBatchSize: 16.00 B (16)
> > CodeGen:(Total: 41.687ms, non-child: 41.687ms, % non-child:
> 100.00%)
> > - CodegenTime: 1.194ms
> > - CompileTime: 7.360ms
> > - LoadTime: 0.000ns
> > - ModuleBitcodeSize: 1.86 MB (1950880)
> > - NumFunctions: 15 (15)
> > - NumInstructions: 214 (214)
> > - OptimizationTime: 13.529ms
> > - PeakMemoryUsage: 107.00 KB (109568)
> > - PrepareTime: 20.084ms
> > AGGREGATION_NODE (id=1):(Total: 283.825ms, non-child: 1.027ms, %
> > non-child: 0.36%)
> > - BuildTime: 1.259ms
> > - GetResultsTime: 0.000ns
> > - HTResizeTime: 0.000ns
> > - HashBuckets: 0 (0)
> > - LargestPartitionPercent: 0 (0)
> > - MaxPartitionLevel: 0 (0)
> > - NumRepartitions: 0 (0)
> > - PartitionsCreated: 0 (0)
> > - PeakMemoryUsage: 20.00 KB (20480)
> > - RowsRepartitioned: 0 (0)
> > - RowsReturned: 1 (1)
> > - RowsReturnedRate: 2.00 /sec
> > - SpilledPartitions: 0 (0)
> > HDFS_SCAN_NODE (id=0):(Total: 282.797ms, non-child: 282.797ms, %
> > non-child: 100.00%)
> > - AverageHdfsReadThreadConcurrency: 0.00
> > - AverageScannerThreadConcurrency: 14.40
> > - BytesRead: 44.52 MB (46686028)
> > - BytesReadDataNodeCache: 0
> > - BytesReadLocal: 44.52 MB (46686028)
> > - BytesReadRemoteUnexpected: 0
> > - BytesReadShortCircuit: 44.52 MB (46686028)
> > - CachedFileHandlesHitCount: 0 (0)
> > - CachedFileHandlesMissCount: 458 (458)
> > - CollectionItemsRead: 0 (0)
> > - DecompressionTime: 0.000ns
> > - MaxCompressedTextFileLength: 0
> > - NumColumns: 0 (0)
> > - NumDictFilteredRowGroups: 0 (0)
> > - NumDisksAccessed: 12 (12)
> > - NumRowGroups: 458 (458)
> > - NumScannerThreadsStarted: 24 (24)
> > - NumScannersWithNoReads: 0 (0)
> > - NumStatsFilteredRowGroups: 0 (0)
> > - PeakMemoryUsage: 2.50 MB (2625539)
> > - PerReadThreadRawHdfsThroughput: 1.29 GB/sec
> > - RemoteScanRanges: 0 (0)
> > - RowBatchQueueGetWaitTime: 269.787ms
> > - RowBatchQueuePutWaitTime: 0.000ns
> > - RowsRead: 0 (0)
> > - RowsReturned: 458 (458)
> > - RowsReturnedRate: 1.64 K/sec
> > - ScanRangesComplete: 458 (458)
> > - ScannerThreadsInvoluntaryContextSwitches: 99 (99)
> > - ScannerThreadsTotalWallClockTime: 6s532ms
> > - MaterializeTupleTime(*): 70.664us
> > - ScannerThreadsSysTime: 26.184ms
> > - ScannerThreadsUserTime: 650.089ms
> > - ScannerThreadsVoluntaryContextSwitches: 1.67K (1674)
> > - TotalRawHdfsReadTime(*): 34.416ms
> > - TotalReadThroughput: 36.86 MB/sec
> > Fragment F00:
> > Instance c843dae48ce9b289:6ad3ffc000000004
> > (host=aopr-dhc004.lpdomain.com:22000):(Total: 377.384ms, non-child:
> 1.619ms,
> > % non-child: 0.43%)
> > Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> > 0:36/21.95 MB 11:33/19.87 MB 9:27/15.16 MB 5:44/28.07 MB 6:56/33.32 MB
> > 1:38/22.24 MB 10:44/27.90 MB 3:34/20.26 MB 2:45/28.21 MB 7:31/19.33 MB
> > 4:42/26.73 MB 8:34/21.25 MB
> > MemoryUsage(500.000ms): 1.91 MB
> > ThreadUsage(500.000ms): 25
> > - AverageThreadTokens: 25.00
> > - BloomFilterBytes: 0
> > - PeakMemoryUsage: 2.41 MB (2531689)
> > - PeakReservation: 0
> > - PeakUsedReservation: 0
> > - PerHostPeakMemUsage: 2.41 MB (2531689)
> > - RowsProduced: 1 (1)
> > - TotalNetworkReceiveTime: 0.000ns
> > - TotalNetworkSendTime: 382.629us
> > - TotalStorageWaitTime: 5s790ms
> > - TotalThreadsInvoluntaryContextSwitches: 115 (115)
> > - TotalThreadsTotalWallClockTime: 8s122ms
> > - TotalThreadsSysTime: 33.984ms
> > - TotalThreadsUserTime: 702.879ms
> > - TotalThreadsVoluntaryContextSwitches: 2.04K (2037)
> > Fragment Instance Lifecycle Timings:
> > - ExecTime: 604.005us
> > - ExecTreeExecTime: 8.338us
> > - OpenTime: 352.927ms
> > - ExecTreeOpenTime: 331.921ms
> > - PrepareTime: 23.783ms
> > - ExecTreePrepareTime: 2.559ms
> > DataStreamSender (dst_id=2):(Total: 331.818us, non-child:
> 331.818us,
> > % non-child: 100.00%)
> > - BytesSent: 16.00 B (16)
> > - NetworkThroughput(*): 289.00 KB/sec
> > - OverallThroughput: 47.09 KB/sec
> > - PeakMemoryUsage: 16.00 KB (16384)
> > - RowsReturned: 1 (1)
> > - SerializeBatchTime: 14.466us
> > - TransmitDataRPCTime: 54.066us
> > - UncompressedRowBatchSize: 16.00 B (16)
> > CodeGen:(Total: 40.950ms, non-child: 40.950ms, % non-child:
> 100.00%)
> > - CodegenTime: 1.122ms
> > - CompileTime: 7.318ms
> > - LoadTime: 0.000ns
> > - ModuleBitcodeSize: 1.86 MB (1950880)
> > - NumFunctions: 15 (15)
> > - NumInstructions: 214 (214)
> > - OptimizationTime: 13.106ms
> > - PeakMemoryUsage: 107.00 KB (109568)
> > - PrepareTime: 19.773ms
> > AGGREGATION_NODE (id=1):(Total: 334.482ms, non-child: 766.527us,
> %
> > non-child: 0.23%)
> > ExecOption: Codegen Enabled
> > - BuildTime: 1.192ms
> > - GetResultsTime: 0.000ns
> > - HTResizeTime: 0.000ns
> > - HashBuckets: 0 (0)
> > - LargestPartitionPercent: 0 (0)
> > - MaxPartitionLevel: 0 (0)
> > - NumRepartitions: 0 (0)
> > - PartitionsCreated: 0 (0)
> > - PeakMemoryUsage: 20.00 KB (20480)
> > - RowsRepartitioned: 0 (0)
> > - RowsReturned: 1 (1)
> > - RowsReturnedRate: 2.00 /sec
> > - SpilledPartitions: 0 (0)
> > HDFS_SCAN_NODE (id=0):(Total: 333.715ms, non-child: 333.715ms, %
> > non-child: 100.00%)
> > Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> > 0:36/21.95 MB 11:33/19.87 MB 9:27/15.16 MB 5:44/28.07 MB 6:56/33.32 MB
> > 1:38/22.24 MB 10:44/27.90 MB 3:34/20.26 MB 2:45/28.21 MB 7:31/19.33 MB
> > 4:42/26.73 MB 8:34/21.25 MB
> > ExecOption: PARQUET Codegen Enabled, Codegen enabled: 464 out
> of
> > 464
> > Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0%
> > 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
> > File Formats: PARQUET/Unknown(Skipped):464
> > BytesRead(500.000ms): 12.27 MB
> > - FooterProcessingTime: (Avg: 13.555ms ; Min: 5.433ms ; Max:
> > 62.957ms ; Number of samples: 464)
> > - AverageHdfsReadThreadConcurrency: 0.00
> > - AverageScannerThreadConcurrency: 24.00
> > - BytesRead: 45.01 MB (47193988)
> > - BytesReadDataNodeCache: 0
> > - BytesReadLocal: 45.01 MB (47193988)
> > - BytesReadRemoteUnexpected: 0
> > - BytesReadShortCircuit: 45.01 MB (47193988)
> > - CachedFileHandlesHitCount: 0 (0)
> > - CachedFileHandlesMissCount: 464 (464)
> > - CollectionItemsRead: 0 (0)
> > - DecompressionTime: 0.000ns
> > - MaxCompressedTextFileLength: 0
> > - NumColumns: 0 (0)
> > - NumDictFilteredRowGroups: 0 (0)
> > - NumDisksAccessed: 12 (12)
> > - NumRowGroups: 464 (464)
> > - NumScannerThreadsStarted: 24 (24)
> > - NumScannersWithNoReads: 0 (0)
> > - NumStatsFilteredRowGroups: 0 (0)
> > - PeakMemoryUsage: 2.38 MB (2490376)
> > - PerReadThreadRawHdfsThroughput: 1016.44 MB/sec
> > - RemoteScanRanges: 0 (0)
> > - RowBatchQueueGetWaitTime: 320.860ms
> > - RowBatchQueuePutWaitTime: 0.000ns
> > - RowsRead: 0 (0)
> > - RowsReturned: 464 (464)
> > - RowsReturnedRate: 1.39 K/sec
> > - ScanRangesComplete: 464 (464)
> > - ScannerThreadsInvoluntaryContextSwitches: 108 (108)
> > - ScannerThreadsTotalWallClockTime: 7s769ms
> > - MaterializeTupleTime(*): 75.514us
> > - ScannerThreadsSysTime: 29.984ms
> > - ScannerThreadsUserTime: 669.884ms
> > - ScannerThreadsVoluntaryContextSwitches: 1.60K (1597)
> > - TotalRawHdfsReadTime(*): 44.279ms
> > - TotalReadThroughput: 24.53 MB/sec
> > Instance c843dae48ce9b289:6ad3ffc000000002
> > (host=aopr-dhc003.lpdomain.com:22000):(Total: 351.014ms, non-child:
> 1.641ms,
> > % non-child: 0.47%)
> > Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> > 0:35/20.89 MB 7:44/26.64 MB 11:27/17.21 MB 9:32/18.61 MB 5:25/14.56 MB
> > 2:38/24.32 MB 6:31/19.85 MB 8:50/30.76 MB 10:41/25.92 MB 3:31/19.78 MB
> > 4:60/37.59 MB 1:44/27.19 MB
> > MemoryUsage(500.000ms): 1.16 MB
> > ThreadUsage(500.000ms): 25
> > - AverageThreadTokens: 25.00
> > - BloomFilterBytes: 0
> > - PeakMemoryUsage: 2.54 MB (2662753)
> > - PeakReservation: 0
> > - PeakUsedReservation: 0
> > - PerHostPeakMemUsage: 2.54 MB (2662753)
> > - RowsProduced: 1 (1)
> > - TotalNetworkReceiveTime: 0.000ns
> > - TotalNetworkSendTime: 342.746us
> > - TotalStorageWaitTime: 4s796ms
> > - TotalThreadsInvoluntaryContextSwitches: 93 (93)
> > - TotalThreadsTotalWallClockTime: 7s286ms
> > - TotalThreadsSysTime: 27.988ms
> > - TotalThreadsUserTime: 671.887ms
> > - TotalThreadsVoluntaryContextSwitches: 2.08K (2079)
> > Fragment Instance Lifecycle Timings:
> > - ExecTime: 524.006us
> > - ExecTreeExecTime: 7.920us
> > - OpenTime: 327.802ms
> > - ExecTreeOpenTime: 307.154ms
> > - PrepareTime: 22.617ms
> > - ExecTreePrepareTime: 2.650ms
> > DataStreamSender (dst_id=2):(Total: 348.990us, non-child:
> 348.990us,
> > % non-child: 100.00%)
> > - BytesSent: 16.00 B (16)
> > - NetworkThroughput(*): 354.47 KB/sec
> > - OverallThroughput: 44.77 KB/sec
> > - PeakMemoryUsage: 16.00 KB (16384)
> > - RowsReturned: 1 (1)
> > - SerializeBatchTime: 15.602us
> > - TransmitDataRPCTime: 44.080us
> > - UncompressedRowBatchSize: 16.00 B (16)
> > CodeGen:(Total: 39.215ms, non-child: 39.215ms, % non-child:
> 100.00%)
> > - CodegenTime: 1.183ms
> > - CompileTime: 7.168ms
> > - LoadTime: 0.000ns
> > - ModuleBitcodeSize: 1.86 MB (1950880)
> > - NumFunctions: 15 (15)
> > - NumInstructions: 214 (214)
> > - OptimizationTime: 12.929ms
> > - PeakMemoryUsage: 107.00 KB (109568)
> > - PrepareTime: 18.453ms
> > AGGREGATION_NODE (id=1):(Total: 309.807ms, non-child: 948.677us,
> %
> > non-child: 0.31%)
> > ExecOption: Codegen Enabled
> > - BuildTime: 1.552ms
> > - GetResultsTime: 0.000ns
> > - HTResizeTime: 0.000ns
> > - HashBuckets: 0 (0)
> > - LargestPartitionPercent: 0 (0)
> > - MaxPartitionLevel: 0 (0)
> > - NumRepartitions: 0 (0)
> > - PartitionsCreated: 0 (0)
> > - PeakMemoryUsage: 20.00 KB (20480)
> > - RowsRepartitioned: 0 (0)
> > - RowsReturned: 1 (1)
> > - RowsReturnedRate: 3.00 /sec
> > - SpilledPartitions: 0 (0)
> > HDFS_SCAN_NODE (id=0):(Total: 308.859ms, non-child: 308.859ms, %
> > non-child: 100.00%)
> > Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> > 0:35/20.89 MB 7:44/26.64 MB 11:27/17.21 MB 9:32/18.61 MB 5:25/14.56 MB
> > 2:38/24.32 MB 6:31/19.85 MB 8:50/30.76 MB 10:41/25.92 MB 3:31/19.78 MB
> > 4:60/37.59 MB 1:44/27.19 MB
> > ExecOption: PARQUET Codegen Enabled, Codegen enabled: 458 out
> of
> > 458
> > Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0%
> > 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
> > File Formats: PARQUET/Unknown(Skipped):458
> > BytesRead(500.000ms): 36.74 MB
> > - FooterProcessingTime: (Avg: 11.502ms ; Min: 3.668ms ; Max:
> > 23.564ms ; Number of samples: 458)
> > - AverageHdfsReadThreadConcurrency: 0.00
> > - AverageScannerThreadConcurrency: 24.00
> > - BytesRead: 44.51 MB (46668432)
> > - BytesReadDataNodeCache: 0
> > - BytesReadLocal: 44.51 MB (46668432)
> > - BytesReadRemoteUnexpected: 0
> > - BytesReadShortCircuit: 44.51 MB (46668432)
> > - CachedFileHandlesHitCount: 0 (0)
> > - CachedFileHandlesMissCount: 458 (458)
> > - CollectionItemsRead: 0 (0)
> > - DecompressionTime: 0.000ns
> > - MaxCompressedTextFileLength: 0
> > - NumColumns: 0 (0)
> > - NumDictFilteredRowGroups: 0 (0)
> > - NumDisksAccessed: 12 (12)
> > - NumRowGroups: 458 (458)
> > - NumScannerThreadsStarted: 24 (24)
> > - NumScannersWithNoReads: 0 (0)
> > - NumStatsFilteredRowGroups: 0 (0)
> > - PeakMemoryUsage: 2.50 MB (2621440)
> > - PerReadThreadRawHdfsThroughput: 1.26 GB/sec
> > - RemoteScanRanges: 0 (0)
> > - RowBatchQueueGetWaitTime: 295.595ms
> > - RowBatchQueuePutWaitTime: 0.000ns
> > - RowsRead: 0 (0)
> > - RowsReturned: 458 (458)
> > - RowsReturnedRate: 1.48 K/sec
> > - ScanRangesComplete: 458 (458)
> > - ScannerThreadsInvoluntaryContextSwitches: 91 (91)
> > - ScannerThreadsTotalWallClockTime: 6s958ms
> > - MaterializeTupleTime(*): 71.630us
> > - ScannerThreadsSysTime: 16.989ms
> > - ScannerThreadsUserTime: 645.891ms
> > - ScannerThreadsVoluntaryContextSwitches: 1.63K (1630)
> > - TotalRawHdfsReadTime(*): 34.377ms
> > - TotalReadThroughput: 73.49 MB/sec
> > Instance c843dae48ce9b289:6ad3ffc000000001
> > (host=aopr-dhc005.lpdomain.com:22000):(Total: 308.195ms, non-child:
> 1.784ms,
> > % non-child: 0.58%)
> > Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> > 2:46/28.97 MB 8:39/24.34 MB 10:33/21.03 MB 3:43/26.50 MB 5:32/20.27 MB
> > 9:32/19.86 MB 6:25/16.01 MB 7:41/24.77 MB 1:42/25.01 MB 0:55/32.51 MB
> > 4:40/24.29 MB 11:33/20.77 MB
> > - AverageThreadTokens: 0.00
> > - BloomFilterBytes: 0
> > - PeakMemoryUsage: 2.56 MB (2683241)
> > - PeakReservation: 0
> > - PeakUsedReservation: 0
> > - PerHostPeakMemUsage: 2.56 MB (2683241)
> > - RowsProduced: 1 (1)
> > - TotalNetworkReceiveTime: 0.000ns
> > - TotalNetworkSendTime: 292.765us
> > - TotalStorageWaitTime: 4s482ms
> > - TotalThreadsInvoluntaryContextSwitches: 113 (113)
> > - TotalThreadsTotalWallClockTime: 6s310ms
> > - TotalThreadsSysTime: 35.983ms
> > - TotalThreadsUserTime: 682.884ms
> > - TotalThreadsVoluntaryContextSwitches: 2.14K (2143)
> > Fragment Instance Lifecycle Timings:
> > - ExecTime: 532.572us
> > - ExecTreeExecTime: 8.377us
> > - OpenTime: 280.241ms
> > - ExecTreeOpenTime: 258.652ms
> > - PrepareTime: 27.342ms
> > - ExecTreePrepareTime: 2.863ms
> > DataStreamSender (dst_id=2):(Total: 309.794us, non-child:
> 309.794us,
> > % non-child: 100.00%)
> > - BytesSent: 16.00 B (16)
> > - NetworkThroughput(*): 356.91 KB/sec
> > - OverallThroughput: 50.44 KB/sec
> > - PeakMemoryUsage: 16.00 KB (16384)
> > - RowsReturned: 1 (1)
> > - SerializeBatchTime: 14.262us
> > - TransmitDataRPCTime: 43.778us
> > - UncompressedRowBatchSize: 16.00 B (16)
> > CodeGen:(Total: 44.584ms, non-child: 44.584ms, % non-child:
> 100.00%)
> > - CodegenTime: 1.382ms
> > - CompileTime: 7.346ms
> > - LoadTime: 0.000ns
> > - ModuleBitcodeSize: 1.86 MB (1950880)
> > - NumFunctions: 15 (15)
> > - NumInstructions: 214 (214)
> > - OptimizationTime: 13.462ms
> > - PeakMemoryUsage: 107.00 KB (109568)
> > - PrepareTime: 23.006ms
> > AGGREGATION_NODE (id=1):(Total: 261.516ms, non-child: 836.656us,
> %
> > non-child: 0.32%)
> > ExecOption: Codegen Enabled
> > - BuildTime: 1.280ms
> > - GetResultsTime: 0.000ns
> > - HTResizeTime: 0.000ns
> > - HashBuckets: 0 (0)
> > - LargestPartitionPercent: 0 (0)
> > - MaxPartitionLevel: 0 (0)
> > - NumRepartitions: 0 (0)
> > - PartitionsCreated: 0 (0)
> > - PeakMemoryUsage: 20.00 KB (20480)
> > - RowsRepartitioned: 0 (0)
> > - RowsReturned: 1 (1)
> > - RowsReturnedRate: 3.00 /sec
> > - SpilledPartitions: 0 (0)
> > HDFS_SCAN_NODE (id=0):(Total: 260.680ms, non-child: 260.680ms, %
> > non-child: 100.00%)
> > Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> > 2:46/28.97 MB 8:39/24.34 MB 10:33/21.03 MB 3:43/26.50 MB 5:32/20.27 MB
> > 9:32/19.86 MB 6:25/16.01 MB 7:41/24.77 MB 1:42/25.01 MB 0:55/32.51 MB
> > 4:40/24.29 MB 11:33/20.77 MB
> > ExecOption: PARQUET Codegen Enabled, Codegen enabled: 461 out
> of
> > 461
> > Hdfs Read Thread Concurrency Bucket: 0:0% 1:0% 2:0% 3:0% 4:0%
> 5:0%
> > 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
> > File Formats: PARQUET/Unknown(Skipped):461
> > - FooterProcessingTime: (Avg: 10.793ms ; Min: 3.087ms ; Max:
> > 20.253ms ; Number of samples: 461)
> > - AverageHdfsReadThreadConcurrency: 0.00
> > - AverageScannerThreadConcurrency: 0.00
> > - BytesRead: 44.73 MB (46906407)
> > - BytesReadDataNodeCache: 0
> > - BytesReadLocal: 44.73 MB (46906407)
> > - BytesReadRemoteUnexpected: 0
> > - BytesReadShortCircuit: 44.73 MB (46906407)
> > - CachedFileHandlesHitCount: 0 (0)
> > - CachedFileHandlesMissCount: 461 (461)
> > - CollectionItemsRead: 0 (0)
> > - DecompressionTime: 0.000ns
> > - MaxCompressedTextFileLength: 0
> > - NumColumns: 0 (0)
> > - NumDictFilteredRowGroups: 0 (0)
> > - NumDisksAccessed: 12 (12)
> > - NumRowGroups: 461 (461)
> > - NumScannerThreadsStarted: 24 (24)
> > - NumScannersWithNoReads: 0 (0)
> > - NumStatsFilteredRowGroups: 0 (0)
> > - PeakMemoryUsage: 2.52 MB (2641928)
> > - PerReadThreadRawHdfsThroughput: 1.31 GB/sec
> > - RemoteScanRanges: 0 (0)
> > - RowBatchQueueGetWaitTime: 246.691ms
> > - RowBatchQueuePutWaitTime: 0.000ns
> > - RowsRead: 0 (0)
> > - RowsReturned: 461 (461)
> > - RowsReturnedRate: 1.77 K/sec
> > - ScanRangesComplete: 461 (461)
> > - ScannerThreadsInvoluntaryContextSwitches: 109 (109)
> > - ScannerThreadsTotalWallClockTime: 6s030ms
> > - MaterializeTupleTime(*): 68.735us
> > - ScannerThreadsSysTime: 33.983ms
> > - ScannerThreadsUserTime: 647.889ms
> > - ScannerThreadsVoluntaryContextSwitches: 1.70K (1696)
> > - TotalRawHdfsReadTime(*): 33.405ms
> > - TotalReadThroughput: 0.00 /sec
> > Instance c843dae48ce9b289:6ad3ffc000000003
> > (host=aopr-dhc002.lpdomain.com:22000):(Total: 302.116ms, non-child:
> 1.604ms,
> > % non-child: 0.53%)
> > Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> > 0:27/16.21 MB 11:33/21.23 MB 6:36/23.15 MB 10:41/25.09 MB 1:41/24.13 MB
> > 9:32/18.79 MB 5:41/25.31 MB 2:42/26.49 MB 7:36/22.87 MB 3:51/31.00 MB
> > 4:41/25.99 MB 8:38/23.79 MB
> > MemoryUsage(500.000ms): 700.35 KB
> > ThreadUsage(500.000ms): 25
> > - AverageThreadTokens: 25.00
> > - BloomFilterBytes: 0
> > - PeakMemoryUsage: 2.54 MB (2662753)
> > - PeakReservation: 0
> > - PeakUsedReservation: 0
> > - PerHostPeakMemUsage: 2.54 MB (2662753)
> > - RowsProduced: 1 (1)
> > - TotalNetworkReceiveTime: 0.000ns
> > - TotalNetworkSendTime: 300.675us
> > - TotalStorageWaitTime: 4s661ms
> > - TotalThreadsInvoluntaryContextSwitches: 90 (90)
> > - TotalThreadsTotalWallClockTime: 6s256ms
> > - TotalThreadsSysTime: 30.983ms
> > - TotalThreadsUserTime: 709.880ms
> > - TotalThreadsVoluntaryContextSwitches: 2.17K (2175)
> > Fragment Instance Lifecycle Timings:
> > - ExecTime: 492.627us
> > - ExecTreeExecTime: 8.191us
> > - OpenTime: 277.922ms
> > - ExecTreeOpenTime: 255.395ms
> > - PrepareTime: 23.647ms
> > - ExecTreePrepareTime: 2.243ms
> > DataStreamSender (dst_id=2):(Total: 194.517us, non-child:
> 194.517us,
> > % non-child: 100.00%)
> > - BytesSent: 16.00 B (16)
> > - NetworkThroughput(*): 371.86 KB/sec
> > - OverallThroughput: 80.33 KB/sec
> > - PeakMemoryUsage: 16.00 KB (16384)
> > - RowsReturned: 1 (1)
> > - SerializeBatchTime: 14.035us
> > - TransmitDataRPCTime: 42.019us
> > - UncompressedRowBatchSize: 16.00 B (16)
> > CodeGen:(Total: 42.676ms, non-child: 42.676ms, % non-child:
> 100.00%)
> > - CodegenTime: 1.236ms
> > - CompileTime: 7.717ms
> > - LoadTime: 0.000ns
> > - ModuleBitcodeSize: 1.86 MB (1950880)
> > - NumFunctions: 15 (15)
> > - NumInstructions: 214 (214)
> > - OptimizationTime: 14.130ms
> > - PeakMemoryUsage: 107.00 KB (109568)
> > - PrepareTime: 20.109ms
> > AGGREGATION_NODE (id=1):(Total: 257.640ms, non-child: 1.541ms, %
> > non-child: 0.60%)
> > ExecOption: Codegen Enabled
> > - BuildTime: 1.252ms
> > - GetResultsTime: 0.000ns
> > - HTResizeTime: 0.000ns
> > - HashBuckets: 0 (0)
> > - LargestPartitionPercent: 0 (0)
> > - MaxPartitionLevel: 0 (0)
> > - NumRepartitions: 0 (0)
> > - PartitionsCreated: 0 (0)
> > - PeakMemoryUsage: 20.00 KB (20480)
> > - RowsRepartitioned: 0 (0)
> > - RowsReturned: 1 (1)
> > - RowsReturnedRate: 3.00 /sec
> > - SpilledPartitions: 0 (0)
> > HDFS_SCAN_NODE (id=0):(Total: 256.099ms, non-child: 256.099ms, %
> > non-child: 100.00%)
> > Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> > 0:27/16.21 MB 11:33/21.23 MB 6:36/23.15 MB 10:41/25.09 MB 1:41/24.13 MB
> > 9:32/18.79 MB 5:41/25.31 MB 2:42/26.49 MB 7:36/22.87 MB 3:51/31.00 MB
> > 4:41/25.99 MB 8:38/23.79 MB
> > ExecOption: PARQUET Codegen Enabled, Codegen enabled: 459 out
> of
> > 459
> > Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0%
> > 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
> > File Formats: PARQUET/Unknown(Skipped):459
> > BytesRead(500.000ms): 43.15 MB
> > - FooterProcessingTime: (Avg: 11.257ms ; Min: 3.989ms ; Max:
> > 23.004ms ; Number of samples: 459)
> > - AverageHdfsReadThreadConcurrency: 0.00
> > - AverageScannerThreadConcurrency: 24.00
> > - BytesRead: 44.52 MB (46678031)
> > - BytesReadDataNodeCache: 0
> > - BytesReadLocal: 44.52 MB (46678031)
> > - BytesReadRemoteUnexpected: 0
> > - BytesReadShortCircuit: 44.52 MB (46678031)
> > - CachedFileHandlesHitCount: 0 (0)
> > - CachedFileHandlesMissCount: 459 (459)
> > - CollectionItemsRead: 0 (0)
> > - DecompressionTime: 0.000ns
> > - MaxCompressedTextFileLength: 0
> > - NumColumns: 0 (0)
> > - NumDictFilteredRowGroups: 0 (0)
> > - NumDisksAccessed: 12 (12)
> > - NumRowGroups: 459 (459)
> > - NumScannerThreadsStarted: 24 (24)
> > - NumScannersWithNoReads: 0 (0)
> > - NumStatsFilteredRowGroups: 0 (0)
> > - PeakMemoryUsage: 2.50 MB (2621440)
> > - PerReadThreadRawHdfsThroughput: 1.33 GB/sec
> > - RemoteScanRanges: 0 (0)
> > - RowBatchQueueGetWaitTime: 243.426ms
> > - RowBatchQueuePutWaitTime: 0.000ns
> > - RowsRead: 0 (0)
> > - RowsReturned: 459 (459)
> > - RowsReturnedRate: 1.79 K/sec
> > - ScanRangesComplete: 459 (459)
> > - ScannerThreadsInvoluntaryContextSwitches: 85 (85)
> > - ScannerThreadsTotalWallClockTime: 5s978ms
> > - MaterializeTupleTime(*): 67.997us
> > - ScannerThreadsSysTime: 22.984ms
> > - ScannerThreadsUserTime: 678.884ms
> > - ScannerThreadsVoluntaryContextSwitches: 1.72K (1724)
> > - TotalRawHdfsReadTime(*): 32.626ms
> > - TotalReadThroughput: 86.30 MB/sec
> > Instance c843dae48ce9b289:6ad3ffc000000005
> > (host=aopr-dhc001.lpdomain.com:22000):(Total: 298.093ms, non-child:
> 1.242ms,
> > % non-child: 0.42%)
> > Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> > 9:55/35.38 MB 0:37/24.23 MB 5:37/23.31 MB 11:22/13.79 MB 8:30/18.53 MB
> > 4:33/20.45 MB 1:38/24.06 MB 10:29/18.40 MB 3:39/24.35 MB 2:41/25.45 MB
> > 6:42/27.03 MB 7:47/29.22 MB
> > - AverageThreadTokens: 0.00
> > - BloomFilterBytes: 0
> > - PeakMemoryUsage: 2.66 MB (2793825)
> > - PeakReservation: 0
> > - PeakUsedReservation: 0
> > - PerHostPeakMemUsage: 2.68 MB (2806113)
> > - RowsProduced: 1 (1)
> > - TotalNetworkReceiveTime: 0.000ns
> > - TotalNetworkSendTime: 198.894us
> > - TotalStorageWaitTime: 4s374ms
> > - TotalThreadsInvoluntaryContextSwitches: 110 (110)
> > - TotalThreadsTotalWallClockTime: 6s201ms
> > - TotalThreadsSysTime: 29.983ms
> > - TotalThreadsUserTime: 642.892ms
> > - TotalThreadsVoluntaryContextSwitches: 2.17K (2174)
> > Fragment Instance Lifecycle Timings:
> > - ExecTime: 350.873us
> > - ExecTreeExecTime: 8.173us
> > - OpenTime: 275.347ms
> > - ExecTreeOpenTime: 253.425ms
> > - PrepareTime: 22.341ms
> > - ExecTreePrepareTime: 2.251ms
> > DataStreamSender (dst_id=2):(Total: 164.394us, non-child:
> 164.394us,
> > % non-child: 100.00%)
> > - BytesSent: 16.00 B (16)
> > - NetworkThroughput(*): 493.42 KB/sec
> > - OverallThroughput: 95.05 KB/sec
> > - PeakMemoryUsage: 16.00 KB (16384)
> > - RowsReturned: 1 (1)
> > - SerializeBatchTime: 14.262us
> > - TransmitDataRPCTime: 31.667us
> > - UncompressedRowBatchSize: 16.00 B (16)
> > CodeGen:(Total: 41.008ms, non-child: 41.008ms, % non-child:
> 100.00%)
> > - CodegenTime: 1.049ms
> > - CompileTime: 7.248ms
> > - LoadTime: 0.000ns
> > - ModuleBitcodeSize: 1.86 MB (1950880)
> > - NumFunctions: 15 (15)
> > - NumInstructions: 214 (214)
> > - OptimizationTime: 14.018ms
> > - PeakMemoryUsage: 107.00 KB (109568)
> > - PrepareTime: 19.080ms
> > AGGREGATION_NODE (id=1):(Total: 255.678ms, non-child: 1.046ms, %
> > non-child: 0.41%)
> > ExecOption: Codegen Enabled
> > - BuildTime: 1.015ms
> > - GetResultsTime: 0.000ns
> > - HTResizeTime: 0.000ns
> > - HashBuckets: 0 (0)
> > - LargestPartitionPercent: 0 (0)
> > - MaxPartitionLevel: 0 (0)
> > - NumRepartitions: 0 (0)
> > - PartitionsCreated: 0 (0)
> > - PeakMemoryUsage: 20.00 KB (20480)
> > - RowsRepartitioned: 0 (0)
> > - RowsReturned: 1 (1)
> > - RowsReturnedRate: 3.00 /sec
> > - SpilledPartitions: 0 (0)
> > HDFS_SCAN_NODE (id=0):(Total: 254.631ms, non-child: 254.631ms, %
> > non-child: 100.00%)
> > Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> > 9:55/35.38 MB 0:37/24.23 MB 5:37/23.31 MB 11:22/13.79 MB 8:30/18.53 MB
> > 4:33/20.45 MB 1:38/24.06 MB 10:29/18.40 MB 3:39/24.35 MB 2:41/25.45 MB
> > 6:42/27.03 MB 7:47/29.22 MB
> > ExecOption: PARQUET Codegen Enabled, Codegen enabled: 450 out
> of
> > 450
> > Hdfs Read Thread Concurrency Bucket: 0:0% 1:0% 2:0% 3:0% 4:0%
> 5:0%
> > 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
> > File Formats: PARQUET/Unknown(Skipped):450
> > - FooterProcessingTime: (Avg: 10.727ms ; Min: 3.813ms ; Max:
> > 23.755ms ; Number of samples: 450)
> > - AverageHdfsReadThreadConcurrency: 0.00
> > - AverageScannerThreadConcurrency: 0.00
> > - BytesRead: 43.85 MB (45983284)
> > - BytesReadDataNodeCache: 0
> > - BytesReadLocal: 43.85 MB (45983284)
> > - BytesReadRemoteUnexpected: 0
> > - BytesReadShortCircuit: 43.85 MB (45983284)
> > - CachedFileHandlesHitCount: 0 (0)
> > - CachedFileHandlesMissCount: 450 (450)
> > - CollectionItemsRead: 0 (0)
> > - DecompressionTime: 0.000ns
> > - MaxCompressedTextFileLength: 0
> > - NumColumns: 0 (0)
> > - NumDictFilteredRowGroups: 0 (0)
> > - NumDisksAccessed: 12 (12)
> > - NumRowGroups: 450 (450)
> > - NumScannerThreadsStarted: 24 (24)
> > - NumScannersWithNoReads: 0 (0)
> > - NumStatsFilteredRowGroups: 0 (0)
> > - PeakMemoryUsage: 2.62 MB (2752512)
> > - PerReadThreadRawHdfsThroughput: 1.56 GB/sec
> > - RemoteScanRanges: 0 (0)
> > - RowBatchQueueGetWaitTime: 242.362ms
> > - RowBatchQueuePutWaitTime: 0.000ns
> > - RowsRead: 0 (0)
> > - RowsReturned: 450 (450)
> > - RowsReturnedRate: 1.77 K/sec
> > - ScanRangesComplete: 450 (450)
> > - ScannerThreadsInvoluntaryContextSwitches: 106 (106)
> > - ScannerThreadsTotalWallClockTime: 5s926ms
> > - MaterializeTupleTime(*): 69.447us
> > - ScannerThreadsSysTime: 26.984ms
> > - ScannerThreadsUserTime: 607.897ms
> > - ScannerThreadsVoluntaryContextSwitches: 1.72K (1723)
> > - TotalRawHdfsReadTime(*): 27.393ms
> > - TotalReadThroughput: 0.00 /sec
> > ===================================================
> >
> > Coordinator log:
> >
> > 01: query (string) = "select count(*) from dv_entersiteevent_prq_remote
> > where year=2017 and month=10",
> > 03: configuration (list) = list<string>[1] {
> > [0] = "REQUEST_POOL=impreports",
> > },
> > 04: hadoop_user (string) = "fawzea",
> > }
> > I0208 01:07:06.321281 7154 impala-beeswax-server.cc:459]
> > TClientRequest.queryOptions: TQueryOptions {
> > 01: abort_on_error (bool) = false,
> > 02: max_errors (i32) = 100,
> > 03: disable_codegen (bool) = false,
> > 04: batch_size (i32) = 0,
> > 05: num_nodes (i32) = 0,
> > 06: max_scan_range_length (i64) = 0,
> > 07: num_scanner_threads (i32) = 0,
> > 08: max_io_buffers (i32) = 0,
> > 09: allow_unsupported_formats (bool) = false,
> > 10: default_order_by_limit (i64) = -1,
> > 11: debug_action (string) = "",
> > 12: mem_limit (i64) = 0,
> > 13: abort_on_default_limit_exceeded (bool) = false,
> > 15: hbase_caching (i32) = 0,
> > 16: hbase_cache_blocks (bool) = false,
> > 17: parquet_file_size (i64) = 0,
> > 18: explain_level (i32) = 1,
> > 19: sync_ddl (bool) = false,
> > 20: request_pool (string) = "impreports",
> > 23: disable_cached_reads (bool) = false,
> > 24: disable_outermost_topn (bool) = false,
> > 25: rm_initial_mem (i64) = 0,
> > 26: query_timeout_s (i32) = 0,
> > 28: appx_count_distinct (bool) = false,
> > 29: disable_unsafe_spills (bool) = false,
> > 31: exec_single_node_rows_threshold (i32) = 100,
> > 32: optimize_partition_key_scans (bool) = false,
> > 33: replica_preference (i32) = 0,
> > 34: schedule_random_replica (bool) = false,
> > 35: scan_node_codegen_threshold (i64) = 1800000,
> > 36: disable_streaming_preaggregations (bool) = false,
> > 37: runtime_filter_mode (i32) = 2,
> > 38: runtime_bloom_filter_size (i32) = 1048576,
> > 39: runtime_filter_wait_time_ms (i32) = 0,
> > 40: disable_row_runtime_filtering (bool) = false,
> > 41: max_num_runtime_filters (i32) = 10,
> > 42: parquet_annotate_strings_utf8 (bool) = false,
> > 43: parquet_fallback_schema_resolution (i32) = 0,
> > 45: s3_skip_insert_staging (bool) = true,
> > 46: runtime_filter_min_size (i32) = 1048576,
> > 47: runtime_filter_max_size (i32) = 16777216,
> > 48: prefetch_mode (i32) = 1,
> > 49: strict_mode (bool) = false,
> > 50: scratch_limit (i64) = -1,
> > 51: enable_expr_rewrites (bool) = true,
> > 52: decimal_v2 (bool) = false,
> > 53: parquet_dictionary_filtering (bool) = true,
> > 54: parquet_array_resolution (i32) = 2,
> > 55: parquet_read_statistics (bool) = true,
> > 56: default_join_distribution_mode (i32) = 0,
> > 57: disable_codegen_rows_threshold (i32) = 50000,
> > 58: default_spillable_buffer_size (i64) = 2097152,
> > 59: min_spillable_buffer_size (i64) = 65536,
> > 60: max_row_size (i64) = 524288,
> > }
> > I0208 01:07:06.324410 7154 Frontend.java:909] Compiling query: select
> > count(*) from dv_entersiteevent_prq_remote where year=2017 and month=10
> > I0208 01:07:06.327203 7154 Frontend.java:948] Compiled query.
> > I0208 01:07:06.359748 7154 admission-controller.cc:510] Schedule for
> > id=5f49e927591a9042:faf8f28600000000 in pool_name=root.impreports
> > cluster_mem_needed=105.00 MB PoolConfig: max_requests=-1 max_queued=200
> > max_mem=-1.00 B
> > I0208 01:07:06.359776 7154 admission-controller.cc:515] Stats:
> > agg_num_running=0, agg_num_queued=0, agg_mem_reserved=0,
> > local_host(local_mem_admitted=0, num_admitted_running=0, num_queued=0,
> > backend_mem_reserved=0)
> > I0208 01:07:06.359784 7154 admission-controller.cc:531] Admitted query
> > id=5f49e927591a9042:faf8f28600000000
> > I0208 01:07:06.359807 7154 coordinator.cc:99] Exec()
> > query_id=5f49e927591a9042:faf8f28600000000 stmt=select count(*) from
> > dv_entersiteevent_prq_remote where year=2017 and month=10
> > I0208 01:07:06.360694 7154 coordinator.cc:357] starting execution on 5
> > backends for query 5f49e927591a9042:faf8f28600000000
> > I0208 01:07:06.363255 63583 impala-internal-service.cc:44]
> > ExecQueryFInstances(): query_id=5f49e927591a9042:faf8f28600000000
> > I0208 01:07:06.363282 63583 query-exec-mgr.cc:46] StartQueryFInstances()
> > query_id=5f49e927591a9042:faf8f28600000000
> > coord=aopr-dhc001.lpdomain.com:22000
> > I0208 01:07:06.363296 63583 query-state.cc:173] Buffer pool limit for
> > 5f49e927591a9042:faf8f28600000000: 10802010521
> > I0208 01:07:06.363364 63583 initial-reservations.cc:60] Successfully
> claimed
> > initial reservations (0) for query 5f49e927591a9042:faf8f28600000000
> > I0208 01:07:06.363544 7155 query-state.cc:286] StartFInstances():
> > query_id=5f49e927591a9042:faf8f28600000000 #instances=2
> > I0208 01:07:06.363844 7155 query-state.cc:299] descriptor table for
> > query=5f49e927591a9042:faf8f28600000000
> > tuples:
> > Tuple(id=2 size=0 slots=[] tuple_path=[])
> > Tuple(id=1 size=8 slots=[Slot(id=2 type=BIGINT col_path=[] offset=0
> > null=(offset=0 mask=0) slot_idx=0 field_idx=-1)] tuple_path=[])
> > Tuple(id=0 size=8 slots=[Slot(id=3 type=BIGINT col_path=[] offset=0
> > null=(offset=0 mask=0) slot_idx=0 field_idx=-1)] tuple_path=[])
> > I0208 01:07:06.364013 7156 query-state.cc:377] Executing instance.
> > instance_id=5f49e927591a9042:faf8f28600000000 fragment_idx=0
> > per_fragment_instance_idx=0 coord_state_idx=0 #in-flight=1
> > I0208 01:07:06.364135 7157 query-state.cc:377] Executing instance.
> > instance_id=5f49e927591a9042:faf8f28600000005 fragment_idx=1
> > per_fragment_instance_idx=4 coord_state_idx=0 #in-flight=2
> > I0208 01:07:06.364241 7157 hdfs-scan-node.cc:160] Max row batch queue
> size
> > for scan node '0' in fragment instance '5f49e927591a9042:
> faf8f28600000005':
> > 160
> > I0208 01:07:06.366775 7154 coordinator.cc:370] started execution on 5
> > backends for query 5f49e927591a9042:faf8f28600000000
> > I0208 01:07:06.385625 7155 query-exec-mgr.cc:149] ReleaseQueryState():
> > query_id=5f49e927591a9042:faf8f28600000000 refcnt=4
> > I0208 01:07:06.671193 7559 impala-server.cc:1796] Connection from client
> > ::ffff:10.16.136.3:55598 closed, closing 1 associated session(s)
> > I0208 01:07:06.676138 544 data-stream-mgr.cc:228] Reduced stream ID
> cache
> > from 2 items, to 0, eviction took: 0
> > I0208 01:07:06.677202 544 coordinator.cc:955] Backend completed:
> > host=aopr-dhc002.lpdomain.com:22000 remaining=4
> > I0208 01:07:06.677214 544 coordinator-backend-state.cc:229]
> > query_id=5f49e927591a9042:faf8f28600000000: first in-progress backend:
> > aopr-dhc001.lpdomain.com:22000
> > I0208 01:07:06.684221 7157 query-state.cc:384] Instance completed.
> > instance_id=5f49e927591a9042:faf8f28600000005 #in-flight=1 status=OK
> > I0208 01:07:06.684237 7157 query-exec-mgr.cc:149] ReleaseQueryState():
> > query_id=5f49e927591a9042:faf8f28600000000 refcnt=3
> > I0208 01:07:06.700275 520 coordinator.cc:955] Backend completed:
> > host=aopr-dhc005.lpdomain.com:22000 remaining=3
> > I0208 01:07:06.700289 520 coordinator-backend-state.cc:229]
> > query_id=5f49e927591a9042:faf8f28600000000: first in-progress backend:
> > aopr-dhc001.lpdomain.com:22000
> > I0208 01:07:06.704620 65258 coordinator.cc:955] Backend completed:
> > host=aopr-dhc003.lpdomain.com:22000 remaining=2
> > I0208 01:07:06.704666 65258 coordinator-backend-state.cc:229]
> > query_id=5f49e927591a9042:faf8f28600000000: first in-progress backend:
> > aopr-dhc001.lpdomain.com:22000
> > I0208 01:07:06.706041 7156 data-stream-mgr.cc:238] DeregisterRecvr():
> > fragment_instance_id=5f49e927591a9042:faf8f28600000000, node=2
> > I0208 01:07:06.706070 7156 data-stream-recvr.cc:235] cancelled stream:
> > fragment_instance_id_=5f49e927591a9042:faf8f28600000000 node_id=2
> > I0208 01:07:06.706809 64916 coordinator.cc:955] Backend completed:
> > host=aopr-dhc004.lpdomain.com:22000 remaining=1
> > I0208 01:07:06.706821 64916 coordinator-backend-state.cc:229]
> > query_id=5f49e927591a9042:faf8f28600000000: first in-progress backend:
> > aopr-dhc001.lpdomain.com:22000
> > I0208 01:07:06.789568 7154 impala-beeswax-server.cc:193]
> > get_results_metadata(): query_id=5f49e927591a9042:faf8f28600000000
> > I0208 01:07:06.790439 7154 coordinator.cc:789] Coordinator waiting for
> > backends to finish, 1 remaining
> > I0208 01:07:06.791052 7154 coordinator.cc:794] All backends finished
> > successfully.
> > I0208 01:07:06.791086 7154 coordinator.cc:1090] Release admssion control
> > resources for query 5f49e927591a9042:faf8f28600000000
> > I0208 01:07:06.791363 7156 query-state.cc:384] Instance completed.
> > instance_id=5f49e927591a9042:faf8f28600000000 #in-flight=0 status=OK
> > I0208 01:07:06.791414 7156 query-exec-mgr.cc:149] ReleaseQueryState():
> > query_id=5f49e927591a9042:faf8f28600000000 refcnt=2
> > I0208 01:07:06.793586 7154 impala-beeswax-server.cc:235] close():
> > query_id=5f49e927591a9042:faf8f28600000000
> > I0208 01:07:06.793602 7154 impala-server.cc:992] UnregisterQuery():
> > query_id=5f49e927591a9042:faf8f28600000000
> > I0208 01:07:06.793614 7154 impala-server.cc:1075] Cancel():
> > query_id=5f49e927591a9042:faf8f28600000000
> > I0208 01:07:06.793624 7154 coordinator.cc:895] Cancel()
> > query_id=5f49e927591a9042:faf8f28600000000
> > I0208 01:07:06.793637 7154 coordinator.cc:905] CancelBackends()
> > query_id=5f49e927591a9042:faf8f28600000000, tried to cancel 0 backends
> > I0208 01:07:06.802776 7154 query-exec-mgr.cc:149] ReleaseQueryState():
> > query_id=5f49e927591a9042:faf8f28600000000 refcnt=1
> > I0208 01:07:06.814214 7154 impala-server.cc:1796] Connection from client
> > ::ffff:10.16.144.131:51012 closed, closing 1 associated session(s)
> > I0208 01:07:07.258891 7567 impala-server.cc:1796] Connection from client
> > ::ffff:10.16.136.2:56375 closed, closing 1 associated session(s)
> > I0208 01:07:11.600131 7819 impala-server.cc:1796] Connection from client
> > ::ffff:10.16.136.3:27460 closed, closing 1 associated session(s)
> > I0208 01:07:12.262739 7821 impala-server.cc:1796] Connection from client
> > ::ffff:10.16.136.2:52672 closed, closing 1 associated session(s)
> > I0208 01:07:16.665451 7822 impala-server.cc:1796] Connection from client
> > ::ffff:10.16.136.3:10693 closed, closing 1 associated session(s)
> > I0208 01:07:17.235321 7825 impala-server.cc:1796] Connection from client
> > ::ffff:10.16.136.2:34752 closed, closing 1 associated session(s)
> > I0208 01:07:21.410611 5858 impala-beeswax-server.cc:193]
> > get_results_metadata(): query_id=1d4fbf0f2571f1e9:ea346f7900000000
> > I0208 01:07:21.414175 5858 impala-beeswax-server.cc:235] close():
> > query_id=1d4fbf0f2571f1e9:ea346f7900000000
> > I0208 01:07:21.414185 5858 impala-server.cc:992] UnregisterQuery():
> > query_id=1d4fbf0f2571f1e9:ea346f7900000000
> > I0208 01:07:21.414191 5858 impala-server.cc:1075] Cancel():
> > query_id=1d4fbf0f2571f1e9:ea346f7900000000
> > I0208 01:07:21.421505 5858 impala-server.cc:1796] Connection from client
> > ::ffff:10.26.180.75:60072 closed, closing 1 associated session(s)
> > I0208 01:07:21.605690 7830 impala-server.cc:1796] Connection from client
> > ::ffff:10.16.136.3:46660 closed, closing 1 associated session(s)
> >
> >
> > On Thu, Feb 8, 2018 at 6:20 AM, Matthew Jacobs <mj...@apache.org>
> wrote:
> >>
> >> Thanks! I think that configuration looks OK.
> >>
> >> 1) What version of Impala is this?
> >>
> >> 2) Can you send the query profile and the Impala coordinator log for
> >> the query where it's not behaving as expected?
> >>
> >> > When i'm running the following with fawzea user:
> >> > set request_pool=impreports;
> >> > select count(*) from dv_entersiteevent_prq_remote where year=2017 and
> >> > month=10;
> >> > I can see that the query is running with impreports pool.
> >>
> >> On Wed, Feb 7, 2018 at 7:59 PM, Fawze Abujaber <fa...@gmail.com>
> wrote:
> >> > Hi Matt,
> >> >
> >> > [root@aopr-dhc001 ~]# cat
> >> >
> >> > /var/run/cloudera-scm-agent/process/4558-impala-IMPALAD/
> impala-conf/fair-scheduler.xml
> >> > <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
> >> > <allocations>
> >> > <queue name="root">
> >> > <weight>1.0</weight>
> >> > <schedulingPolicy>fair</schedulingPolicy>
> >> > <queue name="impiaas">
> >> > <weight>1.0</weight>
> >> > <schedulingPolicy>fair</schedulingPolicy>
> >> > <aclSubmitApps>fawzea </aclSubmitApps>
> >> > </queue>
> >> > <queue name="impreports">
> >> > <weight>1.0</weight>
> >> > <schedulingPolicy>fair</schedulingPolicy>
> >> > <aclSubmitApps>nira </aclSubmitApps>
> >> > </queue>
> >> > </queue>
> >> > <defaultQueueSchedulingPolicy>fair</defaultQueueSchedulingPolicy>
> >> > <queuePlacementPolicy>
> >> > <rule name="specified" create="false"/>
> >> > <rule name="secondaryGroupExistingQueue"/>
> >> > <rule name="default"/>
> >> > </queuePlacementPolicy>
> >> > </allocations>
> >> >
> >> >
> >> > On Thu, Feb 8, 2018 at 4:47 AM, Matthew Jacobs <ja...@gmail.com>
> >> > wrote:
> >> >>
> >> >> Yes, I saw that, but I'd like to see the actual configuration that is
> >> >> passed to Impala because it may not be specified in the right way to
> >> >> accomplish what you're trying to do in "b".
> >> >>
> >> >> On Wed, Feb 7, 2018 at 4:32 PM, Fawze Abujaber <fa...@gmail.com>
> >> >> wrote:
> >> >> > Hi Matt,
> >> >> >
> >> >> > I created the pools manually and it’s completely nfigurations as
> >> >> > shown
> >> >> > in b.
> >> >> >
> >> >> > On Wed, 7 Feb 2018 at 23:17 Matthew Jacobs <ja...@gmail.com>
> >> >> > wrote:
> >> >> >>
> >> >> >> How, more specifically, are you configuring these pools? Can you
> >> >> >> share
> >> >> >> the configuration files?
> >> >> >>
> >> >> >> On Wed, Feb 7, 2018 at 11:42 AM, Fawze Abujaber <
> fawzeaj@gmail.com>
> >> >> >> wrote:
> >> >> >> > Hello All,
> >> >> >> >
> >> >> >> > Will appreciate any insight of what i'm missing in my
> >> >> >> > configuration
> >> >> >> > to
> >> >> >> > get
> >> >> >> > the impala submission control working as expected ....
> >> >> >> >
> >> >> >> > a- Here is my placement Placement Rule:
> >> >> >> > 1- Use the pool Specified at run time., only if the pool exists.
> >> >> >> > 2- Use the pool root.[secondary group].
> >> >> >> > 3- Use the pool root.default.
> >> >> >> >
> >> >> >> > b- Here is the impala resource pools
> >> >> >> >
> >> >> >> > Name Max Memory Max Running Queries Max
> Queued
> >> >> >> > Queries
> >> >> >> > Queue Timeout Default Query Memory Limit Access Control
> >> >> >> > root.impiaas Unlimited Unlimited
> >> >> >> > 200 1 minute(s) No Default
> >> >> >> > Custom
> >> >> >> > root.impreports Unlimited Unlimited
> >> >> >> > 200 1 minute(s) No Default
> >> >> >> > Custom
> >> >> >> >
> >> >> >> >
> >> >> >> > C- Here is my Access control for root.impiaas pool:
> >> >> >> >
> >> >> >> > Allow these users and groups to submit to this pool
> >> >> >> >
> >> >> >> > Users: fawzea
> >> >> >> >
> >> >> >> > The same for impreports the allowed user and groups to submit to
> >> >> >> > pool
> >> >> >> > is
> >> >> >> > only nira.
> >> >> >> >
> >> >> >> >
> >> >> >> > LDAP groups for fawzea user:
> >> >> >> >
> >> >> >> > Member in groups:
> >> >> >> >
> >> >> >> > - dba
> >> >> >> > - impiaas
> >> >> >> > - mysql
> >> >> >> >
> >> >> >> > LDAP group for nira user
> >> >> >> > Member in groups:
> >> >> >> > - dba
> >> >> >> > - impreports
> >> >> >> > - mysql
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> > Impala Queries UI shows:
> >> >> >> >
> >> >> >> > 1- select count(*) from dv_entersiteevent_prq_remote where
> >> >> >> > year=2017
> >> >> >> >
> >> >> >> > User: fawzea
> >> >> >> > Pool: root.impiaas
> >> >> >> >
> >> >> >> > 2- select count(*) from dv_entersiteevent_prq_remote where
> >> >> >> > year=2017
> >> >> >> > and
> >> >> >> > month=10
> >> >> >> > User: nira
> >> >> >> > Pool: root.impreports
> >> >> >> >
> >> >> >> > As shown above that the queries submitted based on the secondary
> >> >> >> > group
> >> >> >> > rule.
> >> >> >> >
> >> >> >> > When i'm running the following with fawzea user:
> >> >> >> >
> >> >> >> > set request_pool=impreports;
> >> >> >> > select count(*) from dv_entersiteevent_prq_remote where
> year=2017
> >> >> >> > and
> >> >> >> > month=10;
> >> >> >> >
> >> >> >> > I can see that the query is running with impreports pool.
> >> >> >> >
> >> >> >> >
> >> >> >> > What i'm missing here in my configuration to get this works as
> >> >> >> > expected.
> >> >> >> >
> >> >
> >> >
> >
> >
>
Re: impala submission control
Posted by Matthew Jacobs <ja...@gmail.com>.
As far as I can tell from this information, your configuration looks
fine. The only other potential issue I can think of is that the
secondary groups placement rule isn't picking up your LDAP groups
properly. Does the ACL behavior work if you remove the secondary
groups rule?
On Wed, Feb 7, 2018 at 10:09 PM, Fawze Abujaber <fa...@gmail.com> wrote:
> Hi Matt,
>
> v2.11.0-cdh5.14.0
>
> Query Runtime Profile:
> Query (id=c843dae48ce9b289:6ad3ffc000000000):
> Summary:
> Session ID: ca420ac477772b56:51ae460ce9e44999
> Session Type: BEESWAX
> Start Time: 2018-02-08 01:02:01.023558000
> End Time: 2018-02-08 01:02:01.501506000
> Query Type: QUERY
> Query State: FINISHED
> Query Status: OK
> Impala Version: impalad version 2.11.0-cdh5.14.0 RELEASE (build
> d68206561bce6b26762d62c01a78e6cd27aa7690)
> User: fawzea
> Connected User: fawzea
> Delegated User:
> Network Address: ::ffff:10.16.144.131:50078
> Default Db: default
> Sql Statement: select count(*) from dv_entersiteevent_prq_remote where
> year=2017 and month=10
> Coordinator: aopr-dhc001.lpdomain.com:22000
> Query Options (set by configuration): REQUEST_POOL=impreports
> Query Options (set by configuration and planner):
> REQUEST_POOL=impreports,MT_DOP=0
> Plan:
> ----------------
> Max Per-Host Resource Reservation: Memory=0B
> Per-Host Resource Estimates: Memory=21.00MB
> WARNING: The following tables are missing relevant table and/or column
> statistics.
> default.dv_entersiteevent_prq_remote
>
> F01:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1
> | Per-Host Resources: mem-estimate=10.00MB mem-reservation=0B
> PLAN-ROOT SINK
> | mem-estimate=0B mem-reservation=0B
> |
> 03:AGGREGATE [FINALIZE]
> | output: count:merge(*)
> | mem-estimate=10.00MB mem-reservation=0B spill-buffer=2.00MB
> | tuple-ids=1 row-size=8B cardinality=1
> |
> 02:EXCHANGE [UNPARTITIONED]
> | mem-estimate=0B mem-reservation=0B
> | tuple-ids=1 row-size=8B cardinality=1
> |
> F00:PLAN FRAGMENT [RANDOM] hosts=5 instances=5
> Per-Host Resources: mem-estimate=11.00MB mem-reservation=0B
> 01:AGGREGATE
> | output: sum_init_zero(default.dv_entersiteevent_prq_remote.parquet-stats:
> num_rows)
> | mem-estimate=10.00MB mem-reservation=0B spill-buffer=2.00MB
> | tuple-ids=1 row-size=8B cardinality=1
> |
> 00:SCAN HDFS [default.dv_entersiteevent_prq_remote, RANDOM]
> partitions=31/441 files=2292 size=1.39GB
> stats-rows=unavailable extrapolated-rows=disabled
> table stats: rows=unavailable size=unavailable
> column stats: all
> mem-estimate=1.00MB mem-reservation=0B
> tuple-ids=0 row-size=8B cardinality=unavailable
> ----------------
> Estimated Per-Host Mem: 22020096
> Tables Missing Stats: default.dv_entersiteevent_prq_remote
> Per Host Min Reservation: aopr-dhc001.lpdomain.com:22000(0)
> aopr-dhc002.lpdomain.com:22000(0) aopr-dhc003.lpdomain.com:22000(0)
> aopr-dhc004.lpdomain.com:22000(0) aopr-dhc005.lpdomain.com:22000(0)
> Request Pool: root.impreports
> Admission result: Admitted immediately
> ExecSummary:
> Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem
> Est. Peak Mem Detail
> ------------------------------------------------------------------------------------------------------------------------------
> 03:AGGREGATE 1 222.958us 222.958us 1 1 36.00 KB
> 10.00 MB FINALIZE
> 02:EXCHANGE 1 357.508ms 357.508ms 5 1 0
> 0 UNPARTITIONED
> 01:AGGREGATE 5 1.027ms 1.541ms 5 1 20.00 KB
> 10.00 MB
> 00:SCAN HDFS 5 282.797ms 333.715ms 2.29K -1 2.62 MB
> 1.00 MB default.dv_entersiteevent_prq_remote
> Errors:
> Planner Timeline: 13.508ms
> - Analysis finished: 3.303ms (3.303ms)
> - Value transfer graph computed: 3.465ms (162.667us)
> - Single node plan created: 10.352ms (6.886ms)
> - Runtime filters computed: 10.677ms (325.364us)
> - Distributed plan created: 10.733ms (55.854us)
> - Lineage info computed: 10.898ms (164.937us)
> - Planning finished: 13.508ms (2.610ms)
> Query Timeline: 481.950ms
> - Query submitted: 79.971us (79.971us)
> - Planning finished: 33.947ms (33.867ms)
> - Submit for admission: 40.067ms (6.120ms)
> - Completed admission: 40.197ms (130.146us)
> - Ready to start on 5 backends: 40.905ms (707.852us)
> - All 5 execution backends (6 fragment instances) started: 46.417ms
> (5.512ms)
> - Rows available: 424.548ms (378.131ms)
> - First row fetched: 473.488ms (48.939ms)
> - Unregister query: 477.951ms (4.462ms)
> - ComputeScanRangeAssignmentTimer: 3.980ms
> ImpalaServer:
> - ClientFetchWaitTimer: 51.659ms
> - RowMaterializationTimer: 1.726ms
> Execution Profile c843dae48ce9b289:6ad3ffc000000000:(Total: 385.029ms,
> non-child: 0.000ns, % non-child: 0.00%)
> Number of filters: 0
> Filter routing table:
> ID Src. Node Tgt. Node(s) Target type Partition filter Pending
> (Expected) First arrived Completed Enabled
> -------------------------------------------------------------------------------------------------------------------
>
> Backend startup latencies: Count: 5, min / max: 3ms / 4ms, 25th %-ile:
> 3ms, 50th %-ile: 4ms, 75th %-ile: 4ms, 90th %-ile: 4ms, 95th %-ile: 4ms,
> 99.9th %-ile: 4ms
> Per Node Peak Memory Usage: aopr-dhc001.lpdomain.com:22000(2.68 MB)
> aopr-dhc002.lpdomain.com:22000(2.54 MB) aopr-dhc003.lpdomain.com:22000(2.54
> MB) aopr-dhc004.lpdomain.com:22000(2.41 MB)
> aopr-dhc005.lpdomain.com:22000(2.56 MB)
> - FiltersReceived: 0 (0)
> - FinalizationTimer: 0.000ns
> - NumBackends: 5 (5)
> - NumFragmentInstances: 6 (6)
> - NumFragments: 2 (2)
> Averaged Fragment F01:(Total: 427.530ms, non-child: 49.428ms, %
> non-child: 11.56%)
> split sizes: min: 0, max: 0, avg: 0, stddev: 0
> completion times: min:428.693ms max:428.693ms mean: 428.693ms
> stddev:0.000ns
> execution rates: min:0.00 /sec max:0.00 /sec mean:0.00 /sec
> stddev:0.00 /sec
> num instances: 1
> - AverageThreadTokens: 0.00
> - BloomFilterBytes: 0
> - PeakMemoryUsage: 44.02 KB (45072)
> - PeakReservation: 0
> - PeakUsedReservation: 0
> - PerHostPeakMemUsage: 2.68 MB (2806113)
> - RowsProduced: 1 (1)
> - TotalNetworkReceiveTime: 357.481ms
> - TotalNetworkSendTime: 0.000ns
> - TotalStorageWaitTime: 0.000ns
> - TotalThreadsInvoluntaryContextSwitches: 0 (0)
> - TotalThreadsTotalWallClockTime: 406.708ms
> - TotalThreadsSysTime: 0.000ns
> - TotalThreadsUserTime: 0.000ns
> - TotalThreadsVoluntaryContextSwitches: 7 (7)
> Fragment Instance Lifecycle Timings:
> - ExecTime: 49.055ms
> - ExecTreeExecTime: 4.251us
> - OpenTime: 357.663ms
> - ExecTreeOpenTime: 357.649ms
> - PrepareTime: 20.764ms
> - ExecTreePrepareTime: 81.759us
> PLAN_ROOT_SINK:
> - PeakMemoryUsage: 0
> CodeGen:(Total: 20.369ms, non-child: 20.369ms, % non-child: 100.00%)
> - CodegenTime: 0.000ns
> - CompileTime: 0.000ns
> - LoadTime: 0.000ns
> - ModuleBitcodeSize: 1.86 MB (1950880)
> - NumFunctions: 0 (0)
> - NumInstructions: 0 (0)
> - OptimizationTime: 0.000ns
> - PeakMemoryUsage: 0
> - PrepareTime: 19.685ms
> AGGREGATION_NODE (id=3):(Total: 357.731ms, non-child: 222.958us, %
> non-child: 0.06%)
> - BuildTime: 7.368us
> - GetResultsTime: 0.000ns
> - HTResizeTime: 0.000ns
> - HashBuckets: 0 (0)
> - LargestPartitionPercent: 0 (0)
> - MaxPartitionLevel: 0 (0)
> - NumRepartitions: 0 (0)
> - PartitionsCreated: 0 (0)
> - PeakMemoryUsage: 36.00 KB (36864)
> - RowsRepartitioned: 0 (0)
> - RowsReturned: 1 (1)
> - RowsReturnedRate: 2.00 /sec
> - SpilledPartitions: 0 (0)
> EXCHANGE_NODE (id=2):(Total: 357.508ms, non-child: 357.508ms, %
> non-child: 100.00%)
> - ConvertRowBatchTime: 8.881us
> - PeakMemoryUsage: 0
> - RowsReturned: 5 (5)
> - RowsReturnedRate: 13.00 /sec
> DataStreamReceiver:
> - BytesReceived: 80.00 B (80)
> - DeserializeRowBatchTimer: 24.165us
> - FirstBatchArrivalWaitTime: 277.194ms
> - PeakMemoryUsage: 4.02 KB (4112)
> - SendersBlockedTimer: 0.000ns
> - SendersBlockedTotalTimer(*): 0.000ns
> Coordinator Fragment F01:
> Instance c843dae48ce9b289:6ad3ffc000000000
> (host=aopr-dhc001.lpdomain.com:22000):(Total: 427.530ms, non-child:
> 49.428ms, % non-child: 11.56%)
> - AverageThreadTokens: 0.00
> - BloomFilterBytes: 0
> - PeakMemoryUsage: 44.02 KB (45072)
> - PeakReservation: 0
> - PeakUsedReservation: 0
> - PerHostPeakMemUsage: 2.68 MB (2806113)
> - RowsProduced: 1 (1)
> - TotalNetworkReceiveTime: 357.481ms
> - TotalNetworkSendTime: 0.000ns
> - TotalStorageWaitTime: 0.000ns
> - TotalThreadsInvoluntaryContextSwitches: 0 (0)
> - TotalThreadsTotalWallClockTime: 406.708ms
> - TotalThreadsSysTime: 0.000ns
> - TotalThreadsUserTime: 0.000ns
> - TotalThreadsVoluntaryContextSwitches: 7 (7)
> Fragment Instance Lifecycle Timings:
> - ExecTime: 49.055ms
> - ExecTreeExecTime: 4.251us
> - OpenTime: 357.663ms
> - ExecTreeOpenTime: 357.649ms
> - PrepareTime: 20.764ms
> - ExecTreePrepareTime: 81.759us
> PLAN_ROOT_SINK:
> - PeakMemoryUsage: 0
> CodeGen:(Total: 20.369ms, non-child: 20.369ms, % non-child: 100.00%)
> - CodegenTime: 0.000ns
> - CompileTime: 0.000ns
> - LoadTime: 0.000ns
> - ModuleBitcodeSize: 1.86 MB (1950880)
> - NumFunctions: 0 (0)
> - NumInstructions: 0 (0)
> - OptimizationTime: 0.000ns
> - PeakMemoryUsage: 0
> - PrepareTime: 19.685ms
> AGGREGATION_NODE (id=3):(Total: 357.731ms, non-child: 222.958us, %
> non-child: 0.06%)
> - BuildTime: 7.368us
> - GetResultsTime: 0.000ns
> - HTResizeTime: 0.000ns
> - HashBuckets: 0 (0)
> - LargestPartitionPercent: 0 (0)
> - MaxPartitionLevel: 0 (0)
> - NumRepartitions: 0 (0)
> - PartitionsCreated: 0 (0)
> - PeakMemoryUsage: 36.00 KB (36864)
> - RowsRepartitioned: 0 (0)
> - RowsReturned: 1 (1)
> - RowsReturnedRate: 2.00 /sec
> - SpilledPartitions: 0 (0)
> EXCHANGE_NODE (id=2):(Total: 357.508ms, non-child: 357.508ms, %
> non-child: 100.00%)
> - ConvertRowBatchTime: 8.881us
> - PeakMemoryUsage: 0
> - RowsReturned: 5 (5)
> - RowsReturnedRate: 13.00 /sec
> DataStreamReceiver:
> - BytesReceived: 80.00 B (80)
> - DeserializeRowBatchTimer: 24.165us
> - FirstBatchArrivalWaitTime: 277.194ms
> - PeakMemoryUsage: 4.02 KB (4112)
> - SendersBlockedTimer: 0.000ns
> - SendersBlockedTotalTimer(*): 0.000ns
> Averaged Fragment F00:(Total: 327.360ms, non-child: 1.578ms, %
> non-child: 0.48%)
> split sizes: min: 283.31 MB, max: 284.33 MB, avg: 284.04 MB, stddev:
> 382.00 KB
> completion times: min:299.457ms max:378.950ms mean: 328.881ms
> stddev:31.507ms
> execution rates: min:750.18 MB/sec max:949.01 MB/sec mean:871.27
> MB/sec stddev:79.74 MB/sec
> num instances: 5
> - AverageThreadTokens: 15.00
> - BloomFilterBytes: 0
> - PeakMemoryUsage: 2.54 MB (2666852)
> - PeakReservation: 0
> - PeakUsedReservation: 0
> - PerHostPeakMemUsage: 2.55 MB (2669309)
> - RowsProduced: 1 (1)
> - TotalNetworkReceiveTime: 0.000ns
> - TotalNetworkSendTime: 303.541us
> - TotalStorageWaitTime: 4s821ms
> - TotalThreadsInvoluntaryContextSwitches: 104 (104)
> - TotalThreadsTotalWallClockTime: 6s835ms
> - TotalThreadsSysTime: 31.784ms
> - TotalThreadsUserTime: 682.084ms
> - TotalThreadsVoluntaryContextSwitches: 2.12K (2121)
> Fragment Instance Lifecycle Timings:
> - ExecTime: 500.816us
> - ExecTreeExecTime: 8.200us
> - OpenTime: 302.848ms
> - ExecTreeOpenTime: 281.309ms
> - PrepareTime: 23.946ms
> - ExecTreePrepareTime: 2.513ms
> DataStreamSender (dst_id=2):(Total: 269.902us, non-child: 269.902us, %
> non-child: 100.00%)
> - BytesSent: 16.00 B (16)
> - NetworkThroughput(*): 373.13 KB/sec
> - OverallThroughput: 63.53 KB/sec
> - PeakMemoryUsage: 16.00 KB (16384)
> - RowsReturned: 1 (1)
> - SerializeBatchTime: 14.525us
> - TransmitDataRPCTime: 43.122us
> - UncompressedRowBatchSize: 16.00 B (16)
> CodeGen:(Total: 41.687ms, non-child: 41.687ms, % non-child: 100.00%)
> - CodegenTime: 1.194ms
> - CompileTime: 7.360ms
> - LoadTime: 0.000ns
> - ModuleBitcodeSize: 1.86 MB (1950880)
> - NumFunctions: 15 (15)
> - NumInstructions: 214 (214)
> - OptimizationTime: 13.529ms
> - PeakMemoryUsage: 107.00 KB (109568)
> - PrepareTime: 20.084ms
> AGGREGATION_NODE (id=1):(Total: 283.825ms, non-child: 1.027ms, %
> non-child: 0.36%)
> - BuildTime: 1.259ms
> - GetResultsTime: 0.000ns
> - HTResizeTime: 0.000ns
> - HashBuckets: 0 (0)
> - LargestPartitionPercent: 0 (0)
> - MaxPartitionLevel: 0 (0)
> - NumRepartitions: 0 (0)
> - PartitionsCreated: 0 (0)
> - PeakMemoryUsage: 20.00 KB (20480)
> - RowsRepartitioned: 0 (0)
> - RowsReturned: 1 (1)
> - RowsReturnedRate: 2.00 /sec
> - SpilledPartitions: 0 (0)
> HDFS_SCAN_NODE (id=0):(Total: 282.797ms, non-child: 282.797ms, %
> non-child: 100.00%)
> - AverageHdfsReadThreadConcurrency: 0.00
> - AverageScannerThreadConcurrency: 14.40
> - BytesRead: 44.52 MB (46686028)
> - BytesReadDataNodeCache: 0
> - BytesReadLocal: 44.52 MB (46686028)
> - BytesReadRemoteUnexpected: 0
> - BytesReadShortCircuit: 44.52 MB (46686028)
> - CachedFileHandlesHitCount: 0 (0)
> - CachedFileHandlesMissCount: 458 (458)
> - CollectionItemsRead: 0 (0)
> - DecompressionTime: 0.000ns
> - MaxCompressedTextFileLength: 0
> - NumColumns: 0 (0)
> - NumDictFilteredRowGroups: 0 (0)
> - NumDisksAccessed: 12 (12)
> - NumRowGroups: 458 (458)
> - NumScannerThreadsStarted: 24 (24)
> - NumScannersWithNoReads: 0 (0)
> - NumStatsFilteredRowGroups: 0 (0)
> - PeakMemoryUsage: 2.50 MB (2625539)
> - PerReadThreadRawHdfsThroughput: 1.29 GB/sec
> - RemoteScanRanges: 0 (0)
> - RowBatchQueueGetWaitTime: 269.787ms
> - RowBatchQueuePutWaitTime: 0.000ns
> - RowsRead: 0 (0)
> - RowsReturned: 458 (458)
> - RowsReturnedRate: 1.64 K/sec
> - ScanRangesComplete: 458 (458)
> - ScannerThreadsInvoluntaryContextSwitches: 99 (99)
> - ScannerThreadsTotalWallClockTime: 6s532ms
> - MaterializeTupleTime(*): 70.664us
> - ScannerThreadsSysTime: 26.184ms
> - ScannerThreadsUserTime: 650.089ms
> - ScannerThreadsVoluntaryContextSwitches: 1.67K (1674)
> - TotalRawHdfsReadTime(*): 34.416ms
> - TotalReadThroughput: 36.86 MB/sec
> Fragment F00:
> Instance c843dae48ce9b289:6ad3ffc000000004
> (host=aopr-dhc004.lpdomain.com:22000):(Total: 377.384ms, non-child: 1.619ms,
> % non-child: 0.43%)
> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> 0:36/21.95 MB 11:33/19.87 MB 9:27/15.16 MB 5:44/28.07 MB 6:56/33.32 MB
> 1:38/22.24 MB 10:44/27.90 MB 3:34/20.26 MB 2:45/28.21 MB 7:31/19.33 MB
> 4:42/26.73 MB 8:34/21.25 MB
> MemoryUsage(500.000ms): 1.91 MB
> ThreadUsage(500.000ms): 25
> - AverageThreadTokens: 25.00
> - BloomFilterBytes: 0
> - PeakMemoryUsage: 2.41 MB (2531689)
> - PeakReservation: 0
> - PeakUsedReservation: 0
> - PerHostPeakMemUsage: 2.41 MB (2531689)
> - RowsProduced: 1 (1)
> - TotalNetworkReceiveTime: 0.000ns
> - TotalNetworkSendTime: 382.629us
> - TotalStorageWaitTime: 5s790ms
> - TotalThreadsInvoluntaryContextSwitches: 115 (115)
> - TotalThreadsTotalWallClockTime: 8s122ms
> - TotalThreadsSysTime: 33.984ms
> - TotalThreadsUserTime: 702.879ms
> - TotalThreadsVoluntaryContextSwitches: 2.04K (2037)
> Fragment Instance Lifecycle Timings:
> - ExecTime: 604.005us
> - ExecTreeExecTime: 8.338us
> - OpenTime: 352.927ms
> - ExecTreeOpenTime: 331.921ms
> - PrepareTime: 23.783ms
> - ExecTreePrepareTime: 2.559ms
> DataStreamSender (dst_id=2):(Total: 331.818us, non-child: 331.818us,
> % non-child: 100.00%)
> - BytesSent: 16.00 B (16)
> - NetworkThroughput(*): 289.00 KB/sec
> - OverallThroughput: 47.09 KB/sec
> - PeakMemoryUsage: 16.00 KB (16384)
> - RowsReturned: 1 (1)
> - SerializeBatchTime: 14.466us
> - TransmitDataRPCTime: 54.066us
> - UncompressedRowBatchSize: 16.00 B (16)
> CodeGen:(Total: 40.950ms, non-child: 40.950ms, % non-child: 100.00%)
> - CodegenTime: 1.122ms
> - CompileTime: 7.318ms
> - LoadTime: 0.000ns
> - ModuleBitcodeSize: 1.86 MB (1950880)
> - NumFunctions: 15 (15)
> - NumInstructions: 214 (214)
> - OptimizationTime: 13.106ms
> - PeakMemoryUsage: 107.00 KB (109568)
> - PrepareTime: 19.773ms
> AGGREGATION_NODE (id=1):(Total: 334.482ms, non-child: 766.527us, %
> non-child: 0.23%)
> ExecOption: Codegen Enabled
> - BuildTime: 1.192ms
> - GetResultsTime: 0.000ns
> - HTResizeTime: 0.000ns
> - HashBuckets: 0 (0)
> - LargestPartitionPercent: 0 (0)
> - MaxPartitionLevel: 0 (0)
> - NumRepartitions: 0 (0)
> - PartitionsCreated: 0 (0)
> - PeakMemoryUsage: 20.00 KB (20480)
> - RowsRepartitioned: 0 (0)
> - RowsReturned: 1 (1)
> - RowsReturnedRate: 2.00 /sec
> - SpilledPartitions: 0 (0)
> HDFS_SCAN_NODE (id=0):(Total: 333.715ms, non-child: 333.715ms, %
> non-child: 100.00%)
> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> 0:36/21.95 MB 11:33/19.87 MB 9:27/15.16 MB 5:44/28.07 MB 6:56/33.32 MB
> 1:38/22.24 MB 10:44/27.90 MB 3:34/20.26 MB 2:45/28.21 MB 7:31/19.33 MB
> 4:42/26.73 MB 8:34/21.25 MB
> ExecOption: PARQUET Codegen Enabled, Codegen enabled: 464 out of
> 464
> Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0%
> 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
> File Formats: PARQUET/Unknown(Skipped):464
> BytesRead(500.000ms): 12.27 MB
> - FooterProcessingTime: (Avg: 13.555ms ; Min: 5.433ms ; Max:
> 62.957ms ; Number of samples: 464)
> - AverageHdfsReadThreadConcurrency: 0.00
> - AverageScannerThreadConcurrency: 24.00
> - BytesRead: 45.01 MB (47193988)
> - BytesReadDataNodeCache: 0
> - BytesReadLocal: 45.01 MB (47193988)
> - BytesReadRemoteUnexpected: 0
> - BytesReadShortCircuit: 45.01 MB (47193988)
> - CachedFileHandlesHitCount: 0 (0)
> - CachedFileHandlesMissCount: 464 (464)
> - CollectionItemsRead: 0 (0)
> - DecompressionTime: 0.000ns
> - MaxCompressedTextFileLength: 0
> - NumColumns: 0 (0)
> - NumDictFilteredRowGroups: 0 (0)
> - NumDisksAccessed: 12 (12)
> - NumRowGroups: 464 (464)
> - NumScannerThreadsStarted: 24 (24)
> - NumScannersWithNoReads: 0 (0)
> - NumStatsFilteredRowGroups: 0 (0)
> - PeakMemoryUsage: 2.38 MB (2490376)
> - PerReadThreadRawHdfsThroughput: 1016.44 MB/sec
> - RemoteScanRanges: 0 (0)
> - RowBatchQueueGetWaitTime: 320.860ms
> - RowBatchQueuePutWaitTime: 0.000ns
> - RowsRead: 0 (0)
> - RowsReturned: 464 (464)
> - RowsReturnedRate: 1.39 K/sec
> - ScanRangesComplete: 464 (464)
> - ScannerThreadsInvoluntaryContextSwitches: 108 (108)
> - ScannerThreadsTotalWallClockTime: 7s769ms
> - MaterializeTupleTime(*): 75.514us
> - ScannerThreadsSysTime: 29.984ms
> - ScannerThreadsUserTime: 669.884ms
> - ScannerThreadsVoluntaryContextSwitches: 1.60K (1597)
> - TotalRawHdfsReadTime(*): 44.279ms
> - TotalReadThroughput: 24.53 MB/sec
> Instance c843dae48ce9b289:6ad3ffc000000002
> (host=aopr-dhc003.lpdomain.com:22000):(Total: 351.014ms, non-child: 1.641ms,
> % non-child: 0.47%)
> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> 0:35/20.89 MB 7:44/26.64 MB 11:27/17.21 MB 9:32/18.61 MB 5:25/14.56 MB
> 2:38/24.32 MB 6:31/19.85 MB 8:50/30.76 MB 10:41/25.92 MB 3:31/19.78 MB
> 4:60/37.59 MB 1:44/27.19 MB
> MemoryUsage(500.000ms): 1.16 MB
> ThreadUsage(500.000ms): 25
> - AverageThreadTokens: 25.00
> - BloomFilterBytes: 0
> - PeakMemoryUsage: 2.54 MB (2662753)
> - PeakReservation: 0
> - PeakUsedReservation: 0
> - PerHostPeakMemUsage: 2.54 MB (2662753)
> - RowsProduced: 1 (1)
> - TotalNetworkReceiveTime: 0.000ns
> - TotalNetworkSendTime: 342.746us
> - TotalStorageWaitTime: 4s796ms
> - TotalThreadsInvoluntaryContextSwitches: 93 (93)
> - TotalThreadsTotalWallClockTime: 7s286ms
> - TotalThreadsSysTime: 27.988ms
> - TotalThreadsUserTime: 671.887ms
> - TotalThreadsVoluntaryContextSwitches: 2.08K (2079)
> Fragment Instance Lifecycle Timings:
> - ExecTime: 524.006us
> - ExecTreeExecTime: 7.920us
> - OpenTime: 327.802ms
> - ExecTreeOpenTime: 307.154ms
> - PrepareTime: 22.617ms
> - ExecTreePrepareTime: 2.650ms
> DataStreamSender (dst_id=2):(Total: 348.990us, non-child: 348.990us,
> % non-child: 100.00%)
> - BytesSent: 16.00 B (16)
> - NetworkThroughput(*): 354.47 KB/sec
> - OverallThroughput: 44.77 KB/sec
> - PeakMemoryUsage: 16.00 KB (16384)
> - RowsReturned: 1 (1)
> - SerializeBatchTime: 15.602us
> - TransmitDataRPCTime: 44.080us
> - UncompressedRowBatchSize: 16.00 B (16)
> CodeGen:(Total: 39.215ms, non-child: 39.215ms, % non-child: 100.00%)
> - CodegenTime: 1.183ms
> - CompileTime: 7.168ms
> - LoadTime: 0.000ns
> - ModuleBitcodeSize: 1.86 MB (1950880)
> - NumFunctions: 15 (15)
> - NumInstructions: 214 (214)
> - OptimizationTime: 12.929ms
> - PeakMemoryUsage: 107.00 KB (109568)
> - PrepareTime: 18.453ms
> AGGREGATION_NODE (id=1):(Total: 309.807ms, non-child: 948.677us, %
> non-child: 0.31%)
> ExecOption: Codegen Enabled
> - BuildTime: 1.552ms
> - GetResultsTime: 0.000ns
> - HTResizeTime: 0.000ns
> - HashBuckets: 0 (0)
> - LargestPartitionPercent: 0 (0)
> - MaxPartitionLevel: 0 (0)
> - NumRepartitions: 0 (0)
> - PartitionsCreated: 0 (0)
> - PeakMemoryUsage: 20.00 KB (20480)
> - RowsRepartitioned: 0 (0)
> - RowsReturned: 1 (1)
> - RowsReturnedRate: 3.00 /sec
> - SpilledPartitions: 0 (0)
> HDFS_SCAN_NODE (id=0):(Total: 308.859ms, non-child: 308.859ms, %
> non-child: 100.00%)
> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> 0:35/20.89 MB 7:44/26.64 MB 11:27/17.21 MB 9:32/18.61 MB 5:25/14.56 MB
> 2:38/24.32 MB 6:31/19.85 MB 8:50/30.76 MB 10:41/25.92 MB 3:31/19.78 MB
> 4:60/37.59 MB 1:44/27.19 MB
> ExecOption: PARQUET Codegen Enabled, Codegen enabled: 458 out of
> 458
> Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0%
> 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
> File Formats: PARQUET/Unknown(Skipped):458
> BytesRead(500.000ms): 36.74 MB
> - FooterProcessingTime: (Avg: 11.502ms ; Min: 3.668ms ; Max:
> 23.564ms ; Number of samples: 458)
> - AverageHdfsReadThreadConcurrency: 0.00
> - AverageScannerThreadConcurrency: 24.00
> - BytesRead: 44.51 MB (46668432)
> - BytesReadDataNodeCache: 0
> - BytesReadLocal: 44.51 MB (46668432)
> - BytesReadRemoteUnexpected: 0
> - BytesReadShortCircuit: 44.51 MB (46668432)
> - CachedFileHandlesHitCount: 0 (0)
> - CachedFileHandlesMissCount: 458 (458)
> - CollectionItemsRead: 0 (0)
> - DecompressionTime: 0.000ns
> - MaxCompressedTextFileLength: 0
> - NumColumns: 0 (0)
> - NumDictFilteredRowGroups: 0 (0)
> - NumDisksAccessed: 12 (12)
> - NumRowGroups: 458 (458)
> - NumScannerThreadsStarted: 24 (24)
> - NumScannersWithNoReads: 0 (0)
> - NumStatsFilteredRowGroups: 0 (0)
> - PeakMemoryUsage: 2.50 MB (2621440)
> - PerReadThreadRawHdfsThroughput: 1.26 GB/sec
> - RemoteScanRanges: 0 (0)
> - RowBatchQueueGetWaitTime: 295.595ms
> - RowBatchQueuePutWaitTime: 0.000ns
> - RowsRead: 0 (0)
> - RowsReturned: 458 (458)
> - RowsReturnedRate: 1.48 K/sec
> - ScanRangesComplete: 458 (458)
> - ScannerThreadsInvoluntaryContextSwitches: 91 (91)
> - ScannerThreadsTotalWallClockTime: 6s958ms
> - MaterializeTupleTime(*): 71.630us
> - ScannerThreadsSysTime: 16.989ms
> - ScannerThreadsUserTime: 645.891ms
> - ScannerThreadsVoluntaryContextSwitches: 1.63K (1630)
> - TotalRawHdfsReadTime(*): 34.377ms
> - TotalReadThroughput: 73.49 MB/sec
> Instance c843dae48ce9b289:6ad3ffc000000001
> (host=aopr-dhc005.lpdomain.com:22000):(Total: 308.195ms, non-child: 1.784ms,
> % non-child: 0.58%)
> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> 2:46/28.97 MB 8:39/24.34 MB 10:33/21.03 MB 3:43/26.50 MB 5:32/20.27 MB
> 9:32/19.86 MB 6:25/16.01 MB 7:41/24.77 MB 1:42/25.01 MB 0:55/32.51 MB
> 4:40/24.29 MB 11:33/20.77 MB
> - AverageThreadTokens: 0.00
> - BloomFilterBytes: 0
> - PeakMemoryUsage: 2.56 MB (2683241)
> - PeakReservation: 0
> - PeakUsedReservation: 0
> - PerHostPeakMemUsage: 2.56 MB (2683241)
> - RowsProduced: 1 (1)
> - TotalNetworkReceiveTime: 0.000ns
> - TotalNetworkSendTime: 292.765us
> - TotalStorageWaitTime: 4s482ms
> - TotalThreadsInvoluntaryContextSwitches: 113 (113)
> - TotalThreadsTotalWallClockTime: 6s310ms
> - TotalThreadsSysTime: 35.983ms
> - TotalThreadsUserTime: 682.884ms
> - TotalThreadsVoluntaryContextSwitches: 2.14K (2143)
> Fragment Instance Lifecycle Timings:
> - ExecTime: 532.572us
> - ExecTreeExecTime: 8.377us
> - OpenTime: 280.241ms
> - ExecTreeOpenTime: 258.652ms
> - PrepareTime: 27.342ms
> - ExecTreePrepareTime: 2.863ms
> DataStreamSender (dst_id=2):(Total: 309.794us, non-child: 309.794us,
> % non-child: 100.00%)
> - BytesSent: 16.00 B (16)
> - NetworkThroughput(*): 356.91 KB/sec
> - OverallThroughput: 50.44 KB/sec
> - PeakMemoryUsage: 16.00 KB (16384)
> - RowsReturned: 1 (1)
> - SerializeBatchTime: 14.262us
> - TransmitDataRPCTime: 43.778us
> - UncompressedRowBatchSize: 16.00 B (16)
> CodeGen:(Total: 44.584ms, non-child: 44.584ms, % non-child: 100.00%)
> - CodegenTime: 1.382ms
> - CompileTime: 7.346ms
> - LoadTime: 0.000ns
> - ModuleBitcodeSize: 1.86 MB (1950880)
> - NumFunctions: 15 (15)
> - NumInstructions: 214 (214)
> - OptimizationTime: 13.462ms
> - PeakMemoryUsage: 107.00 KB (109568)
> - PrepareTime: 23.006ms
> AGGREGATION_NODE (id=1):(Total: 261.516ms, non-child: 836.656us, %
> non-child: 0.32%)
> ExecOption: Codegen Enabled
> - BuildTime: 1.280ms
> - GetResultsTime: 0.000ns
> - HTResizeTime: 0.000ns
> - HashBuckets: 0 (0)
> - LargestPartitionPercent: 0 (0)
> - MaxPartitionLevel: 0 (0)
> - NumRepartitions: 0 (0)
> - PartitionsCreated: 0 (0)
> - PeakMemoryUsage: 20.00 KB (20480)
> - RowsRepartitioned: 0 (0)
> - RowsReturned: 1 (1)
> - RowsReturnedRate: 3.00 /sec
> - SpilledPartitions: 0 (0)
> HDFS_SCAN_NODE (id=0):(Total: 260.680ms, non-child: 260.680ms, %
> non-child: 100.00%)
> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> 2:46/28.97 MB 8:39/24.34 MB 10:33/21.03 MB 3:43/26.50 MB 5:32/20.27 MB
> 9:32/19.86 MB 6:25/16.01 MB 7:41/24.77 MB 1:42/25.01 MB 0:55/32.51 MB
> 4:40/24.29 MB 11:33/20.77 MB
> ExecOption: PARQUET Codegen Enabled, Codegen enabled: 461 out of
> 461
> Hdfs Read Thread Concurrency Bucket: 0:0% 1:0% 2:0% 3:0% 4:0% 5:0%
> 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
> File Formats: PARQUET/Unknown(Skipped):461
> - FooterProcessingTime: (Avg: 10.793ms ; Min: 3.087ms ; Max:
> 20.253ms ; Number of samples: 461)
> - AverageHdfsReadThreadConcurrency: 0.00
> - AverageScannerThreadConcurrency: 0.00
> - BytesRead: 44.73 MB (46906407)
> - BytesReadDataNodeCache: 0
> - BytesReadLocal: 44.73 MB (46906407)
> - BytesReadRemoteUnexpected: 0
> - BytesReadShortCircuit: 44.73 MB (46906407)
> - CachedFileHandlesHitCount: 0 (0)
> - CachedFileHandlesMissCount: 461 (461)
> - CollectionItemsRead: 0 (0)
> - DecompressionTime: 0.000ns
> - MaxCompressedTextFileLength: 0
> - NumColumns: 0 (0)
> - NumDictFilteredRowGroups: 0 (0)
> - NumDisksAccessed: 12 (12)
> - NumRowGroups: 461 (461)
> - NumScannerThreadsStarted: 24 (24)
> - NumScannersWithNoReads: 0 (0)
> - NumStatsFilteredRowGroups: 0 (0)
> - PeakMemoryUsage: 2.52 MB (2641928)
> - PerReadThreadRawHdfsThroughput: 1.31 GB/sec
> - RemoteScanRanges: 0 (0)
> - RowBatchQueueGetWaitTime: 246.691ms
> - RowBatchQueuePutWaitTime: 0.000ns
> - RowsRead: 0 (0)
> - RowsReturned: 461 (461)
> - RowsReturnedRate: 1.77 K/sec
> - ScanRangesComplete: 461 (461)
> - ScannerThreadsInvoluntaryContextSwitches: 109 (109)
> - ScannerThreadsTotalWallClockTime: 6s030ms
> - MaterializeTupleTime(*): 68.735us
> - ScannerThreadsSysTime: 33.983ms
> - ScannerThreadsUserTime: 647.889ms
> - ScannerThreadsVoluntaryContextSwitches: 1.70K (1696)
> - TotalRawHdfsReadTime(*): 33.405ms
> - TotalReadThroughput: 0.00 /sec
> Instance c843dae48ce9b289:6ad3ffc000000003
> (host=aopr-dhc002.lpdomain.com:22000):(Total: 302.116ms, non-child: 1.604ms,
> % non-child: 0.53%)
> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> 0:27/16.21 MB 11:33/21.23 MB 6:36/23.15 MB 10:41/25.09 MB 1:41/24.13 MB
> 9:32/18.79 MB 5:41/25.31 MB 2:42/26.49 MB 7:36/22.87 MB 3:51/31.00 MB
> 4:41/25.99 MB 8:38/23.79 MB
> MemoryUsage(500.000ms): 700.35 KB
> ThreadUsage(500.000ms): 25
> - AverageThreadTokens: 25.00
> - BloomFilterBytes: 0
> - PeakMemoryUsage: 2.54 MB (2662753)
> - PeakReservation: 0
> - PeakUsedReservation: 0
> - PerHostPeakMemUsage: 2.54 MB (2662753)
> - RowsProduced: 1 (1)
> - TotalNetworkReceiveTime: 0.000ns
> - TotalNetworkSendTime: 300.675us
> - TotalStorageWaitTime: 4s661ms
> - TotalThreadsInvoluntaryContextSwitches: 90 (90)
> - TotalThreadsTotalWallClockTime: 6s256ms
> - TotalThreadsSysTime: 30.983ms
> - TotalThreadsUserTime: 709.880ms
> - TotalThreadsVoluntaryContextSwitches: 2.17K (2175)
> Fragment Instance Lifecycle Timings:
> - ExecTime: 492.627us
> - ExecTreeExecTime: 8.191us
> - OpenTime: 277.922ms
> - ExecTreeOpenTime: 255.395ms
> - PrepareTime: 23.647ms
> - ExecTreePrepareTime: 2.243ms
> DataStreamSender (dst_id=2):(Total: 194.517us, non-child: 194.517us,
> % non-child: 100.00%)
> - BytesSent: 16.00 B (16)
> - NetworkThroughput(*): 371.86 KB/sec
> - OverallThroughput: 80.33 KB/sec
> - PeakMemoryUsage: 16.00 KB (16384)
> - RowsReturned: 1 (1)
> - SerializeBatchTime: 14.035us
> - TransmitDataRPCTime: 42.019us
> - UncompressedRowBatchSize: 16.00 B (16)
> CodeGen:(Total: 42.676ms, non-child: 42.676ms, % non-child: 100.00%)
> - CodegenTime: 1.236ms
> - CompileTime: 7.717ms
> - LoadTime: 0.000ns
> - ModuleBitcodeSize: 1.86 MB (1950880)
> - NumFunctions: 15 (15)
> - NumInstructions: 214 (214)
> - OptimizationTime: 14.130ms
> - PeakMemoryUsage: 107.00 KB (109568)
> - PrepareTime: 20.109ms
> AGGREGATION_NODE (id=1):(Total: 257.640ms, non-child: 1.541ms, %
> non-child: 0.60%)
> ExecOption: Codegen Enabled
> - BuildTime: 1.252ms
> - GetResultsTime: 0.000ns
> - HTResizeTime: 0.000ns
> - HashBuckets: 0 (0)
> - LargestPartitionPercent: 0 (0)
> - MaxPartitionLevel: 0 (0)
> - NumRepartitions: 0 (0)
> - PartitionsCreated: 0 (0)
> - PeakMemoryUsage: 20.00 KB (20480)
> - RowsRepartitioned: 0 (0)
> - RowsReturned: 1 (1)
> - RowsReturnedRate: 3.00 /sec
> - SpilledPartitions: 0 (0)
> HDFS_SCAN_NODE (id=0):(Total: 256.099ms, non-child: 256.099ms, %
> non-child: 100.00%)
> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> 0:27/16.21 MB 11:33/21.23 MB 6:36/23.15 MB 10:41/25.09 MB 1:41/24.13 MB
> 9:32/18.79 MB 5:41/25.31 MB 2:42/26.49 MB 7:36/22.87 MB 3:51/31.00 MB
> 4:41/25.99 MB 8:38/23.79 MB
> ExecOption: PARQUET Codegen Enabled, Codegen enabled: 459 out of
> 459
> Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0%
> 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
> File Formats: PARQUET/Unknown(Skipped):459
> BytesRead(500.000ms): 43.15 MB
> - FooterProcessingTime: (Avg: 11.257ms ; Min: 3.989ms ; Max:
> 23.004ms ; Number of samples: 459)
> - AverageHdfsReadThreadConcurrency: 0.00
> - AverageScannerThreadConcurrency: 24.00
> - BytesRead: 44.52 MB (46678031)
> - BytesReadDataNodeCache: 0
> - BytesReadLocal: 44.52 MB (46678031)
> - BytesReadRemoteUnexpected: 0
> - BytesReadShortCircuit: 44.52 MB (46678031)
> - CachedFileHandlesHitCount: 0 (0)
> - CachedFileHandlesMissCount: 459 (459)
> - CollectionItemsRead: 0 (0)
> - DecompressionTime: 0.000ns
> - MaxCompressedTextFileLength: 0
> - NumColumns: 0 (0)
> - NumDictFilteredRowGroups: 0 (0)
> - NumDisksAccessed: 12 (12)
> - NumRowGroups: 459 (459)
> - NumScannerThreadsStarted: 24 (24)
> - NumScannersWithNoReads: 0 (0)
> - NumStatsFilteredRowGroups: 0 (0)
> - PeakMemoryUsage: 2.50 MB (2621440)
> - PerReadThreadRawHdfsThroughput: 1.33 GB/sec
> - RemoteScanRanges: 0 (0)
> - RowBatchQueueGetWaitTime: 243.426ms
> - RowBatchQueuePutWaitTime: 0.000ns
> - RowsRead: 0 (0)
> - RowsReturned: 459 (459)
> - RowsReturnedRate: 1.79 K/sec
> - ScanRangesComplete: 459 (459)
> - ScannerThreadsInvoluntaryContextSwitches: 85 (85)
> - ScannerThreadsTotalWallClockTime: 5s978ms
> - MaterializeTupleTime(*): 67.997us
> - ScannerThreadsSysTime: 22.984ms
> - ScannerThreadsUserTime: 678.884ms
> - ScannerThreadsVoluntaryContextSwitches: 1.72K (1724)
> - TotalRawHdfsReadTime(*): 32.626ms
> - TotalReadThroughput: 86.30 MB/sec
> Instance c843dae48ce9b289:6ad3ffc000000005
> (host=aopr-dhc001.lpdomain.com:22000):(Total: 298.093ms, non-child: 1.242ms,
> % non-child: 0.42%)
> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> 9:55/35.38 MB 0:37/24.23 MB 5:37/23.31 MB 11:22/13.79 MB 8:30/18.53 MB
> 4:33/20.45 MB 1:38/24.06 MB 10:29/18.40 MB 3:39/24.35 MB 2:41/25.45 MB
> 6:42/27.03 MB 7:47/29.22 MB
> - AverageThreadTokens: 0.00
> - BloomFilterBytes: 0
> - PeakMemoryUsage: 2.66 MB (2793825)
> - PeakReservation: 0
> - PeakUsedReservation: 0
> - PerHostPeakMemUsage: 2.68 MB (2806113)
> - RowsProduced: 1 (1)
> - TotalNetworkReceiveTime: 0.000ns
> - TotalNetworkSendTime: 198.894us
> - TotalStorageWaitTime: 4s374ms
> - TotalThreadsInvoluntaryContextSwitches: 110 (110)
> - TotalThreadsTotalWallClockTime: 6s201ms
> - TotalThreadsSysTime: 29.983ms
> - TotalThreadsUserTime: 642.892ms
> - TotalThreadsVoluntaryContextSwitches: 2.17K (2174)
> Fragment Instance Lifecycle Timings:
> - ExecTime: 350.873us
> - ExecTreeExecTime: 8.173us
> - OpenTime: 275.347ms
> - ExecTreeOpenTime: 253.425ms
> - PrepareTime: 22.341ms
> - ExecTreePrepareTime: 2.251ms
> DataStreamSender (dst_id=2):(Total: 164.394us, non-child: 164.394us,
> % non-child: 100.00%)
> - BytesSent: 16.00 B (16)
> - NetworkThroughput(*): 493.42 KB/sec
> - OverallThroughput: 95.05 KB/sec
> - PeakMemoryUsage: 16.00 KB (16384)
> - RowsReturned: 1 (1)
> - SerializeBatchTime: 14.262us
> - TransmitDataRPCTime: 31.667us
> - UncompressedRowBatchSize: 16.00 B (16)
> CodeGen:(Total: 41.008ms, non-child: 41.008ms, % non-child: 100.00%)
> - CodegenTime: 1.049ms
> - CompileTime: 7.248ms
> - LoadTime: 0.000ns
> - ModuleBitcodeSize: 1.86 MB (1950880)
> - NumFunctions: 15 (15)
> - NumInstructions: 214 (214)
> - OptimizationTime: 14.018ms
> - PeakMemoryUsage: 107.00 KB (109568)
> - PrepareTime: 19.080ms
> AGGREGATION_NODE (id=1):(Total: 255.678ms, non-child: 1.046ms, %
> non-child: 0.41%)
> ExecOption: Codegen Enabled
> - BuildTime: 1.015ms
> - GetResultsTime: 0.000ns
> - HTResizeTime: 0.000ns
> - HashBuckets: 0 (0)
> - LargestPartitionPercent: 0 (0)
> - MaxPartitionLevel: 0 (0)
> - NumRepartitions: 0 (0)
> - PartitionsCreated: 0 (0)
> - PeakMemoryUsage: 20.00 KB (20480)
> - RowsRepartitioned: 0 (0)
> - RowsReturned: 1 (1)
> - RowsReturnedRate: 3.00 /sec
> - SpilledPartitions: 0 (0)
> HDFS_SCAN_NODE (id=0):(Total: 254.631ms, non-child: 254.631ms, %
> non-child: 100.00%)
> Hdfs split stats (<volume id>:<# splits>/<split lengths>):
> 9:55/35.38 MB 0:37/24.23 MB 5:37/23.31 MB 11:22/13.79 MB 8:30/18.53 MB
> 4:33/20.45 MB 1:38/24.06 MB 10:29/18.40 MB 3:39/24.35 MB 2:41/25.45 MB
> 6:42/27.03 MB 7:47/29.22 MB
> ExecOption: PARQUET Codegen Enabled, Codegen enabled: 450 out of
> 450
> Hdfs Read Thread Concurrency Bucket: 0:0% 1:0% 2:0% 3:0% 4:0% 5:0%
> 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
> File Formats: PARQUET/Unknown(Skipped):450
> - FooterProcessingTime: (Avg: 10.727ms ; Min: 3.813ms ; Max:
> 23.755ms ; Number of samples: 450)
> - AverageHdfsReadThreadConcurrency: 0.00
> - AverageScannerThreadConcurrency: 0.00
> - BytesRead: 43.85 MB (45983284)
> - BytesReadDataNodeCache: 0
> - BytesReadLocal: 43.85 MB (45983284)
> - BytesReadRemoteUnexpected: 0
> - BytesReadShortCircuit: 43.85 MB (45983284)
> - CachedFileHandlesHitCount: 0 (0)
> - CachedFileHandlesMissCount: 450 (450)
> - CollectionItemsRead: 0 (0)
> - DecompressionTime: 0.000ns
> - MaxCompressedTextFileLength: 0
> - NumColumns: 0 (0)
> - NumDictFilteredRowGroups: 0 (0)
> - NumDisksAccessed: 12 (12)
> - NumRowGroups: 450 (450)
> - NumScannerThreadsStarted: 24 (24)
> - NumScannersWithNoReads: 0 (0)
> - NumStatsFilteredRowGroups: 0 (0)
> - PeakMemoryUsage: 2.62 MB (2752512)
> - PerReadThreadRawHdfsThroughput: 1.56 GB/sec
> - RemoteScanRanges: 0 (0)
> - RowBatchQueueGetWaitTime: 242.362ms
> - RowBatchQueuePutWaitTime: 0.000ns
> - RowsRead: 0 (0)
> - RowsReturned: 450 (450)
> - RowsReturnedRate: 1.77 K/sec
> - ScanRangesComplete: 450 (450)
> - ScannerThreadsInvoluntaryContextSwitches: 106 (106)
> - ScannerThreadsTotalWallClockTime: 5s926ms
> - MaterializeTupleTime(*): 69.447us
> - ScannerThreadsSysTime: 26.984ms
> - ScannerThreadsUserTime: 607.897ms
> - ScannerThreadsVoluntaryContextSwitches: 1.72K (1723)
> - TotalRawHdfsReadTime(*): 27.393ms
> - TotalReadThroughput: 0.00 /sec
> ===================================================
>
> Coordinator log:
>
> 01: query (string) = "select count(*) from dv_entersiteevent_prq_remote
> where year=2017 and month=10",
> 03: configuration (list) = list<string>[1] {
> [0] = "REQUEST_POOL=impreports",
> },
> 04: hadoop_user (string) = "fawzea",
> }
> I0208 01:07:06.321281 7154 impala-beeswax-server.cc:459]
> TClientRequest.queryOptions: TQueryOptions {
> 01: abort_on_error (bool) = false,
> 02: max_errors (i32) = 100,
> 03: disable_codegen (bool) = false,
> 04: batch_size (i32) = 0,
> 05: num_nodes (i32) = 0,
> 06: max_scan_range_length (i64) = 0,
> 07: num_scanner_threads (i32) = 0,
> 08: max_io_buffers (i32) = 0,
> 09: allow_unsupported_formats (bool) = false,
> 10: default_order_by_limit (i64) = -1,
> 11: debug_action (string) = "",
> 12: mem_limit (i64) = 0,
> 13: abort_on_default_limit_exceeded (bool) = false,
> 15: hbase_caching (i32) = 0,
> 16: hbase_cache_blocks (bool) = false,
> 17: parquet_file_size (i64) = 0,
> 18: explain_level (i32) = 1,
> 19: sync_ddl (bool) = false,
> 20: request_pool (string) = "impreports",
> 23: disable_cached_reads (bool) = false,
> 24: disable_outermost_topn (bool) = false,
> 25: rm_initial_mem (i64) = 0,
> 26: query_timeout_s (i32) = 0,
> 28: appx_count_distinct (bool) = false,
> 29: disable_unsafe_spills (bool) = false,
> 31: exec_single_node_rows_threshold (i32) = 100,
> 32: optimize_partition_key_scans (bool) = false,
> 33: replica_preference (i32) = 0,
> 34: schedule_random_replica (bool) = false,
> 35: scan_node_codegen_threshold (i64) = 1800000,
> 36: disable_streaming_preaggregations (bool) = false,
> 37: runtime_filter_mode (i32) = 2,
> 38: runtime_bloom_filter_size (i32) = 1048576,
> 39: runtime_filter_wait_time_ms (i32) = 0,
> 40: disable_row_runtime_filtering (bool) = false,
> 41: max_num_runtime_filters (i32) = 10,
> 42: parquet_annotate_strings_utf8 (bool) = false,
> 43: parquet_fallback_schema_resolution (i32) = 0,
> 45: s3_skip_insert_staging (bool) = true,
> 46: runtime_filter_min_size (i32) = 1048576,
> 47: runtime_filter_max_size (i32) = 16777216,
> 48: prefetch_mode (i32) = 1,
> 49: strict_mode (bool) = false,
> 50: scratch_limit (i64) = -1,
> 51: enable_expr_rewrites (bool) = true,
> 52: decimal_v2 (bool) = false,
> 53: parquet_dictionary_filtering (bool) = true,
> 54: parquet_array_resolution (i32) = 2,
> 55: parquet_read_statistics (bool) = true,
> 56: default_join_distribution_mode (i32) = 0,
> 57: disable_codegen_rows_threshold (i32) = 50000,
> 58: default_spillable_buffer_size (i64) = 2097152,
> 59: min_spillable_buffer_size (i64) = 65536,
> 60: max_row_size (i64) = 524288,
> }
> I0208 01:07:06.324410 7154 Frontend.java:909] Compiling query: select
> count(*) from dv_entersiteevent_prq_remote where year=2017 and month=10
> I0208 01:07:06.327203 7154 Frontend.java:948] Compiled query.
> I0208 01:07:06.359748 7154 admission-controller.cc:510] Schedule for
> id=5f49e927591a9042:faf8f28600000000 in pool_name=root.impreports
> cluster_mem_needed=105.00 MB PoolConfig: max_requests=-1 max_queued=200
> max_mem=-1.00 B
> I0208 01:07:06.359776 7154 admission-controller.cc:515] Stats:
> agg_num_running=0, agg_num_queued=0, agg_mem_reserved=0,
> local_host(local_mem_admitted=0, num_admitted_running=0, num_queued=0,
> backend_mem_reserved=0)
> I0208 01:07:06.359784 7154 admission-controller.cc:531] Admitted query
> id=5f49e927591a9042:faf8f28600000000
> I0208 01:07:06.359807 7154 coordinator.cc:99] Exec()
> query_id=5f49e927591a9042:faf8f28600000000 stmt=select count(*) from
> dv_entersiteevent_prq_remote where year=2017 and month=10
> I0208 01:07:06.360694 7154 coordinator.cc:357] starting execution on 5
> backends for query 5f49e927591a9042:faf8f28600000000
> I0208 01:07:06.363255 63583 impala-internal-service.cc:44]
> ExecQueryFInstances(): query_id=5f49e927591a9042:faf8f28600000000
> I0208 01:07:06.363282 63583 query-exec-mgr.cc:46] StartQueryFInstances()
> query_id=5f49e927591a9042:faf8f28600000000
> coord=aopr-dhc001.lpdomain.com:22000
> I0208 01:07:06.363296 63583 query-state.cc:173] Buffer pool limit for
> 5f49e927591a9042:faf8f28600000000: 10802010521
> I0208 01:07:06.363364 63583 initial-reservations.cc:60] Successfully claimed
> initial reservations (0) for query 5f49e927591a9042:faf8f28600000000
> I0208 01:07:06.363544 7155 query-state.cc:286] StartFInstances():
> query_id=5f49e927591a9042:faf8f28600000000 #instances=2
> I0208 01:07:06.363844 7155 query-state.cc:299] descriptor table for
> query=5f49e927591a9042:faf8f28600000000
> tuples:
> Tuple(id=2 size=0 slots=[] tuple_path=[])
> Tuple(id=1 size=8 slots=[Slot(id=2 type=BIGINT col_path=[] offset=0
> null=(offset=0 mask=0) slot_idx=0 field_idx=-1)] tuple_path=[])
> Tuple(id=0 size=8 slots=[Slot(id=3 type=BIGINT col_path=[] offset=0
> null=(offset=0 mask=0) slot_idx=0 field_idx=-1)] tuple_path=[])
> I0208 01:07:06.364013 7156 query-state.cc:377] Executing instance.
> instance_id=5f49e927591a9042:faf8f28600000000 fragment_idx=0
> per_fragment_instance_idx=0 coord_state_idx=0 #in-flight=1
> I0208 01:07:06.364135 7157 query-state.cc:377] Executing instance.
> instance_id=5f49e927591a9042:faf8f28600000005 fragment_idx=1
> per_fragment_instance_idx=4 coord_state_idx=0 #in-flight=2
> I0208 01:07:06.364241 7157 hdfs-scan-node.cc:160] Max row batch queue size
> for scan node '0' in fragment instance '5f49e927591a9042:faf8f28600000005':
> 160
> I0208 01:07:06.366775 7154 coordinator.cc:370] started execution on 5
> backends for query 5f49e927591a9042:faf8f28600000000
> I0208 01:07:06.385625 7155 query-exec-mgr.cc:149] ReleaseQueryState():
> query_id=5f49e927591a9042:faf8f28600000000 refcnt=4
> I0208 01:07:06.671193 7559 impala-server.cc:1796] Connection from client
> ::ffff:10.16.136.3:55598 closed, closing 1 associated session(s)
> I0208 01:07:06.676138 544 data-stream-mgr.cc:228] Reduced stream ID cache
> from 2 items, to 0, eviction took: 0
> I0208 01:07:06.677202 544 coordinator.cc:955] Backend completed:
> host=aopr-dhc002.lpdomain.com:22000 remaining=4
> I0208 01:07:06.677214 544 coordinator-backend-state.cc:229]
> query_id=5f49e927591a9042:faf8f28600000000: first in-progress backend:
> aopr-dhc001.lpdomain.com:22000
> I0208 01:07:06.684221 7157 query-state.cc:384] Instance completed.
> instance_id=5f49e927591a9042:faf8f28600000005 #in-flight=1 status=OK
> I0208 01:07:06.684237 7157 query-exec-mgr.cc:149] ReleaseQueryState():
> query_id=5f49e927591a9042:faf8f28600000000 refcnt=3
> I0208 01:07:06.700275 520 coordinator.cc:955] Backend completed:
> host=aopr-dhc005.lpdomain.com:22000 remaining=3
> I0208 01:07:06.700289 520 coordinator-backend-state.cc:229]
> query_id=5f49e927591a9042:faf8f28600000000: first in-progress backend:
> aopr-dhc001.lpdomain.com:22000
> I0208 01:07:06.704620 65258 coordinator.cc:955] Backend completed:
> host=aopr-dhc003.lpdomain.com:22000 remaining=2
> I0208 01:07:06.704666 65258 coordinator-backend-state.cc:229]
> query_id=5f49e927591a9042:faf8f28600000000: first in-progress backend:
> aopr-dhc001.lpdomain.com:22000
> I0208 01:07:06.706041 7156 data-stream-mgr.cc:238] DeregisterRecvr():
> fragment_instance_id=5f49e927591a9042:faf8f28600000000, node=2
> I0208 01:07:06.706070 7156 data-stream-recvr.cc:235] cancelled stream:
> fragment_instance_id_=5f49e927591a9042:faf8f28600000000 node_id=2
> I0208 01:07:06.706809 64916 coordinator.cc:955] Backend completed:
> host=aopr-dhc004.lpdomain.com:22000 remaining=1
> I0208 01:07:06.706821 64916 coordinator-backend-state.cc:229]
> query_id=5f49e927591a9042:faf8f28600000000: first in-progress backend:
> aopr-dhc001.lpdomain.com:22000
> I0208 01:07:06.789568 7154 impala-beeswax-server.cc:193]
> get_results_metadata(): query_id=5f49e927591a9042:faf8f28600000000
> I0208 01:07:06.790439 7154 coordinator.cc:789] Coordinator waiting for
> backends to finish, 1 remaining
> I0208 01:07:06.791052 7154 coordinator.cc:794] All backends finished
> successfully.
> I0208 01:07:06.791086 7154 coordinator.cc:1090] Release admssion control
> resources for query 5f49e927591a9042:faf8f28600000000
> I0208 01:07:06.791363 7156 query-state.cc:384] Instance completed.
> instance_id=5f49e927591a9042:faf8f28600000000 #in-flight=0 status=OK
> I0208 01:07:06.791414 7156 query-exec-mgr.cc:149] ReleaseQueryState():
> query_id=5f49e927591a9042:faf8f28600000000 refcnt=2
> I0208 01:07:06.793586 7154 impala-beeswax-server.cc:235] close():
> query_id=5f49e927591a9042:faf8f28600000000
> I0208 01:07:06.793602 7154 impala-server.cc:992] UnregisterQuery():
> query_id=5f49e927591a9042:faf8f28600000000
> I0208 01:07:06.793614 7154 impala-server.cc:1075] Cancel():
> query_id=5f49e927591a9042:faf8f28600000000
> I0208 01:07:06.793624 7154 coordinator.cc:895] Cancel()
> query_id=5f49e927591a9042:faf8f28600000000
> I0208 01:07:06.793637 7154 coordinator.cc:905] CancelBackends()
> query_id=5f49e927591a9042:faf8f28600000000, tried to cancel 0 backends
> I0208 01:07:06.802776 7154 query-exec-mgr.cc:149] ReleaseQueryState():
> query_id=5f49e927591a9042:faf8f28600000000 refcnt=1
> I0208 01:07:06.814214 7154 impala-server.cc:1796] Connection from client
> ::ffff:10.16.144.131:51012 closed, closing 1 associated session(s)
> I0208 01:07:07.258891 7567 impala-server.cc:1796] Connection from client
> ::ffff:10.16.136.2:56375 closed, closing 1 associated session(s)
> I0208 01:07:11.600131 7819 impala-server.cc:1796] Connection from client
> ::ffff:10.16.136.3:27460 closed, closing 1 associated session(s)
> I0208 01:07:12.262739 7821 impala-server.cc:1796] Connection from client
> ::ffff:10.16.136.2:52672 closed, closing 1 associated session(s)
> I0208 01:07:16.665451 7822 impala-server.cc:1796] Connection from client
> ::ffff:10.16.136.3:10693 closed, closing 1 associated session(s)
> I0208 01:07:17.235321 7825 impala-server.cc:1796] Connection from client
> ::ffff:10.16.136.2:34752 closed, closing 1 associated session(s)
> I0208 01:07:21.410611 5858 impala-beeswax-server.cc:193]
> get_results_metadata(): query_id=1d4fbf0f2571f1e9:ea346f7900000000
> I0208 01:07:21.414175 5858 impala-beeswax-server.cc:235] close():
> query_id=1d4fbf0f2571f1e9:ea346f7900000000
> I0208 01:07:21.414185 5858 impala-server.cc:992] UnregisterQuery():
> query_id=1d4fbf0f2571f1e9:ea346f7900000000
> I0208 01:07:21.414191 5858 impala-server.cc:1075] Cancel():
> query_id=1d4fbf0f2571f1e9:ea346f7900000000
> I0208 01:07:21.421505 5858 impala-server.cc:1796] Connection from client
> ::ffff:10.26.180.75:60072 closed, closing 1 associated session(s)
> I0208 01:07:21.605690 7830 impala-server.cc:1796] Connection from client
> ::ffff:10.16.136.3:46660 closed, closing 1 associated session(s)
>
>
> On Thu, Feb 8, 2018 at 6:20 AM, Matthew Jacobs <mj...@apache.org> wrote:
>>
>> Thanks! I think that configuration looks OK.
>>
>> 1) What version of Impala is this?
>>
>> 2) Can you send the query profile and the Impala coordinator log for
>> the query where it's not behaving as expected?
>>
>> > When i'm running the following with fawzea user:
>> > set request_pool=impreports;
>> > select count(*) from dv_entersiteevent_prq_remote where year=2017 and
>> > month=10;
>> > I can see that the query is running with impreports pool.
>>
>> On Wed, Feb 7, 2018 at 7:59 PM, Fawze Abujaber <fa...@gmail.com> wrote:
>> > Hi Matt,
>> >
>> > [root@aopr-dhc001 ~]# cat
>> >
>> > /var/run/cloudera-scm-agent/process/4558-impala-IMPALAD/impala-conf/fair-scheduler.xml
>> > <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
>> > <allocations>
>> > <queue name="root">
>> > <weight>1.0</weight>
>> > <schedulingPolicy>fair</schedulingPolicy>
>> > <queue name="impiaas">
>> > <weight>1.0</weight>
>> > <schedulingPolicy>fair</schedulingPolicy>
>> > <aclSubmitApps>fawzea </aclSubmitApps>
>> > </queue>
>> > <queue name="impreports">
>> > <weight>1.0</weight>
>> > <schedulingPolicy>fair</schedulingPolicy>
>> > <aclSubmitApps>nira </aclSubmitApps>
>> > </queue>
>> > </queue>
>> > <defaultQueueSchedulingPolicy>fair</defaultQueueSchedulingPolicy>
>> > <queuePlacementPolicy>
>> > <rule name="specified" create="false"/>
>> > <rule name="secondaryGroupExistingQueue"/>
>> > <rule name="default"/>
>> > </queuePlacementPolicy>
>> > </allocations>
>> >
>> >
>> > On Thu, Feb 8, 2018 at 4:47 AM, Matthew Jacobs <ja...@gmail.com>
>> > wrote:
>> >>
>> >> Yes, I saw that, but I'd like to see the actual configuration that is
>> >> passed to Impala because it may not be specified in the right way to
>> >> accomplish what you're trying to do in "b".
>> >>
>> >> On Wed, Feb 7, 2018 at 4:32 PM, Fawze Abujaber <fa...@gmail.com>
>> >> wrote:
>> >> > Hi Matt,
>> >> >
>> >> > I created the pools manually and it’s completely nfigurations as
>> >> > shown
>> >> > in b.
>> >> >
>> >> > On Wed, 7 Feb 2018 at 23:17 Matthew Jacobs <ja...@gmail.com>
>> >> > wrote:
>> >> >>
>> >> >> How, more specifically, are you configuring these pools? Can you
>> >> >> share
>> >> >> the configuration files?
>> >> >>
>> >> >> On Wed, Feb 7, 2018 at 11:42 AM, Fawze Abujaber <fa...@gmail.com>
>> >> >> wrote:
>> >> >> > Hello All,
>> >> >> >
>> >> >> > Will appreciate any insight of what i'm missing in my
>> >> >> > configuration
>> >> >> > to
>> >> >> > get
>> >> >> > the impala submission control working as expected ....
>> >> >> >
>> >> >> > a- Here is my placement Placement Rule:
>> >> >> > 1- Use the pool Specified at run time., only if the pool exists.
>> >> >> > 2- Use the pool root.[secondary group].
>> >> >> > 3- Use the pool root.default.
>> >> >> >
>> >> >> > b- Here is the impala resource pools
>> >> >> >
>> >> >> > Name Max Memory Max Running Queries Max Queued
>> >> >> > Queries
>> >> >> > Queue Timeout Default Query Memory Limit Access Control
>> >> >> > root.impiaas Unlimited Unlimited
>> >> >> > 200 1 minute(s) No Default
>> >> >> > Custom
>> >> >> > root.impreports Unlimited Unlimited
>> >> >> > 200 1 minute(s) No Default
>> >> >> > Custom
>> >> >> >
>> >> >> >
>> >> >> > C- Here is my Access control for root.impiaas pool:
>> >> >> >
>> >> >> > Allow these users and groups to submit to this pool
>> >> >> >
>> >> >> > Users: fawzea
>> >> >> >
>> >> >> > The same for impreports the allowed user and groups to submit to
>> >> >> > pool
>> >> >> > is
>> >> >> > only nira.
>> >> >> >
>> >> >> >
>> >> >> > LDAP groups for fawzea user:
>> >> >> >
>> >> >> > Member in groups:
>> >> >> >
>> >> >> > - dba
>> >> >> > - impiaas
>> >> >> > - mysql
>> >> >> >
>> >> >> > LDAP group for nira user
>> >> >> > Member in groups:
>> >> >> > - dba
>> >> >> > - impreports
>> >> >> > - mysql
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > Impala Queries UI shows:
>> >> >> >
>> >> >> > 1- select count(*) from dv_entersiteevent_prq_remote where
>> >> >> > year=2017
>> >> >> >
>> >> >> > User: fawzea
>> >> >> > Pool: root.impiaas
>> >> >> >
>> >> >> > 2- select count(*) from dv_entersiteevent_prq_remote where
>> >> >> > year=2017
>> >> >> > and
>> >> >> > month=10
>> >> >> > User: nira
>> >> >> > Pool: root.impreports
>> >> >> >
>> >> >> > As shown above that the queries submitted based on the secondary
>> >> >> > group
>> >> >> > rule.
>> >> >> >
>> >> >> > When i'm running the following with fawzea user:
>> >> >> >
>> >> >> > set request_pool=impreports;
>> >> >> > select count(*) from dv_entersiteevent_prq_remote where year=2017
>> >> >> > and
>> >> >> > month=10;
>> >> >> >
>> >> >> > I can see that the query is running with impreports pool.
>> >> >> >
>> >> >> >
>> >> >> > What i'm missing here in my configuration to get this works as
>> >> >> > expected.
>> >> >> >
>> >
>> >
>
>
Re: impala submission control
Posted by Fawze Abujaber <fa...@gmail.com>.
Hi Matt,
v2.11.0-cdh5.14.0
Query Runtime Profile:
Query (id=c843dae48ce9b289:6ad3ffc000000000):
Summary:
Session ID: ca420ac477772b56:51ae460ce9e44999
Session Type: BEESWAX
Start Time: 2018-02-08 01:02:01.023558000
End Time: 2018-02-08 01:02:01.501506000
Query Type: QUERY
Query State: FINISHED
Query Status: OK
Impala Version: impalad version 2.11.0-cdh5.14.0 RELEASE (build
d68206561bce6b26762d62c01a78e6cd27aa7690)
User: fawzea
Connected User: fawzea
Delegated User:
Network Address: ::ffff:10.16.144.131:50078
Default Db: default
Sql Statement: select count(*) from dv_entersiteevent_prq_remote where
year=2017 and month=10
Coordinator: aopr-dhc001.lpdomain.com:22000
Query Options (set by configuration): REQUEST_POOL=impreports
Query Options (set by configuration and planner):
REQUEST_POOL=impreports,MT_DOP=0
Plan:
----------------
Max Per-Host Resource Reservation: Memory=0B
Per-Host Resource Estimates: Memory=21.00MB
WARNING: The following tables are missing relevant table and/or column
statistics.
default.dv_entersiteevent_prq_remote
F01:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1
| Per-Host Resources: mem-estimate=10.00MB mem-reservation=0B
PLAN-ROOT SINK
| mem-estimate=0B mem-reservation=0B
|
03:AGGREGATE [FINALIZE]
| output: count:merge(*)
| mem-estimate=10.00MB mem-reservation=0B spill-buffer=2.00MB
| tuple-ids=1 row-size=8B cardinality=1
|
02:EXCHANGE [UNPARTITIONED]
| mem-estimate=0B mem-reservation=0B
| tuple-ids=1 row-size=8B cardinality=1
|
F00:PLAN FRAGMENT [RANDOM] hosts=5 instances=5
Per-Host Resources: mem-estimate=11.00MB mem-reservation=0B
01:AGGREGATE
| output:
sum_init_zero(default.dv_entersiteevent_prq_remote.parquet-stats: num_rows)
| mem-estimate=10.00MB mem-reservation=0B spill-buffer=2.00MB
| tuple-ids=1 row-size=8B cardinality=1
|
00:SCAN HDFS [default.dv_entersiteevent_prq_remote, RANDOM]
partitions=31/441 files=2292 size=1.39GB
stats-rows=unavailable extrapolated-rows=disabled
table stats: rows=unavailable size=unavailable
column stats: all
mem-estimate=1.00MB mem-reservation=0B
tuple-ids=0 row-size=8B cardinality=unavailable
----------------
Estimated Per-Host Mem: 22020096
Tables Missing Stats: default.dv_entersiteevent_prq_remote
Per Host Min Reservation: aopr-dhc001.lpdomain.com:22000(0)
aopr-dhc002.lpdomain.com:22000(0) aopr-dhc003.lpdomain.com:22000(0)
aopr-dhc004.lpdomain.com:22000(0) aopr-dhc005.lpdomain.com:22000(0)
Request Pool: root.impreports
Admission result: Admitted immediately
ExecSummary:
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem
Est. Peak Mem Detail
------------------------------------------------------------------------------------------------------------------------------
03:AGGREGATE 1 222.958us 222.958us 1 1 36.00 KB
10.00 MB FINALIZE
02:EXCHANGE 1 357.508ms 357.508ms 5 1 0
0 UNPARTITIONED
01:AGGREGATE 5 1.027ms 1.541ms 5 1 20.00 KB
10.00 MB
00:SCAN HDFS 5 282.797ms 333.715ms 2.29K -1 2.62 MB
1.00 MB default.dv_entersiteevent_prq_remote
Errors:
Planner Timeline: 13.508ms
- Analysis finished: 3.303ms (3.303ms)
- Value transfer graph computed: 3.465ms (162.667us)
- Single node plan created: 10.352ms (6.886ms)
- Runtime filters computed: 10.677ms (325.364us)
- Distributed plan created: 10.733ms (55.854us)
- Lineage info computed: 10.898ms (164.937us)
- Planning finished: 13.508ms (2.610ms)
Query Timeline: 481.950ms
- Query submitted: 79.971us (79.971us)
- Planning finished: 33.947ms (33.867ms)
- Submit for admission: 40.067ms (6.120ms)
- Completed admission: 40.197ms (130.146us)
- Ready to start on 5 backends: 40.905ms (707.852us)
- All 5 execution backends (6 fragment instances) started: 46.417ms
(5.512ms)
- Rows available: 424.548ms (378.131ms)
- First row fetched: 473.488ms (48.939ms)
- Unregister query: 477.951ms (4.462ms)
- ComputeScanRangeAssignmentTimer: 3.980ms
ImpalaServer:
- ClientFetchWaitTimer: 51.659ms
- RowMaterializationTimer: 1.726ms
Execution Profile c843dae48ce9b289:6ad3ffc000000000:(Total: 385.029ms,
non-child: 0.000ns, % non-child: 0.00%)
Number of filters: 0
Filter routing table:
ID Src. Node Tgt. Node(s) Target type Partition filter Pending
(Expected) First arrived Completed Enabled
-------------------------------------------------------------------------------------------------------------------
Backend startup latencies: Count: 5, min / max: 3ms / 4ms, 25th %-ile:
3ms, 50th %-ile: 4ms, 75th %-ile: 4ms, 90th %-ile: 4ms, 95th %-ile: 4ms,
99.9th %-ile: 4ms
Per Node Peak Memory Usage: aopr-dhc001.lpdomain.com:22000(2.68 MB)
aopr-dhc002.lpdomain.com:22000(2.54 MB) aopr-dhc003.lpdomain.com:22000(2.54
MB) aopr-dhc004.lpdomain.com:22000(2.41 MB) aopr-dhc005.lpdomain.com:22000(2.56
MB)
- FiltersReceived: 0 (0)
- FinalizationTimer: 0.000ns
- NumBackends: 5 (5)
- NumFragmentInstances: 6 (6)
- NumFragments: 2 (2)
Averaged Fragment F01:(Total: 427.530ms, non-child: 49.428ms, %
non-child: 11.56%)
split sizes: min: 0, max: 0, avg: 0, stddev: 0
completion times: min:428.693ms max:428.693ms mean: 428.693ms
stddev:0.000ns
execution rates: min:0.00 /sec max:0.00 /sec mean:0.00 /sec
stddev:0.00 /sec
num instances: 1
- AverageThreadTokens: 0.00
- BloomFilterBytes: 0
- PeakMemoryUsage: 44.02 KB (45072)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 2.68 MB (2806113)
- RowsProduced: 1 (1)
- TotalNetworkReceiveTime: 357.481ms
- TotalNetworkSendTime: 0.000ns
- TotalStorageWaitTime: 0.000ns
- TotalThreadsInvoluntaryContextSwitches: 0 (0)
- TotalThreadsTotalWallClockTime: 406.708ms
- TotalThreadsSysTime: 0.000ns
- TotalThreadsUserTime: 0.000ns
- TotalThreadsVoluntaryContextSwitches: 7 (7)
Fragment Instance Lifecycle Timings:
- ExecTime: 49.055ms
- ExecTreeExecTime: 4.251us
- OpenTime: 357.663ms
- ExecTreeOpenTime: 357.649ms
- PrepareTime: 20.764ms
- ExecTreePrepareTime: 81.759us
PLAN_ROOT_SINK:
- PeakMemoryUsage: 0
CodeGen:(Total: 20.369ms, non-child: 20.369ms, % non-child: 100.00%)
- CodegenTime: 0.000ns
- CompileTime: 0.000ns
- LoadTime: 0.000ns
- ModuleBitcodeSize: 1.86 MB (1950880)
- NumFunctions: 0 (0)
- NumInstructions: 0 (0)
- OptimizationTime: 0.000ns
- PeakMemoryUsage: 0
- PrepareTime: 19.685ms
AGGREGATION_NODE (id=3):(Total: 357.731ms, non-child: 222.958us, %
non-child: 0.06%)
- BuildTime: 7.368us
- GetResultsTime: 0.000ns
- HTResizeTime: 0.000ns
- HashBuckets: 0 (0)
- LargestPartitionPercent: 0 (0)
- MaxPartitionLevel: 0 (0)
- NumRepartitions: 0 (0)
- PartitionsCreated: 0 (0)
- PeakMemoryUsage: 36.00 KB (36864)
- RowsRepartitioned: 0 (0)
- RowsReturned: 1 (1)
- RowsReturnedRate: 2.00 /sec
- SpilledPartitions: 0 (0)
EXCHANGE_NODE (id=2):(Total: 357.508ms, non-child: 357.508ms, %
non-child: 100.00%)
- ConvertRowBatchTime: 8.881us
- PeakMemoryUsage: 0
- RowsReturned: 5 (5)
- RowsReturnedRate: 13.00 /sec
DataStreamReceiver:
- BytesReceived: 80.00 B (80)
- DeserializeRowBatchTimer: 24.165us
- FirstBatchArrivalWaitTime: 277.194ms
- PeakMemoryUsage: 4.02 KB (4112)
- SendersBlockedTimer: 0.000ns
- SendersBlockedTotalTimer(*): 0.000ns
Coordinator Fragment F01:
Instance c843dae48ce9b289:6ad3ffc000000000
(host=aopr-dhc001.lpdomain.com:22000):(Total: 427.530ms, non-child:
49.428ms, % non-child: 11.56%)
- AverageThreadTokens: 0.00
- BloomFilterBytes: 0
- PeakMemoryUsage: 44.02 KB (45072)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 2.68 MB (2806113)
- RowsProduced: 1 (1)
- TotalNetworkReceiveTime: 357.481ms
- TotalNetworkSendTime: 0.000ns
- TotalStorageWaitTime: 0.000ns
- TotalThreadsInvoluntaryContextSwitches: 0 (0)
- TotalThreadsTotalWallClockTime: 406.708ms
- TotalThreadsSysTime: 0.000ns
- TotalThreadsUserTime: 0.000ns
- TotalThreadsVoluntaryContextSwitches: 7 (7)
Fragment Instance Lifecycle Timings:
- ExecTime: 49.055ms
- ExecTreeExecTime: 4.251us
- OpenTime: 357.663ms
- ExecTreeOpenTime: 357.649ms
- PrepareTime: 20.764ms
- ExecTreePrepareTime: 81.759us
PLAN_ROOT_SINK:
- PeakMemoryUsage: 0
CodeGen:(Total: 20.369ms, non-child: 20.369ms, % non-child: 100.00%)
- CodegenTime: 0.000ns
- CompileTime: 0.000ns
- LoadTime: 0.000ns
- ModuleBitcodeSize: 1.86 MB (1950880)
- NumFunctions: 0 (0)
- NumInstructions: 0 (0)
- OptimizationTime: 0.000ns
- PeakMemoryUsage: 0
- PrepareTime: 19.685ms
AGGREGATION_NODE (id=3):(Total: 357.731ms, non-child: 222.958us, %
non-child: 0.06%)
- BuildTime: 7.368us
- GetResultsTime: 0.000ns
- HTResizeTime: 0.000ns
- HashBuckets: 0 (0)
- LargestPartitionPercent: 0 (0)
- MaxPartitionLevel: 0 (0)
- NumRepartitions: 0 (0)
- PartitionsCreated: 0 (0)
- PeakMemoryUsage: 36.00 KB (36864)
- RowsRepartitioned: 0 (0)
- RowsReturned: 1 (1)
- RowsReturnedRate: 2.00 /sec
- SpilledPartitions: 0 (0)
EXCHANGE_NODE (id=2):(Total: 357.508ms, non-child: 357.508ms, %
non-child: 100.00%)
- ConvertRowBatchTime: 8.881us
- PeakMemoryUsage: 0
- RowsReturned: 5 (5)
- RowsReturnedRate: 13.00 /sec
DataStreamReceiver:
- BytesReceived: 80.00 B (80)
- DeserializeRowBatchTimer: 24.165us
- FirstBatchArrivalWaitTime: 277.194ms
- PeakMemoryUsage: 4.02 KB (4112)
- SendersBlockedTimer: 0.000ns
- SendersBlockedTotalTimer(*): 0.000ns
Averaged Fragment F00:(Total: 327.360ms, non-child: 1.578ms, %
non-child: 0.48%)
split sizes: min: 283.31 MB, max: 284.33 MB, avg: 284.04 MB, stddev:
382.00 KB
completion times: min:299.457ms max:378.950ms mean: 328.881ms
stddev:31.507ms
execution rates: min:750.18 MB/sec max:949.01 MB/sec mean:871.27
MB/sec stddev:79.74 MB/sec
num instances: 5
- AverageThreadTokens: 15.00
- BloomFilterBytes: 0
- PeakMemoryUsage: 2.54 MB (2666852)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 2.55 MB (2669309)
- RowsProduced: 1 (1)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 303.541us
- TotalStorageWaitTime: 4s821ms
- TotalThreadsInvoluntaryContextSwitches: 104 (104)
- TotalThreadsTotalWallClockTime: 6s835ms
- TotalThreadsSysTime: 31.784ms
- TotalThreadsUserTime: 682.084ms
- TotalThreadsVoluntaryContextSwitches: 2.12K (2121)
Fragment Instance Lifecycle Timings:
- ExecTime: 500.816us
- ExecTreeExecTime: 8.200us
- OpenTime: 302.848ms
- ExecTreeOpenTime: 281.309ms
- PrepareTime: 23.946ms
- ExecTreePrepareTime: 2.513ms
DataStreamSender (dst_id=2):(Total: 269.902us, non-child: 269.902us,
% non-child: 100.00%)
- BytesSent: 16.00 B (16)
- NetworkThroughput(*): 373.13 KB/sec
- OverallThroughput: 63.53 KB/sec
- PeakMemoryUsage: 16.00 KB (16384)
- RowsReturned: 1 (1)
- SerializeBatchTime: 14.525us
- TransmitDataRPCTime: 43.122us
- UncompressedRowBatchSize: 16.00 B (16)
CodeGen:(Total: 41.687ms, non-child: 41.687ms, % non-child: 100.00%)
- CodegenTime: 1.194ms
- CompileTime: 7.360ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 1.86 MB (1950880)
- NumFunctions: 15 (15)
- NumInstructions: 214 (214)
- OptimizationTime: 13.529ms
- PeakMemoryUsage: 107.00 KB (109568)
- PrepareTime: 20.084ms
AGGREGATION_NODE (id=1):(Total: 283.825ms, non-child: 1.027ms, %
non-child: 0.36%)
- BuildTime: 1.259ms
- GetResultsTime: 0.000ns
- HTResizeTime: 0.000ns
- HashBuckets: 0 (0)
- LargestPartitionPercent: 0 (0)
- MaxPartitionLevel: 0 (0)
- NumRepartitions: 0 (0)
- PartitionsCreated: 0 (0)
- PeakMemoryUsage: 20.00 KB (20480)
- RowsRepartitioned: 0 (0)
- RowsReturned: 1 (1)
- RowsReturnedRate: 2.00 /sec
- SpilledPartitions: 0 (0)
HDFS_SCAN_NODE (id=0):(Total: 282.797ms, non-child: 282.797ms, %
non-child: 100.00%)
- AverageHdfsReadThreadConcurrency: 0.00
- AverageScannerThreadConcurrency: 14.40
- BytesRead: 44.52 MB (46686028)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 44.52 MB (46686028)
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 44.52 MB (46686028)
- CachedFileHandlesHitCount: 0 (0)
- CachedFileHandlesMissCount: 458 (458)
- CollectionItemsRead: 0 (0)
- DecompressionTime: 0.000ns
- MaxCompressedTextFileLength: 0
- NumColumns: 0 (0)
- NumDictFilteredRowGroups: 0 (0)
- NumDisksAccessed: 12 (12)
- NumRowGroups: 458 (458)
- NumScannerThreadsStarted: 24 (24)
- NumScannersWithNoReads: 0 (0)
- NumStatsFilteredRowGroups: 0 (0)
- PeakMemoryUsage: 2.50 MB (2625539)
- PerReadThreadRawHdfsThroughput: 1.29 GB/sec
- RemoteScanRanges: 0 (0)
- RowBatchQueueGetWaitTime: 269.787ms
- RowBatchQueuePutWaitTime: 0.000ns
- RowsRead: 0 (0)
- RowsReturned: 458 (458)
- RowsReturnedRate: 1.64 K/sec
- ScanRangesComplete: 458 (458)
- ScannerThreadsInvoluntaryContextSwitches: 99 (99)
- ScannerThreadsTotalWallClockTime: 6s532ms
- MaterializeTupleTime(*): 70.664us
- ScannerThreadsSysTime: 26.184ms
- ScannerThreadsUserTime: 650.089ms
- ScannerThreadsVoluntaryContextSwitches: 1.67K (1674)
- TotalRawHdfsReadTime(*): 34.416ms
- TotalReadThroughput: 36.86 MB/sec
Fragment F00:
Instance c843dae48ce9b289:6ad3ffc000000004
(host=aopr-dhc004.lpdomain.com:22000):(Total: 377.384ms, non-child:
1.619ms, % non-child: 0.43%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
0:36/21.95 MB 11:33/19.87 MB 9:27/15.16 MB 5:44/28.07 MB 6:56/33.32 MB
1:38/22.24 MB 10:44/27.90 MB 3:34/20.26 MB 2:45/28.21 MB 7:31/19.33 MB
4:42/26.73 MB 8:34/21.25 MB
MemoryUsage(500.000ms): 1.91 MB
ThreadUsage(500.000ms): 25
- AverageThreadTokens: 25.00
- BloomFilterBytes: 0
- PeakMemoryUsage: 2.41 MB (2531689)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 2.41 MB (2531689)
- RowsProduced: 1 (1)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 382.629us
- TotalStorageWaitTime: 5s790ms
- TotalThreadsInvoluntaryContextSwitches: 115 (115)
- TotalThreadsTotalWallClockTime: 8s122ms
- TotalThreadsSysTime: 33.984ms
- TotalThreadsUserTime: 702.879ms
- TotalThreadsVoluntaryContextSwitches: 2.04K (2037)
Fragment Instance Lifecycle Timings:
- ExecTime: 604.005us
- ExecTreeExecTime: 8.338us
- OpenTime: 352.927ms
- ExecTreeOpenTime: 331.921ms
- PrepareTime: 23.783ms
- ExecTreePrepareTime: 2.559ms
DataStreamSender (dst_id=2):(Total: 331.818us, non-child:
331.818us, % non-child: 100.00%)
- BytesSent: 16.00 B (16)
- NetworkThroughput(*): 289.00 KB/sec
- OverallThroughput: 47.09 KB/sec
- PeakMemoryUsage: 16.00 KB (16384)
- RowsReturned: 1 (1)
- SerializeBatchTime: 14.466us
- TransmitDataRPCTime: 54.066us
- UncompressedRowBatchSize: 16.00 B (16)
CodeGen:(Total: 40.950ms, non-child: 40.950ms, % non-child: 100.00%)
- CodegenTime: 1.122ms
- CompileTime: 7.318ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 1.86 MB (1950880)
- NumFunctions: 15 (15)
- NumInstructions: 214 (214)
- OptimizationTime: 13.106ms
- PeakMemoryUsage: 107.00 KB (109568)
- PrepareTime: 19.773ms
AGGREGATION_NODE (id=1):(Total: 334.482ms, non-child: 766.527us, %
non-child: 0.23%)
ExecOption: Codegen Enabled
- BuildTime: 1.192ms
- GetResultsTime: 0.000ns
- HTResizeTime: 0.000ns
- HashBuckets: 0 (0)
- LargestPartitionPercent: 0 (0)
- MaxPartitionLevel: 0 (0)
- NumRepartitions: 0 (0)
- PartitionsCreated: 0 (0)
- PeakMemoryUsage: 20.00 KB (20480)
- RowsRepartitioned: 0 (0)
- RowsReturned: 1 (1)
- RowsReturnedRate: 2.00 /sec
- SpilledPartitions: 0 (0)
HDFS_SCAN_NODE (id=0):(Total: 333.715ms, non-child: 333.715ms, %
non-child: 100.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
0:36/21.95 MB 11:33/19.87 MB 9:27/15.16 MB 5:44/28.07 MB 6:56/33.32 MB
1:38/22.24 MB 10:44/27.90 MB 3:34/20.26 MB 2:45/28.21 MB 7:31/19.33 MB
4:42/26.73 MB 8:34/21.25 MB
ExecOption: PARQUET Codegen Enabled, Codegen enabled: 464 out of
464
Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0%
5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
File Formats: PARQUET/Unknown(Skipped):464
BytesRead(500.000ms): 12.27 MB
- FooterProcessingTime: (Avg: 13.555ms ; Min: 5.433ms ; Max:
62.957ms ; Number of samples: 464)
- AverageHdfsReadThreadConcurrency: 0.00
- AverageScannerThreadConcurrency: 24.00
- BytesRead: 45.01 MB (47193988)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 45.01 MB (47193988)
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 45.01 MB (47193988)
- CachedFileHandlesHitCount: 0 (0)
- CachedFileHandlesMissCount: 464 (464)
- CollectionItemsRead: 0 (0)
- DecompressionTime: 0.000ns
- MaxCompressedTextFileLength: 0
- NumColumns: 0 (0)
- NumDictFilteredRowGroups: 0 (0)
- NumDisksAccessed: 12 (12)
- NumRowGroups: 464 (464)
- NumScannerThreadsStarted: 24 (24)
- NumScannersWithNoReads: 0 (0)
- NumStatsFilteredRowGroups: 0 (0)
- PeakMemoryUsage: 2.38 MB (2490376)
- PerReadThreadRawHdfsThroughput: 1016.44 MB/sec
- RemoteScanRanges: 0 (0)
- RowBatchQueueGetWaitTime: 320.860ms
- RowBatchQueuePutWaitTime: 0.000ns
- RowsRead: 0 (0)
- RowsReturned: 464 (464)
- RowsReturnedRate: 1.39 K/sec
- ScanRangesComplete: 464 (464)
- ScannerThreadsInvoluntaryContextSwitches: 108 (108)
- ScannerThreadsTotalWallClockTime: 7s769ms
- MaterializeTupleTime(*): 75.514us
- ScannerThreadsSysTime: 29.984ms
- ScannerThreadsUserTime: 669.884ms
- ScannerThreadsVoluntaryContextSwitches: 1.60K (1597)
- TotalRawHdfsReadTime(*): 44.279ms
- TotalReadThroughput: 24.53 MB/sec
Instance c843dae48ce9b289:6ad3ffc000000002
(host=aopr-dhc003.lpdomain.com:22000):(Total: 351.014ms, non-child:
1.641ms, % non-child: 0.47%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
0:35/20.89 MB 7:44/26.64 MB 11:27/17.21 MB 9:32/18.61 MB 5:25/14.56 MB
2:38/24.32 MB 6:31/19.85 MB 8:50/30.76 MB 10:41/25.92 MB 3:31/19.78 MB
4:60/37.59 MB 1:44/27.19 MB
MemoryUsage(500.000ms): 1.16 MB
ThreadUsage(500.000ms): 25
- AverageThreadTokens: 25.00
- BloomFilterBytes: 0
- PeakMemoryUsage: 2.54 MB (2662753)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 2.54 MB (2662753)
- RowsProduced: 1 (1)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 342.746us
- TotalStorageWaitTime: 4s796ms
- TotalThreadsInvoluntaryContextSwitches: 93 (93)
- TotalThreadsTotalWallClockTime: 7s286ms
- TotalThreadsSysTime: 27.988ms
- TotalThreadsUserTime: 671.887ms
- TotalThreadsVoluntaryContextSwitches: 2.08K (2079)
Fragment Instance Lifecycle Timings:
- ExecTime: 524.006us
- ExecTreeExecTime: 7.920us
- OpenTime: 327.802ms
- ExecTreeOpenTime: 307.154ms
- PrepareTime: 22.617ms
- ExecTreePrepareTime: 2.650ms
DataStreamSender (dst_id=2):(Total: 348.990us, non-child:
348.990us, % non-child: 100.00%)
- BytesSent: 16.00 B (16)
- NetworkThroughput(*): 354.47 KB/sec
- OverallThroughput: 44.77 KB/sec
- PeakMemoryUsage: 16.00 KB (16384)
- RowsReturned: 1 (1)
- SerializeBatchTime: 15.602us
- TransmitDataRPCTime: 44.080us
- UncompressedRowBatchSize: 16.00 B (16)
CodeGen:(Total: 39.215ms, non-child: 39.215ms, % non-child: 100.00%)
- CodegenTime: 1.183ms
- CompileTime: 7.168ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 1.86 MB (1950880)
- NumFunctions: 15 (15)
- NumInstructions: 214 (214)
- OptimizationTime: 12.929ms
- PeakMemoryUsage: 107.00 KB (109568)
- PrepareTime: 18.453ms
AGGREGATION_NODE (id=1):(Total: 309.807ms, non-child: 948.677us, %
non-child: 0.31%)
ExecOption: Codegen Enabled
- BuildTime: 1.552ms
- GetResultsTime: 0.000ns
- HTResizeTime: 0.000ns
- HashBuckets: 0 (0)
- LargestPartitionPercent: 0 (0)
- MaxPartitionLevel: 0 (0)
- NumRepartitions: 0 (0)
- PartitionsCreated: 0 (0)
- PeakMemoryUsage: 20.00 KB (20480)
- RowsRepartitioned: 0 (0)
- RowsReturned: 1 (1)
- RowsReturnedRate: 3.00 /sec
- SpilledPartitions: 0 (0)
HDFS_SCAN_NODE (id=0):(Total: 308.859ms, non-child: 308.859ms, %
non-child: 100.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
0:35/20.89 MB 7:44/26.64 MB 11:27/17.21 MB 9:32/18.61 MB 5:25/14.56 MB
2:38/24.32 MB 6:31/19.85 MB 8:50/30.76 MB 10:41/25.92 MB 3:31/19.78 MB
4:60/37.59 MB 1:44/27.19 MB
ExecOption: PARQUET Codegen Enabled, Codegen enabled: 458 out of
458
Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0%
5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
File Formats: PARQUET/Unknown(Skipped):458
BytesRead(500.000ms): 36.74 MB
- FooterProcessingTime: (Avg: 11.502ms ; Min: 3.668ms ; Max:
23.564ms ; Number of samples: 458)
- AverageHdfsReadThreadConcurrency: 0.00
- AverageScannerThreadConcurrency: 24.00
- BytesRead: 44.51 MB (46668432)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 44.51 MB (46668432)
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 44.51 MB (46668432)
- CachedFileHandlesHitCount: 0 (0)
- CachedFileHandlesMissCount: 458 (458)
- CollectionItemsRead: 0 (0)
- DecompressionTime: 0.000ns
- MaxCompressedTextFileLength: 0
- NumColumns: 0 (0)
- NumDictFilteredRowGroups: 0 (0)
- NumDisksAccessed: 12 (12)
- NumRowGroups: 458 (458)
- NumScannerThreadsStarted: 24 (24)
- NumScannersWithNoReads: 0 (0)
- NumStatsFilteredRowGroups: 0 (0)
- PeakMemoryUsage: 2.50 MB (2621440)
- PerReadThreadRawHdfsThroughput: 1.26 GB/sec
- RemoteScanRanges: 0 (0)
- RowBatchQueueGetWaitTime: 295.595ms
- RowBatchQueuePutWaitTime: 0.000ns
- RowsRead: 0 (0)
- RowsReturned: 458 (458)
- RowsReturnedRate: 1.48 K/sec
- ScanRangesComplete: 458 (458)
- ScannerThreadsInvoluntaryContextSwitches: 91 (91)
- ScannerThreadsTotalWallClockTime: 6s958ms
- MaterializeTupleTime(*): 71.630us
- ScannerThreadsSysTime: 16.989ms
- ScannerThreadsUserTime: 645.891ms
- ScannerThreadsVoluntaryContextSwitches: 1.63K (1630)
- TotalRawHdfsReadTime(*): 34.377ms
- TotalReadThroughput: 73.49 MB/sec
Instance c843dae48ce9b289:6ad3ffc000000001
(host=aopr-dhc005.lpdomain.com:22000):(Total: 308.195ms, non-child:
1.784ms, % non-child: 0.58%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
2:46/28.97 MB 8:39/24.34 MB 10:33/21.03 MB 3:43/26.50 MB 5:32/20.27 MB
9:32/19.86 MB 6:25/16.01 MB 7:41/24.77 MB 1:42/25.01 MB 0:55/32.51 MB
4:40/24.29 MB 11:33/20.77 MB
- AverageThreadTokens: 0.00
- BloomFilterBytes: 0
- PeakMemoryUsage: 2.56 MB (2683241)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 2.56 MB (2683241)
- RowsProduced: 1 (1)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 292.765us
- TotalStorageWaitTime: 4s482ms
- TotalThreadsInvoluntaryContextSwitches: 113 (113)
- TotalThreadsTotalWallClockTime: 6s310ms
- TotalThreadsSysTime: 35.983ms
- TotalThreadsUserTime: 682.884ms
- TotalThreadsVoluntaryContextSwitches: 2.14K (2143)
Fragment Instance Lifecycle Timings:
- ExecTime: 532.572us
- ExecTreeExecTime: 8.377us
- OpenTime: 280.241ms
- ExecTreeOpenTime: 258.652ms
- PrepareTime: 27.342ms
- ExecTreePrepareTime: 2.863ms
DataStreamSender (dst_id=2):(Total: 309.794us, non-child:
309.794us, % non-child: 100.00%)
- BytesSent: 16.00 B (16)
- NetworkThroughput(*): 356.91 KB/sec
- OverallThroughput: 50.44 KB/sec
- PeakMemoryUsage: 16.00 KB (16384)
- RowsReturned: 1 (1)
- SerializeBatchTime: 14.262us
- TransmitDataRPCTime: 43.778us
- UncompressedRowBatchSize: 16.00 B (16)
CodeGen:(Total: 44.584ms, non-child: 44.584ms, % non-child: 100.00%)
- CodegenTime: 1.382ms
- CompileTime: 7.346ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 1.86 MB (1950880)
- NumFunctions: 15 (15)
- NumInstructions: 214 (214)
- OptimizationTime: 13.462ms
- PeakMemoryUsage: 107.00 KB (109568)
- PrepareTime: 23.006ms
AGGREGATION_NODE (id=1):(Total: 261.516ms, non-child: 836.656us, %
non-child: 0.32%)
ExecOption: Codegen Enabled
- BuildTime: 1.280ms
- GetResultsTime: 0.000ns
- HTResizeTime: 0.000ns
- HashBuckets: 0 (0)
- LargestPartitionPercent: 0 (0)
- MaxPartitionLevel: 0 (0)
- NumRepartitions: 0 (0)
- PartitionsCreated: 0 (0)
- PeakMemoryUsage: 20.00 KB (20480)
- RowsRepartitioned: 0 (0)
- RowsReturned: 1 (1)
- RowsReturnedRate: 3.00 /sec
- SpilledPartitions: 0 (0)
HDFS_SCAN_NODE (id=0):(Total: 260.680ms, non-child: 260.680ms, %
non-child: 100.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
2:46/28.97 MB 8:39/24.34 MB 10:33/21.03 MB 3:43/26.50 MB 5:32/20.27 MB
9:32/19.86 MB 6:25/16.01 MB 7:41/24.77 MB 1:42/25.01 MB 0:55/32.51 MB
4:40/24.29 MB 11:33/20.77 MB
ExecOption: PARQUET Codegen Enabled, Codegen enabled: 461 out of
461
Hdfs Read Thread Concurrency Bucket: 0:0% 1:0% 2:0% 3:0% 4:0%
5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
File Formats: PARQUET/Unknown(Skipped):461
- FooterProcessingTime: (Avg: 10.793ms ; Min: 3.087ms ; Max:
20.253ms ; Number of samples: 461)
- AverageHdfsReadThreadConcurrency: 0.00
- AverageScannerThreadConcurrency: 0.00
- BytesRead: 44.73 MB (46906407)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 44.73 MB (46906407)
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 44.73 MB (46906407)
- CachedFileHandlesHitCount: 0 (0)
- CachedFileHandlesMissCount: 461 (461)
- CollectionItemsRead: 0 (0)
- DecompressionTime: 0.000ns
- MaxCompressedTextFileLength: 0
- NumColumns: 0 (0)
- NumDictFilteredRowGroups: 0 (0)
- NumDisksAccessed: 12 (12)
- NumRowGroups: 461 (461)
- NumScannerThreadsStarted: 24 (24)
- NumScannersWithNoReads: 0 (0)
- NumStatsFilteredRowGroups: 0 (0)
- PeakMemoryUsage: 2.52 MB (2641928)
- PerReadThreadRawHdfsThroughput: 1.31 GB/sec
- RemoteScanRanges: 0 (0)
- RowBatchQueueGetWaitTime: 246.691ms
- RowBatchQueuePutWaitTime: 0.000ns
- RowsRead: 0 (0)
- RowsReturned: 461 (461)
- RowsReturnedRate: 1.77 K/sec
- ScanRangesComplete: 461 (461)
- ScannerThreadsInvoluntaryContextSwitches: 109 (109)
- ScannerThreadsTotalWallClockTime: 6s030ms
- MaterializeTupleTime(*): 68.735us
- ScannerThreadsSysTime: 33.983ms
- ScannerThreadsUserTime: 647.889ms
- ScannerThreadsVoluntaryContextSwitches: 1.70K (1696)
- TotalRawHdfsReadTime(*): 33.405ms
- TotalReadThroughput: 0.00 /sec
Instance c843dae48ce9b289:6ad3ffc000000003
(host=aopr-dhc002.lpdomain.com:22000):(Total: 302.116ms, non-child:
1.604ms, % non-child: 0.53%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
0:27/16.21 MB 11:33/21.23 MB 6:36/23.15 MB 10:41/25.09 MB 1:41/24.13 MB
9:32/18.79 MB 5:41/25.31 MB 2:42/26.49 MB 7:36/22.87 MB 3:51/31.00 MB
4:41/25.99 MB 8:38/23.79 MB
MemoryUsage(500.000ms): 700.35 KB
ThreadUsage(500.000ms): 25
- AverageThreadTokens: 25.00
- BloomFilterBytes: 0
- PeakMemoryUsage: 2.54 MB (2662753)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 2.54 MB (2662753)
- RowsProduced: 1 (1)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 300.675us
- TotalStorageWaitTime: 4s661ms
- TotalThreadsInvoluntaryContextSwitches: 90 (90)
- TotalThreadsTotalWallClockTime: 6s256ms
- TotalThreadsSysTime: 30.983ms
- TotalThreadsUserTime: 709.880ms
- TotalThreadsVoluntaryContextSwitches: 2.17K (2175)
Fragment Instance Lifecycle Timings:
- ExecTime: 492.627us
- ExecTreeExecTime: 8.191us
- OpenTime: 277.922ms
- ExecTreeOpenTime: 255.395ms
- PrepareTime: 23.647ms
- ExecTreePrepareTime: 2.243ms
DataStreamSender (dst_id=2):(Total: 194.517us, non-child:
194.517us, % non-child: 100.00%)
- BytesSent: 16.00 B (16)
- NetworkThroughput(*): 371.86 KB/sec
- OverallThroughput: 80.33 KB/sec
- PeakMemoryUsage: 16.00 KB (16384)
- RowsReturned: 1 (1)
- SerializeBatchTime: 14.035us
- TransmitDataRPCTime: 42.019us
- UncompressedRowBatchSize: 16.00 B (16)
CodeGen:(Total: 42.676ms, non-child: 42.676ms, % non-child: 100.00%)
- CodegenTime: 1.236ms
- CompileTime: 7.717ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 1.86 MB (1950880)
- NumFunctions: 15 (15)
- NumInstructions: 214 (214)
- OptimizationTime: 14.130ms
- PeakMemoryUsage: 107.00 KB (109568)
- PrepareTime: 20.109ms
AGGREGATION_NODE (id=1):(Total: 257.640ms, non-child: 1.541ms, %
non-child: 0.60%)
ExecOption: Codegen Enabled
- BuildTime: 1.252ms
- GetResultsTime: 0.000ns
- HTResizeTime: 0.000ns
- HashBuckets: 0 (0)
- LargestPartitionPercent: 0 (0)
- MaxPartitionLevel: 0 (0)
- NumRepartitions: 0 (0)
- PartitionsCreated: 0 (0)
- PeakMemoryUsage: 20.00 KB (20480)
- RowsRepartitioned: 0 (0)
- RowsReturned: 1 (1)
- RowsReturnedRate: 3.00 /sec
- SpilledPartitions: 0 (0)
HDFS_SCAN_NODE (id=0):(Total: 256.099ms, non-child: 256.099ms, %
non-child: 100.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
0:27/16.21 MB 11:33/21.23 MB 6:36/23.15 MB 10:41/25.09 MB 1:41/24.13 MB
9:32/18.79 MB 5:41/25.31 MB 2:42/26.49 MB 7:36/22.87 MB 3:51/31.00 MB
4:41/25.99 MB 8:38/23.79 MB
ExecOption: PARQUET Codegen Enabled, Codegen enabled: 459 out of
459
Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0%
5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
File Formats: PARQUET/Unknown(Skipped):459
BytesRead(500.000ms): 43.15 MB
- FooterProcessingTime: (Avg: 11.257ms ; Min: 3.989ms ; Max:
23.004ms ; Number of samples: 459)
- AverageHdfsReadThreadConcurrency: 0.00
- AverageScannerThreadConcurrency: 24.00
- BytesRead: 44.52 MB (46678031)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 44.52 MB (46678031)
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 44.52 MB (46678031)
- CachedFileHandlesHitCount: 0 (0)
- CachedFileHandlesMissCount: 459 (459)
- CollectionItemsRead: 0 (0)
- DecompressionTime: 0.000ns
- MaxCompressedTextFileLength: 0
- NumColumns: 0 (0)
- NumDictFilteredRowGroups: 0 (0)
- NumDisksAccessed: 12 (12)
- NumRowGroups: 459 (459)
- NumScannerThreadsStarted: 24 (24)
- NumScannersWithNoReads: 0 (0)
- NumStatsFilteredRowGroups: 0 (0)
- PeakMemoryUsage: 2.50 MB (2621440)
- PerReadThreadRawHdfsThroughput: 1.33 GB/sec
- RemoteScanRanges: 0 (0)
- RowBatchQueueGetWaitTime: 243.426ms
- RowBatchQueuePutWaitTime: 0.000ns
- RowsRead: 0 (0)
- RowsReturned: 459 (459)
- RowsReturnedRate: 1.79 K/sec
- ScanRangesComplete: 459 (459)
- ScannerThreadsInvoluntaryContextSwitches: 85 (85)
- ScannerThreadsTotalWallClockTime: 5s978ms
- MaterializeTupleTime(*): 67.997us
- ScannerThreadsSysTime: 22.984ms
- ScannerThreadsUserTime: 678.884ms
- ScannerThreadsVoluntaryContextSwitches: 1.72K (1724)
- TotalRawHdfsReadTime(*): 32.626ms
- TotalReadThroughput: 86.30 MB/sec
Instance c843dae48ce9b289:6ad3ffc000000005
(host=aopr-dhc001.lpdomain.com:22000):(Total: 298.093ms, non-child:
1.242ms, % non-child: 0.42%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
9:55/35.38 MB 0:37/24.23 MB 5:37/23.31 MB 11:22/13.79 MB 8:30/18.53 MB
4:33/20.45 MB 1:38/24.06 MB 10:29/18.40 MB 3:39/24.35 MB 2:41/25.45 MB
6:42/27.03 MB 7:47/29.22 MB
- AverageThreadTokens: 0.00
- BloomFilterBytes: 0
- PeakMemoryUsage: 2.66 MB (2793825)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 2.68 MB (2806113)
- RowsProduced: 1 (1)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 198.894us
- TotalStorageWaitTime: 4s374ms
- TotalThreadsInvoluntaryContextSwitches: 110 (110)
- TotalThreadsTotalWallClockTime: 6s201ms
- TotalThreadsSysTime: 29.983ms
- TotalThreadsUserTime: 642.892ms
- TotalThreadsVoluntaryContextSwitches: 2.17K (2174)
Fragment Instance Lifecycle Timings:
- ExecTime: 350.873us
- ExecTreeExecTime: 8.173us
- OpenTime: 275.347ms
- ExecTreeOpenTime: 253.425ms
- PrepareTime: 22.341ms
- ExecTreePrepareTime: 2.251ms
DataStreamSender (dst_id=2):(Total: 164.394us, non-child:
164.394us, % non-child: 100.00%)
- BytesSent: 16.00 B (16)
- NetworkThroughput(*): 493.42 KB/sec
- OverallThroughput: 95.05 KB/sec
- PeakMemoryUsage: 16.00 KB (16384)
- RowsReturned: 1 (1)
- SerializeBatchTime: 14.262us
- TransmitDataRPCTime: 31.667us
- UncompressedRowBatchSize: 16.00 B (16)
CodeGen:(Total: 41.008ms, non-child: 41.008ms, % non-child: 100.00%)
- CodegenTime: 1.049ms
- CompileTime: 7.248ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 1.86 MB (1950880)
- NumFunctions: 15 (15)
- NumInstructions: 214 (214)
- OptimizationTime: 14.018ms
- PeakMemoryUsage: 107.00 KB (109568)
- PrepareTime: 19.080ms
AGGREGATION_NODE (id=1):(Total: 255.678ms, non-child: 1.046ms, %
non-child: 0.41%)
ExecOption: Codegen Enabled
- BuildTime: 1.015ms
- GetResultsTime: 0.000ns
- HTResizeTime: 0.000ns
- HashBuckets: 0 (0)
- LargestPartitionPercent: 0 (0)
- MaxPartitionLevel: 0 (0)
- NumRepartitions: 0 (0)
- PartitionsCreated: 0 (0)
- PeakMemoryUsage: 20.00 KB (20480)
- RowsRepartitioned: 0 (0)
- RowsReturned: 1 (1)
- RowsReturnedRate: 3.00 /sec
- SpilledPartitions: 0 (0)
HDFS_SCAN_NODE (id=0):(Total: 254.631ms, non-child: 254.631ms, %
non-child: 100.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
9:55/35.38 MB 0:37/24.23 MB 5:37/23.31 MB 11:22/13.79 MB 8:30/18.53 MB
4:33/20.45 MB 1:38/24.06 MB 10:29/18.40 MB 3:39/24.35 MB 2:41/25.45 MB
6:42/27.03 MB 7:47/29.22 MB
ExecOption: PARQUET Codegen Enabled, Codegen enabled: 450 out of
450
Hdfs Read Thread Concurrency Bucket: 0:0% 1:0% 2:0% 3:0% 4:0%
5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
File Formats: PARQUET/Unknown(Skipped):450
- FooterProcessingTime: (Avg: 10.727ms ; Min: 3.813ms ; Max:
23.755ms ; Number of samples: 450)
- AverageHdfsReadThreadConcurrency: 0.00
- AverageScannerThreadConcurrency: 0.00
- BytesRead: 43.85 MB (45983284)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 43.85 MB (45983284)
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 43.85 MB (45983284)
- CachedFileHandlesHitCount: 0 (0)
- CachedFileHandlesMissCount: 450 (450)
- CollectionItemsRead: 0 (0)
- DecompressionTime: 0.000ns
- MaxCompressedTextFileLength: 0
- NumColumns: 0 (0)
- NumDictFilteredRowGroups: 0 (0)
- NumDisksAccessed: 12 (12)
- NumRowGroups: 450 (450)
- NumScannerThreadsStarted: 24 (24)
- NumScannersWithNoReads: 0 (0)
- NumStatsFilteredRowGroups: 0 (0)
- PeakMemoryUsage: 2.62 MB (2752512)
- PerReadThreadRawHdfsThroughput: 1.56 GB/sec
- RemoteScanRanges: 0 (0)
- RowBatchQueueGetWaitTime: 242.362ms
- RowBatchQueuePutWaitTime: 0.000ns
- RowsRead: 0 (0)
- RowsReturned: 450 (450)
- RowsReturnedRate: 1.77 K/sec
- ScanRangesComplete: 450 (450)
- ScannerThreadsInvoluntaryContextSwitches: 106 (106)
- ScannerThreadsTotalWallClockTime: 5s926ms
- MaterializeTupleTime(*): 69.447us
- ScannerThreadsSysTime: 26.984ms
- ScannerThreadsUserTime: 607.897ms
- ScannerThreadsVoluntaryContextSwitches: 1.72K (1723)
- TotalRawHdfsReadTime(*): 27.393ms
- TotalReadThroughput: 0.00 /sec
===================================================
Coordinator log:
01: query (string) = "select count(*) from dv_entersiteevent_prq_remote
where year=2017 and month=10",
03: configuration (list) = list<string>[1] {
[0] = "REQUEST_POOL=impreports",
},
04: hadoop_user (string) = "fawzea",
}
I0208 01:07:06.321281 7154 impala-beeswax-server.cc:459]
TClientRequest.queryOptions: TQueryOptions {
01: abort_on_error (bool) = false,
02: max_errors (i32) = 100,
03: disable_codegen (bool) = false,
04: batch_size (i32) = 0,
05: num_nodes (i32) = 0,
06: max_scan_range_length (i64) = 0,
07: num_scanner_threads (i32) = 0,
08: max_io_buffers (i32) = 0,
09: allow_unsupported_formats (bool) = false,
10: default_order_by_limit (i64) = -1,
11: debug_action (string) = "",
12: mem_limit (i64) = 0,
13: abort_on_default_limit_exceeded (bool) = false,
15: hbase_caching (i32) = 0,
16: hbase_cache_blocks (bool) = false,
17: parquet_file_size (i64) = 0,
18: explain_level (i32) = 1,
19: sync_ddl (bool) = false,
20: request_pool (string) = "impreports",
23: disable_cached_reads (bool) = false,
24: disable_outermost_topn (bool) = false,
25: rm_initial_mem (i64) = 0,
26: query_timeout_s (i32) = 0,
28: appx_count_distinct (bool) = false,
29: disable_unsafe_spills (bool) = false,
31: exec_single_node_rows_threshold (i32) = 100,
32: optimize_partition_key_scans (bool) = false,
33: replica_preference (i32) = 0,
34: schedule_random_replica (bool) = false,
35: scan_node_codegen_threshold (i64) = 1800000,
36: disable_streaming_preaggregations (bool) = false,
37: runtime_filter_mode (i32) = 2,
38: runtime_bloom_filter_size (i32) = 1048576,
39: runtime_filter_wait_time_ms (i32) = 0,
40: disable_row_runtime_filtering (bool) = false,
41: max_num_runtime_filters (i32) = 10,
42: parquet_annotate_strings_utf8 (bool) = false,
43: parquet_fallback_schema_resolution (i32) = 0,
45: s3_skip_insert_staging (bool) = true,
46: runtime_filter_min_size (i32) = 1048576,
47: runtime_filter_max_size (i32) = 16777216,
48: prefetch_mode (i32) = 1,
49: strict_mode (bool) = false,
50: scratch_limit (i64) = -1,
51: enable_expr_rewrites (bool) = true,
52: decimal_v2 (bool) = false,
53: parquet_dictionary_filtering (bool) = true,
54: parquet_array_resolution (i32) = 2,
55: parquet_read_statistics (bool) = true,
56: default_join_distribution_mode (i32) = 0,
57: disable_codegen_rows_threshold (i32) = 50000,
58: default_spillable_buffer_size (i64) = 2097152,
59: min_spillable_buffer_size (i64) = 65536,
60: max_row_size (i64) = 524288,
}
I0208 01:07:06.324410 7154 Frontend.java:909] Compiling query: select
count(*) from dv_entersiteevent_prq_remote where year=2017 and month=10
I0208 01:07:06.327203 7154 Frontend.java:948] Compiled query.
I0208 01:07:06.359748 7154 admission-controller.cc:510] Schedule for
id=5f49e927591a9042:faf8f28600000000 in pool_name=root.impreports
cluster_mem_needed=105.00 MB PoolConfig: max_requests=-1 max_queued=200
max_mem=-1.00 B
I0208 01:07:06.359776 7154 admission-controller.cc:515] Stats:
agg_num_running=0, agg_num_queued=0, agg_mem_reserved=0,
local_host(local_mem_admitted=0, num_admitted_running=0, num_queued=0,
backend_mem_reserved=0)
I0208 01:07:06.359784 7154 admission-controller.cc:531] Admitted query
id=5f49e927591a9042:faf8f28600000000
I0208 01:07:06.359807 7154 coordinator.cc:99] Exec()
query_id=5f49e927591a9042:faf8f28600000000 stmt=select count(*) from
dv_entersiteevent_prq_remote where year=2017 and month=10
I0208 01:07:06.360694 7154 coordinator.cc:357] starting execution on 5
backends for query 5f49e927591a9042:faf8f28600000000
I0208 01:07:06.363255 63583 impala-internal-service.cc:44]
ExecQueryFInstances(): query_id=5f49e927591a9042:faf8f28600000000
I0208 01:07:06.363282 63583 query-exec-mgr.cc:46] StartQueryFInstances()
query_id=5f49e927591a9042:faf8f28600000000 coord=
aopr-dhc001.lpdomain.com:22000
I0208 01:07:06.363296 63583 query-state.cc:173] Buffer pool limit for
5f49e927591a9042:faf8f28600000000: 10802010521
I0208 01:07:06.363364 63583 initial-reservations.cc:60] Successfully
claimed initial reservations (0) for query 5f49e927591a9042:faf8f28600000000
I0208 01:07:06.363544 7155 query-state.cc:286] StartFInstances():
query_id=5f49e927591a9042:faf8f28600000000 #instances=2
I0208 01:07:06.363844 7155 query-state.cc:299] descriptor table for
query=5f49e927591a9042:faf8f28600000000
tuples:
Tuple(id=2 size=0 slots=[] tuple_path=[])
Tuple(id=1 size=8 slots=[Slot(id=2 type=BIGINT col_path=[] offset=0
null=(offset=0 mask=0) slot_idx=0 field_idx=-1)] tuple_path=[])
Tuple(id=0 size=8 slots=[Slot(id=3 type=BIGINT col_path=[] offset=0
null=(offset=0 mask=0) slot_idx=0 field_idx=-1)] tuple_path=[])
I0208 01:07:06.364013 7156 query-state.cc:377] Executing instance.
instance_id=5f49e927591a9042:faf8f28600000000 fragment_idx=0
per_fragment_instance_idx=0 coord_state_idx=0 #in-flight=1
I0208 01:07:06.364135 7157 query-state.cc:377] Executing instance.
instance_id=5f49e927591a9042:faf8f28600000005 fragment_idx=1
per_fragment_instance_idx=4 coord_state_idx=0 #in-flight=2
I0208 01:07:06.364241 7157 hdfs-scan-node.cc:160] Max row batch queue size
for scan node '0' in fragment instance '5f49e927591a9042:faf8f28600000005':
160
I0208 01:07:06.366775 7154 coordinator.cc:370] started execution on 5
backends for query 5f49e927591a9042:faf8f28600000000
I0208 01:07:06.385625 7155 query-exec-mgr.cc:149] ReleaseQueryState():
query_id=5f49e927591a9042:faf8f28600000000 refcnt=4
I0208 01:07:06.671193 7559 impala-server.cc:1796] Connection from client
::ffff:10.16.136.3:55598 closed, closing 1 associated session(s)
I0208 01:07:06.676138 544 data-stream-mgr.cc:228] Reduced stream ID cache
from 2 items, to 0, eviction took: 0
I0208 01:07:06.677202 544 coordinator.cc:955] Backend completed: host=
aopr-dhc002.lpdomain.com:22000 remaining=4
I0208 01:07:06.677214 544 coordinator-backend-state.cc:229]
query_id=5f49e927591a9042:faf8f28600000000: first in-progress backend:
aopr-dhc001.lpdomain.com:22000
I0208 01:07:06.684221 7157 query-state.cc:384] Instance completed.
instance_id=5f49e927591a9042:faf8f28600000005 #in-flight=1 status=OK
I0208 01:07:06.684237 7157 query-exec-mgr.cc:149] ReleaseQueryState():
query_id=5f49e927591a9042:faf8f28600000000 refcnt=3
I0208 01:07:06.700275 520 coordinator.cc:955] Backend completed: host=
aopr-dhc005.lpdomain.com:22000 remaining=3
I0208 01:07:06.700289 520 coordinator-backend-state.cc:229]
query_id=5f49e927591a9042:faf8f28600000000: first in-progress backend:
aopr-dhc001.lpdomain.com:22000
I0208 01:07:06.704620 65258 coordinator.cc:955] Backend completed: host=
aopr-dhc003.lpdomain.com:22000 remaining=2
I0208 01:07:06.704666 65258 coordinator-backend-state.cc:229]
query_id=5f49e927591a9042:faf8f28600000000: first in-progress backend:
aopr-dhc001.lpdomain.com:22000
I0208 01:07:06.706041 7156 data-stream-mgr.cc:238] DeregisterRecvr():
fragment_instance_id=5f49e927591a9042:faf8f28600000000, node=2
I0208 01:07:06.706070 7156 data-stream-recvr.cc:235] cancelled stream:
fragment_instance_id_=5f49e927591a9042:faf8f28600000000 node_id=2
I0208 01:07:06.706809 64916 coordinator.cc:955] Backend completed: host=
aopr-dhc004.lpdomain.com:22000 remaining=1
I0208 01:07:06.706821 64916 coordinator-backend-state.cc:229]
query_id=5f49e927591a9042:faf8f28600000000: first in-progress backend:
aopr-dhc001.lpdomain.com:22000
I0208 01:07:06.789568 7154 impala-beeswax-server.cc:193]
get_results_metadata(): query_id=5f49e927591a9042:faf8f28600000000
I0208 01:07:06.790439 7154 coordinator.cc:789] Coordinator waiting for
backends to finish, 1 remaining
I0208 01:07:06.791052 7154 coordinator.cc:794] All backends finished
successfully.
I0208 01:07:06.791086 7154 coordinator.cc:1090] Release admssion control
resources for query 5f49e927591a9042:faf8f28600000000
I0208 01:07:06.791363 7156 query-state.cc:384] Instance completed.
instance_id=5f49e927591a9042:faf8f28600000000 #in-flight=0 status=OK
I0208 01:07:06.791414 7156 query-exec-mgr.cc:149] ReleaseQueryState():
query_id=5f49e927591a9042:faf8f28600000000 refcnt=2
I0208 01:07:06.793586 7154 impala-beeswax-server.cc:235] close():
query_id=5f49e927591a9042:faf8f28600000000
I0208 01:07:06.793602 7154 impala-server.cc:992] UnregisterQuery():
query_id=5f49e927591a9042:faf8f28600000000
I0208 01:07:06.793614 7154 impala-server.cc:1075] Cancel():
query_id=5f49e927591a9042:faf8f28600000000
I0208 01:07:06.793624 7154 coordinator.cc:895] Cancel()
query_id=5f49e927591a9042:faf8f28600000000
I0208 01:07:06.793637 7154 coordinator.cc:905] CancelBackends()
query_id=5f49e927591a9042:faf8f28600000000, tried to cancel 0 backends
I0208 01:07:06.802776 7154 query-exec-mgr.cc:149] ReleaseQueryState():
query_id=5f49e927591a9042:faf8f28600000000 refcnt=1
I0208 01:07:06.814214 7154 impala-server.cc:1796] Connection from client
::ffff:10.16.144.131:51012 closed, closing 1 associated session(s)
I0208 01:07:07.258891 7567 impala-server.cc:1796] Connection from client
::ffff:10.16.136.2:56375 closed, closing 1 associated session(s)
I0208 01:07:11.600131 7819 impala-server.cc:1796] Connection from client
::ffff:10.16.136.3:27460 closed, closing 1 associated session(s)
I0208 01:07:12.262739 7821 impala-server.cc:1796] Connection from client
::ffff:10.16.136.2:52672 closed, closing 1 associated session(s)
I0208 01:07:16.665451 7822 impala-server.cc:1796] Connection from client
::ffff:10.16.136.3:10693 closed, closing 1 associated session(s)
I0208 01:07:17.235321 7825 impala-server.cc:1796] Connection from client
::ffff:10.16.136.2:34752 closed, closing 1 associated session(s)
I0208 01:07:21.410611 5858 impala-beeswax-server.cc:193]
get_results_metadata(): query_id=1d4fbf0f2571f1e9:ea346f7900000000
I0208 01:07:21.414175 5858 impala-beeswax-server.cc:235] close():
query_id=1d4fbf0f2571f1e9:ea346f7900000000
I0208 01:07:21.414185 5858 impala-server.cc:992] UnregisterQuery():
query_id=1d4fbf0f2571f1e9:ea346f7900000000
I0208 01:07:21.414191 5858 impala-server.cc:1075] Cancel():
query_id=1d4fbf0f2571f1e9:ea346f7900000000
I0208 01:07:21.421505 5858 impala-server.cc:1796] Connection from client
::ffff:10.26.180.75:60072 closed, closing 1 associated session(s)
I0208 01:07:21.605690 7830 impala-server.cc:1796] Connection from client
::ffff:10.16.136.3:46660 closed, closing 1 associated session(s)
On Thu, Feb 8, 2018 at 6:20 AM, Matthew Jacobs <mj...@apache.org> wrote:
> Thanks! I think that configuration looks OK.
>
> 1) What version of Impala is this?
>
> 2) Can you send the query profile and the Impala coordinator log for
> the query where it's not behaving as expected?
>
> > When i'm running the following with fawzea user:
> > set request_pool=impreports;
> > select count(*) from dv_entersiteevent_prq_remote where year=2017 and
> month=10;
> > I can see that the query is running with impreports pool.
>
> On Wed, Feb 7, 2018 at 7:59 PM, Fawze Abujaber <fa...@gmail.com> wrote:
> > Hi Matt,
> >
> > [root@aopr-dhc001 ~]# cat
> > /var/run/cloudera-scm-agent/process/4558-impala-IMPALAD/
> impala-conf/fair-scheduler.xml
> > <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
> > <allocations>
> > <queue name="root">
> > <weight>1.0</weight>
> > <schedulingPolicy>fair</schedulingPolicy>
> > <queue name="impiaas">
> > <weight>1.0</weight>
> > <schedulingPolicy>fair</schedulingPolicy>
> > <aclSubmitApps>fawzea </aclSubmitApps>
> > </queue>
> > <queue name="impreports">
> > <weight>1.0</weight>
> > <schedulingPolicy>fair</schedulingPolicy>
> > <aclSubmitApps>nira </aclSubmitApps>
> > </queue>
> > </queue>
> > <defaultQueueSchedulingPolicy>fair</defaultQueueSchedulingPolicy>
> > <queuePlacementPolicy>
> > <rule name="specified" create="false"/>
> > <rule name="secondaryGroupExistingQueue"/>
> > <rule name="default"/>
> > </queuePlacementPolicy>
> > </allocations>
> >
> >
> > On Thu, Feb 8, 2018 at 4:47 AM, Matthew Jacobs <ja...@gmail.com>
> wrote:
> >>
> >> Yes, I saw that, but I'd like to see the actual configuration that is
> >> passed to Impala because it may not be specified in the right way to
> >> accomplish what you're trying to do in "b".
> >>
> >> On Wed, Feb 7, 2018 at 4:32 PM, Fawze Abujaber <fa...@gmail.com>
> wrote:
> >> > Hi Matt,
> >> >
> >> > I created the pools manually and it’s completely nfigurations as shown
> >> > in b.
> >> >
> >> > On Wed, 7 Feb 2018 at 23:17 Matthew Jacobs <ja...@gmail.com>
> wrote:
> >> >>
> >> >> How, more specifically, are you configuring these pools? Can you
> share
> >> >> the configuration files?
> >> >>
> >> >> On Wed, Feb 7, 2018 at 11:42 AM, Fawze Abujaber <fa...@gmail.com>
> >> >> wrote:
> >> >> > Hello All,
> >> >> >
> >> >> > Will appreciate any insight of what i'm missing in my configuration
> >> >> > to
> >> >> > get
> >> >> > the impala submission control working as expected ....
> >> >> >
> >> >> > a- Here is my placement Placement Rule:
> >> >> > 1- Use the pool Specified at run time., only if the pool exists.
> >> >> > 2- Use the pool root.[secondary group].
> >> >> > 3- Use the pool root.default.
> >> >> >
> >> >> > b- Here is the impala resource pools
> >> >> >
> >> >> > Name Max Memory Max Running Queries Max Queued
> >> >> > Queries
> >> >> > Queue Timeout Default Query Memory Limit Access Control
> >> >> > root.impiaas Unlimited Unlimited
> >> >> > 200 1 minute(s) No Default
> >> >> > Custom
> >> >> > root.impreports Unlimited Unlimited
> >> >> > 200 1 minute(s) No Default
> >> >> > Custom
> >> >> >
> >> >> >
> >> >> > C- Here is my Access control for root.impiaas pool:
> >> >> >
> >> >> > Allow these users and groups to submit to this pool
> >> >> >
> >> >> > Users: fawzea
> >> >> >
> >> >> > The same for impreports the allowed user and groups to submit to
> pool
> >> >> > is
> >> >> > only nira.
> >> >> >
> >> >> >
> >> >> > LDAP groups for fawzea user:
> >> >> >
> >> >> > Member in groups:
> >> >> >
> >> >> > - dba
> >> >> > - impiaas
> >> >> > - mysql
> >> >> >
> >> >> > LDAP group for nira user
> >> >> > Member in groups:
> >> >> > - dba
> >> >> > - impreports
> >> >> > - mysql
> >> >> >
> >> >> >
> >> >> >
> >> >> > Impala Queries UI shows:
> >> >> >
> >> >> > 1- select count(*) from dv_entersiteevent_prq_remote where
> year=2017
> >> >> >
> >> >> > User: fawzea
> >> >> > Pool: root.impiaas
> >> >> >
> >> >> > 2- select count(*) from dv_entersiteevent_prq_remote where
> year=2017
> >> >> > and
> >> >> > month=10
> >> >> > User: nira
> >> >> > Pool: root.impreports
> >> >> >
> >> >> > As shown above that the queries submitted based on the secondary
> >> >> > group
> >> >> > rule.
> >> >> >
> >> >> > When i'm running the following with fawzea user:
> >> >> >
> >> >> > set request_pool=impreports;
> >> >> > select count(*) from dv_entersiteevent_prq_remote where year=2017
> and
> >> >> > month=10;
> >> >> >
> >> >> > I can see that the query is running with impreports pool.
> >> >> >
> >> >> >
> >> >> > What i'm missing here in my configuration to get this works as
> >> >> > expected.
> >> >> >
> >
> >
>
Re: impala submission control
Posted by Matthew Jacobs <mj...@apache.org>.
Thanks! I think that configuration looks OK.
1) What version of Impala is this?
2) Can you send the query profile and the Impala coordinator log for
the query where it's not behaving as expected?
> When i'm running the following with fawzea user:
> set request_pool=impreports;
> select count(*) from dv_entersiteevent_prq_remote where year=2017 and month=10;
> I can see that the query is running with impreports pool.
On Wed, Feb 7, 2018 at 7:59 PM, Fawze Abujaber <fa...@gmail.com> wrote:
> Hi Matt,
>
> [root@aopr-dhc001 ~]# cat
> /var/run/cloudera-scm-agent/process/4558-impala-IMPALAD/impala-conf/fair-scheduler.xml
> <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
> <allocations>
> <queue name="root">
> <weight>1.0</weight>
> <schedulingPolicy>fair</schedulingPolicy>
> <queue name="impiaas">
> <weight>1.0</weight>
> <schedulingPolicy>fair</schedulingPolicy>
> <aclSubmitApps>fawzea </aclSubmitApps>
> </queue>
> <queue name="impreports">
> <weight>1.0</weight>
> <schedulingPolicy>fair</schedulingPolicy>
> <aclSubmitApps>nira </aclSubmitApps>
> </queue>
> </queue>
> <defaultQueueSchedulingPolicy>fair</defaultQueueSchedulingPolicy>
> <queuePlacementPolicy>
> <rule name="specified" create="false"/>
> <rule name="secondaryGroupExistingQueue"/>
> <rule name="default"/>
> </queuePlacementPolicy>
> </allocations>
>
>
> On Thu, Feb 8, 2018 at 4:47 AM, Matthew Jacobs <ja...@gmail.com> wrote:
>>
>> Yes, I saw that, but I'd like to see the actual configuration that is
>> passed to Impala because it may not be specified in the right way to
>> accomplish what you're trying to do in "b".
>>
>> On Wed, Feb 7, 2018 at 4:32 PM, Fawze Abujaber <fa...@gmail.com> wrote:
>> > Hi Matt,
>> >
>> > I created the pools manually and it’s completely nfigurations as shown
>> > in b.
>> >
>> > On Wed, 7 Feb 2018 at 23:17 Matthew Jacobs <ja...@gmail.com> wrote:
>> >>
>> >> How, more specifically, are you configuring these pools? Can you share
>> >> the configuration files?
>> >>
>> >> On Wed, Feb 7, 2018 at 11:42 AM, Fawze Abujaber <fa...@gmail.com>
>> >> wrote:
>> >> > Hello All,
>> >> >
>> >> > Will appreciate any insight of what i'm missing in my configuration
>> >> > to
>> >> > get
>> >> > the impala submission control working as expected ....
>> >> >
>> >> > a- Here is my placement Placement Rule:
>> >> > 1- Use the pool Specified at run time., only if the pool exists.
>> >> > 2- Use the pool root.[secondary group].
>> >> > 3- Use the pool root.default.
>> >> >
>> >> > b- Here is the impala resource pools
>> >> >
>> >> > Name Max Memory Max Running Queries Max Queued
>> >> > Queries
>> >> > Queue Timeout Default Query Memory Limit Access Control
>> >> > root.impiaas Unlimited Unlimited
>> >> > 200 1 minute(s) No Default
>> >> > Custom
>> >> > root.impreports Unlimited Unlimited
>> >> > 200 1 minute(s) No Default
>> >> > Custom
>> >> >
>> >> >
>> >> > C- Here is my Access control for root.impiaas pool:
>> >> >
>> >> > Allow these users and groups to submit to this pool
>> >> >
>> >> > Users: fawzea
>> >> >
>> >> > The same for impreports the allowed user and groups to submit to pool
>> >> > is
>> >> > only nira.
>> >> >
>> >> >
>> >> > LDAP groups for fawzea user:
>> >> >
>> >> > Member in groups:
>> >> >
>> >> > - dba
>> >> > - impiaas
>> >> > - mysql
>> >> >
>> >> > LDAP group for nira user
>> >> > Member in groups:
>> >> > - dba
>> >> > - impreports
>> >> > - mysql
>> >> >
>> >> >
>> >> >
>> >> > Impala Queries UI shows:
>> >> >
>> >> > 1- select count(*) from dv_entersiteevent_prq_remote where year=2017
>> >> >
>> >> > User: fawzea
>> >> > Pool: root.impiaas
>> >> >
>> >> > 2- select count(*) from dv_entersiteevent_prq_remote where year=2017
>> >> > and
>> >> > month=10
>> >> > User: nira
>> >> > Pool: root.impreports
>> >> >
>> >> > As shown above that the queries submitted based on the secondary
>> >> > group
>> >> > rule.
>> >> >
>> >> > When i'm running the following with fawzea user:
>> >> >
>> >> > set request_pool=impreports;
>> >> > select count(*) from dv_entersiteevent_prq_remote where year=2017 and
>> >> > month=10;
>> >> >
>> >> > I can see that the query is running with impreports pool.
>> >> >
>> >> >
>> >> > What i'm missing here in my configuration to get this works as
>> >> > expected.
>> >> >
>
>
Re: impala submission control
Posted by Fawze Abujaber <fa...@gmail.com>.
Hi Matt,
[root@aopr-dhc001 ~]# cat
/var/run/cloudera-scm-agent/process/4558-impala-IMPALAD/impala-conf/fair-scheduler.xml
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<allocations>
<queue name="root">
<weight>1.0</weight>
<schedulingPolicy>fair</schedulingPolicy>
<queue name="impiaas">
<weight>1.0</weight>
<schedulingPolicy>fair</schedulingPolicy>
<aclSubmitApps>fawzea </aclSubmitApps>
</queue>
<queue name="impreports">
<weight>1.0</weight>
<schedulingPolicy>fair</schedulingPolicy>
<aclSubmitApps>nira </aclSubmitApps>
</queue>
</queue>
<defaultQueueSchedulingPolicy>fair</defaultQueueSchedulingPolicy>
<queuePlacementPolicy>
<rule name="specified" create="false"/>
<rule name="secondaryGroupExistingQueue"/>
<rule name="default"/>
</queuePlacementPolicy>
</allocations>
On Thu, Feb 8, 2018 at 4:47 AM, Matthew Jacobs <ja...@gmail.com> wrote:
> Yes, I saw that, but I'd like to see the actual configuration that is
> passed to Impala because it may not be specified in the right way to
> accomplish what you're trying to do in "b".
>
> On Wed, Feb 7, 2018 at 4:32 PM, Fawze Abujaber <fa...@gmail.com> wrote:
> > Hi Matt,
> >
> > I created the pools manually and it’s completely nfigurations as shown
> in b.
> >
> > On Wed, 7 Feb 2018 at 23:17 Matthew Jacobs <ja...@gmail.com> wrote:
> >>
> >> How, more specifically, are you configuring these pools? Can you share
> >> the configuration files?
> >>
> >> On Wed, Feb 7, 2018 at 11:42 AM, Fawze Abujaber <fa...@gmail.com>
> wrote:
> >> > Hello All,
> >> >
> >> > Will appreciate any insight of what i'm missing in my configuration to
> >> > get
> >> > the impala submission control working as expected ....
> >> >
> >> > a- Here is my placement Placement Rule:
> >> > 1- Use the pool Specified at run time., only if the pool exists.
> >> > 2- Use the pool root.[secondary group].
> >> > 3- Use the pool root.default.
> >> >
> >> > b- Here is the impala resource pools
> >> >
> >> > Name Max Memory Max Running Queries Max Queued
> >> > Queries
> >> > Queue Timeout Default Query Memory Limit Access Control
> >> > root.impiaas Unlimited Unlimited
> >> > 200 1 minute(s) No Default
> >> > Custom
> >> > root.impreports Unlimited Unlimited
> >> > 200 1 minute(s) No Default
> >> > Custom
> >> >
> >> >
> >> > C- Here is my Access control for root.impiaas pool:
> >> >
> >> > Allow these users and groups to submit to this pool
> >> >
> >> > Users: fawzea
> >> >
> >> > The same for impreports the allowed user and groups to submit to pool
> is
> >> > only nira.
> >> >
> >> >
> >> > LDAP groups for fawzea user:
> >> >
> >> > Member in groups:
> >> >
> >> > - dba
> >> > - impiaas
> >> > - mysql
> >> >
> >> > LDAP group for nira user
> >> > Member in groups:
> >> > - dba
> >> > - impreports
> >> > - mysql
> >> >
> >> >
> >> >
> >> > Impala Queries UI shows:
> >> >
> >> > 1- select count(*) from dv_entersiteevent_prq_remote where year=2017
> >> >
> >> > User: fawzea
> >> > Pool: root.impiaas
> >> >
> >> > 2- select count(*) from dv_entersiteevent_prq_remote where year=2017
> and
> >> > month=10
> >> > User: nira
> >> > Pool: root.impreports
> >> >
> >> > As shown above that the queries submitted based on the secondary group
> >> > rule.
> >> >
> >> > When i'm running the following with fawzea user:
> >> >
> >> > set request_pool=impreports;
> >> > select count(*) from dv_entersiteevent_prq_remote where year=2017 and
> >> > month=10;
> >> >
> >> > I can see that the query is running with impreports pool.
> >> >
> >> >
> >> > What i'm missing here in my configuration to get this works as
> expected.
> >> >
>
Re: impala submission control
Posted by Matthew Jacobs <ja...@gmail.com>.
Yes, I saw that, but I'd like to see the actual configuration that is
passed to Impala because it may not be specified in the right way to
accomplish what you're trying to do in "b".
On Wed, Feb 7, 2018 at 4:32 PM, Fawze Abujaber <fa...@gmail.com> wrote:
> Hi Matt,
>
> I created the pools manually and it’s completely nfigurations as shown in b.
>
> On Wed, 7 Feb 2018 at 23:17 Matthew Jacobs <ja...@gmail.com> wrote:
>>
>> How, more specifically, are you configuring these pools? Can you share
>> the configuration files?
>>
>> On Wed, Feb 7, 2018 at 11:42 AM, Fawze Abujaber <fa...@gmail.com> wrote:
>> > Hello All,
>> >
>> > Will appreciate any insight of what i'm missing in my configuration to
>> > get
>> > the impala submission control working as expected ....
>> >
>> > a- Here is my placement Placement Rule:
>> > 1- Use the pool Specified at run time., only if the pool exists.
>> > 2- Use the pool root.[secondary group].
>> > 3- Use the pool root.default.
>> >
>> > b- Here is the impala resource pools
>> >
>> > Name Max Memory Max Running Queries Max Queued
>> > Queries
>> > Queue Timeout Default Query Memory Limit Access Control
>> > root.impiaas Unlimited Unlimited
>> > 200 1 minute(s) No Default
>> > Custom
>> > root.impreports Unlimited Unlimited
>> > 200 1 minute(s) No Default
>> > Custom
>> >
>> >
>> > C- Here is my Access control for root.impiaas pool:
>> >
>> > Allow these users and groups to submit to this pool
>> >
>> > Users: fawzea
>> >
>> > The same for impreports the allowed user and groups to submit to pool is
>> > only nira.
>> >
>> >
>> > LDAP groups for fawzea user:
>> >
>> > Member in groups:
>> >
>> > - dba
>> > - impiaas
>> > - mysql
>> >
>> > LDAP group for nira user
>> > Member in groups:
>> > - dba
>> > - impreports
>> > - mysql
>> >
>> >
>> >
>> > Impala Queries UI shows:
>> >
>> > 1- select count(*) from dv_entersiteevent_prq_remote where year=2017
>> >
>> > User: fawzea
>> > Pool: root.impiaas
>> >
>> > 2- select count(*) from dv_entersiteevent_prq_remote where year=2017 and
>> > month=10
>> > User: nira
>> > Pool: root.impreports
>> >
>> > As shown above that the queries submitted based on the secondary group
>> > rule.
>> >
>> > When i'm running the following with fawzea user:
>> >
>> > set request_pool=impreports;
>> > select count(*) from dv_entersiteevent_prq_remote where year=2017 and
>> > month=10;
>> >
>> > I can see that the query is running with impreports pool.
>> >
>> >
>> > What i'm missing here in my configuration to get this works as expected.
>> >
Re: impala submission control
Posted by Fawze Abujaber <fa...@gmail.com>.
Hi Matt,
I created the pools manually and it’s completely nfigurations as shown in b.
On Wed, 7 Feb 2018 at 23:17 Matthew Jacobs <ja...@gmail.com> wrote:
> How, more specifically, are you configuring these pools? Can you share
> the configuration files?
>
> On Wed, Feb 7, 2018 at 11:42 AM, Fawze Abujaber <fa...@gmail.com> wrote:
> > Hello All,
> >
> > Will appreciate any insight of what i'm missing in my configuration to
> get
> > the impala submission control working as expected ....
> >
> > a- Here is my placement Placement Rule:
> > 1- Use the pool Specified at run time., only if the pool exists.
> > 2- Use the pool root.[secondary group].
> > 3- Use the pool root.default.
> >
> > b- Here is the impala resource pools
> >
> > Name Max Memory Max Running Queries Max Queued Queries
> > Queue Timeout Default Query Memory Limit Access Control
> > root.impiaas Unlimited Unlimited
> > 200 1 minute(s) No Default
> > Custom
> > root.impreports Unlimited Unlimited
> > 200 1 minute(s) No Default
> > Custom
> >
> >
> > C- Here is my Access control for root.impiaas pool:
> >
> > Allow these users and groups to submit to this pool
> >
> > Users: fawzea
> >
> > The same for impreports the allowed user and groups to submit to pool is
> > only nira.
> >
> >
> > LDAP groups for fawzea user:
> >
> > Member in groups:
> >
> > - dba
> > - impiaas
> > - mysql
> >
> > LDAP group for nira user
> > Member in groups:
> > - dba
> > - impreports
> > - mysql
> >
> >
> >
> > Impala Queries UI shows:
> >
> > 1- select count(*) from dv_entersiteevent_prq_remote where year=2017
> >
> > User: fawzea
> > Pool: root.impiaas
> >
> > 2- select count(*) from dv_entersiteevent_prq_remote where year=2017 and
> > month=10
> > User: nira
> > Pool: root.impreports
> >
> > As shown above that the queries submitted based on the secondary group
> rule.
> >
> > When i'm running the following with fawzea user:
> >
> > set request_pool=impreports;
> > select count(*) from dv_entersiteevent_prq_remote where year=2017 and
> > month=10;
> >
> > I can see that the query is running with impreports pool.
> >
> >
> > What i'm missing here in my configuration to get this works as expected.
> >
>
Re: impala submission control
Posted by Matthew Jacobs <ja...@gmail.com>.
How, more specifically, are you configuring these pools? Can you share
the configuration files?
On Wed, Feb 7, 2018 at 11:42 AM, Fawze Abujaber <fa...@gmail.com> wrote:
> Hello All,
>
> Will appreciate any insight of what i'm missing in my configuration to get
> the impala submission control working as expected ....
>
> a- Here is my placement Placement Rule:
> 1- Use the pool Specified at run time., only if the pool exists.
> 2- Use the pool root.[secondary group].
> 3- Use the pool root.default.
>
> b- Here is the impala resource pools
>
> Name Max Memory Max Running Queries Max Queued Queries
> Queue Timeout Default Query Memory Limit Access Control
> root.impiaas Unlimited Unlimited
> 200 1 minute(s) No Default
> Custom
> root.impreports Unlimited Unlimited
> 200 1 minute(s) No Default
> Custom
>
>
> C- Here is my Access control for root.impiaas pool:
>
> Allow these users and groups to submit to this pool
>
> Users: fawzea
>
> The same for impreports the allowed user and groups to submit to pool is
> only nira.
>
>
> LDAP groups for fawzea user:
>
> Member in groups:
>
> - dba
> - impiaas
> - mysql
>
> LDAP group for nira user
> Member in groups:
> - dba
> - impreports
> - mysql
>
>
>
> Impala Queries UI shows:
>
> 1- select count(*) from dv_entersiteevent_prq_remote where year=2017
>
> User: fawzea
> Pool: root.impiaas
>
> 2- select count(*) from dv_entersiteevent_prq_remote where year=2017 and
> month=10
> User: nira
> Pool: root.impreports
>
> As shown above that the queries submitted based on the secondary group rule.
>
> When i'm running the following with fawzea user:
>
> set request_pool=impreports;
> select count(*) from dv_entersiteevent_prq_remote where year=2017 and
> month=10;
>
> I can see that the query is running with impreports pool.
>
>
> What i'm missing here in my configuration to get this works as expected.
>