You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@impala.apache.org by "Lars Volker (JIRA)" <ji...@apache.org> on 2018/03/15 17:39:00 UTC
[jira] [Resolved] (IMPALA-6652) KRPC : Data Stream Manager Deferred
RPCs in memz page should be renamed
[ https://issues.apache.org/jira/browse/IMPALA-6652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Volker resolved IMPALA-6652.
---------------------------------
Resolution: Fixed
Fix Version/s: Impala 2.12.0
IMPALA-6652: Rename label of MemTracker for early RPCs
This change renames the label of the MemTracker in
KrpcDataStreamMgr for tracking payloads of early RPCs
to "Data Stream Manager Early RPCs". This is to distinguish
these RPCs from the deferred RPCs in a receiver. The early
RPCs refer to those RPCs which arrive before a receiver
is ready. The responses to these RPCs are deferred until
the receiver is created. The receiver may also defer
responses to RPCs if the deserialized payloads of RPCs in
an inbound queue exceed FLAGS_exchg_node_buffer_size_bytes.
In this case, the RPCs won't be responded to until the
inbound queue is drained.
Change-Id: I5bb72c28e8d660a6b78543dbc8b5b156e0e7c843
Reviewed-on: http://gerrit.cloudera.org:8080/9633
Reviewed-by: Dan Hecht <dh...@cloudera.com>
Reviewed-by: Michael Ho <kw...@cloudera.com>
Tested-by: Impala Public Jenkins
> KRPC : Data Stream Manager Deferred RPCs in memz page should be renamed
> -----------------------------------------------------------------------
>
> Key: IMPALA-6652
> URL: https://issues.apache.org/jira/browse/IMPALA-6652
> Project: IMPALA
> Issue Type: Sub-task
> Components: Distributed Exec
> Affects Versions: Impala 2.12.0
> Reporter: Mostafa Mokhtar
> Assignee: Lars Volker
> Priority: Major
> Labels: metrics, supportability
> Fix For: Impala 2.12.0
>
> Attachments: Partial query profile.txt
>
>
> While loading data into a Kudu table against the latest Impala 2.11.0 I noticed that "Data Stream Manager Deferred RPCs" in the memz isn't accurate.
>
> From memz on worker
> {code}
> Process: Limit=201.73 GB Total=85.41 GB Peak=85.41 GB
> Buffer Pool: Free Buffers: Total=43.64 MB
> Buffer Pool: Clean Pages: Total=0
> Buffer Pool: Unused Reservation: Total=-17.84 MB
> Data Stream Service Queue: Limit=10.09 GB Total=0 Peak=512.97 MB
> Data Stream Manager Deferred RPCs: Total=0 Peak=0
> TCMalloc Overhead: Total=124.07 MB
> Free Disk IO Buffers: Total=984.97 MB Peak=984.97 MB
> RequestPool=root.default: Total=83.92 GB Peak=83.92 GB
> Query(844a0200d7876345:20bb38b900000000): Reservation=70.44 GB ReservationLimit=161.39 GB OtherMemory=13.48 GB Total=83.92 GB Peak=83.92 GB
> Fragment 844a0200d7876345:20bb38b9000000a3: Reservation=70.44 GB OtherMemory=38.08 MB Total=70.47 GB Peak=70.47 GB
> SORT_NODE (id=2): Reservation=70.44 GB OtherMemory=8.00 KB Total=70.44 GB Peak=70.44 GB
> EXCHANGE_NODE (id=1): Reservation=18.06 MB OtherMemory=0 Total=18.06 MB Peak=19.53 MB
> KrpcDeferredRpcs: Total=0 Peak=1.47 MB
> KuduTableSink: Total=20.00 MB Peak=20.00 MB
> CodeGen: Total=438.00 B Peak=306.00 KB
> Fragment 844a0200d7876345:20bb38b900000022: Reservation=0 OtherMemory=13.44 GB Total=13.44 GB Peak=13.97 GB
> HDFS_SCAN_NODE (id=0): Total=13.44 GB Peak=13.97 GB
> KrpcDataStreamSender (dst_id=1): Total=2.57 MB Peak=3.61 MB
> CodeGen: Total=234.00 B Peak=52.50 KB
> Untracked Memory: Total=389.18 MB
> {code}
>
> And snapshot from query profile
> {code}
> Instance 844a0200d7876345:20bb38b9000000a3 (host=va1030.halxg.cloudera.com:22000):(Total: 1s172ms, non-child: 200.411ms, % non-child: 17.09%)
> Fragment Instance Lifecycle Event Timeline: 1s173ms
> - Prepare Finished: 199.691ms (199.691ms)
> - Open Finished: 1s173ms (973.902ms)
> MemoryUsage(1m4s): 4.77 GB, 13.21 GB, 19.60 GB, 23.70 GB, 26.67 GB, 29.21 GB, 31.50 GB, 33.63 GB, 35.40 GB, 37.14 GB, 38.54 GB, 39.79 GB, 41.09 GB, 42.37 GB, 43.60 GB, 44.80 GB, 45.95 GB, 47.01 GB, 48.09 GB, 49.17 GB, 50.22 GB, 51.21 GB, 52.40 GB, 53.46 GB, 54.58 GB, 55.61 GB, 56.58 GB, 57.53 GB, 58.45 GB, 59.39 GB, 60.31 GB, 61.20 GB, 62.12 GB, 63.04 GB, 64.15 GB, 65.11 GB, 66.15 GB, 67.06 GB, 67.87 GB, 68.66 GB, 69.49 GB, 70.29 GB, 71.07 GB, 71.83 GB, 72.61 GB
> ThreadUsage(1m4s): 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1
> - AverageThreadTokens: 1.00
> - BloomFilterBytes: 0
> - PeakMemoryUsage: 73.69 GB (79123530166)
> - PeakReservation: 73.65 GB (79083601920)
> - PeakUsedReservation: 0
> - PerHostPeakMemUsage: 87.02 GB (93435941671)
> - RowsProduced: 0 (0)
> - TotalNetworkReceiveTime: 38m28s
> - TotalNetworkSendTime: 0.000ns
> - TotalStorageWaitTime: 0.000ns
> - TotalThreadsInvoluntaryContextSwitches: 7 (7)
> - TotalThreadsTotalWallClockTime: 973.873ms
> - TotalThreadsSysTime: 2.000ms
> - TotalThreadsUserTime: 55.991ms
> - TotalThreadsVoluntaryContextSwitches: 25 (25)
> Buffer pool:
> - AllocTime: 0.000ns
> - CumulativeAllocationBytes: 0
> - CumulativeAllocations: 0 (0)
> - PeakReservation: 0
> - PeakUnpinnedBytes: 0
> - PeakUsedReservation: 0
> - ReadIoBytes: 0
> - ReadIoOps: 0 (0)
> - ReadIoWaitTime: 0.000ns
> - ReservationLimit: 0
> - WriteIoBytes: 0
> - WriteIoOps: 0 (0)
> - WriteIoWaitTime: 0.000ns
> Fragment Instance Lifecycle Timings:
> - ExecTime: 0.000ns
> - ExecTreeExecTime: 0.000ns
> - OpenTime: 973.876ms
> - ExecTreeOpenTime: 915.567ms
> - PrepareTime: 198.988ms
> - ExecTreePrepareTime: 155.134us
> KuduTableSink:(Total: 12.589us, non-child: 12.589us, % non-child: 100.00%)
> - KuduApplyTimer: 0.000ns
> - NumRowErrors: 0 (0)
> - PeakMemoryUsage: 20.00 MB (20971520)
> - RowsProcessedRate: 0
> - TotalNumRows: 0 (0)
> SORT_NODE (id=2):(Total: 915.718ms, non-child: 0.000ns, % non-child: 0.00%)
> SortType: Partial
> ExecOption: Codegen Enabled
> - NumRowsPerRun: 0 (0) (Number of samples: 0)
> - InMemorySortTime: 0.000ns
> - PeakMemoryUsage: 73.65 GB (79083610112)
> - RowsReturned: 0 (0)
> - RowsReturnedRate: 0
> - RunsCreated: 1 (1)
> - SortDataSize: 0
> Buffer pool:
> - AllocTime: 3m9s
> - CumulativeAllocationBytes: 73.65 GB (79083601920)
> - CumulativeAllocations: 37.71K (37710)
> - PeakReservation: 73.65 GB (79083601920)
> - PeakUnpinnedBytes: 0
> - PeakUsedReservation: 73.65 GB (79083601920)
> - ReadIoBytes: 0
> - ReadIoOps: 0 (0)
> - ReadIoWaitTime: 0.000ns
> - WriteIoBytes: 0
> - WriteIoOps: 0 (0)
> - WriteIoWaitTime: 0.000ns
> EXCHANGE_NODE (id=1):(Total: 38m39s, non-child: 18s181ms, % non-child: 0.78%)
> - ConvertRowBatchTime: 7s926ms
> - PeakMemoryUsage: 19.53 MB (20481319)
> - RowsReturned: 288.83M (288830892)
> - RowsReturnedRate: 124.54 K/sec
> Buffer pool:
> - AllocTime: 17m12s
> - CumulativeAllocationBytes: 131.17 GB (140839444480)
> - CumulativeAllocations: 6.88M (6876926)
> - PeakReservation: 18.06 MB (18939904)
> - PeakUnpinnedBytes: 0
> - PeakUsedReservation: 18.06 MB (18939904)
> - ReadIoBytes: 0
> - ReadIoOps: 0 (0)
> - ReadIoWaitTime: 0.000ns
> - WriteIoBytes: 0
> - WriteIoOps: 0 (0)
> - WriteIoWaitTime: 0.000ns
> RecvrSide:
> BytesReceived(1m4s): 4.87 GB, 13.37 GB, 19.82 GB, 23.95 GB, 26.95 GB, 29.52 GB, 31.82 GB, 33.98 GB, 35.77 GB, 37.52 GB, 38.94 GB, 40.20 GB, 41.51 GB, 42.80 GB, 44.05 GB, 45.26 GB, 46.42 GB, 47.49 GB, 48.58 GB, 49.68 GB, 50.74 GB, 51.74 GB, 52.94 GB, 54.01 GB, 55.15 GB, 56.19 GB, 57.16 GB, 58.12 GB, 59.05 GB, 60.00 GB, 60.93 GB, 61.84 GB, 62.76 GB, 63.70 GB, 64.82 GB, 65.79 GB, 66.84 GB, 67.76 GB, 68.57 GB, 69.38 GB, 70.22 GB, 71.02 GB, 71.81 GB, 72.58 GB, 73.37 GB
> - FirstBatchArrivalWaitTime: 915.459ms
> - TotalBytesReceived: 74.45 GB (79944017735)
> - TotalGetBatchTime: 38m28s
> - DataArrivalTimer: 38m21s
> SenderSide:
> - DeserializeRowBatchTime: 20m44s
> - NumBatchesArrived: 3.44M (3438463)
> - NumBatchesDeferred: 27.14K (27144)
> - NumBatchesEnqueued: 3.44M (3438463)
> - NumBatchesReceived: 3.44M (3438463)
> - NumEarlySenders: 0 (0)
> - NumEosReceived: 0 (0)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)