You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2021/12/03 12:05:00 UTC

[jira] [Work logged] (HIVE-25666) Realtime memory usage in beeline progress

     [ https://issues.apache.org/jira/browse/HIVE-25666?focusedWorklogId=689952&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-689952 ]

ASF GitHub Bot logged work on HIVE-25666:
-----------------------------------------

                Author: ASF GitHub Bot
            Created on: 03/Dec/21 12:04
            Start Date: 03/Dec/21 12:04
    Worklog Time Spent: 10m 
      Work Description: abstractdog commented on a change in pull request #2812:
URL: https://github.com/apache/hive/pull/2812#discussion_r761878622



##########
File path: llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
##########
@@ -1163,4 +1165,15 @@ private QueryIdentifierProto constructQueryIdentifierProto(int dagIdentifier) {
   public String getAmHostString() {
     return amHost;
   }
+
+  /**
+   * Overrides TezTaskCommunicatorImpl.getTotalUsedMemory in order to provide correct aggregated memory usage.
+   * In LLAP, every container reports the whole used heap of the daemon they're running in, so we need to consider
+   * every usedMemory once per daemon.
+   * @return
+   */
+  @Override
+  public long getTotalUsedMemory() {
+    return pingedNodeMap.values().stream().mapToLong(c -> c.usedMemory).sum();

Review comment:
       also, I need to plan this consciously, because every change needs to flow through tez DagClient API, which is protobuf based and doesn't support map<> with the current protobuf version, so I need to find out another way to propagate arbitrary memory-related data




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Issue Time Tracking
-------------------

    Worklog Id:     (was: 689952)
    Time Spent: 40m  (was: 0.5h)

> Realtime memory usage in beeline progress
> -----------------------------------------
>
>                 Key: HIVE-25666
>                 URL: https://issues.apache.org/jira/browse/HIVE-25666
>             Project: Hive
>          Issue Type: Improvement
>            Reporter: László Bodor
>            Assignee: László Bodor
>            Priority: Major
>              Labels: pull-request-available
>         Attachments: llap_memory_monitoring_beeline.gif
>
>          Time Spent: 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.20.1#820001)