You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by kkhatua <gi...@git.apache.org> on 2018/03/11 23:30:54 UTC

[GitHub] drill pull request #1160: DRILL-6224: Publish current memory usage for Drill...

GitHub user kkhatua opened a pull request:

    https://github.com/apache/drill/pull/1160

    DRILL-6224: Publish current memory usage for Drillbits in webpages

    The `metrics.ftl` page had gauges incorrectly set to near zero values. The commit for metrics.ftl fixes that, and also provides an estimate of the current direct memory actively in use (based on the `drill.allocator.root.used` value reported by the Drillbit)
    The `index.ftl` page expands this by additionally providing the active heap and direct memory usage for all Drillbits. This is done by the JavaScript actively pinging all the Drillbits on their current memory usage in 10 second intervals.
    The percentages in both cases indicate the % usage in terms of the max (heap) or the peak (direct) observed.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/kkhatua/drill DRILL-6224

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/drill/pull/1160.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1160
    
----
commit f098a2dc8644e9d3b11d032f6250c1f12f381f7e
Author: Kunal Khatua <kk...@...>
Date:   2018-03-10T08:04:35Z

    DRILL-6224: Publish current memory usage for Drillbits in webpages
    
    The `metrics.ftl` page had gauges incorrectly set to near zero values. The commit for metrics.ftl fixes that, and also provides an estimate of the current direct memory actively in use (based on the `drill.allocator.root.used` value reported by the Drillbit)
    The `index.ftl` page expands this by additionally providing the active heap and direct memory usage for all Drillbits. This is done by the JavaScript actively pinging all the Drillbits on their current memory usage in 10 second intervals.
    The percentages in both cases indicate the % usage in terms of the max (heap) or the peak (direct) observed.

----


---

[GitHub] drill issue #1160: DRILL-6224: Publish current memory usage for Drillbits in...

Posted by kkhatua <gi...@git.apache.org>.
Github user kkhatua commented on the issue:

    https://github.com/apache/drill/pull/1160
  
    During a freshly setup Drillbit:
    ![image](https://user-images.githubusercontent.com/4335237/37847547-644fa67c-2e8e-11e8-9aa7-2b4adf499cd0.png)
    
    While running 2 large (600M rows) join query on a 4 node setup, and one completes:
    ![image](https://user-images.githubusercontent.com/4335237/37847633-b4dd5a76-2e8e-11e8-9c1d-87c3a0f9f3e7.png)
    
    The Heap Usage percentage is based on {{heapUsed/heapMax}}
    The Actively Used Direct percentage is based on {{currentDirectInUse/PeakDirectUsed}} 
    This is because the actual total direct memory used by Netty is not exposed. However, this does tell how much the Drillbit is using at the moment.


---

[GitHub] drill pull request #1160: DRILL-6224: Publish current memory usage for Drill...

Posted by kkhatua <gi...@git.apache.org>.
Github user kkhatua commented on a diff in the pull request:

    https://github.com/apache/drill/pull/1160#discussion_r176592068
  
    --- Diff: exec/java-exec/src/main/resources/rest/index.ftl ---
    @@ -192,6 +199,8 @@
           var port = getPortNum();
           var timeout;
           var size = $("#size").html();
    +      reloadMemory();
    +      setInterval(reloadMemory, refreshTime);
    --- End diff --
    
    That's because the scheduler for graceful shutdown can change, while in this case we need a periodic refresh that shouldn't change.
    As for a Drillbit going down, it can always come back online. As long as the page continues to list the Drillbit and it does get updated to reflect the status as "ONLINE" again, we should be pinging for the metrics too.


---

[GitHub] drill pull request #1160: DRILL-6224: Publish current memory usage for Drill...

Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on a diff in the pull request:

    https://github.com/apache/drill/pull/1160#discussion_r174440404
  
    --- Diff: exec/java-exec/src/main/resources/rest/index.ftl ---
    @@ -291,6 +300,46 @@
                 var url = protocol + "//" + host + requestPath;
                 return url;
           }
    +
    +      //Iterates through all the nodes for update
    +      function reloadMemory () {
    +          for (i = 1; i <= size; i++) {
    +              var address = $("#row-"+i).find("#address").contents().get(0).nodeValue.trim();
    +              updateMemory(address, i);
    +          }
    +      }
    +
    +      //Update memory
    +      function updateMemory(drillbit,idx) {
    +        var result = $.ajax({
    +          type: 'GET',
    +          url: "http://"+drillbit+":8047/status/metrics",
    --- End diff --
    
    You can not hard-code port, it can be changed using property. See how graceful shutdown is checking drillbits statuses.


---

[GitHub] drill pull request #1160: DRILL-6224: Publish current memory usage for Drill...

Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on a diff in the pull request:

    https://github.com/apache/drill/pull/1160#discussion_r174440089
  
    --- Diff: exec/java-exec/src/main/resources/rest/index.ftl ---
    @@ -53,11 +53,16 @@
                 <tr>
                   <th>#</th>
                   <th>Address</th>
    +              <th>Heap Usage</th>
    +              <th>Direct Usage</th>
                   <th>User Port</th>
                   <th>Control Port</th>
                   <th>Data Port</th>
                   <th>Version</th>
                   <th>Status</th>
    +              <#if model.shouldShowAdminInfo() || !model.isAuthEnabled() >
    --- End diff --
    
    On master we have different line `<#if (model.shouldShowAdminInfo() || !model.isAuthEnabled()) && drillbit.isCurrent() >`


---

[GitHub] drill pull request #1160: DRILL-6224: Publish current memory usage for Drill...

Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on a diff in the pull request:

    https://github.com/apache/drill/pull/1160#discussion_r174440528
  
    --- Diff: exec/java-exec/src/main/resources/rest/index.ftl ---
    @@ -291,6 +300,46 @@
                 var url = protocol + "//" + host + requestPath;
                 return url;
           }
    +
    +      //Iterates through all the nodes for update
    +      function reloadMemory () {
    +          for (i = 1; i <= size; i++) {
    +              var address = $("#row-"+i).find("#address").contents().get(0).nodeValue.trim();
    +              updateMemory(address, i);
    +          }
    +      }
    +
    +      //Update memory
    +      function updateMemory(drillbit,idx) {
    +        var result = $.ajax({
    +          type: 'GET',
    +          url: "http://"+drillbit+":8047/status/metrics",
    +          dataType: "json",
    +          complete: function(data) {
    +            var metrics = data.responseJSON['gauges'];
    +            var usedHeap = metrics['heap.used'].value;
    +            var maxHeap  = metrics['heap.max'].value;
    +            var usedDirect = metrics['drill.allocator.root.used'].value;
    +            var peakDirect = metrics['drill.allocator.root.peak'].value;
    +            var heapUsage    = computeMemUsage(usedHeap,maxHeap);
    --- End diff --
    
    Please remove unnecessary spaces here and in other file as well.


---

[GitHub] drill pull request #1160: DRILL-6224: Publish current memory usage for Drill...

Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on a diff in the pull request:

    https://github.com/apache/drill/pull/1160#discussion_r174444297
  
    --- Diff: exec/java-exec/src/main/resources/rest/index.ftl ---
    @@ -291,6 +300,46 @@
                 var url = protocol + "//" + host + requestPath;
                 return url;
           }
    +
    +      //Iterates through all the nodes for update
    +      function reloadMemory () {
    +          for (i = 1; i <= size; i++) {
    +              var address = $("#row-"+i).find("#address").contents().get(0).nodeValue.trim();
    +              updateMemory(address, i);
    +          }
    +      }
    +
    +      //Update memory
    +      function updateMemory(drillbit,idx) {
    +        var result = $.ajax({
    +          type: 'GET',
    +          url: "http://"+drillbit+":8047/status/metrics",
    +          dataType: "json",
    +          complete: function(data) {
    +            var metrics = data.responseJSON['gauges'];
    +            var usedHeap = metrics['heap.used'].value;
    +            var maxHeap  = metrics['heap.max'].value;
    +            var usedDirect = metrics['drill.allocator.root.used'].value;
    +            var peakDirect = metrics['drill.allocator.root.peak'].value;
    +            var heapUsage    = computeMemUsage(usedHeap,maxHeap);
    +            var directUsage  = computeMemUsage(usedDirect,peakDirect);
    +            var heapElem = document.getElementById("row-"+idx).getElementsByClassName("heap")[0];
    +            heapElem.innerHTML = heapUsage;
    +            var directElem = document.getElementById("row-"+idx).getElementsByClassName("direct")[0];
    +            directElem.innerHTML = directUsage;
    +            }
    +          });
    +      }
    +
    +      //Compute Usage
    +      function computeMemUsage(used, max) {
    +        var percent = 0;
    +        if ( max > 0)
    +          var percent = Math.round((100 * used / max), 2);
    +        var usage   = Math.round((used / 1073741824), 2) + "GB (" + Math.max(0, percent) + "%)";
    --- End diff --
    
    This logic repeats in several files, so can create js function in separate file and import it.


---

[GitHub] drill pull request #1160: DRILL-6224: Publish current memory usage for Drill...

Posted by kkhatua <gi...@git.apache.org>.
Github user kkhatua commented on a diff in the pull request:

    https://github.com/apache/drill/pull/1160#discussion_r176590360
  
    --- Diff: exec/java-exec/src/main/resources/rest/index.ftl ---
    @@ -291,6 +300,46 @@
                 var url = protocol + "//" + host + requestPath;
                 return url;
           }
    +
    +      //Iterates through all the nodes for update
    +      function reloadMemory () {
    +          for (i = 1; i <= size; i++) {
    +              var address = $("#row-"+i).find("#address").contents().get(0).nodeValue.trim();
    +              updateMemory(address, i);
    +          }
    +      }
    +
    +      //Update memory
    +      function updateMemory(drillbit,idx) {
    +        var result = $.ajax({
    +          type: 'GET',
    +          url: "http://"+drillbit+":8047/status/metrics",
    --- End diff --
    
    Actually, this is a challenge. We don't seem to be exposing any information on the http-port.  The only information I am getting back into DrillbitInfo are the other 3 ports: control, data and user. If I need to had HTTP to this (which I think it should), we need to modify the protobuf (UserBitShared). :(
    Do you have any suggestions?


---

[GitHub] drill pull request #1160: DRILL-6224: Publish current memory usage for Drill...

Posted by kkhatua <gi...@git.apache.org>.
Github user kkhatua commented on a diff in the pull request:

    https://github.com/apache/drill/pull/1160#discussion_r176579720
  
    --- Diff: exec/java-exec/src/main/resources/rest/index.ftl ---
    @@ -53,11 +53,16 @@
                 <tr>
                   <th>#</th>
                   <th>Address</th>
    +              <th>Heap Usage</th>
    +              <th>Direct Usage</th>
                   <th>User Port</th>
                   <th>Control Port</th>
                   <th>Data Port</th>
                   <th>Version</th>
                   <th>Status</th>
    +              <#if model.shouldShowAdminInfo() || !model.isAuthEnabled() >
    --- End diff --
    
    I need to rebase. DRILL-6044 fix changed that entry.


---

[GitHub] drill issue #1160: DRILL-6224: Publish current memory usage for Drillbits in...

Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on the issue:

    https://github.com/apache/drill/pull/1160
  
    +1, thanks for making the changes.


---

[GitHub] drill pull request #1160: DRILL-6224: Publish current memory usage for Drill...

Posted by kkhatua <gi...@git.apache.org>.
Github user kkhatua commented on a diff in the pull request:

    https://github.com/apache/drill/pull/1160#discussion_r176593083
  
    --- Diff: exec/java-exec/src/main/resources/rest/index.ftl ---
    @@ -291,6 +300,46 @@
                 var url = protocol + "//" + host + requestPath;
                 return url;
           }
    +
    +      //Iterates through all the nodes for update
    +      function reloadMemory () {
    +          for (i = 1; i <= size; i++) {
    +              var address = $("#row-"+i).find("#address").contents().get(0).nodeValue.trim();
    +              updateMemory(address, i);
    +          }
    +      }
    +
    +      //Update memory
    +      function updateMemory(drillbit,idx) {
    +        var result = $.ajax({
    +          type: 'GET',
    +          url: "http://"+drillbit+":8047/status/metrics",
    --- End diff --
    
    So one way to find out a Drillbit's HTTP port is through this call:
    `http://localhost:8047/portNum `
    to get 
    ```
    {
      "port" : 8047
    }
    ```
    Notice how I need to use the port number in the request's URL to get that very number! 🤣 


---

[GitHub] drill issue #1160: DRILL-6224: Publish current memory usage for Drillbits in...

Posted by kkhatua <gi...@git.apache.org>.
Github user kkhatua commented on the issue:

    https://github.com/apache/drill/pull/1160
  
    I think I need to put up some tooltips.
    
    We're getting the metrics from the `drill.allocator.root.[peak|used]` values. I haven't seen a metric that shows the actual Direct allocation from the system. What Drill seems to report is the current **actively** used Direct memory. So, when running queries on the cluster, these numbers move up. The percentage for Direct is the current usage as a percent of the peak memory that was reported by the root allocator.
    
    Check out any active Drillbit's `#metrics` page. The `drill.allocator.*` entries are what I was referring to. JVM itself doesn't appear to expose the Direct memory information readily.


---

[GitHub] drill issue #1160: DRILL-6224: Publish current memory usage for Drillbits in...

Posted by parthchandra <gi...@git.apache.org>.
Github user parthchandra commented on the issue:

    https://github.com/apache/drill/pull/1160
  
    Looks wrong dunnit? Direct memory is (still?) showing as zero?


---

[GitHub] drill issue #1160: DRILL-6224: Publish current memory usage for Drillbits in...

Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on the issue:

    https://github.com/apache/drill/pull/1160
  
    +1


---

[GitHub] drill issue #1160: DRILL-6224: Publish current memory usage for Drillbits in...

Posted by kkhatua <gi...@git.apache.org>.
Github user kkhatua commented on the issue:

    https://github.com/apache/drill/pull/1160
  
    @parthchandra / @arina-ielchiieva  would you guys want to review this? 
    Here is a preview of how the details appear on the default page:
    ![image](https://user-images.githubusercontent.com/4335237/37267989-c42b7a42-2580-11e8-9581-2023895503eb.png)
    



---

[GitHub] drill pull request #1160: DRILL-6224: Publish current memory usage for Drill...

Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on a diff in the pull request:

    https://github.com/apache/drill/pull/1160#discussion_r176655100
  
    --- Diff: exec/java-exec/src/main/resources/rest/metrics/metrics.ftl ---
    @@ -109,18 +114,23 @@
         };
     
         function updateBars(gauges) {
    -      $.each(["heap","non-heap","total"], function(i, key) {
    +      $.each(["heap","non-heap","total","drill.allocator.root"], function(i, key) {
             var used    = gauges[key + ".used"].value;
    -        var max     = gauges[key + ".max"].value;
    -        var usage   = round((used / 1073741824), 2) + "GB";
    -        var percent = round((used / max), 2);
    -
    -        var styleVal = "width: " + percent + "%;"
    -        $("#" + key + "Usage").attr({
    +        var max;
    +        if (key.startsWith("drill.allocator")) {
    --- End diff --
    
    Please address the below comment.


---

[GitHub] drill pull request #1160: DRILL-6224: Publish current memory usage for Drill...

Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on a diff in the pull request:

    https://github.com/apache/drill/pull/1160#discussion_r174445380
  
    --- Diff: exec/java-exec/src/main/resources/rest/index.ftl ---
    @@ -192,6 +199,8 @@
           var port = getPortNum();
           var timeout;
           var size = $("#size").html();
    +      reloadMemory();
    +      setInterval(reloadMemory, refreshTime);
    --- End diff --
    
    In graceful shutdown `setTimeout` is used. Which one is better? Anayway it's better to be consistent in scheduler usage. Plus we do why need two schedulers? Can one work for both tasks? Plus if one drillbit is unavailable, there is no need to check memory.


---

[GitHub] drill pull request #1160: DRILL-6224: Publish current memory usage for Drill...

Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on a diff in the pull request:

    https://github.com/apache/drill/pull/1160#discussion_r174444069
  
    --- Diff: exec/java-exec/src/main/resources/rest/metrics/metrics.ftl ---
    @@ -109,18 +114,23 @@
         };
     
         function updateBars(gauges) {
    -      $.each(["heap","non-heap","total"], function(i, key) {
    +      $.each(["heap","non-heap","total","drill.allocator.root"], function(i, key) {
             var used    = gauges[key + ".used"].value;
    -        var max     = gauges[key + ".max"].value;
    -        var usage   = round((used / 1073741824), 2) + "GB";
    -        var percent = round((used / max), 2);
    -
    -        var styleVal = "width: " + percent + "%;"
    -        $("#" + key + "Usage").attr({
    +        var max;
    +        if (key.startsWith("drill.allocator")) {
    --- End diff --
    
    Using `key.startsWith("drill.allocator")` several times in this method is not very elegant. Besides it he fact it is repeated action, I do not like such checks in general. I guess that can be re-written.


---

[GitHub] drill pull request #1160: DRILL-6224: Publish current memory usage for Drill...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/drill/pull/1160


---

[GitHub] drill issue #1160: DRILL-6224: Publish current memory usage for Drillbits in...

Posted by kkhatua <gi...@git.apache.org>.
Github user kkhatua commented on the issue:

    https://github.com/apache/drill/pull/1160
  
    @arina-ielchiieva 
    Restricting this PR to address the bug in `metrics.ftl`
    Will create a new PR for [DRILL-6289](https://issues.apache.org/jira/browse/DRILL-6289)


---

[GitHub] drill pull request #1160: DRILL-6224: Publish current memory usage for Drill...

Posted by kkhatua <gi...@git.apache.org>.
Github user kkhatua commented on a diff in the pull request:

    https://github.com/apache/drill/pull/1160#discussion_r176579901
  
    --- Diff: exec/java-exec/src/main/resources/rest/index.ftl ---
    @@ -291,6 +300,46 @@
                 var url = protocol + "//" + host + requestPath;
                 return url;
           }
    +
    +      //Iterates through all the nodes for update
    +      function reloadMemory () {
    +          for (i = 1; i <= size; i++) {
    +              var address = $("#row-"+i).find("#address").contents().get(0).nodeValue.trim();
    +              updateMemory(address, i);
    +          }
    +      }
    +
    +      //Update memory
    +      function updateMemory(drillbit,idx) {
    +        var result = $.ajax({
    +          type: 'GET',
    +          url: "http://"+drillbit+":8047/status/metrics",
    --- End diff --
    
    Good point. I'll follow the same port extraction logic.


---