[jira] [Comment Edited] (SOLR-15251) Metrics changes in Solr 8 cause admin dashboard to render incorrectly

["Dwane Hall (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SOLR-15251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17301292#comment-17301292 ] 

Dwane Hall edited comment on SOLR-15251 at 3/14/21, 10:52 PM:
--------------------------------------------------------------

[~epugh] I completed a bit more troubleshooting this morning and it turns out it's not the browser caching the JavaScript files causing this issue. The metrics API looks to use the java MBeans Interfaces to get OS Metrics ([java.lang.management.OperatingSystemMXBean|https://docs.oracle.com/en/java/javase/11/docs/api/jdk.management/com/sun/management/OperatingSystemMXBean.html]) so I put together a simple class to see what was happening on each docker container when calling this MBean directly.
{code:java}
import java.lang.management.ManagementFactory;
import com.sun.management.OperatingSystemMXBean;

public class OSTest {
public static void main (String[] args) {
    OperatingSystemMXBean bean = (com.sun.management.OperatingSystemMXBean) ManagementFactory.getOperatingSystemMXBean();

    System.out.println("Free swap size: " + bean.getFreeSwapSpaceSize());
    System.out.println("Total swap size: " + bean.getTotalSwapSpaceSize());
    System.out.println("Total free physical memory size: " + bean.getFreePhysicalMemorySize());
    System.out.println("Total physical memory size: " + bean.getTotalPhysicalMemorySize());
    }
}
{code}
Result on Solr 7.7.2 (Java build 11.0.4+11) - As expected
 Free swap size: 0
 Total swap size: 0
 Total free PM size: 760331739136
 Total PM size: 810586099712

Result on Solr 8.8.1 (Java build 11.0.10+9) - MBean collection fails
 Exception in thread "main" java.lang.NullPointerException
 at java.base/jdk.internal.platform.cgroupv1.Metrics.getMemoryAndSwapLimit(Unknown Source)
 at jdk.management/com.sun.management.internal.OperatingSystemImpl.getFreeSwapSpaceSize(Unknown Source)
 at OSTest.main(OSTest.java:10)

It looks like the MBean can't get the metrics from our host with the newer Java version (Currently SUSE12 SP3 - Linux 4.4.180-94.130-default x86_64) which results in the metrics not being collected and as such they're not available in the admin UI's JavaScript to render. As I mentioned above this worked without issue on my Ubuntu environment (5.4 kernel) so there's something environment specific here that our SUSE hosts aren't liking.


was (Author: dartfish):
[~epugh] I completed a bit more troubleshooting this morning and it turns out it's not the browser caching the JavaScript files causing this issue. The metrics API looks to use the java MBeans Interfaces to get OS Metrics ([java.lang.management.OperatingSystemMXBean|https://docs.oracle.com/en/java/javase/11/docs/api/jdk.management/com/sun/management/OperatingSystemMXBean.html]) so I put together a simple class to see what was happening on each docker container when calling this MBean directly.
{code:java}
import java.lang.management.ManagementFactory;
import com.sun.management.OperatingSystemMXBean;

public class OSTest {
public static void main (String[] args) {
    OperatingSystemMXBean bean = (com.sun.management.OperatingSystemMXBean) ManagementFactory.getOperatingSystemMXBean();

    System.out.println("Free swap size: " + bean.getFreeSwapSpaceSize());
    System.out.println("Total swap size: " + bean.getTotalSwapSpaceSize());
	System.out.println("Total free physical memory size: " + bean.getFreePhysicalMemorySize());
	System.out.println("Total physical memory size: " + bean.getTotalPhysicalMemorySize());
    }
}
{code}
Result on Solr 7.7.2 (Java build 11.0.4+11) - As expected
 Free swap size: 0
 Total swap size: 0
 Total free PM size: 760331739136
 Total PM size: 810586099712

Result on Solr 8.8.1 (Java build 11.0.10+9) - MBean collection fails
 Exception in thread "main" java.lang.NullPointerException
 at java.base/jdk.internal.platform.cgroupv1.Metrics.getMemoryAndSwapLimit(Unknown Source)
 at jdk.management/com.sun.management.internal.OperatingSystemImpl.getFreeSwapSpaceSize(Unknown Source)
 at OSTest.main(OSTest.java:10)

It looks like the MBean can't get the metrics from our host with the newer Java version (Currently SUSE12 SP3 - Linux 4.4.180-94.130-default x86_64) which results in the metrics not being collected and as such they're not available in the admin UI's JavaScript to render. As I mentioned above this worked without issue on my Ubuntu environment (5.4 kernel) so there's something environment specific here that our SUSE hosts aren't liking.

> Metrics changes in Solr 8 cause admin dashboard to render incorrectly
> ---------------------------------------------------------------------
>
>                 Key: SOLR-15251
>                 URL: https://issues.apache.org/jira/browse/SOLR-15251
>             Project: Solr
>          Issue Type: Bug
>      Security Level: Public(Default Security Level. Issues are Public) 
>          Components: Admin UI, metrics
>    Affects Versions: 8.8.1
>            Reporter: Dwane Hall
>            Assignee: David Eric Pugh
>            Priority: Minor
>
> The metrics API appears to have changed between Solr 7 and 8. Some of the effected metrics are used on the Admin UI Dashboard view which results in the view not rendering correctly. The main culprits appear to be os.totalPhysicalMemorySize not being calculated in Solr 8 and two missing metrics os.totalSwapSpaceSize and os.freeSwapSpaceSize which are all used in the Admin UI Dashboard view page.   Values memory.heap.committed, memory.heap.init, memory.heap.max also appear to be using the identical value in Solr 8. Below is an extract of the javascrpit used on the Admin Dashboard and a comparison between metrics reported in Solr 7 and 8.  The function "parse_memory_value" is where the JavaScript error appears to be thrown with the missing metrics.
> {code:java}
> Solr 8
> "os.totalPhysicalMemorySize":0, (Not calculated)
> "os.freePhysicalMemorySize":792087998464,
> "os.totalSwapSpaceSize" (Metric not present)
> "os.freeSwapSpaceSize": (Metric not present)
> "memory.heap.committed":8589934592,
> "memory.heap.init":8589934592,
> "memory.heap.max":8589934592,
> "memory.heap.usage":0.006413557566702366,
> "memory.heap.used":55092040,
> "memory.non-heap.committed":97910784,
> "memory.non-heap.init":7667712,
> "memory.non-heap.max":-1,
> "memory.non-heap.usage":-9.2249552E7,
> "memory.non-heap.used":92249712,
>  Solr 7
> "os.totalPhysicalMemorySize":810586099712,
> "os.freePhysicalMemorySize":756665888768,
> "os.totalSwapSpaceSize":0,
> "os.freeSwapSpaceSize":0
> "memory.heap.committed":12348030976,
> "memory.heap.init":12884901888,
> "memory.heap.max":12348030976,
> "memory.heap.usage":0.313836514301922,
> "memory.heap.used":3875263000,
> "memory.non-heap.committed":145039360,
> "memory.non-heap.init":7667712,
> "memory.non-heap.max":-1,
> "memory.non-heap.usage":-1.30145664E8,
> "memory.non-heap.used":130145824, 
> +main.js (Admin Dashboard)+
> // physical memory
> var memoryMax = parse_memory_value(data.system.totalPhysicalMemorySize);
> $scope.memoryTotal = parse_memory_value(data.system.totalPhysicalMemorySize - data.system.freePhysicalMemorySize);
> $scope.memoryPercentage = ($scope.memoryTotal / memoryMax * 100).toFixed(1)+ "%";
> $scope.memoryMax = pretty_print_bytes(memoryMax);
> $scope.memoryTotalDisplay = pretty_print_bytes($scope.memoryTotal);
> // swap space
> var swapMax = parse_memory_value(data.system.totalSwapSpaceSize);
> $scope.swapTotal = parse_memory_value(data.system.totalSwapSpaceSize - data.system.freeSwapSpaceSize);
> $scope.swapPercentage = ($scope.swapTotal / swapMax * 100).toFixed(1)+ "%";
> $scope.swapMax = pretty_print_bytes(swapMax);
> $scope.swapTotalDisplay = pretty_print_bytes($scope.swapTotal); 
> // file handles
> $scope.fileDescriptorPercentage = (data.system.openFileDescriptorCount / data.system.maxFileDescriptorCount *100).toFixed(1) + "%";
> // java memory
> var javaMemoryMax = parse_memory_value(data.jvm.memory.raw.max || data.jvm.memory.max);
> $scope.javaMemoryTotal = parse_memory_value(data.jvm.memory.raw.total || data.jvm.memory.total);
> $scope.javaMemoryUsed = parse_memory_value(data.jvm.memory.raw.used || data.jvm.memory.used);
> $scope.javaMemoryTotalPercentage = ($scope.javaMemoryTotal / javaMemoryMax *100).toFixed(1) + "%";
> $scope.javaMemoryUsedPercentage = ($scope.javaMemoryUsed / $scope.javaMemoryTotal *100).toFixed(1) + "%";
> $scope.javaMemoryPercentage = ($scope.javaMemoryUsed / javaMemoryMax * 100).toFixed(1) + "%";
> $scope.javaMemoryTotalDisplay = pretty_print_bytes($scope.javaMemoryTotal);
> $scope.javaMemoryUsedDisplay = pretty_print_bytes($scope.javaMemoryUsed);  // @todo These should really be an AngularJS Filter: {{ javaMemoryUsed | bytes }}
> $scope.javaMemoryMax = pretty_print_bytes(javaMemoryMax);
> var parse_memory_value = function( value ) {
>   if( value !== Number( value ) )
>   {
>     var units = 'BKMGTPEZY';
>     var match = value.match( /^(\d+([,\.]\d+)?) (\w).*$/ );
>     var value = parseFloat( match[1] ) * Math.pow( 1024, units.indexOf( match[3].toUpperCase() ) );
>   }
>   return value;
> };
> {code}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)