You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Will Xiao (Jira)" <ji...@apache.org> on 2024/03/12 16:53:00 UTC

[jira] [Comment Edited] (HDDS-10311) Speed up TestOmMetrics

    [ https://issues.apache.org/jira/browse/HDDS-10311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17825733#comment-17825733 ] 

Will Xiao edited comment on HDDS-10311 at 3/12/24 4:52 PM:
-----------------------------------------------------------

Hi [~adoroszlai], I tried to use @BeforeAll and @AfterAll to build cluster only once
{code:java}
@TestInstance(Lifecycle.PER_CLASS)
@Timeout(300)
public class TestOmMetrics {
  private MiniOzoneCluster cluster;
  private MiniOzoneCluster.Builder clusterBuilder;
  private OzoneConfiguration conf;
  private OzoneManager ozoneManager;
  private OzoneManagerProtocol writeClient;
  private final OMException exception =
      new OMException("dummyException", OMException.ResultCodes.TIMEOUT);
  private OzoneClient client;

  @BeforeAll
  public void setup() throws Exception {
    conf = new OzoneConfiguration();
    conf.setTimeDuration(OMConfigKeys.OZONE_OM_METRICS_SAVE_INTERVAL,
        1000, TimeUnit.MILLISECONDS);
    clusterBuilder = MiniOzoneCluster.newBuilder(conf).withoutDatanodes();
    startCluster();
  }

  private void startCluster() throws Exception {
    cluster = clusterBuilder.build();
    cluster.waitForClusterToBeReady();
    ozoneManager = cluster.getOzoneManager();
    client = cluster.newClient();
    writeClient = client.getObjectStore()
        .getClientProxy().getOzoneManagerClient();
  }

  @AfterAll
  public void shutdown() {
    IOUtils.closeQuietly(client);
    if (cluster != null) {
      cluster.shutdown();
    }
  }{code}
Only keep testVolumeOps() and testBucketOps() and comment out other test classes.
removed startCluster() from each test class.
The mvn test failed at this line testVolumeOps:122   
{code:java}
MetricsRecordBuilder omMetrics = getMetrics("OMMetrics");{code}
ERROR:
{code:java}
[ERROR] org.apache.hadoop.ozone.om.TestOmMetrics.testBucketOps  Time elapsed: 4.072 s [ERROR] org.apache.hadoop.ozone.om.TestOmMetrics.testVolumeOps  Time elapsed: 4.094 s  <<< ERROR! java.lang.NullPointerException at org.apache.ozone.test.MetricsAsserts.getMetrics(MetricsAsserts.java:101) at org.apache.ozone.test.MetricsAsserts.getMetrics(MetricsAsserts.java:110) at org.apache.ozone.test.MetricsAsserts.getMetrics(MetricsAsserts.java:106) at org.apache.hadoop.ozone.om.TestOmMetrics.testVolumeOps(TestOmMetrics.java:122) at java.lang.reflect.Method.invoke(Method.java:498) at java.util.ArrayList.forEach(ArrayList.java:1259) at java.util.ArrayList.forEach(ArrayList.java:1259)  {code}
I tried to remove all other test classes and only keep testVolumeOps(), and it was successful. looks like getMetrics in both tests are in conflict.
Any idea how to handle getMetrics if we want both test classes to share one cluster?


was (Author: JIRAUSER296695):
Hi [~adoroszlai], I tried to use @BeforeAll and @AfterAll to build cluster only once
{code:java}
@TestInstance(Lifecycle.PER_CLASS)
@Timeout(300)
public class TestOmMetrics {
  private MiniOzoneCluster cluster;
  private MiniOzoneCluster.Builder clusterBuilder;
  private OzoneConfiguration conf;
  private OzoneManager ozoneManager;
  private OzoneManagerProtocol writeClient;
  private final OMException exception =
      new OMException("dummyException", OMException.ResultCodes.TIMEOUT);
  private OzoneClient client;

  @BeforeAll
  public void setup() throws Exception {
    conf = new OzoneConfiguration();
    conf.setTimeDuration(OMConfigKeys.OZONE_OM_METRICS_SAVE_INTERVAL,
        1000, TimeUnit.MILLISECONDS);
    clusterBuilder = MiniOzoneCluster.newBuilder(conf).withoutDatanodes();
    startCluster();
  }

  private void startCluster() throws Exception {
    cluster = clusterBuilder.build();
    cluster.waitForClusterToBeReady();
    ozoneManager = cluster.getOzoneManager();
    client = cluster.newClient();
    writeClient = client.getObjectStore()
        .getClientProxy().getOzoneManagerClient();
  }

  @AfterAll
  public void shutdown() {
    IOUtils.closeQuietly(client);
    if (cluster != null) {
      cluster.shutdown();
    }
  }{code}
Only keep testVolumeOps() and testBucketOps() and comment out other test classes.
removed startCluster() from each test class.
The mvn test failed at this line testVolumeOps:122   
{code:java}
MetricsRecordBuilder omMetrics = getMetrics("OMMetrics");{code}
ERROR:
 
{code:java}
[ERROR] org.apache.hadoop.ozone.om.TestOmMetrics.testBucketOps  Time elapsed: 4.072 s [ERROR] org.apache.hadoop.ozone.om.TestOmMetrics.testVolumeOps  Time elapsed: 4.094 s  <<< ERROR! java.lang.NullPointerException at org.apache.ozone.test.MetricsAsserts.getMetrics(MetricsAsserts.java:101) at org.apache.ozone.test.MetricsAsserts.getMetrics(MetricsAsserts.java:110) at org.apache.ozone.test.MetricsAsserts.getMetrics(MetricsAsserts.java:106) at org.apache.hadoop.ozone.om.TestOmMetrics.testVolumeOps(TestOmMetrics.java:122) at java.lang.reflect.Method.invoke(Method.java:498) at java.util.ArrayList.forEach(ArrayList.java:1259) at java.util.ArrayList.forEach(ArrayList.java:1259)  {code}
I tried to remove all other test classes and only keep testVolumeOps(), and it was successful. looks like getMetrics in both tests are in conflict.
Any idea how to handle getMetrics if we want both test classes to share one cluster?

> Speed up TestOmMetrics
> ----------------------
>
>                 Key: HDDS-10311
>                 URL: https://issues.apache.org/jira/browse/HDDS-10311
>             Project: Apache Ozone
>          Issue Type: Sub-task
>          Components: test
>            Reporter: Attila Doroszlai
>            Assignee: Will Xiao
>            Priority: Major
>
> {code}
> Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 115.365 s - in org.apache.hadoop.ozone.om.TestOmMetrics
> {code}
> # assertions about metrics should be merged into functional tests (no need to test e.g. {{listKeys}} and metrics for {{listKeys}} separately
> # no need to start new cluster for each test case



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org