You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@atlas.apache.org by Paresh Devalia <pa...@freestoneinfotech.com> on 2024/07/11 13:51:45 UTC

Re: Review Request 75014: ATLAS-4867 : Improve Metrics API

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/75014/
-----------------------------------------------------------

(Updated July 11, 2024, 1:51 p.m.)


Review request for atlas, Mandar Ambawane and Pinal Shah.


Bugs: ATLAS-4867
    https://issues.apache.org/jira/browse/ATLAS-4867


Repository: atlas


Description
-------

Metrics API : http://<>/api/atlas/admin/metrics

1. There is some information in the Metrics API response, which is not been used like Count of the typeAndSubTypes, Active, deleted, Shell
Approach:
Identify time taken by the above counts
Take reading of metric api
Take reading of metric api without the above counts


2. Also we can get the total number of count of vertices and edges present in graph
This  helps in estimating the time on Reindexing Patch.
Also it may be helpful while debugging the escalations
Approach:
Embed this count in MetricUtil
Or
Create seperate API


Diffs (updated)
-----

  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java 004cab9c8 
  intg/src/main/java/org/apache/atlas/model/graph/AtlasGraphDetails.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/services/MetricsService.java 9ec2cd2cb 
  repository/src/main/java/org/apache/atlas/util/AtlasMetricsUtil.java a13e07387 
  webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 475756981 


Diff: https://reviews.apache.org/r/75014/diff/2/

Changes: https://reviews.apache.org/r/75014/diff/1-2/


Testing (updated)
-------

With typeAndSubTypes data it took 217ms for Metric API 
Without typeAndSubTypes data, it took 180ms for Metric API 
Without typeAndSubTypes and include the graph vertex and edge count took almost 300++ ms

PC build:- https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1662/console

Create a new API to fetch the graph details.

Graph API Request :- http://<>/api/atlas/admin/graph

Response :- 
{
    "data": {
        "general": {
            "Graph:": {
                "Graph:edgesCount": 286,
                "Graph:verticesCount": 697
            }
        }
    }
}


File Attachments (updated)
----------------

AtlasGraphDetails.java
  https://reviews.apache.org/media/uploaded/files/2024/07/11/3199de1d-ba43-4951-bc01-d42462c38518__AtlasGraphDetails.java


Thanks,

Paresh Devalia

Re: Review Request 75014: ATLAS-4867 : Improve Metrics API

Posted by Pinal Shah <pi...@freestoneinfotech.com>.


> On Jan. 20, 2025, 2:24 p.m., Pinal Shah wrote:
> > Ship It!

fix below


- Pinal


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/75014/#review227185
-----------------------------------------------------------


On Jan. 20, 2025, 6:23 a.m., Paresh Devalia wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/75014/
> -----------------------------------------------------------
> 
> (Updated Jan. 20, 2025, 6:23 a.m.)
> 
> 
> Review request for atlas, Mandar Ambawane and Pinal Shah.
> 
> 
> Bugs: ATLAS-4867
>     https://issues.apache.org/jira/browse/ATLAS-4867
> 
> 
> Repository: atlas
> 
> 
> Description
> -------
> 
> Metrics API : http://<>/api/atlas/admin/metrics
> 
> 1. There is some information in the Metrics API response, which is not been used like Count of the typeAndSubTypes, Active, deleted, Shell
> Approach:
> Identify time taken by the above counts
> Take reading of metric api
> Take reading of metric api without the above counts
> 
> 
> Diffs
> -----
> 
>   repository/src/main/java/org/apache/atlas/services/MetricsService.java 9ec2cd2cb 
> 
> 
> Diff: https://reviews.apache.org/r/75014/diff/3/
> 
> 
> Testing
> -------
> 
> With typeAndSubTypes data it took 4578ms for Active (22,994,801), Deleted (139,398) and Shell (166,280) for Metric API 
> First Time -> Without typeAndSubTypes data, it took 2927ms for Metric API 
> Then two subsequent request took around 2389ms and 2234ms  for metric API.
> 
> +--------------------------------------------------------+--------------------------+-------------------------------+
> |                                                        | Time taken first request | Time taken subsequent request |
> | typeAndSubTypes, Active, deleted, Shell  in Metric API |                          |                               |
> +========================================================+==========================+===============================+
> | With typeAndSubTypes                                   | 4578ms                   | 4152ms and 3914ms             |
> +--------------------------------------------------------+--------------------------+-------------------------------+
> | Without typeAndSubTypes                                | 2927ms                   | 2389ms and 2234ms             |
> +--------------------------------------------------------+--------------------------+-------------------------------+
> 
> 
> PC build:- https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1715/
> 
> 
> Thanks,
> 
> Paresh Devalia
> 
>

Re: Review Request 75014: ATLAS-4867 : Improve Metrics API

Posted by Pinal Shah <pi...@freestoneinfotech.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/75014/#review227185
-----------------------------------------------------------


Ship it!




Ship It!

- Pinal Shah


On Jan. 20, 2025, 6:23 a.m., Paresh Devalia wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/75014/
> -----------------------------------------------------------
> 
> (Updated Jan. 20, 2025, 6:23 a.m.)
> 
> 
> Review request for atlas, Mandar Ambawane and Pinal Shah.
> 
> 
> Bugs: ATLAS-4867
>     https://issues.apache.org/jira/browse/ATLAS-4867
> 
> 
> Repository: atlas
> 
> 
> Description
> -------
> 
> Metrics API : http://<>/api/atlas/admin/metrics
> 
> 1. There is some information in the Metrics API response, which is not been used like Count of the typeAndSubTypes, Active, deleted, Shell
> Approach:
> Identify time taken by the above counts
> Take reading of metric api
> Take reading of metric api without the above counts
> 
> 
> Diffs
> -----
> 
>   repository/src/main/java/org/apache/atlas/services/MetricsService.java 9ec2cd2cb 
> 
> 
> Diff: https://reviews.apache.org/r/75014/diff/3/
> 
> 
> Testing
> -------
> 
> With typeAndSubTypes data it took 4578ms for Active (22,994,801), Deleted (139,398) and Shell (166,280) for Metric API 
> First Time -> Without typeAndSubTypes data, it took 2927ms for Metric API 
> Then two subsequent request took around 2389ms and 2234ms  for metric API.
> 
> +--------------------------------------------------------+--------------------------+-------------------------------+
> |                                                        | Time taken first request | Time taken subsequent request |
> | typeAndSubTypes, Active, deleted, Shell  in Metric API |                          |                               |
> +========================================================+==========================+===============================+
> | With typeAndSubTypes                                   | 4578ms                   | 4152ms and 3914ms             |
> +--------------------------------------------------------+--------------------------+-------------------------------+
> | Without typeAndSubTypes                                | 2927ms                   | 2389ms and 2234ms             |
> +--------------------------------------------------------+--------------------------+-------------------------------+
> 
> 
> PC build:- https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1715/
> 
> 
> Thanks,
> 
> Paresh Devalia
> 
>

Re: Review Request 75014: ATLAS-4867 : Improve Metrics API

Posted by Pinal Shah <pi...@freestoneinfotech.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/75014/#review227188
-----------------------------------------------------------



Please introduce flag, to skip these metrics to support backward compatibility

- Pinal Shah


On Jan. 20, 2025, 6:23 a.m., Paresh Devalia wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/75014/
> -----------------------------------------------------------
> 
> (Updated Jan. 20, 2025, 6:23 a.m.)
> 
> 
> Review request for atlas, Mandar Ambawane and Pinal Shah.
> 
> 
> Bugs: ATLAS-4867
>     https://issues.apache.org/jira/browse/ATLAS-4867
> 
> 
> Repository: atlas
> 
> 
> Description
> -------
> 
> Metrics API : http://<>/api/atlas/admin/metrics
> 
> 1. There is some information in the Metrics API response, which is not been used like Count of the typeAndSubTypes, Active, deleted, Shell
> Approach:
> Identify time taken by the above counts
> Take reading of metric api
> Take reading of metric api without the above counts
> 
> 
> Diffs
> -----
> 
>   repository/src/main/java/org/apache/atlas/services/MetricsService.java 9ec2cd2cb 
> 
> 
> Diff: https://reviews.apache.org/r/75014/diff/3/
> 
> 
> Testing
> -------
> 
> With typeAndSubTypes data it took 4578ms for Active (22,994,801), Deleted (139,398) and Shell (166,280) for Metric API 
> First Time -> Without typeAndSubTypes data, it took 2927ms for Metric API 
> Then two subsequent request took around 2389ms and 2234ms  for metric API.
> 
> +--------------------------------------------------------+--------------------------+-------------------------------+
> |                                                        | Time taken first request | Time taken subsequent request |
> | typeAndSubTypes, Active, deleted, Shell  in Metric API |                          |                               |
> +========================================================+==========================+===============================+
> | With typeAndSubTypes                                   | 4578ms                   | 4152ms and 3914ms             |
> +--------------------------------------------------------+--------------------------+-------------------------------+
> | Without typeAndSubTypes                                | 2927ms                   | 2389ms and 2234ms             |
> +--------------------------------------------------------+--------------------------+-------------------------------+
> 
> 
> PC build:- https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1715/
> 
> 
> Thanks,
> 
> Paresh Devalia
> 
>