You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by Madhan Neethiraj <ma...@apache.org> on 2018/12/18 19:49:51 UTC
Review Request 69587: ATLAS-3002: added instrumentation to collect
time taken for sub-tasks during entity create/update
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69587/
-----------------------------------------------------------
Review request for atlas, Abhishek Kadam, Ashutosh Mestry, keval bhatt, Kapildeo Nayak, Mehul Parikh, Nixon Rodrigues, and Sarath Subramanian.
Bugs: ATLAS-2003
https://issues.apache.org/jira/browse/ATLAS-2003
Repository: atlas
Description
-------
- added instrumentation to collect time taken for sub-tasks during entity create/update - like walkEntityGraph, resolveReferences, preCreateOrUpdate, mapAttributesAndClassifications, fullTextMapping, graphCommit, entityNotification, entityAudit
- updated default atlas-log4j.xml to include loggers for METRICS
- optimization: updated Hive hook to avoid including quertText in hive_column_lineage.name attribute value, as this can unnecessarily bloat the message size for large queries
- optimziation: updated fullTextMapper to skip going through object-id collections; this saves cycles for example while processing hive_table entities with large number of columns
- here is the sample metrics:
```json
{
"walkEntityGraph": { "count": 1, "timeTaken": 1 },
"resolveReferences": { "count": 1, "timeTaken": 670 },
"findByUniqueAttributes": { "count": 217, "timeTaken": 668 },
"findByTypeAndPropertyName": { "count": 217, "timeTaken": 641 },
"findBySuperTypeAndPropertyName": { "count": 74, "timeTaken": 27 },
"createOrUpdate": { "count": 1, "timeTaken": 1637 },
"preCreateOrUpdate": { "count": 1, "timeTaken": 720 },
"mapAttributesAndClassifications": { "count": 1, "timeTaken": 90 },
"fullTextMapping": { "count": 1, "timeTaken": 195 },
"notification-getReferenceable": { "count": 1, "timeTaken": 37 },
"entityAudit": { "count": 1, "timeTaken": 31 },
"entityNotification": { "count": 2, "timeTaken": 8 },
"graphCommit": { "count": 75, "timeTaken": 315 }
}
```
Diffs
-----
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateHiveProcess.java 2ccfff4d1
common/src/main/java/org/apache/atlas/utils/AtlasPerfMetrics.java PRE-CREATION
distro/src/conf/atlas-log4j.xml c183871eb
notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 4bec91709
repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java 4c436779e
repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListener.java dfacb3817
repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java 8ca8c9a0b
repository/src/main/java/org/apache/atlas/repository/graph/FullTextMapperV2.java 08ccd9c73
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java a8c3363d5
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityGraphDiscoveryV2.java 6580beecd
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java d97b74d9d
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java 25770a334
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java 6c830bafe
server-api/src/main/java/org/apache/atlas/RequestContext.java 099d713f6
webapp/src/main/java/org/apache/atlas/notification/EntityNotificationListenerV2.java e0a60a133
webapp/src/main/java/org/apache/atlas/notification/NotificationEntityChangeListener.java b5e7ed871
webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java b95594831
Diff: https://reviews.apache.org/r/69587/diff/1/
Testing
-------
- verified the instumentation output in metric.log
- pre-commit tests run: https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/869
Thanks,
Madhan Neethiraj
Re: Review Request 69587: ATLAS-3002: added instrumentation to
collect time taken for sub-tasks during entity create/update
Posted by Ashutosh Mestry <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69587/#review211414
-----------------------------------------------------------
Ship it!
Current implementation provides lot of value in getting metrics per request. The numbers are cumulative. If same operation is called then the time taken becomes additive. Per request time can be calculated using: timeTaken/count.
While this is helpful, it would be worthwhile having per request metrics as well. This will help in understanding call to call variations.
- Ashutosh Mestry
On Dec. 18, 2018, 7:49 p.m., Madhan Neethiraj wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69587/
> -----------------------------------------------------------
>
> (Updated Dec. 18, 2018, 7:49 p.m.)
>
>
> Review request for atlas, Abhishek Kadam, Ashutosh Mestry, keval bhatt, Kapildeo Nayak, Mehul Parikh, Nixon Rodrigues, and Sarath Subramanian.
>
>
> Bugs: ATLAS-3002
> https://issues.apache.org/jira/browse/ATLAS-3002
>
>
> Repository: atlas
>
>
> Description
> -------
>
> - added instrumentation to collect time taken for sub-tasks during entity create/update - like walkEntityGraph, resolveReferences, preCreateOrUpdate, mapAttributesAndClassifications, fullTextMapping, graphCommit, entityNotification, entityAudit
> - updated default atlas-log4j.xml to include loggers for METRICS
> - optimization: updated Hive hook to avoid including quertText in hive_column_lineage.name attribute value, as this can unnecessarily bloat the message size for large queries
> - optimziation: updated fullTextMapper to skip going through object-id collections; this saves cycles for example while processing hive_table entities with large number of columns
> - here is the sample metrics:
> ```json
> {
> "walkEntityGraph": { "count": 1, "timeTaken": 1 },
> "resolveReferences": { "count": 1, "timeTaken": 670 },
> "findByUniqueAttributes": { "count": 217, "timeTaken": 668 },
> "findByTypeAndPropertyName": { "count": 217, "timeTaken": 641 },
> "findBySuperTypeAndPropertyName": { "count": 74, "timeTaken": 27 },
> "createOrUpdate": { "count": 1, "timeTaken": 1637 },
> "preCreateOrUpdate": { "count": 1, "timeTaken": 720 },
> "mapAttributesAndClassifications": { "count": 1, "timeTaken": 90 },
> "fullTextMapping": { "count": 1, "timeTaken": 195 },
> "notification-getReferenceable": { "count": 1, "timeTaken": 37 },
> "entityAudit": { "count": 1, "timeTaken": 31 },
> "entityNotification": { "count": 2, "timeTaken": 8 },
> "graphCommit": { "count": 75, "timeTaken": 315 }
> }
> ```
>
>
> Diffs
> -----
>
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateHiveProcess.java 2ccfff4d1
> common/src/main/java/org/apache/atlas/utils/AtlasPerfMetrics.java PRE-CREATION
> distro/src/conf/atlas-log4j.xml c183871eb
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 4bec91709
> repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java 4c436779e
> repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListener.java dfacb3817
> repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java 8ca8c9a0b
> repository/src/main/java/org/apache/atlas/repository/graph/FullTextMapperV2.java 08ccd9c73
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java a8c3363d5
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityGraphDiscoveryV2.java 6580beecd
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java d97b74d9d
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java 25770a334
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java 6c830bafe
> server-api/src/main/java/org/apache/atlas/RequestContext.java 099d713f6
> webapp/src/main/java/org/apache/atlas/notification/EntityNotificationListenerV2.java e0a60a133
> webapp/src/main/java/org/apache/atlas/notification/NotificationEntityChangeListener.java b5e7ed871
> webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java b95594831
>
>
> Diff: https://reviews.apache.org/r/69587/diff/1/
>
>
> Testing
> -------
>
> - verified the instumentation output in metric.log
> - pre-commit tests run: https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/869
>
>
> Thanks,
>
> Madhan Neethiraj
>
>
Re: Review Request 69587: ATLAS-3002: added instrumentation to
collect time taken for sub-tasks during entity create/update
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69587/
-----------------------------------------------------------
(Updated Dec. 18, 2018, 11:18 p.m.)
Review request for atlas, Abhishek Kadam, Ashutosh Mestry, keval bhatt, Kapildeo Nayak, Mehul Parikh, Nixon Rodrigues, and Sarath Subramanian.
Changes
-------
Ashutosh - thanks for reviewing and catching the cumulative counter issue in case of import. This update addresses this issue.
Bugs: ATLAS-3002
https://issues.apache.org/jira/browse/ATLAS-3002
Repository: atlas
Description
-------
- added instrumentation to collect time taken for sub-tasks during entity create/update - like walkEntityGraph, resolveReferences, preCreateOrUpdate, mapAttributesAndClassifications, fullTextMapping, graphCommit, entityNotification, entityAudit
- updated default atlas-log4j.xml to include loggers for METRICS
- optimization: updated Hive hook to avoid including quertText in hive_column_lineage.name attribute value, as this can unnecessarily bloat the message size for large queries
- optimziation: updated fullTextMapper to skip going through object-id collections; this saves cycles for example while processing hive_table entities with large number of columns
- here is the sample metrics:
```json
{
"walkEntityGraph": { "count": 1, "timeTaken": 1 },
"resolveReferences": { "count": 1, "timeTaken": 670 },
"findByUniqueAttributes": { "count": 217, "timeTaken": 668 },
"findByTypeAndPropertyName": { "count": 217, "timeTaken": 641 },
"findBySuperTypeAndPropertyName": { "count": 74, "timeTaken": 27 },
"createOrUpdate": { "count": 1, "timeTaken": 1637 },
"preCreateOrUpdate": { "count": 1, "timeTaken": 720 },
"mapAttributesAndClassifications": { "count": 1, "timeTaken": 90 },
"fullTextMapping": { "count": 1, "timeTaken": 195 },
"notification-getReferenceable": { "count": 1, "timeTaken": 37 },
"entityAudit": { "count": 1, "timeTaken": 31 },
"entityNotification": { "count": 2, "timeTaken": 8 },
"graphCommit": { "count": 75, "timeTaken": 315 }
}
```
Diffs (updated)
-----
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateHiveProcess.java 2ccfff4d1
common/src/main/java/org/apache/atlas/utils/AtlasPerfMetrics.java PRE-CREATION
distro/src/conf/atlas-log4j.xml c183871eb
notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 4bec91709
repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java 4c436779e
repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListener.java dfacb3817
repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java 8ca8c9a0b
repository/src/main/java/org/apache/atlas/repository/graph/FullTextMapperV2.java 08ccd9c73
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java a8c3363d5
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityGraphDiscoveryV2.java 6580beecd
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java d97b74d9d
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java 25770a334
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java 6c830bafe
server-api/src/main/java/org/apache/atlas/RequestContext.java 099d713f6
webapp/src/main/java/org/apache/atlas/notification/EntityNotificationListenerV2.java e0a60a133
webapp/src/main/java/org/apache/atlas/notification/NotificationEntityChangeListener.java b5e7ed871
webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java b344c50e6
Diff: https://reviews.apache.org/r/69587/diff/2/
Changes: https://reviews.apache.org/r/69587/diff/1-2/
Testing
-------
- verified the instumentation output in metric.log
- pre-commit tests run: https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/869
Thanks,
Madhan Neethiraj