You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by "Madhan Neethiraj (JIRA)" <ji...@apache.org> on 2018/09/20 21:27:00 UTC
[jira] [Updated] (ATLAS-2883) Export & Import Process: Large Amount
of Data Prevents ExportImport Audits from Being Saved
[ https://issues.apache.org/jira/browse/ATLAS-2883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Madhan Neethiraj updated ATLAS-2883:
------------------------------------
Description:
*Background*
The new audit mechanism saves contents of Export/Import results to _AtlasExportImportAudit_. This is an entity within Atlas, that ends up as a vertex within the database.
*Steps to Reproduce*
* Populate Atlas with _hive_db_ with over 100 tables each with 100 columns.
* Perform an export.
The following is seen in logs:
{noformat}
audit record too long: entityType=__ExportImportAuditEntry, guid=5a5170fa-8f65-4037-a092-1abbaa725b2f, size=2099039; maxSize=1048576. entity attribute values not stored in audit (EntityAuditListener:153)
018-09-20 08:11:33,430 ERROR - [pool-2-thread-5 - f7441d48-1c37-4529-9956-23eaf90634d8:] ~ graph rollback due to exception (GraphTransactionInterceptor:71)
org.apache.atlas.exception.AtlasBaseException: Failed to notify EntityAuditListener for change CREATE
at org.apache.atlas.repository.store.graph.v1.AtlasEntityChangeNotifier.notifyListeners(AtlasEntityChangeNotifier.java:183)
at org.apache.atlas.repository.store.graph.v1.AtlasEntityChangeNotifier.onEntitiesMutated(AtlasEntityChangeNotifier.java:91)
at org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1.createOrUpdate(AtlasEntityStoreV1.java:207)
at org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1.createOrUpdate(AtlasEntityStoreV1.java:215)
at org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1$$FastClassBySpringCGLIB$$80c00649.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.apache.atlas.GraphTransactionInterceptor.invoke(GraphTransactionInterceptor.java:60)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
at org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1$$EnhancerBySpringCGLIB$$59a2a5e8.createOrUpdate(<generated>)
…
Caused by: org.apache.atlas.AtlasException: Unknown datatype {name=long, description=null}
at org.apache.atlas.typesystem.persistence.StructInstance.setNull(StructInstance.java:276)
at org.apache.atlas.repository.audit.EntityAuditListener.clearAttributeValues(EntityAuditListener.java:175)
at org.apache.atlas.repository.audit.EntityAuditListener.getAuditEventDetail(EntityAuditListener.java:158)
at org.apache.atlas.repository.audit.EntityAuditListener.createEvent(EntityAuditListener.java:133)
at org.apache.atlas.repository.audit.EntityAuditListener.onEntitiesAdded(EntityAuditListener.java:63)
at org.apache.atlas.repository.store.graph.v1.AtlasEntityChangeNotifier.notifyListeners(AtlasEntityChangeNotifier.java:172)
... 112 more
{noformat}
was:
*Background*
The new audit mechanism saves contents of Export/Import results to _AtlasExportImportAudit_. This is an entity within Atlas, that ends up as a vertex within the database.
*Steps to Reproduce*
* Populate Atlas with _hive_db_ with over 100 tables each with 100 columns.
* Perform an export.
The following is seen in logs:
_{{audit record too long: entityType=__ExportImportAuditEntry, guid=5a5170fa-8f65-4037-a092-1abbaa725b2f, size=2099039; maxSize=1048576. entity attribute values not stored in audit (EntityAuditListener:153)}}_
> Export & Import Process: Large Amount of Data Prevents ExportImport Audits from Being Saved
> -------------------------------------------------------------------------------------------
>
> Key: ATLAS-2883
> URL: https://issues.apache.org/jira/browse/ATLAS-2883
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
> Affects Versions: 0.8.2
> Reporter: Ashutosh Mestry
> Assignee: Ashutosh Mestry
> Priority: Major
> Fix For: 0.8.3
>
>
> *Background*
> The new audit mechanism saves contents of Export/Import results to _AtlasExportImportAudit_. This is an entity within Atlas, that ends up as a vertex within the database.
> *Steps to Reproduce*
> * Populate Atlas with _hive_db_ with over 100 tables each with 100 columns.
> * Perform an export.
> The following is seen in logs:
> {noformat}
> audit record too long: entityType=__ExportImportAuditEntry, guid=5a5170fa-8f65-4037-a092-1abbaa725b2f, size=2099039; maxSize=1048576. entity attribute values not stored in audit (EntityAuditListener:153)
> 018-09-20 08:11:33,430 ERROR - [pool-2-thread-5 - f7441d48-1c37-4529-9956-23eaf90634d8:] ~ graph rollback due to exception (GraphTransactionInterceptor:71)
> org.apache.atlas.exception.AtlasBaseException: Failed to notify EntityAuditListener for change CREATE
> at org.apache.atlas.repository.store.graph.v1.AtlasEntityChangeNotifier.notifyListeners(AtlasEntityChangeNotifier.java:183)
> at org.apache.atlas.repository.store.graph.v1.AtlasEntityChangeNotifier.onEntitiesMutated(AtlasEntityChangeNotifier.java:91)
> at org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1.createOrUpdate(AtlasEntityStoreV1.java:207)
> at org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1.createOrUpdate(AtlasEntityStoreV1.java:215)
> at org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1$$FastClassBySpringCGLIB$$80c00649.invoke(<generated>)
> at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
> at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
> at org.apache.atlas.GraphTransactionInterceptor.invoke(GraphTransactionInterceptor.java:60)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
> at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
> at org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1$$EnhancerBySpringCGLIB$$59a2a5e8.createOrUpdate(<generated>)
> …
> Caused by: org.apache.atlas.AtlasException: Unknown datatype {name=long, description=null}
> at org.apache.atlas.typesystem.persistence.StructInstance.setNull(StructInstance.java:276)
> at org.apache.atlas.repository.audit.EntityAuditListener.clearAttributeValues(EntityAuditListener.java:175)
> at org.apache.atlas.repository.audit.EntityAuditListener.getAuditEventDetail(EntityAuditListener.java:158)
> at org.apache.atlas.repository.audit.EntityAuditListener.createEvent(EntityAuditListener.java:133)
> at org.apache.atlas.repository.audit.EntityAuditListener.onEntitiesAdded(EntityAuditListener.java:63)
> at org.apache.atlas.repository.store.graph.v1.AtlasEntityChangeNotifier.notifyListeners(AtlasEntityChangeNotifier.java:172)
> ... 112 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)