You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by "Péter Gergő Barna (JIRA)" <ji...@apache.org> on 2017/06/28 13:06:01 UTC
[jira] [Comment Edited] (ATLAS-1879) Updating classification
removes some properties
[ https://issues.apache.org/jira/browse/ATLAS-1879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16066140#comment-16066140 ]
Péter Gergő Barna edited comment on ATLAS-1879 at 6/28/17 1:05 PM:
-------------------------------------------------------------------
[~madhan.neethiraj],
I understand the problem is the updateTypesDef method updates the TypeRegistry first, then the GraphStorage. First the TypeRegistry is updated with the user input, which does not necessarily contain all fields. Then the updateGraphStore method updates the storage, and this method does not updates missing fields back into the registry.
repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasTypeDefGraphStore.java
{noformat}
@Override
@GraphTransaction
public AtlasTypesDef updateTypesDef(AtlasTypesDef typesDef) throws AtlasBaseException {
if (LOG.isDebugEnabled()) {
LOG.debug("==> AtlasTypeDefGraphStore.updateTypesDef(enums={}, structs={}, classfications={}, entities={})",
CollectionUtils.size(typesDef.getEnumDefs()),
CollectionUtils.size(typesDef.getStructDefs()),
CollectionUtils.size(typesDef.getClassificationDefs()),
CollectionUtils.size(typesDef.getEntityDefs()));
}
AtlasTransientTypeRegistry ttr = lockTypeRegistryAndReleasePostCommit();
// Translate any NOT FOUND errors to BAD REQUEST
try {
ttr.updateTypes(typesDef);
} catch (AtlasBaseException e) {
if (AtlasErrorCode.TYPE_NAME_NOT_FOUND == e.getAtlasErrorCode()) {
throw new AtlasBaseException(AtlasErrorCode.BAD_REQUEST, e.getMessage());
} else {
throw e;
}
}
AtlasTypesDef ret = updateGraphStore(typesDef, ttr);
if (LOG.isDebugEnabled()) {
LOG.debug("<== AtlasTypeDefGraphStore.updateTypesDef(enums={}, structs={}, classfications={}, entities={})",
CollectionUtils.size(typesDef.getEnumDefs()),
CollectionUtils.size(typesDef.getStructDefs()),
CollectionUtils.size(typesDef.getClassificationDefs()),
CollectionUtils.size(typesDef.getEntityDefs()));
}
return ret;
{noformat}
I figured, a possible simple solution would be the GraphStorage being updated first, and then the TypeRegistry:
{noformat}
AtlasTypesDef ret = updateGraphStore(typesDef, ttr);
ttr.updateTypes(ret);
{noformat}
Unfortunately, that will cause a NullPointerException:
{noformat}
2017-06-26 19:42:59,802 ERROR - [pool-1-thread-7 - /api/atlas/v2/types/typedefs - 23dc280b-d4e9-4c98-9e80-5496b77a98e3:] ~ graph rollback due to exception (GraphTransactionInterceptor:71)
java.lang.NullPointerException
at org.apache.atlas.repository.store.graph.v1.AtlasStructDefStoreV1.toJsonFromAttribute(AtlasStructDefStoreV1.java:508)
at org.apache.atlas.repository.store.graph.v1.AtlasStructDefStoreV1.updateVertexPreUpdate(AtlasStructDefStoreV1.java:448)
at org.apache.atlas.repository.store.graph.v1.AtlasClassificationDefStoreV1.updateVertexPreUpdate(AtlasClassificationDefStoreV1.java:365)
at org.apache.atlas.repository.store.graph.v1.AtlasClassificationDefStoreV1.updateByName(AtlasClassificationDefStoreV1.java:215)
at org.apache.atlas.repository.store.graph.v1.AtlasClassificationDefStoreV1.update(AtlasClassificationDefStoreV1.java:184)
at org.apache.atlas.repository.store.graph.AtlasTypeDefGraphStore.updateGraphStore(AtlasTypeDefGraphStore.java:793)
at org.apache.atlas.repository.store.graph.AtlasTypeDefGraphStore.updateTypesDef(AtlasTypeDefGraphStore.java:372)
at org.apache.atlas.repository.store.graph.AtlasTypeDefGraphStore$$FastClassBySpringCGLIB$$5226c80b.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.apache.atlas.GraphTransactionInterceptor.invoke(GraphTransactionInterceptor.java:60)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
at org.apache.atlas.repository.store.graph.v1.AtlasTypeDefGraphStoreV1$$EnhancerBySpringCGLIB$$f4b1ae3b.updateTypesDef(<generated>)
at org.apache.atlas.web.rest.TypesREST.updateAtlasTypeDefs(TypesREST.java:326)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
...
{noformat}
I made some tests and I found, that if TypeRegistry update is called both before and after the GraphStorage update, than it will fix the issue. Of course that would be a silly way to fix this problem. Must go deeper...
{noformat}
AtlasTransientTypeRegistry ttr = lockTypeRegistryAndReleasePostCommit();
// Translate any NOT FOUND errors to BAD REQUEST
try {
ttr.updateTypes(typesDef);
} catch (AtlasBaseException e) {
if (AtlasErrorCode.TYPE_NAME_NOT_FOUND == e.getAtlasErrorCode()) {
throw new AtlasBaseException(AtlasErrorCode.BAD_REQUEST, e.getMessage());
} else {
throw e;
}
}
AtlasTypesDef ret = updateGraphStore(typesDef, ttr);
ttr.updateTypes(ret);
{noformat}
I also noticed, that the addToGraphStore method (used during creation of a new type) calls an updateGuid method on the TypeRegistry:
{noformat}
ttr.updateGuid(createdDef.getName(), createdDef.getGuid());
{noformat}
Calling updateGuid in the updateGraphStore method (or in the updateTypesDef method) would possibly fix the Guid problem. I am looking into a similar solution for the createTime, updateTime, etc. fields.
Conclusion:
I am still looking into this to find an elegant solution, any comments or hints are welcome, thx
was (Author: bpgergo):
[~madhan.neethiraj],
I understand the problem may be in the updateTypesDef method:
(repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasTypeDefGraphStore.java)
{noformat}
@Override
@GraphTransaction
public AtlasTypesDef updateTypesDef(AtlasTypesDef typesDef) throws AtlasBaseException {
if (LOG.isDebugEnabled()) {
LOG.debug("==> AtlasTypeDefGraphStore.updateTypesDef(enums={}, structs={}, classfications={}, entities={})",
CollectionUtils.size(typesDef.getEnumDefs()),
CollectionUtils.size(typesDef.getStructDefs()),
CollectionUtils.size(typesDef.getClassificationDefs()),
CollectionUtils.size(typesDef.getEntityDefs()));
}
AtlasTransientTypeRegistry ttr = lockTypeRegistryAndReleasePostCommit();
// Translate any NOT FOUND errors to BAD REQUEST
try {
ttr.updateTypes(typesDef);
} catch (AtlasBaseException e) {
if (AtlasErrorCode.TYPE_NAME_NOT_FOUND == e.getAtlasErrorCode()) {
throw new AtlasBaseException(AtlasErrorCode.BAD_REQUEST, e.getMessage());
} else {
throw e;
}
}
AtlasTypesDef ret = updateGraphStore(typesDef, ttr);
if (LOG.isDebugEnabled()) {
LOG.debug("<== AtlasTypeDefGraphStore.updateTypesDef(enums={}, structs={}, classfications={}, entities={})",
CollectionUtils.size(typesDef.getEnumDefs()),
CollectionUtils.size(typesDef.getStructDefs()),
CollectionUtils.size(typesDef.getClassificationDefs()),
CollectionUtils.size(typesDef.getEntityDefs()));
}
return ret;
{noformat}
This method updates the TypeRegistry first, then the GraphStorage.
The TypeRegistry is updated with the user input, which does not necessarily contain all fields.
I figured, maybe the GraphStorage should be updated first, and then the TypeRegistry:
{noformat}
AtlasTypesDef ret = updateGraphStore(typesDef, ttr);
ttr.updateTypes(ret);
{noformat}
Unfortunately, it will cause a NullPointerException:
{noformat}
2017-06-26 19:42:59,802 ERROR - [pool-1-thread-7 - /api/atlas/v2/types/typedefs - 23dc280b-d4e9-4c98-9e80-5496b77a98e3:] ~ graph rollback due to exception (GraphTransactionInterceptor:71)
java.lang.NullPointerException
at org.apache.atlas.repository.store.graph.v1.AtlasStructDefStoreV1.toJsonFromAttribute(AtlasStructDefStoreV1.java:508)
at org.apache.atlas.repository.store.graph.v1.AtlasStructDefStoreV1.updateVertexPreUpdate(AtlasStructDefStoreV1.java:448)
at org.apache.atlas.repository.store.graph.v1.AtlasClassificationDefStoreV1.updateVertexPreUpdate(AtlasClassificationDefStoreV1.java:365)
at org.apache.atlas.repository.store.graph.v1.AtlasClassificationDefStoreV1.updateByName(AtlasClassificationDefStoreV1.java:215)
at org.apache.atlas.repository.store.graph.v1.AtlasClassificationDefStoreV1.update(AtlasClassificationDefStoreV1.java:184)
at org.apache.atlas.repository.store.graph.AtlasTypeDefGraphStore.updateGraphStore(AtlasTypeDefGraphStore.java:793)
at org.apache.atlas.repository.store.graph.AtlasTypeDefGraphStore.updateTypesDef(AtlasTypeDefGraphStore.java:372)
at org.apache.atlas.repository.store.graph.AtlasTypeDefGraphStore$$FastClassBySpringCGLIB$$5226c80b.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.apache.atlas.GraphTransactionInterceptor.invoke(GraphTransactionInterceptor.java:60)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
at org.apache.atlas.repository.store.graph.v1.AtlasTypeDefGraphStoreV1$$EnhancerBySpringCGLIB$$f4b1ae3b.updateTypesDef(<generated>)
at org.apache.atlas.web.rest.TypesREST.updateAtlasTypeDefs(TypesREST.java:326)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
...
{noformat}
I made some tests and I found, that if TypeRegistry update is called both before and after the GraphStorage update, than it will fix the issue:
{noformat}
AtlasTransientTypeRegistry ttr = lockTypeRegistryAndReleasePostCommit();
// Translate any NOT FOUND errors to BAD REQUEST
try {
ttr.updateTypes(typesDef);
} catch (AtlasBaseException e) {
if (AtlasErrorCode.TYPE_NAME_NOT_FOUND == e.getAtlasErrorCode()) {
throw new AtlasBaseException(AtlasErrorCode.BAD_REQUEST, e.getMessage());
} else {
throw e;
}
}
AtlasTypesDef ret = updateGraphStore(typesDef, ttr);
ttr.updateTypes(ret);
{noformat}
I am still looking into this to find a more elegant solution, any comments or hints are welcome, thx
> Updating classification removes some properties
> -----------------------------------------------
>
> Key: ATLAS-1879
> URL: https://issues.apache.org/jira/browse/ATLAS-1879
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
> Affects Versions: 0.8-incubating
> Reporter: Laura Ngo
> Attachments: Atlas-1789.postman_collection.json
>
>
> * Created classification via POST.
> * Updated via PUT
> * Lost properties
> POST http://127.0.0.1:21000/api/atlas/v2/types/typedefs
> {code}
> {
> "classificationDefs": [{
> "name": "test_classification_11",
> "description": "",
> "createdBy" : "admin",
> "superTypes": [],
> "attributeDefs": [{
> "name" : "test_class_11",
> "typeName" : "string",
> "isOptional" : true,
> "isUnique" : true,
> "isIndexable" : true,
> "cardinality": "SINGLE",
> "valuesMinCount": 0,
> "valuesMaxCount": 1
> }]
> }],
> "entityDefs": [],
> "enumDefs": [],
> "structDefs": []
> }
> {code}
> GET http://127.0.0.1:21000/api/atlas/v2/types/classification/name/test_classification_11
> {code}
> {
> "category": "CLASSIFICATION",
> "guid": "83162fe1-4bb4-4a87-b2b8-364e751a1265",
> "createdBy": "admin",
> "createTime": 1497485890857,
> "updateTime": 1497485890857,
> "version": 1,
> "name": "test_classification_11",
> "description": "",
> "typeVersion": "1.0",
> "attributeDefs": [
> {
> "name": "test_class_11",
> "typeName": "string",
> "isOptional": true,
> "cardinality": "SINGLE",
> "valuesMinCount": 0,
> "valuesMaxCount": 1,
> "isUnique": true,
> "isIndexable": true
> }
> ],
> "superTypes": [],
> }
> {code}
> PUT http://127.0.0.1:21000/api/atlas/v2/types/typedefs
> Update attribute.
> GET http://127.0.0.1:21000/api/atlas/v2/types/classification/name/test_classification_11
> {code}
> {
> "category": "CLASSIFICATION",
> "createdBy": "admin",
> "name": "test_classification_11",
> "description": "",
> "attributeDefs": [
> {
> "name": "test_class_11",
> "typeName": "string",
> "isOptional": true,
> "cardinality": "SINGLE",
> "valuesMinCount": 0,
> "valuesMaxCount": 1,
> "isUnique": true,
> "isIndexable": false
> }
> ],
> "superTypes": [],
> }
> {code}
> Some properties are missing after PUT update of attribute "isIndexable"
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)