You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by "Péter Gergő Barna (JIRA)" <ji...@apache.org> on 2017/08/01 11:26:00 UTC
[jira] [Commented] (ATLAS-1903) When Solr is not available, REST
API type update returns HTTP 200 OK although the update was not successful
[ https://issues.apache.org/jira/browse/ATLAS-1903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108732#comment-16108732 ]
Péter Gergő Barna commented on ATLAS-1903:
------------------------------------------
I tried to debug this issue and it looks to me like the TitanException that can be seen in the log is swallowed, I mean it is simply logged by Titan and it is not re-raised. Atlas will never know about it happened. These parts from the Titan code show the problem:
{noformat}
//com.thinkaurelius.titan.diskstorage.BackendTransaction
public Map<String,Throwable> commitIndexes() {
Map<String,Throwable> exceptions = new HashMap<String, Throwable>(indexTx.size());
for (Map.Entry<String,IndexTransaction> txentry : indexTx.entrySet()) {
try {
txentry.getValue().commit();
} catch (Throwable e) {
exceptions.put(txentry.getKey(),e);
}
}
return exceptions;
}
//com.thinkaurelius.titan.graphdb.database.StandardTitanGraph
public void commit(final Collection<InternalRelation> addedRelations,
final Collection<InternalRelation> deletedRelations, final StandardTitanTx tx) {
...
//2. Commit indexes - [FAILURE] all exceptions are collected and logged but nothing is aborted
indexFailures = mutator.commitIndexes();
if (!indexFailures.isEmpty()) {
status = LogTxStatus.SECONDARY_FAILURE;
for (Map.Entry<String,Throwable> entry : indexFailures.entrySet()) {
log.error("Error while commiting index mutations for transaction ["+transactionId+"] on index: " +entry.getKey(),entry.getValue());
}
}
{noformat}
I am looking into if it is possible to detect index commit failure on the Atlas side, either from the state of the transaction or from the txLog, I am trying something along these lines:
{noformat}
//public class Titan0Graph implements AtlasGraph<Titan0Vertex, Titan0Edge> {
@Override
public void commit() {
TitanTransaction tx = null;
StandardTitanTx stdTx = null;
BackendTransaction bTx = null;
TitanGraph graph = getGraph();
if (graph instanceof StandardTitanGraph) {
StandardTitanGraph stdGraph = (StandardTitanGraph)graph;
tx = stdGraph.getCurrentThreadTx();
LOG.debug("Titan-transaction before commit:" + ReflectionToStringBuilder.toString(tx));
//KCVSLog txLog = stdGraph.getBackend().getSystemTxLog();
//Timestamp startTime = new Timestamp(System.currentTimeMillis(),TimeUnit.MILLISECONDS);
//ReadMarker start = ReadMarker.fromTime(startTime.sinceEpoch(startTime.getNativeUnit()),startTime.getNativeUnit());
//com.thinkaurelius.titan.graphdb.log.StandardTransactionLogProcessor.TxLogMessageReader tlmr;
//txLog.registerReader(start,new TxLogMessageReader());
if (tx instanceof StandardTitanTx) {
stdTx = (StandardTitanTx)tx;
LOG.debug("std Titan-transaction before commit:" + ReflectionToStringBuilder.toString(stdTx));
bTx = stdTx.getTxHandle();
LOG.debug("backend Titan-transaction before commit:" + ReflectionToStringBuilder.toString(bTx));
}
}
getGraph().commit();
LOG.debug("Titan-transaction AFTER commit:" + ReflectionToStringBuilder.toString(tx));
LOG.debug("std Titan-transaction AFTER commit:" + ReflectionToStringBuilder.toString(stdTx));
LOG.debug("backend Titan-transaction AFTER commit:" + ReflectionToStringBuilder.toString(bTx));
}
{noformat}
Any help/hint is welcomed, thanks in advance.
> When Solr is not available, REST API type update returns HTTP 200 OK although the update was not successful
> -----------------------------------------------------------------------------------------------------------
>
> Key: ATLAS-1903
> URL: https://issues.apache.org/jira/browse/ATLAS-1903
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
> Affects Versions: 0.8-incubating
> Reporter: Péter Gergő Barna
> Assignee: Péter Gergő Barna
>
> After Solr is stopped, REST API type update is called:
> PUT http://127.0.0.1:21000/api/atlas/v2/types/typedefs
> It takes a long time to respond (3 mins). Finally it returns HTTP 200 OK, and a message body that suggests that the update was successful.
> However, the update was not successful, and from the logs it can be seen that the problem is Solr being unavailable
> I think, instead of HTTP 200 OK, the response should be HTTP 500 or 503 and a meaningful error message indicating that the problem is Solr being unavailable.
> Log excerpt:
> {noformat}
> 2017-06-28 15:36:52,747 ERROR - [pool-1-thread-8 - /api/atlas/v2/types/typedefs - c6d49838-c0d2-465a-93ce-38e63511fca5:] ~ Error while commiting index mutations for transaction [2] on index: search (StandardTitanGraph:688)
> com.thinkaurelius.titan.core.TitanException: Could not execute operation due to backend exception
> at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:44)
> at com.thinkaurelius.titan.diskstorage.indexing.IndexTransaction.flushInternal(IndexTransaction.java:119)
> at com.thinkaurelius.titan.diskstorage.indexing.IndexTransaction.commit(IndexTransaction.java:100)
> at com.thinkaurelius.titan.diskstorage.BackendTransaction.commitIndexes(BackendTransaction.java:124)
> at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.commit(StandardTitanGraph.java:684)
> at com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx.commit(StandardTitanTx.java:1337)
> at com.thinkaurelius.titan.graphdb.blueprints.TitanBlueprintsGraph.commit(TitanBlueprintsGraph.java:60)
> at org.apache.atlas.repository.graphdb.titan0.Titan0Graph.commit(Titan0Graph.java:154)
> at org.apache.atlas.GraphTransactionInterceptor.invoke(GraphTransactionInterceptor.java:61)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
> at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
> at org.apache.atlas.repository.store.graph.v1.AtlasTypeDefGraphStoreV1$$EnhancerBySpringCGLIB$$f4b1ae3b.updateTypesDef(<generated>)
> at org.apache.atlas.web.rest.TypesREST.updateAtlasTypeDefs(TypesREST.java:326)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
> at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
> at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
> at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
> at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
> at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
> at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
> at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
> at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
> at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:808)
> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
> at org.apache.atlas.web.filters.AuditFilter.doFilter(AuditFilter.java:76)
> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
> at org.apache.atlas.web.filters.AtlasAuthorizationFilter.doFilter(AtlasAuthorizationFilter.java:157)
> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
> at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114)
> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)
> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at org.apache.atlas.web.filters.AtlasCSRFPreventionFilter$ServletFilterHttpInteraction.proceed(AtlasCSRFPreventionFilter.java:235)
> at org.apache.atlas.web.filters.AtlasCSRFPreventionFilter.handleHttpInteraction(AtlasCSRFPreventionFilter.java:177)
> at org.apache.atlas.web.filters.AtlasCSRFPreventionFilter.doFilter(AtlasCSRFPreventionFilter.java:190)
> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at org.apache.atlas.web.filters.AtlasAuthenticationFilter.doFilter(AtlasAuthenticationFilter.java:325)
> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170)
> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at org.apache.atlas.web.filters.AtlasKnoxSSOAuthenticationFilter.doFilter(AtlasKnoxSSOAuthenticationFilter.java:132)
> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at org.apache.atlas.web.filters.StaleTransactionCleanupFilter.doFilter(StaleTransactionCleanupFilter.java:55)
> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:215)
> at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:200)
> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116)
> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)
> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)
> at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
> at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)
> at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
> at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262)
> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> at org.eclipse.jetty.server.Server.handle(Server.java:499)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
> at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: com.thinkaurelius.titan.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after Duration[100 s]
> at com.thinkaurelius.titan.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:86)
> at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:42)
> ... 96 more
> Caused by: com.thinkaurelius.titan.diskstorage.TemporaryBackendException: Unable to complete query on Solr.
> at com.thinkaurelius.titan.diskstorage.solr.Solr5Index.storageException(Solr5Index.java:892)
> at com.thinkaurelius.titan.diskstorage.solr.Solr5Index.mutate(Solr5Index.java:354)
> at com.thinkaurelius.titan.diskstorage.indexing.IndexTransaction$1.call(IndexTransaction.java:122)
> at com.thinkaurelius.titan.diskstorage.indexing.IndexTransaction$1.call(IndexTransaction.java:119)
> at com.thinkaurelius.titan.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:56)
> ... 97 more
> Caused by: org.apache.solr.common.SolrException: Could not load collection from ZK: vertex_index
> at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:992)
> at org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:525)
> at org.apache.solr.common.cloud.ClusterState.getCollectionOrNull(ClusterState.java:189)
> at org.apache.solr.common.cloud.ClusterState.hasCollection(ClusterState.java:119)
> at org.apache.solr.client.solrj.impl.CloudSolrClient.getCollectionNames(CloudSolrClient.java:1111)
> at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:833)
> at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:806)
> at com.thinkaurelius.titan.diskstorage.solr.Solr5Index.commitDocumentChanges(Solr5Index.java:435)
> at com.thinkaurelius.titan.diskstorage.solr.Solr5Index.mutate(Solr5Index.java:351)
> ... 100 more
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /collections/vertex_index/state.json
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:345)
> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:342)
> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:342)
> at org.apache.solr.common.cloud.ZkStateReader.fetchCollectionState(ZkStateReader.java:1004)
> at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:990)
> ... 108 more
> 2017-06-28 15:36:52,748 DEBUG - [pool-1-thread-8 - /api/atlas/v2/types/typedefs - c6d49838-c0d2-465a-93ce-38e63511fca5:] ~ graph commit (GraphTransactionInterceptor:65)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)