You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@asterixdb.apache.org by Siripen Pongpaichet <sp...@gmail.com> on 2015/12/24 03:47:33 UTC
Asterix Cluster Global recovery is not yet complete and The system is
in ACTIVE state
Hi all,
I got this following error when I tried to call the REST api. Could anybody
please help? I included part of the log files at the end. Thank you very
much.
{
• "summary": " Asterix Cluster Global recovery is not yet complete
and The system is in ACTIVE state\n caused by:
edu.uci.ics.asterix.translator.AbstractAqlTranslator.validateOperation(AbstractAqlTranslator.java:97)",
• "error-code":
• [
◦ 99,
◦ " Asterix Cluster Global recovery is not yet complete and The
system is in ACTIVE state [AsterixException]"
• ],
• "stacktrace":
"edu.uci.ics.asterix.common.exceptions.AsterixException: Asterix Cluster
Global recovery is not yet complete and The system is in ACTIVE state\n\tat
edu.uci.ics.asterix.translator.AbstractAqlTranslator.validateOperation(AbstractAqlTranslator.java:97)\n\tat
edu.uci.ics.asterix.aql.translator.AqlTranslator.compileAndExecute(AqlTranslator.java:257)\n\tat
edu.uci.ics.asterix.api.http.servlet.RESTAPIServlet.handleRequest(RESTAPIServlet.java:156)\n\tat
edu.uci.ics.asterix.api.http.servlet.QueryAPIServlet.handleRequest(QueryAPIServlet.java:25)\n\tat
edu.uci.ics.asterix.api.http.servlet.RESTAPIServlet.doGet(RESTAPIServlet.java:129)\n\tat
edu.uci.ics.asterix.api.http.servlet.QueryAPIServlet.doGet(QueryAPIServlet.java:25)\n\tat
javax.servlet.http.HttpServlet.service(HttpServlet.java:734)\n\tat
javax.servlet.http.HttpServlet.service(HttpServlet.java:847)\n\tat
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:546)\n\tat
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:483)\n\tat
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)\n\tat
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:970)\n\tat
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:411)\n\tat
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:192)\n\tat
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:904)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:110)\n\tat
org.eclipse.jetty.server.Server.handle(Server.java:347)\n\tat
org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:439)\n\tat
org.eclipse.jetty.server.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:907)\n\tat
org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:562)\n\tat
org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:214)\n\tat
org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:43)\n\tat
org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:545)\n\tat
org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:43)\n\tat
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529)\n\tat
java.lang.Thread.run(Thread.java:745)\n"
}
I looked at the nodecontroller log. I found a lot of line with.
OpenJDK 64-Bit Server VM warning: Exception java.lang.OutOfMemoryError
occurred dispatching signal SIGTERM to handler- the VM may need to be
forcibly terminated
Here is some log from cc.log
Dec 23, 2015 6:12:20 PM
edu.uci.ics.hyracks.control.common.dataset.ResultStateSweeper sweep
INFO: Result state cleanup instance successfully completed.
Dec 23, 2015 6:12:23 PM edu.uci.ics.asterix.feeds.FeedMessageReceiver
processMessage
INFO: Received message
{"cpu-load":0.86,"nodeId":"my_asterix_node1","heap_usage":0.9068803669364529,"n_runtimes":0,"message-type":"NODE_REPORT"}
Dec 23, 2015 6:12:37 PM edu.uci.ics.asterix.feeds.FeedMessageReceiver
processMessage
INFO: Received message
{"cpu-load":1.05,"nodeId":"my_asterix_node1","heap_usage":0.9068707699501982,"n_runtimes":0,"message-type":"NODE_REPORT"}
Dec 23, 2015 6:13:13 PM edu.uci.ics.asterix.feeds.FeedMessageReceiver
processMessage
INFO: Received message
{"cpu-load":0.82,"nodeId":"my_asterix_node1","heap_usage":0.906874105845857,"n_runtimes":0,"message-type":"NODE_REPORT"}
Dec 23, 2015 6:13:20 PM
edu.uci.ics.hyracks.control.common.dataset.ResultStateSweeper sweep
INFO: Result state cleanup instance successfully completed.
Dec 23, 2015 6:13:47 PM edu.uci.ics.asterix.feeds.FeedMessageReceiver
processMessage
INFO: Received message
{"cpu-load":1.41,"nodeId":"my_asterix_node1","heap_usage":0.9068897459998756,"n_runtimes":0,"message-type":"NODE_REPORT"}
Dec 23, 2015 6:14:07 PM edu.uci.ics.asterix.feeds.FeedMessageReceiver
processMessage
INFO: Received message
{"cpu-load":1.07,"nodeId":"my_asterix_node1","heap_usage":0.9068736071378377,"n_runtimes":0,"message-type":"NODE_REPORT"}
Best,
Siripen
Re: Asterix Cluster Global recovery is not yet complete and The
system is in ACTIVE state
Posted by abdullah alamoudi <ba...@gmail.com>.
Siripen,
This situation can happen in the following scenario:
1. The cluster is active and everything works fine.
2. One or more node controllers get disconnected. This sends a signal to
the GlobalRecoveryManager.
3. GlobalRecoveryManager sets a flag that when the cluster becomes active,
it should run global recovery to clean up hanging global transactions.
4. Nodes which left the cluster join again.
5. The cluster becomes active. This should send a signal to
GlobalRecoveryManager to start global recovery and allow queries to be
executed. Someone disabled this and so GlobalRecoveryManager stops all
queries.
I am going to investigate this and fix it very soon. For the time being,
the only way to handle such situations is to restart the instance.
Cheers,
Abdullah.
Amoudi, Abdullah.
On Thu, Dec 24, 2015 at 5:47 AM, Siripen Pongpaichet <sp...@gmail.com>
wrote:
> Hi all,
>
> I got this following error when I tried to call the REST api. Could anybody
> please help? I included part of the log files at the end. Thank you very
> much.
>
>
> {
> • "summary": " Asterix Cluster Global recovery is not yet complete
> and The system is in ACTIVE state\n caused by:
>
> edu.uci.ics.asterix.translator.AbstractAqlTranslator.validateOperation(AbstractAqlTranslator.java:97)",
> • "error-code":
> • [
> ◦ 99,
> ◦ " Asterix Cluster Global recovery is not yet complete and The
> system is in ACTIVE state [AsterixException]"
> • ],
> • "stacktrace":
> "edu.uci.ics.asterix.common.exceptions.AsterixException: Asterix Cluster
> Global recovery is not yet complete and The system is in ACTIVE state\n\tat
>
> edu.uci.ics.asterix.translator.AbstractAqlTranslator.validateOperation(AbstractAqlTranslator.java:97)\n\tat
>
> edu.uci.ics.asterix.aql.translator.AqlTranslator.compileAndExecute(AqlTranslator.java:257)\n\tat
>
> edu.uci.ics.asterix.api.http.servlet.RESTAPIServlet.handleRequest(RESTAPIServlet.java:156)\n\tat
>
> edu.uci.ics.asterix.api.http.servlet.QueryAPIServlet.handleRequest(QueryAPIServlet.java:25)\n\tat
>
> edu.uci.ics.asterix.api.http.servlet.RESTAPIServlet.doGet(RESTAPIServlet.java:129)\n\tat
>
> edu.uci.ics.asterix.api.http.servlet.QueryAPIServlet.doGet(QueryAPIServlet.java:25)\n\tat
> javax.servlet.http.HttpServlet.service(HttpServlet.java:734)\n\tat
> javax.servlet.http.HttpServlet.service(HttpServlet.java:847)\n\tat
>
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:546)\n\tat
>
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:483)\n\tat
>
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)\n\tat
>
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:970)\n\tat
>
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:411)\n\tat
>
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:192)\n\tat
>
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:904)\n\tat
>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)\n\tat
>
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:110)\n\tat
> org.eclipse.jetty.server.Server.handle(Server.java:347)\n\tat
>
> org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:439)\n\tat
>
> org.eclipse.jetty.server.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:907)\n\tat
> org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:562)\n\tat
> org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:214)\n\tat
>
> org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:43)\n\tat
>
> org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:545)\n\tat
>
> org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:43)\n\tat
>
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529)\n\tat
> java.lang.Thread.run(Thread.java:745)\n"
> }
>
>
> I looked at the nodecontroller log. I found a lot of line with.
>
> OpenJDK 64-Bit Server VM warning: Exception java.lang.OutOfMemoryError
> occurred dispatching signal SIGTERM to handler- the VM may need to be
> forcibly terminated
>
>
> Here is some log from cc.log
>
> Dec 23, 2015 6:12:20 PM
> edu.uci.ics.hyracks.control.common.dataset.ResultStateSweeper sweep
> INFO: Result state cleanup instance successfully completed.
> Dec 23, 2015 6:12:23 PM edu.uci.ics.asterix.feeds.FeedMessageReceiver
> processMessage
> INFO: Received message
>
> {"cpu-load":0.86,"nodeId":"my_asterix_node1","heap_usage":0.9068803669364529,"n_runtimes":0,"message-type":"NODE_REPORT"}
> Dec 23, 2015 6:12:37 PM edu.uci.ics.asterix.feeds.FeedMessageReceiver
> processMessage
> INFO: Received message
>
> {"cpu-load":1.05,"nodeId":"my_asterix_node1","heap_usage":0.9068707699501982,"n_runtimes":0,"message-type":"NODE_REPORT"}
> Dec 23, 2015 6:13:13 PM edu.uci.ics.asterix.feeds.FeedMessageReceiver
> processMessage
> INFO: Received message
>
> {"cpu-load":0.82,"nodeId":"my_asterix_node1","heap_usage":0.906874105845857,"n_runtimes":0,"message-type":"NODE_REPORT"}
> Dec 23, 2015 6:13:20 PM
> edu.uci.ics.hyracks.control.common.dataset.ResultStateSweeper sweep
> INFO: Result state cleanup instance successfully completed.
> Dec 23, 2015 6:13:47 PM edu.uci.ics.asterix.feeds.FeedMessageReceiver
> processMessage
> INFO: Received message
>
> {"cpu-load":1.41,"nodeId":"my_asterix_node1","heap_usage":0.9068897459998756,"n_runtimes":0,"message-type":"NODE_REPORT"}
> Dec 23, 2015 6:14:07 PM edu.uci.ics.asterix.feeds.FeedMessageReceiver
> processMessage
> INFO: Received message
>
> {"cpu-load":1.07,"nodeId":"my_asterix_node1","heap_usage":0.9068736071378377,"n_runtimes":0,"message-type":"NODE_REPORT"}
>
>
> Best,
> Siripen
>