[jira] [Created] (ASTERIXDB-1926) Buffer Cache error after 7 hours of Feed Ingestion

["Steven Jacobs (JIRA)" <ji...@apache.org>]

Steven Jacobs created ASTERIXDB-1926:
----------------------------------------

             Summary: Buffer Cache error after 7 hours of Feed Ingestion
                 Key: ASTERIXDB-1926
                 URL: https://issues.apache.org/jira/browse/ASTERIXDB-1926
             Project: Apache AsterixDB
          Issue Type: Bug
            Reporter: Steven Jacobs


I have a single node, single partition cluster on one machine, and a script sending artificial tweets as fast as possible to a socket feed from a second machine. On the Asterix machine, I have a script performing a single record lookup every second. In both scripts, the key is randomly chosen, so in the read case the key may not exist, and in the write case, duplicate keys are being inserted. I'm using an upsert feed with no secondary keys or filters created. After 7 hours, all queries start to fail with the following (this happens every time for me):

May 26, 2017 5:17:38 PM org.apache.asterix.api.http.server.RestApiServlet doHandle
SEVERE: Unable to find free page in buffer cache after 1000 cycles (buffer cache undersized?)
org.apache.hyracks.algebricks.common.exceptions.AlgebricksException: Unable to find free page in buffer cache after 1000 cycles (buffer cache undersized?)
        at org.apache.asterix.metadata.declared.MetadataManagerUtil.getDatasetIndexes(MetadataManagerUtil.java:159)
        at org.apache.asterix.metadata.declared.MetadataProvider.getDatasetIndexes(MetadataProvider.java:364)
        at org.apache.asterix.optimizer.rules.PushFieldAccessRule.isAccessToIndexedField(PushFieldAccessRule.java:148)
        at org.apache.asterix.optimizer.rules.PushFieldAccessRule.propagateFieldAccessRec(PushFieldAccessRule.java:185)
        at org.apache.asterix.optimizer.rules.PushFieldAccessRule.rewritePost(PushFieldAccessRule.java:96)
        at org.apache.hyracks.algebricks.core.rewriter.base.AbstractRuleController.rewriteOperatorRef(AbstractRuleController.java:126)
        at org.apache.hyracks.algebricks.core.rewriter.base.AbstractRuleController.rewriteOperatorRef(AbstractRuleController.java:100)
        at org.apache.hyracks.algebricks.core.rewriter.base.AbstractRuleController.rewriteOperatorRef(AbstractRuleController.java:100)
        at org.apache.hyracks.algebricks.compiler.rewriter.rulecontrollers.SequentialFixpointRuleController.rewriteWithRuleCollection(SequentialFixpointRuleController.java:53)
        at org.apache.hyracks.algebricks.core.rewriter.base.HeuristicOptimizer.runOptimizationSets(HeuristicOptimizer.java:102)
        at org.apache.hyracks.algebricks.core.rewriter.base.HeuristicOptimizer.optimize(HeuristicOptimizer.java:82)
        at org.apache.hyracks.algebricks.compiler.api.HeuristicCompilerFactoryBuilder$1$1.optimize(HeuristicCompilerFactoryBuilder.java:90)
        at org.apache.asterix.api.common.APIFramework.compileQuery(APIFramework.java:263)
        at org.apache.asterix.app.translator.QueryTranslator.rewriteCompileQuery(QueryTranslator.java:1818)
        at org.apache.asterix.app.translator.QueryTranslator.lambda$handleQuery$2(QueryTranslator.java:2312)
        at org.apache.asterix.app.translator.QueryTranslator.createAndRunJob(QueryTranslator.java:2402)
        at org.apache.asterix.app.translator.QueryTranslator.deliverResult(QueryTranslator.java:2348)
        at org.apache.asterix.app.translator.QueryTranslator.handleQuery(QueryTranslator.java:2324)
        at org.apache.asterix.app.translator.QueryTranslator.compileAndExecute(QueryTranslator.java:362)
        at org.apache.asterix.app.translator.QueryTranslator.compileAndExecute(QueryTranslator.java:246)
        at org.apache.asterix.api.http.server.RestApiServlet.doHandle(RestApiServlet.java:206)
        at org.apache.asterix.api.http.server.RestApiServlet.getOrPost(RestApiServlet.java:176)
        at org.apache.asterix.api.http.server.RestApiServlet.get(RestApiServlet.java:160)
        at org.apache.hyracks.http.server.AbstractServlet.handle(AbstractServlet.java:74)
        at org.apache.hyracks.http.server.HttpRequestHandler.handle(HttpRequestHandler.java:70)
        at org.apache.hyracks.http.server.HttpRequestHandler.call(HttpRequestHandler.java:55)
        at org.apache.hyracks.http.server.HttpRequestHandler.call(HttpRequestHandler.java:36)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

Here are the DDL's and the query:

drop dataverse experiments if exists;
create dataverse experiments;
use dataverse experiments;

create type TweetMessageType as closed {
    id: int64,
    message_text: string,
    created_at: string,
    country:string
}

create feed TweetFeed using socket_adapter
(
    ("sockets"="127.0.0.1:10001"),
    ("address-type"="IP"),
    ("type-name"="TweetMessageType"),
    ("upsert-feed"="true"),
    ("format"="adm")
);
connect feed TweetFeed to dataset Tweets1;
start feed TweetFeed;


Query run every second:
use dataverse experiments; 
for $i in dataset Tweets1 
where  $i.id = <random_int> 
return $i.country;



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)