You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tajo.apache.org by "Keuntae Park (JIRA)" <ji...@apache.org> on 2014/03/07 07:30:46 UTC
[jira] [Updated] (TAJO-678) Too many open files error
[ https://issues.apache.org/jira/browse/TAJO-678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Keuntae Park updated TAJO-678:
------------------------------
Description:
During the fetch phase, too many open file exception occurred.
{noformat}
2014-03-07 15:15:03,346 INFO worker.Task (Task.java:<init>(195)) - Table Id: eb_1394170620737_0001_000001, url: http://skt-rf-52:49073/?qid=q_1394170620737_0001&sid=1&p=486&type=h&ta=4390_0
2014-03-07 15:15:03,346 INFO worker.Task (Task.java:<init>(195)) - Table Id: eb_1394170620737_0001_000001, url: http://skt-rf-55:44134/?qid=q_1394170620737_0001&sid=1&p=270&type=h&ta=9044_0,4744_0,51_0,7815_0,7619_0,6923_0,6735_0,3548_0,7607_0,4013_0,3234_0,2757_0
2014-03-07 15:15:03,346 INFO worker.Task (Task.java:<init>(195)) - Table Id: eb_1394170620737_0001_000001, url: http://skt-rf-83:34835/?qid=q_1394170620737_0001&sid=1&p=111&type=h&ta=7644_0,7739_0,8937_0,4305_0
2014-03-07 15:15:03,346 INFO worker.Task (Task.java:<init>(195)) - Table Id: eb_1394170620737_0001_000001, url: http://skt-rf-112:51947/?qid=q_1394170620737_0001&sid=1&p=30&type=h&ta=3224_0,6164_0,2627_0,2630_0,5929_0,9160_0,5506_0
2014-03-07 15:15:03,346 INFO worker.Task (Task.java:<init>(195)) - Table Id: eb_1394170620737_0001_000001, url: http://skt-rf-64:59899/?qid=q_1394170620737_0001&sid=1&p=12&type=h&ta=7879_0
2014-03-07 15:15:03,346 INFO worker.Task (Task.java:<init>(195)) - Table Id: eb_1394170620737_0001_000001, url: http://skt-rf-88:39979/?qid=q_1394170620737_0001&sid=1&p=373&type=h&ta=7873_0,4879_0,4387_0,7875_0
2014-03-07 15:15:03,346 INFO worker.Task (Task.java:<init>(195)) - Table Id: eb_1394170620737_0001_000001, url: http://skt-rf-55:44134/?qid=q_1394170620737_0001&sid=1&p=428&type=h&ta=2690_0
2014-03-07 15:15:03,346 INFO worker.Task (Task.java:<init>(195)) - Table Id: eb_1394170620737_0001_000001, url: http://skt-rf-82:37300/?qid=q_1394170620737_0001&sid=1&p=154&type=h&ta=1149_0,2679_0,2455_0
2014-03-07 15:15:03,346 INFO worker.Task (Task.java:<init>(197)) - * Local task dir: file:/data1/tajo/tajo-localdir/q_1394170620737_0001/output/2/0_1
2014-03-07 15:15:03,346 INFO worker.Task (Task.java:<init>(202)) - ==================================
2014-03-07 15:15:03,362 INFO worker.Task (Task.java:init(218)) - the directory is created file:/data3/tajo/tajo-localdir/q_1394170620737_0001/in/eb_1394170620737_0001_000002/0/1/eb_1394170620737_0001_000001
2014-03-07 15:15:03,365 ERROR worker.TaskRunner (TaskRunner.java:run(397)) - Failed to create a selector.
org.jboss.netty.channel.ChannelException: Failed to create a selector.
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.<init>(AbstractNioSelector.java:95)
at org.jboss.netty.channel.socket.nio.NioClientBoss.<init>(NioClientBoss.java:63)
at org.jboss.netty.channel.socket.nio.NioClientBossPool.newBoss(NioClientBossPool.java:61)
at org.jboss.netty.channel.socket.nio.NioClientBossPool.newBoss(NioClientBossPool.java:27)
at org.jboss.netty.channel.socket.nio.AbstractNioBossPool.init(AbstractNioBossPool.java:65)
at org.jboss.netty.channel.socket.nio.NioClientBossPool.<init>(NioClientBossPool.java:45)
at org.apache.tajo.rpc.RpcChannelFactory.createClientChannelFactory(RpcChannelFactory.java:69)
at org.apache.tajo.worker.Task.getFetchRunners(Task.java:626)
at org.apache.tajo.worker.Task.localize(Task.java:236)
at org.apache.tajo.worker.Task.init(Task.java:224)
at org.apache.tajo.worker.TaskRunner$1.run(TaskRunner.java:390)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Too many open files
at sun.nio.ch.IOUtil.initPipe(Native Method)
at sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:49)
at sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18)
at java.nio.channels.Selector.open(Selector.java:209)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:335)
... 12 more
2014-03-07 15:15:03,366 INFO worker.TaskRunner (TaskRunner.java:run(336)) - Request GetTask: eb_1394170620737_0001_000002,container_1394170620737_0001_01_000780
2014-03-07 15:15:03,580 INFO worker.TaskRunner (TaskRunner.java:run(374)) - Accumulated Received Task: 3
2014-03-07 15:15:03,580 INFO worker.TaskRunner (TaskRunner.java:run(383)) - Initializing: ta_1394170620737_0001_000002_000000_02
2014-03-07 15:15:03,583 INFO worker.Task (Task.java:<init>(182)) - Output File Path: hdfs://nameservice1/tmp/tajo-tajo/staging/q_1394170620737_0001/RESULT/part-02-000000
2014-03-07 15:15:03,583 INFO worker.TaskAttemptContext (TaskAttemptContext.java:setState(110)) - Query status of ta_1394170620737_0001_000002_000000_02 is changed to TA_PENDING
2014-03-07 15:15:03,583 INFO worker.Task (Task.java:<init>(187)) - ==================================
2014-03-07 15:15:03,583 INFO worker.Task (Task.java:<init>(188)) - * Subquery ta_1394170620737_0001_000002_000000_02 is initialized
2014-03-07 15:15:03,583 INFO worker.Task (Task.java:<init>(189)) - * InterQuery: false
2014-03-07 15:15:03,583 INFO worker.Task (Task.java:<init>(192)) - * Fragments (num: 1)
2014-03-07 15:15:03,587 INFO worker.Task (Task.java:<init>(193)) - * Fetches (total:47922) :
2014-03-07 15:15:03,587 INFO worker.Task (Task.java:<init>(195)) - Table Id: eb_1394170620737_0001_000001, url: http://skt-rf-75:39431/?qid=q_1394170620737_0001&sid=1&p=3&type=h&ta=2925_0,7796_0,9038_0,8593_0,6694_0,9229_0,8360_0,2903_0,574_0,4357_0,9022_0,3137_0
2014-03-07 15:15:03,587 INFO worker.Task (Task.java:<init>(195)) - Table Id: eb_1394170620737_0001_000001, url: http://skt-rf-14:33755/?qid=q_1394170620737_0001&sid=1&p=168&type=h&ta=3103_0
2014-03-07 15:15:03,587 INFO worker.Task (Task.java:<init>(195)) - Table Id: eb_1394170620737_0001_000001, url: http://skt-rf-63:49448/?qid=q_1394170620737_0001&sid=1&p=541&type=h&ta=9039_0,2625_0
2014-03-07 15:15:03,587 INFO worker.Task (Task.java:<init>(195)) - Table Id: eb_1394170620737_0001_000001, url: http://skt-rf-09:38879/?qid=q_1394170620737_0001&sid=1&p=648&type=h&ta=2476_0,50_0,9063_0,4536_0,2685_0,4168_0
2014-03-07 15:15:03,587 INFO worker.Task (Task.java:<init>(195)) - Table Id: eb_1394170620737_0001_000001, url: http://skt-rf-73:34510/?qid=q_1394170620737_0001&sid=1&p=453&type=h&ta=7168_0,3176_0,2626_0,3841_0,7926_0
{noformat}
was:
During the fetch phase, too many open file exception occurred.
{noformat}
2014-03-07 15:15:03,346 INFO worker.Task (Task.java:<init>(195)) - Table Id: eb_1394170620737_0001_000001, url: http://skt-rf-55:44134/?qid=q_1394170620737_0001&sid=1&p=428&type=h&ta=2690_0
148566 2014-03-07 15:15:03,346 INFO worker.Task (Task.java:<init>(195)) - Table Id: eb_1394170620737_0001_000001, url: http://skt-rf-82:37300/?qid=q_1394170620737_0001&sid=1&p=154&type=h&ta=1149_0,2679_0,2455_0
148567 2014-03-07 15:15:03,346 INFO worker.Task (Task.java:<init>(197)) - * Local task dir: file:/data1/tajo/tajo-localdir/q_1394170620737_0001/output/2/0_1
148568 2014-03-07 15:15:03,346 INFO worker.Task (Task.java:<init>(202)) - ==================================
148569 2014-03-07 15:15:03,362 INFO worker.Task (Task.java:init(218)) - the directory is created file:/data3/tajo/tajo-localdir/q_1394170620737_0001/in/eb_1394170620737_0001_000002/0/1/eb_1394170620737_0001_000001
148570 2014-03-07 15:15:03,365 ERROR worker.TaskRunner (TaskRunner.java:run(397)) - Failed to create a selector.
148571 org.jboss.netty.channel.ChannelException: Failed to create a selector.
148572 at org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:337)
148573 at org.jboss.netty.channel.socket.nio.AbstractNioSelector.<init>(AbstractNioSelector.java:95)
148574 at org.jboss.netty.channel.socket.nio.NioClientBoss.<init>(NioClientBoss.java:63)
148575 at org.jboss.netty.channel.socket.nio.NioClientBossPool.newBoss(NioClientBossPool.java:61)
148576 at org.jboss.netty.channel.socket.nio.NioClientBossPool.newBoss(NioClientBossPool.java:27)
148577 at org.jboss.netty.channel.socket.nio.AbstractNioBossPool.init(AbstractNioBossPool.java:65)
148578 at org.jboss.netty.channel.socket.nio.NioClientBossPool.<init>(NioClientBossPool.java:45)
148579 at org.apache.tajo.rpc.RpcChannelFactory.createClientChannelFactory(RpcChannelFactory.java:69)
148580 at org.apache.tajo.worker.Task.getFetchRunners(Task.java:626)
148581 at org.apache.tajo.worker.Task.localize(Task.java:236)
148582 at org.apache.tajo.worker.Task.init(Task.java:224)
148583 at org.apache.tajo.worker.TaskRunner$1.run(TaskRunner.java:390)
148584 at java.lang.Thread.run(Thread.java:662)
148585 Caused by: java.io.IOException: Too many open files
148586 at sun.nio.ch.IOUtil.initPipe(Native Method)
148587 at sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:49)
148588 at sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18)
148589 at java.nio.channels.Selector.open(Selector.java:209)
148590 at org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:335)
148591 ... 12 more
148592 2014-03-07 15:15:03,366 INFO worker.TaskRunner (TaskRunner.java:run(336)) - Request GetTask: eb_1394170620737_0001_000002,container_1394170620737_0001_01_000780
148593 2014-03-07 15:15:03,580 INFO worker.TaskRunner (TaskRunner.java:run(374)) - Accumulated Received Task: 3
148594 2014-03-07 15:15:03,580 INFO worker.TaskRunner (TaskRunner.java:run(383)) - Initializing: ta_1394170620737_0001_000002_000000_02
148595 2014-03-07 15:15:03,583 INFO worker.Task (Task.java:<init>(182)) - Output File Path: hdfs://nameservice1/tmp/tajo-tajo/staging/q_1394170620737_0001/RESULT/part-02-000000
148596 2014-03-07 15:15:03,583 INFO worker.TaskAttemptContext (TaskAttemptContext.java:setState(110)) - Query status of ta_1394170620737_0001_000002_000000_02 is changed to TA_PENDING
{noformat}
> Too many open files error
> -------------------------
>
> Key: TAJO-678
> URL: https://issues.apache.org/jira/browse/TAJO-678
> Project: Tajo
> Issue Type: Bug
> Reporter: Keuntae Park
>
> During the fetch phase, too many open file exception occurred.
> {noformat}
> 2014-03-07 15:15:03,346 INFO worker.Task (Task.java:<init>(195)) - Table Id: eb_1394170620737_0001_000001, url: http://skt-rf-52:49073/?qid=q_1394170620737_0001&sid=1&p=486&type=h&ta=4390_0
> 2014-03-07 15:15:03,346 INFO worker.Task (Task.java:<init>(195)) - Table Id: eb_1394170620737_0001_000001, url: http://skt-rf-55:44134/?qid=q_1394170620737_0001&sid=1&p=270&type=h&ta=9044_0,4744_0,51_0,7815_0,7619_0,6923_0,6735_0,3548_0,7607_0,4013_0,3234_0,2757_0
> 2014-03-07 15:15:03,346 INFO worker.Task (Task.java:<init>(195)) - Table Id: eb_1394170620737_0001_000001, url: http://skt-rf-83:34835/?qid=q_1394170620737_0001&sid=1&p=111&type=h&ta=7644_0,7739_0,8937_0,4305_0
> 2014-03-07 15:15:03,346 INFO worker.Task (Task.java:<init>(195)) - Table Id: eb_1394170620737_0001_000001, url: http://skt-rf-112:51947/?qid=q_1394170620737_0001&sid=1&p=30&type=h&ta=3224_0,6164_0,2627_0,2630_0,5929_0,9160_0,5506_0
> 2014-03-07 15:15:03,346 INFO worker.Task (Task.java:<init>(195)) - Table Id: eb_1394170620737_0001_000001, url: http://skt-rf-64:59899/?qid=q_1394170620737_0001&sid=1&p=12&type=h&ta=7879_0
> 2014-03-07 15:15:03,346 INFO worker.Task (Task.java:<init>(195)) - Table Id: eb_1394170620737_0001_000001, url: http://skt-rf-88:39979/?qid=q_1394170620737_0001&sid=1&p=373&type=h&ta=7873_0,4879_0,4387_0,7875_0
> 2014-03-07 15:15:03,346 INFO worker.Task (Task.java:<init>(195)) - Table Id: eb_1394170620737_0001_000001, url: http://skt-rf-55:44134/?qid=q_1394170620737_0001&sid=1&p=428&type=h&ta=2690_0
> 2014-03-07 15:15:03,346 INFO worker.Task (Task.java:<init>(195)) - Table Id: eb_1394170620737_0001_000001, url: http://skt-rf-82:37300/?qid=q_1394170620737_0001&sid=1&p=154&type=h&ta=1149_0,2679_0,2455_0
> 2014-03-07 15:15:03,346 INFO worker.Task (Task.java:<init>(197)) - * Local task dir: file:/data1/tajo/tajo-localdir/q_1394170620737_0001/output/2/0_1
> 2014-03-07 15:15:03,346 INFO worker.Task (Task.java:<init>(202)) - ==================================
> 2014-03-07 15:15:03,362 INFO worker.Task (Task.java:init(218)) - the directory is created file:/data3/tajo/tajo-localdir/q_1394170620737_0001/in/eb_1394170620737_0001_000002/0/1/eb_1394170620737_0001_000001
> 2014-03-07 15:15:03,365 ERROR worker.TaskRunner (TaskRunner.java:run(397)) - Failed to create a selector.
> org.jboss.netty.channel.ChannelException: Failed to create a selector.
> at org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:337)
> at org.jboss.netty.channel.socket.nio.AbstractNioSelector.<init>(AbstractNioSelector.java:95)
> at org.jboss.netty.channel.socket.nio.NioClientBoss.<init>(NioClientBoss.java:63)
> at org.jboss.netty.channel.socket.nio.NioClientBossPool.newBoss(NioClientBossPool.java:61)
> at org.jboss.netty.channel.socket.nio.NioClientBossPool.newBoss(NioClientBossPool.java:27)
> at org.jboss.netty.channel.socket.nio.AbstractNioBossPool.init(AbstractNioBossPool.java:65)
> at org.jboss.netty.channel.socket.nio.NioClientBossPool.<init>(NioClientBossPool.java:45)
> at org.apache.tajo.rpc.RpcChannelFactory.createClientChannelFactory(RpcChannelFactory.java:69)
> at org.apache.tajo.worker.Task.getFetchRunners(Task.java:626)
> at org.apache.tajo.worker.Task.localize(Task.java:236)
> at org.apache.tajo.worker.Task.init(Task.java:224)
> at org.apache.tajo.worker.TaskRunner$1.run(TaskRunner.java:390)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.io.IOException: Too many open files
> at sun.nio.ch.IOUtil.initPipe(Native Method)
> at sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:49)
> at sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18)
> at java.nio.channels.Selector.open(Selector.java:209)
> at org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:335)
> ... 12 more
> 2014-03-07 15:15:03,366 INFO worker.TaskRunner (TaskRunner.java:run(336)) - Request GetTask: eb_1394170620737_0001_000002,container_1394170620737_0001_01_000780
> 2014-03-07 15:15:03,580 INFO worker.TaskRunner (TaskRunner.java:run(374)) - Accumulated Received Task: 3
> 2014-03-07 15:15:03,580 INFO worker.TaskRunner (TaskRunner.java:run(383)) - Initializing: ta_1394170620737_0001_000002_000000_02
> 2014-03-07 15:15:03,583 INFO worker.Task (Task.java:<init>(182)) - Output File Path: hdfs://nameservice1/tmp/tajo-tajo/staging/q_1394170620737_0001/RESULT/part-02-000000
> 2014-03-07 15:15:03,583 INFO worker.TaskAttemptContext (TaskAttemptContext.java:setState(110)) - Query status of ta_1394170620737_0001_000002_000000_02 is changed to TA_PENDING
> 2014-03-07 15:15:03,583 INFO worker.Task (Task.java:<init>(187)) - ==================================
> 2014-03-07 15:15:03,583 INFO worker.Task (Task.java:<init>(188)) - * Subquery ta_1394170620737_0001_000002_000000_02 is initialized
> 2014-03-07 15:15:03,583 INFO worker.Task (Task.java:<init>(189)) - * InterQuery: false
> 2014-03-07 15:15:03,583 INFO worker.Task (Task.java:<init>(192)) - * Fragments (num: 1)
> 2014-03-07 15:15:03,587 INFO worker.Task (Task.java:<init>(193)) - * Fetches (total:47922) :
> 2014-03-07 15:15:03,587 INFO worker.Task (Task.java:<init>(195)) - Table Id: eb_1394170620737_0001_000001, url: http://skt-rf-75:39431/?qid=q_1394170620737_0001&sid=1&p=3&type=h&ta=2925_0,7796_0,9038_0,8593_0,6694_0,9229_0,8360_0,2903_0,574_0,4357_0,9022_0,3137_0
> 2014-03-07 15:15:03,587 INFO worker.Task (Task.java:<init>(195)) - Table Id: eb_1394170620737_0001_000001, url: http://skt-rf-14:33755/?qid=q_1394170620737_0001&sid=1&p=168&type=h&ta=3103_0
> 2014-03-07 15:15:03,587 INFO worker.Task (Task.java:<init>(195)) - Table Id: eb_1394170620737_0001_000001, url: http://skt-rf-63:49448/?qid=q_1394170620737_0001&sid=1&p=541&type=h&ta=9039_0,2625_0
> 2014-03-07 15:15:03,587 INFO worker.Task (Task.java:<init>(195)) - Table Id: eb_1394170620737_0001_000001, url: http://skt-rf-09:38879/?qid=q_1394170620737_0001&sid=1&p=648&type=h&ta=2476_0,50_0,9063_0,4536_0,2685_0,4168_0
> 2014-03-07 15:15:03,587 INFO worker.Task (Task.java:<init>(195)) - Table Id: eb_1394170620737_0001_000001, url: http://skt-rf-73:34510/?qid=q_1394170620737_0001&sid=1&p=453&type=h&ta=7168_0,3176_0,2626_0,3841_0,7926_0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)