You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@asterixdb.apache.org by "Pouria (JIRA)" <ji...@apache.org> on 2015/10/22 01:45:27 UTC
[jira] [Created] (ASTERIXDB-1146) Cleaning up left-overs once the
query is done - External datasets
Pouria created ASTERIXDB-1146:
---------------------------------
Summary: Cleaning up left-overs once the query is done - External datasets
Key: ASTERIXDB-1146
URL: https://issues.apache.org/jira/browse/ASTERIXDB-1146
Project: Apache AsterixDB
Issue Type: Bug
Components: AsterixDB
Reporter: Pouria
Assignee: Abdullah Alamoudi
Priority: Minor
Running queries which use external datasets for a long time, without restarting the asterixdb instance in between, causes the number of open files grow, and it can eventually break the system.
Using 'lsof' command , it seems that there are left-over 'ESTABLISHED' TCP connections between NC and Datanode:
java 7576 pouria 292u IPv4 393409691 0t0 TCP asterix-10.ics.uci.edu:46965->asterix-10.ics.uci.edu:50010 (CLOSE_WAIT)
java 7576 pouria 293u IPv4 393412907 0t0 TCP asterix-10.ics.uci.edu:47005->asterix-10.ics.uci.edu:50010 (ESTABLISHED)
…
java 32205 pouria 576u IPv4 393415126 0t0 TCP asterix-10.ics.uci.edu:50010->asterix-10.ics.uci.edu:47056 (ESTABLISHED)
java 32205 pouria 586u IPv4 393414645 0t0 TCP asterix-10.ics.uci.edu:50010->asterix-10.ics.uci.edu:47045 (ESTABLISHED)
…
Here is the error upon system breakage from CC logs:
org.apache.hyracks.api.exceptions.HyracksDataException: org.apache.hyracks.api.exceptions.HyracksDataException: org.apache.hyracks.api.exceptions.HyracksDataException: java.io.IOException: Too many open files
at org.apache.hyracks.control.common.utils.ExceptionUtils.setNodeIds(ExceptionUtils.java:45)
at org.apache.hyracks.control.nc.Task.run(Task.java:312)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hyracks.api.exceptions.HyracksDataException: org.apache.hyracks.api.exceptions.HyracksDataException: java.io.IOException: Too many open files
at org.apache.hyracks.control.nc.Task.pushFrames(Task.java:358)
at org.apache.hyracks.control.nc.Task.run(Task.java:290)
... 3 more
Caused by: org.apache.hyracks.api.exceptions.HyracksDataException: java.io.IOException: Too many open files
at org.apache.hyracks.control.nc.io.IOManager.createWorkspaceFile(IOManager.java:171)
at org.apache.hyracks.control.nc.io.WorkspaceFileFactory.createManagedWorkspaceFile(WorkspaceFileFactory.java:39)
at org.apache.hyracks.control.nc.Joblet.createManagedWorkspaceFile(Joblet.java:262)
at org.apache.hyracks.dataflow.std.join.OptimizedHybridHashJoin.buildWrite(OptimizedHybridHashJoin.java:332)
at org.apache.hyracks.dataflow.std.join.OptimizedHybridHashJoin.spillPartition(OptimizedHybridHashJoin.java:311)
at org.apache.hyracks.dataflow.std.join.OptimizedHybridHashJoin.processTuple(OptimizedHybridHashJoin.java:237)
at org.apache.hyracks.dataflow.std.join.OptimizedHybridHashJoin.build(OptimizedHybridHashJoin.java:215)
at org.apache.hyracks.dataflow.std.join.OptimizedHybridHashJoinOperatorDescriptor$PartitionAndBuildActivityNode$1.nextFrame(OptimizedHybridHashJoinOperatorDescriptor.java:313)
at org.apache.hyracks.control.nc.Task.pushFrames(Task.java:342)
... 4 more
Caused by: java.io.IOException: Too many open files
at java.io.UnixFileSystem.createFileExclusively(Native Method)
at java.io.File.createNewFile(File.java:1006)
at java.io.File.createTempFile(File.java:1989)
at org.apache.hyracks.control.nc.io.IOManager.createWorkspaceFile(IOManager.java:169)
... 12 more
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)