You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@asterixdb.apache.org by Taewoo Kim <wa...@gmail.com> on 2017/01/30 00:00:36 UTC
An exception: Input/output error?
Hi all,
I'm trying to load json files into a dataset. I have nine nodes so that I
divide a 250GB file into nine files. After one hour, suddenly, I see the
following exception on one node. Has anyone experienced this symptom?
org.apache.hyracks.api.exceptions.HyracksDataException:
org.apache.hyracks.api.exceptions.HyracksDataException: Input/output error
at
org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.runInParallel(SuperActivityOperatorNodePushable.java:228)
at
org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.initialize(SuperActivityOperatorNodePushable.java:84)
at org.apache.hyracks.control.nc.Task.run(Task.java:273)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.ExecutionException:
org.apache.hyracks.api.exceptions.HyracksDataException: Input/output error
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at
org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.runInParallel(SuperActivityOperatorNodePushable.java:222)
... 5 more
Caused by: org.apache.hyracks.api.exceptions.HyracksDataException:
Input/output error
at
org.apache.asterix.external.operators.ExternalDataScanOperatorDescriptor$1.initialize(ExternalDataScanOperatorDescriptor.java:65)
at
org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.lambda$initialize$0(SuperActivityOperatorNodePushable.java:86)
at
org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.lambda$runInParallel$2(SuperActivityOperatorNodePushable.java:216)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
Caused by: org.apache.hyracks.api.exceptions.HyracksDataException:
Input/output error
at
org.apache.asterix.external.dataflow.RecordDataFlowController.start(RecordDataFlowController.java:63)
at
org.apache.asterix.external.dataset.adapter.GenericAdapter.start(GenericAdapter.java:37)
at
org.apache.asterix.external.operators.ExternalDataScanOperatorDescriptor$1.initialize(ExternalDataScanOperatorDescriptor.java:62)
... 6 more
Caused by: java.io.IOException: Input/output error
at java.io.FileInputStream.readBytes(Native Method)
at java.io.FileInputStream.read(FileInputStream.java:255)
at
org.apache.asterix.external.input.stream.LocalFSInputStream.read(LocalFSInputStream.java:126)
at
org.apache.asterix.external.input.stream.AsterixInputStreamReader.read(AsterixInputStreamReader.java:95)
at
org.apache.asterix.external.input.stream.AsterixInputStreamReader.read(AsterixInputStreamReader.java:67)
at
org.apache.asterix.external.input.record.reader.stream.SemiStructuredRecordReader.hasNext(SemiStructuredRecordReader.java:83)
at
org.apache.asterix.external.dataflow.RecordDataFlowController.start(RecordDataFlowController.java:52)
... 8 more
java.lang.InterruptedException
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1302)
at java.util.concurrent.Semaphore.acquire(Semaphore.java:467)
at org.apache.hyracks.control.nc.Task.run(Task.java:310)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
java.lang.InterruptedException
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1302)
at java.util.concurrent.Semaphore.acquire(Semaphore.java:467)
at org.apache.hyracks.control.nc.Task.run(Task.java:310)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Jan 29, 2017 3:44:26 PM org.apache.hyracks.control.nc.Joblet close
Best,
Taewoo
Re: An exception: Input/output error?
Posted by Taewoo Kim <wa...@gmail.com>.
For the future references, I am sending this message.
As Yingyi mentioned, "Input/Output Error" happened because of a disk error.
I checked "/var/log/messages" as Yingyi suggested. For this time, it didn't
say anything special. I tried the "kmesg" command that were suggested by
Ian and found that there was a disk error. The message is as follows:
INFO: task flush-8:16:14342 blocked for more than 120 seconds.
Not tainted 2.6.32-642.1.1.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
flush-8:16 D 0000000000000000 0 14342 2 0x00000080
ffff8802093e33e0 0000000000000046 0000000000000000 ffff88003de4d8e0
0000000000000000 ffff88015e063520 0044e48cad574dea ffff88005e399840
ffff88015e063520 0000000583cf9f6a ffff88015e063ad8 ffff8802093e3fd8
Call Trace:
[<ffffffff81547db3>] io_schedule+0x73/0xc0
... ... ... ... (omitted) ... ... ... ...
[<ffffffff8100c280>] ? child_rip+0x0/0x20
ata2: lost interrupt (Status 0x51)
ata2.00: limiting speed to UDMA/33:PIO4
ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x40000000 action 0x6 frozen
ata2: SError: { }
ata2.00: failed command: READ DMA EXT
ata2.00: cmd 25/00:08:30:ed:af/00:00:25:00:00/e0 tag 0 dma 4096 in
res 40/00:93:35:ed:af/00:00:25:00:00/05 Emask 0x4 (timeout)
ata2.00: status: { DRDY }
ata2: hard resetting link
ata2: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
ata2.00: configured for UDMA/33
sd 1:0:0:0: [sdb] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
sd 1:0:0:0: [sdb] Sense Key : Aborted Command [current] [descriptor]
Descriptor sense data with sense descriptors (in hex):
72 0b 00 00 00 00 00 0c 00 0a 80 00 00 00 00 00
25 af ed 35
sd 1:0:0:0: [sdb] Add. Sense: No additional sense information
sd 1:0:0:0: [sdb] CDB: Read(10): 28 00 25 af ed 30 00 00 08 00
*end_request: I/O error, dev sdb, sector 632286512*
Best,
Taewoo
On Sun, Jan 29, 2017 at 6:14 PM, Taewoo Kim <wa...@gmail.com> wrote:
> @Yingyi: Thanks. I will look into that error.
>
> Best,
> Taewoo
>
> On Sun, Jan 29, 2017 at 5:50 PM, Yingyi Bu <bu...@gmail.com> wrote:
>
>> Hi Taewoo,
>>
>> This error message usually indicates that there are some disk
>> failures.
>> Please check /var/log/messages on the machine to see if that's the
>> case.
>>
>> Best,
>> Yingyi
>>
>> On Sun, Jan 29, 2017 at 4:00 PM, Taewoo Kim <wa...@gmail.com> wrote:
>>
>> > Hi all,
>> >
>> > I'm trying to load json files into a dataset. I have nine nodes so that
>> I
>> > divide a 250GB file into nine files. After one hour, suddenly, I see the
>> > following exception on one node. Has anyone experienced this symptom?
>> >
>> > org.apache.hyracks.api.exceptions.HyracksDataException:
>> > org.apache.hyracks.api.exceptions.HyracksDataException: Input/output
>> error
>> > at
>> > org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePusha
>> > ble.runInParallel(SuperActivityOperatorNodePushable.java:228)
>> > at
>> > org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePusha
>> > ble.initialize(SuperActivityOperatorNodePushable.java:84)
>> > at org.apache.hyracks.control.nc.Task.run(Task.java:273)
>> > at
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(
>> > ThreadPoolExecutor.java:1142)
>> > at
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(
>> > ThreadPoolExecutor.java:617)
>> > at java.lang.Thread.run(Thread.java:745)
>> > Caused by: java.util.concurrent.ExecutionException:
>> > org.apache.hyracks.api.exceptions.HyracksDataException: Input/output
>> error
>> > at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>> > at java.util.concurrent.FutureTask.get(FutureTask.java:192)
>> > at
>> > org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePusha
>> > ble.runInParallel(SuperActivityOperatorNodePushable.java:222)
>> > ... 5 more
>> > Caused by: org.apache.hyracks.api.exceptions.HyracksDataException:
>> > Input/output error
>> > at
>> > org.apache.asterix.external.operators.ExternalDataScanOperatorDescri
>> > ptor$1.initialize(ExternalDataScanOperatorDescriptor.java:65)
>> > at
>> > org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePusha
>> > ble.lambda$initialize$0(SuperActivityOperatorNodePushable.java:86)
>> > at
>> > org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePusha
>> > ble.lambda$runInParallel$2(SuperActivityOperatorNodePushable.java:216)
>> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> > ... 3 more
>> > Caused by: org.apache.hyracks.api.exceptions.HyracksDataException:
>> > Input/output error
>> > at
>> > org.apache.asterix.external.dataflow.RecordDataFlowController.start(
>> > RecordDataFlowController.java:63)
>> > at
>> > org.apache.asterix.external.dataset.adapter.GenericAdapter.start(
>> > GenericAdapter.java:37)
>> > at
>> > org.apache.asterix.external.operators.ExternalDataScanOperatorDescri
>> > ptor$1.initialize(ExternalDataScanOperatorDescriptor.java:62)
>> > ... 6 more
>> > Caused by: java.io.IOException: Input/output error
>> > at java.io.FileInputStream.readBytes(Native Method)
>> > at java.io.FileInputStream.read(FileInputStream.java:255)
>> > at
>> > org.apache.asterix.external.input.stream.LocalFSInputStream.read(
>> > LocalFSInputStream.java:126)
>> > at
>> > org.apache.asterix.external.input.stream.AsterixInputStreamReader.read(
>> > AsterixInputStreamReader.java:95)
>> > at
>> > org.apache.asterix.external.input.stream.AsterixInputStreamReader.read(
>> > AsterixInputStreamReader.java:67)
>> > at
>> > org.apache.asterix.external.input.record.reader.stream.
>> > SemiStructuredRecordReader.hasNext(SemiStructuredRecordReader.java:83)
>> > at
>> > org.apache.asterix.external.dataflow.RecordDataFlowController.start(
>> > RecordDataFlowController.java:52)
>> > ... 8 more
>> > java.lang.InterruptedException
>> > at
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer.
>> > acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1302)
>> > at java.util.concurrent.Semaphore.acquire(Semaphore.java:467)
>> > at org.apache.hyracks.control.nc.Task.run(Task.java:310)
>> > at
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(
>> > ThreadPoolExecutor.java:1142)
>> > at
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(
>> > ThreadPoolExecutor.java:617)
>> > at java.lang.Thread.run(Thread.java:745)
>> > java.lang.InterruptedException
>> > at
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer.
>> > acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1302)
>> > at java.util.concurrent.Semaphore.acquire(Semaphore.java:467)
>> > at org.apache.hyracks.control.nc.Task.run(Task.java:310)
>> > at
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(
>> > ThreadPoolExecutor.java:1142)
>> > at
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(
>> > ThreadPoolExecutor.java:617)
>> > at java.lang.Thread.run(Thread.java:745)
>> > Jan 29, 2017 3:44:26 PM org.apache.hyracks.control.nc.Joblet close
>> >
>> >
>> > Best,
>> > Taewoo
>> >
>>
>
>
Re: An exception: Input/output error?
Posted by Taewoo Kim <wa...@gmail.com>.
@Yingyi: Thanks. I will look into that error.
Best,
Taewoo
On Sun, Jan 29, 2017 at 5:50 PM, Yingyi Bu <bu...@gmail.com> wrote:
> Hi Taewoo,
>
> This error message usually indicates that there are some disk failures.
> Please check /var/log/messages on the machine to see if that's the
> case.
>
> Best,
> Yingyi
>
> On Sun, Jan 29, 2017 at 4:00 PM, Taewoo Kim <wa...@gmail.com> wrote:
>
> > Hi all,
> >
> > I'm trying to load json files into a dataset. I have nine nodes so that I
> > divide a 250GB file into nine files. After one hour, suddenly, I see the
> > following exception on one node. Has anyone experienced this symptom?
> >
> > org.apache.hyracks.api.exceptions.HyracksDataException:
> > org.apache.hyracks.api.exceptions.HyracksDataException: Input/output
> error
> > at
> > org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePusha
> > ble.runInParallel(SuperActivityOperatorNodePushable.java:228)
> > at
> > org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePusha
> > ble.initialize(SuperActivityOperatorNodePushable.java:84)
> > at org.apache.hyracks.control.nc.Task.run(Task.java:273)
> > at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(
> > ThreadPoolExecutor.java:1142)
> > at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> > Caused by: java.util.concurrent.ExecutionException:
> > org.apache.hyracks.api.exceptions.HyracksDataException: Input/output
> error
> > at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> > at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> > at
> > org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePusha
> > ble.runInParallel(SuperActivityOperatorNodePushable.java:222)
> > ... 5 more
> > Caused by: org.apache.hyracks.api.exceptions.HyracksDataException:
> > Input/output error
> > at
> > org.apache.asterix.external.operators.ExternalDataScanOperatorDescri
> > ptor$1.initialize(ExternalDataScanOperatorDescriptor.java:65)
> > at
> > org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePusha
> > ble.lambda$initialize$0(SuperActivityOperatorNodePushable.java:86)
> > at
> > org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePusha
> > ble.lambda$runInParallel$2(SuperActivityOperatorNodePushable.java:216)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > ... 3 more
> > Caused by: org.apache.hyracks.api.exceptions.HyracksDataException:
> > Input/output error
> > at
> > org.apache.asterix.external.dataflow.RecordDataFlowController.start(
> > RecordDataFlowController.java:63)
> > at
> > org.apache.asterix.external.dataset.adapter.GenericAdapter.start(
> > GenericAdapter.java:37)
> > at
> > org.apache.asterix.external.operators.ExternalDataScanOperatorDescri
> > ptor$1.initialize(ExternalDataScanOperatorDescriptor.java:62)
> > ... 6 more
> > Caused by: java.io.IOException: Input/output error
> > at java.io.FileInputStream.readBytes(Native Method)
> > at java.io.FileInputStream.read(FileInputStream.java:255)
> > at
> > org.apache.asterix.external.input.stream.LocalFSInputStream.read(
> > LocalFSInputStream.java:126)
> > at
> > org.apache.asterix.external.input.stream.AsterixInputStreamReader.read(
> > AsterixInputStreamReader.java:95)
> > at
> > org.apache.asterix.external.input.stream.AsterixInputStreamReader.read(
> > AsterixInputStreamReader.java:67)
> > at
> > org.apache.asterix.external.input.record.reader.stream.
> > SemiStructuredRecordReader.hasNext(SemiStructuredRecordReader.java:83)
> > at
> > org.apache.asterix.external.dataflow.RecordDataFlowController.start(
> > RecordDataFlowController.java:52)
> > ... 8 more
> > java.lang.InterruptedException
> > at
> > java.util.concurrent.locks.AbstractQueuedSynchronizer.
> > acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1302)
> > at java.util.concurrent.Semaphore.acquire(Semaphore.java:467)
> > at org.apache.hyracks.control.nc.Task.run(Task.java:310)
> > at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(
> > ThreadPoolExecutor.java:1142)
> > at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> > java.lang.InterruptedException
> > at
> > java.util.concurrent.locks.AbstractQueuedSynchronizer.
> > acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1302)
> > at java.util.concurrent.Semaphore.acquire(Semaphore.java:467)
> > at org.apache.hyracks.control.nc.Task.run(Task.java:310)
> > at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(
> > ThreadPoolExecutor.java:1142)
> > at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> > Jan 29, 2017 3:44:26 PM org.apache.hyracks.control.nc.Joblet close
> >
> >
> > Best,
> > Taewoo
> >
>
Re: An exception: Input/output error?
Posted by Yingyi Bu <bu...@gmail.com>.
Hi Taewoo,
This error message usually indicates that there are some disk failures.
Please check /var/log/messages on the machine to see if that's the case.
Best,
Yingyi
On Sun, Jan 29, 2017 at 4:00 PM, Taewoo Kim <wa...@gmail.com> wrote:
> Hi all,
>
> I'm trying to load json files into a dataset. I have nine nodes so that I
> divide a 250GB file into nine files. After one hour, suddenly, I see the
> following exception on one node. Has anyone experienced this symptom?
>
> org.apache.hyracks.api.exceptions.HyracksDataException:
> org.apache.hyracks.api.exceptions.HyracksDataException: Input/output error
> at
> org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePusha
> ble.runInParallel(SuperActivityOperatorNodePushable.java:228)
> at
> org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePusha
> ble.initialize(SuperActivityOperatorNodePushable.java:84)
> at org.apache.hyracks.control.nc.Task.run(Task.java:273)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.hyracks.api.exceptions.HyracksDataException: Input/output error
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePusha
> ble.runInParallel(SuperActivityOperatorNodePushable.java:222)
> ... 5 more
> Caused by: org.apache.hyracks.api.exceptions.HyracksDataException:
> Input/output error
> at
> org.apache.asterix.external.operators.ExternalDataScanOperatorDescri
> ptor$1.initialize(ExternalDataScanOperatorDescriptor.java:65)
> at
> org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePusha
> ble.lambda$initialize$0(SuperActivityOperatorNodePushable.java:86)
> at
> org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePusha
> ble.lambda$runInParallel$2(SuperActivityOperatorNodePushable.java:216)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ... 3 more
> Caused by: org.apache.hyracks.api.exceptions.HyracksDataException:
> Input/output error
> at
> org.apache.asterix.external.dataflow.RecordDataFlowController.start(
> RecordDataFlowController.java:63)
> at
> org.apache.asterix.external.dataset.adapter.GenericAdapter.start(
> GenericAdapter.java:37)
> at
> org.apache.asterix.external.operators.ExternalDataScanOperatorDescri
> ptor$1.initialize(ExternalDataScanOperatorDescriptor.java:62)
> ... 6 more
> Caused by: java.io.IOException: Input/output error
> at java.io.FileInputStream.readBytes(Native Method)
> at java.io.FileInputStream.read(FileInputStream.java:255)
> at
> org.apache.asterix.external.input.stream.LocalFSInputStream.read(
> LocalFSInputStream.java:126)
> at
> org.apache.asterix.external.input.stream.AsterixInputStreamReader.read(
> AsterixInputStreamReader.java:95)
> at
> org.apache.asterix.external.input.stream.AsterixInputStreamReader.read(
> AsterixInputStreamReader.java:67)
> at
> org.apache.asterix.external.input.record.reader.stream.
> SemiStructuredRecordReader.hasNext(SemiStructuredRecordReader.java:83)
> at
> org.apache.asterix.external.dataflow.RecordDataFlowController.start(
> RecordDataFlowController.java:52)
> ... 8 more
> java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.
> acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1302)
> at java.util.concurrent.Semaphore.acquire(Semaphore.java:467)
> at org.apache.hyracks.control.nc.Task.run(Task.java:310)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.
> acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1302)
> at java.util.concurrent.Semaphore.acquire(Semaphore.java:467)
> at org.apache.hyracks.control.nc.Task.run(Task.java:310)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Jan 29, 2017 3:44:26 PM org.apache.hyracks.control.nc.Joblet close
>
>
> Best,
> Taewoo
>