You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kylin.apache.org by "Chao Long (JIRA)" <ji...@apache.org> on 2018/07/04 09:00:00 UTC
[jira] [Comment Edited] (KYLIN-3431) Avoid
FileInputStream/FileOutputStream
[ https://issues.apache.org/jira/browse/KYLIN-3431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16532442#comment-16532442 ]
Chao Long edited comment on KYLIN-3431 at 7/4/18 8:59 AM:
----------------------------------------------------------
I have replaced FileInputStream/FileOutputStream with FIles.newInputStream/newOutputStream, but when I run CI, I get error message like this:
2018-07-04 06:10:41,485 INFO [main] threadpool.DefaultScheduler:192 : Shutting down DefaultScheduler ....
2018-07-04 06:10:41,487 ERROR [Scheduler 107702879 Job b036fdd2-73e2-4463-b1a6-841018297f1c-21] dao.ExecutableDao:219 : error update job output id:b036fdd2-73e2-4463-b1a6-841018297f1c-00
java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:216)
at java.nio.channels.Channels.writeFullyImpl(Channels.java:78)
at java.nio.channels.Channels.writeFully(Channels.java:101)
at java.nio.channels.Channels.access$000(Channels.java:61)
at java.nio.channels.Channels$1.write(Channels.java:174)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1793)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1769)
at org.apache.commons.io.IOUtils.copy(IOUtils.java:1744)
at org.apache.kylin.common.persistence.FileResourceStore.putResourceImpl(FileResourceStore.java:161)
at org.apache.kylin.common.persistence.FileResourceStore.checkAndPutResourceImpl(FileResourceStore.java:177)
at org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceCheckpoint(ResourceStore.java:318)
at org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:303)
at org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:282)
at org.apache.kylin.job.dao.ExecutableDao.writeJobOutputResource(ExecutableDao.java:88)
at org.apache.kylin.job.dao.ExecutableDao.updateJobOutput(ExecutableDao.java:216)
at org.apache.kylin.job.execution.ExecutableManager.addJobInfo(ExecutableManager.java:480)
at org.apache.kylin.job.execution.ExecutableManager.addJobInfo(ExecutableManager.java:490)
at org.apache.kylin.job.execution.AbstractExecutable.addExtraInfo(AbstractExecutable.java:403)
at org.apache.kylin.job.execution.AbstractExecutable.setEndTime(AbstractExecutable.java:419)
at org.apache.kylin.job.execution.AbstractExecutable.onExecuteFinished(AbstractExecutable.java:121)
at org.apache.kylin.job.execution.AbstractExecutable.onExecuteFinishedWithRetry(AbstractExecutable.java:98)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:175)
at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:69)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:162)
at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:113)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
code block: FileResourceStore.java->putResourceImpl()
@Override
protected void putResourceImpl(String resPath, InputStream content, long ts) throws IOException {
synchronized (FileResourceStore.class) {
File f = file(resPath);
f.getParentFile().mkdirs();
try (final OutputStream out = Files.newOutputStream(f.toPath())){
IOUtils.copy(content, out);
f.setLastModified(ts);
}
}
}
After checking the code, I find method DefaultSchedulerTest.java->testSchedulerStop() cause this problem. I also find "FIle.newInputStream" use "channal" to do his work, and If a class implements the "InterruptibleChannel" interface, then when the thread on the channel is interrupted, the channel will be closed and the thread will throws "ClosedByInterruptException" exception.
So can I catch this exception["ClosedByInterruptException"] in method["putResourceImpl"] or any other suggestions provided?
was (Author: wayne0101):
I have replaced FileInputStream/FileOutputStream with FIles.newInputStream/newOutputStream, but when I run CI, I get error message like this:
2018-07-04 06:10:41,485 INFO [main] threadpool.DefaultScheduler:192 : Shutting down DefaultScheduler ....
2018-07-04 06:10:41,487 ERROR [Scheduler 107702879 Job b036fdd2-73e2-4463-b1a6-841018297f1c-21] dao.ExecutableDao:219 : error update job output id:b036fdd2-73e2-4463-b1a6-841018297f1c-00
java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:216)
at java.nio.channels.Channels.writeFullyImpl(Channels.java:78)
at java.nio.channels.Channels.writeFully(Channels.java:101)
at java.nio.channels.Channels.access$000(Channels.java:61)
at java.nio.channels.Channels$1.write(Channels.java:174)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1793)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1769)
at org.apache.commons.io.IOUtils.copy(IOUtils.java:1744)
at org.apache.kylin.common.persistence.FileResourceStore.putResourceImpl(FileResourceStore.java:161)
at org.apache.kylin.common.persistence.FileResourceStore.checkAndPutResourceImpl(FileResourceStore.java:177)
at org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceCheckpoint(ResourceStore.java:318)
at org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:303)
at org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:282)
at org.apache.kylin.job.dao.ExecutableDao.writeJobOutputResource(ExecutableDao.java:88)
at org.apache.kylin.job.dao.ExecutableDao.updateJobOutput(ExecutableDao.java:216)
at org.apache.kylin.job.execution.ExecutableManager.addJobInfo(ExecutableManager.java:480)
at org.apache.kylin.job.execution.ExecutableManager.addJobInfo(ExecutableManager.java:490)
at org.apache.kylin.job.execution.AbstractExecutable.addExtraInfo(AbstractExecutable.java:403)
at org.apache.kylin.job.execution.AbstractExecutable.setEndTime(AbstractExecutable.java:419)
at org.apache.kylin.job.execution.AbstractExecutable.onExecuteFinished(AbstractExecutable.java:121)
at org.apache.kylin.job.execution.AbstractExecutable.onExecuteFinishedWithRetry(AbstractExecutable.java:98)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:175)
at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:69)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:162)
at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:113)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
code block: FileResourceStore.java->putResourceImpl()
@Override
protected void putResourceImpl(String resPath, InputStream content, long ts) throws IOException {
synchronized (FileResourceStore.class) {
File f = file(resPath);
f.getParentFile().mkdirs();
try (final OutputStream out = Files.newOutputStream(f.toPath())) {
IOUtils.copy(content, out);
f.setLastModified(ts);
}
}
}
After checking the code, I find method DefaultSchedulerTest.java->testSchedulerStop() cause this problem. I also find "FIle.newInputStream" use "channal" to do his work, and If a class implements the "InterruptibleChannel" interface, then when the thread on the channel is interrupted, the channel will be closed and the thread will throws "ClosedByInterruptException" exception.
So can I catch this exception["ClosedByInterruptException"] in method["putResourceImpl"] or any other suggestions provided?
> Avoid FileInputStream/FileOutputStream
> --------------------------------------
>
> Key: KYLIN-3431
> URL: https://issues.apache.org/jira/browse/KYLIN-3431
> Project: Kylin
> Issue Type: Improvement
> Reporter: Ted Yu
> Priority: Major
> Fix For: v2.5.0
>
> Attachments: image-2018-07-04-16-41-54-234.png
>
>
> They rely on finalizers (before Java 11), which create unnecessary GC load. The alternatives, {{Files.newInputStream}}, are as easy to use and don't have this issue.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)