You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@samza.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2017/04/04 23:58:41 UTC
[jira] [Commented] (SAMZA-1189) Fix file system closed issue on
hdfs system producer
[ https://issues.apache.org/jira/browse/SAMZA-1189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15956077#comment-15956077 ]
ASF GitHub Bot commented on SAMZA-1189:
---------------------------------------
GitHub user jmakes opened a pull request:
https://github.com/apache/samza/pull/111
SAMZA-1189: Fix file system closed issue on hdfs system producer
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jmakes/samza samza-1189
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/samza/pull/111.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #111
----
commit 25e44ef12a65212aabca49d829aa9d79a1aef6b5
Author: Jacob Maes <jm...@linkedin.com>
Date: 2017-04-04T23:57:21Z
SAMZA-1189: Fix file system closed issue on hdfs system producer
----
> Fix file system closed issue on hdfs system producer
> ----------------------------------------------------
>
> Key: SAMZA-1189
> URL: https://issues.apache.org/jira/browse/SAMZA-1189
> Project: Samza
> Issue Type: Bug
> Reporter: Jake Maes
> Assignee: Jake Maes
>
> Under certain circumstances (the actual condition remains unclear), Samza Hdfs producer could hit IOException like this:
> {noFormat}
> 17/03/31 00:20:15 ERROR container.SamzaContainer: Caught exception in process loop.
> java.io.IOException: Filesystem closed
> at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:798)
> at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:2732)
> at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2713)
> at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:870)
> at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:866)
> at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirsInternal(DistributedFileSystem.java:866)
> at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:859)
> at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1817)
> at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:597)
> at org.apache.samza.system.hdfs.writer.JobNameDateTimeBucketer.getNextWritePath(JobNameDateTimeBucketer.scala:64)
> at org.apache.samza.system.hdfs.writer.AvroDataFileHdfsWriter.getNextWriter(AvroDataFileHdfsWriter.scala:69)
> at org.apache.samza.system.hdfs.writer.AvroDataFileHdfsWriter.write(AvroDataFileHdfsWriter.scala:49)
> at org.apache.samza.system.hdfs.HdfsSystemProducer$$anonfun$liftedTree2$1$1.apply$mcV$sp(HdfsSystemProducer.scala:94)
> at org.apache.samza.system.hdfs.HdfsSystemProducer$$anonfun$liftedTree2$1$1.apply(HdfsSystemProducer.scala:94)
> at org.apache.samza.system.hdfs.HdfsSystemProducer$$anonfun$liftedTree2$1$1.apply(HdfsSystemProducer.scala:94)
> at org.apache.samza.util.TimerUtils$class.updateTimer(TimerUtils.scala:37)
> at org.apache.samza.system.hdfs.HdfsSystemProducer.updateTimer(HdfsSystemProducer.scala:34)
> at org.apache.samza.system.hdfs.HdfsSystemProducer.liftedTree2$1(HdfsSystemProducer.scala:93)
> at org.apache.samza.system.hdfs.HdfsSystemProducer.send(HdfsSystemProducer.scala:92)
> at org.apache.samza.system.SystemProducers.send(SystemProducers.scala:87)
> at org.apache.samza.task.TaskInstanceCollector.send(TaskInstanceCollector.scala:60)
> at com.linkedin.samza.example.HelloWorldTask.processAsync(HelloWorldTask.java:58)
> at org.apache.samza.container.TaskInstance$$anonfun$process$1.apply$mcV$sp(TaskInstance.scala:162)
> at org.apache.samza.container.TaskInstanceExceptionHandler.maybeHandle(TaskInstanceExceptionHandler.scala:54)
> at org.apache.samza.container.TaskInstance.process(TaskInstance.scala:160)
> at org.apache.samza.task.AsyncRunLoop$AsyncTaskWorker.process(AsyncRunLoop.java:425)
> at org.apache.samza.task.AsyncRunLoop$AsyncTaskWorker.run(AsyncRunLoop.java:370)
> at org.apache.samza.task.AsyncRunLoop$AsyncTaskWorker.access$300(AsyncRunLoop.java:312)
> at org.apache.samza.task.AsyncRunLoop.runTasks(AsyncRunLoop.java:226)
> at org.apache.samza.task.AsyncRunLoop.run(AsyncRunLoop.java:156)
> at org.apache.samza.container.SamzaContainer.run(SamzaContainer.scala:743)
> at org.apache.samza.container.SamzaContainer$.safeMain(SamzaContainer.scala:114)
> at org.apache.samza.container.SamzaContainer$.main(SamzaContainer.scala:88)
> at org.apache.samza.container.SamzaContainer.main(SamzaContainer.scala)
> {noFormat}
> It appears that the FileSystem used by Hdfs producer was somehow closed before it's used by Hdfs producer. The issue can be addressed if hdfs producer always grabs a new instance of FileSystem during construction. Hence the simple fix on this RB.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)