[jira] [Created] (HIVE-6748) FileSinkOperator needs to cleanup held references for container reuse

["Gopal V (JIRA)" <ji...@apache.org>]

Gopal V created HIVE-6748:
-----------------------------

             Summary: FileSinkOperator needs to cleanup held references for container reuse
                 Key: HIVE-6748
                 URL: https://issues.apache.org/jira/browse/HIVE-6748
             Project: Hive
          Issue Type: Bug
          Components: Tez
    Affects Versions: 0.13.0
            Reporter: Gopal V
            Assignee: Gopal V


The current implementation of FileSinkOperator runs into trouble when reusing the same query pipeline aggressively with container reuse.

This is due to a prevFSP writer which is left referenced after closeOp() and which is not reset even for initializeOp().

{code}
014-03-25 14:46:31,744 FATAL [main] org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor: org.apache.hadoop.hive.ql.metadata.HiveException: java.nio.channels.ClosedChannelException
        at org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.closeWriters(FileSinkOperator.java:170)
        at org.apache.hadoop.hive.ql.exec.FileSinkOperator.getDynOutPaths(FileSinkOperator.java:758)
        at org.apache.hadoop.hive.ql.exec.FileSinkOperator.startGroup(FileSinkOperator.java:833)
        at org.apache.hadoop.hive.ql.exec.Operator.defaultStartGroup(Operator.java:497)
        at org.apache.hadoop.hive.ql.exec.Operator.startGroup(Operator.java:520)
        at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.processKeyValues(ReduceRecordProcessor.java:296)
        at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:223)
        at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:159)
        at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:306)
        at org.apache.hadoop.mapred.YarnTezDagChild$4.run(YarnTezDagChild.java:549)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
        at org.apache.hadoop.mapred.YarnTezDagChild.main(YarnTezDagChild.java:538)
Caused by: java.nio.channels.ClosedChannelException
        at org.apache.hadoop.hdfs.DFSOutputStream.checkClosed(DFSOutputStream.java:1526)
        at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:98)
        at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:58)
        at java.io.DataOutputStream.write(DataOutputStream.java:107)
        at org.apache.hadoop.hive.ql.io.orc.WriterImpl$DirectStream.output(WriterImpl.java:316)
        at org.apache.hadoop.hive.ql.io.orc.OutStream.flush(OutStream.java:242)
        at org.apache.hadoop.hive.ql.io.orc.WriterImpl.writeMetadata(WriterImpl.java:1923)
        at org.apache.hadoop.hive.ql.io.orc.WriterImpl.close(WriterImpl.java:2017)
        at org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat$OrcRecordWriter.close(OrcOutputFormat.java:98)
        at org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.closeWriters(FileSinkOperator.java:167)
        ... 13 more
{code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)