[jira] [Commented] (TEZ-3348) NullPointerException in Tez MROutput while trying to write using Parquet's DeprecatedParquetOutputFormat

["Hitesh Shah (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TEZ-3348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15376096#comment-15376096 ] 

Hitesh Shah commented on TEZ-3348:
----------------------------------

Thanks for reporting the issue [~pnarang] and yes, a PR or a patch attached to this JIRA sounds good. 

> NullPointerException in Tez MROutput while trying to write using Parquet's DeprecatedParquetOutputFormat
> --------------------------------------------------------------------------------------------------------
>
>                 Key: TEZ-3348
>                 URL: https://issues.apache.org/jira/browse/TEZ-3348
>             Project: Apache Tez
>          Issue Type: Bug
>            Reporter: Piyush Narang
>
> Trying to run some Tez MR jobs that write out some data using Parquet to HDFS. When I try to do so, end up seeing a NPE in the Parquet code:
> {code}
> java.lang.NullPointerException
> 	at org.apache.hadoop.fs.Path.<init>(Path.java:105)
> 	at org.apache.hadoop.fs.Path.<init>(Path.java:94)
> 	at org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.getDefaultWorkFile(DeprecatedParquetOutputFormat.java:69)
> 	at org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.access$100(DeprecatedParquetOutputFormat.java:36)
> 	at org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat$RecordWriterWrapper.<init>(DeprecatedParquetOutputFormat.java:89)
> 	at org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.getRecordWriter(DeprecatedParquetOutputFormat.java:77)
> 	at org.apache.tez.mapreduce.output.MROutput.initialize(MROutput.java:416)
> {code}
> The flow seems to be:
> 1) The Parquet deprecated output format class tries to read the workOutputPath - https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/mapred/DeprecatedParquetOutputFormat.java#L69
> 2) This calls FileOutputFormat.getWorkOutputPath(...) - https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileOutputFormat.java#L229
> 3) That in turn tries to read the JobContext.TASK_OUTPUT_DIR ("mapreduce.task.output.dir") constant. 
> 4) This ends up being null and in the Parquet code we end up with an NPE in the Path class. 
> Looking at the Tez code, we are setting the workOutputPath in the MROutput.initCommitter method - https://github.com/apache/tez/blob/master/tez-mapreduce/src/main/java/org/apache/tez/mapreduce/output/MROutput.java#L445. 
> This call however, is made after the call to access the workOutputPath as part of outputFormat.getRecordWriter(). 
> I tried out a run where I moved this initCommitter call up:
> {code}
> else {
>       oldApiTaskAttemptContext =
>           new org.apache.tez.mapreduce.hadoop.mapred.TaskAttemptContextImpl(
>               jobConf, taskAttemptId,
>               new MRTaskReporter(getContext()));
>       initCommitter(jobConf, useNewApi); // before the getRecordWriter call
>       oldOutputFormat = jobConf.getOutputFormat();
>       outputFormatClassName = oldOutputFormat.getClass().getName();
>       FileSystem fs = FileSystem.get(jobConf);
>       String finalName = getOutputName();
>       oldRecordWriter =
>           oldOutputFormat.getRecordWriter(
>               fs, jobConf, finalName, new MRReporter(getContext().getCounters()));
>     }
> {code}
> I tried out a run with this and it seems to succeed. If this sounds reasonable, I can cut a PR. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)