[jira] [Updated] (PARQUET-1281) Jackson dependency

["Qinghui Xu (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/PARQUET-1281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Qinghui Xu updated PARQUET-1281:
--------------------------------
    Affects Version/s: 1.9.0
                       1.10.0
        Fix Version/s: 1.11.0
           Issue Type: Bug  (was: Improvement)

I recently talked to some people who stumbled again into this problem, so I'll put more context here about this ticket.

This problem is impacting 1.9.0 when turning on debug logging on mappers:
{code:java}
2019-04-18 16:09:12,063 ERROR [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.NoSuchMethodError: org.codehaus.jackson.map.ObjectMapper.writerWithDefaultPrettyPrinter()Lorg/codehaus/jackson/map/ObjectWriter;
	at jmoab.org.apache.parquet.hadoop.metadata.ParquetMetadata.toJSON(ParquetMetadata.java:66)
	at jmoab.org.apache.parquet.hadoop.metadata.ParquetMetadata.toPrettyJSON(ParquetMetadata.java:59)
	at jmoab.org.apache.parquet.format.converter.ParquetMetadataConverter.readParquetMetadata(ParquetMetadataConverter.java:880)
	at jmoab.org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:516)
	at jmoab.org.apache.parquet.hadoop.ParquetFileReader.<init>(ParquetFileReader.java:649)
	at jmoab.org.apache.parquet.hadoop.ParquetFileReader.open(ParquetFileReader.java:562)
	at jmoab.org.apache.parquet.hadoop.ParquetRecordReader.initializeInternalReader(ParquetRecordReader.java:164)
	at jmoab.org.apache.parquet.hadoop.ParquetRecordReader.initialize(ParquetRecordReader.java:148)
	at com.criteo.datadisco.mapred.parquetproto.WrappedProtoParquetInputFormat.getRecordReader(WrappedProtoParquetInputFormat.java:115)
	at com.criteo.datadisco.mapred.HiveLogInputFormat.getHiveRecordReader(HiveLogInputFormat.java:133)
	at com.criteo.datadisco.mapred.HiveLogInputFormat.getRecordReader(HiveLogInputFormat.java:105)
	at org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:250)
	at com.criteo.hadoop.hive.ql.io.PailOrCombineHiveInputFormat.getRecordReader(PailOrCombineHiveInputFormat.java:121)
	at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:169)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:438)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158){code}

But it is solved as a side effect in PARQUET-1529 ([https://github.com/apache/parquet-mr/pull/617)|https://github.com/apache/parquet-mr/pull/617], so should be fixed in next release (1.11.0)

> Jackson dependency
> ------------------
>
>                 Key: PARQUET-1281
>                 URL: https://issues.apache.org/jira/browse/PARQUET-1281
>             Project: Parquet
>          Issue Type: Bug
>    Affects Versions: 1.9.0, 1.10.0
>            Reporter: Qinghui Xu
>            Priority: Major
>             Fix For: 1.11.0
>
>
> Currently we shaded jackson in parquet-jackson module (org.codehaus.jackon --> shaded.parquet.org.codehaus.jackson), but in fact we do not use the shaded jackson in parquet-hadoop code. Is that a mistake? (see https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/metadata/ParquetMetadata.java#L26)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)