You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Jerry Lam (JIRA)" <ji...@apache.org> on 2015/09/18 21:31:04 UTC
[jira] [Comment Edited] (SPARK-8118) Turn off noisy log output
produced by Parquet 1.7.0
[ https://issues.apache.org/jira/browse/SPARK-8118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14876198#comment-14876198 ]
Jerry Lam edited comment on SPARK-8118 at 9/18/15 7:30 PM:
-----------------------------------------------------------
I'm trying to turn off parquet logging in spark 1.5.0 by adding these lines into the log4j.properties:
log4j.logger.org.apache.parquet=ERROR
log4j.logger.parquet=ERROR
However, it does not work as expected. I still got A LOT of info logs from parquet. It is ridiculous because it generates more data than the data I'm using for analysis. :)
How to shut parquet it to ERROR only?
was (Author: superwai):
I'm trying to turn off parquet logging by adding these lines into the log4j.properties:
log4j.logger.org.apache.parquet=ERROR
log4j.logger.parquet=ERROR
However, it does not work as expected. I still got A LOT of info logs from parquet. It is ridiculous because it generates more data than the data I'm using for analysis. :)
How to shut parquet it to ERROR only?
> Turn off noisy log output produced by Parquet 1.7.0
> ---------------------------------------------------
>
> Key: SPARK-8118
> URL: https://issues.apache.org/jira/browse/SPARK-8118
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 1.4.1, 1.5.0
> Reporter: Cheng Lian
> Assignee: Cheng Lian
> Priority: Minor
> Fix For: 1.5.0
>
>
> Parquet 1.7.0 renames package name to "org.apache.parquet", need to adjust {{ParquetRelation.enableLogForwarding}} accordingly to avoid noisy log output.
> A better approach than simply muting these log lines is to redirect Parquet logs via SLF4J, so that we can handle them consistently. In general these logs are very useful. Esp. when used to diagnosing Parquet memory issue and filter push-down.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org