You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Xin Ren (JIRA)" <ji...@apache.org> on 2016/07/12 18:05:20 UTC
[jira] [Comment Edited] (SPARK-16437) SparkR read.df() from parquet
got error: SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder"
[ https://issues.apache.org/jira/browse/SPARK-16437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15372210#comment-15372210 ]
Xin Ren edited comment on SPARK-16437 at 7/12/16 6:04 PM:
----------------------------------------------------------
I worked on this for couple days, and I found it's not caused by Spark, but the parquet library "parquet-mr/parquet-hadoop".
I've debug by step, and found this error is from here: https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala#L820
and after digging into "parquet-hadoop", it's mostly probably because this library is missing the slf4j binder:
https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java#L231
But it's technically not a bug, since Spark is using latest version of slf4j and parquet
{code}
<slf4j.version>1.7.16</slf4j.version>
<parquet.version>1.8.1</parquet.version>
{code}
and since 1.6 SLF4J is defaulting to no-operation (NOP) logger implementation, so should be ok.
was (Author: iamshrek):
I worked on this for couple days, and I found it's not caused by Spark, but the parquet library "parquet-mr/parquet-hadoop".
I've debug by step, and found this error is from here: https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala#L820
and after digging into "parquet-hadoop", it's mostly probably because this library is missing the slf4j binder:
https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java#L231
But it's technically not a bug, since Spark is using {code}<slf4j.version>1.7.16</slf4j.version>{code}, and since 1.6 SLF4J is defaulting to no-operation (NOP) logger implementation, so should be ok.
> SparkR read.df() from parquet got error: SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder"
> --------------------------------------------------------------------------------------------------------
>
> Key: SPARK-16437
> URL: https://issues.apache.org/jira/browse/SPARK-16437
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Reporter: Xin Ren
> Priority: Minor
>
> build SparkR with command
> {code}
> build/mvn -DskipTests -Psparkr package
> {code}
> start SparkR console
> {code}
> ./bin/sparkR
> {code}
> then get error
> {code}
> Welcome to
> ____ __
> / __/__ ___ _____/ /__
> _\ \/ _ \/ _ `/ __/ '_/
> /___/ .__/\_,_/_/ /_/\_\ version 2.0.0-SNAPSHOT
> /_/
> SparkSession available as 'spark'.
> >
> >
> > library(SparkR)
> >
> > df <- read.df("examples/src/main/resources/users.parquet")
> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> SLF4J: Defaulting to no-operation (NOP) logger implementation
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
> >
> >
> > head(df)
> 16/07/07 23:20:54 WARN ParquetRecordReader: Can not initialize counter due to context is not a instance of TaskInputOutputContext, but is org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
> name favorite_color favorite_numbers
> 1 Alyssa <NA> 3, 9, 15, 20
> 2 Ben red NULL
> {code}
> Reference
> * seems need to add a lib from slf4j to point to older version
> http://stackoverflow.com/questions/7421612/slf4j-failed-to-load-class-org-slf4j-impl-staticloggerbinder
> * on slf4j official site: http://www.slf4j.org/codes.html#StaticLoggerBinder
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org