You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by FLORANCE Grégory <gr...@cnp.fr.INVALID> on 2022/08/24 14:34:51 UTC

spark-3.2.2-bin-without-hadoop : NoClassDefFoundError: org/apache/log4j/spi/Filter when starting the master

                                          Hi,

              I've downloaded the 3.2.2-without Hadoop Spark distribution in order to test it in a without Hadoop context.
              I tested the version with Hadoop and it worked well.

              When I wanted to start the master, I encountered this basic error : Caused by: java.lang.NoClassDefFoundError: org/apache/log4j/spi/Filter

$ ./start-master.sh
starting org.apache.spark.deploy.master.Master, logging to /projets/kfk/home/kfkusrm1/spark-3.2.2-bin-without-hadoop/logs/spark-kfkusrm1-org.apache.spark.deploy.master.Master-1-pcld1313.angers.cnp.fr.out
failed to launch: nice -n 0 /projets/kfk/home/kfkusrm1/spark-3.2.2-bin-without-hadoop/bin/spark-class org.apache.spark.deploy.master.Master --host pcld1313.angers.cnp.fr --port 7077 --webui-port 8080
  Spark Command: /soft/java/jdk-11.0.11+9/bin/java -cp /projets/kfk/home/kfkusrm1/spark-3.2.2-bin-without-hadoop/conf/:/projets/kfk/home/kfkusrm1/spark-3.2.2-bin-without-hadoop/jars/* -Xmx1g org.apache.spark.deploy.master.Master --host pcld1313.angers.cnp.fr --port 7077 --webui-port 8080
  ========================================
  Error: Unable to initialize main class org.apache.spark.deploy.master.Master
  Caused by: java.lang.NoClassDefFoundError: org/apache/log4j/spi/Filter
full log in /projets/kfk/home/kfkusrm1/spark-3.2.2-bin-without-hadoop/logs/spark-kfkusrm1-org.apache.spark.deploy.master.Master-1-pcld1313.angers.cnp.fr.out

I didn't find any librairies slf4j in the distribution « without hadoop » !
But we can find it in the distribution with Hadoop :
slf4j-log4j12-1.7.30.jar
slf4j-api-1.7.30.jar

              Is it a problem in the distribution available at https://www.apache.org/dyn/closer.lua/spark/spark-3.2.2/spark-3.2.2-bin-without-hadoop.tgz ?

              Thanks for your help.

              Regards

              Grégory


[cid:image001.jpg@01D6E37C.8B6E7490]
Grégory FLORANCE
Service Architecture - YU2
Direction de l'expérience client, des services numériques et de la donnée

CNP Assurances
1 place François Mitterrand, 49100 ANGERS
Tél : 02 41 96 39 64 ou 06 08 02 63 45
gregory.florance@cnp.fr<ma...@cnp.fr>

  Pensez à l'environnement, n'imprimez ce courriel que si nécessaire


Ce message (et toutes ses pièces jointes éventuelles) est confidentiel et établi a l'intention exclusive de ses destinataires.
Toute utilisation de ce message non conforme a sa destination, toute diffusion ou toute publication, totale ou partielle, est
interdite, sauf autorisation expresse.
L'internet ne permettant pas d'assurer l’intégrité de ce message, CNP Assurances et ses filiales déclinent toute responsabilité
au titre de ce message, s'il a été altéré, déformé ou falsifié.

*****************************************************************************
*****************************************************************************

This message and any attachments (the "message") are confidential and intended solely for the addressees.
Any unauthorised use or dissemination is prohibited.
E-mails are susceptible to alteration.
Neither CNP Assurances nor any of its subsidiaries or affiliates shall be liable for the message if altered, changed or falsified.

Re: spark-3.2.2-bin-without-hadoop : NoClassDefFoundError: org/apache/log4j/spi/Filter when starting the master

Posted by Sean Owen <sr...@gmail.com>.
You have to provide your own Hadoop distro and all its dependencies. This
build is intended for use on a Hadoop cluster, really. If you're running
stand-alone, you should not be using it. Use a 'normal' distribution that
bundles Hadoop libs.

On Wed, Aug 24, 2022 at 9:35 AM FLORANCE Grégory
<gr...@cnp.fr.invalid> wrote:

>                                           Hi,
>
>
>
>               I’ve downloaded the 3.2.2-without Hadoop Spark distribution
> in order to test it in a without Hadoop context.
>
>               I tested the version with Hadoop and it worked well.
>
>
>
>               When I wanted to start the master, I encountered this basic
> error : Caused by: java.lang.NoClassDefFoundError:
> org/apache/log4j/spi/Filter
>
>
>
> $ ./start-master.sh
>
> starting org.apache.spark.deploy.master.Master, logging to
> /projets/kfk/home/kfkusrm1/spark-3.2.2-bin-without-hadoop/logs/spark-kfkusrm1-org.apache.spark.deploy.master.Master-1-pcld1313.angers.cnp.fr.out
>
> failed to launch: nice -n 0
> /projets/kfk/home/kfkusrm1/spark-3.2.2-bin-without-hadoop/bin/spark-class
> org.apache.spark.deploy.master.Master --host pcld1313.angers.cnp.fr
> --port 7077 --webui-port 8080
>
>   Spark Command: /soft/java/jdk-11.0.11+9/bin/java -cp
> /projets/kfk/home/kfkusrm1/spark-3.2.2-bin-without-hadoop/conf/:/projets/kfk/home/kfkusrm1/spark-3.2.2-bin-without-hadoop/jars/*
> -Xmx1g org.apache.spark.deploy.master.Master --host pcld1313.angers.cnp.fr
> --port 7077 --webui-port 8080
>
>   ========================================
>
>   Error: Unable to initialize main class
> org.apache.spark.deploy.master.Master
>
>   Caused by: java.lang.NoClassDefFoundError: org/apache/log4j/spi/Filter
>
> full log in
> /projets/kfk/home/kfkusrm1/spark-3.2.2-bin-without-hadoop/logs/spark-kfkusrm1-org.apache.spark.deploy.master.Master-1-pcld1313.angers.cnp.fr.out
>
>
>
> *I didn’t find any librairies slf4j* in the distribution « without
> hadoop » !
>
> But we can find it in the distribution with Hadoop :
>
> slf4j-log4j12-1.7.30.jar
>
> slf4j-api-1.7.30.jar
>
>
>
>               Is it a problem in the distribution available at
> https://www.apache.org/dyn/closer.lua/spark/spark-3.2.2/spark-3.2.2-bin-without-hadoop.tgz
> ?
>
>
>
>               Thanks for your help.
>
>
>
>               Regards
>
>
>
>               Grégory
>
>
>
>
>
> [image: cid:image001.jpg@01D6E37C.8B6E7490]
>
> *Grégory FLORANCE*
> Service Architecture - YU2
>
> Direction de l’expérience client, des services numériques et de la donnée
>
>
> CNP Assurances
> 1 place François Mitterrand, 49100 ANGERS
> Tél : 02 41 96 39 64 ou 06 08 02 63 45
> gregory.florance@cnp.fr
>
>
>
> *  Pensez à l'environnement, n'imprimez ce courriel que si nécessaire*
>
>
>
>
> Ce message (et toutes ses pièces jointes éventuelles) est confidentiel et
> établi a l'intention exclusive de ses destinataires.
> Toute utilisation de ce message non conforme a sa destination, toute
> diffusion ou toute publication, totale ou partielle, est
> interdite, sauf autorisation expresse.
> L'internet ne permettant pas d'assurer l’intégrité de ce message, CNP
> Assurances et ses filiales déclinent toute responsabilité
> au titre de ce message, s'il a été altéré, déformé ou falsifié.
>
>
> *****************************************************************************
>
> *****************************************************************************
>
> This message and any attachments (the "message") are confidential and
> intended solely for the addressees.
> Any unauthorised use or dissemination is prohibited.
> E-mails are susceptible to alteration.
> Neither CNP Assurances nor any of its subsidiaries or affiliates shall be
> liable for the message if altered, changed or falsified.
>