You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by ArtemisDev <ar...@dtechspace.com> on 2020/06/29 14:19:46 UTC
File Not Found: /tmp/spark-events in Spark 3.0
While launching a spark job from Zeppelin against a standalone spark
cluster (Spark 3.0 with multiple workers without hadoop), we have
encountered a Spark interpreter exception caused by a I/O File Not Found
exception due to the non-existence of the /tmp/spark-events directory.
We had to create the /tmp/spark-events directory manually in order to
resolve the problem.
As a reference, the same notebook code run on Spark 2.4.6 (also a
standalone cluster) without any problems.
What is /tmp/spark-events for and is there anyway to pre-define this
directory as some config parameter so we don't end up manually add it in
/tmp?
Thanks!
-- ND
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: File Not Found: /tmp/spark-events in Spark 3.0
Posted by Xin Jinhan <18...@163.com>.
Hi,
First, the '/tmp/spark-events' is the default storage location of spark
eventLog, but the log will be stored in it only when the
'spark.eventLog.enabled' is true, which your spark 2.4.6 may set to false.
So you can try to set false and the error may disappear.
Second, I suggest enable eventLog and you can set the storage location by
set 'spark.eventLog.dir' to a fileSystem or local path, in case you want to
check the log later.(can simplely use spark-history-server)
Regards,
Jinhan
--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: File Not Found: /tmp/spark-events in Spark 3.0
Posted by ArtemisDev <ar...@dtechspace.com>.
Thank you all for the responses. I believe the user shouldn't be
worried about creating the log dir explicitly. The event logging should
behave like other logs (e.g. master or slave) that the directory should
be automatically created if not exist.
-- ND
On 7/2/20 9:19 AM, Zero wrote:
>
> This could be the result of you not setting the location of eventLog
> properly. By default, it's/TMP/Spark-Events, and since the files in
> the/TMP directory are cleaned up regularly, you could have this problem.
>
> ------------------ Original ------------------
> *From:* "Xin Jinhan"<18...@163.com>;
> *Date:* Thu, Jul 2, 2020 08:39 PM
> *To:* "user"<us...@spark.apache.org>;
> *Subject:* Re: File Not Found: /tmp/spark-events in Spark 3.0
>
> Hi,
>
> First, the /tmp/spark-events is the default storage location of spark
> eventLog, but the log is stored only when you set the
> 'spark.eventLog.enabled=true', which maybe your spark 2.4.6 set to
> false. So
> you can just set it to false and the error will disappear.
>
> Second, I suggest to open the eventLog and you can specify the log
> location
> with 'spark.eventLog.dir' either a filesystem or local one, because you
> maybe to check the log later.(can simplely use spark-history-server)
>
> Regards
> Jinhan
>
>
>
> --
> Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
Re: File Not Found: /tmp/spark-events in Spark 3.0
Posted by Zero <81...@qq.com>.
This could be the result of you not setting the location of eventLog properly. By default, it's/TMP/Spark-Events, and since the files in the/TMP directory are cleaned up regularly, you could have this problem.
------------------ Original ------------------
From: "Xin Jinhan"<18183124928@163.com>;
Date: Thu, Jul 2, 2020 08:39 PM
To: "user"<user@spark.apache.org>;
Subject: Re: File Not Found: /tmp/spark-events in Spark 3.0
Hi,
First, the /tmp/spark-events is the default storage location of spark
eventLog, but the log is stored only when you set the
'spark.eventLog.enabled=true', which maybe your spark 2.4.6 set to false. So
you can just set it to false and the error will disappear.
Second, I suggest to open the eventLog and you can specify the log location
with 'spark.eventLog.dir' either a filesystem or local one, because you
maybe to check the log later.(can simplely use spark-history-server)
Regards
Jinhan
--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: File Not Found: /tmp/spark-events in Spark 3.0
Posted by Xin Jinhan <18...@163.com>.
Hi,
First, the /tmp/spark-events is the default storage location of spark
eventLog, but the log is stored only when you set the
'spark.eventLog.enabled=true', which maybe your spark 2.4.6 set to false. So
you can just set it to false and the error will disappear.
Second, I suggest to open the eventLog and you can specify the log location
with 'spark.eventLog.dir' either a filesystem or local one, because you
maybe to check the log later.(can simplely use spark-history-server)
Regards
Jinhan
--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: File Not Found: /tmp/spark-events in Spark 3.0
Posted by Jeff Evans <je...@gmail.com>.
This should only be needed if the spark.eventLog.enabled property was set
to true. Is it possible the job configuration is different between your
two environments?
On Mon, Jun 29, 2020 at 9:21 AM ArtemisDev <ar...@dtechspace.com> wrote:
> While launching a spark job from Zeppelin against a standalone spark
> cluster (Spark 3.0 with multiple workers without hadoop), we have
> encountered a Spark interpreter exception caused by a I/O File Not Found
> exception due to the non-existence of the /tmp/spark-events directory.
> We had to create the /tmp/spark-events directory manually in order to
> resolve the problem.
>
> As a reference, the same notebook code run on Spark 2.4.6 (also a
> standalone cluster) without any problems.
>
> What is /tmp/spark-events for and is there anyway to pre-define this
> directory as some config parameter so we don't end up manually add it in
> /tmp?
>
> Thanks!
>
> -- ND
>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>