You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by TJ Klein <TJ...@gmail.com> on 2015/02/11 17:54:56 UTC
SPARK_LOCAL_DIRS Issue
Hi,
Using Spark 1.2 I ran into issued setting SPARK_LOCAL_DIRS to a different
path then local directory.
On our cluster we have a folder for temporary files (in a central file
system), which is called /scratch.
When setting SPARK_LOCAL_DIRS=/scratch/<node name>
I get:
An error occurred while calling
z:org.apache.spark.api.python.PythonRDD.newAPIHadoopFile.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0
in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0
(TID 3, XXXXXXX): java.io.IOException: Function not implemented
at sun.nio.ch.FileDispatcherImpl.lock0(Native Method)
at sun.nio.ch.FileDispatcherImpl.lock(FileDispatcherImpl.java:91)
at sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:1022)
at java.nio.channels.FileChannel.lock(FileChannel.java:1052)
at org.apache.spark.util.Utils$.fetchFile(Utils.scala:379)
Using SPARK_LOCAL_DIRS=/tmp, however, works perfectly. Any idea?
Best,
Tassilo
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/SPARK-LOCAL-DIRS-Issue-tp21602.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: SPARK_LOCAL_DIRS Issue
Posted by Tassilo Klein <tj...@gmail.com>.
Thanks a lot. I will have a look at it.
On Wed, Feb 11, 2015 at 2:20 PM, Charles Feduke <ch...@gmail.com>
wrote:
> Take a look at this:
>
> http://wiki.lustre.org/index.php/Running_Hadoop_with_Lustre
>
> Particularly: http://wiki.lustre.org/images/1/1b/Hadoop_wp_v0.4.2.pdf
> (linked from that article)
>
> to get a better idea of what your options are.
>
> If its possible to avoid writing to [any] disk I'd recommend that route,
> since that's the performance advantage Spark has over vanilla Hadoop.
>
> On Wed Feb 11 2015 at 2:10:36 PM Tassilo Klein <tj...@gmail.com> wrote:
>
>> Thanks for the info. The file system in use is a Lustre file system.
>>
>> Best,
>> Tassilo
>>
>> On Wed, Feb 11, 2015 at 12:15 PM, Charles Feduke <
>> charles.feduke@gmail.com> wrote:
>>
>>> A central location, such as NFS?
>>>
>>> If they are temporary for the purpose of further job processing you'll
>>> want to keep them local to the node in the cluster, i.e., in /tmp. If they
>>> are centralized you won't be able to take advantage of data locality and
>>> the central file store will become a bottleneck for further processing.
>>>
>>> If /tmp isn't an option because you want to be able to monitor the file
>>> outputs as they occur you can also use HDFS (assuming your Spark nodes are
>>> also HDFS members they will benefit from data locality).
>>>
>>> It looks like the problem you are seeing is that a lock cannot be
>>> acquired on the output file in the central file system.
>>>
>>> On Wed Feb 11 2015 at 11:55:55 AM TJ Klein <TJ...@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> Using Spark 1.2 I ran into issued setting SPARK_LOCAL_DIRS to a
>>>> different
>>>> path then local directory.
>>>>
>>>> On our cluster we have a folder for temporary files (in a central file
>>>> system), which is called /scratch.
>>>>
>>>> When setting SPARK_LOCAL_DIRS=/scratch/<node name>
>>>>
>>>> I get:
>>>>
>>>> An error occurred while calling
>>>> z:org.apache.spark.api.python.PythonRDD.newAPIHadoopFile.
>>>> : org.apache.spark.SparkException: Job aborted due to stage failure:
>>>> Task 0
>>>> in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in
>>>> stage 0.0
>>>> (TID 3, XXXXXXX): java.io.IOException: Function not implemented
>>>> at sun.nio.ch.FileDispatcherImpl.lock0(Native Method)
>>>> at sun.nio.ch.FileDispatcherImpl.lock(FileDispatcherImpl.java:
>>>> 91)
>>>> at sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:1022)
>>>> at java.nio.channels.FileChannel.lock(FileChannel.java:1052)
>>>> at org.apache.spark.util.Utils$.fetchFile(Utils.scala:379)
>>>>
>>>> Using SPARK_LOCAL_DIRS=/tmp, however, works perfectly. Any idea?
>>>>
>>>> Best,
>>>> Tassilo
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> View this message in context: http://apache-spark-user-list.
>>>> 1001560.n3.nabble.com/SPARK-LOCAL-DIRS-Issue-tp21602.html
>>>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>>> For additional commands, e-mail: user-help@spark.apache.org
>>>>
>>>>
>>
Re: SPARK_LOCAL_DIRS Issue
Posted by Tassilo Klein <tj...@gmail.com>.
Thanks. Yes, I think it might not always make sense to lock files,
particularly if every executor is getting its own path.
On Wed, Feb 11, 2015 at 2:31 PM, Charles Feduke <ch...@gmail.com>
wrote:
> And just glancing at the Spark source code around where the stack trace
> originates:
>
> val lockFile = new File(localDir, lockFileName)
> val raf = new RandomAccessFile(lockFile, "rw")
> // Only one executor entry.
> // The FileLock is only used to control synchronization for
> executors download file,
> // it's always safe regardless of lock type (mandatory or advisory).
> val lock = raf.getChannel().lock()
> val cachedFile = new File(localDir, cachedFileName)
> try {
> if (!cachedFile.exists()) {
> doFetchFile(url, localDir, cachedFileName, conf, securityMgr,
> hadoopConf)
> }
> } finally {
> lock.release()
> }
>
> I think Spark is making assumptions about the underlying file system that
> isn't safe to make (locking? I don't know enough about POSIX to know
> whether locking is part of the spec). Maybe file a bug report after someone
> from the dev team chimes in on this issue.
>
>
> On Wed Feb 11 2015 at 2:20:34 PM Charles Feduke <ch...@gmail.com>
> wrote:
>
>> Take a look at this:
>>
>> http://wiki.lustre.org/index.php/Running_Hadoop_with_Lustre
>>
>> Particularly: http://wiki.lustre.org/images/1/1b/Hadoop_wp_v0.4.2.pdf
>> (linked from that article)
>>
>> to get a better idea of what your options are.
>>
>> If its possible to avoid writing to [any] disk I'd recommend that route,
>> since that's the performance advantage Spark has over vanilla Hadoop.
>>
>> On Wed Feb 11 2015 at 2:10:36 PM Tassilo Klein <tj...@gmail.com> wrote:
>>
>>> Thanks for the info. The file system in use is a Lustre file system.
>>>
>>> Best,
>>> Tassilo
>>>
>>> On Wed, Feb 11, 2015 at 12:15 PM, Charles Feduke <
>>> charles.feduke@gmail.com> wrote:
>>>
>>>> A central location, such as NFS?
>>>>
>>>> If they are temporary for the purpose of further job processing you'll
>>>> want to keep them local to the node in the cluster, i.e., in /tmp. If they
>>>> are centralized you won't be able to take advantage of data locality and
>>>> the central file store will become a bottleneck for further processing.
>>>>
>>>> If /tmp isn't an option because you want to be able to monitor the file
>>>> outputs as they occur you can also use HDFS (assuming your Spark nodes are
>>>> also HDFS members they will benefit from data locality).
>>>>
>>>> It looks like the problem you are seeing is that a lock cannot be
>>>> acquired on the output file in the central file system.
>>>>
>>>> On Wed Feb 11 2015 at 11:55:55 AM TJ Klein <TJ...@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Using Spark 1.2 I ran into issued setting SPARK_LOCAL_DIRS to a
>>>>> different
>>>>> path then local directory.
>>>>>
>>>>> On our cluster we have a folder for temporary files (in a central file
>>>>> system), which is called /scratch.
>>>>>
>>>>> When setting SPARK_LOCAL_DIRS=/scratch/<node name>
>>>>>
>>>>> I get:
>>>>>
>>>>> An error occurred while calling
>>>>> z:org.apache.spark.api.python.PythonRDD.newAPIHadoopFile.
>>>>> : org.apache.spark.SparkException: Job aborted due to stage failure:
>>>>> Task 0
>>>>> in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in
>>>>> stage 0.0
>>>>> (TID 3, XXXXXXX): java.io.IOException: Function not implemented
>>>>> at sun.nio.ch.FileDispatcherImpl.lock0(Native Method)
>>>>> at sun.nio.ch.FileDispatcherImpl.lock(FileDispatcherImpl.java:
>>>>> 91)
>>>>> at sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:1022)
>>>>> at java.nio.channels.FileChannel.lock(FileChannel.java:1052)
>>>>> at org.apache.spark.util.Utils$.fetchFile(Utils.scala:379)
>>>>>
>>>>> Using SPARK_LOCAL_DIRS=/tmp, however, works perfectly. Any idea?
>>>>>
>>>>> Best,
>>>>> Tassilo
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> View this message in context: http://apache-spark-user-list.
>>>>> 1001560.n3.nabble.com/SPARK-LOCAL-DIRS-Issue-tp21602.html
>>>>> Sent from the Apache Spark User List mailing list archive at
>>>>> Nabble.com.
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>>>> For additional commands, e-mail: user-help@spark.apache.org
>>>>>
>>>>>
>>>
Re: SPARK_LOCAL_DIRS Issue
Posted by Charles Feduke <ch...@gmail.com>.
And just glancing at the Spark source code around where the stack trace
originates:
val lockFile = new File(localDir, lockFileName)
val raf = new RandomAccessFile(lockFile, "rw")
// Only one executor entry.
// The FileLock is only used to control synchronization for executors
download file,
// it's always safe regardless of lock type (mandatory or advisory).
val lock = raf.getChannel().lock()
val cachedFile = new File(localDir, cachedFileName)
try {
if (!cachedFile.exists()) {
doFetchFile(url, localDir, cachedFileName, conf, securityMgr,
hadoopConf)
}
} finally {
lock.release()
}
I think Spark is making assumptions about the underlying file system that
isn't safe to make (locking? I don't know enough about POSIX to know
whether locking is part of the spec). Maybe file a bug report after someone
from the dev team chimes in on this issue.
On Wed Feb 11 2015 at 2:20:34 PM Charles Feduke <ch...@gmail.com>
wrote:
> Take a look at this:
>
> http://wiki.lustre.org/index.php/Running_Hadoop_with_Lustre
>
> Particularly: http://wiki.lustre.org/images/1/1b/Hadoop_wp_v0.4.2.pdf
> (linked from that article)
>
> to get a better idea of what your options are.
>
> If its possible to avoid writing to [any] disk I'd recommend that route,
> since that's the performance advantage Spark has over vanilla Hadoop.
>
> On Wed Feb 11 2015 at 2:10:36 PM Tassilo Klein <tj...@gmail.com> wrote:
>
>> Thanks for the info. The file system in use is a Lustre file system.
>>
>> Best,
>> Tassilo
>>
>> On Wed, Feb 11, 2015 at 12:15 PM, Charles Feduke <
>> charles.feduke@gmail.com> wrote:
>>
>>> A central location, such as NFS?
>>>
>>> If they are temporary for the purpose of further job processing you'll
>>> want to keep them local to the node in the cluster, i.e., in /tmp. If they
>>> are centralized you won't be able to take advantage of data locality and
>>> the central file store will become a bottleneck for further processing.
>>>
>>> If /tmp isn't an option because you want to be able to monitor the file
>>> outputs as they occur you can also use HDFS (assuming your Spark nodes are
>>> also HDFS members they will benefit from data locality).
>>>
>>> It looks like the problem you are seeing is that a lock cannot be
>>> acquired on the output file in the central file system.
>>>
>>> On Wed Feb 11 2015 at 11:55:55 AM TJ Klein <TJ...@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> Using Spark 1.2 I ran into issued setting SPARK_LOCAL_DIRS to a
>>>> different
>>>> path then local directory.
>>>>
>>>> On our cluster we have a folder for temporary files (in a central file
>>>> system), which is called /scratch.
>>>>
>>>> When setting SPARK_LOCAL_DIRS=/scratch/<node name>
>>>>
>>>> I get:
>>>>
>>>> An error occurred while calling
>>>> z:org.apache.spark.api.python.PythonRDD.newAPIHadoopFile.
>>>> : org.apache.spark.SparkException: Job aborted due to stage failure:
>>>> Task 0
>>>> in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in
>>>> stage 0.0
>>>> (TID 3, XXXXXXX): java.io.IOException: Function not implemented
>>>> at sun.nio.ch.FileDispatcherImpl.lock0(Native Method)
>>>> at sun.nio.ch.FileDispatcherImpl.lock(FileDispatcherImpl.java:9
>>>> 1)
>>>> at sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:1022)
>>>> at java.nio.channels.FileChannel.lock(FileChannel.java:1052)
>>>> at org.apache.spark.util.Utils$.fetchFile(Utils.scala:379)
>>>>
>>>> Using SPARK_LOCAL_DIRS=/tmp, however, works perfectly. Any idea?
>>>>
>>>> Best,
>>>> Tassilo
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> View this message in context: http://apache-spark-user-list.
>>>> 1001560.n3.nabble.com/SPARK-LOCAL-DIRS-Issue-tp21602.html
>>>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>>> For additional commands, e-mail: user-help@spark.apache.org
>>>>
>>>>
>>
Re: SPARK_LOCAL_DIRS Issue
Posted by Charles Feduke <ch...@gmail.com>.
Take a look at this:
http://wiki.lustre.org/index.php/Running_Hadoop_with_Lustre
Particularly: http://wiki.lustre.org/images/1/1b/Hadoop_wp_v0.4.2.pdf
(linked from that article)
to get a better idea of what your options are.
If its possible to avoid writing to [any] disk I'd recommend that route,
since that's the performance advantage Spark has over vanilla Hadoop.
On Wed Feb 11 2015 at 2:10:36 PM Tassilo Klein <tj...@gmail.com> wrote:
> Thanks for the info. The file system in use is a Lustre file system.
>
> Best,
> Tassilo
>
> On Wed, Feb 11, 2015 at 12:15 PM, Charles Feduke <charles.feduke@gmail.com
> > wrote:
>
>> A central location, such as NFS?
>>
>> If they are temporary for the purpose of further job processing you'll
>> want to keep them local to the node in the cluster, i.e., in /tmp. If they
>> are centralized you won't be able to take advantage of data locality and
>> the central file store will become a bottleneck for further processing.
>>
>> If /tmp isn't an option because you want to be able to monitor the file
>> outputs as they occur you can also use HDFS (assuming your Spark nodes are
>> also HDFS members they will benefit from data locality).
>>
>> It looks like the problem you are seeing is that a lock cannot be
>> acquired on the output file in the central file system.
>>
>> On Wed Feb 11 2015 at 11:55:55 AM TJ Klein <TJ...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> Using Spark 1.2 I ran into issued setting SPARK_LOCAL_DIRS to a different
>>> path then local directory.
>>>
>>> On our cluster we have a folder for temporary files (in a central file
>>> system), which is called /scratch.
>>>
>>> When setting SPARK_LOCAL_DIRS=/scratch/<node name>
>>>
>>> I get:
>>>
>>> An error occurred while calling
>>> z:org.apache.spark.api.python.PythonRDD.newAPIHadoopFile.
>>> : org.apache.spark.SparkException: Job aborted due to stage failure:
>>> Task 0
>>> in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage
>>> 0.0
>>> (TID 3, XXXXXXX): java.io.IOException: Function not implemented
>>> at sun.nio.ch.FileDispatcherImpl.lock0(Native Method)
>>> at sun.nio.ch.FileDispatcherImpl.lock(FileDispatcherImpl.java:
>>> 91)
>>> at sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:1022)
>>> at java.nio.channels.FileChannel.lock(FileChannel.java:1052)
>>> at org.apache.spark.util.Utils$.fetchFile(Utils.scala:379)
>>>
>>> Using SPARK_LOCAL_DIRS=/tmp, however, works perfectly. Any idea?
>>>
>>> Best,
>>> Tassilo
>>>
>>>
>>>
>>>
>>>
>>> --
>>> View this message in context: http://apache-spark-user-list.
>>> 1001560.n3.nabble.com/SPARK-LOCAL-DIRS-Issue-tp21602.html
>>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>> For additional commands, e-mail: user-help@spark.apache.org
>>>
>>>
>
Re: SPARK_LOCAL_DIRS Issue
Posted by Tassilo Klein <tj...@gmail.com>.
Thanks for the info. The file system in use is a Lustre file system.
Best,
Tassilo
On Wed, Feb 11, 2015 at 12:15 PM, Charles Feduke <ch...@gmail.com>
wrote:
> A central location, such as NFS?
>
> If they are temporary for the purpose of further job processing you'll
> want to keep them local to the node in the cluster, i.e., in /tmp. If they
> are centralized you won't be able to take advantage of data locality and
> the central file store will become a bottleneck for further processing.
>
> If /tmp isn't an option because you want to be able to monitor the file
> outputs as they occur you can also use HDFS (assuming your Spark nodes are
> also HDFS members they will benefit from data locality).
>
> It looks like the problem you are seeing is that a lock cannot be acquired
> on the output file in the central file system.
>
> On Wed Feb 11 2015 at 11:55:55 AM TJ Klein <TJ...@gmail.com> wrote:
>
>> Hi,
>>
>> Using Spark 1.2 I ran into issued setting SPARK_LOCAL_DIRS to a different
>> path then local directory.
>>
>> On our cluster we have a folder for temporary files (in a central file
>> system), which is called /scratch.
>>
>> When setting SPARK_LOCAL_DIRS=/scratch/<node name>
>>
>> I get:
>>
>> An error occurred while calling
>> z:org.apache.spark.api.python.PythonRDD.newAPIHadoopFile.
>> : org.apache.spark.SparkException: Job aborted due to stage failure:
>> Task 0
>> in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage
>> 0.0
>> (TID 3, XXXXXXX): java.io.IOException: Function not implemented
>> at sun.nio.ch.FileDispatcherImpl.lock0(Native Method)
>> at sun.nio.ch.FileDispatcherImpl.lock(FileDispatcherImpl.java:91)
>> at sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:1022)
>> at java.nio.channels.FileChannel.lock(FileChannel.java:1052)
>> at org.apache.spark.util.Utils$.fetchFile(Utils.scala:379)
>>
>> Using SPARK_LOCAL_DIRS=/tmp, however, works perfectly. Any idea?
>>
>> Best,
>> Tassilo
>>
>>
>>
>>
>>
>> --
>> View this message in context: http://apache-spark-user-list.
>> 1001560.n3.nabble.com/SPARK-LOCAL-DIRS-Issue-tp21602.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
Re: SPARK_LOCAL_DIRS Issue
Posted by Charles Feduke <ch...@gmail.com>.
A central location, such as NFS?
If they are temporary for the purpose of further job processing you'll want
to keep them local to the node in the cluster, i.e., in /tmp. If they are
centralized you won't be able to take advantage of data locality and the
central file store will become a bottleneck for further processing.
If /tmp isn't an option because you want to be able to monitor the file
outputs as they occur you can also use HDFS (assuming your Spark nodes are
also HDFS members they will benefit from data locality).
It looks like the problem you are seeing is that a lock cannot be acquired
on the output file in the central file system.
On Wed Feb 11 2015 at 11:55:55 AM TJ Klein <TJ...@gmail.com> wrote:
> Hi,
>
> Using Spark 1.2 I ran into issued setting SPARK_LOCAL_DIRS to a different
> path then local directory.
>
> On our cluster we have a folder for temporary files (in a central file
> system), which is called /scratch.
>
> When setting SPARK_LOCAL_DIRS=/scratch/<node name>
>
> I get:
>
> An error occurred while calling
> z:org.apache.spark.api.python.PythonRDD.newAPIHadoopFile.
> : org.apache.spark.SparkException: Job aborted due to stage failure: Task
> 0
> in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage
> 0.0
> (TID 3, XXXXXXX): java.io.IOException: Function not implemented
> at sun.nio.ch.FileDispatcherImpl.lock0(Native Method)
> at sun.nio.ch.FileDispatcherImpl.lock(FileDispatcherImpl.java:91)
> at sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:1022)
> at java.nio.channels.FileChannel.lock(FileChannel.java:1052)
> at org.apache.spark.util.Utils$.fetchFile(Utils.scala:379)
>
> Using SPARK_LOCAL_DIRS=/tmp, however, works perfectly. Any idea?
>
> Best,
> Tassilo
>
>
>
>
>
> --
> View this message in context: http://apache-spark-user-list.
> 1001560.n3.nabble.com/SPARK-LOCAL-DIRS-Issue-tp21602.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>