You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by andrem <am...@gmail.com> on 2015/04/02 17:23:35 UTC
Spark Streaming Worker runs out of inodes
Apparently Spark Streaming 1.3.0 is not cleaning up its internal files and
the worker nodes eventually run out of inodes.
We see tons of old shuffle_*.data and *.index files that are never deleted.
How do we get Spark to remove these files?
We have a simple standalone app with one RabbitMQ receiver and a two node
cluster (2 x r3large AWS instances).
Batch interval is 10 minutes after which we process data and write results
to DB. No windowing or state mgmt is used.
I've poured over the documentation and tried setting the following
properties but they have not helped.
As a work around we're using a cron script that periodically cleans up old
files but this has a bad smell to it.
SPARK_WORKER_OPTS in spark-env.sh on every worker node
spark.worker.cleanup.enabled true
spark.worker.cleanup.interval
spark.worker.cleanup.appDataTtl
Also tried on the driver side:
spark.cleaner.ttl
spark.shuffle.consolidateFiles true
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-Worker-runs-out-of-inodes-tp22355.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Spark Streaming Worker runs out of inodes
Posted by Charles Feduke <ch...@gmail.com>.
You could also try setting your `nofile` value in /etc/security/limits.conf
for `soft` to some ridiculously high value if you haven't done so already.
On Fri, Apr 3, 2015 at 2:09 AM Akhil Das <ak...@sigmoidanalytics.com> wrote:
> Did you try these?
>
> - Disable shuffle : spark.shuffle.spill=false
> - Enable log rotation:
>
> sparkConf.set("spark.executor.logs.rolling.strategy", "size")
> .set("spark.executor.logs.rolling.size.maxBytes", "1024")
> .set("spark.executor.logs.rolling.maxRetainedFiles", "3")
>
>
> Thanks
> Best Regards
>
> On Fri, Apr 3, 2015 at 9:09 AM, a mesar <am...@gmail.com> wrote:
>
>> Yes, with spark.cleaner.ttl set there is no cleanup. We pass --properties-file
>> spark-dev.conf to spark-submit where spark-dev.conf contains:
>>
>> spark.master spark://10.250.241.66:7077
>> spark.logConf true
>> spark.cleaner.ttl 1800
>> spark.executor.memory 10709m
>> spark.cores.max 4
>> spark.shuffle.consolidateFiles true
>>
>> On Thu, Apr 2, 2015 at 7:12 PM, Tathagata Das <td...@databricks.com>
>> wrote:
>>
>>> Are you saying that even with the spark.cleaner.ttl set your files are
>>> not getting cleaned up?
>>>
>>> TD
>>>
>>> On Thu, Apr 2, 2015 at 8:23 AM, andrem <am...@gmail.com> wrote:
>>>
>>>> Apparently Spark Streaming 1.3.0 is not cleaning up its internal files
>>>> and
>>>> the worker nodes eventually run out of inodes.
>>>> We see tons of old shuffle_*.data and *.index files that are never
>>>> deleted.
>>>> How do we get Spark to remove these files?
>>>>
>>>> We have a simple standalone app with one RabbitMQ receiver and a two
>>>> node
>>>> cluster (2 x r3large AWS instances).
>>>> Batch interval is 10 minutes after which we process data and write
>>>> results
>>>> to DB. No windowing or state mgmt is used.
>>>>
>>>> I've poured over the documentation and tried setting the following
>>>> properties but they have not helped.
>>>> As a work around we're using a cron script that periodically cleans up
>>>> old
>>>> files but this has a bad smell to it.
>>>>
>>>> SPARK_WORKER_OPTS in spark-env.sh on every worker node
>>>> spark.worker.cleanup.enabled true
>>>> spark.worker.cleanup.interval
>>>> spark.worker.cleanup.appDataTtl
>>>>
>>>> Also tried on the driver side:
>>>> spark.cleaner.ttl
>>>> spark.shuffle.consolidateFiles true
>>>>
>>>>
>>>>
>>>> --
>>>> View this message in context: http://apache-spark-user-list.
>>>> 1001560.n3.nabble.com/Spark-Streaming-Worker-runs-out-of-
>>>> inodes-tp22355.html
>>>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>>> For additional commands, e-mail: user-help@spark.apache.org
>>>>
>>>>
>>>
>>
>
Re: Spark Streaming Worker runs out of inodes
Posted by Akhil Das <ak...@sigmoidanalytics.com>.
Did you try these?
- Disable shuffle : spark.shuffle.spill=false
- Enable log rotation:
sparkConf.set("spark.executor.logs.rolling.strategy", "size")
.set("spark.executor.logs.rolling.size.maxBytes", "1024")
.set("spark.executor.logs.rolling.maxRetainedFiles", "3")
Thanks
Best Regards
On Fri, Apr 3, 2015 at 9:09 AM, a mesar <am...@gmail.com> wrote:
> Yes, with spark.cleaner.ttl set there is no cleanup. We pass --properties-file
> spark-dev.conf to spark-submit where spark-dev.conf contains:
>
> spark.master spark://10.250.241.66:7077
> spark.logConf true
> spark.cleaner.ttl 1800
> spark.executor.memory 10709m
> spark.cores.max 4
> spark.shuffle.consolidateFiles true
>
> On Thu, Apr 2, 2015 at 7:12 PM, Tathagata Das <td...@databricks.com> wrote:
>
>> Are you saying that even with the spark.cleaner.ttl set your files are
>> not getting cleaned up?
>>
>> TD
>>
>> On Thu, Apr 2, 2015 at 8:23 AM, andrem <am...@gmail.com> wrote:
>>
>>> Apparently Spark Streaming 1.3.0 is not cleaning up its internal files
>>> and
>>> the worker nodes eventually run out of inodes.
>>> We see tons of old shuffle_*.data and *.index files that are never
>>> deleted.
>>> How do we get Spark to remove these files?
>>>
>>> We have a simple standalone app with one RabbitMQ receiver and a two node
>>> cluster (2 x r3large AWS instances).
>>> Batch interval is 10 minutes after which we process data and write
>>> results
>>> to DB. No windowing or state mgmt is used.
>>>
>>> I've poured over the documentation and tried setting the following
>>> properties but they have not helped.
>>> As a work around we're using a cron script that periodically cleans up
>>> old
>>> files but this has a bad smell to it.
>>>
>>> SPARK_WORKER_OPTS in spark-env.sh on every worker node
>>> spark.worker.cleanup.enabled true
>>> spark.worker.cleanup.interval
>>> spark.worker.cleanup.appDataTtl
>>>
>>> Also tried on the driver side:
>>> spark.cleaner.ttl
>>> spark.shuffle.consolidateFiles true
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-Worker-runs-out-of-inodes-tp22355.html
>>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>> For additional commands, e-mail: user-help@spark.apache.org
>>>
>>>
>>
>
Re: Spark Streaming Worker runs out of inodes
Posted by a mesar <am...@gmail.com>.
Yes, with spark.cleaner.ttl set there is no cleanup. We pass --properties-file
spark-dev.conf to spark-submit where spark-dev.conf contains:
spark.master spark://10.250.241.66:7077
spark.logConf true
spark.cleaner.ttl 1800
spark.executor.memory 10709m
spark.cores.max 4
spark.shuffle.consolidateFiles true
On Thu, Apr 2, 2015 at 7:12 PM, Tathagata Das <td...@databricks.com> wrote:
> Are you saying that even with the spark.cleaner.ttl set your files are not
> getting cleaned up?
>
> TD
>
> On Thu, Apr 2, 2015 at 8:23 AM, andrem <am...@gmail.com> wrote:
>
>> Apparently Spark Streaming 1.3.0 is not cleaning up its internal files and
>> the worker nodes eventually run out of inodes.
>> We see tons of old shuffle_*.data and *.index files that are never
>> deleted.
>> How do we get Spark to remove these files?
>>
>> We have a simple standalone app with one RabbitMQ receiver and a two node
>> cluster (2 x r3large AWS instances).
>> Batch interval is 10 minutes after which we process data and write results
>> to DB. No windowing or state mgmt is used.
>>
>> I've poured over the documentation and tried setting the following
>> properties but they have not helped.
>> As a work around we're using a cron script that periodically cleans up old
>> files but this has a bad smell to it.
>>
>> SPARK_WORKER_OPTS in spark-env.sh on every worker node
>> spark.worker.cleanup.enabled true
>> spark.worker.cleanup.interval
>> spark.worker.cleanup.appDataTtl
>>
>> Also tried on the driver side:
>> spark.cleaner.ttl
>> spark.shuffle.consolidateFiles true
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-Worker-runs-out-of-inodes-tp22355.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
>
Re: Spark Streaming Worker runs out of inodes
Posted by Tathagata Das <td...@databricks.com>.
Are you saying that even with the spark.cleaner.ttl set your files are not
getting cleaned up?
TD
On Thu, Apr 2, 2015 at 8:23 AM, andrem <am...@gmail.com> wrote:
> Apparently Spark Streaming 1.3.0 is not cleaning up its internal files and
> the worker nodes eventually run out of inodes.
> We see tons of old shuffle_*.data and *.index files that are never deleted.
> How do we get Spark to remove these files?
>
> We have a simple standalone app with one RabbitMQ receiver and a two node
> cluster (2 x r3large AWS instances).
> Batch interval is 10 minutes after which we process data and write results
> to DB. No windowing or state mgmt is used.
>
> I've poured over the documentation and tried setting the following
> properties but they have not helped.
> As a work around we're using a cron script that periodically cleans up old
> files but this has a bad smell to it.
>
> SPARK_WORKER_OPTS in spark-env.sh on every worker node
> spark.worker.cleanup.enabled true
> spark.worker.cleanup.interval
> spark.worker.cleanup.appDataTtl
>
> Also tried on the driver side:
> spark.cleaner.ttl
> spark.shuffle.consolidateFiles true
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-Worker-runs-out-of-inodes-tp22355.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>