You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Georgios Samaras <ge...@gmail.com> on 2016/09/01 22:35:23 UTC
Is Spark's KMeans unable to handle bigdata?
Dear all,
the random initialization works well, but the default initialization is
k-means|| and has made me struggle. Also, I had heard people one year ago
struggling with it too, and everybody would just skip it and use random,
but I cannot keep it inside me!
I have posted a minimal example here
<http://stackoverflow.com/questions/39260820/is-sparks-kmeans-unable-to-handle-bigdata>
..
Please advice,
George Samaras
Re: Is Spark's KMeans unable to handle bigdata?
Posted by Georgios Samaras <ge...@gmail.com>.
Thank you very much Sean! If you would like, this could serve as an answer
in StackOverflow's question:
[Is Spark's kMeans unable to handle bigdata?](
http://stackoverflow.com/questions/39260820/is-sparks-kmeans-unable-to-handle-bigdata
).
Enjoy your weekend,
George
On Sat, Sep 3, 2016 at 1:22 AM, Sean Owen <so...@cloudera.com> wrote:
> I opened https://issues.apache.org/jira/browse/SPARK-17389 to track
> some improvements, but by far the big one is that the init steps
> defaults to 5, when the paper says that 2 is pretty much optimal here.
> It's much faster with that setting.
>
> On Fri, Sep 2, 2016 at 6:45 PM, Georgios Samaras
> <ge...@gmail.com> wrote:
> > I am not using the "runs" parameter anyway, but I see your point. If you
> > could point out any modifications in the minimal example I posted, I
> would
> > be more than interested to try them!
> >
>
Re: Is Spark's KMeans unable to handle bigdata?
Posted by Sean Owen <so...@cloudera.com>.
I opened https://issues.apache.org/jira/browse/SPARK-17389 to track
some improvements, but by far the big one is that the init steps
defaults to 5, when the paper says that 2 is pretty much optimal here.
It's much faster with that setting.
On Fri, Sep 2, 2016 at 6:45 PM, Georgios Samaras
<ge...@gmail.com> wrote:
> I am not using the "runs" parameter anyway, but I see your point. If you
> could point out any modifications in the minimal example I posted, I would
> be more than interested to try them!
>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Is Spark's KMeans unable to handle bigdata?
Posted by Georgios Samaras <ge...@gmail.com>.
I am not using the "runs" parameter anyway, but I see your point. If you
could point out any modifications in the minimal example I posted, I would
be more than interested to try them!
On Fri, Sep 2, 2016 at 10:43 AM, Sean Owen <so...@cloudera.com> wrote:
> Eh... more specifically, since Spark 2.0 the "runs" parameter in the
> KMeans mllib implementation has been ignored and is always 1. This
> means a lot of code that wraps this stuff up in arrays could be
> simplified quite a lot. I'll take a shot at optimizing this code and
> see if I can measure an effect.
>
> On Fri, Sep 2, 2016 at 6:33 PM, Sean Owen <so...@cloudera.com> wrote:
> > Yes it works fine, though each iteration of the parallel init step is
> > slow indeed -- about 5 minutes on my cluster. Given your question I
> > think you are actually 'hanging' because resources are being killed.
> >
> > I think this init may need some love and optimization. For example, I
> > think treeAggregate might work better. An Array[Float] may be just
> > fine and cut down memory usage, etc.
> >
> > On Fri, Sep 2, 2016 at 5:47 PM, Georgios Samaras
> > <ge...@gmail.com> wrote:
> >> So you were able to execute the minimal example I posted?
> >>
> >> I mean that the application doesn't progresses, it hangs (I would be OK
> if
> >> it was just slower). It doesn't seem to me a configuration issue.
> >>
> >> On Fri, Sep 2, 2016 at 1:07 AM, Sean Owen <so...@cloudera.com> wrote:
> >>>
> >>> Hm, what do you mean? k-means|| init is certainly slower because it's
> >>> making passes over the data in order to pick better initial centroids.
> >>> The idea is that you might then spend fewer iterations converging
> >>> later, and converge to a better clustering.
> >>>
> >>> Your problem doesn't seem to be related to scale. You aren't even
> >>> running out of memory it seems. Your memory settings are causing YARN
> >>> to kill the executors for using more memory than they advertise. That
> >>> could mean it never proceeds if this happens a lot.
> >>>
> >>> I don't have any problems with it.
> >>>
> >>> On Thu, Sep 1, 2016 at 11:35 PM, Georgios Samaras
> >>> <ge...@gmail.com> wrote:
> >>> > Dear all,
> >>> >
> >>> > the random initialization works well, but the default
> initialization
> >>> > is
> >>> > k-means|| and has made me struggle. Also, I had heard people one year
> >>> > ago
> >>> > struggling with it too, and everybody would just skip it and use
> random,
> >>> > but
> >>> > I cannot keep it inside me!
> >>> >
> >>> > I have posted a minimal example here..
> >>> >
> >>> > Please advice,
> >>> > George Samaras
> >>
> >>
>
Re: Is Spark's KMeans unable to handle bigdata?
Posted by Sean Owen <so...@cloudera.com>.
Eh... more specifically, since Spark 2.0 the "runs" parameter in the
KMeans mllib implementation has been ignored and is always 1. This
means a lot of code that wraps this stuff up in arrays could be
simplified quite a lot. I'll take a shot at optimizing this code and
see if I can measure an effect.
On Fri, Sep 2, 2016 at 6:33 PM, Sean Owen <so...@cloudera.com> wrote:
> Yes it works fine, though each iteration of the parallel init step is
> slow indeed -- about 5 minutes on my cluster. Given your question I
> think you are actually 'hanging' because resources are being killed.
>
> I think this init may need some love and optimization. For example, I
> think treeAggregate might work better. An Array[Float] may be just
> fine and cut down memory usage, etc.
>
> On Fri, Sep 2, 2016 at 5:47 PM, Georgios Samaras
> <ge...@gmail.com> wrote:
>> So you were able to execute the minimal example I posted?
>>
>> I mean that the application doesn't progresses, it hangs (I would be OK if
>> it was just slower). It doesn't seem to me a configuration issue.
>>
>> On Fri, Sep 2, 2016 at 1:07 AM, Sean Owen <so...@cloudera.com> wrote:
>>>
>>> Hm, what do you mean? k-means|| init is certainly slower because it's
>>> making passes over the data in order to pick better initial centroids.
>>> The idea is that you might then spend fewer iterations converging
>>> later, and converge to a better clustering.
>>>
>>> Your problem doesn't seem to be related to scale. You aren't even
>>> running out of memory it seems. Your memory settings are causing YARN
>>> to kill the executors for using more memory than they advertise. That
>>> could mean it never proceeds if this happens a lot.
>>>
>>> I don't have any problems with it.
>>>
>>> On Thu, Sep 1, 2016 at 11:35 PM, Georgios Samaras
>>> <ge...@gmail.com> wrote:
>>> > Dear all,
>>> >
>>> > the random initialization works well, but the default initialization
>>> > is
>>> > k-means|| and has made me struggle. Also, I had heard people one year
>>> > ago
>>> > struggling with it too, and everybody would just skip it and use random,
>>> > but
>>> > I cannot keep it inside me!
>>> >
>>> > I have posted a minimal example here..
>>> >
>>> > Please advice,
>>> > George Samaras
>>
>>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Is Spark's KMeans unable to handle bigdata?
Posted by Sean Owen <so...@cloudera.com>.
Yes it works fine, though each iteration of the parallel init step is
slow indeed -- about 5 minutes on my cluster. Given your question I
think you are actually 'hanging' because resources are being killed.
I think this init may need some love and optimization. For example, I
think treeAggregate might work better. An Array[Float] may be just
fine and cut down memory usage, etc.
On Fri, Sep 2, 2016 at 5:47 PM, Georgios Samaras
<ge...@gmail.com> wrote:
> So you were able to execute the minimal example I posted?
>
> I mean that the application doesn't progresses, it hangs (I would be OK if
> it was just slower). It doesn't seem to me a configuration issue.
>
> On Fri, Sep 2, 2016 at 1:07 AM, Sean Owen <so...@cloudera.com> wrote:
>>
>> Hm, what do you mean? k-means|| init is certainly slower because it's
>> making passes over the data in order to pick better initial centroids.
>> The idea is that you might then spend fewer iterations converging
>> later, and converge to a better clustering.
>>
>> Your problem doesn't seem to be related to scale. You aren't even
>> running out of memory it seems. Your memory settings are causing YARN
>> to kill the executors for using more memory than they advertise. That
>> could mean it never proceeds if this happens a lot.
>>
>> I don't have any problems with it.
>>
>> On Thu, Sep 1, 2016 at 11:35 PM, Georgios Samaras
>> <ge...@gmail.com> wrote:
>> > Dear all,
>> >
>> > the random initialization works well, but the default initialization
>> > is
>> > k-means|| and has made me struggle. Also, I had heard people one year
>> > ago
>> > struggling with it too, and everybody would just skip it and use random,
>> > but
>> > I cannot keep it inside me!
>> >
>> > I have posted a minimal example here..
>> >
>> > Please advice,
>> > George Samaras
>
>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Is Spark's KMeans unable to handle bigdata?
Posted by Georgios Samaras <ge...@gmail.com>.
So you were able to execute the minimal example I posted?
I mean that the application doesn't progresses, it hangs (I would be OK if
it was just slower). It doesn't seem to me a configuration issue.
On Fri, Sep 2, 2016 at 1:07 AM, Sean Owen <so...@cloudera.com> wrote:
> Hm, what do you mean? k-means|| init is certainly slower because it's
> making passes over the data in order to pick better initial centroids.
> The idea is that you might then spend fewer iterations converging
> later, and converge to a better clustering.
>
> Your problem doesn't seem to be related to scale. You aren't even
> running out of memory it seems. Your memory settings are causing YARN
> to kill the executors for using more memory than they advertise. That
> could mean it never proceeds if this happens a lot.
>
> I don't have any problems with it.
>
> On Thu, Sep 1, 2016 at 11:35 PM, Georgios Samaras
> <ge...@gmail.com> wrote:
> > Dear all,
> >
> > the random initialization works well, but the default initialization is
> > k-means|| and has made me struggle. Also, I had heard people one year ago
> > struggling with it too, and everybody would just skip it and use random,
> but
> > I cannot keep it inside me!
> >
> > I have posted a minimal example here..
> >
> > Please advice,
> > George Samaras
>
Re: Is Spark's KMeans unable to handle bigdata?
Posted by Sean Owen <so...@cloudera.com>.
Hm, what do you mean? k-means|| init is certainly slower because it's
making passes over the data in order to pick better initial centroids.
The idea is that you might then spend fewer iterations converging
later, and converge to a better clustering.
Your problem doesn't seem to be related to scale. You aren't even
running out of memory it seems. Your memory settings are causing YARN
to kill the executors for using more memory than they advertise. That
could mean it never proceeds if this happens a lot.
I don't have any problems with it.
On Thu, Sep 1, 2016 at 11:35 PM, Georgios Samaras
<ge...@gmail.com> wrote:
> Dear all,
>
> the random initialization works well, but the default initialization is
> k-means|| and has made me struggle. Also, I had heard people one year ago
> struggling with it too, and everybody would just skip it and use random, but
> I cannot keep it inside me!
>
> I have posted a minimal example here..
>
> Please advice,
> George Samaras
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: shuffle files not deleted after executor restarted
Posted by Artur Sukhenko <ar...@gmail.com>.
I believe in your case it will help, as executor's shuffle files will be
managed by external service.
It is described in spark docs: graceful-decommission-of-executors
<http://spark.apache.org/docs/latest/job-scheduling.html#graceful-decommission-of-executors>
Artur
On Fri, Sep 2, 2016 at 1:01 PM 汪洋 <ti...@icloud.com> wrote:
> 在 2016年9月2日，下午5:58，汪洋 <ti...@icloud.com> 写道：
>
> Yeah, using external shuffle service is a reasonable choice but I think we
> will still face the same problems. We use SSDs to store shuffle files for
> performance considerations. If the shuffle files are not going to be used
> anymore, we want them to be deleted instead of taking up valuable SSD space.
>
> Not very familiar with external shuffle service though. Is it going to
> help in this case? -:)
>
> 在 2016年9月2日，下午5:40，Artur Sukhenko <ar...@gmail.com> 写道：
>
> Hi Yang,
>
> Isn't external shuffle service better for long running applications?
> "It runs as a standalone application and manages shuffle output files so
> they are available for executors at all time"
>
> It is described here:
>
> https://jaceklaskowski.gitbooks.io/mastering-apache-spark/content/spark-ExternalShuffleService.html
>
> ---
> Artur
>
> On Fri, Sep 2, 2016 at 12:30 PM 汪洋 <ti...@icloud.com> wrote:
>
>> Thank you for you response.
>>
>> We are using spark-1.6.2 on standalone deploy mode with dynamic
>> allocation disabled.
>>
>> I have traced the code. IMHO, it seems this cleanup is not handled by
>> shutdown hooks directly. The shutdown hooks only send a
>> “ExecutorStateChanged” message to the worker and if the worker see the
>> message, it will cleanup the directory *only when this application is
>> finished*. In our case, the application is not finished (long running).
>> The executor exits due to some unknown error and it is restarted by worker
>> right away. In this scenario, those old directories are not going to be
>> deleted.
>>
>> If the application is still running, is it safe to delete the old
>> “blockmgr” directory and leaving only the newest one?
>>
>> Our temporary solution is to restart our application regularly and we are
>> seeking a more elegant way.
>>
>> Thanks.
>>
>> Yang
>>
>>
>> 在 2016年9月2日，下午4:11，Sun Rui <su...@163.com> 写道：
>>
>> Hi,
>> Could you give more information about your Spark environment? cluster
>> manager, spark version, using dynamic allocation or not, etc..
>>
>> Generally, executors will delete temporary directories for shuffle files
>> on exit because JVM shutdown hooks are registered. Unless they are brutally
>> killed.
>>
>> You can safely delete the directories when you are sure that the spark
>> applications related to them have finished. A crontab task may be used for
>> automatic clean up.
>>
>> On Sep 2, 2016, at 12:18, 汪洋 <ti...@icloud.com> wrote:
>>
>> Hi all,
>>
>> I discovered that sometimes executor exits unexpectedly and when it is
>> restarted, it will create another blockmgr directory without deleting the
>> old ones. Thus, for a long running application, some shuffle files will
>> never be cleaned up. Sometimes those files could take up the whole disk.
>>
>> Is there a way to clean up those unused file automatically? Or is it safe
>> to delete the old directory manually only leaving the newest one？
>>
>> Here is the executor’s local directory.
>> <D7718580-FF26-47F8-B6F8-00FB1F20A8C0.png>
>>
>> Any advice on this?
>>
>> Thanks.
>>
>> Yang
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
>>
>> --
> --
> Artur Sukhenko
>
>
> --
--
Artur Sukhenko
Re: shuffle files not deleted after executor restarted
Posted by 汪洋 <ti...@icloud.com>.
> 在 2016年9月2日，下午5:58，汪洋 <ti...@icloud.com> 写道：
>
> Yeah, using external shuffle service is a reasonable choice but I think we will still face the same problems. We use SSDs to store shuffle files for performance considerations. If the shuffle files are not going to be used anymore, we want them to be deleted instead of taking up valuable SSD space.
>
Not very familiar with external shuffle service though. Is it going to help in this case? -:)
>> 在 2016年9月2日，下午5:40，Artur Sukhenko <artur.sukhenko@gmail.com <ma...@gmail.com>> 写道：
>>
>> Hi Yang,
>>
>> Isn't external shuffle service better for long running applications?
>> "It runs as a standalone application and manages shuffle output files so they are available for executors at all time"
>>
>> It is described here:
>> https://jaceklaskowski.gitbooks.io/mastering-apache-spark/content/spark-ExternalShuffleService.html <https://jaceklaskowski.gitbooks.io/mastering-apache-spark/content/spark-ExternalShuffleService.html>
>>
>> ---
>> Artur
>>
>> On Fri, Sep 2, 2016 at 12:30 PM 汪洋 <tiandiwoxin@icloud.com <ma...@icloud.com>> wrote:
>> Thank you for you response.
>>
>> We are using spark-1.6.2 on standalone deploy mode with dynamic allocation disabled.
>>
>> I have traced the code. IMHO, it seems this cleanup is not handled by shutdown hooks directly. The shutdown hooks only send a “ExecutorStateChanged” message to the worker and if the worker see the message, it will cleanup the directory only when this application is finished. In our case, the application is not finished (long running). The executor exits due to some unknown error and it is restarted by worker right away. In this scenario, those old directories are not going to be deleted.
>>
>> If the application is still running, is it safe to delete the old “blockmgr” directory and leaving only the newest one?
>>
>> Our temporary solution is to restart our application regularly and we are seeking a more elegant way.
>>
>> Thanks.
>>
>> Yang
>>
>>
>>> 在 2016年9月2日，下午4:11，Sun Rui <sunrise_win@163.com <ma...@163.com>> 写道：
>>>
>>> Hi,
>>> Could you give more information about your Spark environment? cluster manager, spark version, using dynamic allocation or not, etc..
>>>
>>> Generally, executors will delete temporary directories for shuffle files on exit because JVM shutdown hooks are registered. Unless they are brutally killed.
>>>
>>> You can safely delete the directories when you are sure that the spark applications related to them have finished. A crontab task may be used for automatic clean up.
>>>
>>>> On Sep 2, 2016, at 12:18, 汪洋 <tiandiwoxin@icloud.com <ma...@icloud.com>> wrote:
>>>>
>>>> Hi all,
>>>>
>>>> I discovered that sometimes executor exits unexpectedly and when it is restarted, it will create another blockmgr directory without deleting the old ones. Thus, for a long running application, some shuffle files will never be cleaned up. Sometimes those files could take up the whole disk.
>>>>
>>>> Is there a way to clean up those unused file automatically? Or is it safe to delete the old directory manually only leaving the newest one？
>>>>
>>>> Here is the executor’s local directory.
>>>> <D7718580-FF26-47F8-B6F8-00FB1F20A8C0.png>
>>>>
>>>> Any advice on this?
>>>>
>>>> Thanks.
>>>>
>>>> Yang
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org <ma...@spark.apache.org>
>>>
>>
>> --
>> --
>> Artur Sukhenko
>
Re: shuffle files not deleted after executor restarted
Posted by 汪洋 <ti...@icloud.com>.
Yeah, using external shuffle service is a reasonable choice but I think we will still face the same problems. We use SSDs to store shuffle files for performance considerations. If the shuffle files are not going to be used anymore, we want them to be deleted instead of taking up valuable SSD space.
> 在 2016年9月2日，下午5:40，Artur Sukhenko <ar...@gmail.com> 写道：
>
> Hi Yang,
>
> Isn't external shuffle service better for long running applications?
> "It runs as a standalone application and manages shuffle output files so they are available for executors at all time"
>
> It is described here:
> https://jaceklaskowski.gitbooks.io/mastering-apache-spark/content/spark-ExternalShuffleService.html <https://jaceklaskowski.gitbooks.io/mastering-apache-spark/content/spark-ExternalShuffleService.html>
>
> ---
> Artur
>
> On Fri, Sep 2, 2016 at 12:30 PM 汪洋 <tiandiwoxin@icloud.com <ma...@icloud.com>> wrote:
> Thank you for you response.
>
> We are using spark-1.6.2 on standalone deploy mode with dynamic allocation disabled.
>
> I have traced the code. IMHO, it seems this cleanup is not handled by shutdown hooks directly. The shutdown hooks only send a “ExecutorStateChanged” message to the worker and if the worker see the message, it will cleanup the directory only when this application is finished. In our case, the application is not finished (long running). The executor exits due to some unknown error and it is restarted by worker right away. In this scenario, those old directories are not going to be deleted.
>
> If the application is still running, is it safe to delete the old “blockmgr” directory and leaving only the newest one?
>
> Our temporary solution is to restart our application regularly and we are seeking a more elegant way.
>
> Thanks.
>
> Yang
>
>
>> 在 2016年9月2日，下午4:11，Sun Rui <sunrise_win@163.com <ma...@163.com>> 写道：
>>
>> Hi,
>> Could you give more information about your Spark environment? cluster manager, spark version, using dynamic allocation or not, etc..
>>
>> Generally, executors will delete temporary directories for shuffle files on exit because JVM shutdown hooks are registered. Unless they are brutally killed.
>>
>> You can safely delete the directories when you are sure that the spark applications related to them have finished. A crontab task may be used for automatic clean up.
>>
>>> On Sep 2, 2016, at 12:18, 汪洋 <tiandiwoxin@icloud.com <ma...@icloud.com>> wrote:
>>>
>>> Hi all,
>>>
>>> I discovered that sometimes executor exits unexpectedly and when it is restarted, it will create another blockmgr directory without deleting the old ones. Thus, for a long running application, some shuffle files will never be cleaned up. Sometimes those files could take up the whole disk.
>>>
>>> Is there a way to clean up those unused file automatically? Or is it safe to delete the old directory manually only leaving the newest one？
>>>
>>> Here is the executor’s local directory.
>>> <D7718580-FF26-47F8-B6F8-00FB1F20A8C0.png>
>>>
>>> Any advice on this?
>>>
>>> Thanks.
>>>
>>> Yang
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org <ma...@spark.apache.org>
>>
>
> --
> --
> Artur Sukhenko
Re: shuffle files not deleted after executor restarted
Posted by Artur Sukhenko <ar...@gmail.com>.
Hi Yang,
Isn't external shuffle service better for long running applications?
"It runs as a standalone application and manages shuffle output files so
they are available for executors at all time"
It is described here:
https://jaceklaskowski.gitbooks.io/mastering-apache-spark/content/spark-ExternalShuffleService.html
---
Artur
On Fri, Sep 2, 2016 at 12:30 PM 汪洋 <ti...@icloud.com> wrote:
> Thank you for you response.
>
> We are using spark-1.6.2 on standalone deploy mode with dynamic allocation
> disabled.
>
> I have traced the code. IMHO, it seems this cleanup is not handled by
> shutdown hooks directly. The shutdown hooks only send a
> “ExecutorStateChanged” message to the worker and if the worker see the
> message, it will cleanup the directory *only when this application is
> finished*. In our case, the application is not finished (long running).
> The executor exits due to some unknown error and it is restarted by worker
> right away. In this scenario, those old directories are not going to be
> deleted.
>
> If the application is still running, is it safe to delete the old
> “blockmgr” directory and leaving only the newest one?
>
> Our temporary solution is to restart our application regularly and we are
> seeking a more elegant way.
>
> Thanks.
>
> Yang
>
>
> 在 2016年9月2日，下午4:11，Sun Rui <su...@163.com> 写道：
>
> Hi,
> Could you give more information about your Spark environment? cluster
> manager, spark version, using dynamic allocation or not, etc..
>
> Generally, executors will delete temporary directories for shuffle files
> on exit because JVM shutdown hooks are registered. Unless they are brutally
> killed.
>
> You can safely delete the directories when you are sure that the spark
> applications related to them have finished. A crontab task may be used for
> automatic clean up.
>
> On Sep 2, 2016, at 12:18, 汪洋 <ti...@icloud.com> wrote:
>
> Hi all,
>
> I discovered that sometimes executor exits unexpectedly and when it is
> restarted, it will create another blockmgr directory without deleting the
> old ones. Thus, for a long running application, some shuffle files will
> never be cleaned up. Sometimes those files could take up the whole disk.
>
> Is there a way to clean up those unused file automatically? Or is it safe
> to delete the old directory manually only leaving the newest one？
>
> Here is the executor’s local directory.
> <D7718580-FF26-47F8-B6F8-00FB1F20A8C0.png>
>
> Any advice on this?
>
> Thanks.
>
> Yang
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
> --
--
Artur Sukhenko
Re: shuffle files not deleted after executor restarted
Posted by 汪洋 <ti...@icloud.com>.
Thank you for you response.
We are using spark-1.6.2 on standalone deploy mode with dynamic allocation disabled.
I have traced the code. IMHO, it seems this cleanup is not handled by shutdown hooks directly. The shutdown hooks only send a “ExecutorStateChanged” message to the worker and if the worker see the message, it will cleanup the directory only when this application is finished. In our case, the application is not finished (long running). The executor exits due to some unknown error and it is restarted by worker right away. In this scenario, those old directories are not going to be deleted.
If the application is still running, is it safe to delete the old “blockmgr” directory and leaving only the newest one?
Our temporary solution is to restart our application regularly and we are seeking a more elegant way.
Thanks.
Yang
> 在 2016年9月2日，下午4:11，Sun Rui <su...@163.com> 写道：
>
> Hi,
> Could you give more information about your Spark environment? cluster manager, spark version, using dynamic allocation or not, etc..
>
> Generally, executors will delete temporary directories for shuffle files on exit because JVM shutdown hooks are registered. Unless they are brutally killed.
>
> You can safely delete the directories when you are sure that the spark applications related to them have finished. A crontab task may be used for automatic clean up.
>
>> On Sep 2, 2016, at 12:18, 汪洋 <ti...@icloud.com> wrote:
>>
>> Hi all,
>>
>> I discovered that sometimes executor exits unexpectedly and when it is restarted, it will create another blockmgr directory without deleting the old ones. Thus, for a long running application, some shuffle files will never be cleaned up. Sometimes those files could take up the whole disk.
>>
>> Is there a way to clean up those unused file automatically? Or is it safe to delete the old directory manually only leaving the newest one？
>>
>> Here is the executor’s local directory.
>> <D7718580-FF26-47F8-B6F8-00FB1F20A8C0.png>
>>
>> Any advice on this?
>>
>> Thanks.
>>
>> Yang
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
Re: shuffle files not deleted after executor restarted
Posted by Sun Rui <su...@163.com>.
Hi,
Could you give more information about your Spark environment? cluster manager, spark version, using dynamic allocation or not, etc..
Generally, executors will delete temporary directories for shuffle files on exit because JVM shutdown hooks are registered. Unless they are brutally killed.
You can safely delete the directories when you are sure that the spark applications related to them have finished. A crontab task may be used for automatic clean up.
> On Sep 2, 2016, at 12:18, 汪洋 <ti...@icloud.com> wrote:
>
> Hi all,
>
> I discovered that sometimes executor exits unexpectedly and when it is restarted, it will create another blockmgr directory without deleting the old ones. Thus, for a long running application, some shuffle files will never be cleaned up. Sometimes those files could take up the whole disk.
>
> Is there a way to clean up those unused file automatically? Or is it safe to delete the old directory manually only leaving the newest one？
>
> Here is the executor’s local directory.
> <D7718580-FF26-47F8-B6F8-00FB1F20A8C0.png>
>
> Any advice on this?
>
> Thanks.
>
> Yang
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
shuffle files not deleted after executor restarted
Posted by 汪洋 <ti...@icloud.com>.
Hi all,
I discovered that sometimes executor exits unexpectedly and when it is restarted, it will create another blockmgr directory without deleting the old ones. Thus, for a long running application, some shuffle files will never be cleaned up. Sometimes those files could take up the whole disk.
Is there a way to clean up those unused file automatically? Or is it safe to delete the old directory manually only leaving the newest one？
Here is the executor’s local directory.
Any advice on this?
Thanks.
Yang