You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Lukas Kairies <lu...@googlemail.com> on 2014/01/21 11:36:34 UTC
Shutdown hook for FileSystems
Hey,
I use Hadoop with XtreemFS (with a corresponding FileSystem
implementation). The XtreemFS client uses several non-deamon Threads eg.
for communication. Therefore the shutdown hooks do not start after a
mapper/reducer is finished and the Child processes do not terminate.
My question:
Is there any way to determine if the mapper/reducer child processes do
not need the file system any more?
Best regards,
Lukas K
Re: Shutdown hook for FileSystems
Posted by Oleg Zhurakousky <ol...@gmail.com>.
Sure it does. Your application most certainly has a starting point (some
Main), which starts-up everything else. So what you need to do is create
your own shutdown hook during the startup of the application so it could
catch Ctrl+C kind of termination and then within such shutdown hook
initiate a graceful shutdown.
Now I don't know how you create threads. I would certainly hope you are
using java.util.concurrent package and if so ExecutorService has few
shutdown(..) methods. This means you can call it inside of your Main's
shutdown hook. That of course would be a very, very bad idea since it would
require you to expose all executors to the Main. But Main obviously starts
something (a process) which means it has reference to it and such process
most definitely has reference to its supporting classes and so on. So what
I would recommend is for the Main's process to have a stop() method which
could be invoked in the Main's shutdown hook and such process can propagate
its own invocation of stop() to other "stoppable" components by invoking
their stop() methods and so on, thus achieving a graceful shutdown without
escaping your Executors, thus not breaking any encapsulation rules
Cheers
Oleg.
On Wed, Jan 22, 2014 at 12:37 PM, Lukas Kairies <
lukas.xtreemfs@googlemail.com> wrote:
> Thanks for your answer, but I do not see how this could work in my case.
> My problem is that XtreemFS uses Threads (RPC clients) for method calls and
> therefore I have to stop this threads by myself to invoke the shutdown
> hooks. I think replacing one shutdown hook with another does not work in my
> case.My idea was to determine if the file system can be closed, because the
> mapper/reducer is done instead of waiting for the shutdown hook. So is this
> possible? E.g. getting the status of the task from somewhere an checking if
> it is finished?
>
> Am 22.01.2014 01:38, schrieb Oleg Zhurakousky:
>
> I am not sure either, you have to ask Hadoop guys, but it was giving me a
> hard time so I found a way around it.
>
>
> On Tue, Jan 21, 2014 at 6:05 PM, Jay Vyas <ja...@gmail.com> wrote:
>
>> I guess im not sure what the ShutdownHook actually is there for.... Thats
>> the real question im asking .
>>
>>
>> On Tue, Jan 21, 2014 at 5:58 PM, Oleg Zhurakousky <
>> oleg.zhurakousky@gmail.com> wrote:
>>
>>> No, all I do is have my own shutdown hook in the main which closes the
>>> FSDataOutputStream. Before I did that it would throw an ugly exception when
>>> I hit Ctrl+C, telling me that the stream is already closed, because of this
>>> shutdown hook (bad design on the hadoop part), so removing it keeps it open
>>> until the client's shutdown hook will close it at due time. So now it shuts
>>> down gracefully and under complete control of the client.
>>>
>>>
>>> On Tue, Jan 21, 2014 at 12:30 PM, Jay Vyas <ja...@gmail.com> wrote:
>>>
>>>> what is happening when you remove the shutdown hook ? is that
>>>> supposed to trigger an exception -
>>>>
>>>
>>>
>>
>>
>> --
>> Jay Vyas
>> http://jayunit100.blogspot.com
>>
>
>
>
Re: Shutdown hook for FileSystems
Posted by Oleg Zhurakousky <ol...@gmail.com>.
Sure it does. Your application most certainly has a starting point (some
Main), which starts-up everything else. So what you need to do is create
your own shutdown hook during the startup of the application so it could
catch Ctrl+C kind of termination and then within such shutdown hook
initiate a graceful shutdown.
Now I don't know how you create threads. I would certainly hope you are
using java.util.concurrent package and if so ExecutorService has few
shutdown(..) methods. This means you can call it inside of your Main's
shutdown hook. That of course would be a very, very bad idea since it would
require you to expose all executors to the Main. But Main obviously starts
something (a process) which means it has reference to it and such process
most definitely has reference to its supporting classes and so on. So what
I would recommend is for the Main's process to have a stop() method which
could be invoked in the Main's shutdown hook and such process can propagate
its own invocation of stop() to other "stoppable" components by invoking
their stop() methods and so on, thus achieving a graceful shutdown without
escaping your Executors, thus not breaking any encapsulation rules
Cheers
Oleg.
On Wed, Jan 22, 2014 at 12:37 PM, Lukas Kairies <
lukas.xtreemfs@googlemail.com> wrote:
> Thanks for your answer, but I do not see how this could work in my case.
> My problem is that XtreemFS uses Threads (RPC clients) for method calls and
> therefore I have to stop this threads by myself to invoke the shutdown
> hooks. I think replacing one shutdown hook with another does not work in my
> case.My idea was to determine if the file system can be closed, because the
> mapper/reducer is done instead of waiting for the shutdown hook. So is this
> possible? E.g. getting the status of the task from somewhere an checking if
> it is finished?
>
> Am 22.01.2014 01:38, schrieb Oleg Zhurakousky:
>
> I am not sure either, you have to ask Hadoop guys, but it was giving me a
> hard time so I found a way around it.
>
>
> On Tue, Jan 21, 2014 at 6:05 PM, Jay Vyas <ja...@gmail.com> wrote:
>
>> I guess im not sure what the ShutdownHook actually is there for.... Thats
>> the real question im asking .
>>
>>
>> On Tue, Jan 21, 2014 at 5:58 PM, Oleg Zhurakousky <
>> oleg.zhurakousky@gmail.com> wrote:
>>
>>> No, all I do is have my own shutdown hook in the main which closes the
>>> FSDataOutputStream. Before I did that it would throw an ugly exception when
>>> I hit Ctrl+C, telling me that the stream is already closed, because of this
>>> shutdown hook (bad design on the hadoop part), so removing it keeps it open
>>> until the client's shutdown hook will close it at due time. So now it shuts
>>> down gracefully and under complete control of the client.
>>>
>>>
>>> On Tue, Jan 21, 2014 at 12:30 PM, Jay Vyas <ja...@gmail.com> wrote:
>>>
>>>> what is happening when you remove the shutdown hook ? is that
>>>> supposed to trigger an exception -
>>>>
>>>
>>>
>>
>>
>> --
>> Jay Vyas
>> http://jayunit100.blogspot.com
>>
>
>
>
Re: Shutdown hook for FileSystems
Posted by Oleg Zhurakousky <ol...@gmail.com>.
Sure it does. Your application most certainly has a starting point (some
Main), which starts-up everything else. So what you need to do is create
your own shutdown hook during the startup of the application so it could
catch Ctrl+C kind of termination and then within such shutdown hook
initiate a graceful shutdown.
Now I don't know how you create threads. I would certainly hope you are
using java.util.concurrent package and if so ExecutorService has few
shutdown(..) methods. This means you can call it inside of your Main's
shutdown hook. That of course would be a very, very bad idea since it would
require you to expose all executors to the Main. But Main obviously starts
something (a process) which means it has reference to it and such process
most definitely has reference to its supporting classes and so on. So what
I would recommend is for the Main's process to have a stop() method which
could be invoked in the Main's shutdown hook and such process can propagate
its own invocation of stop() to other "stoppable" components by invoking
their stop() methods and so on, thus achieving a graceful shutdown without
escaping your Executors, thus not breaking any encapsulation rules
Cheers
Oleg.
On Wed, Jan 22, 2014 at 12:37 PM, Lukas Kairies <
lukas.xtreemfs@googlemail.com> wrote:
> Thanks for your answer, but I do not see how this could work in my case.
> My problem is that XtreemFS uses Threads (RPC clients) for method calls and
> therefore I have to stop this threads by myself to invoke the shutdown
> hooks. I think replacing one shutdown hook with another does not work in my
> case.My idea was to determine if the file system can be closed, because the
> mapper/reducer is done instead of waiting for the shutdown hook. So is this
> possible? E.g. getting the status of the task from somewhere an checking if
> it is finished?
>
> Am 22.01.2014 01:38, schrieb Oleg Zhurakousky:
>
> I am not sure either, you have to ask Hadoop guys, but it was giving me a
> hard time so I found a way around it.
>
>
> On Tue, Jan 21, 2014 at 6:05 PM, Jay Vyas <ja...@gmail.com> wrote:
>
>> I guess im not sure what the ShutdownHook actually is there for.... Thats
>> the real question im asking .
>>
>>
>> On Tue, Jan 21, 2014 at 5:58 PM, Oleg Zhurakousky <
>> oleg.zhurakousky@gmail.com> wrote:
>>
>>> No, all I do is have my own shutdown hook in the main which closes the
>>> FSDataOutputStream. Before I did that it would throw an ugly exception when
>>> I hit Ctrl+C, telling me that the stream is already closed, because of this
>>> shutdown hook (bad design on the hadoop part), so removing it keeps it open
>>> until the client's shutdown hook will close it at due time. So now it shuts
>>> down gracefully and under complete control of the client.
>>>
>>>
>>> On Tue, Jan 21, 2014 at 12:30 PM, Jay Vyas <ja...@gmail.com> wrote:
>>>
>>>> what is happening when you remove the shutdown hook ? is that
>>>> supposed to trigger an exception -
>>>>
>>>
>>>
>>
>>
>> --
>> Jay Vyas
>> http://jayunit100.blogspot.com
>>
>
>
>
Re: Shutdown hook for FileSystems
Posted by Oleg Zhurakousky <ol...@gmail.com>.
Sure it does. Your application most certainly has a starting point (some
Main), which starts-up everything else. So what you need to do is create
your own shutdown hook during the startup of the application so it could
catch Ctrl+C kind of termination and then within such shutdown hook
initiate a graceful shutdown.
Now I don't know how you create threads. I would certainly hope you are
using java.util.concurrent package and if so ExecutorService has few
shutdown(..) methods. This means you can call it inside of your Main's
shutdown hook. That of course would be a very, very bad idea since it would
require you to expose all executors to the Main. But Main obviously starts
something (a process) which means it has reference to it and such process
most definitely has reference to its supporting classes and so on. So what
I would recommend is for the Main's process to have a stop() method which
could be invoked in the Main's shutdown hook and such process can propagate
its own invocation of stop() to other "stoppable" components by invoking
their stop() methods and so on, thus achieving a graceful shutdown without
escaping your Executors, thus not breaking any encapsulation rules
Cheers
Oleg.
On Wed, Jan 22, 2014 at 12:37 PM, Lukas Kairies <
lukas.xtreemfs@googlemail.com> wrote:
> Thanks for your answer, but I do not see how this could work in my case.
> My problem is that XtreemFS uses Threads (RPC clients) for method calls and
> therefore I have to stop this threads by myself to invoke the shutdown
> hooks. I think replacing one shutdown hook with another does not work in my
> case.My idea was to determine if the file system can be closed, because the
> mapper/reducer is done instead of waiting for the shutdown hook. So is this
> possible? E.g. getting the status of the task from somewhere an checking if
> it is finished?
>
> Am 22.01.2014 01:38, schrieb Oleg Zhurakousky:
>
> I am not sure either, you have to ask Hadoop guys, but it was giving me a
> hard time so I found a way around it.
>
>
> On Tue, Jan 21, 2014 at 6:05 PM, Jay Vyas <ja...@gmail.com> wrote:
>
>> I guess im not sure what the ShutdownHook actually is there for.... Thats
>> the real question im asking .
>>
>>
>> On Tue, Jan 21, 2014 at 5:58 PM, Oleg Zhurakousky <
>> oleg.zhurakousky@gmail.com> wrote:
>>
>>> No, all I do is have my own shutdown hook in the main which closes the
>>> FSDataOutputStream. Before I did that it would throw an ugly exception when
>>> I hit Ctrl+C, telling me that the stream is already closed, because of this
>>> shutdown hook (bad design on the hadoop part), so removing it keeps it open
>>> until the client's shutdown hook will close it at due time. So now it shuts
>>> down gracefully and under complete control of the client.
>>>
>>>
>>> On Tue, Jan 21, 2014 at 12:30 PM, Jay Vyas <ja...@gmail.com> wrote:
>>>
>>>> what is happening when you remove the shutdown hook ? is that
>>>> supposed to trigger an exception -
>>>>
>>>
>>>
>>
>>
>> --
>> Jay Vyas
>> http://jayunit100.blogspot.com
>>
>
>
>
Re: Shutdown hook for FileSystems
Posted by Lukas Kairies <lu...@googlemail.com>.
Thanks for your answer, but I do not see how this could work in my case.
My problem is that XtreemFS uses Threads (RPC clients) for method calls
and therefore I have to stop this threads by myself to invoke the
shutdown hooks. I think replacing one shutdown hook with another does
not work in my case.My idea was to determine if the file system can be
closed, because the mapper/reducer is done instead of waiting for the
shutdown hook. So is this possible? E.g. getting the status of the task
from somewhere an checking if it is finished?
Am 22.01.2014 01:38, schrieb Oleg Zhurakousky:
> I am not sure either, you have to ask Hadoop guys, but it was giving
> me a hard time so I found a way around it.
>
>
> On Tue, Jan 21, 2014 at 6:05 PM, Jay Vyas <jayunit100@gmail.com
> <ma...@gmail.com>> wrote:
>
> I guess im not sure what the ShutdownHook actually is there
> for.... Thats the real question im asking .
>
>
> On Tue, Jan 21, 2014 at 5:58 PM, Oleg Zhurakousky
> <oleg.zhurakousky@gmail.com <ma...@gmail.com>>
> wrote:
>
> No, all I do is have my own shutdown hook in the main which
> closes the FSDataOutputStream. Before I did that it would
> throw an ugly exception when I hit Ctrl+C, telling me that the
> stream is already closed, because of this shutdown hook (bad
> design on the hadoop part), so removing it keeps it open until
> the client's shutdown hook will close it at due time. So now
> it shuts down gracefully and under complete control of the
> client.
>
>
> On Tue, Jan 21, 2014 at 12:30 PM, Jay Vyas
> <jayunit100@gmail.com <ma...@gmail.com>> wrote:
>
> what is happening when you remove the shutdown hook ?
> is that supposed to trigger an exception -
>
>
>
>
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
>
>
Re: Shutdown hook for FileSystems
Posted by Lukas Kairies <lu...@googlemail.com>.
Thanks for your answer, but I do not see how this could work in my case.
My problem is that XtreemFS uses Threads (RPC clients) for method calls
and therefore I have to stop this threads by myself to invoke the
shutdown hooks. I think replacing one shutdown hook with another does
not work in my case.My idea was to determine if the file system can be
closed, because the mapper/reducer is done instead of waiting for the
shutdown hook. So is this possible? E.g. getting the status of the task
from somewhere an checking if it is finished?
Am 22.01.2014 01:38, schrieb Oleg Zhurakousky:
> I am not sure either, you have to ask Hadoop guys, but it was giving
> me a hard time so I found a way around it.
>
>
> On Tue, Jan 21, 2014 at 6:05 PM, Jay Vyas <jayunit100@gmail.com
> <ma...@gmail.com>> wrote:
>
> I guess im not sure what the ShutdownHook actually is there
> for.... Thats the real question im asking .
>
>
> On Tue, Jan 21, 2014 at 5:58 PM, Oleg Zhurakousky
> <oleg.zhurakousky@gmail.com <ma...@gmail.com>>
> wrote:
>
> No, all I do is have my own shutdown hook in the main which
> closes the FSDataOutputStream. Before I did that it would
> throw an ugly exception when I hit Ctrl+C, telling me that the
> stream is already closed, because of this shutdown hook (bad
> design on the hadoop part), so removing it keeps it open until
> the client's shutdown hook will close it at due time. So now
> it shuts down gracefully and under complete control of the
> client.
>
>
> On Tue, Jan 21, 2014 at 12:30 PM, Jay Vyas
> <jayunit100@gmail.com <ma...@gmail.com>> wrote:
>
> what is happening when you remove the shutdown hook ?
> is that supposed to trigger an exception -
>
>
>
>
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
>
>
Re: Shutdown hook for FileSystems
Posted by Lukas Kairies <lu...@googlemail.com>.
Thanks for your answer, but I do not see how this could work in my case.
My problem is that XtreemFS uses Threads (RPC clients) for method calls
and therefore I have to stop this threads by myself to invoke the
shutdown hooks. I think replacing one shutdown hook with another does
not work in my case.My idea was to determine if the file system can be
closed, because the mapper/reducer is done instead of waiting for the
shutdown hook. So is this possible? E.g. getting the status of the task
from somewhere an checking if it is finished?
Am 22.01.2014 01:38, schrieb Oleg Zhurakousky:
> I am not sure either, you have to ask Hadoop guys, but it was giving
> me a hard time so I found a way around it.
>
>
> On Tue, Jan 21, 2014 at 6:05 PM, Jay Vyas <jayunit100@gmail.com
> <ma...@gmail.com>> wrote:
>
> I guess im not sure what the ShutdownHook actually is there
> for.... Thats the real question im asking .
>
>
> On Tue, Jan 21, 2014 at 5:58 PM, Oleg Zhurakousky
> <oleg.zhurakousky@gmail.com <ma...@gmail.com>>
> wrote:
>
> No, all I do is have my own shutdown hook in the main which
> closes the FSDataOutputStream. Before I did that it would
> throw an ugly exception when I hit Ctrl+C, telling me that the
> stream is already closed, because of this shutdown hook (bad
> design on the hadoop part), so removing it keeps it open until
> the client's shutdown hook will close it at due time. So now
> it shuts down gracefully and under complete control of the
> client.
>
>
> On Tue, Jan 21, 2014 at 12:30 PM, Jay Vyas
> <jayunit100@gmail.com <ma...@gmail.com>> wrote:
>
> what is happening when you remove the shutdown hook ?
> is that supposed to trigger an exception -
>
>
>
>
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
>
>
Re: Shutdown hook for FileSystems
Posted by Lukas Kairies <lu...@googlemail.com>.
Thanks for your answer, but I do not see how this could work in my case.
My problem is that XtreemFS uses Threads (RPC clients) for method calls
and therefore I have to stop this threads by myself to invoke the
shutdown hooks. I think replacing one shutdown hook with another does
not work in my case.My idea was to determine if the file system can be
closed, because the mapper/reducer is done instead of waiting for the
shutdown hook. So is this possible? E.g. getting the status of the task
from somewhere an checking if it is finished?
Am 22.01.2014 01:38, schrieb Oleg Zhurakousky:
> I am not sure either, you have to ask Hadoop guys, but it was giving
> me a hard time so I found a way around it.
>
>
> On Tue, Jan 21, 2014 at 6:05 PM, Jay Vyas <jayunit100@gmail.com
> <ma...@gmail.com>> wrote:
>
> I guess im not sure what the ShutdownHook actually is there
> for.... Thats the real question im asking .
>
>
> On Tue, Jan 21, 2014 at 5:58 PM, Oleg Zhurakousky
> <oleg.zhurakousky@gmail.com <ma...@gmail.com>>
> wrote:
>
> No, all I do is have my own shutdown hook in the main which
> closes the FSDataOutputStream. Before I did that it would
> throw an ugly exception when I hit Ctrl+C, telling me that the
> stream is already closed, because of this shutdown hook (bad
> design on the hadoop part), so removing it keeps it open until
> the client's shutdown hook will close it at due time. So now
> it shuts down gracefully and under complete control of the
> client.
>
>
> On Tue, Jan 21, 2014 at 12:30 PM, Jay Vyas
> <jayunit100@gmail.com <ma...@gmail.com>> wrote:
>
> what is happening when you remove the shutdown hook ?
> is that supposed to trigger an exception -
>
>
>
>
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
>
>
Re: Shutdown hook for FileSystems
Posted by Oleg Zhurakousky <ol...@gmail.com>.
I am not sure either, you have to ask Hadoop guys, but it was giving me a
hard time so I found a way around it.
On Tue, Jan 21, 2014 at 6:05 PM, Jay Vyas <ja...@gmail.com> wrote:
> I guess im not sure what the ShutdownHook actually is there for.... Thats
> the real question im asking .
>
>
> On Tue, Jan 21, 2014 at 5:58 PM, Oleg Zhurakousky <
> oleg.zhurakousky@gmail.com> wrote:
>
>> No, all I do is have my own shutdown hook in the main which closes the
>> FSDataOutputStream. Before I did that it would throw an ugly exception when
>> I hit Ctrl+C, telling me that the stream is already closed, because of this
>> shutdown hook (bad design on the hadoop part), so removing it keeps it open
>> until the client's shutdown hook will close it at due time. So now it shuts
>> down gracefully and under complete control of the client.
>>
>>
>> On Tue, Jan 21, 2014 at 12:30 PM, Jay Vyas <ja...@gmail.com> wrote:
>>
>>> what is happening when you remove the shutdown hook ? is that
>>> supposed to trigger an exception -
>>>
>>
>>
>
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
>
Re: Shutdown hook for FileSystems
Posted by Oleg Zhurakousky <ol...@gmail.com>.
I am not sure either, you have to ask Hadoop guys, but it was giving me a
hard time so I found a way around it.
On Tue, Jan 21, 2014 at 6:05 PM, Jay Vyas <ja...@gmail.com> wrote:
> I guess im not sure what the ShutdownHook actually is there for.... Thats
> the real question im asking .
>
>
> On Tue, Jan 21, 2014 at 5:58 PM, Oleg Zhurakousky <
> oleg.zhurakousky@gmail.com> wrote:
>
>> No, all I do is have my own shutdown hook in the main which closes the
>> FSDataOutputStream. Before I did that it would throw an ugly exception when
>> I hit Ctrl+C, telling me that the stream is already closed, because of this
>> shutdown hook (bad design on the hadoop part), so removing it keeps it open
>> until the client's shutdown hook will close it at due time. So now it shuts
>> down gracefully and under complete control of the client.
>>
>>
>> On Tue, Jan 21, 2014 at 12:30 PM, Jay Vyas <ja...@gmail.com> wrote:
>>
>>> what is happening when you remove the shutdown hook ? is that
>>> supposed to trigger an exception -
>>>
>>
>>
>
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
>
Re: Shutdown hook for FileSystems
Posted by Oleg Zhurakousky <ol...@gmail.com>.
I am not sure either, you have to ask Hadoop guys, but it was giving me a
hard time so I found a way around it.
On Tue, Jan 21, 2014 at 6:05 PM, Jay Vyas <ja...@gmail.com> wrote:
> I guess im not sure what the ShutdownHook actually is there for.... Thats
> the real question im asking .
>
>
> On Tue, Jan 21, 2014 at 5:58 PM, Oleg Zhurakousky <
> oleg.zhurakousky@gmail.com> wrote:
>
>> No, all I do is have my own shutdown hook in the main which closes the
>> FSDataOutputStream. Before I did that it would throw an ugly exception when
>> I hit Ctrl+C, telling me that the stream is already closed, because of this
>> shutdown hook (bad design on the hadoop part), so removing it keeps it open
>> until the client's shutdown hook will close it at due time. So now it shuts
>> down gracefully and under complete control of the client.
>>
>>
>> On Tue, Jan 21, 2014 at 12:30 PM, Jay Vyas <ja...@gmail.com> wrote:
>>
>>> what is happening when you remove the shutdown hook ? is that
>>> supposed to trigger an exception -
>>>
>>
>>
>
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
>
Re: Shutdown hook for FileSystems
Posted by Oleg Zhurakousky <ol...@gmail.com>.
I am not sure either, you have to ask Hadoop guys, but it was giving me a
hard time so I found a way around it.
On Tue, Jan 21, 2014 at 6:05 PM, Jay Vyas <ja...@gmail.com> wrote:
> I guess im not sure what the ShutdownHook actually is there for.... Thats
> the real question im asking .
>
>
> On Tue, Jan 21, 2014 at 5:58 PM, Oleg Zhurakousky <
> oleg.zhurakousky@gmail.com> wrote:
>
>> No, all I do is have my own shutdown hook in the main which closes the
>> FSDataOutputStream. Before I did that it would throw an ugly exception when
>> I hit Ctrl+C, telling me that the stream is already closed, because of this
>> shutdown hook (bad design on the hadoop part), so removing it keeps it open
>> until the client's shutdown hook will close it at due time. So now it shuts
>> down gracefully and under complete control of the client.
>>
>>
>> On Tue, Jan 21, 2014 at 12:30 PM, Jay Vyas <ja...@gmail.com> wrote:
>>
>>> what is happening when you remove the shutdown hook ? is that
>>> supposed to trigger an exception -
>>>
>>
>>
>
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
>
Re: Shutdown hook for FileSystems
Posted by Jay Vyas <ja...@gmail.com>.
I guess im not sure what the ShutdownHook actually is there for.... Thats
the real question im asking .
On Tue, Jan 21, 2014 at 5:58 PM, Oleg Zhurakousky <
oleg.zhurakousky@gmail.com> wrote:
> No, all I do is have my own shutdown hook in the main which closes the
> FSDataOutputStream. Before I did that it would throw an ugly exception when
> I hit Ctrl+C, telling me that the stream is already closed, because of this
> shutdown hook (bad design on the hadoop part), so removing it keeps it open
> until the client's shutdown hook will close it at due time. So now it shuts
> down gracefully and under complete control of the client.
>
>
> On Tue, Jan 21, 2014 at 12:30 PM, Jay Vyas <ja...@gmail.com> wrote:
>
>> what is happening when you remove the shutdown hook ? is that supposed
>> to trigger an exception -
>>
>
>
--
Jay Vyas
http://jayunit100.blogspot.com
Re: Shutdown hook for FileSystems
Posted by Jay Vyas <ja...@gmail.com>.
I guess im not sure what the ShutdownHook actually is there for.... Thats
the real question im asking .
On Tue, Jan 21, 2014 at 5:58 PM, Oleg Zhurakousky <
oleg.zhurakousky@gmail.com> wrote:
> No, all I do is have my own shutdown hook in the main which closes the
> FSDataOutputStream. Before I did that it would throw an ugly exception when
> I hit Ctrl+C, telling me that the stream is already closed, because of this
> shutdown hook (bad design on the hadoop part), so removing it keeps it open
> until the client's shutdown hook will close it at due time. So now it shuts
> down gracefully and under complete control of the client.
>
>
> On Tue, Jan 21, 2014 at 12:30 PM, Jay Vyas <ja...@gmail.com> wrote:
>
>> what is happening when you remove the shutdown hook ? is that supposed
>> to trigger an exception -
>>
>
>
--
Jay Vyas
http://jayunit100.blogspot.com
Re: Shutdown hook for FileSystems
Posted by Jay Vyas <ja...@gmail.com>.
I guess im not sure what the ShutdownHook actually is there for.... Thats
the real question im asking .
On Tue, Jan 21, 2014 at 5:58 PM, Oleg Zhurakousky <
oleg.zhurakousky@gmail.com> wrote:
> No, all I do is have my own shutdown hook in the main which closes the
> FSDataOutputStream. Before I did that it would throw an ugly exception when
> I hit Ctrl+C, telling me that the stream is already closed, because of this
> shutdown hook (bad design on the hadoop part), so removing it keeps it open
> until the client's shutdown hook will close it at due time. So now it shuts
> down gracefully and under complete control of the client.
>
>
> On Tue, Jan 21, 2014 at 12:30 PM, Jay Vyas <ja...@gmail.com> wrote:
>
>> what is happening when you remove the shutdown hook ? is that supposed
>> to trigger an exception -
>>
>
>
--
Jay Vyas
http://jayunit100.blogspot.com
Re: Shutdown hook for FileSystems
Posted by Jay Vyas <ja...@gmail.com>.
I guess im not sure what the ShutdownHook actually is there for.... Thats
the real question im asking .
On Tue, Jan 21, 2014 at 5:58 PM, Oleg Zhurakousky <
oleg.zhurakousky@gmail.com> wrote:
> No, all I do is have my own shutdown hook in the main which closes the
> FSDataOutputStream. Before I did that it would throw an ugly exception when
> I hit Ctrl+C, telling me that the stream is already closed, because of this
> shutdown hook (bad design on the hadoop part), so removing it keeps it open
> until the client's shutdown hook will close it at due time. So now it shuts
> down gracefully and under complete control of the client.
>
>
> On Tue, Jan 21, 2014 at 12:30 PM, Jay Vyas <ja...@gmail.com> wrote:
>
>> what is happening when you remove the shutdown hook ? is that supposed
>> to trigger an exception -
>>
>
>
--
Jay Vyas
http://jayunit100.blogspot.com
Re: Shutdown hook for FileSystems
Posted by Oleg Zhurakousky <ol...@gmail.com>.
No, all I do is have my own shutdown hook in the main which closes the
FSDataOutputStream. Before I did that it would throw an ugly exception when
I hit Ctrl+C, telling me that the stream is already closed, because of this
shutdown hook (bad design on the hadoop part), so removing it keeps it open
until the client's shutdown hook will close it at due time. So now it shuts
down gracefully and under complete control of the client.
On Tue, Jan 21, 2014 at 12:30 PM, Jay Vyas <ja...@gmail.com> wrote:
> what is happening when you remove the shutdown hook ? is that supposed
> to trigger an exception -
>
Re: Shutdown hook for FileSystems
Posted by Oleg Zhurakousky <ol...@gmail.com>.
No, all I do is have my own shutdown hook in the main which closes the
FSDataOutputStream. Before I did that it would throw an ugly exception when
I hit Ctrl+C, telling me that the stream is already closed, because of this
shutdown hook (bad design on the hadoop part), so removing it keeps it open
until the client's shutdown hook will close it at due time. So now it shuts
down gracefully and under complete control of the client.
On Tue, Jan 21, 2014 at 12:30 PM, Jay Vyas <ja...@gmail.com> wrote:
> what is happening when you remove the shutdown hook ? is that supposed
> to trigger an exception -
>
Re: Shutdown hook for FileSystems
Posted by Oleg Zhurakousky <ol...@gmail.com>.
No, all I do is have my own shutdown hook in the main which closes the
FSDataOutputStream. Before I did that it would throw an ugly exception when
I hit Ctrl+C, telling me that the stream is already closed, because of this
shutdown hook (bad design on the hadoop part), so removing it keeps it open
until the client's shutdown hook will close it at due time. So now it shuts
down gracefully and under complete control of the client.
On Tue, Jan 21, 2014 at 12:30 PM, Jay Vyas <ja...@gmail.com> wrote:
> what is happening when you remove the shutdown hook ? is that supposed
> to trigger an exception -
>
Re: Shutdown hook for FileSystems
Posted by Oleg Zhurakousky <ol...@gmail.com>.
No, all I do is have my own shutdown hook in the main which closes the
FSDataOutputStream. Before I did that it would throw an ugly exception when
I hit Ctrl+C, telling me that the stream is already closed, because of this
shutdown hook (bad design on the hadoop part), so removing it keeps it open
until the client's shutdown hook will close it at due time. So now it shuts
down gracefully and under complete control of the client.
On Tue, Jan 21, 2014 at 12:30 PM, Jay Vyas <ja...@gmail.com> wrote:
> what is happening when you remove the shutdown hook ? is that supposed
> to trigger an exception -
>
Re: Shutdown hook for FileSystems
Posted by Jay Vyas <ja...@gmail.com>.
what is happening when you remove the shutdown hook ? is that supposed
to trigger an exception -
Re: Shutdown hook for FileSystems
Posted by Jay Vyas <ja...@gmail.com>.
what is happening when you remove the shutdown hook ? is that supposed
to trigger an exception -
Re: Shutdown hook for FileSystems
Posted by Jay Vyas <ja...@gmail.com>.
what is happening when you remove the shutdown hook ? is that supposed
to trigger an exception -
Re: Shutdown hook for FileSystems
Posted by Jay Vyas <ja...@gmail.com>.
what is happening when you remove the shutdown hook ? is that supposed
to trigger an exception -
Re: Shutdown hook for FileSystems
Posted by Oleg Zhurakousky <ol...@gmail.com>.
Had the same problem recently
Here is how you can get around it.
I am using org.springframework.util.ReflectionUtils here just for
convenience (to traverse class hierarchy if necessary), but you can just
use raw reflection.
try {
Field clientFinalizerField = ReflectionUtils.findField(fileSystem.getClass(),
"clientFinalizer");
clientFinalizerField.setAccessible(true);
Thread clientFinalizer = (Thread) clientFinalizerField.get(null);
Runtime.getRuntime().removeShutdownHook(clientFinalizer);
} catch (Exception e) {
// handle
}
On Tue, Jan 21, 2014 at 5:36 AM, Lukas Kairies <
lukas.xtreemfs@googlemail.com> wrote:
> Hey,
>
> I use Hadoop with XtreemFS (with a corresponding FileSystem
> implementation). The XtreemFS client uses several non-deamon Threads eg.
> for communication. Therefore the shutdown hooks do not start after a
> mapper/reducer is finished and the Child processes do not terminate.
>
> My question:
> Is there any way to determine if the mapper/reducer child processes do not
> need the file system any more?
>
> Best regards,
> Lukas K
>
Re: Shutdown hook for FileSystems
Posted by Oleg Zhurakousky <ol...@gmail.com>.
Had the same problem recently
Here is how you can get around it.
I am using org.springframework.util.ReflectionUtils here just for
convenience (to traverse class hierarchy if necessary), but you can just
use raw reflection.
try {
Field clientFinalizerField = ReflectionUtils.findField(fileSystem.getClass(),
"clientFinalizer");
clientFinalizerField.setAccessible(true);
Thread clientFinalizer = (Thread) clientFinalizerField.get(null);
Runtime.getRuntime().removeShutdownHook(clientFinalizer);
} catch (Exception e) {
// handle
}
On Tue, Jan 21, 2014 at 5:36 AM, Lukas Kairies <
lukas.xtreemfs@googlemail.com> wrote:
> Hey,
>
> I use Hadoop with XtreemFS (with a corresponding FileSystem
> implementation). The XtreemFS client uses several non-deamon Threads eg.
> for communication. Therefore the shutdown hooks do not start after a
> mapper/reducer is finished and the Child processes do not terminate.
>
> My question:
> Is there any way to determine if the mapper/reducer child processes do not
> need the file system any more?
>
> Best regards,
> Lukas K
>
Re: Shutdown hook for FileSystems
Posted by Oleg Zhurakousky <ol...@gmail.com>.
Had the same problem recently
Here is how you can get around it.
I am using org.springframework.util.ReflectionUtils here just for
convenience (to traverse class hierarchy if necessary), but you can just
use raw reflection.
try {
Field clientFinalizerField = ReflectionUtils.findField(fileSystem.getClass(),
"clientFinalizer");
clientFinalizerField.setAccessible(true);
Thread clientFinalizer = (Thread) clientFinalizerField.get(null);
Runtime.getRuntime().removeShutdownHook(clientFinalizer);
} catch (Exception e) {
// handle
}
On Tue, Jan 21, 2014 at 5:36 AM, Lukas Kairies <
lukas.xtreemfs@googlemail.com> wrote:
> Hey,
>
> I use Hadoop with XtreemFS (with a corresponding FileSystem
> implementation). The XtreemFS client uses several non-deamon Threads eg.
> for communication. Therefore the shutdown hooks do not start after a
> mapper/reducer is finished and the Child processes do not terminate.
>
> My question:
> Is there any way to determine if the mapper/reducer child processes do not
> need the file system any more?
>
> Best regards,
> Lukas K
>
Re: Shutdown hook for FileSystems
Posted by Oleg Zhurakousky <ol...@gmail.com>.
Had the same problem recently
Here is how you can get around it.
I am using org.springframework.util.ReflectionUtils here just for
convenience (to traverse class hierarchy if necessary), but you can just
use raw reflection.
try {
Field clientFinalizerField = ReflectionUtils.findField(fileSystem.getClass(),
"clientFinalizer");
clientFinalizerField.setAccessible(true);
Thread clientFinalizer = (Thread) clientFinalizerField.get(null);
Runtime.getRuntime().removeShutdownHook(clientFinalizer);
} catch (Exception e) {
// handle
}
On Tue, Jan 21, 2014 at 5:36 AM, Lukas Kairies <
lukas.xtreemfs@googlemail.com> wrote:
> Hey,
>
> I use Hadoop with XtreemFS (with a corresponding FileSystem
> implementation). The XtreemFS client uses several non-deamon Threads eg.
> for communication. Therefore the shutdown hooks do not start after a
> mapper/reducer is finished and the Child processes do not terminate.
>
> My question:
> Is there any way to determine if the mapper/reducer child processes do not
> need the file system any more?
>
> Best regards,
> Lukas K
>