You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hama.apache.org by "Edward J. Yoon" <ed...@apache.org> on 2011/09/23 12:19:15 UTC
Hang problem
Hi,
Today I ran the sssp example with 4GB sample file.
At 32th step, some tasks are finished and others hang forever.
Could anyone figure out this problem?
Plus, there're too many INFO-level logs. Let's reduce them.
Thanks.
--
Best Regards, Edward J. Yoon
@eddieyoon
Re: Hang problem
Posted by Thomas Jungblut <th...@googlemail.com>.
No, actually not.
2011/9/24 Edward J. Yoon <ed...@apache.org>
> > There is a while(updated) loop.
> > Updated is just getting false when globally no updates were made.
> > Same logic in Pagerank.
>
> Is this mean that some process can be finished earlier than others?
>
> On Sat, Sep 24, 2011 at 12:50 AM, Thomas Jungblut
> <th...@googlemail.com> wrote:
> > If it is still about SSSP:
> > Well, I took that into account. That is the reason why there is a master
> > task.
> > There is a while(updated) loop.
> > Updated is just getting false when globally no updates were made.
> > Same logic in Pagerank.
> > This is totally failsafe :p
> >
> > 2011/9/23 Edward J. Yoon <ed...@apache.org>
> >
> >> In other words, all tasks should be entered into next step until whole
> >> job is completed successfully.
> >>
> >> On Sat, Sep 24, 2011 at 12:37 AM, Edward J. Yoon <edwardyoon@apache.org
> >
> >> wrote:
> >> > According to BSPMaster log messages, a few tasks of all are finished
> >> > with SUCCEEDED status during the iterations. If I remember correctly,
> >> > child processes calls bspPeer.close() finally.
> >> >
> >> > Then yes, others will be hanged at the step of comparing the size of
> >> > znode and initial task size.
> >> >
> >> > I wonder what happens if some task no longer need to communicate with
> >> others?
> >> >
> >> > On Fri, Sep 23, 2011 at 11:59 PM, Thomas Jungblut
> >> > <th...@googlemail.com> wrote:
> >> >> Well, for SSSP example it might be correct.
> >> >> But you faced the hanging problems in randbench, too.
> >> >>
> >> >> Moreover, we have to implement our own mechanisms for high
> availability
> >> if
> >> >>> we have own sync master server.
> >> >>>
> >> >>
> >> >> +1
> >> >>
> >> >> 2011/9/23 Edward J. Yoon <ed...@apache.org>
> >> >>
> >> >>> As I mentioned before, it's not a ZK problem.
> >> >>>
> >> >>> Moreover, we have to implement our own mechanisms for high
> availability
> >> if
> >> >>> we have own sync master server.
> >> >>>
> >> >>> Sent from my iPad
> >> >>>
> >> >>> On Sep 23, 2011, at 11:01 PM, Thomas Jungblut <
> >> >>> thomas.jungblut@googlemail.com> wrote:
> >> >>>
> >> >>> > I have made a github for that:
> >> >>> > https://github.com/thomasjungblut/barriersync
> >> >>> >
> >> >>> > Check it out into your eclipse (the root directory failed for
> >> whatever
> >> >>> > reason).
> >> >>> > Start the server and then the clientemulator.
> >> >>> > Works like a real charm.
> >> >>> >
> >> >>> > Please consider this as an alternative. We should not roll out a
> 4.0
> >> >>> release
> >> >>> > with a not working barrier sync.
> >> >>> >
> >> >>> > 2011/9/23 Thomas Jungblut <th...@googlemail.com>
> >> >>> >
> >> >>> >> Won't much different.
> >> >>> >>>
> >> >>> >>
> >> >>> >> Let's see.
> >> >>> >>
> >> >>> >> 2011/9/23 Edward J. Yoon <ed...@apache.org>
> >> >>> >>
> >> >>> >>> What happens if some task no longer need to communicate with
> >> others?
> >> >>> >>>
> >> >>> >>> I didn't look at the code recently but I guess that the problem
> is
> >> >>> >>> related with comparison of znode size and task size.
> >> >>> >>>
> >> >>> >>>> I am going to write a RPC barrier sync. Zookeeper sucks in this
> >> case.
> >> >>> >>>
> >> >>> >>> Won't much different. Let's focusing on NG integration and
> >> In/Output
> >> >>> >>> system.
> >> >>> >>>
> >> >>> >>> On Fri, Sep 23, 2011 at 8:21 PM, Thomas Jungblut
> >> >>> >>> <th...@googlemail.com> wrote:
> >> >>> >>>> I am going to write a RPC barrier sync. Zookeeper sucks in this
> >> case.
> >> >>> >>>>
> >> >>> >>>> 2011/9/23 Edward J. Yoon <ed...@apache.org>
> >> >>> >>>>
> >> >>> >>>>> P.S., Tested on 16 nodes using 10 tasks per node.
> >> >>> >>>>>
> >> >>> >>>>> On Fri, Sep 23, 2011 at 7:19 PM, Edward J. Yoon <
> >> >>> edwardyoon@apache.org
> >> >>> >>>>
> >> >>> >>>>> wrote:
> >> >>> >>>>>> Hi,
> >> >>> >>>>>>
> >> >>> >>>>>> Today I ran the sssp example with 4GB sample file.
> >> >>> >>>>>>
> >> >>> >>>>>> At 32th step, some tasks are finished and others hang
> forever.
> >> >>> >>>>>>
> >> >>> >>>>>> Could anyone figure out this problem?
> >> >>> >>>>>>
> >> >>> >>>>>> Plus, there're too many INFO-level logs. Let's reduce them.
> >> >>> >>>>>>
> >> >>> >>>>>> Thanks.
> >> >>> >>>>>>
> >> >>> >>>>>> --
> >> >>> >>>>>> Best Regards, Edward J. Yoon
> >> >>> >>>>>> @eddieyoon
> >> >>> >>>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>> --
> >> >>> >>>>> Best Regards, Edward J. Yoon
> >> >>> >>>>> @eddieyoon
> >> >>> >>>>>
> >> >>> >>>>
> >> >>> >>>>
> >> >>> >>>>
> >> >>> >>>> --
> >> >>> >>>> Thomas Jungblut
> >> >>> >>>> Berlin
> >> >>> >>>>
> >> >>> >>>> mobile: 0170-3081070
> >> >>> >>>>
> >> >>> >>>> business: thomas.jungblut@testberichte.de
> >> >>> >>>> private: thomas.jungblut@gmail.com
> >> >>> >>>>
> >> >>> >>>
> >> >>> >>>
> >> >>> >>>
> >> >>> >>> --
> >> >>> >>> Best Regards, Edward J. Yoon
> >> >>> >>> @eddieyoon
> >> >>> >>>
> >> >>> >>
> >> >>> >>
> >> >>> >>
> >> >>> >> --
> >> >>> >> Thomas Jungblut
> >> >>> >> Berlin
> >> >>> >>
> >> >>> >> mobile: 0170-3081070
> >> >>> >>
> >> >>> >> business: thomas.jungblut@testberichte.de
> >> >>> >> private: thomas.jungblut@gmail.com
> >> >>> >>
> >> >>> >
> >> >>> >
> >> >>> >
> >> >>> > --
> >> >>> > Thomas Jungblut
> >> >>> > Berlin
> >> >>> >
> >> >>> > mobile: 0170-3081070
> >> >>> >
> >> >>> > business: thomas.jungblut@testberichte.de
> >> >>> > private: thomas.jungblut@gmail.com
> >> >>>
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Thomas Jungblut
> >> >> Berlin
> >> >>
> >> >> mobile: 0170-3081070
> >> >>
> >> >> business: thomas.jungblut@testberichte.de
> >> >> private: thomas.jungblut@gmail.com
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > Best Regards, Edward J. Yoon
> >> > @eddieyoon
> >> >
> >>
> >>
> >>
> >> --
> >> Best Regards, Edward J. Yoon
> >> @eddieyoon
> >>
> >
> >
> >
> > --
> > Thomas Jungblut
> > Berlin
> >
> > mobile: 0170-3081070
> >
> > business: thomas.jungblut@testberichte.de
> > private: thomas.jungblut@gmail.com
> >
>
>
>
> --
> Best Regards, Edward J. Yoon
> @eddieyoon
>
--
Thomas Jungblut
Berlin
mobile: 0170-3081070
business: thomas.jungblut@testberichte.de
private: thomas.jungblut@gmail.com
Re: Hang problem
Posted by "Edward J. Yoon" <ed...@apache.org>.
> There is a while(updated) loop.
> Updated is just getting false when globally no updates were made.
> Same logic in Pagerank.
Is this mean that some process can be finished earlier than others?
On Sat, Sep 24, 2011 at 12:50 AM, Thomas Jungblut
<th...@googlemail.com> wrote:
> If it is still about SSSP:
> Well, I took that into account. That is the reason why there is a master
> task.
> There is a while(updated) loop.
> Updated is just getting false when globally no updates were made.
> Same logic in Pagerank.
> This is totally failsafe :p
>
> 2011/9/23 Edward J. Yoon <ed...@apache.org>
>
>> In other words, all tasks should be entered into next step until whole
>> job is completed successfully.
>>
>> On Sat, Sep 24, 2011 at 12:37 AM, Edward J. Yoon <ed...@apache.org>
>> wrote:
>> > According to BSPMaster log messages, a few tasks of all are finished
>> > with SUCCEEDED status during the iterations. If I remember correctly,
>> > child processes calls bspPeer.close() finally.
>> >
>> > Then yes, others will be hanged at the step of comparing the size of
>> > znode and initial task size.
>> >
>> > I wonder what happens if some task no longer need to communicate with
>> others?
>> >
>> > On Fri, Sep 23, 2011 at 11:59 PM, Thomas Jungblut
>> > <th...@googlemail.com> wrote:
>> >> Well, for SSSP example it might be correct.
>> >> But you faced the hanging problems in randbench, too.
>> >>
>> >> Moreover, we have to implement our own mechanisms for high availability
>> if
>> >>> we have own sync master server.
>> >>>
>> >>
>> >> +1
>> >>
>> >> 2011/9/23 Edward J. Yoon <ed...@apache.org>
>> >>
>> >>> As I mentioned before, it's not a ZK problem.
>> >>>
>> >>> Moreover, we have to implement our own mechanisms for high availability
>> if
>> >>> we have own sync master server.
>> >>>
>> >>> Sent from my iPad
>> >>>
>> >>> On Sep 23, 2011, at 11:01 PM, Thomas Jungblut <
>> >>> thomas.jungblut@googlemail.com> wrote:
>> >>>
>> >>> > I have made a github for that:
>> >>> > https://github.com/thomasjungblut/barriersync
>> >>> >
>> >>> > Check it out into your eclipse (the root directory failed for
>> whatever
>> >>> > reason).
>> >>> > Start the server and then the clientemulator.
>> >>> > Works like a real charm.
>> >>> >
>> >>> > Please consider this as an alternative. We should not roll out a 4.0
>> >>> release
>> >>> > with a not working barrier sync.
>> >>> >
>> >>> > 2011/9/23 Thomas Jungblut <th...@googlemail.com>
>> >>> >
>> >>> >> Won't much different.
>> >>> >>>
>> >>> >>
>> >>> >> Let's see.
>> >>> >>
>> >>> >> 2011/9/23 Edward J. Yoon <ed...@apache.org>
>> >>> >>
>> >>> >>> What happens if some task no longer need to communicate with
>> others?
>> >>> >>>
>> >>> >>> I didn't look at the code recently but I guess that the problem is
>> >>> >>> related with comparison of znode size and task size.
>> >>> >>>
>> >>> >>>> I am going to write a RPC barrier sync. Zookeeper sucks in this
>> case.
>> >>> >>>
>> >>> >>> Won't much different. Let's focusing on NG integration and
>> In/Output
>> >>> >>> system.
>> >>> >>>
>> >>> >>> On Fri, Sep 23, 2011 at 8:21 PM, Thomas Jungblut
>> >>> >>> <th...@googlemail.com> wrote:
>> >>> >>>> I am going to write a RPC barrier sync. Zookeeper sucks in this
>> case.
>> >>> >>>>
>> >>> >>>> 2011/9/23 Edward J. Yoon <ed...@apache.org>
>> >>> >>>>
>> >>> >>>>> P.S., Tested on 16 nodes using 10 tasks per node.
>> >>> >>>>>
>> >>> >>>>> On Fri, Sep 23, 2011 at 7:19 PM, Edward J. Yoon <
>> >>> edwardyoon@apache.org
>> >>> >>>>
>> >>> >>>>> wrote:
>> >>> >>>>>> Hi,
>> >>> >>>>>>
>> >>> >>>>>> Today I ran the sssp example with 4GB sample file.
>> >>> >>>>>>
>> >>> >>>>>> At 32th step, some tasks are finished and others hang forever.
>> >>> >>>>>>
>> >>> >>>>>> Could anyone figure out this problem?
>> >>> >>>>>>
>> >>> >>>>>> Plus, there're too many INFO-level logs. Let's reduce them.
>> >>> >>>>>>
>> >>> >>>>>> Thanks.
>> >>> >>>>>>
>> >>> >>>>>> --
>> >>> >>>>>> Best Regards, Edward J. Yoon
>> >>> >>>>>> @eddieyoon
>> >>> >>>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> --
>> >>> >>>>> Best Regards, Edward J. Yoon
>> >>> >>>>> @eddieyoon
>> >>> >>>>>
>> >>> >>>>
>> >>> >>>>
>> >>> >>>>
>> >>> >>>> --
>> >>> >>>> Thomas Jungblut
>> >>> >>>> Berlin
>> >>> >>>>
>> >>> >>>> mobile: 0170-3081070
>> >>> >>>>
>> >>> >>>> business: thomas.jungblut@testberichte.de
>> >>> >>>> private: thomas.jungblut@gmail.com
>> >>> >>>>
>> >>> >>>
>> >>> >>>
>> >>> >>>
>> >>> >>> --
>> >>> >>> Best Regards, Edward J. Yoon
>> >>> >>> @eddieyoon
>> >>> >>>
>> >>> >>
>> >>> >>
>> >>> >>
>> >>> >> --
>> >>> >> Thomas Jungblut
>> >>> >> Berlin
>> >>> >>
>> >>> >> mobile: 0170-3081070
>> >>> >>
>> >>> >> business: thomas.jungblut@testberichte.de
>> >>> >> private: thomas.jungblut@gmail.com
>> >>> >>
>> >>> >
>> >>> >
>> >>> >
>> >>> > --
>> >>> > Thomas Jungblut
>> >>> > Berlin
>> >>> >
>> >>> > mobile: 0170-3081070
>> >>> >
>> >>> > business: thomas.jungblut@testberichte.de
>> >>> > private: thomas.jungblut@gmail.com
>> >>>
>> >>
>> >>
>> >>
>> >> --
>> >> Thomas Jungblut
>> >> Berlin
>> >>
>> >> mobile: 0170-3081070
>> >>
>> >> business: thomas.jungblut@testberichte.de
>> >> private: thomas.jungblut@gmail.com
>> >>
>> >
>> >
>> >
>> > --
>> > Best Regards, Edward J. Yoon
>> > @eddieyoon
>> >
>>
>>
>>
>> --
>> Best Regards, Edward J. Yoon
>> @eddieyoon
>>
>
>
>
> --
> Thomas Jungblut
> Berlin
>
> mobile: 0170-3081070
>
> business: thomas.jungblut@testberichte.de
> private: thomas.jungblut@gmail.com
>
--
Best Regards, Edward J. Yoon
@eddieyoon
Re: Hang problem
Posted by Thomas Jungblut <th...@googlemail.com>.
If it is still about SSSP:
Well, I took that into account. That is the reason why there is a master
task.
There is a while(updated) loop.
Updated is just getting false when globally no updates were made.
Same logic in Pagerank.
This is totally failsafe :p
2011/9/23 Edward J. Yoon <ed...@apache.org>
> In other words, all tasks should be entered into next step until whole
> job is completed successfully.
>
> On Sat, Sep 24, 2011 at 12:37 AM, Edward J. Yoon <ed...@apache.org>
> wrote:
> > According to BSPMaster log messages, a few tasks of all are finished
> > with SUCCEEDED status during the iterations. If I remember correctly,
> > child processes calls bspPeer.close() finally.
> >
> > Then yes, others will be hanged at the step of comparing the size of
> > znode and initial task size.
> >
> > I wonder what happens if some task no longer need to communicate with
> others?
> >
> > On Fri, Sep 23, 2011 at 11:59 PM, Thomas Jungblut
> > <th...@googlemail.com> wrote:
> >> Well, for SSSP example it might be correct.
> >> But you faced the hanging problems in randbench, too.
> >>
> >> Moreover, we have to implement our own mechanisms for high availability
> if
> >>> we have own sync master server.
> >>>
> >>
> >> +1
> >>
> >> 2011/9/23 Edward J. Yoon <ed...@apache.org>
> >>
> >>> As I mentioned before, it's not a ZK problem.
> >>>
> >>> Moreover, we have to implement our own mechanisms for high availability
> if
> >>> we have own sync master server.
> >>>
> >>> Sent from my iPad
> >>>
> >>> On Sep 23, 2011, at 11:01 PM, Thomas Jungblut <
> >>> thomas.jungblut@googlemail.com> wrote:
> >>>
> >>> > I have made a github for that:
> >>> > https://github.com/thomasjungblut/barriersync
> >>> >
> >>> > Check it out into your eclipse (the root directory failed for
> whatever
> >>> > reason).
> >>> > Start the server and then the clientemulator.
> >>> > Works like a real charm.
> >>> >
> >>> > Please consider this as an alternative. We should not roll out a 4.0
> >>> release
> >>> > with a not working barrier sync.
> >>> >
> >>> > 2011/9/23 Thomas Jungblut <th...@googlemail.com>
> >>> >
> >>> >> Won't much different.
> >>> >>>
> >>> >>
> >>> >> Let's see.
> >>> >>
> >>> >> 2011/9/23 Edward J. Yoon <ed...@apache.org>
> >>> >>
> >>> >>> What happens if some task no longer need to communicate with
> others?
> >>> >>>
> >>> >>> I didn't look at the code recently but I guess that the problem is
> >>> >>> related with comparison of znode size and task size.
> >>> >>>
> >>> >>>> I am going to write a RPC barrier sync. Zookeeper sucks in this
> case.
> >>> >>>
> >>> >>> Won't much different. Let's focusing on NG integration and
> In/Output
> >>> >>> system.
> >>> >>>
> >>> >>> On Fri, Sep 23, 2011 at 8:21 PM, Thomas Jungblut
> >>> >>> <th...@googlemail.com> wrote:
> >>> >>>> I am going to write a RPC barrier sync. Zookeeper sucks in this
> case.
> >>> >>>>
> >>> >>>> 2011/9/23 Edward J. Yoon <ed...@apache.org>
> >>> >>>>
> >>> >>>>> P.S., Tested on 16 nodes using 10 tasks per node.
> >>> >>>>>
> >>> >>>>> On Fri, Sep 23, 2011 at 7:19 PM, Edward J. Yoon <
> >>> edwardyoon@apache.org
> >>> >>>>
> >>> >>>>> wrote:
> >>> >>>>>> Hi,
> >>> >>>>>>
> >>> >>>>>> Today I ran the sssp example with 4GB sample file.
> >>> >>>>>>
> >>> >>>>>> At 32th step, some tasks are finished and others hang forever.
> >>> >>>>>>
> >>> >>>>>> Could anyone figure out this problem?
> >>> >>>>>>
> >>> >>>>>> Plus, there're too many INFO-level logs. Let's reduce them.
> >>> >>>>>>
> >>> >>>>>> Thanks.
> >>> >>>>>>
> >>> >>>>>> --
> >>> >>>>>> Best Regards, Edward J. Yoon
> >>> >>>>>> @eddieyoon
> >>> >>>>>>
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> >>> >>>>> --
> >>> >>>>> Best Regards, Edward J. Yoon
> >>> >>>>> @eddieyoon
> >>> >>>>>
> >>> >>>>
> >>> >>>>
> >>> >>>>
> >>> >>>> --
> >>> >>>> Thomas Jungblut
> >>> >>>> Berlin
> >>> >>>>
> >>> >>>> mobile: 0170-3081070
> >>> >>>>
> >>> >>>> business: thomas.jungblut@testberichte.de
> >>> >>>> private: thomas.jungblut@gmail.com
> >>> >>>>
> >>> >>>
> >>> >>>
> >>> >>>
> >>> >>> --
> >>> >>> Best Regards, Edward J. Yoon
> >>> >>> @eddieyoon
> >>> >>>
> >>> >>
> >>> >>
> >>> >>
> >>> >> --
> >>> >> Thomas Jungblut
> >>> >> Berlin
> >>> >>
> >>> >> mobile: 0170-3081070
> >>> >>
> >>> >> business: thomas.jungblut@testberichte.de
> >>> >> private: thomas.jungblut@gmail.com
> >>> >>
> >>> >
> >>> >
> >>> >
> >>> > --
> >>> > Thomas Jungblut
> >>> > Berlin
> >>> >
> >>> > mobile: 0170-3081070
> >>> >
> >>> > business: thomas.jungblut@testberichte.de
> >>> > private: thomas.jungblut@gmail.com
> >>>
> >>
> >>
> >>
> >> --
> >> Thomas Jungblut
> >> Berlin
> >>
> >> mobile: 0170-3081070
> >>
> >> business: thomas.jungblut@testberichte.de
> >> private: thomas.jungblut@gmail.com
> >>
> >
> >
> >
> > --
> > Best Regards, Edward J. Yoon
> > @eddieyoon
> >
>
>
>
> --
> Best Regards, Edward J. Yoon
> @eddieyoon
>
--
Thomas Jungblut
Berlin
mobile: 0170-3081070
business: thomas.jungblut@testberichte.de
private: thomas.jungblut@gmail.com
Re: Hang problem
Posted by "Edward J. Yoon" <ed...@apache.org>.
In other words, all tasks should be entered into next step until whole
job is completed successfully.
On Sat, Sep 24, 2011 at 12:37 AM, Edward J. Yoon <ed...@apache.org> wrote:
> According to BSPMaster log messages, a few tasks of all are finished
> with SUCCEEDED status during the iterations. If I remember correctly,
> child processes calls bspPeer.close() finally.
>
> Then yes, others will be hanged at the step of comparing the size of
> znode and initial task size.
>
> I wonder what happens if some task no longer need to communicate with others?
>
> On Fri, Sep 23, 2011 at 11:59 PM, Thomas Jungblut
> <th...@googlemail.com> wrote:
>> Well, for SSSP example it might be correct.
>> But you faced the hanging problems in randbench, too.
>>
>> Moreover, we have to implement our own mechanisms for high availability if
>>> we have own sync master server.
>>>
>>
>> +1
>>
>> 2011/9/23 Edward J. Yoon <ed...@apache.org>
>>
>>> As I mentioned before, it's not a ZK problem.
>>>
>>> Moreover, we have to implement our own mechanisms for high availability if
>>> we have own sync master server.
>>>
>>> Sent from my iPad
>>>
>>> On Sep 23, 2011, at 11:01 PM, Thomas Jungblut <
>>> thomas.jungblut@googlemail.com> wrote:
>>>
>>> > I have made a github for that:
>>> > https://github.com/thomasjungblut/barriersync
>>> >
>>> > Check it out into your eclipse (the root directory failed for whatever
>>> > reason).
>>> > Start the server and then the clientemulator.
>>> > Works like a real charm.
>>> >
>>> > Please consider this as an alternative. We should not roll out a 4.0
>>> release
>>> > with a not working barrier sync.
>>> >
>>> > 2011/9/23 Thomas Jungblut <th...@googlemail.com>
>>> >
>>> >> Won't much different.
>>> >>>
>>> >>
>>> >> Let's see.
>>> >>
>>> >> 2011/9/23 Edward J. Yoon <ed...@apache.org>
>>> >>
>>> >>> What happens if some task no longer need to communicate with others?
>>> >>>
>>> >>> I didn't look at the code recently but I guess that the problem is
>>> >>> related with comparison of znode size and task size.
>>> >>>
>>> >>>> I am going to write a RPC barrier sync. Zookeeper sucks in this case.
>>> >>>
>>> >>> Won't much different. Let's focusing on NG integration and In/Output
>>> >>> system.
>>> >>>
>>> >>> On Fri, Sep 23, 2011 at 8:21 PM, Thomas Jungblut
>>> >>> <th...@googlemail.com> wrote:
>>> >>>> I am going to write a RPC barrier sync. Zookeeper sucks in this case.
>>> >>>>
>>> >>>> 2011/9/23 Edward J. Yoon <ed...@apache.org>
>>> >>>>
>>> >>>>> P.S., Tested on 16 nodes using 10 tasks per node.
>>> >>>>>
>>> >>>>> On Fri, Sep 23, 2011 at 7:19 PM, Edward J. Yoon <
>>> edwardyoon@apache.org
>>> >>>>
>>> >>>>> wrote:
>>> >>>>>> Hi,
>>> >>>>>>
>>> >>>>>> Today I ran the sssp example with 4GB sample file.
>>> >>>>>>
>>> >>>>>> At 32th step, some tasks are finished and others hang forever.
>>> >>>>>>
>>> >>>>>> Could anyone figure out this problem?
>>> >>>>>>
>>> >>>>>> Plus, there're too many INFO-level logs. Let's reduce them.
>>> >>>>>>
>>> >>>>>> Thanks.
>>> >>>>>>
>>> >>>>>> --
>>> >>>>>> Best Regards, Edward J. Yoon
>>> >>>>>> @eddieyoon
>>> >>>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>> --
>>> >>>>> Best Regards, Edward J. Yoon
>>> >>>>> @eddieyoon
>>> >>>>>
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> --
>>> >>>> Thomas Jungblut
>>> >>>> Berlin
>>> >>>>
>>> >>>> mobile: 0170-3081070
>>> >>>>
>>> >>>> business: thomas.jungblut@testberichte.de
>>> >>>> private: thomas.jungblut@gmail.com
>>> >>>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> --
>>> >>> Best Regards, Edward J. Yoon
>>> >>> @eddieyoon
>>> >>>
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> Thomas Jungblut
>>> >> Berlin
>>> >>
>>> >> mobile: 0170-3081070
>>> >>
>>> >> business: thomas.jungblut@testberichte.de
>>> >> private: thomas.jungblut@gmail.com
>>> >>
>>> >
>>> >
>>> >
>>> > --
>>> > Thomas Jungblut
>>> > Berlin
>>> >
>>> > mobile: 0170-3081070
>>> >
>>> > business: thomas.jungblut@testberichte.de
>>> > private: thomas.jungblut@gmail.com
>>>
>>
>>
>>
>> --
>> Thomas Jungblut
>> Berlin
>>
>> mobile: 0170-3081070
>>
>> business: thomas.jungblut@testberichte.de
>> private: thomas.jungblut@gmail.com
>>
>
>
>
> --
> Best Regards, Edward J. Yoon
> @eddieyoon
>
--
Best Regards, Edward J. Yoon
@eddieyoon
Re: Hang problem
Posted by "Edward J. Yoon" <ed...@apache.org>.
According to BSPMaster log messages, a few tasks of all are finished
with SUCCEEDED status during the iterations. If I remember correctly,
child processes calls bspPeer.close() finally.
Then yes, others will be hanged at the step of comparing the size of
znode and initial task size.
I wonder what happens if some task no longer need to communicate with others?
On Fri, Sep 23, 2011 at 11:59 PM, Thomas Jungblut
<th...@googlemail.com> wrote:
> Well, for SSSP example it might be correct.
> But you faced the hanging problems in randbench, too.
>
> Moreover, we have to implement our own mechanisms for high availability if
>> we have own sync master server.
>>
>
> +1
>
> 2011/9/23 Edward J. Yoon <ed...@apache.org>
>
>> As I mentioned before, it's not a ZK problem.
>>
>> Moreover, we have to implement our own mechanisms for high availability if
>> we have own sync master server.
>>
>> Sent from my iPad
>>
>> On Sep 23, 2011, at 11:01 PM, Thomas Jungblut <
>> thomas.jungblut@googlemail.com> wrote:
>>
>> > I have made a github for that:
>> > https://github.com/thomasjungblut/barriersync
>> >
>> > Check it out into your eclipse (the root directory failed for whatever
>> > reason).
>> > Start the server and then the clientemulator.
>> > Works like a real charm.
>> >
>> > Please consider this as an alternative. We should not roll out a 4.0
>> release
>> > with a not working barrier sync.
>> >
>> > 2011/9/23 Thomas Jungblut <th...@googlemail.com>
>> >
>> >> Won't much different.
>> >>>
>> >>
>> >> Let's see.
>> >>
>> >> 2011/9/23 Edward J. Yoon <ed...@apache.org>
>> >>
>> >>> What happens if some task no longer need to communicate with others?
>> >>>
>> >>> I didn't look at the code recently but I guess that the problem is
>> >>> related with comparison of znode size and task size.
>> >>>
>> >>>> I am going to write a RPC barrier sync. Zookeeper sucks in this case.
>> >>>
>> >>> Won't much different. Let's focusing on NG integration and In/Output
>> >>> system.
>> >>>
>> >>> On Fri, Sep 23, 2011 at 8:21 PM, Thomas Jungblut
>> >>> <th...@googlemail.com> wrote:
>> >>>> I am going to write a RPC barrier sync. Zookeeper sucks in this case.
>> >>>>
>> >>>> 2011/9/23 Edward J. Yoon <ed...@apache.org>
>> >>>>
>> >>>>> P.S., Tested on 16 nodes using 10 tasks per node.
>> >>>>>
>> >>>>> On Fri, Sep 23, 2011 at 7:19 PM, Edward J. Yoon <
>> edwardyoon@apache.org
>> >>>>
>> >>>>> wrote:
>> >>>>>> Hi,
>> >>>>>>
>> >>>>>> Today I ran the sssp example with 4GB sample file.
>> >>>>>>
>> >>>>>> At 32th step, some tasks are finished and others hang forever.
>> >>>>>>
>> >>>>>> Could anyone figure out this problem?
>> >>>>>>
>> >>>>>> Plus, there're too many INFO-level logs. Let's reduce them.
>> >>>>>>
>> >>>>>> Thanks.
>> >>>>>>
>> >>>>>> --
>> >>>>>> Best Regards, Edward J. Yoon
>> >>>>>> @eddieyoon
>> >>>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> --
>> >>>>> Best Regards, Edward J. Yoon
>> >>>>> @eddieyoon
>> >>>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>> --
>> >>>> Thomas Jungblut
>> >>>> Berlin
>> >>>>
>> >>>> mobile: 0170-3081070
>> >>>>
>> >>>> business: thomas.jungblut@testberichte.de
>> >>>> private: thomas.jungblut@gmail.com
>> >>>>
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Best Regards, Edward J. Yoon
>> >>> @eddieyoon
>> >>>
>> >>
>> >>
>> >>
>> >> --
>> >> Thomas Jungblut
>> >> Berlin
>> >>
>> >> mobile: 0170-3081070
>> >>
>> >> business: thomas.jungblut@testberichte.de
>> >> private: thomas.jungblut@gmail.com
>> >>
>> >
>> >
>> >
>> > --
>> > Thomas Jungblut
>> > Berlin
>> >
>> > mobile: 0170-3081070
>> >
>> > business: thomas.jungblut@testberichte.de
>> > private: thomas.jungblut@gmail.com
>>
>
>
>
> --
> Thomas Jungblut
> Berlin
>
> mobile: 0170-3081070
>
> business: thomas.jungblut@testberichte.de
> private: thomas.jungblut@gmail.com
>
--
Best Regards, Edward J. Yoon
@eddieyoon
Re: Hang problem
Posted by Thomas Jungblut <th...@googlemail.com>.
Well, for SSSP example it might be correct.
But you faced the hanging problems in randbench, too.
Moreover, we have to implement our own mechanisms for high availability if
> we have own sync master server.
>
+1
2011/9/23 Edward J. Yoon <ed...@apache.org>
> As I mentioned before, it's not a ZK problem.
>
> Moreover, we have to implement our own mechanisms for high availability if
> we have own sync master server.
>
> Sent from my iPad
>
> On Sep 23, 2011, at 11:01 PM, Thomas Jungblut <
> thomas.jungblut@googlemail.com> wrote:
>
> > I have made a github for that:
> > https://github.com/thomasjungblut/barriersync
> >
> > Check it out into your eclipse (the root directory failed for whatever
> > reason).
> > Start the server and then the clientemulator.
> > Works like a real charm.
> >
> > Please consider this as an alternative. We should not roll out a 4.0
> release
> > with a not working barrier sync.
> >
> > 2011/9/23 Thomas Jungblut <th...@googlemail.com>
> >
> >> Won't much different.
> >>>
> >>
> >> Let's see.
> >>
> >> 2011/9/23 Edward J. Yoon <ed...@apache.org>
> >>
> >>> What happens if some task no longer need to communicate with others?
> >>>
> >>> I didn't look at the code recently but I guess that the problem is
> >>> related with comparison of znode size and task size.
> >>>
> >>>> I am going to write a RPC barrier sync. Zookeeper sucks in this case.
> >>>
> >>> Won't much different. Let's focusing on NG integration and In/Output
> >>> system.
> >>>
> >>> On Fri, Sep 23, 2011 at 8:21 PM, Thomas Jungblut
> >>> <th...@googlemail.com> wrote:
> >>>> I am going to write a RPC barrier sync. Zookeeper sucks in this case.
> >>>>
> >>>> 2011/9/23 Edward J. Yoon <ed...@apache.org>
> >>>>
> >>>>> P.S., Tested on 16 nodes using 10 tasks per node.
> >>>>>
> >>>>> On Fri, Sep 23, 2011 at 7:19 PM, Edward J. Yoon <
> edwardyoon@apache.org
> >>>>
> >>>>> wrote:
> >>>>>> Hi,
> >>>>>>
> >>>>>> Today I ran the sssp example with 4GB sample file.
> >>>>>>
> >>>>>> At 32th step, some tasks are finished and others hang forever.
> >>>>>>
> >>>>>> Could anyone figure out this problem?
> >>>>>>
> >>>>>> Plus, there're too many INFO-level logs. Let's reduce them.
> >>>>>>
> >>>>>> Thanks.
> >>>>>>
> >>>>>> --
> >>>>>> Best Regards, Edward J. Yoon
> >>>>>> @eddieyoon
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> Best Regards, Edward J. Yoon
> >>>>> @eddieyoon
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Thomas Jungblut
> >>>> Berlin
> >>>>
> >>>> mobile: 0170-3081070
> >>>>
> >>>> business: thomas.jungblut@testberichte.de
> >>>> private: thomas.jungblut@gmail.com
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Best Regards, Edward J. Yoon
> >>> @eddieyoon
> >>>
> >>
> >>
> >>
> >> --
> >> Thomas Jungblut
> >> Berlin
> >>
> >> mobile: 0170-3081070
> >>
> >> business: thomas.jungblut@testberichte.de
> >> private: thomas.jungblut@gmail.com
> >>
> >
> >
> >
> > --
> > Thomas Jungblut
> > Berlin
> >
> > mobile: 0170-3081070
> >
> > business: thomas.jungblut@testberichte.de
> > private: thomas.jungblut@gmail.com
>
--
Thomas Jungblut
Berlin
mobile: 0170-3081070
business: thomas.jungblut@testberichte.de
private: thomas.jungblut@gmail.com
Re: Hang problem
Posted by "Edward J. Yoon" <ed...@apache.org>.
As I mentioned before, it's not a ZK problem.
Moreover, we have to implement our own mechanisms for high availability if we have own sync master server.
Sent from my iPad
On Sep 23, 2011, at 11:01 PM, Thomas Jungblut <th...@googlemail.com> wrote:
> I have made a github for that:
> https://github.com/thomasjungblut/barriersync
>
> Check it out into your eclipse (the root directory failed for whatever
> reason).
> Start the server and then the clientemulator.
> Works like a real charm.
>
> Please consider this as an alternative. We should not roll out a 4.0 release
> with a not working barrier sync.
>
> 2011/9/23 Thomas Jungblut <th...@googlemail.com>
>
>> Won't much different.
>>>
>>
>> Let's see.
>>
>> 2011/9/23 Edward J. Yoon <ed...@apache.org>
>>
>>> What happens if some task no longer need to communicate with others?
>>>
>>> I didn't look at the code recently but I guess that the problem is
>>> related with comparison of znode size and task size.
>>>
>>>> I am going to write a RPC barrier sync. Zookeeper sucks in this case.
>>>
>>> Won't much different. Let's focusing on NG integration and In/Output
>>> system.
>>>
>>> On Fri, Sep 23, 2011 at 8:21 PM, Thomas Jungblut
>>> <th...@googlemail.com> wrote:
>>>> I am going to write a RPC barrier sync. Zookeeper sucks in this case.
>>>>
>>>> 2011/9/23 Edward J. Yoon <ed...@apache.org>
>>>>
>>>>> P.S., Tested on 16 nodes using 10 tasks per node.
>>>>>
>>>>> On Fri, Sep 23, 2011 at 7:19 PM, Edward J. Yoon <edwardyoon@apache.org
>>>>
>>>>> wrote:
>>>>>> Hi,
>>>>>>
>>>>>> Today I ran the sssp example with 4GB sample file.
>>>>>>
>>>>>> At 32th step, some tasks are finished and others hang forever.
>>>>>>
>>>>>> Could anyone figure out this problem?
>>>>>>
>>>>>> Plus, there're too many INFO-level logs. Let's reduce them.
>>>>>>
>>>>>> Thanks.
>>>>>>
>>>>>> --
>>>>>> Best Regards, Edward J. Yoon
>>>>>> @eddieyoon
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Best Regards, Edward J. Yoon
>>>>> @eddieyoon
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Thomas Jungblut
>>>> Berlin
>>>>
>>>> mobile: 0170-3081070
>>>>
>>>> business: thomas.jungblut@testberichte.de
>>>> private: thomas.jungblut@gmail.com
>>>>
>>>
>>>
>>>
>>> --
>>> Best Regards, Edward J. Yoon
>>> @eddieyoon
>>>
>>
>>
>>
>> --
>> Thomas Jungblut
>> Berlin
>>
>> mobile: 0170-3081070
>>
>> business: thomas.jungblut@testberichte.de
>> private: thomas.jungblut@gmail.com
>>
>
>
>
> --
> Thomas Jungblut
> Berlin
>
> mobile: 0170-3081070
>
> business: thomas.jungblut@testberichte.de
> private: thomas.jungblut@gmail.com
Re: Hang problem
Posted by Thomas Jungblut <th...@googlemail.com>.
I have made a github for that:
https://github.com/thomasjungblut/barriersync
Check it out into your eclipse (the root directory failed for whatever
reason).
Start the server and then the clientemulator.
Works like a real charm.
Please consider this as an alternative. We should not roll out a 4.0 release
with a not working barrier sync.
2011/9/23 Thomas Jungblut <th...@googlemail.com>
> Won't much different.
>>
>
> Let's see.
>
> 2011/9/23 Edward J. Yoon <ed...@apache.org>
>
>> What happens if some task no longer need to communicate with others?
>>
>> I didn't look at the code recently but I guess that the problem is
>> related with comparison of znode size and task size.
>>
>> > I am going to write a RPC barrier sync. Zookeeper sucks in this case.
>>
>> Won't much different. Let's focusing on NG integration and In/Output
>> system.
>>
>> On Fri, Sep 23, 2011 at 8:21 PM, Thomas Jungblut
>> <th...@googlemail.com> wrote:
>> > I am going to write a RPC barrier sync. Zookeeper sucks in this case.
>> >
>> > 2011/9/23 Edward J. Yoon <ed...@apache.org>
>> >
>> >> P.S., Tested on 16 nodes using 10 tasks per node.
>> >>
>> >> On Fri, Sep 23, 2011 at 7:19 PM, Edward J. Yoon <edwardyoon@apache.org
>> >
>> >> wrote:
>> >> > Hi,
>> >> >
>> >> > Today I ran the sssp example with 4GB sample file.
>> >> >
>> >> > At 32th step, some tasks are finished and others hang forever.
>> >> >
>> >> > Could anyone figure out this problem?
>> >> >
>> >> > Plus, there're too many INFO-level logs. Let's reduce them.
>> >> >
>> >> > Thanks.
>> >> >
>> >> > --
>> >> > Best Regards, Edward J. Yoon
>> >> > @eddieyoon
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Best Regards, Edward J. Yoon
>> >> @eddieyoon
>> >>
>> >
>> >
>> >
>> > --
>> > Thomas Jungblut
>> > Berlin
>> >
>> > mobile: 0170-3081070
>> >
>> > business: thomas.jungblut@testberichte.de
>> > private: thomas.jungblut@gmail.com
>> >
>>
>>
>>
>> --
>> Best Regards, Edward J. Yoon
>> @eddieyoon
>>
>
>
>
> --
> Thomas Jungblut
> Berlin
>
> mobile: 0170-3081070
>
> business: thomas.jungblut@testberichte.de
> private: thomas.jungblut@gmail.com
>
--
Thomas Jungblut
Berlin
mobile: 0170-3081070
business: thomas.jungblut@testberichte.de
private: thomas.jungblut@gmail.com
Re: Hang problem
Posted by Thomas Jungblut <th...@googlemail.com>.
>
> Won't much different.
>
Let's see.
2011/9/23 Edward J. Yoon <ed...@apache.org>
> What happens if some task no longer need to communicate with others?
>
> I didn't look at the code recently but I guess that the problem is
> related with comparison of znode size and task size.
>
> > I am going to write a RPC barrier sync. Zookeeper sucks in this case.
>
> Won't much different. Let's focusing on NG integration and In/Output
> system.
>
> On Fri, Sep 23, 2011 at 8:21 PM, Thomas Jungblut
> <th...@googlemail.com> wrote:
> > I am going to write a RPC barrier sync. Zookeeper sucks in this case.
> >
> > 2011/9/23 Edward J. Yoon <ed...@apache.org>
> >
> >> P.S., Tested on 16 nodes using 10 tasks per node.
> >>
> >> On Fri, Sep 23, 2011 at 7:19 PM, Edward J. Yoon <ed...@apache.org>
> >> wrote:
> >> > Hi,
> >> >
> >> > Today I ran the sssp example with 4GB sample file.
> >> >
> >> > At 32th step, some tasks are finished and others hang forever.
> >> >
> >> > Could anyone figure out this problem?
> >> >
> >> > Plus, there're too many INFO-level logs. Let's reduce them.
> >> >
> >> > Thanks.
> >> >
> >> > --
> >> > Best Regards, Edward J. Yoon
> >> > @eddieyoon
> >> >
> >>
> >>
> >>
> >> --
> >> Best Regards, Edward J. Yoon
> >> @eddieyoon
> >>
> >
> >
> >
> > --
> > Thomas Jungblut
> > Berlin
> >
> > mobile: 0170-3081070
> >
> > business: thomas.jungblut@testberichte.de
> > private: thomas.jungblut@gmail.com
> >
>
>
>
> --
> Best Regards, Edward J. Yoon
> @eddieyoon
>
--
Thomas Jungblut
Berlin
mobile: 0170-3081070
business: thomas.jungblut@testberichte.de
private: thomas.jungblut@gmail.com
Re: Hang problem
Posted by "Edward J. Yoon" <ed...@apache.org>.
What happens if some task no longer need to communicate with others?
I didn't look at the code recently but I guess that the problem is
related with comparison of znode size and task size.
> I am going to write a RPC barrier sync. Zookeeper sucks in this case.
Won't much different. Let's focusing on NG integration and In/Output system.
On Fri, Sep 23, 2011 at 8:21 PM, Thomas Jungblut
<th...@googlemail.com> wrote:
> I am going to write a RPC barrier sync. Zookeeper sucks in this case.
>
> 2011/9/23 Edward J. Yoon <ed...@apache.org>
>
>> P.S., Tested on 16 nodes using 10 tasks per node.
>>
>> On Fri, Sep 23, 2011 at 7:19 PM, Edward J. Yoon <ed...@apache.org>
>> wrote:
>> > Hi,
>> >
>> > Today I ran the sssp example with 4GB sample file.
>> >
>> > At 32th step, some tasks are finished and others hang forever.
>> >
>> > Could anyone figure out this problem?
>> >
>> > Plus, there're too many INFO-level logs. Let's reduce them.
>> >
>> > Thanks.
>> >
>> > --
>> > Best Regards, Edward J. Yoon
>> > @eddieyoon
>> >
>>
>>
>>
>> --
>> Best Regards, Edward J. Yoon
>> @eddieyoon
>>
>
>
>
> --
> Thomas Jungblut
> Berlin
>
> mobile: 0170-3081070
>
> business: thomas.jungblut@testberichte.de
> private: thomas.jungblut@gmail.com
>
--
Best Regards, Edward J. Yoon
@eddieyoon
Re: Hang problem
Posted by Thomas Jungblut <th...@googlemail.com>.
I am going to write a RPC barrier sync. Zookeeper sucks in this case.
2011/9/23 Edward J. Yoon <ed...@apache.org>
> P.S., Tested on 16 nodes using 10 tasks per node.
>
> On Fri, Sep 23, 2011 at 7:19 PM, Edward J. Yoon <ed...@apache.org>
> wrote:
> > Hi,
> >
> > Today I ran the sssp example with 4GB sample file.
> >
> > At 32th step, some tasks are finished and others hang forever.
> >
> > Could anyone figure out this problem?
> >
> > Plus, there're too many INFO-level logs. Let's reduce them.
> >
> > Thanks.
> >
> > --
> > Best Regards, Edward J. Yoon
> > @eddieyoon
> >
>
>
>
> --
> Best Regards, Edward J. Yoon
> @eddieyoon
>
--
Thomas Jungblut
Berlin
mobile: 0170-3081070
business: thomas.jungblut@testberichte.de
private: thomas.jungblut@gmail.com
Re: Hang problem
Posted by ChiaHung Lin <ch...@nuk.edu.tw>.
Do we have log and some detail such as how it is executed (or some custom debug output for figuring out its execution order), etc.?
In addition, It might help check other settings e.g. if /tmp is full. Previously I encountered job hanging forever issue, but that's because the /tmp size is too small. Cleaning /tmp or switching temp dir where bsp writes to fixes the problem.
But it may be also the flaw in code (it would be good if anyone can help review the patch that may spot some issues that I did not notice) or if in the case that double barrier in fact does not work, then we need to switch using e.g. polling that Vinod suggests.
-----Original message-----
From:Edward J. Yoon <ed...@apache.org>
To:hama-dev@incubator.apache.org
Date:Fri, 23 Sep 2011 19:20:58 +0900
Subject:Re: Hang problem
P.S., Tested on 16 nodes using 10 tasks per node.
On Fri, Sep 23, 2011 at 7:19 PM, Edward J. Yoon <ed...@apache.org> wrote:
> Hi,
>
> Today I ran the sssp example with 4GB sample file.
>
> At 32th step, some tasks are finished and others hang forever.
>
> Could anyone figure out this problem?
>
> Plus, there're too many INFO-level logs. Let's reduce them.
>
> Thanks.
>
> --
> Best Regards, Edward J. Yoon
> @eddieyoon
>
--
Best Regards, Edward J. Yoon
@eddieyoon
--
ChiaHung Lin
Department of Information Management
National University of Kaohsiung
Taiwan
Re: Hang problem
Posted by "Edward J. Yoon" <ed...@apache.org>.
P.S., Tested on 16 nodes using 10 tasks per node.
On Fri, Sep 23, 2011 at 7:19 PM, Edward J. Yoon <ed...@apache.org> wrote:
> Hi,
>
> Today I ran the sssp example with 4GB sample file.
>
> At 32th step, some tasks are finished and others hang forever.
>
> Could anyone figure out this problem?
>
> Plus, there're too many INFO-level logs. Let's reduce them.
>
> Thanks.
>
> --
> Best Regards, Edward J. Yoon
> @eddieyoon
>
--
Best Regards, Edward J. Yoon
@eddieyoon