You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hama.apache.org by Thomas Jungblut <th...@gmail.com> on 2012/07/09 09:22:10 UTC
Roadmap and Release Manager
So now we released 0.5.0 what are our plans for 0.6.0?
And who is going to do the release managing?
Re: Roadmap and Release Manager
Posted by Suraj Menon <su...@apache.org>.
I suggest we do a code freeze in the last week of August for 0.6.0. and
spend the next week fixing and hardening the release candidate(s).
And then we release 0.6.1 by the end of September. Version 0.6.1 would have
our examples with Fault tolerance and other new features and API changes we
put by then.
This gives a chance to wider audiences to try our release early and give
their feedback.
Based on our progress till Aug 20th, we can decide the JIRA issues that
would get fixed in 0.6.0 before the code freeze and which of them would be
shipped in 0.6.1.
Let me know your thoughts on this.
Thanks,
Suraj
On Thu, Aug 9, 2012 at 7:46 AM, Tommaso Teofili
<to...@gmail.com>wrote:
> same here, I'll most likely submit my work w.r.t. ML module in September.
> Tommaso
>
> 2012/8/9 Edward J. Yoon <ed...@apache.org>
>
> > >> - Data repartitioning
> > >> - Hadoop 2.0 compatibility
> >
> > P.S., My summer vacation will start from Aug 11th, so I need more time.
> >
> > On Mon, Jul 9, 2012 at 10:49 PM, Tommaso Teofili
> > <to...@gmail.com> wrote:
> > > 2012/7/9 Edward J. Yoon <ed...@apache.org>
> > >
> > >> Then, let's try to release 0.6 by end of August or September 2012. I
> > >> heard/saw some CDH4 plans/activities, we should not hesitate. :)
> > >>
> > >> Our TODO list:
> > >>
> > >> >> - Fault tolerance
> > >> >> - Data repartitioning
> > >> >> - Hadoop 2.0 compatibility
> > >> >> - Hama documentation book in html
> > >> >> - Bug fixes if any reported and few code improvements.
> > >> > adding the streaming API.
> > >>
> > >
> > > +1 for this roadmap.
> > > I'd like to put some machine learning algorithm in there but I'm not
> sure
> > > about the timing for it.
> > > Regards,
> > > Tommaso
> > >
> > >
> > >>
> > >> I'll work on yarn module and (large input data) re-partitioning issue.
> > >>
> > >> If you need my help such as test on large cluster, Just let me know.
> > >>
> > >> On Mon, Jul 9, 2012 at 9:37 PM, Thomas Jungblut
> > >> <th...@gmail.com> wrote:
> > >> > Suraj's plan would also be my favorite.
> > >> > We also could split two faster releases from fault tolerance and
> > Hadoop
> > >> 2.0.
> > >> >
> > >> > If we have sorted out the fault tolerance I would like to spend my
> > time
> > >> on
> > >> > adding the streaming API.
> > >> >
> > >> > 2012/7/9 Suraj Menon <su...@apache.org>
> > >> >
> > >> >> Sorry, I was looking at more time (~6 months) for the next release.
> > This
> > >> >> implies that are we are looking for a release in August end. The
> > desired
> > >> >> feature set:
> > >> >>
> > >> >> - Fault tolerance
> > >> >> - Data repartitioning
> > >> >> - Hadoop 2.0 compatibility
> > >> >> - Hama documentation book in html
> > >> >> - Bug fixes if any reported and few code improvements.
> > >> >>
> > >> >> I have to work on my Jenkins permissions. The last time I tried, it
> > did
> > >> not
> > >> >> work for me.
> > >> >>
> > >> >> -Suraj
> > >> >>
> > >> >> On Mon, Jul 9, 2012 at 6:02 AM, Edward J. Yoon <
> > edward.yoon@oracle.com
> > >> >> >wrote:
> > >> >>
> > >> >> > We only need to discuss about 0.6 release roadmap. Any committer
> > may
> > >> >> > create a release candidate. Please feel free to create your own
> RC
> > and
> > >> >> open
> > >> >> > the vote in future.
> > >> >> >
> > >> >> > Personally, I would like to release along with support for Hadoop
> > 2.0,
> > >> >> > ASAP. Hope I can finish by this end of month or next month.
> > >> >> >
> > >> >> > And IMO, the current high-priority tasks are Input
> re-partitioning
> > and
> > >> >> > exception handling.
> > >> >> >
> > >> >> > Thanks,
> > >> >> > Ed
> > >> >> >
> > >> >> >
> > >> >> > On 7/9/2012 5:15 PM, Suraj Menon wrote:
> > >> >> >
> > >> >> >> Hello,
> > >> >> >>
> > >> >> >> Fault tolerance patch should be in this week.
> > >> >> >>
> > >> >> >> The other features I want to throw for consideration are written
> > >> below.
> > >> >> >> Please feel free to prioritize for 0.6:
> > >> >> >>
> > >> >> >> BSPPeer API Improvements
> > >> >> >> - Broadcast API
> > >> >> >> We have to experiment a broadcast API that does not use the
> > >> Messaging
> > >> >> >> queue. Few strategies for the purpose could be brainstormed.
> > >> >> >> - HAMA-546
> > >> >> >>
> > >> >> >> Task Scheduler Improvements
> > >> >> >> - Task Allocation as feature of BSP job instead of system
> (Coming
> > >> with
> > >> >> the
> > >> >> >> Fault tolerance patch) Today the task allocation responsibility
> is
> > >> >> wrongly
> > >> >> >> implemented in TaskInProgress class. With this feature we can
> have
> > >> >> >> pluggable task allocation strategy with data-local allocation as
> > >> default
> > >> >> >> strategy.
> > >> >> >> - To support real time jobs, we would need to implement resource
> > >> local
> > >> >> >> scheduler, where resource here is more than the file input
> split.
> > >> Based
> > >> >> on
> > >> >> >> the nature of job, it could be nodes with bigger pipe for
> > >> communication,
> > >> >> >> nature of the machine(GPU), Nodes with SSDs, etc.
> > >> >> >>
> > >> >> >> Job Submission Improvements
> > >> >> >> - A Job Submission API, that gives users the flexibility to
> define
> > >> the
> > >> >> >> nature of messaging queue and other attributes per superstep
> > >> >> >>
> > >> >> >> Improve Performance And Scalability
> > >> >> >> - Benchmark on more nodes
> > >> >> >> - Try Asynchronous communication
> > >> >> >> - HAMA-593
> > >> >> >> - Spilling message queue and External Sorting Queue.
> > >> >> >>
> > >> >> >> Apart from this, there could be more graph algorithms to be
> > >> implemented,
> > >> >> >> ml
> > >> >> >> algorithms to be implemented and Mikalai's work integrated in
> the
> > >> >> release.
> > >> >> >> When are we planning to have next release?
> > >> >> >>
> > >> >> >> I can take a shot at release management, but I would definitely
> > need
> > >> >> help
> > >> >> >> for the first time.
> > >> >> >>
> > >> >> >> Thanks,
> > >> >> >> Suraj
> > >> >> >>
> > >> >> >> On Mon, Jul 9, 2012 at 3:22 AM, Thomas Jungblut
> > >> >> >> <th...@gmail.com>**wrote:
> > >> >> >>
> > >> >> >> So now we released 0.5.0 what are our plans for 0.6.0?
> > >> >> >>> And who is going to do the release managing?
> > >> >> >>>
> > >> >> >>>
> > >> >> > --
> > >> >> > Best Regards, Edward J. Yoon
> > >> >> > @eddieyoon
> > >> >> >
> > >> >> >
> > >> >>
> > >>
> > >>
> > >>
> > >> --
> > >> Best Regards, Edward J. Yoon
> > >> @eddieyoon
> > >>
> >
> >
> >
> > --
> > Best Regards, Edward J. Yoon
> > @eddieyoon
> >
>
Re: Roadmap and Release Manager
Posted by Tommaso Teofili <to...@gmail.com>.
same here, I'll most likely submit my work w.r.t. ML module in September.
Tommaso
2012/8/9 Edward J. Yoon <ed...@apache.org>
> >> - Data repartitioning
> >> - Hadoop 2.0 compatibility
>
> P.S., My summer vacation will start from Aug 11th, so I need more time.
>
> On Mon, Jul 9, 2012 at 10:49 PM, Tommaso Teofili
> <to...@gmail.com> wrote:
> > 2012/7/9 Edward J. Yoon <ed...@apache.org>
> >
> >> Then, let's try to release 0.6 by end of August or September 2012. I
> >> heard/saw some CDH4 plans/activities, we should not hesitate. :)
> >>
> >> Our TODO list:
> >>
> >> >> - Fault tolerance
> >> >> - Data repartitioning
> >> >> - Hadoop 2.0 compatibility
> >> >> - Hama documentation book in html
> >> >> - Bug fixes if any reported and few code improvements.
> >> > adding the streaming API.
> >>
> >
> > +1 for this roadmap.
> > I'd like to put some machine learning algorithm in there but I'm not sure
> > about the timing for it.
> > Regards,
> > Tommaso
> >
> >
> >>
> >> I'll work on yarn module and (large input data) re-partitioning issue.
> >>
> >> If you need my help such as test on large cluster, Just let me know.
> >>
> >> On Mon, Jul 9, 2012 at 9:37 PM, Thomas Jungblut
> >> <th...@gmail.com> wrote:
> >> > Suraj's plan would also be my favorite.
> >> > We also could split two faster releases from fault tolerance and
> Hadoop
> >> 2.0.
> >> >
> >> > If we have sorted out the fault tolerance I would like to spend my
> time
> >> on
> >> > adding the streaming API.
> >> >
> >> > 2012/7/9 Suraj Menon <su...@apache.org>
> >> >
> >> >> Sorry, I was looking at more time (~6 months) for the next release.
> This
> >> >> implies that are we are looking for a release in August end. The
> desired
> >> >> feature set:
> >> >>
> >> >> - Fault tolerance
> >> >> - Data repartitioning
> >> >> - Hadoop 2.0 compatibility
> >> >> - Hama documentation book in html
> >> >> - Bug fixes if any reported and few code improvements.
> >> >>
> >> >> I have to work on my Jenkins permissions. The last time I tried, it
> did
> >> not
> >> >> work for me.
> >> >>
> >> >> -Suraj
> >> >>
> >> >> On Mon, Jul 9, 2012 at 6:02 AM, Edward J. Yoon <
> edward.yoon@oracle.com
> >> >> >wrote:
> >> >>
> >> >> > We only need to discuss about 0.6 release roadmap. Any committer
> may
> >> >> > create a release candidate. Please feel free to create your own RC
> and
> >> >> open
> >> >> > the vote in future.
> >> >> >
> >> >> > Personally, I would like to release along with support for Hadoop
> 2.0,
> >> >> > ASAP. Hope I can finish by this end of month or next month.
> >> >> >
> >> >> > And IMO, the current high-priority tasks are Input re-partitioning
> and
> >> >> > exception handling.
> >> >> >
> >> >> > Thanks,
> >> >> > Ed
> >> >> >
> >> >> >
> >> >> > On 7/9/2012 5:15 PM, Suraj Menon wrote:
> >> >> >
> >> >> >> Hello,
> >> >> >>
> >> >> >> Fault tolerance patch should be in this week.
> >> >> >>
> >> >> >> The other features I want to throw for consideration are written
> >> below.
> >> >> >> Please feel free to prioritize for 0.6:
> >> >> >>
> >> >> >> BSPPeer API Improvements
> >> >> >> - Broadcast API
> >> >> >> We have to experiment a broadcast API that does not use the
> >> Messaging
> >> >> >> queue. Few strategies for the purpose could be brainstormed.
> >> >> >> - HAMA-546
> >> >> >>
> >> >> >> Task Scheduler Improvements
> >> >> >> - Task Allocation as feature of BSP job instead of system (Coming
> >> with
> >> >> the
> >> >> >> Fault tolerance patch) Today the task allocation responsibility is
> >> >> wrongly
> >> >> >> implemented in TaskInProgress class. With this feature we can have
> >> >> >> pluggable task allocation strategy with data-local allocation as
> >> default
> >> >> >> strategy.
> >> >> >> - To support real time jobs, we would need to implement resource
> >> local
> >> >> >> scheduler, where resource here is more than the file input split.
> >> Based
> >> >> on
> >> >> >> the nature of job, it could be nodes with bigger pipe for
> >> communication,
> >> >> >> nature of the machine(GPU), Nodes with SSDs, etc.
> >> >> >>
> >> >> >> Job Submission Improvements
> >> >> >> - A Job Submission API, that gives users the flexibility to define
> >> the
> >> >> >> nature of messaging queue and other attributes per superstep
> >> >> >>
> >> >> >> Improve Performance And Scalability
> >> >> >> - Benchmark on more nodes
> >> >> >> - Try Asynchronous communication
> >> >> >> - HAMA-593
> >> >> >> - Spilling message queue and External Sorting Queue.
> >> >> >>
> >> >> >> Apart from this, there could be more graph algorithms to be
> >> implemented,
> >> >> >> ml
> >> >> >> algorithms to be implemented and Mikalai's work integrated in the
> >> >> release.
> >> >> >> When are we planning to have next release?
> >> >> >>
> >> >> >> I can take a shot at release management, but I would definitely
> need
> >> >> help
> >> >> >> for the first time.
> >> >> >>
> >> >> >> Thanks,
> >> >> >> Suraj
> >> >> >>
> >> >> >> On Mon, Jul 9, 2012 at 3:22 AM, Thomas Jungblut
> >> >> >> <th...@gmail.com>**wrote:
> >> >> >>
> >> >> >> So now we released 0.5.0 what are our plans for 0.6.0?
> >> >> >>> And who is going to do the release managing?
> >> >> >>>
> >> >> >>>
> >> >> > --
> >> >> > Best Regards, Edward J. Yoon
> >> >> > @eddieyoon
> >> >> >
> >> >> >
> >> >>
> >>
> >>
> >>
> >> --
> >> Best Regards, Edward J. Yoon
> >> @eddieyoon
> >>
>
>
>
> --
> Best Regards, Edward J. Yoon
> @eddieyoon
>
Re: Roadmap and Release Manager
Posted by "Edward J. Yoon" <ed...@apache.org>.
>> - Data repartitioning
>> - Hadoop 2.0 compatibility
P.S., My summer vacation will start from Aug 11th, so I need more time.
On Mon, Jul 9, 2012 at 10:49 PM, Tommaso Teofili
<to...@gmail.com> wrote:
> 2012/7/9 Edward J. Yoon <ed...@apache.org>
>
>> Then, let's try to release 0.6 by end of August or September 2012. I
>> heard/saw some CDH4 plans/activities, we should not hesitate. :)
>>
>> Our TODO list:
>>
>> >> - Fault tolerance
>> >> - Data repartitioning
>> >> - Hadoop 2.0 compatibility
>> >> - Hama documentation book in html
>> >> - Bug fixes if any reported and few code improvements.
>> > adding the streaming API.
>>
>
> +1 for this roadmap.
> I'd like to put some machine learning algorithm in there but I'm not sure
> about the timing for it.
> Regards,
> Tommaso
>
>
>>
>> I'll work on yarn module and (large input data) re-partitioning issue.
>>
>> If you need my help such as test on large cluster, Just let me know.
>>
>> On Mon, Jul 9, 2012 at 9:37 PM, Thomas Jungblut
>> <th...@gmail.com> wrote:
>> > Suraj's plan would also be my favorite.
>> > We also could split two faster releases from fault tolerance and Hadoop
>> 2.0.
>> >
>> > If we have sorted out the fault tolerance I would like to spend my time
>> on
>> > adding the streaming API.
>> >
>> > 2012/7/9 Suraj Menon <su...@apache.org>
>> >
>> >> Sorry, I was looking at more time (~6 months) for the next release. This
>> >> implies that are we are looking for a release in August end. The desired
>> >> feature set:
>> >>
>> >> - Fault tolerance
>> >> - Data repartitioning
>> >> - Hadoop 2.0 compatibility
>> >> - Hama documentation book in html
>> >> - Bug fixes if any reported and few code improvements.
>> >>
>> >> I have to work on my Jenkins permissions. The last time I tried, it did
>> not
>> >> work for me.
>> >>
>> >> -Suraj
>> >>
>> >> On Mon, Jul 9, 2012 at 6:02 AM, Edward J. Yoon <edward.yoon@oracle.com
>> >> >wrote:
>> >>
>> >> > We only need to discuss about 0.6 release roadmap. Any committer may
>> >> > create a release candidate. Please feel free to create your own RC and
>> >> open
>> >> > the vote in future.
>> >> >
>> >> > Personally, I would like to release along with support for Hadoop 2.0,
>> >> > ASAP. Hope I can finish by this end of month or next month.
>> >> >
>> >> > And IMO, the current high-priority tasks are Input re-partitioning and
>> >> > exception handling.
>> >> >
>> >> > Thanks,
>> >> > Ed
>> >> >
>> >> >
>> >> > On 7/9/2012 5:15 PM, Suraj Menon wrote:
>> >> >
>> >> >> Hello,
>> >> >>
>> >> >> Fault tolerance patch should be in this week.
>> >> >>
>> >> >> The other features I want to throw for consideration are written
>> below.
>> >> >> Please feel free to prioritize for 0.6:
>> >> >>
>> >> >> BSPPeer API Improvements
>> >> >> - Broadcast API
>> >> >> We have to experiment a broadcast API that does not use the
>> Messaging
>> >> >> queue. Few strategies for the purpose could be brainstormed.
>> >> >> - HAMA-546
>> >> >>
>> >> >> Task Scheduler Improvements
>> >> >> - Task Allocation as feature of BSP job instead of system (Coming
>> with
>> >> the
>> >> >> Fault tolerance patch) Today the task allocation responsibility is
>> >> wrongly
>> >> >> implemented in TaskInProgress class. With this feature we can have
>> >> >> pluggable task allocation strategy with data-local allocation as
>> default
>> >> >> strategy.
>> >> >> - To support real time jobs, we would need to implement resource
>> local
>> >> >> scheduler, where resource here is more than the file input split.
>> Based
>> >> on
>> >> >> the nature of job, it could be nodes with bigger pipe for
>> communication,
>> >> >> nature of the machine(GPU), Nodes with SSDs, etc.
>> >> >>
>> >> >> Job Submission Improvements
>> >> >> - A Job Submission API, that gives users the flexibility to define
>> the
>> >> >> nature of messaging queue and other attributes per superstep
>> >> >>
>> >> >> Improve Performance And Scalability
>> >> >> - Benchmark on more nodes
>> >> >> - Try Asynchronous communication
>> >> >> - HAMA-593
>> >> >> - Spilling message queue and External Sorting Queue.
>> >> >>
>> >> >> Apart from this, there could be more graph algorithms to be
>> implemented,
>> >> >> ml
>> >> >> algorithms to be implemented and Mikalai's work integrated in the
>> >> release.
>> >> >> When are we planning to have next release?
>> >> >>
>> >> >> I can take a shot at release management, but I would definitely need
>> >> help
>> >> >> for the first time.
>> >> >>
>> >> >> Thanks,
>> >> >> Suraj
>> >> >>
>> >> >> On Mon, Jul 9, 2012 at 3:22 AM, Thomas Jungblut
>> >> >> <th...@gmail.com>**wrote:
>> >> >>
>> >> >> So now we released 0.5.0 what are our plans for 0.6.0?
>> >> >>> And who is going to do the release managing?
>> >> >>>
>> >> >>>
>> >> > --
>> >> > Best Regards, Edward J. Yoon
>> >> > @eddieyoon
>> >> >
>> >> >
>> >>
>>
>>
>>
>> --
>> Best Regards, Edward J. Yoon
>> @eddieyoon
>>
--
Best Regards, Edward J. Yoon
@eddieyoon
Re: Roadmap and Release Manager
Posted by Tommaso Teofili <to...@gmail.com>.
2012/7/9 Edward J. Yoon <ed...@apache.org>
> Then, let's try to release 0.6 by end of August or September 2012. I
> heard/saw some CDH4 plans/activities, we should not hesitate. :)
>
> Our TODO list:
>
> >> - Fault tolerance
> >> - Data repartitioning
> >> - Hadoop 2.0 compatibility
> >> - Hama documentation book in html
> >> - Bug fixes if any reported and few code improvements.
> > adding the streaming API.
>
+1 for this roadmap.
I'd like to put some machine learning algorithm in there but I'm not sure
about the timing for it.
Regards,
Tommaso
>
> I'll work on yarn module and (large input data) re-partitioning issue.
>
> If you need my help such as test on large cluster, Just let me know.
>
> On Mon, Jul 9, 2012 at 9:37 PM, Thomas Jungblut
> <th...@gmail.com> wrote:
> > Suraj's plan would also be my favorite.
> > We also could split two faster releases from fault tolerance and Hadoop
> 2.0.
> >
> > If we have sorted out the fault tolerance I would like to spend my time
> on
> > adding the streaming API.
> >
> > 2012/7/9 Suraj Menon <su...@apache.org>
> >
> >> Sorry, I was looking at more time (~6 months) for the next release. This
> >> implies that are we are looking for a release in August end. The desired
> >> feature set:
> >>
> >> - Fault tolerance
> >> - Data repartitioning
> >> - Hadoop 2.0 compatibility
> >> - Hama documentation book in html
> >> - Bug fixes if any reported and few code improvements.
> >>
> >> I have to work on my Jenkins permissions. The last time I tried, it did
> not
> >> work for me.
> >>
> >> -Suraj
> >>
> >> On Mon, Jul 9, 2012 at 6:02 AM, Edward J. Yoon <edward.yoon@oracle.com
> >> >wrote:
> >>
> >> > We only need to discuss about 0.6 release roadmap. Any committer may
> >> > create a release candidate. Please feel free to create your own RC and
> >> open
> >> > the vote in future.
> >> >
> >> > Personally, I would like to release along with support for Hadoop 2.0,
> >> > ASAP. Hope I can finish by this end of month or next month.
> >> >
> >> > And IMO, the current high-priority tasks are Input re-partitioning and
> >> > exception handling.
> >> >
> >> > Thanks,
> >> > Ed
> >> >
> >> >
> >> > On 7/9/2012 5:15 PM, Suraj Menon wrote:
> >> >
> >> >> Hello,
> >> >>
> >> >> Fault tolerance patch should be in this week.
> >> >>
> >> >> The other features I want to throw for consideration are written
> below.
> >> >> Please feel free to prioritize for 0.6:
> >> >>
> >> >> BSPPeer API Improvements
> >> >> - Broadcast API
> >> >> We have to experiment a broadcast API that does not use the
> Messaging
> >> >> queue. Few strategies for the purpose could be brainstormed.
> >> >> - HAMA-546
> >> >>
> >> >> Task Scheduler Improvements
> >> >> - Task Allocation as feature of BSP job instead of system (Coming
> with
> >> the
> >> >> Fault tolerance patch) Today the task allocation responsibility is
> >> wrongly
> >> >> implemented in TaskInProgress class. With this feature we can have
> >> >> pluggable task allocation strategy with data-local allocation as
> default
> >> >> strategy.
> >> >> - To support real time jobs, we would need to implement resource
> local
> >> >> scheduler, where resource here is more than the file input split.
> Based
> >> on
> >> >> the nature of job, it could be nodes with bigger pipe for
> communication,
> >> >> nature of the machine(GPU), Nodes with SSDs, etc.
> >> >>
> >> >> Job Submission Improvements
> >> >> - A Job Submission API, that gives users the flexibility to define
> the
> >> >> nature of messaging queue and other attributes per superstep
> >> >>
> >> >> Improve Performance And Scalability
> >> >> - Benchmark on more nodes
> >> >> - Try Asynchronous communication
> >> >> - HAMA-593
> >> >> - Spilling message queue and External Sorting Queue.
> >> >>
> >> >> Apart from this, there could be more graph algorithms to be
> implemented,
> >> >> ml
> >> >> algorithms to be implemented and Mikalai's work integrated in the
> >> release.
> >> >> When are we planning to have next release?
> >> >>
> >> >> I can take a shot at release management, but I would definitely need
> >> help
> >> >> for the first time.
> >> >>
> >> >> Thanks,
> >> >> Suraj
> >> >>
> >> >> On Mon, Jul 9, 2012 at 3:22 AM, Thomas Jungblut
> >> >> <th...@gmail.com>**wrote:
> >> >>
> >> >> So now we released 0.5.0 what are our plans for 0.6.0?
> >> >>> And who is going to do the release managing?
> >> >>>
> >> >>>
> >> > --
> >> > Best Regards, Edward J. Yoon
> >> > @eddieyoon
> >> >
> >> >
> >>
>
>
>
> --
> Best Regards, Edward J. Yoon
> @eddieyoon
>
Re: Roadmap and Release Manager
Posted by "Edward J. Yoon" <ed...@apache.org>.
Then, let's try to release 0.6 by end of August or September 2012. I
heard/saw some CDH4 plans/activities, we should not hesitate. :)
Our TODO list:
>> - Fault tolerance
>> - Data repartitioning
>> - Hadoop 2.0 compatibility
>> - Hama documentation book in html
>> - Bug fixes if any reported and few code improvements.
> adding the streaming API.
I'll work on yarn module and (large input data) re-partitioning issue.
If you need my help such as test on large cluster, Just let me know.
On Mon, Jul 9, 2012 at 9:37 PM, Thomas Jungblut
<th...@gmail.com> wrote:
> Suraj's plan would also be my favorite.
> We also could split two faster releases from fault tolerance and Hadoop 2.0.
>
> If we have sorted out the fault tolerance I would like to spend my time on
> adding the streaming API.
>
> 2012/7/9 Suraj Menon <su...@apache.org>
>
>> Sorry, I was looking at more time (~6 months) for the next release. This
>> implies that are we are looking for a release in August end. The desired
>> feature set:
>>
>> - Fault tolerance
>> - Data repartitioning
>> - Hadoop 2.0 compatibility
>> - Hama documentation book in html
>> - Bug fixes if any reported and few code improvements.
>>
>> I have to work on my Jenkins permissions. The last time I tried, it did not
>> work for me.
>>
>> -Suraj
>>
>> On Mon, Jul 9, 2012 at 6:02 AM, Edward J. Yoon <edward.yoon@oracle.com
>> >wrote:
>>
>> > We only need to discuss about 0.6 release roadmap. Any committer may
>> > create a release candidate. Please feel free to create your own RC and
>> open
>> > the vote in future.
>> >
>> > Personally, I would like to release along with support for Hadoop 2.0,
>> > ASAP. Hope I can finish by this end of month or next month.
>> >
>> > And IMO, the current high-priority tasks are Input re-partitioning and
>> > exception handling.
>> >
>> > Thanks,
>> > Ed
>> >
>> >
>> > On 7/9/2012 5:15 PM, Suraj Menon wrote:
>> >
>> >> Hello,
>> >>
>> >> Fault tolerance patch should be in this week.
>> >>
>> >> The other features I want to throw for consideration are written below.
>> >> Please feel free to prioritize for 0.6:
>> >>
>> >> BSPPeer API Improvements
>> >> - Broadcast API
>> >> We have to experiment a broadcast API that does not use the Messaging
>> >> queue. Few strategies for the purpose could be brainstormed.
>> >> - HAMA-546
>> >>
>> >> Task Scheduler Improvements
>> >> - Task Allocation as feature of BSP job instead of system (Coming with
>> the
>> >> Fault tolerance patch) Today the task allocation responsibility is
>> wrongly
>> >> implemented in TaskInProgress class. With this feature we can have
>> >> pluggable task allocation strategy with data-local allocation as default
>> >> strategy.
>> >> - To support real time jobs, we would need to implement resource local
>> >> scheduler, where resource here is more than the file input split. Based
>> on
>> >> the nature of job, it could be nodes with bigger pipe for communication,
>> >> nature of the machine(GPU), Nodes with SSDs, etc.
>> >>
>> >> Job Submission Improvements
>> >> - A Job Submission API, that gives users the flexibility to define the
>> >> nature of messaging queue and other attributes per superstep
>> >>
>> >> Improve Performance And Scalability
>> >> - Benchmark on more nodes
>> >> - Try Asynchronous communication
>> >> - HAMA-593
>> >> - Spilling message queue and External Sorting Queue.
>> >>
>> >> Apart from this, there could be more graph algorithms to be implemented,
>> >> ml
>> >> algorithms to be implemented and Mikalai's work integrated in the
>> release.
>> >> When are we planning to have next release?
>> >>
>> >> I can take a shot at release management, but I would definitely need
>> help
>> >> for the first time.
>> >>
>> >> Thanks,
>> >> Suraj
>> >>
>> >> On Mon, Jul 9, 2012 at 3:22 AM, Thomas Jungblut
>> >> <th...@gmail.com>**wrote:
>> >>
>> >> So now we released 0.5.0 what are our plans for 0.6.0?
>> >>> And who is going to do the release managing?
>> >>>
>> >>>
>> > --
>> > Best Regards, Edward J. Yoon
>> > @eddieyoon
>> >
>> >
>>
--
Best Regards, Edward J. Yoon
@eddieyoon
Re: Roadmap and Release Manager
Posted by Thomas Jungblut <th...@gmail.com>.
Suraj's plan would also be my favorite.
We also could split two faster releases from fault tolerance and Hadoop 2.0.
If we have sorted out the fault tolerance I would like to spend my time on
adding the streaming API.
2012/7/9 Suraj Menon <su...@apache.org>
> Sorry, I was looking at more time (~6 months) for the next release. This
> implies that are we are looking for a release in August end. The desired
> feature set:
>
> - Fault tolerance
> - Data repartitioning
> - Hadoop 2.0 compatibility
> - Hama documentation book in html
> - Bug fixes if any reported and few code improvements.
>
> I have to work on my Jenkins permissions. The last time I tried, it did not
> work for me.
>
> -Suraj
>
> On Mon, Jul 9, 2012 at 6:02 AM, Edward J. Yoon <edward.yoon@oracle.com
> >wrote:
>
> > We only need to discuss about 0.6 release roadmap. Any committer may
> > create a release candidate. Please feel free to create your own RC and
> open
> > the vote in future.
> >
> > Personally, I would like to release along with support for Hadoop 2.0,
> > ASAP. Hope I can finish by this end of month or next month.
> >
> > And IMO, the current high-priority tasks are Input re-partitioning and
> > exception handling.
> >
> > Thanks,
> > Ed
> >
> >
> > On 7/9/2012 5:15 PM, Suraj Menon wrote:
> >
> >> Hello,
> >>
> >> Fault tolerance patch should be in this week.
> >>
> >> The other features I want to throw for consideration are written below.
> >> Please feel free to prioritize for 0.6:
> >>
> >> BSPPeer API Improvements
> >> - Broadcast API
> >> We have to experiment a broadcast API that does not use the Messaging
> >> queue. Few strategies for the purpose could be brainstormed.
> >> - HAMA-546
> >>
> >> Task Scheduler Improvements
> >> - Task Allocation as feature of BSP job instead of system (Coming with
> the
> >> Fault tolerance patch) Today the task allocation responsibility is
> wrongly
> >> implemented in TaskInProgress class. With this feature we can have
> >> pluggable task allocation strategy with data-local allocation as default
> >> strategy.
> >> - To support real time jobs, we would need to implement resource local
> >> scheduler, where resource here is more than the file input split. Based
> on
> >> the nature of job, it could be nodes with bigger pipe for communication,
> >> nature of the machine(GPU), Nodes with SSDs, etc.
> >>
> >> Job Submission Improvements
> >> - A Job Submission API, that gives users the flexibility to define the
> >> nature of messaging queue and other attributes per superstep
> >>
> >> Improve Performance And Scalability
> >> - Benchmark on more nodes
> >> - Try Asynchronous communication
> >> - HAMA-593
> >> - Spilling message queue and External Sorting Queue.
> >>
> >> Apart from this, there could be more graph algorithms to be implemented,
> >> ml
> >> algorithms to be implemented and Mikalai's work integrated in the
> release.
> >> When are we planning to have next release?
> >>
> >> I can take a shot at release management, but I would definitely need
> help
> >> for the first time.
> >>
> >> Thanks,
> >> Suraj
> >>
> >> On Mon, Jul 9, 2012 at 3:22 AM, Thomas Jungblut
> >> <th...@gmail.com>**wrote:
> >>
> >> So now we released 0.5.0 what are our plans for 0.6.0?
> >>> And who is going to do the release managing?
> >>>
> >>>
> > --
> > Best Regards, Edward J. Yoon
> > @eddieyoon
> >
> >
>
Re: Roadmap and Release Manager
Posted by Suraj Menon <su...@apache.org>.
Sorry, I was looking at more time (~6 months) for the next release. This
implies that are we are looking for a release in August end. The desired
feature set:
- Fault tolerance
- Data repartitioning
- Hadoop 2.0 compatibility
- Hama documentation book in html
- Bug fixes if any reported and few code improvements.
I have to work on my Jenkins permissions. The last time I tried, it did not
work for me.
-Suraj
On Mon, Jul 9, 2012 at 6:02 AM, Edward J. Yoon <ed...@oracle.com>wrote:
> We only need to discuss about 0.6 release roadmap. Any committer may
> create a release candidate. Please feel free to create your own RC and open
> the vote in future.
>
> Personally, I would like to release along with support for Hadoop 2.0,
> ASAP. Hope I can finish by this end of month or next month.
>
> And IMO, the current high-priority tasks are Input re-partitioning and
> exception handling.
>
> Thanks,
> Ed
>
>
> On 7/9/2012 5:15 PM, Suraj Menon wrote:
>
>> Hello,
>>
>> Fault tolerance patch should be in this week.
>>
>> The other features I want to throw for consideration are written below.
>> Please feel free to prioritize for 0.6:
>>
>> BSPPeer API Improvements
>> - Broadcast API
>> We have to experiment a broadcast API that does not use the Messaging
>> queue. Few strategies for the purpose could be brainstormed.
>> - HAMA-546
>>
>> Task Scheduler Improvements
>> - Task Allocation as feature of BSP job instead of system (Coming with the
>> Fault tolerance patch) Today the task allocation responsibility is wrongly
>> implemented in TaskInProgress class. With this feature we can have
>> pluggable task allocation strategy with data-local allocation as default
>> strategy.
>> - To support real time jobs, we would need to implement resource local
>> scheduler, where resource here is more than the file input split. Based on
>> the nature of job, it could be nodes with bigger pipe for communication,
>> nature of the machine(GPU), Nodes with SSDs, etc.
>>
>> Job Submission Improvements
>> - A Job Submission API, that gives users the flexibility to define the
>> nature of messaging queue and other attributes per superstep
>>
>> Improve Performance And Scalability
>> - Benchmark on more nodes
>> - Try Asynchronous communication
>> - HAMA-593
>> - Spilling message queue and External Sorting Queue.
>>
>> Apart from this, there could be more graph algorithms to be implemented,
>> ml
>> algorithms to be implemented and Mikalai's work integrated in the release.
>> When are we planning to have next release?
>>
>> I can take a shot at release management, but I would definitely need help
>> for the first time.
>>
>> Thanks,
>> Suraj
>>
>> On Mon, Jul 9, 2012 at 3:22 AM, Thomas Jungblut
>> <th...@gmail.com>**wrote:
>>
>> So now we released 0.5.0 what are our plans for 0.6.0?
>>> And who is going to do the release managing?
>>>
>>>
> --
> Best Regards, Edward J. Yoon
> @eddieyoon
>
>
Re: Roadmap and Release Manager
Posted by "Edward J. Yoon" <ed...@oracle.com>.
We only need to discuss about 0.6 release roadmap. Any committer may
create a release candidate. Please feel free to create your own RC and
open the vote in future.
Personally, I would like to release along with support for Hadoop 2.0,
ASAP. Hope I can finish by this end of month or next month.
And IMO, the current high-priority tasks are Input re-partitioning and
exception handling.
Thanks,
Ed
On 7/9/2012 5:15 PM, Suraj Menon wrote:
> Hello,
>
> Fault tolerance patch should be in this week.
>
> The other features I want to throw for consideration are written below.
> Please feel free to prioritize for 0.6:
>
> BSPPeer API Improvements
> - Broadcast API
> We have to experiment a broadcast API that does not use the Messaging
> queue. Few strategies for the purpose could be brainstormed.
> - HAMA-546
>
> Task Scheduler Improvements
> - Task Allocation as feature of BSP job instead of system (Coming with the
> Fault tolerance patch) Today the task allocation responsibility is wrongly
> implemented in TaskInProgress class. With this feature we can have
> pluggable task allocation strategy with data-local allocation as default
> strategy.
> - To support real time jobs, we would need to implement resource local
> scheduler, where resource here is more than the file input split. Based on
> the nature of job, it could be nodes with bigger pipe for communication,
> nature of the machine(GPU), Nodes with SSDs, etc.
>
> Job Submission Improvements
> - A Job Submission API, that gives users the flexibility to define the
> nature of messaging queue and other attributes per superstep
>
> Improve Performance And Scalability
> - Benchmark on more nodes
> - Try Asynchronous communication
> - HAMA-593
> - Spilling message queue and External Sorting Queue.
>
> Apart from this, there could be more graph algorithms to be implemented, ml
> algorithms to be implemented and Mikalai's work integrated in the release.
> When are we planning to have next release?
>
> I can take a shot at release management, but I would definitely need help
> for the first time.
>
> Thanks,
> Suraj
>
> On Mon, Jul 9, 2012 at 3:22 AM, Thomas Jungblut
> <th...@gmail.com>wrote:
>
>> So now we released 0.5.0 what are our plans for 0.6.0?
>> And who is going to do the release managing?
>>
--
Best Regards, Edward J. Yoon
@eddieyoon
Re: Roadmap and Release Manager
Posted by Suraj Menon <su...@apache.org>.
Hello,
Fault tolerance patch should be in this week.
The other features I want to throw for consideration are written below.
Please feel free to prioritize for 0.6:
BSPPeer API Improvements
- Broadcast API
We have to experiment a broadcast API that does not use the Messaging
queue. Few strategies for the purpose could be brainstormed.
- HAMA-546
Task Scheduler Improvements
- Task Allocation as feature of BSP job instead of system (Coming with the
Fault tolerance patch) Today the task allocation responsibility is wrongly
implemented in TaskInProgress class. With this feature we can have
pluggable task allocation strategy with data-local allocation as default
strategy.
- To support real time jobs, we would need to implement resource local
scheduler, where resource here is more than the file input split. Based on
the nature of job, it could be nodes with bigger pipe for communication,
nature of the machine(GPU), Nodes with SSDs, etc.
Job Submission Improvements
- A Job Submission API, that gives users the flexibility to define the
nature of messaging queue and other attributes per superstep
Improve Performance And Scalability
- Benchmark on more nodes
- Try Asynchronous communication
- HAMA-593
- Spilling message queue and External Sorting Queue.
Apart from this, there could be more graph algorithms to be implemented, ml
algorithms to be implemented and Mikalai's work integrated in the release.
When are we planning to have next release?
I can take a shot at release management, but I would definitely need help
for the first time.
Thanks,
Suraj
On Mon, Jul 9, 2012 at 3:22 AM, Thomas Jungblut
<th...@gmail.com>wrote:
> So now we released 0.5.0 what are our plans for 0.6.0?
> And who is going to do the release managing?
>