You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@reef.apache.org by Markus Weimer <ma...@weimo.de> on 2015/05/21 17:46:33 UTC
REEF meetup in Seoul
Hi,
I'll be in Seoul on June 11th and 12th for a visit to Gon's lab.
Maybe we can use some of this time to discuss some of the more tricky
issues face to face. Let's use this thread to assemble the list of
topics we want to cover. Here are my initial entrances (centered on
Wake, obviously):
* Wake instantiation: Which classes are singletons, which can't be?
* Non-Singletons and Tang: What is a good pattern? Do we need
something better than injected factories?
* How do we best enable multiplexing of different types across the
same network connection?
* What's so bad about using HTTP(2) for our control flow?
Please add to this list, we can then weed through it and come up with an
agenda.
Let me also state the (Apache) obvious: We should strive to document the
f2f discussions best we can for the benefit of the community members not
in Seoul.
Thanks,
Markus
Re: REEF meetup in Seoul
Posted by Markus Weimer <ma...@weimo.de>.
Hi,
I believe we have quite the list now:
* [REEF-290] Wake instantiation: Which classes are singletons, which
can't be?
* [REEF-???] Non-Singletons and Tang: What is a good pattern? Do we
need something better than injected factories?
* [REEF-181] How do we best enable multiplexing of different types
across the same network connection?
* [REEF-???] What's so bad about using HTTP(2) for our control flow?
* Fixing reef locking issue(s) - REEF-61, 62, 167, 294, Wake clock fix
* [REEF-???] FS output service
* [REEF-42] YARN resource request handling
* [REEF-31] Tang namespaces
* [REEF-???] Tasklets
How do we want to proceed on those? We could for instance have one
person in charge of each topic, which means that they file a JIRA where
none exists and update it with the discussion results for future reference.
What do you think?
Markus
Re: REEF meetup in Seoul
Posted by John Yang <jo...@gmail.com>.
Hi,
Regarding YARN cgroup container support, it seems that this is more about
YARN configuration rather than REEF. I believe with the configuration in
http://blog.cloudera.com/blog/2013/12/managing-multiple-resources-in-hadoop-2-with-yarn/,
the current REEF can use the cgroup cpu isolation(vcores) fine. So I don't
think we need to discuss this topic.
Sorry for the confusion.
Thanks,
John
On May 27, 2015 7:10 PM, "Byung-Gon Chun" <bg...@gmail.com> wrote:
> On Wed, May 27, 2015 at 5:54 AM, Markus Weimer <ma...@weimo.de> wrote:
>
> > Hi,
> >
> >
> > On 2015-05-26 11:01, Byung-Gon Chun wrote:
> > > - FS output service - (this's done in another project and it'd be
> > > good to incorporate it into REEF)
> >
> > FS means filesystem?
> >
>
> Yes.
>
>
> >
> > > Tasklet: run multiple Tasklets (tiny tasks) concurrently inside an
> > > Evaluator, there's a queue to hold Tasklets
> >
> > Is there some documentation on this idea? Is it similar to the Sparrow
> > work over in Spark?
> >
> >
> We don't have a document. It has some similarity.
> The important part is that there's a queue holding tasklets. When one
> tasklet finishes, we should be able to execute the next one without any
> delay. Concurrent execution is an extra feature we target.
>
> I forgot one topic John mentioned.
> - YARN cgroup container support
>
> Thanks.
> -Gon
>
>
>
>
>
> > Thanks,
> >
> > Markus
> >
>
>
>
> --
> Byung-Gon Chun
>
Re: REEF meetup in Seoul
Posted by Byung-Gon Chun <bg...@gmail.com>.
On Wed, May 27, 2015 at 5:54 AM, Markus Weimer <ma...@weimo.de> wrote:
> Hi,
>
>
> On 2015-05-26 11:01, Byung-Gon Chun wrote:
> > - FS output service - (this's done in another project and it'd be
> > good to incorporate it into REEF)
>
> FS means filesystem?
>
Yes.
>
> > Tasklet: run multiple Tasklets (tiny tasks) concurrently inside an
> > Evaluator, there's a queue to hold Tasklets
>
> Is there some documentation on this idea? Is it similar to the Sparrow
> work over in Spark?
>
>
We don't have a document. It has some similarity.
The important part is that there's a queue holding tasklets. When one
tasklet finishes, we should be able to execute the next one without any
delay. Concurrent execution is an extra feature we target.
I forgot one topic John mentioned.
- YARN cgroup container support
Thanks.
-Gon
> Thanks,
>
> Markus
>
--
Byung-Gon Chun
Re: REEF meetup in Seoul
Posted by Markus Weimer <ma...@weimo.de>.
Hi,
On 2015-05-26 11:01, Byung-Gon Chun wrote:
> - FS output service - (this's done in another project and it'd be
> good to incorporate it into REEF)
FS means filesystem?
> Tasklet: run multiple Tasklets (tiny tasks) concurrently inside an
> Evaluator, there's a queue to hold Tasklets
Is there some documentation on this idea? Is it similar to the Sparrow
work over in Spark?
Thanks,
Markus
Re: REEF meetup in Seoul
Posted by Byung-Gon Chun <bg...@gmail.com>.
In addition to the topics, I would like to add the topics my group
discussed.
- Fixing reef locking issue(s) - REEF-61, 62, 167, 294, Wake clock fix
- FS output service - (this's done in another project and it'd be good to
incorporate it into REEF)
- YARN resource request handling - REEF-42, YARN request handling that does
not have allocation rate limiting for each node manager
- Tang namespace(?)
- Tasklet: run multiple Tasklets (tiny tasks) concurrently inside an
Evaluator, there's a queue to hold Tasklets
Thanks.
-Gon
On Fri, May 22, 2015 at 12:52 PM, Byung-Gon Chun <bg...@gmail.com> wrote:
> Sure. I will arrange a video conference call for the REEF .NET part. It's
> late night in PDT. :)
>
> On Fri, May 22, 2015 at 5:48 AM, Markus Weimer <ma...@weimo.de> wrote:
>
>> Hi,
>>
>> On 2015-05-21 20:01, Byung-Gon Chun wrote:
>> > The current (tentative) schedule is as follows.
>> > June 11 - Research chat (private), Deep dive on REEF .NET (public)
>>
>> For that part, can we arrange for a video conference to Redmond? It
>> would be great to have Beysim and Julia patched in as well.
>>
>> Markus
>>
>
>
>
> --
> Byung-Gon Chun
>
--
Byung-Gon Chun
Re: REEF meetup in Seoul
Posted by Byung-Gon Chun <bg...@gmail.com>.
Sure. I will arrange a video conference call for the REEF .NET part. It's
late night in PDT. :)
On Fri, May 22, 2015 at 5:48 AM, Markus Weimer <ma...@weimo.de> wrote:
> Hi,
>
> On 2015-05-21 20:01, Byung-Gon Chun wrote:
> > The current (tentative) schedule is as follows.
> > June 11 - Research chat (private), Deep dive on REEF .NET (public)
>
> For that part, can we arrange for a video conference to Redmond? It
> would be great to have Beysim and Julia patched in as well.
>
> Markus
>
--
Byung-Gon Chun
Re: REEF meetup in Seoul
Posted by Markus Weimer <ma...@weimo.de>.
Hi,
On 2015-05-21 20:01, Byung-Gon Chun wrote:
> The current (tentative) schedule is as follows.
> June 11 - Research chat (private), Deep dive on REEF .NET (public)
For that part, can we arrange for a video conference to Redmond? It
would be great to have Beysim and Julia patched in as well.
Markus
Re: REEF meetup in Seoul
Posted by Byung-Gon Chun <bg...@gmail.com>.
Thanks, Markus. The list covers important (wake-related) topics. :)
My group has also been discussing potential topics to work on in the REEF
hackathon. I will share them with the community once I'm back to Korea.
The current (tentative) schedule is as follows.
June 11 - Research chat (private), Deep dive on REEF .NET (public)
June 12 - REEF hackathon (public)
On Thu, May 21, 2015 at 5:46 PM, Markus Weimer <ma...@weimo.de> wrote:
> Hi,
>
> I'll be in Seoul on June 11th and 12th for a visit to Gon's lab.
>
> Maybe we can use some of this time to discuss some of the more tricky
> issues face to face. Let's use this thread to assemble the list of
> topics we want to cover. Here are my initial entrances (centered on
> Wake, obviously):
>
> * Wake instantiation: Which classes are singletons, which can't be?
> * Non-Singletons and Tang: What is a good pattern? Do we need
> something better than injected factories?
> * How do we best enable multiplexing of different types across the
> same network connection?
> * What's so bad about using HTTP(2) for our control flow?
>
> Please add to this list, we can then weed through it and come up with an
> agenda.
>
> Let me also state the (Apache) obvious: We should strive to document the
> f2f discussions best we can for the benefit of the community members not
> in Seoul.
>
> Thanks,
>
> Markus
>
--
Byung-Gon Chun