You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@bigtop.apache.org by patrick stuedi <ps...@gmail.com> on 2017/07/03 09:44:06 UTC

Crail as part of BigTop

Dear BigTop Devs,

I'm a member of the Big Data / High-Performance I/O team at IBM Research
that develops the Crail open source project (*www.crail.io*
<http://www.crail.io>). Together with PMC member Amir Sanjar we have been
exploring the possibility of donating the Crail code to BigTop and we would
like to hear your thoughts about such a potential integration.

Let me briefly explain what Crail is about: Crail is a platform for storing
(and orchestrating access to) performance critical data in big data
analytics jobs at very high speed. Crail is designed from ground up to
leverage (a) high-speed networking and storage hardware like 100Gbps
networks and NVMe flash, and (b) new modes of data center deployments like
disaggregated data centers, rack-scale deployments and heterogeneous
deployments (e.g., including hardware accelerators), etc. At the same time,
Crail also works in traditional deployments across commodity network and
storage hardware. While currently targeting Linux systems, Crail is
otherwise architecturally independent and has been tested on both x86 and
ppc64 based architectures.

Let me give a few concrete examples of how Crail is used. For instance,
Crail is used to orchestrate access to temporary intra-job data like
shuffle data in Hadoop or Spark or broadcast objects, cached RDDs, etc. in
Spark. Crail is also used to store temporary inter-job data in complex
multi-stage jobs (e.g. a map/reduce job followed by a Spark SQL job). Crail
can generally be used as a storage space for data that is repeatedly
processed like SQL parquet tables.

Today, such data orchestration is done using various techniques. Intra-job
data is typically managed by the compute frameworks through point-to-point
data exchanges implemented using TCP sockets for communication and local
file systems as a temporary storage for the data. For sharing inter-job
data there are systems like Alluxio (formerly Tachyon) or distributed
caches (e.g. Redis). The problem is, each of these solutions can only be
used for one particular type of data or use case (latency sensitive data,
large data, shuffle, etc.). Additionally, new high-performance hardware and
new data center architectures are not leveraged or used very poorly.

Crail, on the other hand, is a platform that implements very efficient data
orchestration for both small and large data sets at the speed of the
hardware, while supporting resource disaggregation and other new forms of
deployments. To give you an idea, Crail–based data shuffling in Spark
reaches a network usage of 80Gbps all-to-all, whereas - on the very same
hardware - the traditional shuffle implementation is stuck at 10-20Gbps or
lower. These improvements at all-to-all data sharing eventually resulted in
a 6x gain at the workload runtime level. A blog post about this can be
found on the Crail website (*http://www.crail.io/blog/2017/01/sorting.html*
<http://www.crail.io/blog/2017/01/sorting.html>). Another example is
storage disaggregation: Crail supports complete disaggregation of memory
and flash for input/output and intra-job data in a Hadoop/Spark/Hive/etc.
job with literally zero performance costs, as shown in our recent Spark
Summit presentation (
*https://www.slideshare.net/databricks/running-apache-spark-on-a-highperformance-cluster-using-rdma-and-nvme-flash-with-patrick-stuedi*
<https://www.slideshare.net/databricks/running-apache-spark-on-a-highperformance-cluster-using-rdma-and-nvme-flash-with-patrick-stuedi>
).

Crail is developed (actively!) under the Apache licence, but it's not yet
an Apache project or an Apache incubator project. However, we would like it
to become one and we are looking for support along that direction. We are
also committed to provide long-term support for Crail and of course we are
very much interested in expanding the community behind Crail. We are
currently a team of about 8 developers.

We think Crail could be beneficial for BigTop as it helps to unleash
performance gains of data processing frameworks on modern high-performance
clusters. Already many of the hardware trends Crail is designed for are
making inroads to data centers. For instance, Amazon offers high-speed
instances with 40Gbps network and NVMe flash. Microsoft Azure offers RDMA
networking as part of their cloud offering. We believe this trend is going
to continue and Crail is the platform that allows these hardware trends to
be consumed by the big data frameworks.

We are looking forward to hear from you guys and get feedback about a
potential integration of Crail with BigTop.

Regards,
Patrick

Re: Crail as part of BigTop

Posted by Animesh Trivedi <an...@gmail.com>.

Hello Everyone,

We have submitted the grant application. Thanks a lot Roman for pointing it
out. The deadline extension was helpful for us to get our application
together.

PS~ I had to re-submit the grant application because I did not receive any
confirmation email even after 24 hours.

Regards,
--
Animesh


On Tue, Jul 11, 2017 at 2:26 PM, Animesh Trivedi <an...@gmail.com>
wrote:

> Hi Roman,
>
> Many thanks for pointing out the grant program. We are certainly
> interested, and barring any unforeseen circumstances, we hope to submit an
> application by the coming Friday !
>
> Thanks,
> --
> Animesh
>
> On Mon, Jul 10, 2017 at 10:10 PM, Roman Shaposhnik <ro...@shaposhnik.org>
> wrote:
>
>> Piling on top of others: I think this also will make an excellent
>> proposal. Any chance you can consider this for:
>>    http://www.prnewswire.com/news-releases/odpi-launches-apach
>> e-bigtop-grant-fund-program-300472753.html
>>
>> ?
>>
>> Thanks,
>> Roman.
>>
>> On Mon, Jul 10, 2017 at 4:15 PM, Animesh Trivedi
>> <an...@gmail.com> wrote:
>> > Hello all,
>> >
>> > many thanks for positive responses. We have created a JIRA for tracking
>> the
>> > progress of Crail and BigTop integration here:
>> >
>> > https://issues.apache.org/jira/browse/BIGTOP-2837
>> >
>> > Any feedback/suggestions/recommendations are appreciated.
>> >
>> > Thanks,
>> > --
>> > Animesh
>> >
>> > On Mon, Jul 10, 2017 at 2:16 PM, MrAsanjar . <af...@gmail.com>
>> wrote:
>> >
>> >> Thanks Cos,
>> >>
>> >>
>> >> On Fri, Jul 7, 2017 at 3:39 PM, Konstantin Boudnik <co...@apache.org>
>> wrote:
>> >>
>> >> > This is a good example [1] of what we were referring to.
>> >> >
>> >> > Thanks,
>> >> >   Cos
>> >> >
>> >> > [1] https://issues.apache.org/jira/browse/BIGTOP-2283
>> >> > --
>> >> >   Take care,
>> >> > Konstantin (Cos) Boudnik
>> >> > 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
>> >> >
>> >> > Disclaimer: Opinions expressed in this email are those of the author,
>> >> > and do not necessarily represent the views of any company the author
>> >> > might be affiliated with at the moment of writing.
>> >> >
>> >> >
>> >> > On Fri, Jul 7, 2017 at 1:28 PM, patrick stuedi <ps...@gmail.com>
>> >> wrote:
>> >> > > Cos, RJ, thanks for the response/advice..what I hear is that
>> opening a
>> >> > JIRA
>> >> > > ticket on the crail integration would be a good starting
>> point..we'll
>> >> do
>> >> > so
>> >> > > and take it from it there..
>> >> > >
>> >> > > On Jul 5, 2017 8:06 PM, "Konstantin Boudnik" <co...@apache.org>
>> wrote:
>> >> > >
>> >> > > Thanks for bringing this to the table, Patrick!
>> >> > >
>> >> > > A couple of comments:
>> >> > >  - adding to what Olaf said earlier: there are two ways of
>> donating a
>> >> > > codebase
>> >> > >    to ASF. One is to go through the Incubator, learn all the ropes
>> and
>> >> > > become
>> >> > >    a TLP. Another one, is by joining an existing TLP as a
>> subproject.
>> >> > > There's
>> >> > >    a number of steps being involved wrt IP clearance, license
>> checks,
>> >> > SGA,
>> >> > >    CCLA, yada-yada. I am not going into more details like this,
>> because
>> >> > > you're
>> >> > >    not considering this path anyway ;)
>> >> > >  - from a quick glance at the project, looks like it fits the
>> general
>> >> > bucket
>> >> > >    of data fabric platforms. Something like Apache Ignite comes to
>> >> mind,
>> >> > > while
>> >> > >    Aluxio, as a simple caching solution, only has partial target
>> >> > >    functionality. While there's no limitation on having multiple
>> >> > components
>> >> > >    with overlapping functionality in a given Bigtop stack (after
>> all,
>> >> we
>> >> > > have
>> >> > >    HDFS and QFS), it's an aspect worthy of some consideration.
>> >> > >  - and to the RJ's point: with source code licensing question out
>> of
>> >> the
>> >> > > way,
>> >> > >    it'd desirable to make sure that Crail doesn't re-distribute
>> >> anything
>> >> > > under
>> >> > >    conflicting licences. The ones in question are of GPL family and
>> >> some
>> >> > > more
>> >> > >    as you can see in [1]. While this isn't much of a concern for
>> >> Bigtop,
>> >> > you
>> >> > >    might benefit from doing this for the project itself.
>> >> > >
>> >> > > Regards,
>> >> > >   Cos
>> >> > >
>> >> > > On Wed, Jul 05, 2017 at 10:25AM, RJ Nowling wrote:
>> >> > >> I believe we already package several other open source projects
>> that
>> >> are
>> >> > >> not Apache projects.  That should not be a hurdle.  We do have
>> >> > > restrictions
>> >> > >> on licenses, but it seems that Crail is licensed under the Apache
>> >> > License:
>> >> > >>
>> >> > >> https://github.com/zrlio/crail/blob/master/LICENSE
>> >> > >>
>> >> > >> I do not believe we have a formal process for accepting /
>> declining.
>> >> > The
>> >> > >> key considerations, as Olaf mentioned, are licensing and
>> willingness
>> >> to
>> >> > >> maintain the packages.  We do have a policy that any packages
>> which
>> >> > break
>> >> > >> and do not have maintainers (or the maintainers do not respond)
>> will
>> >> be
>> >> > >> removed in the following release.
>> >> > >>
>> >> > >> I would suggest creating some JIRA tickets around the integration
>> of
>> >> > Crail
>> >> > >> and your plans for each step.  e.g., creating packages (RPMs,
>> DEBs),
>> >> > >> deploying via our Puppet scripts, and some support in our
>> integration
>> >> > >> testing frameworks.  This would allow other community members to
>> >> comment
>> >> > >> and provide feedback on the technical aspects.
>> >> > >>
>> >> > >>
>> >> > >> On Tue, Jul 4, 2017 at 6:18 AM, patrick stuedi <pstuedi@gmail.com
>> >
>> >> > wrote:
>> >> > >>
>> >> > >> > Olaf,
>> >> > >> >
>> >> > >> > many thanks for the quick reply!
>> >> > >> >
>> >> > >> > let me try to clarify -- the wording "donating the Crail code"
>> was
>> >> > >> > misleading...what we propose is actually to contribute the
>> >> packaging,
>> >> > >> > integration, automation and deployment of Crail to Bigtop, and
>> of
>> >> > course
>> >> > >> > the continuous maintenance of it. Being an optional addon to the
>> >> > regular
>> >> > >> > BigTop stack would be absolutely fine. Question, do you guys
>> have a
>> >> > >> > democratic process where you accept/decline the integration of
>> new
>> >> > > projects
>> >> > >> > like Crail? What next steps do you recommend?
>> >> > >> >
>> >> > >> > For us, integrating Crail into Bigtop would be interesting as it
>> >> makes
>> >> > > the
>> >> > >> > deployment and management of Crail easier. Currently deploying
>> and
>> >> > >> > configuring Crail on a system/cluster is a rather complex and
>> task.
>> >> > >> >
>> >> > >> > Regarding the RDMA integration, yes Crail at its core is based
>> on
>> >> > native
>> >> > >> > verbs, but it is written in Java and uses DiSNI (
>> >> > >> > https://github.com/zrlio/disni) which essentially is a
>> one-to-one
>> >> > > mapping
>> >> > >> > of RDMA verbs into Java implemented on top of native verbs. But
>> I do
>> >> > > want
>> >> > >> > to clarify that Crail is not just about RDMA, it's a fast
>> storage
>> >> > > platform
>> >> > >> > designed for user-level APIs (RDMA, DPDK, SPDK, etc.) and high
>> speed
>> >> > >> > networking and storage hardware (100GbE, NVMe, NVMe-oF, etc.).
>> >> > >> >
>> >> > >> > Thank you for mentioning Roman in the context of Apache
>> incubator,
>> >> > we'll
>> >> > >> > reach out to him. Applying for Apache incubator is one of our
>> >> absolute
>> >> > >> > goals. What is unclear to me is whether an integration of Crail
>> into
>> >> > > BigTop
>> >> > >> > is gated by Crail being Apache incubator already, or whether
>> those
>> >> > > efforts
>> >> > >> > can go on in parallel, can you say something about this?
>> >> > >> >
>> >> > >> >
>> >> > >> > On Mon, Jul 3, 2017 at 10:22 PM, Olaf Flebbe <of...@oflebbe.de>
>> wrote:
>> >> > >> > >
>> >> > >> > > Well, let me speak up as an individual:
>> >> > >> > >
>> >> > >> > > I thank you for making this substantial project available
>> under
>> >> > Apache
>> >> > >> > License!
>> >> > >> > >
>> >> > >> > > -- In my former job I was part of an HPC  team and have some
>> >> cursory
>> >> > >> > ideas of RDMA mostly with MPI or for cluster filesystems. At
>> first
>> >> > > glance
>> >> > >> > it looks to me like the first fully open sourced RDMA support
>> for
>> >> Big
>> >> > > Data.
>> >> > >> > I am assuming the library is based on the standard RDMA "Verbs"
>> API
>> >> > and
>> >> > > not
>> >> > >> > on proprietary extensions of the IBM JDK like efforts before. --
>> >> > >> > >
>> >> > >> > > The big question is: what do you expect from Apache Bigtop /
>> >> Apache
>> >> > >> > Software Foundation?
>> >> > >> > >
>> >> > >> > > If you like to contribute all your code to the Apache
>> Foundation
>> >> in
>> >> > > order
>> >> > >> > to have a platform for you and other contributors to work on the
>> >> code,
>> >> > > then
>> >> > >> > I would recommend you to check out Apache Incubator. Roman can
>> give
>> >> > you
>> >> > >> > more information, since he is PMC of Apache Incubator as well.
>> >> > >> > >
>> >> > >> > > Contributing all the things to Bigtop: I would say: Thank
>> you, we
>> >> > are
>> >> > >> > honoured, but it does not fit. We are an integration and
>> automation
>> >> > >> > project.
>> >> > >> > >
>> >> > >> > > If you like you to contribute packaging, integration,
>> automation
>> >> of
>> >> > >> > configuration and deployment of your code (All these seems to be
>> >> > > missing,
>> >> > >> > at first glance) to Bigtop, feel free: We love to see innovative
>> >> > > usecases
>> >> > >> > built with our stack. We surely can only accept this
>> contribution as
>> >> > an
>> >> > >> > optional addon to our regular stack. And we will ask your group
>> to
>> >> > > maintain
>> >> > >> > it in future (see MAINTAINERS at our toplevel directory), since
>> the
>> >> > code
>> >> > >> > and needed (virtual/real) infrastructure is not middle of the
>> road.
>> >> > > There
>> >> > >> > are a couple of contributions to Bigtop where the developers
>> became
>> >> > >> > committers and are now actively working within Bigtop.
>> >> > >> > >
>> >> > >> > > Be please aware that this is my personal opinion, there may be
>> >> other
>> >> > >> > views.
>> >> > >> > >
>> >> > >> > > Olaf
>> >> > >> > >
>> >> > >> > >
>> >> > >> >
>> >> >
>> >>
>>
>
>

Re: Crail as part of BigTop

Posted by Animesh Trivedi <an...@gmail.com>.

Hi Roman,

Many thanks for pointing out the grant program. We are certainly
interested, and barring any unforeseen circumstances, we hope to submit an
application by the coming Friday !

Thanks,
--
Animesh

On Mon, Jul 10, 2017 at 10:10 PM, Roman Shaposhnik <ro...@shaposhnik.org>
wrote:

> Piling on top of others: I think this also will make an excellent
> proposal. Any chance you can consider this for:
>    http://www.prnewswire.com/news-releases/odpi-launches-
> apache-bigtop-grant-fund-program-300472753.html
>
> ?
>
> Thanks,
> Roman.
>
> On Mon, Jul 10, 2017 at 4:15 PM, Animesh Trivedi
> <an...@gmail.com> wrote:
> > Hello all,
> >
> > many thanks for positive responses. We have created a JIRA for tracking
> the
> > progress of Crail and BigTop integration here:
> >
> > https://issues.apache.org/jira/browse/BIGTOP-2837
> >
> > Any feedback/suggestions/recommendations are appreciated.
> >
> > Thanks,
> > --
> > Animesh
> >
> > On Mon, Jul 10, 2017 at 2:16 PM, MrAsanjar . <af...@gmail.com> wrote:
> >
> >> Thanks Cos,
> >>
> >>
> >> On Fri, Jul 7, 2017 at 3:39 PM, Konstantin Boudnik <co...@apache.org>
> wrote:
> >>
> >> > This is a good example [1] of what we were referring to.
> >> >
> >> > Thanks,
> >> >   Cos
> >> >
> >> > [1] https://issues.apache.org/jira/browse/BIGTOP-2283
> >> > --
> >> >   Take care,
> >> > Konstantin (Cos) Boudnik
> >> > 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
> >> >
> >> > Disclaimer: Opinions expressed in this email are those of the author,
> >> > and do not necessarily represent the views of any company the author
> >> > might be affiliated with at the moment of writing.
> >> >
> >> >
> >> > On Fri, Jul 7, 2017 at 1:28 PM, patrick stuedi <ps...@gmail.com>
> >> wrote:
> >> > > Cos, RJ, thanks for the response/advice..what I hear is that
> opening a
> >> > JIRA
> >> > > ticket on the crail integration would be a good starting
> point..we'll
> >> do
> >> > so
> >> > > and take it from it there..
> >> > >
> >> > > On Jul 5, 2017 8:06 PM, "Konstantin Boudnik" <co...@apache.org>
> wrote:
> >> > >
> >> > > Thanks for bringing this to the table, Patrick!
> >> > >
> >> > > A couple of comments:
> >> > >  - adding to what Olaf said earlier: there are two ways of donating
> a
> >> > > codebase
> >> > >    to ASF. One is to go through the Incubator, learn all the ropes
> and
> >> > > become
> >> > >    a TLP. Another one, is by joining an existing TLP as a
> subproject.
> >> > > There's
> >> > >    a number of steps being involved wrt IP clearance, license
> checks,
> >> > SGA,
> >> > >    CCLA, yada-yada. I am not going into more details like this,
> because
> >> > > you're
> >> > >    not considering this path anyway ;)
> >> > >  - from a quick glance at the project, looks like it fits the
> general
> >> > bucket
> >> > >    of data fabric platforms. Something like Apache Ignite comes to
> >> mind,
> >> > > while
> >> > >    Aluxio, as a simple caching solution, only has partial target
> >> > >    functionality. While there's no limitation on having multiple
> >> > components
> >> > >    with overlapping functionality in a given Bigtop stack (after
> all,
> >> we
> >> > > have
> >> > >    HDFS and QFS), it's an aspect worthy of some consideration.
> >> > >  - and to the RJ's point: with source code licensing question out of
> >> the
> >> > > way,
> >> > >    it'd desirable to make sure that Crail doesn't re-distribute
> >> anything
> >> > > under
> >> > >    conflicting licences. The ones in question are of GPL family and
> >> some
> >> > > more
> >> > >    as you can see in [1]. While this isn't much of a concern for
> >> Bigtop,
> >> > you
> >> > >    might benefit from doing this for the project itself.
> >> > >
> >> > > Regards,
> >> > >   Cos
> >> > >
> >> > > On Wed, Jul 05, 2017 at 10:25AM, RJ Nowling wrote:
> >> > >> I believe we already package several other open source projects
> that
> >> are
> >> > >> not Apache projects.  That should not be a hurdle.  We do have
> >> > > restrictions
> >> > >> on licenses, but it seems that Crail is licensed under the Apache
> >> > License:
> >> > >>
> >> > >> https://github.com/zrlio/crail/blob/master/LICENSE
> >> > >>
> >> > >> I do not believe we have a formal process for accepting /
> declining.
> >> > The
> >> > >> key considerations, as Olaf mentioned, are licensing and
> willingness
> >> to
> >> > >> maintain the packages.  We do have a policy that any packages which
> >> > break
> >> > >> and do not have maintainers (or the maintainers do not respond)
> will
> >> be
> >> > >> removed in the following release.
> >> > >>
> >> > >> I would suggest creating some JIRA tickets around the integration
> of
> >> > Crail
> >> > >> and your plans for each step.  e.g., creating packages (RPMs,
> DEBs),
> >> > >> deploying via our Puppet scripts, and some support in our
> integration
> >> > >> testing frameworks.  This would allow other community members to
> >> comment
> >> > >> and provide feedback on the technical aspects.
> >> > >>
> >> > >>
> >> > >> On Tue, Jul 4, 2017 at 6:18 AM, patrick stuedi <ps...@gmail.com>
> >> > wrote:
> >> > >>
> >> > >> > Olaf,
> >> > >> >
> >> > >> > many thanks for the quick reply!
> >> > >> >
> >> > >> > let me try to clarify -- the wording "donating the Crail code"
> was
> >> > >> > misleading...what we propose is actually to contribute the
> >> packaging,
> >> > >> > integration, automation and deployment of Crail to Bigtop, and of
> >> > course
> >> > >> > the continuous maintenance of it. Being an optional addon to the
> >> > regular
> >> > >> > BigTop stack would be absolutely fine. Question, do you guys
> have a
> >> > >> > democratic process where you accept/decline the integration of
> new
> >> > > projects
> >> > >> > like Crail? What next steps do you recommend?
> >> > >> >
> >> > >> > For us, integrating Crail into Bigtop would be interesting as it
> >> makes
> >> > > the
> >> > >> > deployment and management of Crail easier. Currently deploying
> and
> >> > >> > configuring Crail on a system/cluster is a rather complex and
> task.
> >> > >> >
> >> > >> > Regarding the RDMA integration, yes Crail at its core is based on
> >> > native
> >> > >> > verbs, but it is written in Java and uses DiSNI (
> >> > >> > https://github.com/zrlio/disni) which essentially is a
> one-to-one
> >> > > mapping
> >> > >> > of RDMA verbs into Java implemented on top of native verbs. But
> I do
> >> > > want
> >> > >> > to clarify that Crail is not just about RDMA, it's a fast storage
> >> > > platform
> >> > >> > designed for user-level APIs (RDMA, DPDK, SPDK, etc.) and high
> speed
> >> > >> > networking and storage hardware (100GbE, NVMe, NVMe-oF, etc.).
> >> > >> >
> >> > >> > Thank you for mentioning Roman in the context of Apache
> incubator,
> >> > we'll
> >> > >> > reach out to him. Applying for Apache incubator is one of our
> >> absolute
> >> > >> > goals. What is unclear to me is whether an integration of Crail
> into
> >> > > BigTop
> >> > >> > is gated by Crail being Apache incubator already, or whether
> those
> >> > > efforts
> >> > >> > can go on in parallel, can you say something about this?
> >> > >> >
> >> > >> >
> >> > >> > On Mon, Jul 3, 2017 at 10:22 PM, Olaf Flebbe <of...@oflebbe.de>
> wrote:
> >> > >> > >
> >> > >> > > Well, let me speak up as an individual:
> >> > >> > >
> >> > >> > > I thank you for making this substantial project available under
> >> > Apache
> >> > >> > License!
> >> > >> > >
> >> > >> > > -- In my former job I was part of an HPC  team and have some
> >> cursory
> >> > >> > ideas of RDMA mostly with MPI or for cluster filesystems. At
> first
> >> > > glance
> >> > >> > it looks to me like the first fully open sourced RDMA support for
> >> Big
> >> > > Data.
> >> > >> > I am assuming the library is based on the standard RDMA "Verbs"
> API
> >> > and
> >> > > not
> >> > >> > on proprietary extensions of the IBM JDK like efforts before. --
> >> > >> > >
> >> > >> > > The big question is: what do you expect from Apache Bigtop /
> >> Apache
> >> > >> > Software Foundation?
> >> > >> > >
> >> > >> > > If you like to contribute all your code to the Apache
> Foundation
> >> in
> >> > > order
> >> > >> > to have a platform for you and other contributors to work on the
> >> code,
> >> > > then
> >> > >> > I would recommend you to check out Apache Incubator. Roman can
> give
> >> > you
> >> > >> > more information, since he is PMC of Apache Incubator as well.
> >> > >> > >
> >> > >> > > Contributing all the things to Bigtop: I would say: Thank you,
> we
> >> > are
> >> > >> > honoured, but it does not fit. We are an integration and
> automation
> >> > >> > project.
> >> > >> > >
> >> > >> > > If you like you to contribute packaging, integration,
> automation
> >> of
> >> > >> > configuration and deployment of your code (All these seems to be
> >> > > missing,
> >> > >> > at first glance) to Bigtop, feel free: We love to see innovative
> >> > > usecases
> >> > >> > built with our stack. We surely can only accept this
> contribution as
> >> > an
> >> > >> > optional addon to our regular stack. And we will ask your group
> to
> >> > > maintain
> >> > >> > it in future (see MAINTAINERS at our toplevel directory), since
> the
> >> > code
> >> > >> > and needed (virtual/real) infrastructure is not middle of the
> road.
> >> > > There
> >> > >> > are a couple of contributions to Bigtop where the developers
> became
> >> > >> > committers and are now actively working within Bigtop.
> >> > >> > >
> >> > >> > > Be please aware that this is my personal opinion, there may be
> >> other
> >> > >> > views.
> >> > >> > >
> >> > >> > > Olaf
> >> > >> > >
> >> > >> > >
> >> > >> >
> >> >
> >>
>

Re: Crail as part of BigTop

Posted by Roman Shaposhnik <ro...@shaposhnik.org>.

Piling on top of others: I think this also will make an excellent
proposal. Any chance you can consider this for:
   http://www.prnewswire.com/news-releases/odpi-launches-apache-bigtop-grant-fund-program-300472753.html

?

Thanks,
Roman.

On Mon, Jul 10, 2017 at 4:15 PM, Animesh Trivedi
<an...@gmail.com> wrote:
> Hello all,
>
> many thanks for positive responses. We have created a JIRA for tracking the
> progress of Crail and BigTop integration here:
>
> https://issues.apache.org/jira/browse/BIGTOP-2837
>
> Any feedback/suggestions/recommendations are appreciated.
>
> Thanks,
> --
> Animesh
>
> On Mon, Jul 10, 2017 at 2:16 PM, MrAsanjar . <af...@gmail.com> wrote:
>
>> Thanks Cos,
>>
>>
>> On Fri, Jul 7, 2017 at 3:39 PM, Konstantin Boudnik <co...@apache.org> wrote:
>>
>> > This is a good example [1] of what we were referring to.
>> >
>> > Thanks,
>> >   Cos
>> >
>> > [1] https://issues.apache.org/jira/browse/BIGTOP-2283
>> > --
>> >   Take care,
>> > Konstantin (Cos) Boudnik
>> > 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
>> >
>> > Disclaimer: Opinions expressed in this email are those of the author,
>> > and do not necessarily represent the views of any company the author
>> > might be affiliated with at the moment of writing.
>> >
>> >
>> > On Fri, Jul 7, 2017 at 1:28 PM, patrick stuedi <ps...@gmail.com>
>> wrote:
>> > > Cos, RJ, thanks for the response/advice..what I hear is that opening a
>> > JIRA
>> > > ticket on the crail integration would be a good starting point..we'll
>> do
>> > so
>> > > and take it from it there..
>> > >
>> > > On Jul 5, 2017 8:06 PM, "Konstantin Boudnik" <co...@apache.org> wrote:
>> > >
>> > > Thanks for bringing this to the table, Patrick!
>> > >
>> > > A couple of comments:
>> > >  - adding to what Olaf said earlier: there are two ways of donating a
>> > > codebase
>> > >    to ASF. One is to go through the Incubator, learn all the ropes and
>> > > become
>> > >    a TLP. Another one, is by joining an existing TLP as a subproject.
>> > > There's
>> > >    a number of steps being involved wrt IP clearance, license checks,
>> > SGA,
>> > >    CCLA, yada-yada. I am not going into more details like this, because
>> > > you're
>> > >    not considering this path anyway ;)
>> > >  - from a quick glance at the project, looks like it fits the general
>> > bucket
>> > >    of data fabric platforms. Something like Apache Ignite comes to
>> mind,
>> > > while
>> > >    Aluxio, as a simple caching solution, only has partial target
>> > >    functionality. While there's no limitation on having multiple
>> > components
>> > >    with overlapping functionality in a given Bigtop stack (after all,
>> we
>> > > have
>> > >    HDFS and QFS), it's an aspect worthy of some consideration.
>> > >  - and to the RJ's point: with source code licensing question out of
>> the
>> > > way,
>> > >    it'd desirable to make sure that Crail doesn't re-distribute
>> anything
>> > > under
>> > >    conflicting licences. The ones in question are of GPL family and
>> some
>> > > more
>> > >    as you can see in [1]. While this isn't much of a concern for
>> Bigtop,
>> > you
>> > >    might benefit from doing this for the project itself.
>> > >
>> > > Regards,
>> > >   Cos
>> > >
>> > > On Wed, Jul 05, 2017 at 10:25AM, RJ Nowling wrote:
>> > >> I believe we already package several other open source projects that
>> are
>> > >> not Apache projects.  That should not be a hurdle.  We do have
>> > > restrictions
>> > >> on licenses, but it seems that Crail is licensed under the Apache
>> > License:
>> > >>
>> > >> https://github.com/zrlio/crail/blob/master/LICENSE
>> > >>
>> > >> I do not believe we have a formal process for accepting / declining.
>> > The
>> > >> key considerations, as Olaf mentioned, are licensing and willingness
>> to
>> > >> maintain the packages.  We do have a policy that any packages which
>> > break
>> > >> and do not have maintainers (or the maintainers do not respond) will
>> be
>> > >> removed in the following release.
>> > >>
>> > >> I would suggest creating some JIRA tickets around the integration of
>> > Crail
>> > >> and your plans for each step.  e.g., creating packages (RPMs, DEBs),
>> > >> deploying via our Puppet scripts, and some support in our integration
>> > >> testing frameworks.  This would allow other community members to
>> comment
>> > >> and provide feedback on the technical aspects.
>> > >>
>> > >>
>> > >> On Tue, Jul 4, 2017 at 6:18 AM, patrick stuedi <ps...@gmail.com>
>> > wrote:
>> > >>
>> > >> > Olaf,
>> > >> >
>> > >> > many thanks for the quick reply!
>> > >> >
>> > >> > let me try to clarify -- the wording "donating the Crail code" was
>> > >> > misleading...what we propose is actually to contribute the
>> packaging,
>> > >> > integration, automation and deployment of Crail to Bigtop, and of
>> > course
>> > >> > the continuous maintenance of it. Being an optional addon to the
>> > regular
>> > >> > BigTop stack would be absolutely fine. Question, do you guys have a
>> > >> > democratic process where you accept/decline the integration of new
>> > > projects
>> > >> > like Crail? What next steps do you recommend?
>> > >> >
>> > >> > For us, integrating Crail into Bigtop would be interesting as it
>> makes
>> > > the
>> > >> > deployment and management of Crail easier. Currently deploying and
>> > >> > configuring Crail on a system/cluster is a rather complex and task.
>> > >> >
>> > >> > Regarding the RDMA integration, yes Crail at its core is based on
>> > native
>> > >> > verbs, but it is written in Java and uses DiSNI (
>> > >> > https://github.com/zrlio/disni) which essentially is a one-to-one
>> > > mapping
>> > >> > of RDMA verbs into Java implemented on top of native verbs. But I do
>> > > want
>> > >> > to clarify that Crail is not just about RDMA, it's a fast storage
>> > > platform
>> > >> > designed for user-level APIs (RDMA, DPDK, SPDK, etc.) and high speed
>> > >> > networking and storage hardware (100GbE, NVMe, NVMe-oF, etc.).
>> > >> >
>> > >> > Thank you for mentioning Roman in the context of Apache incubator,
>> > we'll
>> > >> > reach out to him. Applying for Apache incubator is one of our
>> absolute
>> > >> > goals. What is unclear to me is whether an integration of Crail into
>> > > BigTop
>> > >> > is gated by Crail being Apache incubator already, or whether those
>> > > efforts
>> > >> > can go on in parallel, can you say something about this?
>> > >> >
>> > >> >
>> > >> > On Mon, Jul 3, 2017 at 10:22 PM, Olaf Flebbe <of...@oflebbe.de> wrote:
>> > >> > >
>> > >> > > Well, let me speak up as an individual:
>> > >> > >
>> > >> > > I thank you for making this substantial project available under
>> > Apache
>> > >> > License!
>> > >> > >
>> > >> > > -- In my former job I was part of an HPC  team and have some
>> cursory
>> > >> > ideas of RDMA mostly with MPI or for cluster filesystems. At first
>> > > glance
>> > >> > it looks to me like the first fully open sourced RDMA support for
>> Big
>> > > Data.
>> > >> > I am assuming the library is based on the standard RDMA "Verbs" API
>> > and
>> > > not
>> > >> > on proprietary extensions of the IBM JDK like efforts before. --
>> > >> > >
>> > >> > > The big question is: what do you expect from Apache Bigtop /
>> Apache
>> > >> > Software Foundation?
>> > >> > >
>> > >> > > If you like to contribute all your code to the Apache Foundation
>> in
>> > > order
>> > >> > to have a platform for you and other contributors to work on the
>> code,
>> > > then
>> > >> > I would recommend you to check out Apache Incubator. Roman can give
>> > you
>> > >> > more information, since he is PMC of Apache Incubator as well.
>> > >> > >
>> > >> > > Contributing all the things to Bigtop: I would say: Thank you, we
>> > are
>> > >> > honoured, but it does not fit. We are an integration and automation
>> > >> > project.
>> > >> > >
>> > >> > > If you like you to contribute packaging, integration, automation
>> of
>> > >> > configuration and deployment of your code (All these seems to be
>> > > missing,
>> > >> > at first glance) to Bigtop, feel free: We love to see innovative
>> > > usecases
>> > >> > built with our stack. We surely can only accept this contribution as
>> > an
>> > >> > optional addon to our regular stack. And we will ask your group to
>> > > maintain
>> > >> > it in future (see MAINTAINERS at our toplevel directory), since the
>> > code
>> > >> > and needed (virtual/real) infrastructure is not middle of the road.
>> > > There
>> > >> > are a couple of contributions to Bigtop where the developers became
>> > >> > committers and are now actively working within Bigtop.
>> > >> > >
>> > >> > > Be please aware that this is my personal opinion, there may be
>> other
>> > >> > views.
>> > >> > >
>> > >> > > Olaf
>> > >> > >
>> > >> > >
>> > >> >
>> >
>>

Re: Crail as part of BigTop

Posted by Animesh Trivedi <an...@gmail.com>.

Hello all,

many thanks for positive responses. We have created a JIRA for tracking the
progress of Crail and BigTop integration here:

https://issues.apache.org/jira/browse/BIGTOP-2837

Any feedback/suggestions/recommendations are appreciated.

Thanks,
--
Animesh

On Mon, Jul 10, 2017 at 2:16 PM, MrAsanjar . <af...@gmail.com> wrote:

> Thanks Cos,
>
>
> On Fri, Jul 7, 2017 at 3:39 PM, Konstantin Boudnik <co...@apache.org> wrote:
>
> > This is a good example [1] of what we were referring to.
> >
> > Thanks,
> >   Cos
> >
> > [1] https://issues.apache.org/jira/browse/BIGTOP-2283
> > --
> >   Take care,
> > Konstantin (Cos) Boudnik
> > 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
> >
> > Disclaimer: Opinions expressed in this email are those of the author,
> > and do not necessarily represent the views of any company the author
> > might be affiliated with at the moment of writing.
> >
> >
> > On Fri, Jul 7, 2017 at 1:28 PM, patrick stuedi <ps...@gmail.com>
> wrote:
> > > Cos, RJ, thanks for the response/advice..what I hear is that opening a
> > JIRA
> > > ticket on the crail integration would be a good starting point..we'll
> do
> > so
> > > and take it from it there..
> > >
> > > On Jul 5, 2017 8:06 PM, "Konstantin Boudnik" <co...@apache.org> wrote:
> > >
> > > Thanks for bringing this to the table, Patrick!
> > >
> > > A couple of comments:
> > >  - adding to what Olaf said earlier: there are two ways of donating a
> > > codebase
> > >    to ASF. One is to go through the Incubator, learn all the ropes and
> > > become
> > >    a TLP. Another one, is by joining an existing TLP as a subproject.
> > > There's
> > >    a number of steps being involved wrt IP clearance, license checks,
> > SGA,
> > >    CCLA, yada-yada. I am not going into more details like this, because
> > > you're
> > >    not considering this path anyway ;)
> > >  - from a quick glance at the project, looks like it fits the general
> > bucket
> > >    of data fabric platforms. Something like Apache Ignite comes to
> mind,
> > > while
> > >    Aluxio, as a simple caching solution, only has partial target
> > >    functionality. While there's no limitation on having multiple
> > components
> > >    with overlapping functionality in a given Bigtop stack (after all,
> we
> > > have
> > >    HDFS and QFS), it's an aspect worthy of some consideration.
> > >  - and to the RJ's point: with source code licensing question out of
> the
> > > way,
> > >    it'd desirable to make sure that Crail doesn't re-distribute
> anything
> > > under
> > >    conflicting licences. The ones in question are of GPL family and
> some
> > > more
> > >    as you can see in [1]. While this isn't much of a concern for
> Bigtop,
> > you
> > >    might benefit from doing this for the project itself.
> > >
> > > Regards,
> > >   Cos
> > >
> > > On Wed, Jul 05, 2017 at 10:25AM, RJ Nowling wrote:
> > >> I believe we already package several other open source projects that
> are
> > >> not Apache projects.  That should not be a hurdle.  We do have
> > > restrictions
> > >> on licenses, but it seems that Crail is licensed under the Apache
> > License:
> > >>
> > >> https://github.com/zrlio/crail/blob/master/LICENSE
> > >>
> > >> I do not believe we have a formal process for accepting / declining.
> > The
> > >> key considerations, as Olaf mentioned, are licensing and willingness
> to
> > >> maintain the packages.  We do have a policy that any packages which
> > break
> > >> and do not have maintainers (or the maintainers do not respond) will
> be
> > >> removed in the following release.
> > >>
> > >> I would suggest creating some JIRA tickets around the integration of
> > Crail
> > >> and your plans for each step.  e.g., creating packages (RPMs, DEBs),
> > >> deploying via our Puppet scripts, and some support in our integration
> > >> testing frameworks.  This would allow other community members to
> comment
> > >> and provide feedback on the technical aspects.
> > >>
> > >>
> > >> On Tue, Jul 4, 2017 at 6:18 AM, patrick stuedi <ps...@gmail.com>
> > wrote:
> > >>
> > >> > Olaf,
> > >> >
> > >> > many thanks for the quick reply!
> > >> >
> > >> > let me try to clarify -- the wording "donating the Crail code" was
> > >> > misleading...what we propose is actually to contribute the
> packaging,
> > >> > integration, automation and deployment of Crail to Bigtop, and of
> > course
> > >> > the continuous maintenance of it. Being an optional addon to the
> > regular
> > >> > BigTop stack would be absolutely fine. Question, do you guys have a
> > >> > democratic process where you accept/decline the integration of new
> > > projects
> > >> > like Crail? What next steps do you recommend?
> > >> >
> > >> > For us, integrating Crail into Bigtop would be interesting as it
> makes
> > > the
> > >> > deployment and management of Crail easier. Currently deploying and
> > >> > configuring Crail on a system/cluster is a rather complex and task.
> > >> >
> > >> > Regarding the RDMA integration, yes Crail at its core is based on
> > native
> > >> > verbs, but it is written in Java and uses DiSNI (
> > >> > https://github.com/zrlio/disni) which essentially is a one-to-one
> > > mapping
> > >> > of RDMA verbs into Java implemented on top of native verbs. But I do
> > > want
> > >> > to clarify that Crail is not just about RDMA, it's a fast storage
> > > platform
> > >> > designed for user-level APIs (RDMA, DPDK, SPDK, etc.) and high speed
> > >> > networking and storage hardware (100GbE, NVMe, NVMe-oF, etc.).
> > >> >
> > >> > Thank you for mentioning Roman in the context of Apache incubator,
> > we'll
> > >> > reach out to him. Applying for Apache incubator is one of our
> absolute
> > >> > goals. What is unclear to me is whether an integration of Crail into
> > > BigTop
> > >> > is gated by Crail being Apache incubator already, or whether those
> > > efforts
> > >> > can go on in parallel, can you say something about this?
> > >> >
> > >> >
> > >> > On Mon, Jul 3, 2017 at 10:22 PM, Olaf Flebbe <of...@oflebbe.de> wrote:
> > >> > >
> > >> > > Well, let me speak up as an individual:
> > >> > >
> > >> > > I thank you for making this substantial project available under
> > Apache
> > >> > License!
> > >> > >
> > >> > > -- In my former job I was part of an HPC  team and have some
> cursory
> > >> > ideas of RDMA mostly with MPI or for cluster filesystems. At first
> > > glance
> > >> > it looks to me like the first fully open sourced RDMA support for
> Big
> > > Data.
> > >> > I am assuming the library is based on the standard RDMA "Verbs" API
> > and
> > > not
> > >> > on proprietary extensions of the IBM JDK like efforts before. --
> > >> > >
> > >> > > The big question is: what do you expect from Apache Bigtop /
> Apache
> > >> > Software Foundation?
> > >> > >
> > >> > > If you like to contribute all your code to the Apache Foundation
> in
> > > order
> > >> > to have a platform for you and other contributors to work on the
> code,
> > > then
> > >> > I would recommend you to check out Apache Incubator. Roman can give
> > you
> > >> > more information, since he is PMC of Apache Incubator as well.
> > >> > >
> > >> > > Contributing all the things to Bigtop: I would say: Thank you, we
> > are
> > >> > honoured, but it does not fit. We are an integration and automation
> > >> > project.
> > >> > >
> > >> > > If you like you to contribute packaging, integration, automation
> of
> > >> > configuration and deployment of your code (All these seems to be
> > > missing,
> > >> > at first glance) to Bigtop, feel free: We love to see innovative
> > > usecases
> > >> > built with our stack. We surely can only accept this contribution as
> > an
> > >> > optional addon to our regular stack. And we will ask your group to
> > > maintain
> > >> > it in future (see MAINTAINERS at our toplevel directory), since the
> > code
> > >> > and needed (virtual/real) infrastructure is not middle of the road.
> > > There
> > >> > are a couple of contributions to Bigtop where the developers became
> > >> > committers and are now actively working within Bigtop.
> > >> > >
> > >> > > Be please aware that this is my personal opinion, there may be
> other
> > >> > views.
> > >> > >
> > >> > > Olaf
> > >> > >
> > >> > >
> > >> >
> >
>

Re: Crail as part of BigTop

Posted by "MrAsanjar ." <af...@gmail.com>.

Thanks Cos,


On Fri, Jul 7, 2017 at 3:39 PM, Konstantin Boudnik <co...@apache.org> wrote:

> This is a good example [1] of what we were referring to.
>
> Thanks,
>   Cos
>
> [1] https://issues.apache.org/jira/browse/BIGTOP-2283
> --
>   Take care,
> Konstantin (Cos) Boudnik
> 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
>
> Disclaimer: Opinions expressed in this email are those of the author,
> and do not necessarily represent the views of any company the author
> might be affiliated with at the moment of writing.
>
>
> On Fri, Jul 7, 2017 at 1:28 PM, patrick stuedi <ps...@gmail.com> wrote:
> > Cos, RJ, thanks for the response/advice..what I hear is that opening a
> JIRA
> > ticket on the crail integration would be a good starting point..we'll do
> so
> > and take it from it there..
> >
> > On Jul 5, 2017 8:06 PM, "Konstantin Boudnik" <co...@apache.org> wrote:
> >
> > Thanks for bringing this to the table, Patrick!
> >
> > A couple of comments:
> >  - adding to what Olaf said earlier: there are two ways of donating a
> > codebase
> >    to ASF. One is to go through the Incubator, learn all the ropes and
> > become
> >    a TLP. Another one, is by joining an existing TLP as a subproject.
> > There's
> >    a number of steps being involved wrt IP clearance, license checks,
> SGA,
> >    CCLA, yada-yada. I am not going into more details like this, because
> > you're
> >    not considering this path anyway ;)
> >  - from a quick glance at the project, looks like it fits the general
> bucket
> >    of data fabric platforms. Something like Apache Ignite comes to mind,
> > while
> >    Aluxio, as a simple caching solution, only has partial target
> >    functionality. While there's no limitation on having multiple
> components
> >    with overlapping functionality in a given Bigtop stack (after all, we
> > have
> >    HDFS and QFS), it's an aspect worthy of some consideration.
> >  - and to the RJ's point: with source code licensing question out of the
> > way,
> >    it'd desirable to make sure that Crail doesn't re-distribute anything
> > under
> >    conflicting licences. The ones in question are of GPL family and some
> > more
> >    as you can see in [1]. While this isn't much of a concern for Bigtop,
> you
> >    might benefit from doing this for the project itself.
> >
> > Regards,
> >   Cos
> >
> > On Wed, Jul 05, 2017 at 10:25AM, RJ Nowling wrote:
> >> I believe we already package several other open source projects that are
> >> not Apache projects.  That should not be a hurdle.  We do have
> > restrictions
> >> on licenses, but it seems that Crail is licensed under the Apache
> License:
> >>
> >> https://github.com/zrlio/crail/blob/master/LICENSE
> >>
> >> I do not believe we have a formal process for accepting / declining.
> The
> >> key considerations, as Olaf mentioned, are licensing and willingness to
> >> maintain the packages.  We do have a policy that any packages which
> break
> >> and do not have maintainers (or the maintainers do not respond) will be
> >> removed in the following release.
> >>
> >> I would suggest creating some JIRA tickets around the integration of
> Crail
> >> and your plans for each step.  e.g., creating packages (RPMs, DEBs),
> >> deploying via our Puppet scripts, and some support in our integration
> >> testing frameworks.  This would allow other community members to comment
> >> and provide feedback on the technical aspects.
> >>
> >>
> >> On Tue, Jul 4, 2017 at 6:18 AM, patrick stuedi <ps...@gmail.com>
> wrote:
> >>
> >> > Olaf,
> >> >
> >> > many thanks for the quick reply!
> >> >
> >> > let me try to clarify -- the wording "donating the Crail code" was
> >> > misleading...what we propose is actually to contribute the packaging,
> >> > integration, automation and deployment of Crail to Bigtop, and of
> course
> >> > the continuous maintenance of it. Being an optional addon to the
> regular
> >> > BigTop stack would be absolutely fine. Question, do you guys have a
> >> > democratic process where you accept/decline the integration of new
> > projects
> >> > like Crail? What next steps do you recommend?
> >> >
> >> > For us, integrating Crail into Bigtop would be interesting as it makes
> > the
> >> > deployment and management of Crail easier. Currently deploying and
> >> > configuring Crail on a system/cluster is a rather complex and task.
> >> >
> >> > Regarding the RDMA integration, yes Crail at its core is based on
> native
> >> > verbs, but it is written in Java and uses DiSNI (
> >> > https://github.com/zrlio/disni) which essentially is a one-to-one
> > mapping
> >> > of RDMA verbs into Java implemented on top of native verbs. But I do
> > want
> >> > to clarify that Crail is not just about RDMA, it's a fast storage
> > platform
> >> > designed for user-level APIs (RDMA, DPDK, SPDK, etc.) and high speed
> >> > networking and storage hardware (100GbE, NVMe, NVMe-oF, etc.).
> >> >
> >> > Thank you for mentioning Roman in the context of Apache incubator,
> we'll
> >> > reach out to him. Applying for Apache incubator is one of our absolute
> >> > goals. What is unclear to me is whether an integration of Crail into
> > BigTop
> >> > is gated by Crail being Apache incubator already, or whether those
> > efforts
> >> > can go on in parallel, can you say something about this?
> >> >
> >> >
> >> > On Mon, Jul 3, 2017 at 10:22 PM, Olaf Flebbe <of...@oflebbe.de> wrote:
> >> > >
> >> > > Well, let me speak up as an individual:
> >> > >
> >> > > I thank you for making this substantial project available under
> Apache
> >> > License!
> >> > >
> >> > > -- In my former job I was part of an HPC  team and have some cursory
> >> > ideas of RDMA mostly with MPI or for cluster filesystems. At first
> > glance
> >> > it looks to me like the first fully open sourced RDMA support for Big
> > Data.
> >> > I am assuming the library is based on the standard RDMA "Verbs" API
> and
> > not
> >> > on proprietary extensions of the IBM JDK like efforts before. --
> >> > >
> >> > > The big question is: what do you expect from Apache Bigtop / Apache
> >> > Software Foundation?
> >> > >
> >> > > If you like to contribute all your code to the Apache Foundation in
> > order
> >> > to have a platform for you and other contributors to work on the code,
> > then
> >> > I would recommend you to check out Apache Incubator. Roman can give
> you
> >> > more information, since he is PMC of Apache Incubator as well.
> >> > >
> >> > > Contributing all the things to Bigtop: I would say: Thank you, we
> are
> >> > honoured, but it does not fit. We are an integration and automation
> >> > project.
> >> > >
> >> > > If you like you to contribute packaging, integration, automation of
> >> > configuration and deployment of your code (All these seems to be
> > missing,
> >> > at first glance) to Bigtop, feel free: We love to see innovative
> > usecases
> >> > built with our stack. We surely can only accept this contribution as
> an
> >> > optional addon to our regular stack. And we will ask your group to
> > maintain
> >> > it in future (see MAINTAINERS at our toplevel directory), since the
> code
> >> > and needed (virtual/real) infrastructure is not middle of the road.
> > There
> >> > are a couple of contributions to Bigtop where the developers became
> >> > committers and are now actively working within Bigtop.
> >> > >
> >> > > Be please aware that this is my personal opinion, there may be other
> >> > views.
> >> > >
> >> > > Olaf
> >> > >
> >> > >
> >> >
>

Re: Crail as part of BigTop

Posted by Konstantin Boudnik <co...@apache.org>.

This is a good example [1] of what we were referring to.

Thanks,
  Cos

[1] https://issues.apache.org/jira/browse/BIGTOP-2283
--
  Take care,
Konstantin (Cos) Boudnik
2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622

Disclaimer: Opinions expressed in this email are those of the author,
and do not necessarily represent the views of any company the author
might be affiliated with at the moment of writing.


On Fri, Jul 7, 2017 at 1:28 PM, patrick stuedi <ps...@gmail.com> wrote:
> Cos, RJ, thanks for the response/advice..what I hear is that opening a JIRA
> ticket on the crail integration would be a good starting point..we'll do so
> and take it from it there..
>
> On Jul 5, 2017 8:06 PM, "Konstantin Boudnik" <co...@apache.org> wrote:
>
> Thanks for bringing this to the table, Patrick!
>
> A couple of comments:
>  - adding to what Olaf said earlier: there are two ways of donating a
> codebase
>    to ASF. One is to go through the Incubator, learn all the ropes and
> become
>    a TLP. Another one, is by joining an existing TLP as a subproject.
> There's
>    a number of steps being involved wrt IP clearance, license checks, SGA,
>    CCLA, yada-yada. I am not going into more details like this, because
> you're
>    not considering this path anyway ;)
>  - from a quick glance at the project, looks like it fits the general bucket
>    of data fabric platforms. Something like Apache Ignite comes to mind,
> while
>    Aluxio, as a simple caching solution, only has partial target
>    functionality. While there's no limitation on having multiple components
>    with overlapping functionality in a given Bigtop stack (after all, we
> have
>    HDFS and QFS), it's an aspect worthy of some consideration.
>  - and to the RJ's point: with source code licensing question out of the
> way,
>    it'd desirable to make sure that Crail doesn't re-distribute anything
> under
>    conflicting licences. The ones in question are of GPL family and some
> more
>    as you can see in [1]. While this isn't much of a concern for Bigtop, you
>    might benefit from doing this for the project itself.
>
> Regards,
>   Cos
>
> On Wed, Jul 05, 2017 at 10:25AM, RJ Nowling wrote:
>> I believe we already package several other open source projects that are
>> not Apache projects.  That should not be a hurdle.  We do have
> restrictions
>> on licenses, but it seems that Crail is licensed under the Apache License:
>>
>> https://github.com/zrlio/crail/blob/master/LICENSE
>>
>> I do not believe we have a formal process for accepting / declining.  The
>> key considerations, as Olaf mentioned, are licensing and willingness to
>> maintain the packages.  We do have a policy that any packages which break
>> and do not have maintainers (or the maintainers do not respond) will be
>> removed in the following release.
>>
>> I would suggest creating some JIRA tickets around the integration of Crail
>> and your plans for each step.  e.g., creating packages (RPMs, DEBs),
>> deploying via our Puppet scripts, and some support in our integration
>> testing frameworks.  This would allow other community members to comment
>> and provide feedback on the technical aspects.
>>
>>
>> On Tue, Jul 4, 2017 at 6:18 AM, patrick stuedi <ps...@gmail.com> wrote:
>>
>> > Olaf,
>> >
>> > many thanks for the quick reply!
>> >
>> > let me try to clarify -- the wording "donating the Crail code" was
>> > misleading...what we propose is actually to contribute the packaging,
>> > integration, automation and deployment of Crail to Bigtop, and of course
>> > the continuous maintenance of it. Being an optional addon to the regular
>> > BigTop stack would be absolutely fine. Question, do you guys have a
>> > democratic process where you accept/decline the integration of new
> projects
>> > like Crail? What next steps do you recommend?
>> >
>> > For us, integrating Crail into Bigtop would be interesting as it makes
> the
>> > deployment and management of Crail easier. Currently deploying and
>> > configuring Crail on a system/cluster is a rather complex and task.
>> >
>> > Regarding the RDMA integration, yes Crail at its core is based on native
>> > verbs, but it is written in Java and uses DiSNI (
>> > https://github.com/zrlio/disni) which essentially is a one-to-one
> mapping
>> > of RDMA verbs into Java implemented on top of native verbs. But I do
> want
>> > to clarify that Crail is not just about RDMA, it's a fast storage
> platform
>> > designed for user-level APIs (RDMA, DPDK, SPDK, etc.) and high speed
>> > networking and storage hardware (100GbE, NVMe, NVMe-oF, etc.).
>> >
>> > Thank you for mentioning Roman in the context of Apache incubator, we'll
>> > reach out to him. Applying for Apache incubator is one of our absolute
>> > goals. What is unclear to me is whether an integration of Crail into
> BigTop
>> > is gated by Crail being Apache incubator already, or whether those
> efforts
>> > can go on in parallel, can you say something about this?
>> >
>> >
>> > On Mon, Jul 3, 2017 at 10:22 PM, Olaf Flebbe <of...@oflebbe.de> wrote:
>> > >
>> > > Well, let me speak up as an individual:
>> > >
>> > > I thank you for making this substantial project available under Apache
>> > License!
>> > >
>> > > -- In my former job I was part of an HPC  team and have some cursory
>> > ideas of RDMA mostly with MPI or for cluster filesystems. At first
> glance
>> > it looks to me like the first fully open sourced RDMA support for Big
> Data.
>> > I am assuming the library is based on the standard RDMA "Verbs" API and
> not
>> > on proprietary extensions of the IBM JDK like efforts before. --
>> > >
>> > > The big question is: what do you expect from Apache Bigtop / Apache
>> > Software Foundation?
>> > >
>> > > If you like to contribute all your code to the Apache Foundation in
> order
>> > to have a platform for you and other contributors to work on the code,
> then
>> > I would recommend you to check out Apache Incubator. Roman can give you
>> > more information, since he is PMC of Apache Incubator as well.
>> > >
>> > > Contributing all the things to Bigtop: I would say: Thank you, we are
>> > honoured, but it does not fit. We are an integration and automation
>> > project.
>> > >
>> > > If you like you to contribute packaging, integration, automation of
>> > configuration and deployment of your code (All these seems to be
> missing,
>> > at first glance) to Bigtop, feel free: We love to see innovative
> usecases
>> > built with our stack. We surely can only accept this contribution as an
>> > optional addon to our regular stack. And we will ask your group to
> maintain
>> > it in future (see MAINTAINERS at our toplevel directory), since the code
>> > and needed (virtual/real) infrastructure is not middle of the road.
> There
>> > are a couple of contributions to Bigtop where the developers became
>> > committers and are now actively working within Bigtop.
>> > >
>> > > Be please aware that this is my personal opinion, there may be other
>> > views.
>> > >
>> > > Olaf
>> > >
>> > >
>> >

Re: Crail as part of BigTop

Posted by patrick stuedi <ps...@gmail.com>.

Cos, RJ, thanks for the response/advice..what I hear is that opening a JIRA
ticket on the crail integration would be a good starting point..we'll do so
and take it from it there..

On Jul 5, 2017 8:06 PM, "Konstantin Boudnik" <co...@apache.org> wrote:

Thanks for bringing this to the table, Patrick!

A couple of comments:
 - adding to what Olaf said earlier: there are two ways of donating a
codebase
   to ASF. One is to go through the Incubator, learn all the ropes and
become
   a TLP. Another one, is by joining an existing TLP as a subproject.
There's
   a number of steps being involved wrt IP clearance, license checks, SGA,
   CCLA, yada-yada. I am not going into more details like this, because
you're
   not considering this path anyway ;)
 - from a quick glance at the project, looks like it fits the general bucket
   of data fabric platforms. Something like Apache Ignite comes to mind,
while
   Aluxio, as a simple caching solution, only has partial target
   functionality. While there's no limitation on having multiple components
   with overlapping functionality in a given Bigtop stack (after all, we
have
   HDFS and QFS), it's an aspect worthy of some consideration.
 - and to the RJ's point: with source code licensing question out of the
way,
   it'd desirable to make sure that Crail doesn't re-distribute anything
under
   conflicting licences. The ones in question are of GPL family and some
more
   as you can see in [1]. While this isn't much of a concern for Bigtop, you
   might benefit from doing this for the project itself.

Regards,
  Cos

On Wed, Jul 05, 2017 at 10:25AM, RJ Nowling wrote:
> I believe we already package several other open source projects that are
> not Apache projects.  That should not be a hurdle.  We do have
restrictions
> on licenses, but it seems that Crail is licensed under the Apache License:
>
> https://github.com/zrlio/crail/blob/master/LICENSE
>
> I do not believe we have a formal process for accepting / declining.  The
> key considerations, as Olaf mentioned, are licensing and willingness to
> maintain the packages.  We do have a policy that any packages which break
> and do not have maintainers (or the maintainers do not respond) will be
> removed in the following release.
>
> I would suggest creating some JIRA tickets around the integration of Crail
> and your plans for each step.  e.g., creating packages (RPMs, DEBs),
> deploying via our Puppet scripts, and some support in our integration
> testing frameworks.  This would allow other community members to comment
> and provide feedback on the technical aspects.
>
>
> On Tue, Jul 4, 2017 at 6:18 AM, patrick stuedi <ps...@gmail.com> wrote:
>
> > Olaf,
> >
> > many thanks for the quick reply!
> >
> > let me try to clarify -- the wording "donating the Crail code" was
> > misleading...what we propose is actually to contribute the packaging,
> > integration, automation and deployment of Crail to Bigtop, and of course
> > the continuous maintenance of it. Being an optional addon to the regular
> > BigTop stack would be absolutely fine. Question, do you guys have a
> > democratic process where you accept/decline the integration of new
projects
> > like Crail? What next steps do you recommend?
> >
> > For us, integrating Crail into Bigtop would be interesting as it makes
the
> > deployment and management of Crail easier. Currently deploying and
> > configuring Crail on a system/cluster is a rather complex and task.
> >
> > Regarding the RDMA integration, yes Crail at its core is based on native
> > verbs, but it is written in Java and uses DiSNI (
> > https://github.com/zrlio/disni) which essentially is a one-to-one
mapping
> > of RDMA verbs into Java implemented on top of native verbs. But I do
want
> > to clarify that Crail is not just about RDMA, it's a fast storage
platform
> > designed for user-level APIs (RDMA, DPDK, SPDK, etc.) and high speed
> > networking and storage hardware (100GbE, NVMe, NVMe-oF, etc.).
> >
> > Thank you for mentioning Roman in the context of Apache incubator, we'll
> > reach out to him. Applying for Apache incubator is one of our absolute
> > goals. What is unclear to me is whether an integration of Crail into
BigTop
> > is gated by Crail being Apache incubator already, or whether those
efforts
> > can go on in parallel, can you say something about this?
> >
> >
> > On Mon, Jul 3, 2017 at 10:22 PM, Olaf Flebbe <of...@oflebbe.de> wrote:
> > >
> > > Well, let me speak up as an individual:
> > >
> > > I thank you for making this substantial project available under Apache
> > License!
> > >
> > > -- In my former job I was part of an HPC  team and have some cursory
> > ideas of RDMA mostly with MPI or for cluster filesystems. At first
glance
> > it looks to me like the first fully open sourced RDMA support for Big
Data.
> > I am assuming the library is based on the standard RDMA "Verbs" API and
not
> > on proprietary extensions of the IBM JDK like efforts before. --
> > >
> > > The big question is: what do you expect from Apache Bigtop / Apache
> > Software Foundation?
> > >
> > > If you like to contribute all your code to the Apache Foundation in
order
> > to have a platform for you and other contributors to work on the code,
then
> > I would recommend you to check out Apache Incubator. Roman can give you
> > more information, since he is PMC of Apache Incubator as well.
> > >
> > > Contributing all the things to Bigtop: I would say: Thank you, we are
> > honoured, but it does not fit. We are an integration and automation
> > project.
> > >
> > > If you like you to contribute packaging, integration, automation of
> > configuration and deployment of your code (All these seems to be
missing,
> > at first glance) to Bigtop, feel free: We love to see innovative
usecases
> > built with our stack. We surely can only accept this contribution as an
> > optional addon to our regular stack. And we will ask your group to
maintain
> > it in future (see MAINTAINERS at our toplevel directory), since the code
> > and needed (virtual/real) infrastructure is not middle of the road.
There
> > are a couple of contributions to Bigtop where the developers became
> > committers and are now actively working within Bigtop.
> > >
> > > Be please aware that this is my personal opinion, there may be other
> > views.
> > >
> > > Olaf
> > >
> > >
> >

Re: Crail as part of BigTop

Posted by Konstantin Boudnik <co...@apache.org>.

Thanks for bringing this to the table, Patrick!

A couple of comments:
 - adding to what Olaf said earlier: there are two ways of donating a codebase
   to ASF. One is to go through the Incubator, learn all the ropes and become
   a TLP. Another one, is by joining an existing TLP as a subproject. There's
   a number of steps being involved wrt IP clearance, license checks, SGA,
   CCLA, yada-yada. I am not going into more details like this, because you're
   not considering this path anyway ;)
 - from a quick glance at the project, looks like it fits the general bucket
   of data fabric platforms. Something like Apache Ignite comes to mind, while
   Aluxio, as a simple caching solution, only has partial target
   functionality. While there's no limitation on having multiple components
   with overlapping functionality in a given Bigtop stack (after all, we have
   HDFS and QFS), it's an aspect worthy of some consideration.
 - and to the RJ's point: with source code licensing question out of the way,
   it'd desirable to make sure that Crail doesn't re-distribute anything under
   conflicting licences. The ones in question are of GPL family and some more
   as you can see in [1]. While this isn't much of a concern for Bigtop, you
   might benefit from doing this for the project itself.

Regards,
  Cos
    
On Wed, Jul 05, 2017 at 10:25AM, RJ Nowling wrote:
> I believe we already package several other open source projects that are
> not Apache projects.  That should not be a hurdle.  We do have restrictions
> on licenses, but it seems that Crail is licensed under the Apache License:
> 
> https://github.com/zrlio/crail/blob/master/LICENSE
> 
> I do not believe we have a formal process for accepting / declining.  The
> key considerations, as Olaf mentioned, are licensing and willingness to
> maintain the packages.  We do have a policy that any packages which break
> and do not have maintainers (or the maintainers do not respond) will be
> removed in the following release.
> 
> I would suggest creating some JIRA tickets around the integration of Crail
> and your plans for each step.  e.g., creating packages (RPMs, DEBs),
> deploying via our Puppet scripts, and some support in our integration
> testing frameworks.  This would allow other community members to comment
> and provide feedback on the technical aspects.
> 
> 
> On Tue, Jul 4, 2017 at 6:18 AM, patrick stuedi <ps...@gmail.com> wrote:
> 
> > Olaf,
> >
> > many thanks for the quick reply!
> >
> > let me try to clarify -- the wording "donating the Crail code" was
> > misleading...what we propose is actually to contribute the packaging,
> > integration, automation and deployment of Crail to Bigtop, and of course
> > the continuous maintenance of it. Being an optional addon to the regular
> > BigTop stack would be absolutely fine. Question, do you guys have a
> > democratic process where you accept/decline the integration of new projects
> > like Crail? What next steps do you recommend?
> >
> > For us, integrating Crail into Bigtop would be interesting as it makes the
> > deployment and management of Crail easier. Currently deploying and
> > configuring Crail on a system/cluster is a rather complex and task.
> >
> > Regarding the RDMA integration, yes Crail at its core is based on native
> > verbs, but it is written in Java and uses DiSNI (
> > https://github.com/zrlio/disni) which essentially is a one-to-one mapping
> > of RDMA verbs into Java implemented on top of native verbs. But I do want
> > to clarify that Crail is not just about RDMA, it's a fast storage platform
> > designed for user-level APIs (RDMA, DPDK, SPDK, etc.) and high speed
> > networking and storage hardware (100GbE, NVMe, NVMe-oF, etc.).
> >
> > Thank you for mentioning Roman in the context of Apache incubator, we'll
> > reach out to him. Applying for Apache incubator is one of our absolute
> > goals. What is unclear to me is whether an integration of Crail into BigTop
> > is gated by Crail being Apache incubator already, or whether those efforts
> > can go on in parallel, can you say something about this?
> >
> >
> > On Mon, Jul 3, 2017 at 10:22 PM, Olaf Flebbe <of...@oflebbe.de> wrote:
> > >
> > > Well, let me speak up as an individual:
> > >
> > > I thank you for making this substantial project available under Apache
> > License!
> > >
> > > -- In my former job I was part of an HPC  team and have some cursory
> > ideas of RDMA mostly with MPI or for cluster filesystems. At first glance
> > it looks to me like the first fully open sourced RDMA support for Big Data.
> > I am assuming the library is based on the standard RDMA "Verbs" API and not
> > on proprietary extensions of the IBM JDK like efforts before. --
> > >
> > > The big question is: what do you expect from Apache Bigtop / Apache
> > Software Foundation?
> > >
> > > If you like to contribute all your code to the Apache Foundation in order
> > to have a platform for you and other contributors to work on the code, then
> > I would recommend you to check out Apache Incubator. Roman can give you
> > more information, since he is PMC of Apache Incubator as well.
> > >
> > > Contributing all the things to Bigtop: I would say: Thank you, we are
> > honoured, but it does not fit. We are an integration and automation
> > project.
> > >
> > > If you like you to contribute packaging, integration, automation of
> > configuration and deployment of your code (All these seems to be missing,
> > at first glance) to Bigtop, feel free: We love to see innovative usecases
> > built with our stack. We surely can only accept this contribution as an
> > optional addon to our regular stack. And we will ask your group to maintain
> > it in future (see MAINTAINERS at our toplevel directory), since the code
> > and needed (virtual/real) infrastructure is not middle of the road. There
> > are a couple of contributions to Bigtop where the developers became
> > committers and are now actively working within Bigtop.
> > >
> > > Be please aware that this is my personal opinion, there may be other
> > views.
> > >
> > > Olaf
> > >
> > >
> >

Re: Crail as part of BigTop

Posted by RJ Nowling <rn...@gmail.com>.

I believe we already package several other open source projects that are
not Apache projects.  That should not be a hurdle.  We do have restrictions
on licenses, but it seems that Crail is licensed under the Apache License:

https://github.com/zrlio/crail/blob/master/LICENSE

I do not believe we have a formal process for accepting / declining.  The
key considerations, as Olaf mentioned, are licensing and willingness to
maintain the packages.  We do have a policy that any packages which break
and do not have maintainers (or the maintainers do not respond) will be
removed in the following release.

I would suggest creating some JIRA tickets around the integration of Crail
and your plans for each step.  e.g., creating packages (RPMs, DEBs),
deploying via our Puppet scripts, and some support in our integration
testing frameworks.  This would allow other community members to comment
and provide feedback on the technical aspects.


On Tue, Jul 4, 2017 at 6:18 AM, patrick stuedi <ps...@gmail.com> wrote:

> Olaf,
>
> many thanks for the quick reply!
>
> let me try to clarify -- the wording "donating the Crail code" was
> misleading...what we propose is actually to contribute the packaging,
> integration, automation and deployment of Crail to Bigtop, and of course
> the continuous maintenance of it. Being an optional addon to the regular
> BigTop stack would be absolutely fine. Question, do you guys have a
> democratic process where you accept/decline the integration of new projects
> like Crail? What next steps do you recommend?
>
> For us, integrating Crail into Bigtop would be interesting as it makes the
> deployment and management of Crail easier. Currently deploying and
> configuring Crail on a system/cluster is a rather complex and task.
>
> Regarding the RDMA integration, yes Crail at its core is based on native
> verbs, but it is written in Java and uses DiSNI (
> https://github.com/zrlio/disni) which essentially is a one-to-one mapping
> of RDMA verbs into Java implemented on top of native verbs. But I do want
> to clarify that Crail is not just about RDMA, it's a fast storage platform
> designed for user-level APIs (RDMA, DPDK, SPDK, etc.) and high speed
> networking and storage hardware (100GbE, NVMe, NVMe-oF, etc.).
>
> Thank you for mentioning Roman in the context of Apache incubator, we'll
> reach out to him. Applying for Apache incubator is one of our absolute
> goals. What is unclear to me is whether an integration of Crail into BigTop
> is gated by Crail being Apache incubator already, or whether those efforts
> can go on in parallel, can you say something about this?
>
>
> On Mon, Jul 3, 2017 at 10:22 PM, Olaf Flebbe <of...@oflebbe.de> wrote:
> >
> > Well, let me speak up as an individual:
> >
> > I thank you for making this substantial project available under Apache
> License!
> >
> > -- In my former job I was part of an HPC  team and have some cursory
> ideas of RDMA mostly with MPI or for cluster filesystems. At first glance
> it looks to me like the first fully open sourced RDMA support for Big Data.
> I am assuming the library is based on the standard RDMA "Verbs" API and not
> on proprietary extensions of the IBM JDK like efforts before. --
> >
> > The big question is: what do you expect from Apache Bigtop / Apache
> Software Foundation?
> >
> > If you like to contribute all your code to the Apache Foundation in order
> to have a platform for you and other contributors to work on the code, then
> I would recommend you to check out Apache Incubator. Roman can give you
> more information, since he is PMC of Apache Incubator as well.
> >
> > Contributing all the things to Bigtop: I would say: Thank you, we are
> honoured, but it does not fit. We are an integration and automation
> project.
> >
> > If you like you to contribute packaging, integration, automation of
> configuration and deployment of your code (All these seems to be missing,
> at first glance) to Bigtop, feel free: We love to see innovative usecases
> built with our stack. We surely can only accept this contribution as an
> optional addon to our regular stack. And we will ask your group to maintain
> it in future (see MAINTAINERS at our toplevel directory), since the code
> and needed (virtual/real) infrastructure is not middle of the road. There
> are a couple of contributions to Bigtop where the developers became
> committers and are now actively working within Bigtop.
> >
> > Be please aware that this is my personal opinion, there may be other
> views.
> >
> > Olaf
> >
> >
>

Re: Crail as part of BigTop

Posted by patrick stuedi <ps...@gmail.com>.

Olaf,

many thanks for the quick reply!

let me try to clarify -- the wording "donating the Crail code" was
misleading...what we propose is actually to contribute the packaging,
integration, automation and deployment of Crail to Bigtop, and of course
the continuous maintenance of it. Being an optional addon to the regular
BigTop stack would be absolutely fine. Question, do you guys have a
democratic process where you accept/decline the integration of new projects
like Crail? What next steps do you recommend?

For us, integrating Crail into Bigtop would be interesting as it makes the
deployment and management of Crail easier. Currently deploying and
configuring Crail on a system/cluster is a rather complex and task.

Regarding the RDMA integration, yes Crail at its core is based on native
verbs, but it is written in Java and uses DiSNI (
https://github.com/zrlio/disni) which essentially is a one-to-one mapping
of RDMA verbs into Java implemented on top of native verbs. But I do want
to clarify that Crail is not just about RDMA, it's a fast storage platform
designed for user-level APIs (RDMA, DPDK, SPDK, etc.) and high speed
networking and storage hardware (100GbE, NVMe, NVMe-oF, etc.).

Thank you for mentioning Roman in the context of Apache incubator, we'll
reach out to him. Applying for Apache incubator is one of our absolute
goals. What is unclear to me is whether an integration of Crail into BigTop
is gated by Crail being Apache incubator already, or whether those efforts
can go on in parallel, can you say something about this?

On Mon, Jul 3, 2017 at 10:22 PM, Olaf Flebbe <of...@oflebbe.de> wrote:
>
> Well, let me speak up as an individual:
>
> I thank you for making this substantial project available under Apache
License!
>
> -- In my former job I was part of an HPC  team and have some cursory
ideas of RDMA mostly with MPI or for cluster filesystems. At first glance
it looks to me like the first fully open sourced RDMA support for Big Data.
I am assuming the library is based on the standard RDMA "Verbs" API and not
on proprietary extensions of the IBM JDK like efforts before. --
>
> The big question is: what do you expect from Apache Bigtop / Apache
Software Foundation?
>
> If you like to contribute all your code to the Apache Foundation in order
to have a platform for you and other contributors to work on the code, then
I would recommend you to check out Apache Incubator. Roman can give you
more information, since he is PMC of Apache Incubator as well.
>
> Contributing all the things to Bigtop: I would say: Thank you, we are
honoured, but it does not fit. We are an integration and automation project.
>
> If you like you to contribute packaging, integration, automation of
configuration and deployment of your code (All these seems to be missing,
at first glance) to Bigtop, feel free: We love to see innovative usecases
built with our stack. We surely can only accept this contribution as an
optional addon to our regular stack. And we will ask your group to maintain
it in future (see MAINTAINERS at our toplevel directory), since the code
and needed (virtual/real) infrastructure is not middle of the road. There
are a couple of contributions to Bigtop where the developers became
committers and are now actively working within Bigtop.
>
> Be please aware that this is my personal opinion, there may be other
views.
>
> Olaf
>
>

Re: Crail as part of BigTop

Posted by Olaf Flebbe <of...@oflebbe.de>.

Well, let me speak up as an individual:

I thank you for making this substantial project available under Apache License!

-- In my former job I was part of an HPC team and have some cursory ideas of RDMA mostly with MPI or for cluster filesystems. At first glance it looks to me like the first fully open sourced RDMA support for Big Data. I am assuming the library is based on the standard RDMA "Verbs" API and not on proprietary extensions of the IBM JDK like efforts before. --

The big question is: what do you expect from Apache Bigtop / Apache Software Foundation?

If you like to contribute all your code to the Apache Foundation in order to have a platform for you and other contributors to work on the code, then I would recommend you to check out Apache Incubator. Roman can give you more information, since he is PMC of Apache Incubator as well.

Contributing all the things to Bigtop: I would say: Thank you, we are honoured, but it does not fit. We are an integration and automation project.

If you like you to contribute packaging, integration, automation of configuration and deployment of your code (All these seems to be missing, at first glance) to Bigtop, feel free: We love to see innovative usecases built with our stack. We surely can only accept this contribution as an optional addon to our regular stack. And we will ask your group to maintain it in future (see MAINTAINERS at our toplevel directory), since the code and needed (virtual/real) infrastructure is not middle of the road. There are a couple of contributions to Bigtop where the developers became committers and are now actively working within Bigtop.

Be please aware that this is my personal opinion, there may be other views.

Olaf