You are viewing a plain text version of this content. The canonical link for it is here.
Posted to general@incubator.apache.org by Luciano Resende <lu...@gmail.com> on 2020/03/24 04:12:31 UTC
Re: [DISCUSS] Project proposal - Apache Rainbow
Could you speak a little more about the current status of the project?
From the, now available, repository, it seems that it is in very early stages?
https://github.com/Natural-Intelligence/rainbow/graphs/contributors
On Mon, Feb 24, 2020 at 3:45 AM Aviem Zur <av...@apache.org> wrote:
>
> Hi all,
>
> Thanks for the feedback.
>
> 1. Indeed, the codebase is still under a private repository. We intend to
> have it ready to share publicly later this March.
> 2. The project is built in Python and Java.This is due to the fact that we
> have deep integrations with open source projects written in these languages.
> We also considered the fact that it is used by both data scientists and
> data engineers and we believe a combination of Python/Java will promote
> collaboration and contribution.
> 3. Rainbow project intends to facilitate and simplify the composition of
> complex pipelines, which are based on other open source projects.
> As such it does not compete or overlap but rather complement these projects.
> 4. Re: DLAB project - as we see it this project focuses in the research
> phase, while Rainbow's focus is in the production phase.
> Seems the 2 projects complement each other and it would be very interesting
> for us to collaborate with the DLAB team.
> 5. We will adjust the proposal to provide more details on how other Apache
> projects are used in Rainbow.
> We currently mainly use Apache Airflow in order to run pipelines defined by
> users in our APIs (YAML, with plans of UI/REST), this reduces the
> engineering requirements for transitioning data science code into
> production. We also leverage Apache Spark and Apache Hive for data
> preparation features and there are plans to integrate with Apache Karaf as
> well.
>
> Thanks,
> Aviem
>
> On Sat, Feb 22, 2020 at 4:29 AM Paul King <pa...@asert.com.au> wrote:
>
> > Indeed, it does sound interesting.
> >
> > I would find it useful if the "existing Apache projects" bit of "Rainbow is
> > in development, leveraging existing Apache projects." could be expanded in
> > any way. I know there is a list of external dependencies later but any
> > further description of how those technologies are used would be helpful.
> >
> > Also, I'd be interested in knowing how the proposal relates to DLAB:
> > https://dlab.apache.org/
> >
> > Nice work.
> >
> > Cheers, Paul.
> >
> >
> >
> > On Sat, Feb 22, 2020 at 2:34 AM larry mccay <lm...@apache.org> wrote:
> >
> > > This seems like an interesting proposal.
> > >
> > > Couple points/questions:
> > >
> > > * The existing source is not available for viewing as it is still in
> > > private repos?
> > > * Is it a primarily java project?
> > > * It seems the intent of Rainbow is to not compete or overlap with the
> > > Hadoop ecosystem projects but rather to provide an efficient interface
> > > above them - correct?
> > >
> > >
> > > On Fri, Feb 21, 2020 at 8:51 AM Aviem Zur <av...@gmail.com> wrote:
> > >
> > > > Hi,
> > > >
> > > > We would like to propose Rainbow as an Apache incubator project.
> > Rainbow
> > > is
> > > > an end-to-end platform for data engineers & scientists, allowing them
> > to
> > > > build, train and deploy machine learning models in a robust and agile
> > > way.
> > > > The project's goal is to operationalize the machine learning process,
> > > > allowing data scientists to quickly transition from a successful
> > > experiment
> > > > to an automated pipeline in production.
> > > >
> > > > The proposal can be found here:
> > > > https://cwiki.apache.org/confluence/display/INCUBATOR/Apache+Rainbow
> > > >
> > > > We would appreciate your feedback and thoughts on the proposal.
> > > >
> > > > Thanks,
> > > > Aviem
> > > >
> > >
> >
--
Luciano Resende
http://twitter.com/lresende1975
http://lresende.blogspot.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: [DISCUSS] Project proposal - Apache Rainbow
Posted by Justin Mclean <ju...@classsoftware.com>.
Hi,
The main Issue I see with the proposal is that is only a small (if any) community around the project and only a couple of committers. Projects like this can fail to attract a developer community and end up exiting the incubator. So I guess the question is does the Incubator want to accept a project with little community around it and take that risk? We generally don’t do this, but there have been some exceptions to this rule in the past. What do other IPMC members think?
Thanks,
Justin
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: [DISCUSS] Project proposal - Apache Rainbow
Posted by Jean-Baptiste Onofre <jb...@nanthrax.net>.
Agree, that was my first thought: work during the incubation phase to build project/community.
Let’s say what the other IPMC members think (maybe some would like to join the project ;) ).
Regards
JB
> Le 25 mars 2020 à 06:31, Justin Mclean <ju...@classsoftware.com> a écrit :
>
> Hi,
>
> Really up to the project on what process would fit them best. If it ends up an Incubator project that may mean more work for the mentors. I think as long as they are committed to the project we could accept it, but I’d like to hear what other IPMC members think.
>
> Thanks,
> justin
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: [DISCUSS] Project proposal - Apache Rainbow
Posted by Justin Mclean <ju...@classsoftware.com>.
Hi,
One other thing you may need to consider is your name. Changing a projects name is expensive in term of infra time and being such a generic name there may be issues with it.
Thanks,
Justin
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: [DISCUSS] Project proposal - Apache Rainbow
Posted by Lior Schachter <li...@gmail.com>.
Hi all,
At this stage the community around rainbow is indeed rather small. However,
I believe the incubator platform is where we can and should build a
substantial community.
Natural Intelligence is committed to invest in this project as it is used
as the foundation of our production pipelines.
Moreover we have been discussing the viability of this project with several
Israeli companies which all showed a great interest in contributing and
joining the community.
We think using the incubator platform will expedite the process of forming
a community as clearly this project is needed and has a great value to many
companies.
Regards,
Lior
On Wed, Mar 25, 2020 at 2:30 PM Paul King <pa...@asert.com.au> wrote:
> I'm probably leaning towards accepting. Given that the closest fit to me
> seems the DLAB project, I would be keen to see if anyone in that project
> has an interest in keeping abreast of developments in Rainbow. DLAB is
> itself also incubating so we wouldn't want to burden them with too many
> more official responsibilities, so just informal interest would be all I
> would hope we could encourage.
>
> Cheers, Paul.
>
> On Wed, Mar 25, 2020 at 3:31 PM Justin Mclean <ju...@classsoftware.com>
> wrote:
>
> > Hi,
> >
> > Really up to the project on what process would fit them best. If it ends
> > up an Incubator project that may mean more work for the mentors. I think
> > as long as they are committed to the project we could accept it, but I’d
> > like to hear what other IPMC members think.
> >
> > Thanks,
> > justin
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: general-help@incubator.apache.org
> >
> >
>
Re: [DISCUSS] Project proposal - Apache Rainbow
Posted by Paul King <pa...@asert.com.au>.
I'm probably leaning towards accepting. Given that the closest fit to me
seems the DLAB project, I would be keen to see if anyone in that project
has an interest in keeping abreast of developments in Rainbow. DLAB is
itself also incubating so we wouldn't want to burden them with too many
more official responsibilities, so just informal interest would be all I
would hope we could encourage.
Cheers, Paul.
On Wed, Mar 25, 2020 at 3:31 PM Justin Mclean <ju...@classsoftware.com>
wrote:
> Hi,
>
> Really up to the project on what process would fit them best. If it ends
> up an Incubator project that may mean more work for the mentors. I think
> as long as they are committed to the project we could accept it, but I’d
> like to hear what other IPMC members think.
>
> Thanks,
> justin
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>
Re: [DISCUSS] Project proposal - Apache Rainbow
Posted by Justin Mclean <ju...@classsoftware.com>.
Hi,
Really up to the project on what process would fit them best. If it ends up an Incubator project that may mean more work for the mentors. I think as long as they are committed to the project we could accept it, but I’d like to hear what other IPMC members think.
Thanks,
justin
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: [DISCUSS] Project proposal - Apache Rainbow
Posted by Jean-Baptiste Onofre <jb...@nanthrax.net>.
Hi,
Yes, it’s a very early stage project, discussed between several Apache contributor.
We wanted to use the incubator as the "platform" to build the project and the community around the project.
Do you think it’s more a "lab" related project ?
Regards
JB
> Le 24 mars 2020 à 05:12, Luciano Resende <lu...@gmail.com> a écrit :
>
> Could you speak a little more about the current status of the project?
>
> From the, now available, repository, it seems that it is in very early stages?
>
> https://github.com/Natural-Intelligence/rainbow/graphs/contributors
>
>
> On Mon, Feb 24, 2020 at 3:45 AM Aviem Zur <av...@apache.org> wrote:
>>
>> Hi all,
>>
>> Thanks for the feedback.
>>
>> 1. Indeed, the codebase is still under a private repository. We intend to
>> have it ready to share publicly later this March.
>> 2. The project is built in Python and Java.This is due to the fact that we
>> have deep integrations with open source projects written in these languages.
>> We also considered the fact that it is used by both data scientists and
>> data engineers and we believe a combination of Python/Java will promote
>> collaboration and contribution.
>> 3. Rainbow project intends to facilitate and simplify the composition of
>> complex pipelines, which are based on other open source projects.
>> As such it does not compete or overlap but rather complement these projects.
>> 4. Re: DLAB project - as we see it this project focuses in the research
>> phase, while Rainbow's focus is in the production phase.
>> Seems the 2 projects complement each other and it would be very interesting
>> for us to collaborate with the DLAB team.
>> 5. We will adjust the proposal to provide more details on how other Apache
>> projects are used in Rainbow.
>> We currently mainly use Apache Airflow in order to run pipelines defined by
>> users in our APIs (YAML, with plans of UI/REST), this reduces the
>> engineering requirements for transitioning data science code into
>> production. We also leverage Apache Spark and Apache Hive for data
>> preparation features and there are plans to integrate with Apache Karaf as
>> well.
>>
>> Thanks,
>> Aviem
>>
>> On Sat, Feb 22, 2020 at 4:29 AM Paul King <pa...@asert.com.au> wrote:
>>
>>> Indeed, it does sound interesting.
>>>
>>> I would find it useful if the "existing Apache projects" bit of "Rainbow is
>>> in development, leveraging existing Apache projects." could be expanded in
>>> any way. I know there is a list of external dependencies later but any
>>> further description of how those technologies are used would be helpful.
>>>
>>> Also, I'd be interested in knowing how the proposal relates to DLAB:
>>> https://dlab.apache.org/
>>>
>>> Nice work.
>>>
>>> Cheers, Paul.
>>>
>>>
>>>
>>> On Sat, Feb 22, 2020 at 2:34 AM larry mccay <lm...@apache.org> wrote:
>>>
>>>> This seems like an interesting proposal.
>>>>
>>>> Couple points/questions:
>>>>
>>>> * The existing source is not available for viewing as it is still in
>>>> private repos?
>>>> * Is it a primarily java project?
>>>> * It seems the intent of Rainbow is to not compete or overlap with the
>>>> Hadoop ecosystem projects but rather to provide an efficient interface
>>>> above them - correct?
>>>>
>>>>
>>>> On Fri, Feb 21, 2020 at 8:51 AM Aviem Zur <av...@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> We would like to propose Rainbow as an Apache incubator project.
>>> Rainbow
>>>> is
>>>>> an end-to-end platform for data engineers & scientists, allowing them
>>> to
>>>>> build, train and deploy machine learning models in a robust and agile
>>>> way.
>>>>> The project's goal is to operationalize the machine learning process,
>>>>> allowing data scientists to quickly transition from a successful
>>>> experiment
>>>>> to an automated pipeline in production.
>>>>>
>>>>> The proposal can be found here:
>>>>> https://cwiki.apache.org/confluence/display/INCUBATOR/Apache+Rainbow
>>>>>
>>>>> We would appreciate your feedback and thoughts on the proposal.
>>>>>
>>>>> Thanks,
>>>>> Aviem
>>>>>
>>>>
>>>
>
>
>
> --
> Luciano Resende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org