You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by Maximilian Michels <mx...@apache.org> on 2016/04/05 11:25:37 UTC
Re: Flink on Apache Bigtop project
Hi,
I'm an Apache Flink committer and I'd be very happy to see Flink enter
Bigtop. We have seen quite some interest in Bigtop in the Flink
community. I've been checking out Bhupendra Singh's pull request which
followed this thread: https://github.com/apache/bigtop/pull/93
The packaging of Flink remains one of the biggest hurdles for people
who want to install and run Flink on a cluster. Thus, that was my main
focus when reviewing the PR. Apart from a few issues I found, the pull
request looks good. It would be great if we could bring it into a
mergeable state.
I wonder if it makes sense to break this pull request into several
pull requests? For example, one for the packaging, one for the puppet
scripts, and another one for the smoke tests. That could make
reviewing of the changes easier and people could already try out
incremental changes. I'd be happy to help out with the packaging and
scripting.
What do you think about that?
Thanks,
Max
Re: Flink on Apache Bigtop project
Posted by Olaf Flebbe <of...@oflebbe.de>.
Done.
Olaf
> Am 24.04.2016 um 11:25 schrieb Robert Metzger <rm...@apache.org>:
>
> Can somebody give me Contributor permissions in the Bigtop JIRA?
> I can not comment to issues anymore because of the JIRA spam lockdown :(
>
> On Thu, Apr 21, 2016 at 11:16 AM, Robert Metzger <rm...@apache.org>
> wrote:
>
>> Thank you for your responses. We use shading for two main purposes:
>> - Relocating dependencies
>> - building fat-jars / uber jars
>>
>> I understand that it doesn't make sense to deploy fat-jars in a controlled
>> environment like bigtop. The dist jar of Flink is currently ~80MB, so its
>> not as bad as with the others you've mentioned.
>> Sadly, the build process of Flink is pretty complicated and has a lot of
>> implications, so changing this is certainly not trivial.
>> But its definitively on our TODO list to address the issue.
>>
>> Regarding the dependency relocation: We sadly have to do this because of
>> projects like Guava, which do not provide stable interfaces across releases.
>> Flink is shading away Hadoop's guava so that it doesn't conflict with our
>> guava. We in turn also shade away our guava dependency so that our users
>> can use whatever guava version they want to use.
>>
>> I'll keep building Flink from source for the bigtop packaging.
>>
>>
>>
>> On Wed, Apr 20, 2016 at 11:32 PM, Konstantin Boudnik <co...@apache.org>
>> wrote:
>>
>>> +1 what Olaf said! We already see monstrosities that Spark builds and
>>> provides
>>> (close to 400MB binary blobs) or Zeppelin, which goes over half a gig
>>> (sic!)
>>>
>>> We are producing the stack for very exact specifications, so can afford
>>> removing a bunch of redistributed bits and pieces, as well to avoid the
>>> shading nonsense. Build time is a very small price to pay to get clean
>>> deployment.
>>>
>>> Cos
>>>
>>> On Wed, Apr 20, 2016 at 08:16PM, Olaf Flebbe wrote:
>>>> Hi,
>>>>
>>>> Yep, we like to rebuild from source. AFAIK shading dependencies has to
>>> do
>>>> with uber jars. Since we do not use uber jars, can we please have simple
>>>> jars, since we use a packaging system to do the deployment?
>>>>
>>>> Regards,
>>>> Olaf
>>>>
>>>>
>>>>> Am 20.04.2016 um 16:24 schrieb Robert Metzger <rm...@apache.org>:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I'm currently looking into the existing pull request for the Flink
>>>>> integration [1].
>>>>> While doing that, I wondered if its a requirement from Bigtop to build
>>>>> Flink from source when building the rpm / deb packages?
>>>>> Not building from source would speed up the build a lot and it would
>>> ensure
>>>>> that the binary build is verified by the Flink community (there are
>>> some
>>>>> issues with maven dependency shading when building Flink with maven
>>> 3.3.x)
>>>>>
>>>>>
>>>>>
>>>>> [1] https://github.com/apache/bigtop/pull/93/files
>>>>>
>>>>> On Wed, Apr 6, 2016 at 11:16 AM, Maximilian Michels <mx...@apache.org>
>>> wrote:
>>>>>
>>>>>> Hi Konstantin, hi Jay,
>>>>>>
>>>>>> Thanks for the replies and the prompt progress on the pull request.
>>> It
>>>>>> looks like we will have a Flink RPM package in Bigtop very soon. For
>>>>>> the remaining integration we can open follow-up pull requests.
>>>>>>
>>>>>> If anything comes up while finishing up the work, please ping me!
>>>>>>
>>>>>> Cheers,
>>>>>> Max
>>>>>>
>>>>>> On Tue, Apr 5, 2016 at 6:01 PM, jay vyas <
>>> jayunit100.apache@gmail.com>
>>>>>> wrote:
>>>>>>> Yup , we're getting there ! I tend to this sometimes in after work
>>> hours,
>>>>>>> I'll call the grad students tonite, and see if they want to make a
>>> final
>>>>>>> push this wknd on it, i can help them.
>>>>>>>
>>>>>>> On Tue, Apr 5, 2016 at 11:56 AM, Konstantin Boudnik <cos@apache.org
>>>>
>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi Maximilian.
>>>>>>>>
>>>>>>>> Indeed. the saga of getting Flink into the Apache Bigdata stack
>>> has a
>>>>>> long
>>>>>>>> history ;) It's good to see it's finally converges. Your proposal
>>> of
>>>>>>>> breaking
>>>>>>>> the existing PR in peces certainly makes sense! That's how we
>>> prefer to
>>>>>> do
>>>>>>>> things as well - smaller changes are easier to check, fix, and even
>>>>>> revert
>>>>>>>> if
>>>>>>>> needed.
>>>>>>>>
>>>>>>>> Another thing: we normally would expect to have a single commit for
>>>>>> JIRA,
>>>>>>>> so
>>>>>>>> it would make sense to squash (rebase) the existing PR into smaller
>>>>>> number
>>>>>>>> of
>>>>>>>> commits. Otherwise, it is pretty hard to navigate through all of
>>> them.
>>>>>>>>
>>>>>>>> Please don't hesitate to ping the list shall you need any
>>> assistance.
>>>>>>>> Regards,
>>>>>>>> Cos
>>>>>>>>
>>>>>>>> On Tue, Apr 05, 2016 at 11:25AM, Maximilian Michels wrote:
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I'm an Apache Flink committer and I'd be very happy to see Flink
>>> enter
>>>>>>>>> Bigtop. We have seen quite some interest in Bigtop in the Flink
>>>>>>>>> community. I've been checking out Bhupendra Singh's pull request
>>> which
>>>>>>>>> followed this thread: https://github.com/apache/bigtop/pull/93
>>>>>>>>>
>>>>>>>>> The packaging of Flink remains one of the biggest hurdles for
>>> people
>>>>>>>>> who want to install and run Flink on a cluster. Thus, that was my
>>> main
>>>>>>>>> focus when reviewing the PR. Apart from a few issues I found, the
>>> pull
>>>>>>>>> request looks good. It would be great if we could bring it into a
>>>>>>>>> mergeable state.
>>>>>>>>>
>>>>>>>>> I wonder if it makes sense to break this pull request into several
>>>>>>>>> pull requests? For example, one for the packaging, one for the
>>> puppet
>>>>>>>>> scripts, and another one for the smoke tests. That could make
>>>>>>>>> reviewing of the changes easier and people could already try out
>>>>>>>>> incremental changes. I'd be happy to help out with the packaging
>>> and
>>>>>>>>> scripting.
>>>>>>>>>
>>>>>>>>> What do you think about that?
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Max
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> jay vyas
>>>>>>
>>>>
>>>
>>>
>>>
>>
Re: Flink on Apache Bigtop project
Posted by Robert Metzger <rm...@apache.org>.
Can somebody give me Contributor permissions in the Bigtop JIRA?
I can not comment to issues anymore because of the JIRA spam lockdown :(
On Thu, Apr 21, 2016 at 11:16 AM, Robert Metzger <rm...@apache.org>
wrote:
> Thank you for your responses. We use shading for two main purposes:
> - Relocating dependencies
> - building fat-jars / uber jars
>
> I understand that it doesn't make sense to deploy fat-jars in a controlled
> environment like bigtop. The dist jar of Flink is currently ~80MB, so its
> not as bad as with the others you've mentioned.
> Sadly, the build process of Flink is pretty complicated and has a lot of
> implications, so changing this is certainly not trivial.
> But its definitively on our TODO list to address the issue.
>
> Regarding the dependency relocation: We sadly have to do this because of
> projects like Guava, which do not provide stable interfaces across releases.
> Flink is shading away Hadoop's guava so that it doesn't conflict with our
> guava. We in turn also shade away our guava dependency so that our users
> can use whatever guava version they want to use.
>
> I'll keep building Flink from source for the bigtop packaging.
>
>
>
> On Wed, Apr 20, 2016 at 11:32 PM, Konstantin Boudnik <co...@apache.org>
> wrote:
>
>> +1 what Olaf said! We already see monstrosities that Spark builds and
>> provides
>> (close to 400MB binary blobs) or Zeppelin, which goes over half a gig
>> (sic!)
>>
>> We are producing the stack for very exact specifications, so can afford
>> removing a bunch of redistributed bits and pieces, as well to avoid the
>> shading nonsense. Build time is a very small price to pay to get clean
>> deployment.
>>
>> Cos
>>
>> On Wed, Apr 20, 2016 at 08:16PM, Olaf Flebbe wrote:
>> > Hi,
>> >
>> > Yep, we like to rebuild from source. AFAIK shading dependencies has to
>> do
>> > with uber jars. Since we do not use uber jars, can we please have simple
>> > jars, since we use a packaging system to do the deployment?
>> >
>> > Regards,
>> > Olaf
>> >
>> >
>> > > Am 20.04.2016 um 16:24 schrieb Robert Metzger <rm...@apache.org>:
>> > >
>> > > Hi,
>> > >
>> > > I'm currently looking into the existing pull request for the Flink
>> > > integration [1].
>> > > While doing that, I wondered if its a requirement from Bigtop to build
>> > > Flink from source when building the rpm / deb packages?
>> > > Not building from source would speed up the build a lot and it would
>> ensure
>> > > that the binary build is verified by the Flink community (there are
>> some
>> > > issues with maven dependency shading when building Flink with maven
>> 3.3.x)
>> > >
>> > >
>> > >
>> > > [1] https://github.com/apache/bigtop/pull/93/files
>> > >
>> > > On Wed, Apr 6, 2016 at 11:16 AM, Maximilian Michels <mx...@apache.org>
>> wrote:
>> > >
>> > >> Hi Konstantin, hi Jay,
>> > >>
>> > >> Thanks for the replies and the prompt progress on the pull request.
>> It
>> > >> looks like we will have a Flink RPM package in Bigtop very soon. For
>> > >> the remaining integration we can open follow-up pull requests.
>> > >>
>> > >> If anything comes up while finishing up the work, please ping me!
>> > >>
>> > >> Cheers,
>> > >> Max
>> > >>
>> > >> On Tue, Apr 5, 2016 at 6:01 PM, jay vyas <
>> jayunit100.apache@gmail.com>
>> > >> wrote:
>> > >>> Yup , we're getting there ! I tend to this sometimes in after work
>> hours,
>> > >>> I'll call the grad students tonite, and see if they want to make a
>> final
>> > >>> push this wknd on it, i can help them.
>> > >>>
>> > >>> On Tue, Apr 5, 2016 at 11:56 AM, Konstantin Boudnik <cos@apache.org
>> >
>> > >> wrote:
>> > >>>
>> > >>>> Hi Maximilian.
>> > >>>>
>> > >>>> Indeed. the saga of getting Flink into the Apache Bigdata stack
>> has a
>> > >> long
>> > >>>> history ;) It's good to see it's finally converges. Your proposal
>> of
>> > >>>> breaking
>> > >>>> the existing PR in peces certainly makes sense! That's how we
>> prefer to
>> > >> do
>> > >>>> things as well - smaller changes are easier to check, fix, and even
>> > >> revert
>> > >>>> if
>> > >>>> needed.
>> > >>>>
>> > >>>> Another thing: we normally would expect to have a single commit for
>> > >> JIRA,
>> > >>>> so
>> > >>>> it would make sense to squash (rebase) the existing PR into smaller
>> > >> number
>> > >>>> of
>> > >>>> commits. Otherwise, it is pretty hard to navigate through all of
>> them.
>> > >>>>
>> > >>>> Please don't hesitate to ping the list shall you need any
>> assistance.
>> > >>>> Regards,
>> > >>>> Cos
>> > >>>>
>> > >>>> On Tue, Apr 05, 2016 at 11:25AM, Maximilian Michels wrote:
>> > >>>>> Hi,
>> > >>>>>
>> > >>>>> I'm an Apache Flink committer and I'd be very happy to see Flink
>> enter
>> > >>>>> Bigtop. We have seen quite some interest in Bigtop in the Flink
>> > >>>>> community. I've been checking out Bhupendra Singh's pull request
>> which
>> > >>>>> followed this thread: https://github.com/apache/bigtop/pull/93
>> > >>>>>
>> > >>>>> The packaging of Flink remains one of the biggest hurdles for
>> people
>> > >>>>> who want to install and run Flink on a cluster. Thus, that was my
>> main
>> > >>>>> focus when reviewing the PR. Apart from a few issues I found, the
>> pull
>> > >>>>> request looks good. It would be great if we could bring it into a
>> > >>>>> mergeable state.
>> > >>>>>
>> > >>>>> I wonder if it makes sense to break this pull request into several
>> > >>>>> pull requests? For example, one for the packaging, one for the
>> puppet
>> > >>>>> scripts, and another one for the smoke tests. That could make
>> > >>>>> reviewing of the changes easier and people could already try out
>> > >>>>> incremental changes. I'd be happy to help out with the packaging
>> and
>> > >>>>> scripting.
>> > >>>>>
>> > >>>>> What do you think about that?
>> > >>>>>
>> > >>>>> Thanks,
>> > >>>>> Max
>> > >>>>
>> > >>>
>> > >>>
>> > >>>
>> > >>> --
>> > >>> jay vyas
>> > >>
>> >
>>
>>
>>
>
Re: Flink on Apache Bigtop project
Posted by Robert Metzger <rm...@apache.org>.
Thank you for your responses. We use shading for two main purposes:
- Relocating dependencies
- building fat-jars / uber jars
I understand that it doesn't make sense to deploy fat-jars in a controlled
environment like bigtop. The dist jar of Flink is currently ~80MB, so its
not as bad as with the others you've mentioned.
Sadly, the build process of Flink is pretty complicated and has a lot of
implications, so changing this is certainly not trivial.
But its definitively on our TODO list to address the issue.
Regarding the dependency relocation: We sadly have to do this because of
projects like Guava, which do not provide stable interfaces across releases.
Flink is shading away Hadoop's guava so that it doesn't conflict with our
guava. We in turn also shade away our guava dependency so that our users
can use whatever guava version they want to use.
I'll keep building Flink from source for the bigtop packaging.
On Wed, Apr 20, 2016 at 11:32 PM, Konstantin Boudnik <co...@apache.org> wrote:
> +1 what Olaf said! We already see monstrosities that Spark builds and
> provides
> (close to 400MB binary blobs) or Zeppelin, which goes over half a gig
> (sic!)
>
> We are producing the stack for very exact specifications, so can afford
> removing a bunch of redistributed bits and pieces, as well to avoid the
> shading nonsense. Build time is a very small price to pay to get clean
> deployment.
>
> Cos
>
> On Wed, Apr 20, 2016 at 08:16PM, Olaf Flebbe wrote:
> > Hi,
> >
> > Yep, we like to rebuild from source. AFAIK shading dependencies has to do
> > with uber jars. Since we do not use uber jars, can we please have simple
> > jars, since we use a packaging system to do the deployment?
> >
> > Regards,
> > Olaf
> >
> >
> > > Am 20.04.2016 um 16:24 schrieb Robert Metzger <rm...@apache.org>:
> > >
> > > Hi,
> > >
> > > I'm currently looking into the existing pull request for the Flink
> > > integration [1].
> > > While doing that, I wondered if its a requirement from Bigtop to build
> > > Flink from source when building the rpm / deb packages?
> > > Not building from source would speed up the build a lot and it would
> ensure
> > > that the binary build is verified by the Flink community (there are
> some
> > > issues with maven dependency shading when building Flink with maven
> 3.3.x)
> > >
> > >
> > >
> > > [1] https://github.com/apache/bigtop/pull/93/files
> > >
> > > On Wed, Apr 6, 2016 at 11:16 AM, Maximilian Michels <mx...@apache.org>
> wrote:
> > >
> > >> Hi Konstantin, hi Jay,
> > >>
> > >> Thanks for the replies and the prompt progress on the pull request. It
> > >> looks like we will have a Flink RPM package in Bigtop very soon. For
> > >> the remaining integration we can open follow-up pull requests.
> > >>
> > >> If anything comes up while finishing up the work, please ping me!
> > >>
> > >> Cheers,
> > >> Max
> > >>
> > >> On Tue, Apr 5, 2016 at 6:01 PM, jay vyas <jayunit100.apache@gmail.com
> >
> > >> wrote:
> > >>> Yup , we're getting there ! I tend to this sometimes in after work
> hours,
> > >>> I'll call the grad students tonite, and see if they want to make a
> final
> > >>> push this wknd on it, i can help them.
> > >>>
> > >>> On Tue, Apr 5, 2016 at 11:56 AM, Konstantin Boudnik <co...@apache.org>
> > >> wrote:
> > >>>
> > >>>> Hi Maximilian.
> > >>>>
> > >>>> Indeed. the saga of getting Flink into the Apache Bigdata stack has
> a
> > >> long
> > >>>> history ;) It's good to see it's finally converges. Your proposal of
> > >>>> breaking
> > >>>> the existing PR in peces certainly makes sense! That's how we
> prefer to
> > >> do
> > >>>> things as well - smaller changes are easier to check, fix, and even
> > >> revert
> > >>>> if
> > >>>> needed.
> > >>>>
> > >>>> Another thing: we normally would expect to have a single commit for
> > >> JIRA,
> > >>>> so
> > >>>> it would make sense to squash (rebase) the existing PR into smaller
> > >> number
> > >>>> of
> > >>>> commits. Otherwise, it is pretty hard to navigate through all of
> them.
> > >>>>
> > >>>> Please don't hesitate to ping the list shall you need any
> assistance.
> > >>>> Regards,
> > >>>> Cos
> > >>>>
> > >>>> On Tue, Apr 05, 2016 at 11:25AM, Maximilian Michels wrote:
> > >>>>> Hi,
> > >>>>>
> > >>>>> I'm an Apache Flink committer and I'd be very happy to see Flink
> enter
> > >>>>> Bigtop. We have seen quite some interest in Bigtop in the Flink
> > >>>>> community. I've been checking out Bhupendra Singh's pull request
> which
> > >>>>> followed this thread: https://github.com/apache/bigtop/pull/93
> > >>>>>
> > >>>>> The packaging of Flink remains one of the biggest hurdles for
> people
> > >>>>> who want to install and run Flink on a cluster. Thus, that was my
> main
> > >>>>> focus when reviewing the PR. Apart from a few issues I found, the
> pull
> > >>>>> request looks good. It would be great if we could bring it into a
> > >>>>> mergeable state.
> > >>>>>
> > >>>>> I wonder if it makes sense to break this pull request into several
> > >>>>> pull requests? For example, one for the packaging, one for the
> puppet
> > >>>>> scripts, and another one for the smoke tests. That could make
> > >>>>> reviewing of the changes easier and people could already try out
> > >>>>> incremental changes. I'd be happy to help out with the packaging
> and
> > >>>>> scripting.
> > >>>>>
> > >>>>> What do you think about that?
> > >>>>>
> > >>>>> Thanks,
> > >>>>> Max
> > >>>>
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>> jay vyas
> > >>
> >
>
>
>
Re: Flink on Apache Bigtop project
Posted by Konstantin Boudnik <co...@apache.org>.
+1 what Olaf said! We already see monstrosities that Spark builds and provides
(close to 400MB binary blobs) or Zeppelin, which goes over half a gig (sic!)
We are producing the stack for very exact specifications, so can afford
removing a bunch of redistributed bits and pieces, as well to avoid the
shading nonsense. Build time is a very small price to pay to get clean
deployment.
Cos
On Wed, Apr 20, 2016 at 08:16PM, Olaf Flebbe wrote:
> Hi,
>
> Yep, we like to rebuild from source. AFAIK shading dependencies has to do
> with uber jars. Since we do not use uber jars, can we please have simple
> jars, since we use a packaging system to do the deployment?
>
> Regards,
> Olaf
>
>
> > Am 20.04.2016 um 16:24 schrieb Robert Metzger <rm...@apache.org>:
> >
> > Hi,
> >
> > I'm currently looking into the existing pull request for the Flink
> > integration [1].
> > While doing that, I wondered if its a requirement from Bigtop to build
> > Flink from source when building the rpm / deb packages?
> > Not building from source would speed up the build a lot and it would ensure
> > that the binary build is verified by the Flink community (there are some
> > issues with maven dependency shading when building Flink with maven 3.3.x)
> >
> >
> >
> > [1] https://github.com/apache/bigtop/pull/93/files
> >
> > On Wed, Apr 6, 2016 at 11:16 AM, Maximilian Michels <mx...@apache.org> wrote:
> >
> >> Hi Konstantin, hi Jay,
> >>
> >> Thanks for the replies and the prompt progress on the pull request. It
> >> looks like we will have a Flink RPM package in Bigtop very soon. For
> >> the remaining integration we can open follow-up pull requests.
> >>
> >> If anything comes up while finishing up the work, please ping me!
> >>
> >> Cheers,
> >> Max
> >>
> >> On Tue, Apr 5, 2016 at 6:01 PM, jay vyas <ja...@gmail.com>
> >> wrote:
> >>> Yup , we're getting there ! I tend to this sometimes in after work hours,
> >>> I'll call the grad students tonite, and see if they want to make a final
> >>> push this wknd on it, i can help them.
> >>>
> >>> On Tue, Apr 5, 2016 at 11:56 AM, Konstantin Boudnik <co...@apache.org>
> >> wrote:
> >>>
> >>>> Hi Maximilian.
> >>>>
> >>>> Indeed. the saga of getting Flink into the Apache Bigdata stack has a
> >> long
> >>>> history ;) It's good to see it's finally converges. Your proposal of
> >>>> breaking
> >>>> the existing PR in peces certainly makes sense! That's how we prefer to
> >> do
> >>>> things as well - smaller changes are easier to check, fix, and even
> >> revert
> >>>> if
> >>>> needed.
> >>>>
> >>>> Another thing: we normally would expect to have a single commit for
> >> JIRA,
> >>>> so
> >>>> it would make sense to squash (rebase) the existing PR into smaller
> >> number
> >>>> of
> >>>> commits. Otherwise, it is pretty hard to navigate through all of them.
> >>>>
> >>>> Please don't hesitate to ping the list shall you need any assistance.
> >>>> Regards,
> >>>> Cos
> >>>>
> >>>> On Tue, Apr 05, 2016 at 11:25AM, Maximilian Michels wrote:
> >>>>> Hi,
> >>>>>
> >>>>> I'm an Apache Flink committer and I'd be very happy to see Flink enter
> >>>>> Bigtop. We have seen quite some interest in Bigtop in the Flink
> >>>>> community. I've been checking out Bhupendra Singh's pull request which
> >>>>> followed this thread: https://github.com/apache/bigtop/pull/93
> >>>>>
> >>>>> The packaging of Flink remains one of the biggest hurdles for people
> >>>>> who want to install and run Flink on a cluster. Thus, that was my main
> >>>>> focus when reviewing the PR. Apart from a few issues I found, the pull
> >>>>> request looks good. It would be great if we could bring it into a
> >>>>> mergeable state.
> >>>>>
> >>>>> I wonder if it makes sense to break this pull request into several
> >>>>> pull requests? For example, one for the packaging, one for the puppet
> >>>>> scripts, and another one for the smoke tests. That could make
> >>>>> reviewing of the changes easier and people could already try out
> >>>>> incremental changes. I'd be happy to help out with the packaging and
> >>>>> scripting.
> >>>>>
> >>>>> What do you think about that?
> >>>>>
> >>>>> Thanks,
> >>>>> Max
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> jay vyas
> >>
>
Re: Flink on Apache Bigtop project
Posted by Olaf Flebbe <of...@oflebbe.de>.
Hi,
Yep, we like to rebuild from source. AFAIK shading dependencies has to do with uber jars. Since we do not use uber jars, can we please have simple jars, since we use a packaging system to do the deployment?
Regards,
Olaf
> Am 20.04.2016 um 16:24 schrieb Robert Metzger <rm...@apache.org>:
>
> Hi,
>
> I'm currently looking into the existing pull request for the Flink
> integration [1].
> While doing that, I wondered if its a requirement from Bigtop to build
> Flink from source when building the rpm / deb packages?
> Not building from source would speed up the build a lot and it would ensure
> that the binary build is verified by the Flink community (there are some
> issues with maven dependency shading when building Flink with maven 3.3.x)
>
>
>
> [1] https://github.com/apache/bigtop/pull/93/files
>
> On Wed, Apr 6, 2016 at 11:16 AM, Maximilian Michels <mx...@apache.org> wrote:
>
>> Hi Konstantin, hi Jay,
>>
>> Thanks for the replies and the prompt progress on the pull request. It
>> looks like we will have a Flink RPM package in Bigtop very soon. For
>> the remaining integration we can open follow-up pull requests.
>>
>> If anything comes up while finishing up the work, please ping me!
>>
>> Cheers,
>> Max
>>
>> On Tue, Apr 5, 2016 at 6:01 PM, jay vyas <ja...@gmail.com>
>> wrote:
>>> Yup , we're getting there ! I tend to this sometimes in after work hours,
>>> I'll call the grad students tonite, and see if they want to make a final
>>> push this wknd on it, i can help them.
>>>
>>> On Tue, Apr 5, 2016 at 11:56 AM, Konstantin Boudnik <co...@apache.org>
>> wrote:
>>>
>>>> Hi Maximilian.
>>>>
>>>> Indeed. the saga of getting Flink into the Apache Bigdata stack has a
>> long
>>>> history ;) It's good to see it's finally converges. Your proposal of
>>>> breaking
>>>> the existing PR in peces certainly makes sense! That's how we prefer to
>> do
>>>> things as well - smaller changes are easier to check, fix, and even
>> revert
>>>> if
>>>> needed.
>>>>
>>>> Another thing: we normally would expect to have a single commit for
>> JIRA,
>>>> so
>>>> it would make sense to squash (rebase) the existing PR into smaller
>> number
>>>> of
>>>> commits. Otherwise, it is pretty hard to navigate through all of them.
>>>>
>>>> Please don't hesitate to ping the list shall you need any assistance.
>>>> Regards,
>>>> Cos
>>>>
>>>> On Tue, Apr 05, 2016 at 11:25AM, Maximilian Michels wrote:
>>>>> Hi,
>>>>>
>>>>> I'm an Apache Flink committer and I'd be very happy to see Flink enter
>>>>> Bigtop. We have seen quite some interest in Bigtop in the Flink
>>>>> community. I've been checking out Bhupendra Singh's pull request which
>>>>> followed this thread: https://github.com/apache/bigtop/pull/93
>>>>>
>>>>> The packaging of Flink remains one of the biggest hurdles for people
>>>>> who want to install and run Flink on a cluster. Thus, that was my main
>>>>> focus when reviewing the PR. Apart from a few issues I found, the pull
>>>>> request looks good. It would be great if we could bring it into a
>>>>> mergeable state.
>>>>>
>>>>> I wonder if it makes sense to break this pull request into several
>>>>> pull requests? For example, one for the packaging, one for the puppet
>>>>> scripts, and another one for the smoke tests. That could make
>>>>> reviewing of the changes easier and people could already try out
>>>>> incremental changes. I'd be happy to help out with the packaging and
>>>>> scripting.
>>>>>
>>>>> What do you think about that?
>>>>>
>>>>> Thanks,
>>>>> Max
>>>>
>>>
>>>
>>>
>>> --
>>> jay vyas
>>
Re: Flink on Apache Bigtop project
Posted by Robert Metzger <rm...@apache.org>.
Hi,
I'm currently looking into the existing pull request for the Flink
integration [1].
While doing that, I wondered if its a requirement from Bigtop to build
Flink from source when building the rpm / deb packages?
Not building from source would speed up the build a lot and it would ensure
that the binary build is verified by the Flink community (there are some
issues with maven dependency shading when building Flink with maven 3.3.x)
[1] https://github.com/apache/bigtop/pull/93/files
On Wed, Apr 6, 2016 at 11:16 AM, Maximilian Michels <mx...@apache.org> wrote:
> Hi Konstantin, hi Jay,
>
> Thanks for the replies and the prompt progress on the pull request. It
> looks like we will have a Flink RPM package in Bigtop very soon. For
> the remaining integration we can open follow-up pull requests.
>
> If anything comes up while finishing up the work, please ping me!
>
> Cheers,
> Max
>
> On Tue, Apr 5, 2016 at 6:01 PM, jay vyas <ja...@gmail.com>
> wrote:
> > Yup , we're getting there ! I tend to this sometimes in after work hours,
> > I'll call the grad students tonite, and see if they want to make a final
> > push this wknd on it, i can help them.
> >
> > On Tue, Apr 5, 2016 at 11:56 AM, Konstantin Boudnik <co...@apache.org>
> wrote:
> >
> >> Hi Maximilian.
> >>
> >> Indeed. the saga of getting Flink into the Apache Bigdata stack has a
> long
> >> history ;) It's good to see it's finally converges. Your proposal of
> >> breaking
> >> the existing PR in peces certainly makes sense! That's how we prefer to
> do
> >> things as well - smaller changes are easier to check, fix, and even
> revert
> >> if
> >> needed.
> >>
> >> Another thing: we normally would expect to have a single commit for
> JIRA,
> >> so
> >> it would make sense to squash (rebase) the existing PR into smaller
> number
> >> of
> >> commits. Otherwise, it is pretty hard to navigate through all of them.
> >>
> >> Please don't hesitate to ping the list shall you need any assistance.
> >> Regards,
> >> Cos
> >>
> >> On Tue, Apr 05, 2016 at 11:25AM, Maximilian Michels wrote:
> >> > Hi,
> >> >
> >> > I'm an Apache Flink committer and I'd be very happy to see Flink enter
> >> > Bigtop. We have seen quite some interest in Bigtop in the Flink
> >> > community. I've been checking out Bhupendra Singh's pull request which
> >> > followed this thread: https://github.com/apache/bigtop/pull/93
> >> >
> >> > The packaging of Flink remains one of the biggest hurdles for people
> >> > who want to install and run Flink on a cluster. Thus, that was my main
> >> > focus when reviewing the PR. Apart from a few issues I found, the pull
> >> > request looks good. It would be great if we could bring it into a
> >> > mergeable state.
> >> >
> >> > I wonder if it makes sense to break this pull request into several
> >> > pull requests? For example, one for the packaging, one for the puppet
> >> > scripts, and another one for the smoke tests. That could make
> >> > reviewing of the changes easier and people could already try out
> >> > incremental changes. I'd be happy to help out with the packaging and
> >> > scripting.
> >> >
> >> > What do you think about that?
> >> >
> >> > Thanks,
> >> > Max
> >>
> >
> >
> >
> > --
> > jay vyas
>
Re: Flink on Apache Bigtop project
Posted by Maximilian Michels <mx...@apache.org>.
Hi Konstantin, hi Jay,
Thanks for the replies and the prompt progress on the pull request. It
looks like we will have a Flink RPM package in Bigtop very soon. For
the remaining integration we can open follow-up pull requests.
If anything comes up while finishing up the work, please ping me!
Cheers,
Max
On Tue, Apr 5, 2016 at 6:01 PM, jay vyas <ja...@gmail.com> wrote:
> Yup , we're getting there ! I tend to this sometimes in after work hours,
> I'll call the grad students tonite, and see if they want to make a final
> push this wknd on it, i can help them.
>
> On Tue, Apr 5, 2016 at 11:56 AM, Konstantin Boudnik <co...@apache.org> wrote:
>
>> Hi Maximilian.
>>
>> Indeed. the saga of getting Flink into the Apache Bigdata stack has a long
>> history ;) It's good to see it's finally converges. Your proposal of
>> breaking
>> the existing PR in peces certainly makes sense! That's how we prefer to do
>> things as well - smaller changes are easier to check, fix, and even revert
>> if
>> needed.
>>
>> Another thing: we normally would expect to have a single commit for JIRA,
>> so
>> it would make sense to squash (rebase) the existing PR into smaller number
>> of
>> commits. Otherwise, it is pretty hard to navigate through all of them.
>>
>> Please don't hesitate to ping the list shall you need any assistance.
>> Regards,
>> Cos
>>
>> On Tue, Apr 05, 2016 at 11:25AM, Maximilian Michels wrote:
>> > Hi,
>> >
>> > I'm an Apache Flink committer and I'd be very happy to see Flink enter
>> > Bigtop. We have seen quite some interest in Bigtop in the Flink
>> > community. I've been checking out Bhupendra Singh's pull request which
>> > followed this thread: https://github.com/apache/bigtop/pull/93
>> >
>> > The packaging of Flink remains one of the biggest hurdles for people
>> > who want to install and run Flink on a cluster. Thus, that was my main
>> > focus when reviewing the PR. Apart from a few issues I found, the pull
>> > request looks good. It would be great if we could bring it into a
>> > mergeable state.
>> >
>> > I wonder if it makes sense to break this pull request into several
>> > pull requests? For example, one for the packaging, one for the puppet
>> > scripts, and another one for the smoke tests. That could make
>> > reviewing of the changes easier and people could already try out
>> > incremental changes. I'd be happy to help out with the packaging and
>> > scripting.
>> >
>> > What do you think about that?
>> >
>> > Thanks,
>> > Max
>>
>
>
>
> --
> jay vyas
Re: Flink on Apache Bigtop project
Posted by jay vyas <ja...@gmail.com>.
Yup , we're getting there ! I tend to this sometimes in after work hours,
I'll call the grad students tonite, and see if they want to make a final
push this wknd on it, i can help them.
On Tue, Apr 5, 2016 at 11:56 AM, Konstantin Boudnik <co...@apache.org> wrote:
> Hi Maximilian.
>
> Indeed. the saga of getting Flink into the Apache Bigdata stack has a long
> history ;) It's good to see it's finally converges. Your proposal of
> breaking
> the existing PR in peces certainly makes sense! That's how we prefer to do
> things as well - smaller changes are easier to check, fix, and even revert
> if
> needed.
>
> Another thing: we normally would expect to have a single commit for JIRA,
> so
> it would make sense to squash (rebase) the existing PR into smaller number
> of
> commits. Otherwise, it is pretty hard to navigate through all of them.
>
> Please don't hesitate to ping the list shall you need any assistance.
> Regards,
> Cos
>
> On Tue, Apr 05, 2016 at 11:25AM, Maximilian Michels wrote:
> > Hi,
> >
> > I'm an Apache Flink committer and I'd be very happy to see Flink enter
> > Bigtop. We have seen quite some interest in Bigtop in the Flink
> > community. I've been checking out Bhupendra Singh's pull request which
> > followed this thread: https://github.com/apache/bigtop/pull/93
> >
> > The packaging of Flink remains one of the biggest hurdles for people
> > who want to install and run Flink on a cluster. Thus, that was my main
> > focus when reviewing the PR. Apart from a few issues I found, the pull
> > request looks good. It would be great if we could bring it into a
> > mergeable state.
> >
> > I wonder if it makes sense to break this pull request into several
> > pull requests? For example, one for the packaging, one for the puppet
> > scripts, and another one for the smoke tests. That could make
> > reviewing of the changes easier and people could already try out
> > incremental changes. I'd be happy to help out with the packaging and
> > scripting.
> >
> > What do you think about that?
> >
> > Thanks,
> > Max
>
--
jay vyas
Re: Flink on Apache Bigtop project
Posted by Konstantin Boudnik <co...@apache.org>.
Hi Maximilian.
Indeed. the saga of getting Flink into the Apache Bigdata stack has a long
history ;) It's good to see it's finally converges. Your proposal of breaking
the existing PR in peces certainly makes sense! That's how we prefer to do
things as well - smaller changes are easier to check, fix, and even revert if
needed.
Another thing: we normally would expect to have a single commit for JIRA, so
it would make sense to squash (rebase) the existing PR into smaller number of
commits. Otherwise, it is pretty hard to navigate through all of them.
Please don't hesitate to ping the list shall you need any assistance.
Regards,
Cos
On Tue, Apr 05, 2016 at 11:25AM, Maximilian Michels wrote:
> Hi,
>
> I'm an Apache Flink committer and I'd be very happy to see Flink enter
> Bigtop. We have seen quite some interest in Bigtop in the Flink
> community. I've been checking out Bhupendra Singh's pull request which
> followed this thread: https://github.com/apache/bigtop/pull/93
>
> The packaging of Flink remains one of the biggest hurdles for people
> who want to install and run Flink on a cluster. Thus, that was my main
> focus when reviewing the PR. Apart from a few issues I found, the pull
> request looks good. It would be great if we could bring it into a
> mergeable state.
>
> I wonder if it makes sense to break this pull request into several
> pull requests? For example, one for the packaging, one for the puppet
> scripts, and another one for the smoke tests. That could make
> reviewing of the changes easier and people could already try out
> incremental changes. I'd be happy to help out with the packaging and
> scripting.
>
> What do you think about that?
>
> Thanks,
> Max