You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by Manfred PAUL <ma...@gmail.com> on 2020/09/22 13:48:34 UTC
Developer documenation
Hi,
I would like to participate in developing bigtop. For that I am searching
for some material (doc, wiki, video, etc) to get all the information needed
to build the bigtop distribution.
All the information I found was for older versions of bigtop (1.1, 1.2 and
1.3) but might still be up to date, so I am asking.
Is written that a developer should use the pre-built docker images for
building the distribution and not set up his or her own toolchain. Or
should we use only a CI? Is it even possible to build everything on a 16GB
Macbook Pro? Etc.
Thanks
Manfred
Re: Developer documenation
Posted by Matt Andruff <ma...@andruffsolutions.com>.
Olaf, I'd love to update the cwik myself and rearrange some of the cwiki.
Here's a ticket describing what I'd like to start off doing:
https://issues.apache.org/jira/browse/BIGTOP-3424
I'd also like to complete these changes:
https://issues.apache.org/jira/browse/BIGTOP-3360
Actually I'd like to fill in more documentation on
https://cwiki.apache.org/confluence/display/BIGTOP/Quickstart+Guide%3A+Bigtop+Integration+Test+Framework+2.0
to help round out the info.
Hope that helps to explain. My user is mandruff for cwiki
On Sat, Sep 26, 2020 at 10:41 AM Olaf Flebbe <of...@oflebbe.de> wrote:
> Hi,
>
> you could either
>
> add more / updated Markdown files in the repository by an pull request.
> Update / Fix the website in src/site in the repository with an pull
> request.
> Move/copy info from the cwiki/confluence to Markdown into the repository.
> or if you would like to work on our cwiki please send me your account and
> the pages you like to update.
>
> Olaf
>
> > Am 23.09.2020 um 01:03 schrieb Matt Andruff <ma...@andruffsolutions.com>:
> >
> > Olaf, I'm glad we are in agreement.
> >
> > I'm not sure what to ask for but I definitely feel we can de-emphasize
> the
> > 0.x stuff and promote the new stuff... Happy to help, just point me in
> the
> > right direction. I am excited to contribute and it's really a dream come
> > true. (sorry for admitting this out loud)
> >
> > On Tue, Sep 22, 2020 at 3:38 PM Olaf Flebbe <of...@oflebbe.de> wrote:
> >
> >> Hi Matt,
> >>
> >> I totally agree that documentation needs to be updated.
> >>
> >> From my perspective markdown files in the git repository are most
> suitable
> >> for users and accessible to our users. I am very open to accept Pull
> >> requests / Patches for docs.
> >>
> >> Contributors can ask for confluence (cwiki) privileges as well, if we
> are
> >> aligned on what they want to work on. (At least that’s my opinion, not
> >> necessary representative for the project)
> >>
> >> Best
> >> Olaf
> >>
> >>
> >>
> >>> Am 22.09.2020 um 19:55 schrieb Matt Andruff <matt@andruffsolutions.com
> >:
> >>>
> >>> This goes back to why we need to update the documentation... I'll
> update
> >>> that soon so it is easier to find.
> >>>
> >>> Essentially I suggest using the instructions on git to get going:
> >>>
> >>>
> >>
> https://github.com/apache/bigtop#for-developers-building-a-component-from-git-repository
> >>>
> >>> For more info on the commands used (on that git page use this helpful
> gem
> >>> (that is buried a bit)
> >>>
> >>
> https://cwiki.apache.org/confluence/display/BIGTOP/Quickstart+Guide%3A+Bigtop+Integration+Test+Framework+2.0
> >>>
> >>> Whatever you do, please use docker to build it.... It will save you a
> >> heap
> >>> of dependency issues. Please learn from my pain.
> >>>
> >>> Hope this helps
> >>>
> >>> Matt
> >>>
> >>>
> >>>
> >>>
> >>> On Tue., Sep. 22, 2020, 09:51 Manfred PAUL, <ma...@gmail.com>
> >> wrote:
> >>>
> >>>> Hi,
> >>>>
> >>>> I would like to participate in developing bigtop. For that I am
> >> searching
> >>>> for some material (doc, wiki, video, etc) to get all the information
> >> needed
> >>>> to build the bigtop distribution.
> >>>>
> >>>> All the information I found was for older versions of bigtop (1.1, 1.2
> >> and
> >>>> 1.3) but might still be up to date, so I am asking.
> >>>>
> >>>> Is written that a developer should use the pre-built docker images for
> >>>> building the distribution and not set up his or her own toolchain. Or
> >>>> should we use only a CI? Is it even possible to build everything on a
> >> 16GB
> >>>> Macbook Pro? Etc.
> >>>>
> >>>> Thanks
> >>>>
> >>>> Manfred
> >>>>
> >>
> >>
>
>
Re: Developer documenation
Posted by Olaf Flebbe <of...@oflebbe.de>.
Hi,
you could either
add more / updated Markdown files in the repository by an pull request.
Update / Fix the website in src/site in the repository with an pull request.
Move/copy info from the cwiki/confluence to Markdown into the repository.
or if you would like to work on our cwiki please send me your account and the pages you like to update.
Olaf
> Am 23.09.2020 um 01:03 schrieb Matt Andruff <ma...@andruffsolutions.com>:
>
> Olaf, I'm glad we are in agreement.
>
> I'm not sure what to ask for but I definitely feel we can de-emphasize the
> 0.x stuff and promote the new stuff... Happy to help, just point me in the
> right direction. I am excited to contribute and it's really a dream come
> true. (sorry for admitting this out loud)
>
> On Tue, Sep 22, 2020 at 3:38 PM Olaf Flebbe <of...@oflebbe.de> wrote:
>
>> Hi Matt,
>>
>> I totally agree that documentation needs to be updated.
>>
>> From my perspective markdown files in the git repository are most suitable
>> for users and accessible to our users. I am very open to accept Pull
>> requests / Patches for docs.
>>
>> Contributors can ask for confluence (cwiki) privileges as well, if we are
>> aligned on what they want to work on. (At least that’s my opinion, not
>> necessary representative for the project)
>>
>> Best
>> Olaf
>>
>>
>>
>>> Am 22.09.2020 um 19:55 schrieb Matt Andruff <ma...@andruffsolutions.com>:
>>>
>>> This goes back to why we need to update the documentation... I'll update
>>> that soon so it is easier to find.
>>>
>>> Essentially I suggest using the instructions on git to get going:
>>>
>>>
>> https://github.com/apache/bigtop#for-developers-building-a-component-from-git-repository
>>>
>>> For more info on the commands used (on that git page use this helpful gem
>>> (that is buried a bit)
>>>
>> https://cwiki.apache.org/confluence/display/BIGTOP/Quickstart+Guide%3A+Bigtop+Integration+Test+Framework+2.0
>>>
>>> Whatever you do, please use docker to build it.... It will save you a
>> heap
>>> of dependency issues. Please learn from my pain.
>>>
>>> Hope this helps
>>>
>>> Matt
>>>
>>>
>>>
>>>
>>> On Tue., Sep. 22, 2020, 09:51 Manfred PAUL, <ma...@gmail.com>
>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I would like to participate in developing bigtop. For that I am
>> searching
>>>> for some material (doc, wiki, video, etc) to get all the information
>> needed
>>>> to build the bigtop distribution.
>>>>
>>>> All the information I found was for older versions of bigtop (1.1, 1.2
>> and
>>>> 1.3) but might still be up to date, so I am asking.
>>>>
>>>> Is written that a developer should use the pre-built docker images for
>>>> building the distribution and not set up his or her own toolchain. Or
>>>> should we use only a CI? Is it even possible to build everything on a
>> 16GB
>>>> Macbook Pro? Etc.
>>>>
>>>> Thanks
>>>>
>>>> Manfred
>>>>
>>
>>
Re: Developer documenation
Posted by Matt Andruff <ma...@andruffsolutions.com>.
Olaf, I'm glad we are in agreement.
I'm not sure what to ask for but I definitely feel we can de-emphasize the
0.x stuff and promote the new stuff... Happy to help, just point me in the
right direction. I am excited to contribute and it's really a dream come
true. (sorry for admitting this out loud)
On Tue, Sep 22, 2020 at 3:38 PM Olaf Flebbe <of...@oflebbe.de> wrote:
> Hi Matt,
>
> I totally agree that documentation needs to be updated.
>
> From my perspective markdown files in the git repository are most suitable
> for users and accessible to our users. I am very open to accept Pull
> requests / Patches for docs.
>
> Contributors can ask for confluence (cwiki) privileges as well, if we are
> aligned on what they want to work on. (At least that’s my opinion, not
> necessary representative for the project)
>
> Best
> Olaf
>
>
>
> > Am 22.09.2020 um 19:55 schrieb Matt Andruff <ma...@andruffsolutions.com>:
> >
> > This goes back to why we need to update the documentation... I'll update
> > that soon so it is easier to find.
> >
> > Essentially I suggest using the instructions on git to get going:
> >
> >
> https://github.com/apache/bigtop#for-developers-building-a-component-from-git-repository
> >
> > For more info on the commands used (on that git page use this helpful gem
> > (that is buried a bit)
> >
> https://cwiki.apache.org/confluence/display/BIGTOP/Quickstart+Guide%3A+Bigtop+Integration+Test+Framework+2.0
> >
> > Whatever you do, please use docker to build it.... It will save you a
> heap
> > of dependency issues. Please learn from my pain.
> >
> > Hope this helps
> >
> > Matt
> >
> >
> >
> >
> > On Tue., Sep. 22, 2020, 09:51 Manfred PAUL, <ma...@gmail.com>
> wrote:
> >
> >> Hi,
> >>
> >> I would like to participate in developing bigtop. For that I am
> searching
> >> for some material (doc, wiki, video, etc) to get all the information
> needed
> >> to build the bigtop distribution.
> >>
> >> All the information I found was for older versions of bigtop (1.1, 1.2
> and
> >> 1.3) but might still be up to date, so I am asking.
> >>
> >> Is written that a developer should use the pre-built docker images for
> >> building the distribution and not set up his or her own toolchain. Or
> >> should we use only a CI? Is it even possible to build everything on a
> 16GB
> >> Macbook Pro? Etc.
> >>
> >> Thanks
> >>
> >> Manfred
> >>
>
>
Re: Developer documenation
Posted by Olaf Flebbe <of...@oflebbe.de>.
Hi Matt,
I totally agree that documentation needs to be updated.
From my perspective markdown files in the git repository are most suitable for users and accessible to our users. I am very open to accept Pull requests / Patches for docs.
Contributors can ask for confluence (cwiki) privileges as well, if we are aligned on what they want to work on. (At least that’s my opinion, not necessary representative for the project)
Best
Olaf
> Am 22.09.2020 um 19:55 schrieb Matt Andruff <ma...@andruffsolutions.com>:
>
> This goes back to why we need to update the documentation... I'll update
> that soon so it is easier to find.
>
> Essentially I suggest using the instructions on git to get going:
>
> https://github.com/apache/bigtop#for-developers-building-a-component-from-git-repository
>
> For more info on the commands used (on that git page use this helpful gem
> (that is buried a bit)
> https://cwiki.apache.org/confluence/display/BIGTOP/Quickstart+Guide%3A+Bigtop+Integration+Test+Framework+2.0
>
> Whatever you do, please use docker to build it.... It will save you a heap
> of dependency issues. Please learn from my pain.
>
> Hope this helps
>
> Matt
>
>
>
>
> On Tue., Sep. 22, 2020, 09:51 Manfred PAUL, <ma...@gmail.com> wrote:
>
>> Hi,
>>
>> I would like to participate in developing bigtop. For that I am searching
>> for some material (doc, wiki, video, etc) to get all the information needed
>> to build the bigtop distribution.
>>
>> All the information I found was for older versions of bigtop (1.1, 1.2 and
>> 1.3) but might still be up to date, so I am asking.
>>
>> Is written that a developer should use the pre-built docker images for
>> building the distribution and not set up his or her own toolchain. Or
>> should we use only a CI? Is it even possible to build everything on a 16GB
>> Macbook Pro? Etc.
>>
>> Thanks
>>
>> Manfred
>>
Re: Developer documenation
Posted by Matt Andruff <ma...@andruffsolutions.com>.
This goes back to why we need to update the documentation... I'll update
that soon so it is easier to find.
Essentially I suggest using the instructions on git to get going:
https://github.com/apache/bigtop#for-developers-building-a-component-from-git-repository
For more info on the commands used (on that git page use this helpful gem
(that is buried a bit)
https://cwiki.apache.org/confluence/display/BIGTOP/Quickstart+Guide%3A+Bigtop+Integration+Test+Framework+2.0
Whatever you do, please use docker to build it.... It will save you a heap
of dependency issues. Please learn from my pain.
Hope this helps
Matt
On Tue., Sep. 22, 2020, 09:51 Manfred PAUL, <ma...@gmail.com> wrote:
> Hi,
>
> I would like to participate in developing bigtop. For that I am searching
> for some material (doc, wiki, video, etc) to get all the information needed
> to build the bigtop distribution.
>
> All the information I found was for older versions of bigtop (1.1, 1.2 and
> 1.3) but might still be up to date, so I am asking.
>
> Is written that a developer should use the pre-built docker images for
> building the distribution and not set up his or her own toolchain. Or
> should we use only a CI? Is it even possible to build everything on a 16GB
> Macbook Pro? Etc.
>
> Thanks
>
> Manfred
>
Re: Developer documenation
Posted by Evans Ye <ev...@apache.org>.
Awesome!
Staring from docker images and build up packages would be recommended so
that you can get the idea about how it works.
When you'd like to customize your own big data stack based on bigtop, you
start to build your own docker images instead.
- Evans
Olaf Flebbe <of...@oflebbe.de> 於 2020年9月23日 週三 上午12:05寫道：
> Hi Manfred,
>
> Great!
>
> AFAIK the most up to date information on how to (re)compile an package you
> will find here
>
> https://cwiki.apache.org/confluence/display/BIGTOP/How+to+build+Bigtop-trunk
> <
> https://cwiki.apache.org/confluence/display/BIGTOP/How+to+build+Bigtop-trunk
> >
>
> I am assuming you are targeting linux, not macos, right?
>
> IMO When developing it makes sense to compile „within" the provided docker
> images or improving the docker images.
> The CI’s job is to ensure changes do work on all platforms.
>
> I was using a macbook pro 16GB when I developed packages, so it is
> possible, though not always optimal, since docker filesystem (volume mounts
> from macos to docker guest) is buggy and slow.
>
> Running smoke tests on a mac is possible with workarounds.
>
> Needed workarounds depend very much on what you are trying to accomplish:
> VMware Fusion, docker4mac or depending on CI are at least 3 options I would
> recommend depending on usecase.
>
> Best
> Olaf
>
>
>
>
> > Am 22.09.2020 um 15:48 schrieb Manfred PAUL <ma...@gmail.com>:
> >
> > Hi,
> >
> > I would like to participate in developing bigtop. For that I am searching
> > for some material (doc, wiki, video, etc) to get all the information
> needed
> > to build the bigtop distribution.
> >
> > All the information I found was for older versions of bigtop (1.1, 1.2
> and
> > 1.3) but might still be up to date, so I am asking.
> >
> > Is written that a developer should use the pre-built docker images for
> > building the distribution and not set up his or her own toolchain. Or
> > should we use only a CI? Is it even possible to build everything on a
> 16GB
> > Macbook Pro? Etc.
> >
> > Thanks
> >
> > Manfred
>
>
Re: Developer documenation
Posted by Olaf Flebbe <of...@oflebbe.de>.
Hi Manfred,
Great!
AFAIK the most up to date information on how to (re)compile an package you will find here
https://cwiki.apache.org/confluence/display/BIGTOP/How+to+build+Bigtop-trunk <https://cwiki.apache.org/confluence/display/BIGTOP/How+to+build+Bigtop-trunk>
I am assuming you are targeting linux, not macos, right?
IMO When developing it makes sense to compile „within" the provided docker images or improving the docker images.
The CI’s job is to ensure changes do work on all platforms.
I was using a macbook pro 16GB when I developed packages, so it is possible, though not always optimal, since docker filesystem (volume mounts from macos to docker guest) is buggy and slow.
Running smoke tests on a mac is possible with workarounds.
Needed workarounds depend very much on what you are trying to accomplish: VMware Fusion, docker4mac or depending on CI are at least 3 options I would recommend depending on usecase.
Best
Olaf
> Am 22.09.2020 um 15:48 schrieb Manfred PAUL <ma...@gmail.com>:
>
> Hi,
>
> I would like to participate in developing bigtop. For that I am searching
> for some material (doc, wiki, video, etc) to get all the information needed
> to build the bigtop distribution.
>
> All the information I found was for older versions of bigtop (1.1, 1.2 and
> 1.3) but might still be up to date, so I am asking.
>
> Is written that a developer should use the pre-built docker images for
> building the distribution and not set up his or her own toolchain. Or
> should we use only a CI? Is it even possible to build everything on a 16GB
> Macbook Pro? Etc.
>
> Thanks
>
> Manfred