You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@myriad.apache.org by Javi Roman <jr...@gmail.com> on 2018/02/04 09:09:13 UTC
Regarding local environment based on Vagrant
I would like share with you the strategy that I'm following for a
powerful local development environment for Myriad.
The idea is to create an easy-to-use environment for new contributors,
setting up a Mesos cluster, or DCOS cluster (with hadoop running which
is a tedious task). I guess automate this task is a key point for
accelerating the development and focusing in the Myriad hacking.
In this first task [1] I am working on Vagrant libvirt support for a
plain Mesos cluster. However the roadmap for this environment would
be:
1. vagrant-libvirt based cluster for Mesos, provisioned with Ansible
(already working in my environment).
2. vagrant-libvirt based cluster for DC/OS provisioned with Ansible:
Easier environment for testing Myriad with Marathon, Mesos-DNS, VIPS,
.... and so on.
3. vagrant-virtualbox based cluster (maintain the original environment
used right now).
The layout would be something like this:
vagrant/
├── libvirt
│ ├── dcos
│ │ └── Vagrantfile
│ └── mesos
│ └── Vagrantfile
└── virtualbox
└── Vagrantfile
In a second stage, the final scenario would be, only one folder with
one "smart" Vagrantfile for supporting Mesos and DC/OS for Linux and
OSX, the tool will be able of selecting the suitable virtualization
technology by user configuration.
This is the idea.
[1] https://issues.apache.org/jira/browse/MYRIAD-257
--
Javi Roman
Twitter: @javiromanrh
GitHub: github.com/javiroman
Linkedin: es.linkedin.com/in/javiroman
Big Data Blog: dataintensive.info
Re: Regarding local environment based on Vagrant
Posted by Juan P <jp...@gmail.com>.
+1
On 4 April 2018 at 09:10, Javi Roman <jr...@gmail.com> wrote:
> I going to merge the PR [1] related with the Vagrant environment reworking.
>
> Please feel free to comment whatever enhance or inconvenient you think
> about the commit. I will merge he PR if there are no negative
> comments/votes.
>
>
> [1] https://github.com/apache/incubator-myriad/pull/107
> --
> Javi Roman
>
> Twitter: @javiromanrh
> GitHub: github.com/javiroman
> Linkedin: es.linkedin.com/in/javiroman
> Big Data Blog: dataintensive.info
>
>
> On Mon, Feb 5, 2018 at 2:36 PM, Javi Roman <jr...@gmail.com>
> wrote:
> > For sure, this is something to include in the vagrant testbed. Good
> point.
> > --
> > Javi Roman
> >
> > Twitter: @javiromanrh
> > GitHub: github.com/javiroman
> > Linkedin: es.linkedin.com/in/javiroman
> > Big Data Blog: dataintensive.info
> >
> >
> > On Mon, Feb 5, 2018 at 10:01 AM, Juan P <jp...@gmail.com> wrote:
> >> Hello Javi, I think it's a great idea.
> >>
> >> Is it possible to prepare the automatic environment to perform automatic
> >> end-to-end tests (acceptance) or with a minimum of configuration?
> >>
> >> In later versions, would it be possible to include all the known Yarn
> >> frameworks and prepare some automatic use cases that serve as an example
> >> for the community and regression / integration test?
> >>
> >> Thank you
> >>
> >> On 4 February 2018 at 10:09, Javi Roman <jr...@gmail.com>
> wrote:
> >>
> >>> I would like share with you the strategy that I'm following for a
> >>> powerful local development environment for Myriad.
> >>>
> >>> The idea is to create an easy-to-use environment for new contributors,
> >>> setting up a Mesos cluster, or DCOS cluster (with hadoop running which
> >>> is a tedious task). I guess automate this task is a key point for
> >>> accelerating the development and focusing in the Myriad hacking.
> >>>
> >>> In this first task [1] I am working on Vagrant libvirt support for a
> >>> plain Mesos cluster. However the roadmap for this environment would
> >>> be:
> >>>
> >>> 1. vagrant-libvirt based cluster for Mesos, provisioned with Ansible
> >>> (already working in my environment).
> >>> 2. vagrant-libvirt based cluster for DC/OS provisioned with Ansible:
> >>> Easier environment for testing Myriad with Marathon, Mesos-DNS, VIPS,
> >>> .... and so on.
> >>> 3. vagrant-virtualbox based cluster (maintain the original environment
> >>> used right now).
> >>>
> >>> The layout would be something like this:
> >>>
> >>> vagrant/
> >>> ├── libvirt
> >>> │ ├── dcos
> >>> │ │ └── Vagrantfile
> >>> │ └── mesos
> >>> │ └── Vagrantfile
> >>> └── virtualbox
> >>> └── Vagrantfile
> >>>
> >>>
> >>> In a second stage, the final scenario would be, only one folder with
> >>> one "smart" Vagrantfile for supporting Mesos and DC/OS for Linux and
> >>> OSX, the tool will be able of selecting the suitable virtualization
> >>> technology by user configuration.
> >>>
> >>> This is the idea.
> >>>
> >>> [1] https://issues.apache.org/jira/browse/MYRIAD-257
> >>>
> >>> --
> >>> Javi Roman
> >>>
> >>> Twitter: @javiromanrh
> >>> GitHub: github.com/javiroman
> >>> Linkedin: es.linkedin.com/in/javiroman
> >>> Big Data Blog: dataintensive.info
> >>>
>
Re: Regarding local environment based on Vagrant
Posted by Javi Roman <jr...@gmail.com>.
I going to merge the PR [1] related with the Vagrant environment reworking.
Please feel free to comment whatever enhance or inconvenient you think
about the commit. I will merge he PR if there are no negative
comments/votes.
[1] https://github.com/apache/incubator-myriad/pull/107
--
Javi Roman
Twitter: @javiromanrh
GitHub: github.com/javiroman
Linkedin: es.linkedin.com/in/javiroman
Big Data Blog: dataintensive.info
On Mon, Feb 5, 2018 at 2:36 PM, Javi Roman <jr...@gmail.com> wrote:
> For sure, this is something to include in the vagrant testbed. Good point.
> --
> Javi Roman
>
> Twitter: @javiromanrh
> GitHub: github.com/javiroman
> Linkedin: es.linkedin.com/in/javiroman
> Big Data Blog: dataintensive.info
>
>
> On Mon, Feb 5, 2018 at 10:01 AM, Juan P <jp...@gmail.com> wrote:
>> Hello Javi, I think it's a great idea.
>>
>> Is it possible to prepare the automatic environment to perform automatic
>> end-to-end tests (acceptance) or with a minimum of configuration?
>>
>> In later versions, would it be possible to include all the known Yarn
>> frameworks and prepare some automatic use cases that serve as an example
>> for the community and regression / integration test?
>>
>> Thank you
>>
>> On 4 February 2018 at 10:09, Javi Roman <jr...@gmail.com> wrote:
>>
>>> I would like share with you the strategy that I'm following for a
>>> powerful local development environment for Myriad.
>>>
>>> The idea is to create an easy-to-use environment for new contributors,
>>> setting up a Mesos cluster, or DCOS cluster (with hadoop running which
>>> is a tedious task). I guess automate this task is a key point for
>>> accelerating the development and focusing in the Myriad hacking.
>>>
>>> In this first task [1] I am working on Vagrant libvirt support for a
>>> plain Mesos cluster. However the roadmap for this environment would
>>> be:
>>>
>>> 1. vagrant-libvirt based cluster for Mesos, provisioned with Ansible
>>> (already working in my environment).
>>> 2. vagrant-libvirt based cluster for DC/OS provisioned with Ansible:
>>> Easier environment for testing Myriad with Marathon, Mesos-DNS, VIPS,
>>> .... and so on.
>>> 3. vagrant-virtualbox based cluster (maintain the original environment
>>> used right now).
>>>
>>> The layout would be something like this:
>>>
>>> vagrant/
>>> ├── libvirt
>>> │ ├── dcos
>>> │ │ └── Vagrantfile
>>> │ └── mesos
>>> │ └── Vagrantfile
>>> └── virtualbox
>>> └── Vagrantfile
>>>
>>>
>>> In a second stage, the final scenario would be, only one folder with
>>> one "smart" Vagrantfile for supporting Mesos and DC/OS for Linux and
>>> OSX, the tool will be able of selecting the suitable virtualization
>>> technology by user configuration.
>>>
>>> This is the idea.
>>>
>>> [1] https://issues.apache.org/jira/browse/MYRIAD-257
>>>
>>> --
>>> Javi Roman
>>>
>>> Twitter: @javiromanrh
>>> GitHub: github.com/javiroman
>>> Linkedin: es.linkedin.com/in/javiroman
>>> Big Data Blog: dataintensive.info
>>>
Re: Regarding local environment based on Vagrant
Posted by Javi Roman <jr...@gmail.com>.
For sure, this is something to include in the vagrant testbed. Good point.
--
Javi Roman
Twitter: @javiromanrh
GitHub: github.com/javiroman
Linkedin: es.linkedin.com/in/javiroman
Big Data Blog: dataintensive.info
On Mon, Feb 5, 2018 at 10:01 AM, Juan P <jp...@gmail.com> wrote:
> Hello Javi, I think it's a great idea.
>
> Is it possible to prepare the automatic environment to perform automatic
> end-to-end tests (acceptance) or with a minimum of configuration?
>
> In later versions, would it be possible to include all the known Yarn
> frameworks and prepare some automatic use cases that serve as an example
> for the community and regression / integration test?
>
> Thank you
>
> On 4 February 2018 at 10:09, Javi Roman <jr...@gmail.com> wrote:
>
>> I would like share with you the strategy that I'm following for a
>> powerful local development environment for Myriad.
>>
>> The idea is to create an easy-to-use environment for new contributors,
>> setting up a Mesos cluster, or DCOS cluster (with hadoop running which
>> is a tedious task). I guess automate this task is a key point for
>> accelerating the development and focusing in the Myriad hacking.
>>
>> In this first task [1] I am working on Vagrant libvirt support for a
>> plain Mesos cluster. However the roadmap for this environment would
>> be:
>>
>> 1. vagrant-libvirt based cluster for Mesos, provisioned with Ansible
>> (already working in my environment).
>> 2. vagrant-libvirt based cluster for DC/OS provisioned with Ansible:
>> Easier environment for testing Myriad with Marathon, Mesos-DNS, VIPS,
>> .... and so on.
>> 3. vagrant-virtualbox based cluster (maintain the original environment
>> used right now).
>>
>> The layout would be something like this:
>>
>> vagrant/
>> ├── libvirt
>> │ ├── dcos
>> │ │ └── Vagrantfile
>> │ └── mesos
>> │ └── Vagrantfile
>> └── virtualbox
>> └── Vagrantfile
>>
>>
>> In a second stage, the final scenario would be, only one folder with
>> one "smart" Vagrantfile for supporting Mesos and DC/OS for Linux and
>> OSX, the tool will be able of selecting the suitable virtualization
>> technology by user configuration.
>>
>> This is the idea.
>>
>> [1] https://issues.apache.org/jira/browse/MYRIAD-257
>>
>> --
>> Javi Roman
>>
>> Twitter: @javiromanrh
>> GitHub: github.com/javiroman
>> Linkedin: es.linkedin.com/in/javiroman
>> Big Data Blog: dataintensive.info
>>
Re: Regarding local environment based on Vagrant
Posted by Juan P <jp...@gmail.com>.
Hello Javi, I think it's a great idea.
Is it possible to prepare the automatic environment to perform automatic
end-to-end tests (acceptance) or with a minimum of configuration?
In later versions, would it be possible to include all the known Yarn
frameworks and prepare some automatic use cases that serve as an example
for the community and regression / integration test?
Thank you
On 4 February 2018 at 10:09, Javi Roman <jr...@gmail.com> wrote:
> I would like share with you the strategy that I'm following for a
> powerful local development environment for Myriad.
>
> The idea is to create an easy-to-use environment for new contributors,
> setting up a Mesos cluster, or DCOS cluster (with hadoop running which
> is a tedious task). I guess automate this task is a key point for
> accelerating the development and focusing in the Myriad hacking.
>
> In this first task [1] I am working on Vagrant libvirt support for a
> plain Mesos cluster. However the roadmap for this environment would
> be:
>
> 1. vagrant-libvirt based cluster for Mesos, provisioned with Ansible
> (already working in my environment).
> 2. vagrant-libvirt based cluster for DC/OS provisioned with Ansible:
> Easier environment for testing Myriad with Marathon, Mesos-DNS, VIPS,
> .... and so on.
> 3. vagrant-virtualbox based cluster (maintain the original environment
> used right now).
>
> The layout would be something like this:
>
> vagrant/
> ├── libvirt
> │ ├── dcos
> │ │ └── Vagrantfile
> │ └── mesos
> │ └── Vagrantfile
> └── virtualbox
> └── Vagrantfile
>
>
> In a second stage, the final scenario would be, only one folder with
> one "smart" Vagrantfile for supporting Mesos and DC/OS for Linux and
> OSX, the tool will be able of selecting the suitable virtualization
> technology by user configuration.
>
> This is the idea.
>
> [1] https://issues.apache.org/jira/browse/MYRIAD-257
>
> --
> Javi Roman
>
> Twitter: @javiromanrh
> GitHub: github.com/javiroman
> Linkedin: es.linkedin.com/in/javiroman
> Big Data Blog: dataintensive.info
>