You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by moshe jarusalem <tu...@gmail.com> on 2017/05/13 10:50:26 UTC
integration testing framework
Hi All,
I have been looking for an easy way to test metron in the development
environment. Using "full-dev-environment" is a bit painful because you each
time need to copy artifacts etc. I tried to understand integration testing
framework but it a little complex for a newbie.
Would you guys describe how to utilize it? broadly how to quickly write and
test? How do you manage?
Regards,
Re: integration testing framework
Posted by moshe jarusalem <tu...@gmail.com>.
I have solved the issue by following
https://github.com/docker/compose/issues/2431
On Wed, May 17, 2017 at 4:43 PM, moshe jarusalem <tu...@gmail.com> wrote:
> Hi Otto,
> I am getting the following error during setting up metron-docker . my
> machine's ip address is "192.168.17.151". I have changed the IP address in
> ".docker/machine/machines/metron-machine/config.json" but did not help.
>
> docker-compose up -d
> Traceback (most recent call last):
> File "/usr/bin/docker-compose", line 9, in <module>
> load_entry_point('docker-compose==1.8.0', 'console_scripts',
> 'docker-compose')()
> File "/usr/lib/python2.7/dist-packages/compose/cli/main.py", line 61,
> in main
> command()
> File "/usr/lib/python2.7/dist-packages/compose/cli/main.py", line 113,
> in perform_command
> handler(command, command_options)
> File "/usr/lib/python2.7/dist-packages/compose/cli/main.py", line 835,
> in up
> remove_orphans=remove_orphans)
> File "/usr/lib/python2.7/dist-packages/compose/project.py", line 372,
> in up
> warn_for_swarm_mode(self.client)
> File "/usr/lib/python2.7/dist-packages/compose/project.py", line 539,
> in warn_for_swarm_mode
> info = client.info()
> File "/usr/lib/python2.7/dist-packages/docker/api/daemon.py", line 33,
> in info
> return self._result(self._get(self._url("/info")), True)
> File "/usr/lib/python2.7/dist-packages/docker/utils/decorators.py",
> line 47, in inner
> return f(self, *args, **kwargs)
> File "/usr/lib/python2.7/dist-packages/docker/client.py", line 140, in
> _get
> return self.get(url, **self._set_request_timeout(kwargs))
> File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 480,
> in get
> return self.request('GET', url, **kwargs)
> File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 468,
> in request
> resp = self.send(prep, **send_kwargs)
> File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 576,
> in send
> r = adapter.send(request, **kwargs)
> File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 376,
> in send
> timeout=timeout
> File "/usr/lib/python2.7/dist-packages/urllib3/connectionpool.py", line
> 560, in urlopen
> body=body, headers=headers)
> File "/usr/lib/python2.7/dist-packages/urllib3/connectionpool.py", line
> 346, in _make_request
> self._validate_conn(conn)
> File "/usr/lib/python2.7/dist-packages/urllib3/connectionpool.py", line
> 787, in _validate_conn
> conn.connect()
> File "/usr/lib/python2.7/dist-packages/urllib3/connection.py", line
> 277, in connect
> match_hostname(cert, asserted_hostname)
> File "/usr/lib/python2.7/dist-packages/backports/ssl_match_hostname/__init__.py",
> line 99, in match_hostname
> % (hostname, dnsnames[0]))
> backports.ssl_match_hostname.CertificateError: hostname '192.168.99.100'
> doesn't match 'localhost'
>
>
> On Mon, May 15, 2017 at 3:34 PM, Otto Fowler <ot...@gmail.com>
> wrote:
>
>> Hi Moshe,
>>
>> I don’t know of anyone who installs metron on their dev machine, there
>> are a couple of ways to run it on a dev machine
>> that I would not call ‘installing’
>>
>> 1. The vagrant environments, full or quick dev
>> 2. metron-docker
>>
>> Both have readme that you can look at, after you do you can ask some
>> specific questions.
>>
>> The integration testing does not require metron to be installed to run it.
>> You only need to be able to build Metron to run the integration tests.
>>
>> The integration tests attempt to run a faux metron system in memory to
>> allow us
>> to test end to end without an install. Most of our platform components
>> have an
>> in-memory testing component that can be used for such purposes.
>>
>> The tests therefore are concerned at first with setting up the
>> environment and services such as:
>>
>> Storm, Zookeeper, Kafka, HDFS, YARN
>>
>> And configuring them ( pushing things that should be in zookeeper, or
>> hdfs, creating the topologies etc. )
>>
>> Many of the tests then use something called the ComponentRunner to run
>> these services and shut them down in order.
>> The component service can then run tests to execute testing code.
>>
>> By examining these tests you can follow through the requirements and
>> logic around the tests.
>> Then, you can create your own tests, but copying and modifying tests for
>> your own use.
>>
>> The metron-platform/metron-integration-testing project has the
>> primitives of the test components and is used
>> in pretty much all the integration tests ( ComponentRunner etc ).
>>
>> I don’t know what you are interested in hacking on, but you may want to
>> look at
>> metron-platform/metron-enrichment
>> metron-platform/metron-parsers
>>
>> to start.
>>
>> Now, if you want to test beyond what you can do in integration tests, you
>> should look at metron-docker and vagrant.
>>
>>
>>
>>
>>
>> On May 15, 2017 at 08:07:52, moshe jarusalem (tuutdo@gmail.com) wrote:
>>
>> I would like to ask another question related to this topic.
>> If I am going to install metron on a single machine (the same machine as
>> development) which operating distribution would be best centos7 or ubuntu
>> 16.4 or another?
>> And is there any document to follow for such an installation ?
>>
>> Regards,
>>
>> On Sat, May 13, 2017 at 1:50 PM, moshe jarusalem <tu...@gmail.com>
>> wrote:
>>
>> > Hi All,
>> > I have been looking for an easy way to test metron in the development
>> > environment. Using "full-dev-environment" is a bit painful because you
>> each
>> > time need to copy artifacts etc. I tried to understand integration
>> testing
>> > framework but it a little complex for a newbie.
>> >
>> > Would you guys describe how to utilize it? broadly how to quickly write
>> > and test? How do you manage?
>> >
>> > Regards,
>> >
>>
>>
>
Re: integration testing framework
Posted by moshe jarusalem <tu...@gmail.com>.
Hi Otto,
I am getting the following error during setting up metron-docker . my
machine's ip address is "192.168.17.151". I have changed the IP address in
".docker/machine/machines/metron-machine/config.json" but did not help.
docker-compose up -d
Traceback (most recent call last):
File "/usr/bin/docker-compose", line 9, in <module>
load_entry_point('docker-compose==1.8.0', 'console_scripts',
'docker-compose')()
File "/usr/lib/python2.7/dist-packages/compose/cli/main.py", line 61, in
main
command()
File "/usr/lib/python2.7/dist-packages/compose/cli/main.py", line 113, in
perform_command
handler(command, command_options)
File "/usr/lib/python2.7/dist-packages/compose/cli/main.py", line 835, in
up
remove_orphans=remove_orphans)
File "/usr/lib/python2.7/dist-packages/compose/project.py", line 372, in
up
warn_for_swarm_mode(self.client)
File "/usr/lib/python2.7/dist-packages/compose/project.py", line 539, in
warn_for_swarm_mode
info = client.info()
File "/usr/lib/python2.7/dist-packages/docker/api/daemon.py", line 33, in
info
return self._result(self._get(self._url("/info")), True)
File "/usr/lib/python2.7/dist-packages/docker/utils/decorators.py", line
47, in inner
return f(self, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/docker/client.py", line 140, in
_get
return self.get(url, **self._set_request_timeout(kwargs))
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 480,
in get
return self.request('GET', url, **kwargs)
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 468,
in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 576,
in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 376,
in send
timeout=timeout
File "/usr/lib/python2.7/dist-packages/urllib3/connectionpool.py", line
560, in urlopen
body=body, headers=headers)
File "/usr/lib/python2.7/dist-packages/urllib3/connectionpool.py", line
346, in _make_request
self._validate_conn(conn)
File "/usr/lib/python2.7/dist-packages/urllib3/connectionpool.py", line
787, in _validate_conn
conn.connect()
File "/usr/lib/python2.7/dist-packages/urllib3/connection.py", line 277,
in connect
match_hostname(cert, asserted_hostname)
File
"/usr/lib/python2.7/dist-packages/backports/ssl_match_hostname/__init__.py",
line 99, in match_hostname
% (hostname, dnsnames[0]))
backports.ssl_match_hostname.CertificateError: hostname '192.168.99.100'
doesn't match 'localhost'
On Mon, May 15, 2017 at 3:34 PM, Otto Fowler <ot...@gmail.com>
wrote:
> Hi Moshe,
>
> I don’t know of anyone who installs metron on their dev machine, there are
> a couple of ways to run it on a dev machine
> that I would not call ‘installing’
>
> 1. The vagrant environments, full or quick dev
> 2. metron-docker
>
> Both have readme that you can look at, after you do you can ask some
> specific questions.
>
> The integration testing does not require metron to be installed to run it.
> You only need to be able to build Metron to run the integration tests.
>
> The integration tests attempt to run a faux metron system in memory to
> allow us
> to test end to end without an install. Most of our platform components
> have an
> in-memory testing component that can be used for such purposes.
>
> The tests therefore are concerned at first with setting up the environment
> and services such as:
>
> Storm, Zookeeper, Kafka, HDFS, YARN
>
> And configuring them ( pushing things that should be in zookeeper, or
> hdfs, creating the topologies etc. )
>
> Many of the tests then use something called the ComponentRunner to run
> these services and shut them down in order.
> The component service can then run tests to execute testing code.
>
> By examining these tests you can follow through the requirements and logic
> around the tests.
> Then, you can create your own tests, but copying and modifying tests for
> your own use.
>
> The metron-platform/metron-integration-testing project has the
> primitives of the test components and is used
> in pretty much all the integration tests ( ComponentRunner etc ).
>
> I don’t know what you are interested in hacking on, but you may want to
> look at
> metron-platform/metron-enrichment
> metron-platform/metron-parsers
>
> to start.
>
> Now, if you want to test beyond what you can do in integration tests, you
> should look at metron-docker and vagrant.
>
>
>
>
>
> On May 15, 2017 at 08:07:52, moshe jarusalem (tuutdo@gmail.com) wrote:
>
> I would like to ask another question related to this topic.
> If I am going to install metron on a single machine (the same machine as
> development) which operating distribution would be best centos7 or ubuntu
> 16.4 or another?
> And is there any document to follow for such an installation ?
>
> Regards,
>
> On Sat, May 13, 2017 at 1:50 PM, moshe jarusalem <tu...@gmail.com>
> wrote:
>
> > Hi All,
> > I have been looking for an easy way to test metron in the development
> > environment. Using "full-dev-environment" is a bit painful because you
> each
> > time need to copy artifacts etc. I tried to understand integration
> testing
> > framework but it a little complex for a newbie.
> >
> > Would you guys describe how to utilize it? broadly how to quickly write
> > and test? How do you manage?
> >
> > Regards,
> >
>
>
Re: integration testing framework
Posted by Otto Fowler <ot...@gmail.com>.
Hi Moshe,
I don’t know of anyone who installs metron on their dev machine, there are a couple of ways to run it on a dev machine
that I would not call ‘installing’
1. The vagrant environments, full or quick dev
2. metron-docker
Both have readme that you can look at, after you do you can ask some specific questions.
The integration testing does not require metron to be installed to run it.
You only need to be able to build Metron to run the integration tests.
The integration tests attempt to run a faux metron system in memory to allow us
to test end to end without an install. Most of our platform components have an
in-memory testing component that can be used for such purposes.
The tests therefore are concerned at first with setting up the environment and services such as:
Storm, Zookeeper, Kafka, HDFS, YARN
And configuring them ( pushing things that should be in zookeeper, or hdfs, creating the topologies etc. )
Many of the tests then use something called the ComponentRunner to run these services and shut them down in order.
The component service can then run tests to execute testing code.
By examining these tests you can follow through the requirements and logic around the tests.
Then, you can create your own tests, but copying and modifying tests for your own use.
The metron-platform/metron-integration-testing project has the primitives of the test components and is used
in pretty much all the integration tests ( ComponentRunner etc ).
I don’t know what you are interested in hacking on, but you may want to look at
metron-platform/metron-enrichment
metron-platform/metron-parsers
to start.
Now, if you want to test beyond what you can do in integration tests, you should look at metron-docker and vagrant.
On May 15, 2017 at 08:07:52, moshe jarusalem (tuutdo@gmail.com) wrote:
I would like to ask another question related to this topic.
If I am going to install metron on a single machine (the same machine as
development) which operating distribution would be best centos7 or ubuntu
16.4 or another?
And is there any document to follow for such an installation ?
Regards,
On Sat, May 13, 2017 at 1:50 PM, moshe jarusalem <tu...@gmail.com> wrote:
> Hi All,
> I have been looking for an easy way to test metron in the development
> environment. Using "full-dev-environment" is a bit painful because you each
> time need to copy artifacts etc. I tried to understand integration testing
> framework but it a little complex for a newbie.
>
> Would you guys describe how to utilize it? broadly how to quickly write
> and test? How do you manage?
>
> Regards,
>
Re: integration testing framework
Posted by "Zeolla@GMail.com" <ze...@gmail.com>.
The standard has been centos6 for installing Metron up to this point.
There are some Ubuntu guides floating around as well.
Jon
On Mon, May 15, 2017, 8:07 AM moshe jarusalem <tu...@gmail.com> wrote:
> I would like to ask another question related to this topic.
> If I am going to install metron on a single machine (the same machine as
> development) which operating distribution would be best centos7 or ubuntu
> 16.4 or another?
> And is there any document to follow for such an installation ?
>
> Regards,
>
> On Sat, May 13, 2017 at 1:50 PM, moshe jarusalem <tu...@gmail.com> wrote:
>
> > Hi All,
> > I have been looking for an easy way to test metron in the development
> > environment. Using "full-dev-environment" is a bit painful because you
> each
> > time need to copy artifacts etc. I tried to understand integration
> testing
> > framework but it a little complex for a newbie.
> >
> > Would you guys describe how to utilize it? broadly how to quickly write
> > and test? How do you manage?
> >
> > Regards,
> >
>
--
Jon
Re: integration testing framework
Posted by moshe jarusalem <tu...@gmail.com>.
I would like to ask another question related to this topic.
If I am going to install metron on a single machine (the same machine as
development) which operating distribution would be best centos7 or ubuntu
16.4 or another?
And is there any document to follow for such an installation ?
Regards,
On Sat, May 13, 2017 at 1:50 PM, moshe jarusalem <tu...@gmail.com> wrote:
> Hi All,
> I have been looking for an easy way to test metron in the development
> environment. Using "full-dev-environment" is a bit painful because you each
> time need to copy artifacts etc. I tried to understand integration testing
> framework but it a little complex for a newbie.
>
> Would you guys describe how to utilize it? broadly how to quickly write
> and test? How do you manage?
>
> Regards,
>
Re: integration testing framework
Posted by moshe jarusalem <tu...@gmail.com>.
Would one of you please promptly answer this?
On Sat, May 13, 2017 at 1:50 PM, moshe jarusalem <tu...@gmail.com> wrote:
> Hi All,
> I have been looking for an easy way to test metron in the development
> environment. Using "full-dev-environment" is a bit painful because you each
> time need to copy artifacts etc. I tried to understand integration testing
> framework but it a little complex for a newbie.
>
> Would you guys describe how to utilize it? broadly how to quickly write
> and test? How do you manage?
>
> Regards,
>