You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@metron.apache.org by moshe jarusalem <tu...@gmail.com> on 2017/03/05 09:23:17 UTC
a proper entry level documentation
Hi All,
I have been trying to get involved with the metron project for a while. I
have tried release 0.3.0 and 0.3.1. Unfortunately, it is very difficult to
set up a running environment and understand the picture how it works. There
are many technologies prerequisite such as Hadoop, nifi, storm, Kafka etc. I
assume most of you guys are expert at these technologies so you solve your
problems.
I have also browsed the user's list and there are many people struggling to
set up a proper environment.
Would you guys make a document describing step by step setting up a
development environment with a specific release (0.3.1 as its the latest) ?
Can you describe each technology shortly and give references and most
importantly metron's usage of them?
Indeed a video recording or a webinar would be much more useful so that
there can be some discussions.
Regards,
Moshe
Re: a proper entry level documentation
Posted by Umesh Kaushik <um...@bhujang.net>.
Thanks for the word man. I have added in the thread.
On 8 March 2017 at 14:54, Khurram Ahmed <kh...@gmail.com> wrote:
> Umesh Ubuntu docs would be helpful
>
>
> On Mar 8, 2017 2:16 PM, "Umesh Kaushik" <um...@bhujang.net> wrote:
>
>> I tried installing on ubuntu, and I documented all the steps and issues I
>> faced, I would love to attach it in the email thread, please let me know if
>> that suits you all, but that is basic installation. I did not document
>> configuring sqid and other addons. Please let me know if it is needed.
>>
>> On 6 March 2017 at 21:56, Earl Hinkle <ea...@thehinkles.org> wrote:
>>
>>> I tried on centos 6 and had problems getting python 2.7 installed and
>>> some of the other prerequisite. So i triend centos 7, got all the packages
>>> installed (python, ansible,etc) and got further into the build but it still
>>> failed.
>>>
>>> I just saw the previous email about a centos 6 script so will try that
>>> this week and see how it goes.
>>>
>>> Sent with AquaMail for Android
>>> http://www.aqua-mail.com
>>>
>>> On March 5, 2017 3:10:08 PM "Zeolla@GMail.com" <ze...@gmail.com> wrote:
>>>
>>>> What OS are you trying with now? Like I mentioned, the scripts I put
>>>> together before will do Mac and centos 6 but it should be possible to get
>>>> working on more than that. Your choice, but regardless I plan to work
>>>> further on documentation. If there's anything specific you'd like
>>>> addressed or shown please let me know.
>>>>
>>>> Jon
>>>>
>>>> On Sun, Mar 5, 2017, 4:31 PM Earl Hinkle <ea...@thehinkles.org> wrote:
>>>>
>>>>> I would be willing to help using centos if i could get it up and
>>>>> working.
>>>>>
>>>>> Sent with AquaMail for Android
>>>>> http://www.aqua-mail.com
>>>>>
>>>>> On March 5, 2017 2:23:32 AM moshe jarusalem <tu...@gmail.com> wrote:
>>>>>
>>>>> Hi All,
>>>>> I have been trying to get involved with the metron project for a
>>>>> while. I have tried release 0.3.0 and 0.3.1. Unfortunately, it is very
>>>>> difficult to set up a running environment and understand the picture how it
>>>>> works. There are many technologies prerequisite such as Hadoop, nifi,
>>>>> storm, Kafka etc. I assume most of you guys are expert at these
>>>>> technologies so you solve your problems.
>>>>> I have also browsed the user's list and there are many people
>>>>> struggling to set up a proper environment.
>>>>>
>>>>> Would you guys make a document describing step by step setting up a
>>>>> development environment with a specific release (0.3.1 as its the latest) ?
>>>>> Can you describe each technology shortly and give references and most
>>>>> importantly metron's usage of them?
>>>>> Indeed a video recording or a webinar would be much more useful so
>>>>> that there can be some discussions.
>>>>>
>>>>> Regards,
>>>>> Moshe
>>>>>
>>>>> --
>>>>
>>>> Jon
>>>>
>>>> Sent from my mobile device
>>>>
>>>
>>
>>
>> --
>> Cheers,
>> Umesh Kaushik
>> (9620023458)
>>
>
--
Cheers,
Umesh Kaushik
(9620023458)
Re: a proper entry level documentation
Posted by Khurram Ahmed <kh...@gmail.com>.
Umesh Ubuntu docs would be helpful
On Mar 8, 2017 2:16 PM, "Umesh Kaushik" <um...@bhujang.net> wrote:
> I tried installing on ubuntu, and I documented all the steps and issues I
> faced, I would love to attach it in the email thread, please let me know if
> that suits you all, but that is basic installation. I did not document
> configuring sqid and other addons. Please let me know if it is needed.
>
> On 6 March 2017 at 21:56, Earl Hinkle <ea...@thehinkles.org> wrote:
>
>> I tried on centos 6 and had problems getting python 2.7 installed and
>> some of the other prerequisite. So i triend centos 7, got all the packages
>> installed (python, ansible,etc) and got further into the build but it still
>> failed.
>>
>> I just saw the previous email about a centos 6 script so will try that
>> this week and see how it goes.
>>
>> Sent with AquaMail for Android
>> http://www.aqua-mail.com
>>
>> On March 5, 2017 3:10:08 PM "Zeolla@GMail.com" <ze...@gmail.com> wrote:
>>
>>> What OS are you trying with now? Like I mentioned, the scripts I put
>>> together before will do Mac and centos 6 but it should be possible to get
>>> working on more than that. Your choice, but regardless I plan to work
>>> further on documentation. If there's anything specific you'd like
>>> addressed or shown please let me know.
>>>
>>> Jon
>>>
>>> On Sun, Mar 5, 2017, 4:31 PM Earl Hinkle <ea...@thehinkles.org> wrote:
>>>
>>>> I would be willing to help using centos if i could get it up and
>>>> working.
>>>>
>>>> Sent with AquaMail for Android
>>>> http://www.aqua-mail.com
>>>>
>>>> On March 5, 2017 2:23:32 AM moshe jarusalem <tu...@gmail.com> wrote:
>>>>
>>>> Hi All,
>>>> I have been trying to get involved with the metron project for a while.
>>>> I have tried release 0.3.0 and 0.3.1. Unfortunately, it is very difficult
>>>> to set up a running environment and understand the picture how it works.
>>>> There are many technologies prerequisite such as Hadoop, nifi, storm, Kafka
>>>> etc. I assume most of you guys are expert at these technologies so you
>>>> solve your problems.
>>>> I have also browsed the user's list and there are many people
>>>> struggling to set up a proper environment.
>>>>
>>>> Would you guys make a document describing step by step setting up a
>>>> development environment with a specific release (0.3.1 as its the latest) ?
>>>> Can you describe each technology shortly and give references and most
>>>> importantly metron's usage of them?
>>>> Indeed a video recording or a webinar would be much more useful so that
>>>> there can be some discussions.
>>>>
>>>> Regards,
>>>> Moshe
>>>>
>>>> --
>>>
>>> Jon
>>>
>>> Sent from my mobile device
>>>
>>
>
>
> --
> Cheers,
> Umesh Kaushik
> (9620023458)
>
Re: a proper entry level documentation
Posted by Khurram Ahmed <kh...@gmail.com>.
Thanks
On Mar 8, 2017 3:10 PM, "Umesh Kaushik" <um...@bhujang.net> wrote:
> Hi Khurram Ahmed,
>
> Pardon me for the vague answer, however I just created a random Vm, with
> 100 gb hdd and 20 gb ram, I have a server with 128 gb of ram and 1tb hdd.
> So, I just chose a random config.
>
> On 8 March 2017 at 15:13, Khurram Ahmed <kh...@gmail.com> wrote:
>
>> Excellent could you mention the specs that allowed you to run the
>> platform without a hitch? Mention RAM, CPU and hard drive space plz.
>>
>> On Mar 8, 2017 2:38 PM, "Umesh Kaushik" <um...@bhujang.net>
>> wrote:
>>
>>> Hi Guys,
>>>
>>> This is the bare minimum installation document I prepared while
>>> installing metron on ubuntu. Please pardon if there are typos or
>>> unprofessionalism in the document. Hope this helps!
>>>
>>> On 8 March 2017 at 14:19, Umesh Kaushik <um...@bhujang.net>
>>> wrote:
>>>
>>>> I tried installing on ubuntu, and I documented all the steps and issues
>>>> I faced, I would love to attach it in the email thread, please let me know
>>>> if that suits you all, but that is basic installation. I did not document
>>>> configuring sqid and other addons. Please let me know if it is needed.
>>>>
>>>> On 6 March 2017 at 21:56, Earl Hinkle <ea...@thehinkles.org> wrote:
>>>>
>>>>> I tried on centos 6 and had problems getting python 2.7 installed and
>>>>> some of the other prerequisite. So i triend centos 7, got all the packages
>>>>> installed (python, ansible,etc) and got further into the build but it still
>>>>> failed.
>>>>>
>>>>> I just saw the previous email about a centos 6 script so will try that
>>>>> this week and see how it goes.
>>>>>
>>>>> Sent with AquaMail for Android
>>>>> http://www.aqua-mail.com
>>>>>
>>>>> On March 5, 2017 3:10:08 PM "Zeolla@GMail.com" <ze...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> What OS are you trying with now? Like I mentioned, the scripts I put
>>>>>> together before will do Mac and centos 6 but it should be possible to get
>>>>>> working on more than that. Your choice, but regardless I plan to work
>>>>>> further on documentation. If there's anything specific you'd like
>>>>>> addressed or shown please let me know.
>>>>>>
>>>>>> Jon
>>>>>>
>>>>>> On Sun, Mar 5, 2017, 4:31 PM Earl Hinkle <ea...@thehinkles.org> wrote:
>>>>>>
>>>>>>> I would be willing to help using centos if i could get it up and
>>>>>>> working.
>>>>>>>
>>>>>>> Sent with AquaMail for Android
>>>>>>> http://www.aqua-mail.com
>>>>>>>
>>>>>>> On March 5, 2017 2:23:32 AM moshe jarusalem <tu...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Hi All,
>>>>>>> I have been trying to get involved with the metron project for a
>>>>>>> while. I have tried release 0.3.0 and 0.3.1. Unfortunately, it is very
>>>>>>> difficult to set up a running environment and understand the picture how it
>>>>>>> works. There are many technologies prerequisite such as Hadoop, nifi,
>>>>>>> storm, Kafka etc. I assume most of you guys are expert at these
>>>>>>> technologies so you solve your problems.
>>>>>>> I have also browsed the user's list and there are many people
>>>>>>> struggling to set up a proper environment.
>>>>>>>
>>>>>>> Would you guys make a document describing step by step setting up a
>>>>>>> development environment with a specific release (0.3.1 as its the latest) ?
>>>>>>> Can you describe each technology shortly and give references and most
>>>>>>> importantly metron's usage of them?
>>>>>>> Indeed a video recording or a webinar would be much more useful so
>>>>>>> that there can be some discussions.
>>>>>>>
>>>>>>> Regards,
>>>>>>> Moshe
>>>>>>>
>>>>>>> --
>>>>>>
>>>>>> Jon
>>>>>>
>>>>>> Sent from my mobile device
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Cheers,
>>>> Umesh Kaushik
>>>> (9620023458)
>>>>
>>>
>>>
>>>
>>> --
>>> Cheers,
>>> Umesh Kaushik
>>> (9620023458)
>>>
>>
>
>
> --
> Cheers,
> Umesh Kaushik
> (9620023458)
>
Re: a proper entry level documentation
Posted by Michael Miklavcic <mi...@gmail.com>.
Hi Moshe,
The error you're seeing should not affect your topologies. If you'd like to
eliminate that error message, build Metron with the HDP profile.
mvn clean install -PHDP-2.5.0.0
The reason you are seeing this is due to a feature made available in the
HDP cluster you're running on that is not part of the same version in the
OSS version.
Best,
Michael Miklavcic
On Sun, Mar 12, 2017 at 12:46 AM, moshe jarusalem <tu...@gmail.com> wrote:
> Thanks for all your efforts.
> I have followed https://cwiki.apache.org/confluence/display/METRON/
> 2016/04/28/Metron+Tutorial+-+Fundamentals+Part+2%3A+
> Creating+a+New+Enrichment
> Unfortuanatly it is not working as expected. I get one error from kafka
> "Offset lags for kafka not supported for older versions. Please update
> kafka spout to latest version." I checked storm kafka version but it seems
> our storm version already contains the fix.
>
> Yet another question is elastic search port is defined to be 9300 in
> configurations but actually it runs only on 9200.
> http://stackoverflow.com/questions/24624980/port-9300-on-elasticsearch
>
> Would you please also tell how to enable debug logs for topologies so that
> we can flow message flow and troubleshoot ?
>
> Cheers,
>
>
> On Fri, Mar 10, 2017 at 9:22 PM, Umesh Kaushik <um...@bhujang.net>
> wrote:
>
>> Thanks Matt! Kudos!
>>
>> Cheers,
>> Umesh Kaushik
>> (Full stack developer- cyber security analyst: Bhujang innovations)
>> (9620023458)
>> Sent from mobile device, kindly excuse any typographic error.
>>
>> On Mar 10, 2017 11:49 PM, "Matt Foley" <ma...@apache.org> wrote:
>>
>>> Thanks for this contribution, Umesh.
>>>
>>> I’ve edited it lightly, and posted it to the wiki at
>>>
>>> https://cwiki.apache.org/confluence/pages/viewpage.actio
>>> n?pageId=68718548
>>>
>>> along with the other installation-related help files.
>>>
>>>
>>>
>>> Regards,
>>>
>>> --Matt
>>>
>>>
>>>
>>> *From: *Umesh Kaushik <um...@bhujang.net>
>>> *Date: *Wednesday, March 8, 2017 at 5:18 PM
>>> *To: *Matt Foley <ma...@apache.org>
>>> *Cc: *"user@metron.incubator.apache.org" <user@metron.incubator.apache.
>>> org>
>>> *Subject: *Re: a proper entry level documentation
>>>
>>>
>>>
>>> Hi Matt,
>>>
>>> Yes I have no issues in that, in fact I would be happy if this document
>>> could be even a little bit of useful.
>>>
>>> Yes, it was default master version of metron at the time I installed. It
>>> was in the beginning of January for poc and yes ambari was also default.
>>> Please let me know if I could assist you any further.
>>>
>>> Cheers,
>>> Umesh Kaushik
>>> (9620023458)
>>> Sent from mobile device, kindly excuse any typographic error.
>>>
>>>
>>>
>>> On Mar 9, 2017 12:57 AM, "Matt Foley" <ma...@apache.org> wrote:
>>>
>>> Umesh, thanks very much for writing this up. I’d like to add it to the
>>> Metron wiki. Is that okay with you?
>>>
>>>
>>>
>>> Also, what version of Metron was used? Your instructions imply you
>>> built from “master” branch of github site, which means you would be on the
>>> newest post-0.3.1 code (call it 0.3.2-SNAPSHOT). Is that correct?
>>>
>>>
>>>
>>> And what version of Ambari did you use? I think the packaging defaults
>>> to version 2.4.0.0, is that what was used?
>>>
>>>
>>>
>>> Thanks,
>>>
>>> --Matt
>>>
>>>
>>>
>>> *From: *Umesh Kaushik <um...@bhujang.net>
>>> *Reply-To: *"user@metron.incubator.apache.org" <
>>> user@metron.incubator.apache.org>
>>> *Date: *Wednesday, March 8, 2017 at 2:04 AM
>>> *To: *"user@metron.incubator.apache.org" <user@metron.incubator.apache.
>>> org>
>>> *Subject: *Re: a proper entry level documentation
>>>
>>>
>>>
>>> and allotted 2 cores of processor.
>>>
>>>
>>>
>>> On 8 March 2017 at 15:32, Umesh Kaushik <um...@bhujang.net>
>>> wrote:
>>>
>>> Hi Khurram Ahmed,
>>>
>>>
>>>
>>> Pardon me for the vague answer, however I just created a random Vm, with
>>> 100 gb hdd and 20 gb ram, I have a server with 128 gb of ram and 1tb hdd.
>>> So, I just chose a random config.
>>>
>>>
>>>
>>> On 8 March 2017 at 15:13, Khurram Ahmed <kh...@gmail.com> wrote:
>>>
>>> Excellent could you mention the specs that allowed you to run the
>>> platform without a hitch? Mention RAM, CPU and hard drive space plz.
>>>
>>>
>>>
>>> On Mar 8, 2017 2:38 PM, "Umesh Kaushik" <um...@bhujang.net>
>>> wrote:
>>>
>>> Hi Guys,
>>>
>>>
>>>
>>> This is the bare minimum installation document I prepared while
>>> installing metron on ubuntu. Please pardon if there are typos or
>>> unprofessionalism in the document. Hope this helps!
>>>
>>>
>>>
>>> On 8 March 2017 at 14:19, Umesh Kaushik <um...@bhujang.net>
>>> wrote:
>>>
>>> I tried installing on ubuntu, and I documented all the steps and issues
>>> I faced, I would love to attach it in the email thread, please let me know
>>> if that suits you all, but that is basic installation. I did not document
>>> configuring sqid and other addons. Please let me know if it is needed.
>>>
>>>
>>>
>>> On 6 March 2017 at 21:56, Earl Hinkle <ea...@thehinkles.org> wrote:
>>>
>>> I tried on centos 6 and had problems getting python 2.7 installed and
>>> some of the other prerequisite. So i triend centos 7, got all the packages
>>> installed (python, ansible,etc) and got further into the build but it still
>>> failed.
>>>
>>> I just saw the previous email about a centos 6 script so will try that
>>> this week and see how it goes.
>>>
>>> Sent with AquaMail for Android
>>> http://www.aqua-mail.com
>>>
>>> On March 5, 2017 3:10:08 PM "Zeolla@GMail.com" <ze...@gmail.com> wrote:
>>>
>>> What OS are you trying with now? Like I mentioned, the scripts I put
>>> together before will do Mac and centos 6 but it should be possible to get
>>> working on more than that. Your choice, but regardless I plan to work
>>> further on documentation. If there's anything specific you'd like
>>> addressed or shown please let me know.
>>>
>>> Jon
>>>
>>>
>>>
>>> On Sun, Mar 5, 2017, 4:31 PM Earl Hinkle <ea...@thehinkles.org> wrote:
>>>
>>> I would be willing to help using centos if i could get it up and working.
>>>
>>> Sent with AquaMail for Android
>>> http://www.aqua-mail.com
>>>
>>> On March 5, 2017 2:23:32 AM moshe jarusalem <tu...@gmail.com> wrote:
>>>
>>> Hi All,
>>>
>>> I have been trying to get involved with the metron project for a while.
>>> I have tried release 0.3.0 and 0.3.1. Unfortunately, it is very difficult
>>> to set up a running environment and understand the picture how it works.
>>> There are many technologies prerequisite such as Hadoop, nifi, storm, Kafka
>>> etc. I assume most of you guys are expert at these technologies so you
>>> solve your problems.
>>>
>>> I have also browsed the user's list and there are many people struggling
>>> to set up a proper environment.
>>>
>>>
>>>
>>> Would you guys make a document describing step by step setting up a
>>> development environment with a specific release (0.3.1 as its the latest) ?
>>> Can you describe each technology shortly and give references and most
>>> importantly metron's usage of them?
>>>
>>> Indeed a video recording or a webinar would be much more useful so that
>>> there can be some discussions.
>>>
>>>
>>>
>>> Regards,
>>>
>>> Moshe
>>>
>>> --
>>>
>>> Jon
>>>
>>> Sent from my mobile device
>>>
>>>
>>>
>>>
>>>
>>> --
>>>
>>> Cheers,
>>>
>>> Umesh Kaushik
>>>
>>> (9620023458)
>>>
>>>
>>>
>>>
>>>
>>> --
>>>
>>> Cheers,
>>>
>>> Umesh Kaushik
>>>
>>> (9620023458)
>>>
>>>
>>>
>>>
>>>
>>> --
>>>
>>> Cheers,
>>>
>>> Umesh Kaushik
>>>
>>> (9620023458)
>>>
>>>
>>>
>>>
>>>
>>> --
>>>
>>> Cheers,
>>>
>>> Umesh Kaushik
>>>
>>> (9620023458)
>>>
>>>
>
Re: a proper entry level documentation
Posted by Anand Subramanian <as...@hortonworks.com>.
Hi Moshe,
I have seen that the offset lag error goes away once you push a few events into your topic. See if that works for you.
For enabling debug logs on the topologies, check out this article:
https://community.hortonworks.com/articles/36151/debugging-an-apache-storm-topology.html
Regards,
Anand
From: Matt Foley <mf...@hortonworks.com>>
Reply-To: "user@metron.incubator.apache.org<ma...@metron.incubator.apache.org>" <us...@metron.incubator.apache.org>>
Date: Monday, March 13, 2017 at 11:12 AM
To: "user@metron.incubator.apache.org<ma...@metron.incubator.apache.org>" <us...@metron.incubator.apache.org>>
Subject: Re: a proper entry level documentation
Hi Moshe,
Re the Elastic Search port question, ES uses both 9200 and 9300. This is consistent with the answer to the stackoverflow question you cited. Port 9200 is used for RESTful API and Browser interactions. Port 9300 is used for Java Client APIs and intra-cluster communications.
The RESTful port configuration is stated to be 9200-9300 (meaning, I think, 9200-9299, which should have been the answer to that stackoverflow question), I’m not sure how anything but port 9200 would actually be used given the behavior of REST, but perhaps it is.
However, the Java/intra-cluster port usage is stated to be 9300-9400 (probably also meaning 9300-9399), and these clients, which are provided by the ES application code, apparently do know how to switch ports within that range, in cooperation with the server, to support multiple simultaneous persistent connections.
Both port ranges must be configured for ES to work correctly, but of course you can change the actual port range values if needful in your environment.
From: moshe jarusalem <tu...@gmail.com>>
Reply-To: "user@metron.incubator.apache.org<ma...@metron.incubator.apache.org>" <us...@metron.incubator.apache.org>>
Date: Sunday, March 12, 2017 at 12:46 AM
To: "user@metron.incubator.apache.org<ma...@metron.incubator.apache.org>" <us...@metron.incubator.apache.org>>
Subject: Re: a proper entry level documentation
Thanks for all your efforts.
I have followed https://cwiki.apache.org/confluence/display/METRON/2016/04/28/Metron+Tutorial+-+Fundamentals+Part+2%3A+Creating+a+New+Enrichment
Unfortuanatly it is not working as expected. I get one error from kafka
"Offset lags for kafka not supported for older versions. Please update kafka spout to latest version." I checked storm kafka version but it seems our storm version already contains the fix.
Yet another question is elastic search port is defined to be 9300 in configurations but actually it runs only on 9200.
http://stackoverflow.com/questions/24624980/port-9300-on-elasticsearch
Would you please also tell how to enable debug logs for topologies so that we can flow message flow and troubleshoot ?
Cheers,
On Fri, Mar 10, 2017 at 9:22 PM, Umesh Kaushik <um...@bhujang.net>> wrote:
Thanks Matt! Kudos!
Cheers,
Umesh Kaushik
(Full stack developer- cyber security analyst: Bhujang innovations)
(9620023458)
Sent from mobile device, kindly excuse any typographic error.
On Mar 10, 2017 11:49 PM, "Matt Foley" <ma...@apache.org>> wrote:
Thanks for this contribution, Umesh.
I’ve edited it lightly, and posted it to the wiki at
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=68718548
along with the other installation-related help files.
Regards,
--Matt
From: Umesh Kaushik <um...@bhujang.net>>
Date: Wednesday, March 8, 2017 at 5:18 PM
To: Matt Foley <ma...@apache.org>>
Cc: "user@metron.incubator.apache.org<ma...@metron.incubator.apache.org>" <us...@metron.incubator.apache.org>>
Subject: Re: a proper entry level documentation
Hi Matt,
Yes I have no issues in that, in fact I would be happy if this document could be even a little bit of useful.
Yes, it was default master version of metron at the time I installed. It was in the beginning of January for poc and yes ambari was also default. Please let me know if I could assist you any further.
Cheers,
Umesh Kaushik
(9620023458)
Sent from mobile device, kindly excuse any typographic error.
On Mar 9, 2017 12:57 AM, "Matt Foley" <ma...@apache.org>> wrote:
Umesh, thanks very much for writing this up. I’d like to add it to the Metron wiki. Is that okay with you?
Also, what version of Metron was used? Your instructions imply you built from “master” branch of github site, which means you would be on the newest post-0.3.1 code (call it 0.3.2-SNAPSHOT). Is that correct?
And what version of Ambari did you use? I think the packaging defaults to version 2.4.0.0, is that what was used?
Thanks,
--Matt
From: Umesh Kaushik <um...@bhujang.net>>
Reply-To: "user@metron.incubator.apache.org<ma...@metron.incubator.apache.org>" <us...@metron.incubator.apache.org>>
Date: Wednesday, March 8, 2017 at 2:04 AM
To: "user@metron.incubator.apache.org<ma...@metron.incubator.apache.org>" <us...@metron.incubator.apache.org>>
Subject: Re: a proper entry level documentation
and allotted 2 cores of processor.
On 8 March 2017 at 15:32, Umesh Kaushik <um...@bhujang.net>> wrote:
Hi Khurram Ahmed,
Pardon me for the vague answer, however I just created a random Vm, with 100 gb hdd and 20 gb ram, I have a server with 128 gb of ram and 1tb hdd. So, I just chose a random config.
On 8 March 2017 at 15:13, Khurram Ahmed <kh...@gmail.com>> wrote:
Excellent could you mention the specs that allowed you to run the platform without a hitch? Mention RAM, CPU and hard drive space plz.
On Mar 8, 2017 2:38 PM, "Umesh Kaushik" <um...@bhujang.net>> wrote:
Hi Guys,
This is the bare minimum installation document I prepared while installing metron on ubuntu. Please pardon if there are typos or unprofessionalism in the document. Hope this helps!
On 8 March 2017 at 14:19, Umesh Kaushik <um...@bhujang.net>> wrote:
I tried installing on ubuntu, and I documented all the steps and issues I faced, I would love to attach it in the email thread, please let me know if that suits you all, but that is basic installation. I did not document configuring sqid and other addons. Please let me know if it is needed.
On 6 March 2017 at 21:56, Earl Hinkle <ea...@thehinkles.org>> wrote:
I tried on centos 6 and had problems getting python 2.7 installed and some of the other prerequisite. So i triend centos 7, got all the packages installed (python, ansible,etc) and got further into the build but it still failed.
I just saw the previous email about a centos 6 script so will try that this week and see how it goes.
Sent with AquaMail for Android
http://www.aqua-mail.com
On March 5, 2017 3:10:08 PM "Zeolla@GMail.com<ma...@GMail.com>" <ze...@gmail.com>> wrote:
What OS are you trying with now? Like I mentioned, the scripts I put together before will do Mac and centos 6 but it should be possible to get working on more than that. Your choice, but regardless I plan to work further on documentation. If there's anything specific you'd like addressed or shown please let me know.
Jon
On Sun, Mar 5, 2017, 4:31 PM Earl Hinkle <ea...@thehinkles.org>> wrote:
I would be willing to help using centos if i could get it up and working.
Sent with AquaMail for Android
http://www.aqua-mail.com
On March 5, 2017 2:23:32 AM moshe jarusalem <tu...@gmail.com>> wrote:
Hi All,
I have been trying to get involved with the metron project for a while. I have tried release 0.3.0 and 0.3.1. Unfortunately, it is very difficult to set up a running environment and understand the picture how it works. There are many technologies prerequisite such as Hadoop, nifi, storm, Kafka etc. I assume most of you guys are expert at these technologies so you solve your problems.
I have also browsed the user's list and there are many people struggling to set up a proper environment.
Would you guys make a document describing step by step setting up a development environment with a specific release (0.3.1 as its the latest) ? Can you describe each technology shortly and give references and most importantly metron's usage of them?
Indeed a video recording or a webinar would be much more useful so that there can be some discussions.
Regards,
Moshe
--
Jon
Sent from my mobile device
--
Cheers,
Umesh Kaushik
(9620023458)
--
Cheers,
Umesh Kaushik
(9620023458)
--
Cheers,
Umesh Kaushik
(9620023458)
--
Cheers,
Umesh Kaushik
(9620023458)
Re: a proper entry level documentation
Posted by Matt Foley <mf...@hortonworks.com>.
Hi Moshe,
Re the Elastic Search port question, ES uses both 9200 and 9300. This is consistent with the answer to the stackoverflow question you cited. Port 9200 is used for RESTful API and Browser interactions. Port 9300 is used for Java Client APIs and intra-cluster communications.
The RESTful port configuration is stated to be 9200-9300 (meaning, I think, 9200-9299, which should have been the answer to that stackoverflow question), I’m not sure how anything but port 9200 would actually be used given the behavior of REST, but perhaps it is.
However, the Java/intra-cluster port usage is stated to be 9300-9400 (probably also meaning 9300-9399), and these clients, which are provided by the ES application code, apparently do know how to switch ports within that range, in cooperation with the server, to support multiple simultaneous persistent connections.
Both port ranges must be configured for ES to work correctly, but of course you can change the actual port range values if needful in your environment.
From: moshe jarusalem <tu...@gmail.com>
Reply-To: "user@metron.incubator.apache.org" <us...@metron.incubator.apache.org>
Date: Sunday, March 12, 2017 at 12:46 AM
To: "user@metron.incubator.apache.org" <us...@metron.incubator.apache.org>
Subject: Re: a proper entry level documentation
Thanks for all your efforts.
I have followed https://cwiki.apache.org/confluence/display/METRON/2016/04/28/Metron+Tutorial+-+Fundamentals+Part+2%3A+Creating+a+New+Enrichment
Unfortuanatly it is not working as expected. I get one error from kafka
"Offset lags for kafka not supported for older versions. Please update kafka spout to latest version." I checked storm kafka version but it seems our storm version already contains the fix.
Yet another question is elastic search port is defined to be 9300 in configurations but actually it runs only on 9200.
http://stackoverflow.com/questions/24624980/port-9300-on-elasticsearch
Would you please also tell how to enable debug logs for topologies so that we can flow message flow and troubleshoot ?
Cheers,
On Fri, Mar 10, 2017 at 9:22 PM, Umesh Kaushik <um...@bhujang.net>> wrote:
Thanks Matt! Kudos!
Cheers,
Umesh Kaushik
(Full stack developer- cyber security analyst: Bhujang innovations)
(9620023458)
Sent from mobile device, kindly excuse any typographic error.
On Mar 10, 2017 11:49 PM, "Matt Foley" <ma...@apache.org>> wrote:
Thanks for this contribution, Umesh.
I’ve edited it lightly, and posted it to the wiki at
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=68718548
along with the other installation-related help files.
Regards,
--Matt
From: Umesh Kaushik <um...@bhujang.net>>
Date: Wednesday, March 8, 2017 at 5:18 PM
To: Matt Foley <ma...@apache.org>>
Cc: "user@metron.incubator.apache.org<ma...@metron.incubator.apache.org>" <us...@metron.incubator.apache.org>>
Subject: Re: a proper entry level documentation
Hi Matt,
Yes I have no issues in that, in fact I would be happy if this document could be even a little bit of useful.
Yes, it was default master version of metron at the time I installed. It was in the beginning of January for poc and yes ambari was also default. Please let me know if I could assist you any further.
Cheers,
Umesh Kaushik
(9620023458)
Sent from mobile device, kindly excuse any typographic error.
On Mar 9, 2017 12:57 AM, "Matt Foley" <ma...@apache.org>> wrote:
Umesh, thanks very much for writing this up. I’d like to add it to the Metron wiki. Is that okay with you?
Also, what version of Metron was used? Your instructions imply you built from “master” branch of github site, which means you would be on the newest post-0.3.1 code (call it 0.3.2-SNAPSHOT). Is that correct?
And what version of Ambari did you use? I think the packaging defaults to version 2.4.0.0, is that what was used?
Thanks,
--Matt
From: Umesh Kaushik <um...@bhujang.net>>
Reply-To: "user@metron.incubator.apache.org<ma...@metron.incubator.apache.org>" <us...@metron.incubator.apache.org>>
Date: Wednesday, March 8, 2017 at 2:04 AM
To: "user@metron.incubator.apache.org<ma...@metron.incubator.apache.org>" <us...@metron.incubator.apache.org>>
Subject: Re: a proper entry level documentation
and allotted 2 cores of processor.
On 8 March 2017 at 15:32, Umesh Kaushik <um...@bhujang.net>> wrote:
Hi Khurram Ahmed,
Pardon me for the vague answer, however I just created a random Vm, with 100 gb hdd and 20 gb ram, I have a server with 128 gb of ram and 1tb hdd. So, I just chose a random config.
On 8 March 2017 at 15:13, Khurram Ahmed <kh...@gmail.com>> wrote:
Excellent could you mention the specs that allowed you to run the platform without a hitch? Mention RAM, CPU and hard drive space plz.
On Mar 8, 2017 2:38 PM, "Umesh Kaushik" <um...@bhujang.net>> wrote:
Hi Guys,
This is the bare minimum installation document I prepared while installing metron on ubuntu. Please pardon if there are typos or unprofessionalism in the document. Hope this helps!
On 8 March 2017 at 14:19, Umesh Kaushik <um...@bhujang.net>> wrote:
I tried installing on ubuntu, and I documented all the steps and issues I faced, I would love to attach it in the email thread, please let me know if that suits you all, but that is basic installation. I did not document configuring sqid and other addons. Please let me know if it is needed.
On 6 March 2017 at 21:56, Earl Hinkle <ea...@thehinkles.org>> wrote:
I tried on centos 6 and had problems getting python 2.7 installed and some of the other prerequisite. So i triend centos 7, got all the packages installed (python, ansible,etc) and got further into the build but it still failed.
I just saw the previous email about a centos 6 script so will try that this week and see how it goes.
Sent with AquaMail for Android
http://www.aqua-mail.com
On March 5, 2017 3:10:08 PM "Zeolla@GMail.com" <ze...@gmail.com>> wrote:
What OS are you trying with now? Like I mentioned, the scripts I put together before will do Mac and centos 6 but it should be possible to get working on more than that. Your choice, but regardless I plan to work further on documentation. If there's anything specific you'd like addressed or shown please let me know.
Jon
On Sun, Mar 5, 2017, 4:31 PM Earl Hinkle <ea...@thehinkles.org>> wrote:
I would be willing to help using centos if i could get it up and working.
Sent with AquaMail for Android
http://www.aqua-mail.com
On March 5, 2017 2:23:32 AM moshe jarusalem <tu...@gmail.com>> wrote:
Hi All,
I have been trying to get involved with the metron project for a while. I have tried release 0.3.0 and 0.3.1. Unfortunately, it is very difficult to set up a running environment and understand the picture how it works. There are many technologies prerequisite such as Hadoop, nifi, storm, Kafka etc. I assume most of you guys are expert at these technologies so you solve your problems.
I have also browsed the user's list and there are many people struggling to set up a proper environment.
Would you guys make a document describing step by step setting up a development environment with a specific release (0.3.1 as its the latest) ? Can you describe each technology shortly and give references and most importantly metron's usage of them?
Indeed a video recording or a webinar would be much more useful so that there can be some discussions.
Regards,
Moshe
--
Jon
Sent from my mobile device
--
Cheers,
Umesh Kaushik
(9620023458)
--
Cheers,
Umesh Kaushik
(9620023458)
--
Cheers,
Umesh Kaushik
(9620023458)
--
Cheers,
Umesh Kaushik
(9620023458)
Re: a proper entry level documentation
Posted by moshe jarusalem <tu...@gmail.com>.
Thanks for all your efforts.
I have followed
https://cwiki.apache.org/confluence/display/METRON/2016/04/28/Metron+Tutorial+-+Fundamentals+Part+2%3A+Creating+a+New+Enrichment
Unfortuanatly it is not working as expected. I get one error from kafka
"Offset lags for kafka not supported for older versions. Please update
kafka spout to latest version." I checked storm kafka version but it seems
our storm version already contains the fix.
Yet another question is elastic search port is defined to be 9300 in
configurations but actually it runs only on 9200.
http://stackoverflow.com/questions/24624980/port-9300-on-elasticsearch
Would you please also tell how to enable debug logs for topologies so that
we can flow message flow and troubleshoot ?
Cheers,
On Fri, Mar 10, 2017 at 9:22 PM, Umesh Kaushik <um...@bhujang.net>
wrote:
> Thanks Matt! Kudos!
>
> Cheers,
> Umesh Kaushik
> (Full stack developer- cyber security analyst: Bhujang innovations)
> (9620023458)
> Sent from mobile device, kindly excuse any typographic error.
>
> On Mar 10, 2017 11:49 PM, "Matt Foley" <ma...@apache.org> wrote:
>
>> Thanks for this contribution, Umesh.
>>
>> I’ve edited it lightly, and posted it to the wiki at
>>
>> https://cwiki.apache.org/confluence/pages/viewpage.actio
>> n?pageId=68718548
>>
>> along with the other installation-related help files.
>>
>>
>>
>> Regards,
>>
>> --Matt
>>
>>
>>
>> *From: *Umesh Kaushik <um...@bhujang.net>
>> *Date: *Wednesday, March 8, 2017 at 5:18 PM
>> *To: *Matt Foley <ma...@apache.org>
>> *Cc: *"user@metron.incubator.apache.org" <user@metron.incubator.apache.
>> org>
>> *Subject: *Re: a proper entry level documentation
>>
>>
>>
>> Hi Matt,
>>
>> Yes I have no issues in that, in fact I would be happy if this document
>> could be even a little bit of useful.
>>
>> Yes, it was default master version of metron at the time I installed. It
>> was in the beginning of January for poc and yes ambari was also default.
>> Please let me know if I could assist you any further.
>>
>> Cheers,
>> Umesh Kaushik
>> (9620023458)
>> Sent from mobile device, kindly excuse any typographic error.
>>
>>
>>
>> On Mar 9, 2017 12:57 AM, "Matt Foley" <ma...@apache.org> wrote:
>>
>> Umesh, thanks very much for writing this up. I’d like to add it to the
>> Metron wiki. Is that okay with you?
>>
>>
>>
>> Also, what version of Metron was used? Your instructions imply you built
>> from “master” branch of github site, which means you would be on the newest
>> post-0.3.1 code (call it 0.3.2-SNAPSHOT). Is that correct?
>>
>>
>>
>> And what version of Ambari did you use? I think the packaging defaults
>> to version 2.4.0.0, is that what was used?
>>
>>
>>
>> Thanks,
>>
>> --Matt
>>
>>
>>
>> *From: *Umesh Kaushik <um...@bhujang.net>
>> *Reply-To: *"user@metron.incubator.apache.org" <
>> user@metron.incubator.apache.org>
>> *Date: *Wednesday, March 8, 2017 at 2:04 AM
>> *To: *"user@metron.incubator.apache.org" <user@metron.incubator.apache.
>> org>
>> *Subject: *Re: a proper entry level documentation
>>
>>
>>
>> and allotted 2 cores of processor.
>>
>>
>>
>> On 8 March 2017 at 15:32, Umesh Kaushik <um...@bhujang.net>
>> wrote:
>>
>> Hi Khurram Ahmed,
>>
>>
>>
>> Pardon me for the vague answer, however I just created a random Vm, with
>> 100 gb hdd and 20 gb ram, I have a server with 128 gb of ram and 1tb hdd.
>> So, I just chose a random config.
>>
>>
>>
>> On 8 March 2017 at 15:13, Khurram Ahmed <kh...@gmail.com> wrote:
>>
>> Excellent could you mention the specs that allowed you to run the
>> platform without a hitch? Mention RAM, CPU and hard drive space plz.
>>
>>
>>
>> On Mar 8, 2017 2:38 PM, "Umesh Kaushik" <um...@bhujang.net>
>> wrote:
>>
>> Hi Guys,
>>
>>
>>
>> This is the bare minimum installation document I prepared while
>> installing metron on ubuntu. Please pardon if there are typos or
>> unprofessionalism in the document. Hope this helps!
>>
>>
>>
>> On 8 March 2017 at 14:19, Umesh Kaushik <um...@bhujang.net>
>> wrote:
>>
>> I tried installing on ubuntu, and I documented all the steps and issues I
>> faced, I would love to attach it in the email thread, please let me know if
>> that suits you all, but that is basic installation. I did not document
>> configuring sqid and other addons. Please let me know if it is needed.
>>
>>
>>
>> On 6 March 2017 at 21:56, Earl Hinkle <ea...@thehinkles.org> wrote:
>>
>> I tried on centos 6 and had problems getting python 2.7 installed and
>> some of the other prerequisite. So i triend centos 7, got all the packages
>> installed (python, ansible,etc) and got further into the build but it still
>> failed.
>>
>> I just saw the previous email about a centos 6 script so will try that
>> this week and see how it goes.
>>
>> Sent with AquaMail for Android
>> http://www.aqua-mail.com
>>
>> On March 5, 2017 3:10:08 PM "Zeolla@GMail.com" <ze...@gmail.com> wrote:
>>
>> What OS are you trying with now? Like I mentioned, the scripts I put
>> together before will do Mac and centos 6 but it should be possible to get
>> working on more than that. Your choice, but regardless I plan to work
>> further on documentation. If there's anything specific you'd like
>> addressed or shown please let me know.
>>
>> Jon
>>
>>
>>
>> On Sun, Mar 5, 2017, 4:31 PM Earl Hinkle <ea...@thehinkles.org> wrote:
>>
>> I would be willing to help using centos if i could get it up and working.
>>
>> Sent with AquaMail for Android
>> http://www.aqua-mail.com
>>
>> On March 5, 2017 2:23:32 AM moshe jarusalem <tu...@gmail.com> wrote:
>>
>> Hi All,
>>
>> I have been trying to get involved with the metron project for a while. I
>> have tried release 0.3.0 and 0.3.1. Unfortunately, it is very difficult to
>> set up a running environment and understand the picture how it works. There
>> are many technologies prerequisite such as Hadoop, nifi, storm, Kafka etc. I
>> assume most of you guys are expert at these technologies so you solve your
>> problems.
>>
>> I have also browsed the user's list and there are many people struggling
>> to set up a proper environment.
>>
>>
>>
>> Would you guys make a document describing step by step setting up a
>> development environment with a specific release (0.3.1 as its the latest) ?
>> Can you describe each technology shortly and give references and most
>> importantly metron's usage of them?
>>
>> Indeed a video recording or a webinar would be much more useful so that
>> there can be some discussions.
>>
>>
>>
>> Regards,
>>
>> Moshe
>>
>> --
>>
>> Jon
>>
>> Sent from my mobile device
>>
>>
>>
>>
>>
>> --
>>
>> Cheers,
>>
>> Umesh Kaushik
>>
>> (9620023458)
>>
>>
>>
>>
>>
>> --
>>
>> Cheers,
>>
>> Umesh Kaushik
>>
>> (9620023458)
>>
>>
>>
>>
>>
>> --
>>
>> Cheers,
>>
>> Umesh Kaushik
>>
>> (9620023458)
>>
>>
>>
>>
>>
>> --
>>
>> Cheers,
>>
>> Umesh Kaushik
>>
>> (9620023458)
>>
>>
Re: a proper entry level documentation
Posted by Umesh Kaushik <um...@bhujang.net>.
Thanks Matt! Kudos!
Cheers,
Umesh Kaushik
(Full stack developer- cyber security analyst: Bhujang innovations)
(9620023458)
Sent from mobile device, kindly excuse any typographic error.
On Mar 10, 2017 11:49 PM, "Matt Foley" <ma...@apache.org> wrote:
> Thanks for this contribution, Umesh.
>
> I’ve edited it lightly, and posted it to the wiki at
>
> https://cwiki.apache.org/confluence/pages/viewpage.
> action?pageId=68718548
>
> along with the other installation-related help files.
>
>
>
> Regards,
>
> --Matt
>
>
>
> *From: *Umesh Kaushik <um...@bhujang.net>
> *Date: *Wednesday, March 8, 2017 at 5:18 PM
> *To: *Matt Foley <ma...@apache.org>
> *Cc: *"user@metron.incubator.apache.org" <user@metron.incubator.apache.org
> >
> *Subject: *Re: a proper entry level documentation
>
>
>
> Hi Matt,
>
> Yes I have no issues in that, in fact I would be happy if this document
> could be even a little bit of useful.
>
> Yes, it was default master version of metron at the time I installed. It
> was in the beginning of January for poc and yes ambari was also default.
> Please let me know if I could assist you any further.
>
> Cheers,
> Umesh Kaushik
> (9620023458)
> Sent from mobile device, kindly excuse any typographic error.
>
>
>
> On Mar 9, 2017 12:57 AM, "Matt Foley" <ma...@apache.org> wrote:
>
> Umesh, thanks very much for writing this up. I’d like to add it to the
> Metron wiki. Is that okay with you?
>
>
>
> Also, what version of Metron was used? Your instructions imply you built
> from “master” branch of github site, which means you would be on the newest
> post-0.3.1 code (call it 0.3.2-SNAPSHOT). Is that correct?
>
>
>
> And what version of Ambari did you use? I think the packaging defaults to
> version 2.4.0.0, is that what was used?
>
>
>
> Thanks,
>
> --Matt
>
>
>
> *From: *Umesh Kaushik <um...@bhujang.net>
> *Reply-To: *"user@metron.incubator.apache.org" <
> user@metron.incubator.apache.org>
> *Date: *Wednesday, March 8, 2017 at 2:04 AM
> *To: *"user@metron.incubator.apache.org" <user@metron.incubator.apache.org
> >
> *Subject: *Re: a proper entry level documentation
>
>
>
> and allotted 2 cores of processor.
>
>
>
> On 8 March 2017 at 15:32, Umesh Kaushik <um...@bhujang.net> wrote:
>
> Hi Khurram Ahmed,
>
>
>
> Pardon me for the vague answer, however I just created a random Vm, with
> 100 gb hdd and 20 gb ram, I have a server with 128 gb of ram and 1tb hdd.
> So, I just chose a random config.
>
>
>
> On 8 March 2017 at 15:13, Khurram Ahmed <kh...@gmail.com> wrote:
>
> Excellent could you mention the specs that allowed you to run the platform
> without a hitch? Mention RAM, CPU and hard drive space plz.
>
>
>
> On Mar 8, 2017 2:38 PM, "Umesh Kaushik" <um...@bhujang.net> wrote:
>
> Hi Guys,
>
>
>
> This is the bare minimum installation document I prepared while installing
> metron on ubuntu. Please pardon if there are typos or unprofessionalism in
> the document. Hope this helps!
>
>
>
> On 8 March 2017 at 14:19, Umesh Kaushik <um...@bhujang.net> wrote:
>
> I tried installing on ubuntu, and I documented all the steps and issues I
> faced, I would love to attach it in the email thread, please let me know if
> that suits you all, but that is basic installation. I did not document
> configuring sqid and other addons. Please let me know if it is needed.
>
>
>
> On 6 March 2017 at 21:56, Earl Hinkle <ea...@thehinkles.org> wrote:
>
> I tried on centos 6 and had problems getting python 2.7 installed and some
> of the other prerequisite. So i triend centos 7, got all the packages
> installed (python, ansible,etc) and got further into the build but it still
> failed.
>
> I just saw the previous email about a centos 6 script so will try that
> this week and see how it goes.
>
> Sent with AquaMail for Android
> http://www.aqua-mail.com
>
> On March 5, 2017 3:10:08 PM "Zeolla@GMail.com" <ze...@gmail.com> wrote:
>
> What OS are you trying with now? Like I mentioned, the scripts I put
> together before will do Mac and centos 6 but it should be possible to get
> working on more than that. Your choice, but regardless I plan to work
> further on documentation. If there's anything specific you'd like
> addressed or shown please let me know.
>
> Jon
>
>
>
> On Sun, Mar 5, 2017, 4:31 PM Earl Hinkle <ea...@thehinkles.org> wrote:
>
> I would be willing to help using centos if i could get it up and working.
>
> Sent with AquaMail for Android
> http://www.aqua-mail.com
>
> On March 5, 2017 2:23:32 AM moshe jarusalem <tu...@gmail.com> wrote:
>
> Hi All,
>
> I have been trying to get involved with the metron project for a while. I
> have tried release 0.3.0 and 0.3.1. Unfortunately, it is very difficult to
> set up a running environment and understand the picture how it works. There
> are many technologies prerequisite such as Hadoop, nifi, storm, Kafka etc. I
> assume most of you guys are expert at these technologies so you solve your
> problems.
>
> I have also browsed the user's list and there are many people struggling
> to set up a proper environment.
>
>
>
> Would you guys make a document describing step by step setting up a
> development environment with a specific release (0.3.1 as its the latest) ?
> Can you describe each technology shortly and give references and most
> importantly metron's usage of them?
>
> Indeed a video recording or a webinar would be much more useful so that
> there can be some discussions.
>
>
>
> Regards,
>
> Moshe
>
> --
>
> Jon
>
> Sent from my mobile device
>
>
>
>
>
> --
>
> Cheers,
>
> Umesh Kaushik
>
> (9620023458)
>
>
>
>
>
> --
>
> Cheers,
>
> Umesh Kaushik
>
> (9620023458)
>
>
>
>
>
> --
>
> Cheers,
>
> Umesh Kaushik
>
> (9620023458)
>
>
>
>
>
> --
>
> Cheers,
>
> Umesh Kaushik
>
> (9620023458)
>
>
Re: a proper entry level documentation
Posted by Matt Foley <ma...@apache.org>.
Thanks for this contribution, Umesh.
I’ve edited it lightly, and posted it to the wiki at
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=68718548
along with the other installation-related help files.
Regards,
--Matt
From: Umesh Kaushik <um...@bhujang.net>
Date: Wednesday, March 8, 2017 at 5:18 PM
To: Matt Foley <ma...@apache.org>
Cc: "user@metron.incubator.apache.org" <us...@metron.incubator.apache.org>
Subject: Re: a proper entry level documentation
Hi Matt,
Yes I have no issues in that, in fact I would be happy if this document could be even a little bit of useful.
Yes, it was default master version of metron at the time I installed. It was in the beginning of January for poc and yes ambari was also default. Please let me know if I could assist you any further.
Cheers,
Umesh Kaushik
(9620023458)
Sent from mobile device, kindly excuse any typographic error.
On Mar 9, 2017 12:57 AM, "Matt Foley" <ma...@apache.org> wrote:
Umesh, thanks very much for writing this up. I’d like to add it to the Metron wiki. Is that okay with you?
Also, what version of Metron was used? Your instructions imply you built from “master” branch of github site, which means you would be on the newest post-0.3.1 code (call it 0.3.2-SNAPSHOT). Is that correct?
And what version of Ambari did you use? I think the packaging defaults to version 2.4.0.0, is that what was used?
Thanks,
--Matt
From: Umesh Kaushik <um...@bhujang.net>
Reply-To: "user@metron.incubator.apache.org" <us...@metron.incubator.apache.org>
Date: Wednesday, March 8, 2017 at 2:04 AM
To: "user@metron.incubator.apache.org" <us...@metron.incubator.apache.org>
Subject: Re: a proper entry level documentation
and allotted 2 cores of processor.
On 8 March 2017 at 15:32, Umesh Kaushik <um...@bhujang.net> wrote:
Hi Khurram Ahmed,
Pardon me for the vague answer, however I just created a random Vm, with 100 gb hdd and 20 gb ram, I have a server with 128 gb of ram and 1tb hdd. So, I just chose a random config.
On 8 March 2017 at 15:13, Khurram Ahmed <kh...@gmail.com> wrote:
Excellent could you mention the specs that allowed you to run the platform without a hitch? Mention RAM, CPU and hard drive space plz.
On Mar 8, 2017 2:38 PM, "Umesh Kaushik" <um...@bhujang.net> wrote:
Hi Guys,
This is the bare minimum installation document I prepared while installing metron on ubuntu. Please pardon if there are typos or unprofessionalism in the document. Hope this helps!
On 8 March 2017 at 14:19, Umesh Kaushik <um...@bhujang.net> wrote:
I tried installing on ubuntu, and I documented all the steps and issues I faced, I would love to attach it in the email thread, please let me know if that suits you all, but that is basic installation. I did not document configuring sqid and other addons. Please let me know if it is needed.
On 6 March 2017 at 21:56, Earl Hinkle <ea...@thehinkles.org> wrote:
I tried on centos 6 and had problems getting python 2.7 installed and some of the other prerequisite. So i triend centos 7, got all the packages installed (python, ansible,etc) and got further into the build but it still failed.
I just saw the previous email about a centos 6 script so will try that this week and see how it goes.
Sent with AquaMail for Android
http://www.aqua-mail.com
On March 5, 2017 3:10:08 PM "Zeolla@GMail.com" <ze...@gmail.com> wrote:
What OS are you trying with now? Like I mentioned, the scripts I put together before will do Mac and centos 6 but it should be possible to get working on more than that. Your choice, but regardless I plan to work further on documentation. If there's anything specific you'd like addressed or shown please let me know.
Jon
On Sun, Mar 5, 2017, 4:31 PM Earl Hinkle <ea...@thehinkles.org> wrote:
I would be willing to help using centos if i could get it up and working.
Sent with AquaMail for Android
http://www.aqua-mail.com
On March 5, 2017 2:23:32 AM moshe jarusalem <tu...@gmail.com> wrote:
Hi All,
I have been trying to get involved with the metron project for a while. I have tried release 0.3.0 and 0.3.1. Unfortunately, it is very difficult to set up a running environment and understand the picture how it works. There are many technologies prerequisite such as Hadoop, nifi, storm, Kafka etc. I assume most of you guys are expert at these technologies so you solve your problems.
I have also browsed the user's list and there are many people struggling to set up a proper environment.
Would you guys make a document describing step by step setting up a development environment with a specific release (0.3.1 as its the latest) ? Can you describe each technology shortly and give references and most importantly metron's usage of them?
Indeed a video recording or a webinar would be much more useful so that there can be some discussions.
Regards,
Moshe
--
Jon
Sent from my mobile device
--
Cheers,
Umesh Kaushik
(9620023458)
--
Cheers,
Umesh Kaushik
(9620023458)
--
Cheers,
Umesh Kaushik
(9620023458)
--
Cheers,
Umesh Kaushik
(9620023458)
Re: a proper entry level documentation
Posted by Umesh Kaushik <um...@bhujang.net>.
Hi Matt,
Yes I have no issues in that, in fact I would be happy if this document
could be even a little bit of useful.
Yes, it was default master version of metron at the time I installed. It
was in the beginning of January for poc and yes ambari was also default.
Please let me know if I could assist you any further.
Cheers,
Umesh Kaushik
(9620023458)
Sent from mobile device, kindly excuse any typographic error.
On Mar 9, 2017 12:57 AM, "Matt Foley" <ma...@apache.org> wrote:
> Umesh, thanks very much for writing this up. I’d like to add it to the
> Metron wiki. Is that okay with you?
>
>
>
> Also, what version of Metron was used? Your instructions imply you built
> from “master” branch of github site, which means you would be on the newest
> post-0.3.1 code (call it 0.3.2-SNAPSHOT). Is that correct?
>
>
>
> And what version of Ambari did you use? I think the packaging defaults to
> version 2.4.0.0, is that what was used?
>
>
>
> Thanks,
>
> --Matt
>
>
>
> *From: *Umesh Kaushik <um...@bhujang.net>
> *Reply-To: *"user@metron.incubator.apache.org" <
> user@metron.incubator.apache.org>
> *Date: *Wednesday, March 8, 2017 at 2:04 AM
> *To: *"user@metron.incubator.apache.org" <user@metron.incubator.apache.org
> >
> *Subject: *Re: a proper entry level documentation
>
>
>
> and allotted 2 cores of processor.
>
>
>
> On 8 March 2017 at 15:32, Umesh Kaushik <um...@bhujang.net> wrote:
>
> Hi Khurram Ahmed,
>
>
>
> Pardon me for the vague answer, however I just created a random Vm, with
> 100 gb hdd and 20 gb ram, I have a server with 128 gb of ram and 1tb hdd.
> So, I just chose a random config.
>
>
>
> On 8 March 2017 at 15:13, Khurram Ahmed <kh...@gmail.com> wrote:
>
> Excellent could you mention the specs that allowed you to run the platform
> without a hitch? Mention RAM, CPU and hard drive space plz.
>
>
>
> On Mar 8, 2017 2:38 PM, "Umesh Kaushik" <um...@bhujang.net> wrote:
>
> Hi Guys,
>
>
>
> This is the bare minimum installation document I prepared while installing
> metron on ubuntu. Please pardon if there are typos or unprofessionalism in
> the document. Hope this helps!
>
>
>
> On 8 March 2017 at 14:19, Umesh Kaushik <um...@bhujang.net> wrote:
>
> I tried installing on ubuntu, and I documented all the steps and issues I
> faced, I would love to attach it in the email thread, please let me know if
> that suits you all, but that is basic installation. I did not document
> configuring sqid and other addons. Please let me know if it is needed.
>
>
>
> On 6 March 2017 at 21:56, Earl Hinkle <ea...@thehinkles.org> wrote:
>
> I tried on centos 6 and had problems getting python 2.7 installed and some
> of the other prerequisite. So i triend centos 7, got all the packages
> installed (python, ansible,etc) and got further into the build but it still
> failed.
>
> I just saw the previous email about a centos 6 script so will try that
> this week and see how it goes.
>
> Sent with AquaMail for Android
> http://www.aqua-mail.com
>
> On March 5, 2017 3:10:08 PM "Zeolla@GMail.com" <ze...@gmail.com> wrote:
>
> What OS are you trying with now? Like I mentioned, the scripts I put
> together before will do Mac and centos 6 but it should be possible to get
> working on more than that. Your choice, but regardless I plan to work
> further on documentation. If there's anything specific you'd like
> addressed or shown please let me know.
>
> Jon
>
>
>
> On Sun, Mar 5, 2017, 4:31 PM Earl Hinkle <ea...@thehinkles.org> wrote:
>
> I would be willing to help using centos if i could get it up and working.
>
> Sent with AquaMail for Android
> http://www.aqua-mail.com
>
> On March 5, 2017 2:23:32 AM moshe jarusalem <tu...@gmail.com> wrote:
>
> Hi All,
>
> I have been trying to get involved with the metron project for a while. I
> have tried release 0.3.0 and 0.3.1. Unfortunately, it is very difficult to
> set up a running environment and understand the picture how it works. There
> are many technologies prerequisite such as Hadoop, nifi, storm, Kafka etc. I
> assume most of you guys are expert at these technologies so you solve your
> problems.
>
> I have also browsed the user's list and there are many people struggling
> to set up a proper environment.
>
>
>
> Would you guys make a document describing step by step setting up a
> development environment with a specific release (0.3.1 as its the latest) ?
> Can you describe each technology shortly and give references and most
> importantly metron's usage of them?
>
> Indeed a video recording or a webinar would be much more useful so that
> there can be some discussions.
>
>
>
> Regards,
>
> Moshe
>
> --
>
> Jon
>
> Sent from my mobile device
>
>
>
>
>
> --
>
> Cheers,
>
> Umesh Kaushik
>
> (9620023458)
>
>
>
>
>
> --
>
> Cheers,
>
> Umesh Kaushik
>
> (9620023458)
>
>
>
>
>
> --
>
> Cheers,
>
> Umesh Kaushik
>
> (9620023458)
>
>
>
>
>
> --
>
> Cheers,
>
> Umesh Kaushik
>
> (9620023458)
>
Re: a proper entry level documentation
Posted by Matt Foley <ma...@apache.org>.
Umesh, thanks very much for writing this up. I’d like to add it to the Metron wiki. Is that okay with you?
Also, what version of Metron was used? Your instructions imply you built from “master” branch of github site, which means you would be on the newest post-0.3.1 code (call it 0.3.2-SNAPSHOT). Is that correct?
And what version of Ambari did you use? I think the packaging defaults to version 2.4.0.0, is that what was used?
Thanks,
--Matt
From: Umesh Kaushik <um...@bhujang.net>
Reply-To: "user@metron.incubator.apache.org" <us...@metron.incubator.apache.org>
Date: Wednesday, March 8, 2017 at 2:04 AM
To: "user@metron.incubator.apache.org" <us...@metron.incubator.apache.org>
Subject: Re: a proper entry level documentation
and allotted 2 cores of processor.
On 8 March 2017 at 15:32, Umesh Kaushik <um...@bhujang.net> wrote:
Hi Khurram Ahmed,
Pardon me for the vague answer, however I just created a random Vm, with 100 gb hdd and 20 gb ram, I have a server with 128 gb of ram and 1tb hdd. So, I just chose a random config.
On 8 March 2017 at 15:13, Khurram Ahmed <kh...@gmail.com> wrote:
Excellent could you mention the specs that allowed you to run the platform without a hitch? Mention RAM, CPU and hard drive space plz.
On Mar 8, 2017 2:38 PM, "Umesh Kaushik" <um...@bhujang.net> wrote:
Hi Guys,
This is the bare minimum installation document I prepared while installing metron on ubuntu. Please pardon if there are typos or unprofessionalism in the document. Hope this helps!
On 8 March 2017 at 14:19, Umesh Kaushik <um...@bhujang.net> wrote:
I tried installing on ubuntu, and I documented all the steps and issues I faced, I would love to attach it in the email thread, please let me know if that suits you all, but that is basic installation. I did not document configuring sqid and other addons. Please let me know if it is needed.
On 6 March 2017 at 21:56, Earl Hinkle <ea...@thehinkles.org> wrote:
I tried on centos 6 and had problems getting python 2.7 installed and some of the other prerequisite. So i triend centos 7, got all the packages installed (python, ansible,etc) and got further into the build but it still failed.
I just saw the previous email about a centos 6 script so will try that this week and see how it goes.
Sent with AquaMail for Android
http://www.aqua-mail.com
On March 5, 2017 3:10:08 PM "Zeolla@GMail.com" <ze...@gmail.com> wrote:
What OS are you trying with now? Like I mentioned, the scripts I put together before will do Mac and centos 6 but it should be possible to get working on more than that. Your choice, but regardless I plan to work further on documentation. If there's anything specific you'd like addressed or shown please let me know.
Jon
On Sun, Mar 5, 2017, 4:31 PM Earl Hinkle <ea...@thehinkles.org> wrote:
I would be willing to help using centos if i could get it up and working.
Sent with AquaMail for Android
http://www.aqua-mail.com
On March 5, 2017 2:23:32 AM moshe jarusalem <tu...@gmail.com> wrote:
Hi All,
I have been trying to get involved with the metron project for a while. I have tried release 0.3.0 and 0.3.1. Unfortunately, it is very difficult to set up a running environment and understand the picture how it works. There are many technologies prerequisite such as Hadoop, nifi, storm, Kafka etc. I assume most of you guys are expert at these technologies so you solve your problems.
I have also browsed the user's list and there are many people struggling to set up a proper environment.
Would you guys make a document describing step by step setting up a development environment with a specific release (0.3.1 as its the latest) ? Can you describe each technology shortly and give references and most importantly metron's usage of them?
Indeed a video recording or a webinar would be much more useful so that there can be some discussions.
Regards,
Moshe
--
Jon
Sent from my mobile device
--
Cheers,
Umesh Kaushik
(9620023458)
--
Cheers,
Umesh Kaushik
(9620023458)
--
Cheers,
Umesh Kaushik
(9620023458)
--
Cheers,
Umesh Kaushik
(9620023458)
Re: a proper entry level documentation
Posted by Umesh Kaushik <um...@bhujang.net>.
and allotted 2 cores of processor.
On 8 March 2017 at 15:32, Umesh Kaushik <um...@bhujang.net> wrote:
> Hi Khurram Ahmed,
>
> Pardon me for the vague answer, however I just created a random Vm, with
> 100 gb hdd and 20 gb ram, I have a server with 128 gb of ram and 1tb hdd.
> So, I just chose a random config.
>
> On 8 March 2017 at 15:13, Khurram Ahmed <kh...@gmail.com> wrote:
>
>> Excellent could you mention the specs that allowed you to run the
>> platform without a hitch? Mention RAM, CPU and hard drive space plz.
>>
>> On Mar 8, 2017 2:38 PM, "Umesh Kaushik" <um...@bhujang.net>
>> wrote:
>>
>>> Hi Guys,
>>>
>>> This is the bare minimum installation document I prepared while
>>> installing metron on ubuntu. Please pardon if there are typos or
>>> unprofessionalism in the document. Hope this helps!
>>>
>>> On 8 March 2017 at 14:19, Umesh Kaushik <um...@bhujang.net>
>>> wrote:
>>>
>>>> I tried installing on ubuntu, and I documented all the steps and issues
>>>> I faced, I would love to attach it in the email thread, please let me know
>>>> if that suits you all, but that is basic installation. I did not document
>>>> configuring sqid and other addons. Please let me know if it is needed.
>>>>
>>>> On 6 March 2017 at 21:56, Earl Hinkle <ea...@thehinkles.org> wrote:
>>>>
>>>>> I tried on centos 6 and had problems getting python 2.7 installed and
>>>>> some of the other prerequisite. So i triend centos 7, got all the packages
>>>>> installed (python, ansible,etc) and got further into the build but it still
>>>>> failed.
>>>>>
>>>>> I just saw the previous email about a centos 6 script so will try that
>>>>> this week and see how it goes.
>>>>>
>>>>> Sent with AquaMail for Android
>>>>> http://www.aqua-mail.com
>>>>>
>>>>> On March 5, 2017 3:10:08 PM "Zeolla@GMail.com" <ze...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> What OS are you trying with now? Like I mentioned, the scripts I put
>>>>>> together before will do Mac and centos 6 but it should be possible to get
>>>>>> working on more than that. Your choice, but regardless I plan to work
>>>>>> further on documentation. If there's anything specific you'd like
>>>>>> addressed or shown please let me know.
>>>>>>
>>>>>> Jon
>>>>>>
>>>>>> On Sun, Mar 5, 2017, 4:31 PM Earl Hinkle <ea...@thehinkles.org> wrote:
>>>>>>
>>>>>>> I would be willing to help using centos if i could get it up and
>>>>>>> working.
>>>>>>>
>>>>>>> Sent with AquaMail for Android
>>>>>>> http://www.aqua-mail.com
>>>>>>>
>>>>>>> On March 5, 2017 2:23:32 AM moshe jarusalem <tu...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Hi All,
>>>>>>> I have been trying to get involved with the metron project for a
>>>>>>> while. I have tried release 0.3.0 and 0.3.1. Unfortunately, it is very
>>>>>>> difficult to set up a running environment and understand the picture how it
>>>>>>> works. There are many technologies prerequisite such as Hadoop, nifi,
>>>>>>> storm, Kafka etc. I assume most of you guys are expert at these
>>>>>>> technologies so you solve your problems.
>>>>>>> I have also browsed the user's list and there are many people
>>>>>>> struggling to set up a proper environment.
>>>>>>>
>>>>>>> Would you guys make a document describing step by step setting up a
>>>>>>> development environment with a specific release (0.3.1 as its the latest) ?
>>>>>>> Can you describe each technology shortly and give references and most
>>>>>>> importantly metron's usage of them?
>>>>>>> Indeed a video recording or a webinar would be much more useful so
>>>>>>> that there can be some discussions.
>>>>>>>
>>>>>>> Regards,
>>>>>>> Moshe
>>>>>>>
>>>>>>> --
>>>>>>
>>>>>> Jon
>>>>>>
>>>>>> Sent from my mobile device
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Cheers,
>>>> Umesh Kaushik
>>>> (9620023458)
>>>>
>>>
>>>
>>>
>>> --
>>> Cheers,
>>> Umesh Kaushik
>>> (9620023458)
>>>
>>
>
>
> --
> Cheers,
> Umesh Kaushik
> (9620023458)
>
--
Cheers,
Umesh Kaushik
(9620023458)
Re: a proper entry level documentation
Posted by Umesh Kaushik <um...@bhujang.net>.
Hi Khurram Ahmed,
Pardon me for the vague answer, however I just created a random Vm, with
100 gb hdd and 20 gb ram, I have a server with 128 gb of ram and 1tb hdd.
So, I just chose a random config.
On 8 March 2017 at 15:13, Khurram Ahmed <kh...@gmail.com> wrote:
> Excellent could you mention the specs that allowed you to run the platform
> without a hitch? Mention RAM, CPU and hard drive space plz.
>
> On Mar 8, 2017 2:38 PM, "Umesh Kaushik" <um...@bhujang.net> wrote:
>
>> Hi Guys,
>>
>> This is the bare minimum installation document I prepared while
>> installing metron on ubuntu. Please pardon if there are typos or
>> unprofessionalism in the document. Hope this helps!
>>
>> On 8 March 2017 at 14:19, Umesh Kaushik <um...@bhujang.net>
>> wrote:
>>
>>> I tried installing on ubuntu, and I documented all the steps and issues
>>> I faced, I would love to attach it in the email thread, please let me know
>>> if that suits you all, but that is basic installation. I did not document
>>> configuring sqid and other addons. Please let me know if it is needed.
>>>
>>> On 6 March 2017 at 21:56, Earl Hinkle <ea...@thehinkles.org> wrote:
>>>
>>>> I tried on centos 6 and had problems getting python 2.7 installed and
>>>> some of the other prerequisite. So i triend centos 7, got all the packages
>>>> installed (python, ansible,etc) and got further into the build but it still
>>>> failed.
>>>>
>>>> I just saw the previous email about a centos 6 script so will try that
>>>> this week and see how it goes.
>>>>
>>>> Sent with AquaMail for Android
>>>> http://www.aqua-mail.com
>>>>
>>>> On March 5, 2017 3:10:08 PM "Zeolla@GMail.com" <ze...@gmail.com>
>>>> wrote:
>>>>
>>>>> What OS are you trying with now? Like I mentioned, the scripts I put
>>>>> together before will do Mac and centos 6 but it should be possible to get
>>>>> working on more than that. Your choice, but regardless I plan to work
>>>>> further on documentation. If there's anything specific you'd like
>>>>> addressed or shown please let me know.
>>>>>
>>>>> Jon
>>>>>
>>>>> On Sun, Mar 5, 2017, 4:31 PM Earl Hinkle <ea...@thehinkles.org> wrote:
>>>>>
>>>>>> I would be willing to help using centos if i could get it up and
>>>>>> working.
>>>>>>
>>>>>> Sent with AquaMail for Android
>>>>>> http://www.aqua-mail.com
>>>>>>
>>>>>> On March 5, 2017 2:23:32 AM moshe jarusalem <tu...@gmail.com> wrote:
>>>>>>
>>>>>> Hi All,
>>>>>> I have been trying to get involved with the metron project for a
>>>>>> while. I have tried release 0.3.0 and 0.3.1. Unfortunately, it is very
>>>>>> difficult to set up a running environment and understand the picture how it
>>>>>> works. There are many technologies prerequisite such as Hadoop, nifi,
>>>>>> storm, Kafka etc. I assume most of you guys are expert at these
>>>>>> technologies so you solve your problems.
>>>>>> I have also browsed the user's list and there are many people
>>>>>> struggling to set up a proper environment.
>>>>>>
>>>>>> Would you guys make a document describing step by step setting up a
>>>>>> development environment with a specific release (0.3.1 as its the latest) ?
>>>>>> Can you describe each technology shortly and give references and most
>>>>>> importantly metron's usage of them?
>>>>>> Indeed a video recording or a webinar would be much more useful so
>>>>>> that there can be some discussions.
>>>>>>
>>>>>> Regards,
>>>>>> Moshe
>>>>>>
>>>>>> --
>>>>>
>>>>> Jon
>>>>>
>>>>> Sent from my mobile device
>>>>>
>>>>
>>>
>>>
>>> --
>>> Cheers,
>>> Umesh Kaushik
>>> (9620023458)
>>>
>>
>>
>>
>> --
>> Cheers,
>> Umesh Kaushik
>> (9620023458)
>>
>
--
Cheers,
Umesh Kaushik
(9620023458)
Re: a proper entry level documentation
Posted by Khurram Ahmed <kh...@gmail.com>.
Excellent could you mention the specs that allowed you to run the platform
without a hitch? Mention RAM, CPU and hard drive space plz.
On Mar 8, 2017 2:38 PM, "Umesh Kaushik" <um...@bhujang.net> wrote:
> Hi Guys,
>
> This is the bare minimum installation document I prepared while installing
> metron on ubuntu. Please pardon if there are typos or unprofessionalism in
> the document. Hope this helps!
>
> On 8 March 2017 at 14:19, Umesh Kaushik <um...@bhujang.net> wrote:
>
>> I tried installing on ubuntu, and I documented all the steps and issues I
>> faced, I would love to attach it in the email thread, please let me know if
>> that suits you all, but that is basic installation. I did not document
>> configuring sqid and other addons. Please let me know if it is needed.
>>
>> On 6 March 2017 at 21:56, Earl Hinkle <ea...@thehinkles.org> wrote:
>>
>>> I tried on centos 6 and had problems getting python 2.7 installed and
>>> some of the other prerequisite. So i triend centos 7, got all the packages
>>> installed (python, ansible,etc) and got further into the build but it still
>>> failed.
>>>
>>> I just saw the previous email about a centos 6 script so will try that
>>> this week and see how it goes.
>>>
>>> Sent with AquaMail for Android
>>> http://www.aqua-mail.com
>>>
>>> On March 5, 2017 3:10:08 PM "Zeolla@GMail.com" <ze...@gmail.com> wrote:
>>>
>>>> What OS are you trying with now? Like I mentioned, the scripts I put
>>>> together before will do Mac and centos 6 but it should be possible to get
>>>> working on more than that. Your choice, but regardless I plan to work
>>>> further on documentation. If there's anything specific you'd like
>>>> addressed or shown please let me know.
>>>>
>>>> Jon
>>>>
>>>> On Sun, Mar 5, 2017, 4:31 PM Earl Hinkle <ea...@thehinkles.org> wrote:
>>>>
>>>>> I would be willing to help using centos if i could get it up and
>>>>> working.
>>>>>
>>>>> Sent with AquaMail for Android
>>>>> http://www.aqua-mail.com
>>>>>
>>>>> On March 5, 2017 2:23:32 AM moshe jarusalem <tu...@gmail.com> wrote:
>>>>>
>>>>> Hi All,
>>>>> I have been trying to get involved with the metron project for a
>>>>> while. I have tried release 0.3.0 and 0.3.1. Unfortunately, it is very
>>>>> difficult to set up a running environment and understand the picture how it
>>>>> works. There are many technologies prerequisite such as Hadoop, nifi,
>>>>> storm, Kafka etc. I assume most of you guys are expert at these
>>>>> technologies so you solve your problems.
>>>>> I have also browsed the user's list and there are many people
>>>>> struggling to set up a proper environment.
>>>>>
>>>>> Would you guys make a document describing step by step setting up a
>>>>> development environment with a specific release (0.3.1 as its the latest) ?
>>>>> Can you describe each technology shortly and give references and most
>>>>> importantly metron's usage of them?
>>>>> Indeed a video recording or a webinar would be much more useful so
>>>>> that there can be some discussions.
>>>>>
>>>>> Regards,
>>>>> Moshe
>>>>>
>>>>> --
>>>>
>>>> Jon
>>>>
>>>> Sent from my mobile device
>>>>
>>>
>>
>>
>> --
>> Cheers,
>> Umesh Kaushik
>> (9620023458)
>>
>
>
>
> --
> Cheers,
> Umesh Kaushik
> (9620023458)
>
Re: a proper entry level documentation
Posted by Umesh Kaushik <um...@bhujang.net>.
Hi Guys,
This is the bare minimum installation document I prepared while installing
metron on ubuntu. Please pardon if there are typos or unprofessionalism in
the document. Hope this helps!
On 8 March 2017 at 14:19, Umesh Kaushik <um...@bhujang.net> wrote:
> I tried installing on ubuntu, and I documented all the steps and issues I
> faced, I would love to attach it in the email thread, please let me know if
> that suits you all, but that is basic installation. I did not document
> configuring sqid and other addons. Please let me know if it is needed.
>
> On 6 March 2017 at 21:56, Earl Hinkle <ea...@thehinkles.org> wrote:
>
>> I tried on centos 6 and had problems getting python 2.7 installed and
>> some of the other prerequisite. So i triend centos 7, got all the packages
>> installed (python, ansible,etc) and got further into the build but it still
>> failed.
>>
>> I just saw the previous email about a centos 6 script so will try that
>> this week and see how it goes.
>>
>> Sent with AquaMail for Android
>> http://www.aqua-mail.com
>>
>> On March 5, 2017 3:10:08 PM "Zeolla@GMail.com" <ze...@gmail.com> wrote:
>>
>>> What OS are you trying with now? Like I mentioned, the scripts I put
>>> together before will do Mac and centos 6 but it should be possible to get
>>> working on more than that. Your choice, but regardless I plan to work
>>> further on documentation. If there's anything specific you'd like
>>> addressed or shown please let me know.
>>>
>>> Jon
>>>
>>> On Sun, Mar 5, 2017, 4:31 PM Earl Hinkle <ea...@thehinkles.org> wrote:
>>>
>>>> I would be willing to help using centos if i could get it up and
>>>> working.
>>>>
>>>> Sent with AquaMail for Android
>>>> http://www.aqua-mail.com
>>>>
>>>> On March 5, 2017 2:23:32 AM moshe jarusalem <tu...@gmail.com> wrote:
>>>>
>>>> Hi All,
>>>> I have been trying to get involved with the metron project for a while.
>>>> I have tried release 0.3.0 and 0.3.1. Unfortunately, it is very difficult
>>>> to set up a running environment and understand the picture how it works.
>>>> There are many technologies prerequisite such as Hadoop, nifi, storm, Kafka
>>>> etc. I assume most of you guys are expert at these technologies so you
>>>> solve your problems.
>>>> I have also browsed the user's list and there are many people
>>>> struggling to set up a proper environment.
>>>>
>>>> Would you guys make a document describing step by step setting up a
>>>> development environment with a specific release (0.3.1 as its the latest) ?
>>>> Can you describe each technology shortly and give references and most
>>>> importantly metron's usage of them?
>>>> Indeed a video recording or a webinar would be much more useful so that
>>>> there can be some discussions.
>>>>
>>>> Regards,
>>>> Moshe
>>>>
>>>> --
>>>
>>> Jon
>>>
>>> Sent from my mobile device
>>>
>>
>
>
> --
> Cheers,
> Umesh Kaushik
> (9620023458)
>
--
Cheers,
Umesh Kaushik
(9620023458)
Re: a proper entry level documentation
Posted by Umesh Kaushik <um...@bhujang.net>.
I tried installing on ubuntu, and I documented all the steps and issues I
faced, I would love to attach it in the email thread, please let me know if
that suits you all, but that is basic installation. I did not document
configuring sqid and other addons. Please let me know if it is needed.
On 6 March 2017 at 21:56, Earl Hinkle <ea...@thehinkles.org> wrote:
> I tried on centos 6 and had problems getting python 2.7 installed and some
> of the other prerequisite. So i triend centos 7, got all the packages
> installed (python, ansible,etc) and got further into the build but it still
> failed.
>
> I just saw the previous email about a centos 6 script so will try that
> this week and see how it goes.
>
> Sent with AquaMail for Android
> http://www.aqua-mail.com
>
> On March 5, 2017 3:10:08 PM "Zeolla@GMail.com" <ze...@gmail.com> wrote:
>
>> What OS are you trying with now? Like I mentioned, the scripts I put
>> together before will do Mac and centos 6 but it should be possible to get
>> working on more than that. Your choice, but regardless I plan to work
>> further on documentation. If there's anything specific you'd like
>> addressed or shown please let me know.
>>
>> Jon
>>
>> On Sun, Mar 5, 2017, 4:31 PM Earl Hinkle <ea...@thehinkles.org> wrote:
>>
>>> I would be willing to help using centos if i could get it up and working.
>>>
>>> Sent with AquaMail for Android
>>> http://www.aqua-mail.com
>>>
>>> On March 5, 2017 2:23:32 AM moshe jarusalem <tu...@gmail.com> wrote:
>>>
>>> Hi All,
>>> I have been trying to get involved with the metron project for a while.
>>> I have tried release 0.3.0 and 0.3.1. Unfortunately, it is very difficult
>>> to set up a running environment and understand the picture how it works.
>>> There are many technologies prerequisite such as Hadoop, nifi, storm, Kafka
>>> etc. I assume most of you guys are expert at these technologies so you
>>> solve your problems.
>>> I have also browsed the user's list and there are many people struggling
>>> to set up a proper environment.
>>>
>>> Would you guys make a document describing step by step setting up a
>>> development environment with a specific release (0.3.1 as its the latest) ?
>>> Can you describe each technology shortly and give references and most
>>> importantly metron's usage of them?
>>> Indeed a video recording or a webinar would be much more useful so that
>>> there can be some discussions.
>>>
>>> Regards,
>>> Moshe
>>>
>>> --
>>
>> Jon
>>
>> Sent from my mobile device
>>
>
--
Cheers,
Umesh Kaushik
(9620023458)
Re: a proper entry level documentation
Posted by Earl Hinkle <ea...@thehinkles.org>.
I tried on centos 6 and had problems getting python 2.7 installed and some of the other prerequisite. So i triend centos 7, got all the packages installed (python, ansible,etc) and got further into the build but it still failed.
I just saw the previous email about a centos 6 script so will try that this week and see how it goes.
Sent with AquaMail for Android
http://www.aqua-mail.com
On March 5, 2017 3:10:08 PM "Zeolla@GMail.com" <ze...@gmail.com> wrote:
What OS are you trying with now? Like I mentioned, the scripts I put together before will do Mac and centos 6 but it should be possible to get working on more than that. Your choice, but regardless I plan to work further on documentation. If there's anything specific you'd like addressed or shown please let me know.
Jon
On Sun, Mar 5, 2017, 4:31 PM Earl Hinkle <ea...@thehinkles.org>> wrote:
I would be willing to help using centos if i could get it up and working.
Sent with AquaMail for Android
http://www.aqua-mail.com
On March 5, 2017 2:23:32 AM moshe jarusalem <tu...@gmail.com>> wrote:
Hi All,
I have been trying to get involved with the metron project for a while. I have tried release 0.3.0 and 0.3.1. Unfortunately, it is very difficult to set up a running environment and understand the picture how it works. There are many technologies prerequisite such as Hadoop, nifi, storm, Kafka etc. I assume most of you guys are expert at these technologies so you solve your problems.
I have also browsed the user's list and there are many people struggling to set up a proper environment.
Would you guys make a document describing step by step setting up a development environment with a specific release (0.3.1 as its the latest) ? Can you describe each technology shortly and give references and most importantly metron's usage of them?
Indeed a video recording or a webinar would be much more useful so that there can be some discussions.
Regards,
Moshe
--
Jon
Sent from my mobile device
Re: a proper entry level documentation
Posted by "Zeolla@GMail.com" <ze...@gmail.com>.
What OS are you trying with now? Like I mentioned, the scripts I put
together before will do Mac and centos 6 but it should be possible to get
working on more than that. Your choice, but regardless I plan to work
further on documentation. If there's anything specific you'd like
addressed or shown please let me know.
Jon
On Sun, Mar 5, 2017, 4:31 PM Earl Hinkle <ea...@thehinkles.org> wrote:
> I would be willing to help using centos if i could get it up and working.
>
> Sent with AquaMail for Android
> http://www.aqua-mail.com
>
> On March 5, 2017 2:23:32 AM moshe jarusalem <tu...@gmail.com> wrote:
>
> Hi All,
> I have been trying to get involved with the metron project for a while. I
> have tried release 0.3.0 and 0.3.1. Unfortunately, it is very difficult to
> set up a running environment and understand the picture how it works. There
> are many technologies prerequisite such as Hadoop, nifi, storm, Kafka etc. I
> assume most of you guys are expert at these technologies so you solve your
> problems.
> I have also browsed the user's list and there are many people struggling
> to set up a proper environment.
>
> Would you guys make a document describing step by step setting up a
> development environment with a specific release (0.3.1 as its the latest) ?
> Can you describe each technology shortly and give references and most
> importantly metron's usage of them?
> Indeed a video recording or a webinar would be much more useful so that
> there can be some discussions.
>
> Regards,
> Moshe
>
> --
Jon
Sent from my mobile device
Re: a proper entry level documentation
Posted by Earl Hinkle <ea...@thehinkles.org>.
I would be willing to help using centos if i could get it up and working.
Sent with AquaMail for Android
http://www.aqua-mail.com
On March 5, 2017 2:23:32 AM moshe jarusalem <tu...@gmail.com> wrote:
Hi All,
I have been trying to get involved with the metron project for a while. I have tried release 0.3.0 and 0.3.1. Unfortunately, it is very difficult to set up a running environment and understand the picture how it works. There are many technologies prerequisite such as Hadoop, nifi, storm, Kafka etc. I assume most of you guys are expert at these technologies so you solve your problems.
I have also browsed the user's list and there are many people struggling to set up a proper environment.
Would you guys make a document describing step by step setting up a development environment with a specific release (0.3.1 as its the latest) ? Can you describe each technology shortly and give references and most importantly metron's usage of them?
Indeed a video recording or a webinar would be much more useful so that there can be some discussions.
Regards,
Moshe
Re: a proper entry level documentation
Posted by "Zeolla@GMail.com" <ze...@gmail.com>.
I totally agree and I would be happy to help. For setup specifically,
should we prioritize a certain OS? To date the focus has mostly been macOS.
In order to help my personal development process, I scripted the install
process
<https://github.com/JonZeolla/lab-SecurityDataAnalysis/blob/master/setup/setup.sh>
for a fresh CentOS 6.8 box. You should be able to run that script with
some basic arguments and get Metron up and running. It also has some nice
features for testing new PRs or branches, but you can safely ignore those
if you just want to get things up and running.
In response to this email I also wrote a simple Mac installation script
<https://gist.github.com/JonZeolla/4abcbfee96437b948ab46b6782f00caa#file-metron-mac-setup-sh>
that should get things up and running. On my local machine it looked like
it was working - if you give it a shot, please let me know if it worked for
you.
A huge focus of the project has been to make things more simple to run (and
believe me, it is many many times easier than it used to be in the OpenSOC
days), in fact, there are numerous efforts being undertaken right now to
continue to make this easier. However you are definitely right, it is
still too complicated, and our documentation is severely lacking.
What do you think about the following regarding tutorials/videos:
- An Introduction to Hadoop (focused on the aspects that Metron uses)
- A step by step explanation of data flow in Metron (Parsers, Enrichment,
Indexing).
- Data Ingestion (getting data into the cluster)
- Data Retrieval and Visualization
- Cluster Troubleshooting
Is there anything missing? Thanks,
Jon
On Sun, Mar 5, 2017, 4:39 AM Khurram Ahmed <kh...@gmail.com> wrote:
I agree. The barrier to entry for new contributors is prohibitive due to
lack of accurate documentation for setting up dev environment.
On Mar 5, 2017 2:23 PM, "moshe jarusalem" <tu...@gmail.com> wrote:
Hi All,
I have been trying to get involved with the metron project for a while. I
have tried release 0.3.0 and 0.3.1. Unfortunately, it is very difficult to
set up a running environment and understand the picture how it works. There
are many technologies prerequisite such as Hadoop, nifi, storm, Kafka etc. I
assume most of you guys are expert at these technologies so you solve your
problems.
I have also browsed the user's list and there are many people struggling to
set up a proper environment.
Would you guys make a document describing step by step setting up a
development environment with a specific release (0.3.1 as its the latest) ?
Can you describe each technology shortly and give references and most
importantly metron's usage of them?
Indeed a video recording or a webinar would be much more useful so that
there can be some discussions.
Regards,
Moshe
--
Jon
Sent from my mobile device
Re: a proper entry level documentation
Posted by Nick Allen <ni...@nickallen.org>.
Feel free to send an email to the list describing what you are
experiencing. Also send the output of running the script at
metron-deployment/scripts/platform-info.sh.
On Mar 5, 2017 3:49 PM, "Earl Hinkle" <ea...@thehinkles.org> wrote:
> I have tried several times to get it up and running with no success.
>
> Sent with AquaMail for Android
> http://www.aqua-mail.com
>
> On March 5, 2017 7:39:37 AM Nick Allen <ni...@nickallen.org> wrote:
>
>> Khurram - Have you tried getting the "Quick Dev" environment up and
>> running?
>>
>> https://github.com/apache/incubator-metron/tree/master/
>> metron-deployment/vagrant/quick-dev-platform
>>
>>
>>
>>
>> On Sun, Mar 5, 2017 at 4:39 AM, Khurram Ahmed <kh...@gmail.com>
>> wrote:
>>
>>> I agree. The barrier to entry for new contributors is prohibitive due to
>>> lack of accurate documentation for setting up dev environment.
>>>
>>> On Mar 5, 2017 2:23 PM, "moshe jarusalem" <tu...@gmail.com> wrote:
>>>
>>>> Hi All,
>>>> I have been trying to get involved with the metron project for a while.
>>>> I have tried release 0.3.0 and 0.3.1. Unfortunately, it is very difficult
>>>> to set up a running environment and understand the picture how it works.
>>>> There are many technologies prerequisite such as Hadoop, nifi, storm, Kafka
>>>> etc. I assume most of you guys are expert at these technologies so you
>>>> solve your problems.
>>>> I have also browsed the user's list and there are many people
>>>> struggling to set up a proper environment.
>>>>
>>>> Would you guys make a document describing step by step setting up a
>>>> development environment with a specific release (0.3.1 as its the latest) ?
>>>> Can you describe each technology shortly and give references and most
>>>> importantly metron's usage of them?
>>>> Indeed a video recording or a webinar would be much more useful so that
>>>> there can be some discussions.
>>>>
>>>> Regards,
>>>> Moshe
>>>>
>>>
>>
Re: a proper entry level documentation
Posted by Earl Hinkle <ea...@thehinkles.org>.
I have tried several times to get it up and running with no success.
Sent with AquaMail for Android
http://www.aqua-mail.com
On March 5, 2017 7:39:37 AM Nick Allen <ni...@nickallen.org> wrote:
Khurram - Have you tried getting the "Quick Dev" environment up and running?
https://github.com/apache/incubator-metron/tree/master/metron-deployment/vagrant/quick-dev-platform
On Sun, Mar 5, 2017 at 4:39 AM, Khurram Ahmed <kh...@gmail.com>> wrote:
I agree. The barrier to entry for new contributors is prohibitive due to lack of accurate documentation for setting up dev environment.
On Mar 5, 2017 2:23 PM, "moshe jarusalem" <tu...@gmail.com>> wrote:
Hi All,
I have been trying to get involved with the metron project for a while. I have tried release 0.3.0 and 0.3.1. Unfortunately, it is very difficult to set up a running environment and understand the picture how it works. There are many technologies prerequisite such as Hadoop, nifi, storm, Kafka etc. I assume most of you guys are expert at these technologies so you solve your problems.
I have also browsed the user's list and there are many people struggling to set up a proper environment.
Would you guys make a document describing step by step setting up a development environment with a specific release (0.3.1 as its the latest) ? Can you describe each technology shortly and give references and most importantly metron's usage of them?
Indeed a video recording or a webinar would be much more useful so that there can be some discussions.
Regards,
Moshe
Re: a proper entry level documentation
Posted by Nick Allen <ni...@nickallen.org>.
Khurram - Have you tried getting the "Quick Dev" environment up and running?
https://github.com/apache/incubator-metron/tree/master/metron-deployment/vagrant/quick-dev-platform
On Sun, Mar 5, 2017 at 4:39 AM, Khurram Ahmed <kh...@gmail.com>
wrote:
> I agree. The barrier to entry for new contributors is prohibitive due to
> lack of accurate documentation for setting up dev environment.
>
> On Mar 5, 2017 2:23 PM, "moshe jarusalem" <tu...@gmail.com> wrote:
>
>> Hi All,
>> I have been trying to get involved with the metron project for a while. I
>> have tried release 0.3.0 and 0.3.1. Unfortunately, it is very difficult to
>> set up a running environment and understand the picture how it works. There
>> are many technologies prerequisite such as Hadoop, nifi, storm, Kafka etc. I
>> assume most of you guys are expert at these technologies so you solve your
>> problems.
>> I have also browsed the user's list and there are many people struggling
>> to set up a proper environment.
>>
>> Would you guys make a document describing step by step setting up a
>> development environment with a specific release (0.3.1 as its the latest) ?
>> Can you describe each technology shortly and give references and most
>> importantly metron's usage of them?
>> Indeed a video recording or a webinar would be much more useful so that
>> there can be some discussions.
>>
>> Regards,
>> Moshe
>>
>
Re: a proper entry level documentation
Posted by Khurram Ahmed <kh...@gmail.com>.
I agree. The barrier to entry for new contributors is prohibitive due to
lack of accurate documentation for setting up dev environment.
On Mar 5, 2017 2:23 PM, "moshe jarusalem" <tu...@gmail.com> wrote:
> Hi All,
> I have been trying to get involved with the metron project for a while. I
> have tried release 0.3.0 and 0.3.1. Unfortunately, it is very difficult to
> set up a running environment and understand the picture how it works. There
> are many technologies prerequisite such as Hadoop, nifi, storm, Kafka etc. I
> assume most of you guys are expert at these technologies so you solve your
> problems.
> I have also browsed the user's list and there are many people struggling
> to set up a proper environment.
>
> Would you guys make a document describing step by step setting up a
> development environment with a specific release (0.3.1 as its the latest) ?
> Can you describe each technology shortly and give references and most
> importantly metron's usage of them?
> Indeed a video recording or a webinar would be much more useful so that
> there can be some discussions.
>
> Regards,
> Moshe
>