You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@metron.apache.org by "Yerex, Tom" <to...@ubc.ca> on 2019/11/04 21:54:19 UTC
Re-establish /apps/metron directory
Good afternoon,
During development I squashed the /apps/metron/… directory in HDFS. Would someone be able to pinpoint for me the code that builds the /apps/metron directory and sub-directories so I can fix the state of those folders without a reinstallation?
Cheers,
Tom.
Re: Re-establish /apps/metron directory
Posted by "Yerex, Tom" <to...@ubc.ca>.
Thanks Michael, this is what I was hoping for. I reckon that “intelligent compromise” is what goes into a lot of the best software solutions out there.
--Tom.
--
Tom Yerex
Cybersecurity Analyst, Information Technology
Cybersecurity | CISO Office
The University of British Columbia | Musqueam Traditional Territory
Ponderosa Office Annex A | Vancouver BC | V6T1Z2 Canada
Phone 604 822 6531
Privacy Matters @ UBC
From: Michael Miklavcic <mi...@gmail.com>
Reply-To: "user@metron.apache.org" <us...@metron.apache.org>
Date: Tuesday, November 5, 2019 at 11:05 AM
To: "user@metron.apache.org" <us...@metron.apache.org>
Subject: Re: Re-establish /apps/metron directory
Hey Tom,
Just an fyi, the Ambari install MPack code lays down a series of "configured" flag files when various bits of the system have been configured. Ambari has no concept of this natively, so the Metron community had to come up with an intelligent compromise to work around that limitation. If you look in the $METRON_HOME/config directory you should see a series of flag files that indicate what has/has not been successfully configured. This is a non-standard way to workaround issues, but if you delete 1 or more of those files, Ambari will re-attempt to perform the operation when you restart any services that depend on that particular setup. e.g. here is a list I pulled from a full dev install:
metron_elasticsearch_template_installed_flag_file
metron_enrichment_hbase_configured
metron_enrichment_hbase_coprocessor_configured
metron_enrichment_kafka_configured
metron_enrichment_maxmind_configured
metron_indexing_configured
metron_indexing_hbase_configured
metron_parsers_configured
metron_pcap_configured
metron_profiler_configured
metron_profiler_hbase_configured
metron_rest_hbase_configured
metron_zookeeper_configured
HDFS setup occurs here
https://github.com/apache/metron/blob/master/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/indexing_master.py#L76
https://github.com/apache/metron/blob/master/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/indexing_commands.py#L183
Best,
Mike Miklavcic
On Tue, Nov 5, 2019 at 10:44 AM Yerex, Tom <to...@ubc.ca> wrote:
Good morning,
I wanted to share with the group that once the top-level “/apps/metron” folder is created, the rest of the folders seem to eventually reappear on their own after a few app restarts in Ambari.
I have spun up a development instance of Metron (Centos7), to compare, so far it seems to be working well.
Cheers,
Tom.
From: Thomas Yerex <to...@ubc.ca>
Reply-To: <us...@metron.apache.org>
Date: Monday, November 4, 2019 at 1:54 PM
To: "user@metron.apache.org" <us...@metron.apache.org>
Subject: Re-establish /apps/metron directory
Good afternoon,
During development I squashed the /apps/metron/… directory in HDFS. Would someone be able to pinpoint for me the code that builds the /apps/metron directory and sub-directories so I can fix the state of those folders without a reinstallation?
Cheers,
Tom.
Re: Re-establish /apps/metron directory
Posted by Michael Miklavcic <mi...@gmail.com>.
Hey Tom,
Just an fyi, the Ambari install MPack code lays down a series of
"configured" flag files when various bits of the system have been
configured. Ambari has no concept of this natively, so the Metron community
had to come up with an intelligent compromise to work around that
limitation. If you look in the $METRON_HOME/config directory you should see
a series of flag files that indicate what has/has not been successfully
configured. This is a non-standard way to workaround issues, but if you
delete 1 or more of those files, Ambari will re-attempt to perform the
operation when you restart any services that depend on that particular
setup. e.g. here is a list I pulled from a full dev install:
metron_elasticsearch_template_installed_flag_file
metron_enrichment_hbase_configured
metron_enrichment_hbase_coprocessor_configured
metron_enrichment_kafka_configured
metron_enrichment_maxmind_configured
metron_indexing_configured
metron_indexing_hbase_configured
metron_parsers_configured
metron_pcap_configured
metron_profiler_configured
metron_profiler_hbase_configured
metron_rest_hbase_configured
metron_zookeeper_configured
HDFS setup occurs here
https://github.com/apache/metron/blob/master/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/indexing_master.py#L76
https://github.com/apache/metron/blob/master/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/indexing_commands.py#L183
Best,
Mike Miklavcic
On Tue, Nov 5, 2019 at 10:44 AM Yerex, Tom <to...@ubc.ca> wrote:
> Good morning,
>
>
>
> I wanted to share with the group that once the top-level “/apps/metron”
> folder is created, the rest of the folders seem to eventually reappear on
> their own after a few app restarts in Ambari.
>
>
>
> I have spun up a development instance of Metron (Centos7), to compare, so
> far it seems to be working well.
>
>
>
> Cheers,
>
>
>
> Tom.
>
>
>
> *From: *Thomas Yerex <to...@ubc.ca>
> *Reply-To: *<us...@metron.apache.org>
> *Date: *Monday, November 4, 2019 at 1:54 PM
> *To: *"user@metron.apache.org" <us...@metron.apache.org>
> *Subject: *Re-establish /apps/metron directory
>
>
>
> Good afternoon,
>
>
>
> During development I squashed the /apps/metron/… directory in HDFS. Would
> someone be able to pinpoint for me the code that builds the /apps/metron
> directory and sub-directories so I can fix the state of those folders
> without a reinstallation?
>
>
>
> Cheers,
>
>
>
> Tom.
>
>
>
Re: Re-establish /apps/metron directory
Posted by "Yerex, Tom" <to...@ubc.ca>.
Good morning,
I wanted to share with the group that once the top-level “/apps/metron” folder is created, the rest of the folders seem to eventually reappear on their own after a few app restarts in Ambari.
I have spun up a development instance of Metron (Centos7), to compare, so far it seems to be working well.
Cheers,
Tom.
From: Thomas Yerex <to...@ubc.ca>
Reply-To: <us...@metron.apache.org>
Date: Monday, November 4, 2019 at 1:54 PM
To: "user@metron.apache.org" <us...@metron.apache.org>
Subject: Re-establish /apps/metron directory
Good afternoon,
During development I squashed the /apps/metron/… directory in HDFS. Would someone be able to pinpoint for me the code that builds the /apps/metron directory and sub-directories so I can fix the state of those folders without a reinstallation?
Cheers,
Tom.