You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@eagle.apache.org by Edward Zhang <yo...@apache.org> on 2019/03/18 18:57:48 UTC
Re: Apache Eagle GUI, operation issues ; unable to define Policy
I think @hao <ha...@apache.org> has set up eagle in local environment well.
The problem is like you said, Eagle local mode has some issues because
originally we want to run everything in one box including storm, zookeeper,
which has made things difficult to manage. There is a discussion about
simplifying the set up, but that really needs some time and you may not
want to wait.
But if @hao <ha...@apache.org> can quickly identify the issue, probably you
can move forward.
Thanks
Edward
On Mon, Mar 18, 2019 at 11:47 AM tusharintech@gmail.com <
tusharintech@gmail.com> wrote:
>
> Thank you very much for the help at the Apache Eagle Forum (
> https://lists.apache.org/list.html?dev@eagle.apache.org) from Jayesh. .
> I am trying to do Proof of concept with Eagle and encountering constant
> issues.
>
> I can successfully compile Eagle 0.50 and Launch Eagle (
> bin/eagle-server.sh start). . I have new HDP 2.6 cluster running with
> normal Ambari based GUI.
> I am following http://eagle.apache.org/docs/latest/using-eagle/ as my
> reference .
> The basic Apache Eagle GUI works. However I have several major
> issues.
>
> 1) I can not define the sites ( like Sandbox), define
> applications ( HDFS Audit log, Hadoop Metrics Monitor
> (HADOOP_METRIC_MONITOR) but can not define policy . “Apply” button
> remains greyed out.
> 2) Some of the web pages work intermittently, I can not navigate
> but can circumvent intermittently (http://<IP>:9090/#/login and
> http://<ip:9090/ /#//integration/siteList) vis URL .
> 3) The moment I start the Eagle server. The server logs keeps
> getting flooded with Zookeper connection refused errors. I am not running
> any firewall.as belwo.
>
>
> I wonder who else have been user of the Apache Eagle besides eBay,
> PayPal, DataGuise ? Few individuals I communicated told me that they gave
> up due to lack of user participation at the Eagle forum and so lack of
> support. Not sure if that is an accurate assesment.
> Tushar Shah
> ts7237@att.com
>
>
Re: Apache Eagle GUI, operation issues ; unable to define Policy
Posted by Hao Chen <ha...@apache.org>.
Hi Tushar,
Sorry for delayed response. Thanks very much for using eagle and I are very
glad to support you.
How do you starting eagle locally? If starting from shell scripts we
provided, then you need configure correctly about dependencies like
zookeeper/kafka, etc, as eagle standalone model is based on storm
standalone execution model, especially like zookeeper if not correctly
setup then there will be zookeeper connection logging.
Do you think still have a problem about installing Site? I think it's quite
easy for this step to single add a unique *siteId*/*siteName*/.. attributes.
What's your problem to install Application? Have you replaced the
*{SITE_ID}* with correct *siteId* registered in the previous step?
What's the issue for kafka publishing and collection? What's your data
sources?
- Hao
On Wed, Mar 20, 2019 at 11:07 AM tusharintech@gmail.com <
tusharintech@gmail.com> wrote:
>
>
> On 2019/03/18 20:18:20, tusharintech@gmail.com <tu...@gmail.com>
> wrote:
> >
> >
> > On 2019/03/18 18:57:48, Edward Zhang <yo...@apache.org> wrote:
> > > I think @hao <ha...@apache.org> has set up eagle in local environment
> well.
> > > The problem is like you said, Eagle local mode has some issues because
> > > originally we want to run everything in one box including storm,
> zookeeper,
> > > which has made things difficult to manage. There is a discussion about
> > > simplifying the set up, but that really needs some time and you may not
> > > want to wait.
> > >
> > > But if @hao <ha...@apache.org> can quickly identify the issue, probably
> you
> > > can move forward.
> > >
> > > Thanks
> > > Edward
> > >
> > > On Mon, Mar 18, 2019 at 11:47 AM tusharintech@gmail.com <
> > > tusharintech@gmail.com> wrote:
> > >
> > > >
> > > > Thank you very much for the help at the Apache Eagle Forum (
> > > > https://lists.apache.org/list.html?dev@eagle.apache.org) from
> Jayesh. .
> > > > I am trying to do Proof of concept with Eagle and encountering
> constant
> > > > issues.
> > > >
> > > > I can successfully compile Eagle 0.50 and Launch Eagle (
> > > > bin/eagle-server.sh start). . I have new HDP 2.6 cluster running
> with
> > > > normal Ambari based GUI.
> > > > I am following http://eagle.apache.org/docs/latest/using-eagle/ as
> my
> > > > reference .
> > > > The basic Apache Eagle GUI works. However I have several major
> > > > issues.
> > > >
> > > > 1) I can not define the sites ( like Sandbox), define
> > > > applications ( HDFS Audit log, Hadoop Metrics Monitor
> > > > (HADOOP_METRIC_MONITOR) but can not define policy . “Apply” button
> > > > remains greyed out.
> > > > 2) Some of the web pages work intermittently, I can not
> navigate
> > > > but can circumvent intermittently (http://<IP>:9090/#/login and
> > > > http://<ip:9090/ /#//integration/siteList) vis URL .
> > > > 3) The moment I start the Eagle server. The server logs keeps
> > > > getting flooded with Zookeper connection refused errors. I am not
> running
> > > > any firewall.as belwo.
> > > >
> > > >
> > > > I wonder who else have been user of the Apache Eagle besides eBay,
> > > > PayPal, DataGuise ? Few individuals I communicated told me that
> they gave
> > > > up due to lack of user participation at the Eagle forum and so
> lack of
> > > > support. Not sure if that is an accurate assesment.
> > > > Tushar Shah
> > > > ts7237@att.com
> > > >
> > > >
> > > Edward ,
> > Thank you for such a prompt reply. I am running Hortonworks 2.6 in a
> cluster of 10 nodes. I installed Eagle on node 1 whichis running the
> Ambari. Zookeper is rnning on node 2. I changed Eagle .cnf to point to my
> zookeper at node 2. Stll same problem after eagle restart. Orignal
> eage.conf
> > eagle.conf
> > Tushar(3:20:30 PM): zookeeper {
> > zkQuorum = "localhost:2181"
> > zkSessionTimeoutMs : 15000
> > zkRetryTimes : 3
> > zkRetryInterval : 20000
> > }
> >
> > Changed from from zkQuorum = "localhost:2181" to
> 101-02-08-node2.internal.bigtdata.io:2181.
> > port 2181 is listening but same problem.
> >
> > You may share your eagle .conf file. I tried sandbox at VM aswell,
> There I can not even bring up the Eagle GUI.
> >
> > Thanks again
> > Tushar Shah
> > ts7237@att.com
> >
> >
> Hi Edward
> Thanks again for your reply. Wonder if you or Hao plan to work on
> issue with Eagle in loal mode as you mentioned above.
>
> Meanwhile I, could circumvent the problem of the log flooding with
> connection refused errors for the zookeeper by specifying the the
> zookeeper at the actual node ( and not the local host).
>
> However I still have issue in defining the application. After defining
> the site m defining apllication, I am try to install using the following
> stes at the Web interface.
> How to Install
> 1. Create two kafka topics: hdfs_audit_log_{SITE_ID},
> hdfs_audit_log_enriched_{SITE_ID}
> 2. Setup a log collecting tool you like to stream audit log into
> topic hdfs_audit_log_{SITE_ID}
> 3. Click "Install" button and edit configurations in general and
> advanced lists according to your requirements
> 4. Check the new generated stream
> HDFS_AUDIT_LOG_ENRICHED_STREAM_{SITE_ID} at Alert -> Streams
>
> is it necessary for Eagle to have Kafka topics and collection working
> before one can define a policy ? At present I have issue in making kafka
> publishing and collection work.
>
> Tushar Shah
> ts7237@att.com
> 732 420 3139
>
>
>
>
Re: Apache Eagle GUI, operation issues ; unable to define Policy
Posted by tu...@gmail.com,
tu...@gmail.com.
On 2019/03/18 20:18:20, tusharintech@gmail.com <tu...@gmail.com> wrote:
>
>
> On 2019/03/18 18:57:48, Edward Zhang <yo...@apache.org> wrote:
> > I think @hao <ha...@apache.org> has set up eagle in local environment well.
> > The problem is like you said, Eagle local mode has some issues because
> > originally we want to run everything in one box including storm, zookeeper,
> > which has made things difficult to manage. There is a discussion about
> > simplifying the set up, but that really needs some time and you may not
> > want to wait.
> >
> > But if @hao <ha...@apache.org> can quickly identify the issue, probably you
> > can move forward.
> >
> > Thanks
> > Edward
> >
> > On Mon, Mar 18, 2019 at 11:47 AM tusharintech@gmail.com <
> > tusharintech@gmail.com> wrote:
> >
> > >
> > > Thank you very much for the help at the Apache Eagle Forum (
> > > https://lists.apache.org/list.html?dev@eagle.apache.org) from Jayesh. .
> > > I am trying to do Proof of concept with Eagle and encountering constant
> > > issues.
> > >
> > > I can successfully compile Eagle 0.50 and Launch Eagle (
> > > bin/eagle-server.sh start). . I have new HDP 2.6 cluster running with
> > > normal Ambari based GUI.
> > > I am following http://eagle.apache.org/docs/latest/using-eagle/ as my
> > > reference .
> > > The basic Apache Eagle GUI works. However I have several major
> > > issues.
> > >
> > > 1) I can not define the sites ( like Sandbox), define
> > > applications ( HDFS Audit log, Hadoop Metrics Monitor
> > > (HADOOP_METRIC_MONITOR) but can not define policy . “Apply” button
> > > remains greyed out.
> > > 2) Some of the web pages work intermittently, I can not navigate
> > > but can circumvent intermittently (http://<IP>:9090/#/login and
> > > http://<ip:9090/ /#//integration/siteList) vis URL .
> > > 3) The moment I start the Eagle server. The server logs keeps
> > > getting flooded with Zookeper connection refused errors. I am not running
> > > any firewall.as belwo.
> > >
> > >
> > > I wonder who else have been user of the Apache Eagle besides eBay,
> > > PayPal, DataGuise ? Few individuals I communicated told me that they gave
> > > up due to lack of user participation at the Eagle forum and so lack of
> > > support. Not sure if that is an accurate assesment.
> > > Tushar Shah
> > > ts7237@att.com
> > >
> > >
> > Edward ,
> Thank you for such a prompt reply. I am running Hortonworks 2.6 in a cluster of 10 nodes. I installed Eagle on node 1 whichis running the Ambari. Zookeper is rnning on node 2. I changed Eagle .cnf to point to my zookeper at node 2. Stll same problem after eagle restart. Orignal eage.conf
> eagle.conf
> Tushar(3:20:30 PM): zookeeper {
> zkQuorum = "localhost:2181"
> zkSessionTimeoutMs : 15000
> zkRetryTimes : 3
> zkRetryInterval : 20000
> }
>
> Changed from from zkQuorum = "localhost:2181" to 101-02-08-node2.internal.bigtdata.io:2181.
> port 2181 is listening but same problem.
>
> You may share your eagle .conf file. I tried sandbox at VM aswell, There I can not even bring up the Eagle GUI.
>
> Thanks again
> Tushar Shah
> ts7237@att.com
>
>
Hi Edward
Thanks again for your reply. Wonder if you or Hao plan to work on issue with Eagle in loal mode as you mentioned above.
Meanwhile I, could circumvent the problem of the log flooding with connection refused errors for the zookeeper by specifying the the zookeeper at the actual node ( and not the local host).
However I still have issue in defining the application. After defining the site m defining apllication, I am try to install using the following stes at the Web interface.
How to Install
1. Create two kafka topics: hdfs_audit_log_{SITE_ID}, hdfs_audit_log_enriched_{SITE_ID}
2. Setup a log collecting tool you like to stream audit log into topic hdfs_audit_log_{SITE_ID}
3. Click "Install" button and edit configurations in general and advanced lists according to your requirements
4. Check the new generated stream HDFS_AUDIT_LOG_ENRICHED_STREAM_{SITE_ID} at Alert -> Streams
is it necessary for Eagle to have Kafka topics and collection working before one can define a policy ? At present I have issue in making kafka publishing and collection work.
Tushar Shah
ts7237@att.com
732 420 3139
Re: Apache Eagle GUI, operation issues ; unable to define Policy
Posted by tu...@gmail.com,
tu...@gmail.com.
On 2019/03/18 18:57:48, Edward Zhang <yo...@apache.org> wrote:
> I think @hao <ha...@apache.org> has set up eagle in local environment well.
> The problem is like you said, Eagle local mode has some issues because
> originally we want to run everything in one box including storm, zookeeper,
> which has made things difficult to manage. There is a discussion about
> simplifying the set up, but that really needs some time and you may not
> want to wait.
>
> But if @hao <ha...@apache.org> can quickly identify the issue, probably you
> can move forward.
>
> Thanks
> Edward
>
> On Mon, Mar 18, 2019 at 11:47 AM tusharintech@gmail.com <
> tusharintech@gmail.com> wrote:
>
> >
> > Thank you very much for the help at the Apache Eagle Forum (
> > https://lists.apache.org/list.html?dev@eagle.apache.org) from Jayesh. .
> > I am trying to do Proof of concept with Eagle and encountering constant
> > issues.
> >
> > I can successfully compile Eagle 0.50 and Launch Eagle (
> > bin/eagle-server.sh start). . I have new HDP 2.6 cluster running with
> > normal Ambari based GUI.
> > I am following http://eagle.apache.org/docs/latest/using-eagle/ as my
> > reference .
> > The basic Apache Eagle GUI works. However I have several major
> > issues.
> >
> > 1) I can not define the sites ( like Sandbox), define
> > applications ( HDFS Audit log, Hadoop Metrics Monitor
> > (HADOOP_METRIC_MONITOR) but can not define policy . “Apply” button
> > remains greyed out.
> > 2) Some of the web pages work intermittently, I can not navigate
> > but can circumvent intermittently (http://<IP>:9090/#/login and
> > http://<ip:9090/ /#//integration/siteList) vis URL .
> > 3) The moment I start the Eagle server. The server logs keeps
> > getting flooded with Zookeper connection refused errors. I am not running
> > any firewall.as belwo.
> >
> >
> > I wonder who else have been user of the Apache Eagle besides eBay,
> > PayPal, DataGuise ? Few individuals I communicated told me that they gave
> > up due to lack of user participation at the Eagle forum and so lack of
> > support. Not sure if that is an accurate assesment.
> > Tushar Shah
> > ts7237@att.com
> >
> >
> Edward ,
Thank you for such a prompt reply. I am running Hortonworks 2.6 in a cluster of 10 nodes. I installed Eagle on node 1 whichis running the Ambari. Zookeper is rnning on node 2. I changed Eagle .cnf to point to my zookeper at node 2. Stll same problem after eagle restart. Orignal eage.conf
eagle.conf
Tushar(3:20:30 PM): zookeeper {
zkQuorum = "localhost:2181"
zkSessionTimeoutMs : 15000
zkRetryTimes : 3
zkRetryInterval : 20000
}
Changed from from zkQuorum = "localhost:2181" to 101-02-08-node2.internal.bigtdata.io:2181.
port 2181 is listening but same problem.
You may share your eagle .conf file. I tried sandbox at VM aswell, There I can not even bring up the Eagle GUI.
Thanks again
Tushar Shah
ts7237@att.com