You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@eagle.apache.org by Colm O hEigeartaigh <co...@apache.org> on 2018/01/16 11:34:49 UTC

Unable to get 0.5.0 release working

Hi all,

I'm trying to play around a bit with Apache Eagle 0.5.0 to no avail. Here
are the problems I've run into so far:

a) The official docker image uses 0.5.0-SNAPSHOT and not the released
version.

b) Aside from the above, the official docker image uses a mix of "
server.eagle.apache.org" and "sandbox.eagle.apache.org" as the host name.
The HBase service doesn't start by default in Ambari as a result.

c) The UI seems quite buggy. On both chromium and firefox, I only see links
to "Sandbox" and "Alert" on the left hand-side. Once I click on "Alert" I
have no way of going back to see the applications. I don't see the links to
"integration" or "sites" as in the picture here:
http://eagle.apache.org/docs/latest/applications/#jmx-monitoring

d) In chromium, the button to create a new policy does not exist - I can
only see it on Firefox.

e) I'm trying to get the "Hdfs Audit Log Monitor" use-case working, but it
seems to be stuck in "Initialized".

Could someone fill me in on what the "recommended" way is to start Apache
Eagle so that I can play around with the functionality that it offers?
Clearly the docker approach is buggy. Also, what browser should be used?

Thanks,

Colm.


-- 
Colm O hEigeartaigh

Talend Community Coder
http://coders.talend.com

Re: Unable to get 0.5.0 release working

Posted by Colm O hEigeartaigh <co...@apache.org>.
OK cool. I have submitted  a few other PRs:

https://github.com/apache/eagle/pull/986
https://github.com/apache/eagle/pull/983
https://github.com/apache/eagle/pull/985

The first one is a fix for the issue reported on the dev list recently,
where you can't start the eagle-server.sh script from the same directory. I
changed it to get the actual directory name using "readlink -f". I tested
it works OK when run via "./eagle-server.sh start" and "bin/eagle-server.sh
start". The other two are checkstyle fixes for two different modules.

Colm.

On Fri, Feb 2, 2018 at 5:25 PM, Jayesh Senjaliya <ja...@apache.org> wrote:

> Ya slackpublisher has some issue and we have fixed it and also changed the
> implementation to use rest api instad of sdk.
> We will push the patch soon.
>
> Thanks
> Jayesh
>
> On Fri, Feb 2, 2018 at 9:02 AM Colm O hEigeartaigh <co...@apache.org>
> wrote:
>
> > Thanks. I've submitted an initial PR to fix the Slack
> > ClassNotFoundException issue here:
> >
> > https://issues.apache.org/jira/browse/EAGLE-879
> > https://github.com/apache/eagle/pull/984
> >
> > However, no actual Slack messages are sent. The problem is that
> > AlertSlackPublisher only sends the message if the event contains a
> > "severity" that matches the configured "severity" on the publisher.
> However
> > the event contains neither "severity" (or "message") so the message never
> > gets sent. This is for an HDFS audit log - I'm not sure if there are
> other
> > scenarios where there is a "severity" column in the event? Either that or
> > it looks like the SlackPublisher was written before the event column
> > headers were changed.
> >
> > Colm.
> >
> > On Fri, Feb 2, 2018 at 12:54 AM, Jayesh Senjaliya <ja...@apache.org>
> > wrote:
> >
> >> resolved those tickets now.
> >>
> >> I have asked the developer to rebase the PR #941, if he doesnt get to it
> >> by
> >> this week, i will take care of, its long pending one.
> >> Thanks for verifying though.
> >>
> >> - Jayesh
> >>
> >> On Thu, Feb 1, 2018 at 8:56 AM, Colm O hEigeartaigh <
> coheigea@apache.org>
> >> wrote:
> >>
> >> > Thanks Jayesh. I have two more PRs awaiting review:
> >> >
> >> > https://github.com/apache/eagle/pull/981
> >> > https://github.com/apache/eagle/pull/982
> >> >
> >> > Thanks for the JIRA privileges, I can now assign issues to me + change
> >> the
> >> > versions. However, I can't "resolve" JIRAs that weren't reported by me
> >> > which is annoying. These 3 JIRAs should be resolved as they are
> already
> >> > merged:
> >> >
> >> > https://issues.apache.org/jira/browse/EAGLE-445
> >> > https://issues.apache.org/jira/browse/EAGLE-476
> >> > https://issues.apache.org/jira/browse/EAGLE-331
> >> >
> >> > In addition, I tested the fix for the Email issue and it works
> >> correctly.
> >> > The PR (https://github.com/apache/eagle/pull/941) just needs to have
> >> the
> >> > extra commits stripped away - I attached a version of the patch on the
> >> > JIRA.
> >> >
> >> > Colm.
> >> >
> >> > On Wed, Jan 31, 2018 at 10:08 PM, Jayesh Senjaliya <jaysen@apache.org
> >
> >> > wrote:
> >> >
> >> > > Thanks for the PRs. I have merged them.
> >> > >
> >> > > welcome to the developer community Colm. I have also added you to
> jira
> >> > > project so can assign the tasks to yourself.
> >> > >
> >> > > lets create ticket to fix the dedup functionality, I m actually
> >> surprised
> >> > > we havent hit this issue yet. we do use multiple publishers but
> >> someone
> >> > can
> >> > > verify this.
> >> > >
> >> > > Thanks
> >> > > Jayesh
> >> > >
> >> > >
> >> > >
> >> > > On Wed, Jan 31, 2018 at 9:25 AM, Colm O hEigeartaigh <
> >> > coheigea@apache.org>
> >> > > wrote:
> >> > >
> >> > >> Thanks Jayesh. I've created a JIRA here for some admin work for
> some
> >> > >> issues
> >> > >> that were incorrectly flagged as "fix for" 0.5.1/0.6.0:
> >> > >>
> >> > >> https://issues.apache.org/jira/browse/EAGLE-1076
> >> > >>
> >> > >> I've submitted the following (fairly trivial) pull requests. Could
> I
> >> ask
> >> > >> that you or one of the other committers review?
> >> > >>
> >> > >> https://github.com/apache/eagle/pull/978
> >> > >> https://github.com/apache/eagle/pull/979
> >> > >> https://github.com/apache/eagle/pull/980
> >> > >>
> >> > >> It would be good to try to inject some energy into the project. We
> >> need
> >> > >> more than one active committer though.
> >> > >>
> >> > >> Just in terms of the Alert Deduplication issue. The
> >> DefaultDeDuplicator
> >> > >> works per "output" in the policy rule. So if you have more than one
> >> > >> AlertPublisher, I think it is guaranteed to only publish to one of
> >> them.
> >> > >> Instead, surely it would make more sense to work per publisher?
> >> > >>
> >> > >> Colm.
> >> > >>
> >> > >> On Tue, Jan 30, 2018 at 10:39 PM, Jayesh Senjaliya <
> >> jaysen@apache.org>
> >> > >> wrote:
> >> > >>
> >> > >> > Hi Colm,
> >> > >> >
> >> > >> > appreciate your suggestions/ efforts in looking into this
> project,
> >> > >> > putting my comments inline...
> >> > >> >
> >> > >> > a) There is already a JIRA to bump the version here, although the
> >> PR
> >> > >> does
> >> > >> > not apply as it is too old: https://issues.apache.org/
> >> > >> > jira/browse/EAGLE-1025
> >> > >> > .
> >> > >> > I can submit a new PR, but should the version be 0.6.0 or 0.5.1?
> >> > >> >
> >> > >> > *since there are still minor issues, i would say, we put up 0.5.1
> >> as
> >> > >> next
> >> > >> > version. I've updated/rebased the PR (
> >> > >> > https://github.com/apache/eagle/pull/936
> >> > >> > <https://github.com/apache/eagle/pull/936> )*
> >> > >> >
> >> > >> >
> >> > >> > b) The issues that are "resolved" for the 0.5.1 release in JIRA
> are
> >> > >> > actually already fixed in 0.5.0, so they should be updated (
> >> > >> > https://issues.apache.org/jira/projects/EAGLE/versions/12341128
> ).
> >> > >> However,
> >> > >> > the following two issues are resolved even though they are not
> >> merged
> >> > to
> >> > >> > master?
> >> > >> >   https://issues.apache.org/jira/browse/EAGLE-1051 .  - * this
> was
> >> > >> pending
> >> > >> > from developer;s response but i think this is reviewed, so I have
> >> > merged
> >> > >> > it.*
> >> > >> >   https://issues.apache.org/jira/browse/EAGLE-1068 .  - * this
> is
> >> > >> reopened
> >> > >> > now. I dont think this is done yet. Also this is big change.*
> >> > >> >
> >> > >> >
> >> > >> > Like I said I can submit PRs but I'm not convinced there is any
> >> > >> activity on
> >> > >> > the project. Where are the rest of the committers?
> >> > >> >
> >> > >> > *let me give you some ocontext on this. so there were lot of
> >> > development
> >> > >> > happened during last releases, and most of applications that were
> >> > added
> >> > >> are
> >> > >> > being used in production at multiple enterprise companies, but we
> >> are
> >> > >> out
> >> > >> > of ideas on new apps, so at this point we are only focusing on
> bug
> >> > fixes
> >> > >> > and tech upgrades until we get some new ideas to brainstorm and
> >> add.*
> >> > >> >
> >> > >> > *I think current community's thinking is based on their own
> >> industries
> >> > >> > use-cases, but there is definitely room for new features and
> >> > integration
> >> > >> > with other monitoring and security components like grafana and
> >> > rangers.*
> >> > >> >
> >> > >> >
> >> > >> > *Thanks,*
> >> > >> > *Jayesh*
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> > On Tue, Jan 30, 2018 at 8:11 AM, Colm O hEigeartaigh <
> >> > >> coheigea@apache.org>
> >> > >> > wrote:
> >> > >> >
> >> > >> > > Hi Jayesh,
> >> > >> > >
> >> > >> > > Dev suggestions:
> >> > >> > >
> >> > >> > > a) There is already a JIRA to bump the version here, although
> >> the PR
> >> > >> does
> >> > >> > > not apply as it is too old: https://issues.apache.org/
> >> > >> > > jira/browse/EAGLE-1025.
> >> > >> > > I can submit a new PR, but should the version be 0.6.0 or
> 0.5.1?
> >> > >> > > b) The issues that are "resolved" for the 0.5.1 release in JIRA
> >> are
> >> > >> > > actually already fixed in 0.5.0, so they should be updated (
> >> > >> > > https://issues.apache.org/jira/projects/EAGLE/versions/
> 12341128
> >> ).
> >> > >> > However,
> >> > >> > > the following two issues are resolved even though they are not
> >> > merged
> >> > >> to
> >> > >> > > master?
> >> > >> > >   https://issues.apache.org/jira/browse/EAGLE-1051
> >> > >> > >   https://issues.apache.org/jira/browse/EAGLE-1068
> >> > >> > >
> >> > >> > > Like I said I can submit PRs but I'm not convinced there is any
> >> > >> activity
> >> > >> > on
> >> > >> > > the project. Where are the rest of the committers?
> >> > >> > >
> >> > >> > > Multiple Publisher issue:
> >> > >> > >
> >> > >> > > If I assign two publishers for one policy, the alert only goes
> to
> >> > the
> >> > >> > first
> >> > >> > > policy. In the logs I see:
> >> > >> > >
> >> > >> > > 2018-01-30T15:52:45.835+0000 o.a.e.a.e.p.d.DefaultDeduplicator
> >> > [INFO]
> >> > >> > > Alert
> >> > >> > > event is skipped because it's duplicated: Alert {site=sandbox,
> >> > >> > > stream=eagle_output,timestamp=2018-01-30
> >> > >> > > 00:00:11,300,data={securityZone=NA, dst=null,
> sensitivityType=NA,
> >> > >> > > src=/apps/hbase/data/archive/data/default/ambarismoketest,
> >> > >> allowed=true,
> >> > >> > > host=172.22.7.129, cmd=listStatus, user=SOMETHING7.COM,
> >> > >> > > timestamp=1517270411300}, policyId=test,
> >> > >> > > createdBy=alertBolt3-evaluator_stage1, metaVersion=null}
> >> > >> > >
> >> > >> > > It looks like this deduplicator is not working properly, as I'm
> >> > >> guessing
> >> > >> > it
> >> > >> > > should only be used to de-duplicate events for a single
> >> publisher?
> >> > >> > >
> >> > >> > > Incognito mode: Already tried it but with the same result.
> Could
> >> I
> >> > ask
> >> > >> > you
> >> > >> > > to try the docker image to see if the UI is working correctly
> for
> >> > you
> >> > >> > > there?
> >> > >> > >
> >> > >> > > Colm.
> >> > >> > >
> >> > >> > > On Mon, Jan 29, 2018 at 6:46 PM, Jayesh Senjaliya <
> >> > jaysen@apache.org>
> >> > >> > > wrote:
> >> > >> > >
> >> > >> > > > Hi Colm,
> >> > >> > > >
> >> > >> > > > Thanks for the list of dev suggestions, I think we should
> take
> >> > care
> >> > >> of
> >> > >> > > > those. even better if you can provide PR with the changes or
> at
> >> > >> keast
> >> > >> > can
> >> > >> > > > you please create a ticket so we can track it?
> >> > >> > > >
> >> > >> > > > for other issues.
> >> > >> > > >
> >> > >> > > > - I dont have any issue with multiple publisher, but if there
> >> is
> >> > any
> >> > >> > > error
> >> > >> > > > updating the publisher info in storm topology, i might try
> >> > >> restarting
> >> > >> > the
> >> > >> > > > topology and see if that works.
> >> > >> > > > - for us, chrome works as fine as firefox.  can u try
> incognito
> >> > >> mode?
> >> > >> > > just
> >> > >> > > > to be sure to have clean cache?
> >> > >> > > >
> >> > >> > > > Thanks
> >> > >> > > > Jayesh
> >> > >> > > >
> >> > >> > > >
> >> > >> > > > On Thu, Jan 25, 2018 at 4:19 AM, Colm O hEigeartaigh <
> >> > >> > > coheigea@apache.org>
> >> > >> > > > wrote:
> >> > >> > > >
> >> > >> > > > > Thanks again for your feedback. Jayesh, adding
> >> > >> AlertEagleStorePlugin
> >> > >> > > did
> >> > >> > > > > the trick, I can now see alerts in the UI, thanks! By the
> >> way, I
> >> > >> > can't
> >> > >> > > > > configure two Alert Publishers, or else the Alert
> >> DeDuplicator
> >> > >> bins
> >> > >> > the
> >> > >> > > > > alert. Is this a known issue?
> >> > >> > > > >
> >> > >> > > > > Could I ask which browser people are using with the UI?
> There
> >> > >> appears
> >> > >> > > to
> >> > >> > > > be
> >> > >> > > > > a  bug with Chromium where it doesn't list the pages under
> >> > >> > Auth.isAdmin
> >> > >> > > > > even though I am logged on as an administrator. It works OK
> >> in
> >> > >> > Firefox.
> >> > >> > > > > Even with Firefox though, I only see a limited number of
> >> links
> >> > in
> >> > >> the
> >> > >> > > > > left-hand column - I can't get back to the "integration"
> >> page.
> >> > Can
> >> > >> > > > someone
> >> > >> > > > > else confirm this please?
> >> > >> > > > >
> >> > >> > > > > Could I suggest the devs do some basic house-keeping tasks:
> >> > >> > > > >
> >> > >> > > > > a) "Release" version 0.5.0 in JIRA (it's still listed as
> >> > >> > "unreleased").
> >> > >> > > > > b) Figure out whether the next version will be 0.5.1 or
> 0.6.0
> >> > and
> >> > >> > > update
> >> > >> > > > > the versions on Master accordingly with 0.5.1-SNAPSHOT or
> >> > >> > > 0.6.0-SNAPSHOT.
> >> > >> > > > > There are some issues marked here as resolved for 0.5.1 -
> >> > >> > > > >
> >> https://issues.apache.org/jira/projects/EAGLE/versions/12341128
> >> > ),
> >> > >> > > > however
> >> > >> > > > > I
> >> > >> > > > > don't see a branch for 0.5.x?
> >> > >> > > > >
> >> > >> > > > > Colm.
> >> > >> > > > >
> >> > >> > > > > On Thu, Jan 25, 2018 at 8:16 AM, Jayesh Senjaliya <
> >> > >> jaysen@apache.org
> >> > >> > >
> >> > >> > > > > wrote:
> >> > >> > > > >
> >> > >> > > > > > Hi,
> >> > >> > > > > >
> >> > >> > > > > > we do use eagle 0.5 in production although we dont use
> all
> >> the
> >> > >> > > > available
> >> > >> > > > > > hadoop applications.
> >> > >> > > > > >
> >> > >> > > > > > EAGLE-968 <https://issues.apache.org/
> jira/browse/EAGLE-968
> >> >
> >> > is
> >> > >> a
> >> > >> > fix
> >> > >> > > > for
> >> > >> > > > > > email issue we found while our testing. should be merged
> >> soon
> >> > >> > after a
> >> > >> > > > > > rebase.
> >> > >> > > > > >
> >> > >> > > > > > @Colm, did you tried adding storage publisher
> >> > >> > > (AlertEagleStorePlugin)?
> >> > >> > > > to
> >> > >> > > > > > see alerts on UI ?
> >> > >> > > > > >
> >> > >> > > > > > Thanks
> >> > >> > > > > > Jayesh
> >> > >> > > > > >
> >> > >> > > > > >
> >> > >> > > > > >
> >> > >> > > > > >
> >> > >> > > > > >
> >> > >> > > > > >
> >> > >> > > > > > On Wed, Jan 24, 2018 at 7:08 PM, Edward Zhang <
> >> > >> > > yonzhang2012@gmail.com>
> >> > >> > > > > > wrote:
> >> > >> > > > > >
> >> > >> > > > > >> Eagle 0.5 was deployed in production as far as I know,
> >> but it
> >> > >> may
> >> > >> > > not
> >> > >> > > > be
> >> > >> > > > > >> exact the current version in master branch.
> >> > >> > > > > >>
> >> > >> > > > > >> Thanks for your investigation, seems there is still some
> >> bug
> >> > in
> >> > >> > 0.5,
> >> > >> > > > but
> >> > >> > > > > >> this particular issue seems is due to dependent
> components
> >> > >> version
> >> > >> > > > > conflict.
> >> > >> > > > > >>
> >> > >> > > > > >> @Jayesh is this Jira ready for merge to master?
> >> > >> > > https://issues.apache
> >> > >> > > > .
> >> > >> > > > > >> org/jira/browse/EAGLE-968
> >> > >> > > > > >>
> >> > >> > > > > >>
> >> > >> > > > > >> Thanks
> >> > >> > > > > >> Edward
> >> > >> > > > > >>
> >> > >> > > > > >> On Tue, Jan 23, 2018 at 5:10 AM, Colm O hEigeartaigh <
> >> > >> > > > > coheigea@apache.org
> >> > >> > > > > >> > wrote:
> >> > >> > > > > >>
> >> > >> > > > > >>> OK I've made some more progress. I wasn't seeing any
> >> email
> >> > >> alerts
> >> > >> > > due
> >> > >> > > > > to
> >> > >> > > > > >>> https://issues.apache.org/jira/browse/EAGLE-968. Once
> I
> >> > >> > configure
> >> > >> > > a
> >> > >> > > > > >>> Kafka
> >> > >> > > > > >>> alert, I can see the alerts flowing into my topic. It's
> >> > still
> >> > >> not
> >> > >> > > > clear
> >> > >> > > > > >>> to
> >> > >> > > > > >>> me however where the policy "output" is going. I also
> >> don't
> >> > >> see
> >> > >> > any
> >> > >> > > > > >>> alerts
> >> > >> > > > > >>> in the UI window.
> >> > >> > > > > >>>
> >> > >> > > > > >>> Could I ask what the status of the project is in
> general?
> >> > >> There
> >> > >> > > have
> >> > >> > > > > been
> >> > >> > > > > >>> no commits to master since November, so I'm not sure if
> >> > there
> >> > >> is
> >> > >> > > any
> >> > >> > > > > >>> point
> >> > >> > > > > >>> in submitting Pull Requests for outstanding bugs? Are
> >> recent
> >> > >> > > versions
> >> > >> > > > > of
> >> > >> > > > > >>> Apache Eagle used in production?
> >> > >> > > > > >>>
> >> > >> > > > > >>> Colm.
> >> > >> > > > > >>>
> >> > >> > > > > >>> On Mon, Jan 22, 2018 at 1:07 PM, Colm O hEigeartaigh <
> >> > >> > > > > >>> coheigea@apache.org>
> >> > >> > > > > >>> wrote:
> >> > >> > > > > >>>
> >> > >> > > > > >>> >
> >> > >> > > > > >>> > I've done that but I'm not seeing any alerts, which
> is
> >> > why I
> >> > >> > want
> >> > >> > > > to
> >> > >> > > > > >>> find
> >> > >> > > > > >>> > out what the "output" of a policy is and where I can
> >> check
> >> > >> > this.
> >> > >> > > > > >>> >
> >> > >> > > > > >>> > Colm.
> >> > >> > > > > >>> >
> >> > >> > > > > >>> > On Mon, Jan 22, 2018 at 1:05 PM, SUDHA JENSLIN <
> >> > >> > > sjenslin@gmail.com
> >> > >> > > > >
> >> > >> > > > > >>> wrote:
> >> > >> > > > > >>> >
> >> > >> > > > > >>> >> Create and add a publisher to see the output.
> >> > >> > > > > >>> >>
> >> > >> > > > > >>> >>
> >> > >> > > > > >>> >>
> >> > >> > > > > >>> >> Regards,
> >> > >> > > > > >>> >> Sudha jenslin
> >> > >> > > > > >>> >>
> >> > >> > > > > >>> >> On Jan 22, 2018 6:31 PM, "Colm O hEigeartaigh" <
> >> > >> > > > coheigea@apache.org
> >> > >> > > > > >
> >> > >> > > > > >>> >> wrote:
> >> > >> > > > > >>> >>
> >> > >> > > > > >>> >> Thanks - the error was due to a problem running
> Storm
> >> > with
> >> > >> > Java
> >> > >> > > > 1.8.
> >> > >> > > > > >>> I've
> >> > >> > > > > >>> >> abandoned the docker image for now, and I'm trying
> to
> >> get
> >> > >> it
> >> > >> > > > working
> >> > >> > > > > >>> >> locally.
> >> > >> > > > > >>> >>
> >> > >> > > > > >>> >> There are two things I'm not clear on currently, if
> >> > someone
> >> > >> > > could
> >> > >> > > > > >>> fill me
> >> > >> > > > > >>> >> in:
> >> > >> > > > > >>> >>
> >> > >> > > > > >>> >> a) For the  'Hdfs Audit Log Monitor' application,
> the
> >> > Kafka
> >> > >> > > > Consumer
> >> > >> > > > > >>> Topic
> >> > >> > > > > >>> >> is 'hdfs_audit_log_sandbox'. Under 'Kafka Topic for
> >> > >> Auditlog
> >> > >> > > Event
> >> > >> > > > > >>> Sink'
> >> > >> > > > > >>> >> it
> >> > >> > > > > >>> >> also specifies 'hdfs_audit_event_sandbox'. However
> the
> >> > >> > > > documentation
> >> > >> > > > > >>> for
> >> > >> > > > > >>> >> the application mentions
> >> 'hdfs_audit_log_enriched_sandb
> >> > >> ox'?
> >> > >> > > > > >>> >>
> >> > >> > > > > >>> >> When I click on "STREAMS", the
> >> > >> "HDFS_AUDIT_LOG_ENRICHED_STREA
> >> > >> > > > > >>> M_SANDBOX"
> >> > >> > > > > >>> >> uses the topic "hdfs_audit_event_sandbox". And
> indeed
> >> > when
> >> > >> I
> >> > >> > run
> >> > >> > > > the
> >> > >> > > > > >>> >> application, I can see cleansed log data appearing
> in
> >> > >> > > > > >>> >> "hdfs_audit_event_sandbox". So I'm thinking here
> that
> >> > >> > > > > >>> >> 'hdfs_audit_log_enriched_sandbox' is not correct or
> >> > >> > necessary?
> >> > >> > > > > >>> >>
> >> > >> > > > > >>> >> b) It's unclear to me where the output data goes
> when
> >> you
> >> > >> > > create a
> >> > >> > > > > >>> policy.
> >> > >> > > > > >>> >> E.g. say I have:
> >> > >> > > > > >>> >>
> >> > >> > > > > >>> >> from HDFS_AUDIT_LOG_ENRICHED_
> >> > STREAM_SANDBOX[str:contains(
> >> > >> > > src,'/hb
> >> > >> > > > > >>> ase')]
> >> > >> > > > > >>> >> select * group by user insert into
> >> > hdfs_audit_log_enriched_
> >> > >> > > > > stream_out
> >> > >> > > > > >>> >>
> >> > >> > > > > >>> >> Where is "hdfs_audit_log_enriched_stream_out"
> defined
> >> > (is
> >> > >> it
> >> > >> > a
> >> > >> > > > > Kafka
> >> > >> > > > > >>> >> topic?). How can I check the output to make sure the
> >> > >> policy is
> >> > >> > > > > working
> >> > >> > > > > >>> >> correctly?
> >> > >> > > > > >>> >>
> >> > >> > > > > >>> >> Thanks,
> >> > >> > > > > >>> >>
> >> > >> > > > > >>> >> Colm.
> >> > >> > > > > >>> >>
> >> > >> > > > > >>> >> On Wed, Jan 17, 2018 at 10:32 PM, Edward Zhang <
> >> > >> > > > > >>> yonzhang2012@gmail.com>
> >> > >> > > > > >>> >> wrote:
> >> > >> > > > > >>> >>
> >> > >> > > > > >>> >> > There is a data preparation stage between data
> >> > >> source(HDFS
> >> > >> > > audit
> >> > >> > > > > >>> log)
> >> > >> > > > > >>> >> and
> >> > >> > > > > >>> >> > Alert Engine. This stage is running in Storm and
> >> > >> transform
> >> > >> > the
> >> > >> > > > raw
> >> > >> > > > > >>> HDFS
> >> > >> > > > > >>> >> log
> >> > >> > > > > >>> >> > into something which can be alerted.
> >> > >> > > > > >>> >> >
> >> > >> > > > > >>> >> > The input for data preparation is
> >> > hdfs_audit_log_sandbox
> >> > >> > topic
> >> > >> > > > and
> >> > >> > > > > >>> >> output
> >> > >> > > > > >>> >> > is
> >> > >> > > > > >>> >> >  hdfs_audit_log_enriched_sandbox.
> >> > >> > > > > >>> >> > The input for Alert Engine is
> >> hdfs_audit_log_enriched_
> >> > >> > sandbox
> >> > >> > > > and
> >> > >> > > > > >>> >> output
> >> > >> > > > > >>> >> > is
> >> > >> > > > > >>> >> > hdfs_audit_log_alert_sandbox.
> >> > >> > > > > >>> >> >
> >> > >> > > > > >>> >> > Seems in your case, the data preparation staging
> is
> >> not
> >> > >> > > working.
> >> > >> > > > > We
> >> > >> > > > > >>> >> > probably need look at Storm console and figure out
> >> if
> >> > >> that
> >> > >> > > part
> >> > >> > > > is
> >> > >> > > > > >>> >> working.
> >> > >> > > > > >>> >> >
> >> > >> > > > > >>> >> > Thanks
> >> > >> > > > > >>> >> > Edward
> >> > >> > > > > >>> >> >
> >> > >> > > > > >>> >> > On Wed, Jan 17, 2018 at 7:19 AM, Colm O
> >> hEigeartaigh <
> >> > >> > > > > >>> >> coheigea@apache.org>
> >> > >> > > > > >>> >> > wrote:
> >> > >> > > > > >>> >> >
> >> > >> > > > > >>> >> > > Hi Jayesh,
> >> > >> > > > > >>> >> > >
> >> > >> > > > > >>> >> > > Many thanks for your feedback! I was able to
> make
> >> a
> >> > >> little
> >> > >> > > > > further
> >> > >> > > > > >>> >> > headway.
> >> > >> > > > > >>> >> > > There are two configuration problems with the
> >> > official
> >> > >> > > docker
> >> > >> > > > > >>> image:
> >> > >> > > > > >>> >> > >
> >> > >> > > > > >>> >> > > a) A mix of "sandbox.eagle.apache.org" and "
> >> > >> > > > > >>> server.eagle.apache.org"
> >> > >> > > > > >>> >> > (this
> >> > >> > > > > >>> >> > > only occurs in the instructions for running the
> >> > docker
> >> > >> > > image.
> >> > >> > > > > The
> >> > >> > > > > >>> >> version
> >> > >> > > > > >>> >> > > that can be started via the script in the eagle
> >> > source
> >> > >> is
> >> > >> > > OK).
> >> > >> > > > > >>> I'll
> >> > >> > > > > >>> >> > submit
> >> > >> > > > > >>> >> > > a PR to fix this once I get a basic use-case
> >> working.
> >> > >> > > > > >>> >> > > b) For the audit case, it automatically logs
> HDFS
> >> > audit
> >> > >> > logs
> >> > >> > > > to
> >> > >> > > > > >>> the
> >> > >> > > > > >>> >> KAFKA
> >> > >> > > > > >>> >> > > topic sandbox_hdfs_audit_log instead of the
> >> expected
> >> > >> > > > > >>> >> > hdfs_audit_log_sandbox
> >> > >> > > > > >>> >> > >
> >> > >> > > > > >>> >> > > I've fixed these things locally and I can verify
> >> that
> >> > >> > > > everything
> >> > >> > > > > >>> is
> >> > >> > > > > >>> >> > started
> >> > >> > > > > >>> >> > > correctly in Ambari. I log into the docker
> >> container
> >> > >> and
> >> > >> > > > create
> >> > >> > > > > >>> >> > > hdfs_audit_log_sandbox and
> >> hdfs_audit_log_enriched_
> >> > >> > sandbox
> >> > >> > > > > >>> topics,
> >> > >> > > > > >>> >> and
> >> > >> > > > > >>> >> > > verify that the HDFS audit logs are flowing into
> >> the
> >> > >> first
> >> > >> > > > > topic.
> >> > >> > > > > >>> >> Then in
> >> > >> > > > > >>> >> > > the UI I start the Alert Engine and then the
> HDFS
> >> > Audit
> >> > >> > Log
> >> > >> > > > > >>> Monitor
> >> > >> > > > > >>> >> > > application (changing localhost:6667 to
> >> > >> > > > > >>> server.eagle.apache.org:6667
> >> > >> > > > > >>> >> ).
> >> > >> > > > > >>> >> > > Both
> >> > >> > > > > >>> >> > > applications start up correctly and show
> >> "running".
> >> > >> > > > > >>> >> > >
> >> > >> > > > > >>> >> > > I then create a policy with an email alert along
> >> the
> >> > >> lines
> >> > >> > > of
> >> > >> > > > > from
> >> > >> > > > > >>> >> > > "HDFS_AUDIT_LOG_ENRICHED_
> >> > STREAM_SANDBOX[str:contains(
> >> > >> > > src,'/h
> >> > >> > > > > >>> base')]
> >> > >> > > > > >>> >> > select
> >> > >> > > > > >>> >> > > * group by user insert into
> >> hdfs_audit_log_enriched_
> >> > >> > > > > stream_out".
> >> > >> > > > > >>> >> However
> >> > >> > > > > >>> >> > > at
> >> > >> > > > > >>> >> > > this point I'm stuck - nothing appears in the
> >> alert
> >> > >> > window.
> >> > >> > > Is
> >> > >> > > > > >>> there
> >> > >> > > > > >>> >> > > anything obvious I'm doing wrong, or how can I
> get
> >> > >> access
> >> > >> > to
> >> > >> > > > > logs
> >> > >> > > > > >>> to
> >> > >> > > > > >>> >> > figure
> >> > >> > > > > >>> >> > > out what the problem is? Other topics such as
> >> > >> > > > > >>> >> "hdfs_audit_event_sandbox"
> >> > >> > > > > >>> >> > > are mentioned in the streams window, but the
> >> > >> documentation
> >> > >> > > > > doesn't
> >> > >> > > > > >>> >> say to
> >> > >> > > > > >>> >> > > create them.
> >> > >> > > > > >>> >> > >
> >> > >> > > > > >>> >> > > The UI is buggy though on both Firefox and
> >> Chromium
> >> > on
> >> > >> > > Linux.
> >> > >> > > > > What
> >> > >> > > > > >>> >> > > browser/platform are people using with the UI?
> >> > >> > > > > >>> >> > >
> >> > >> > > > > >>> >> > > Colm.
> >> > >> > > > > >>> >> > >
> >> > >> > > > > >>> >> > > On Wed, Jan 17, 2018 at 12:27 AM, Jayesh
> >> Senjaliya <
> >> > >> > > > > >>> jaysen@apache.org
> >> > >> > > > > >>> >> >
> >> > >> > > > > >>> >> > > wrote:
> >> > >> > > > > >>> >> > >
> >> > >> > > > > >>> >> > > > Hi Colm,
> >> > >> > > > > >>> >> > > >
> >> > >> > > > > >>> >> > > > Please find my comments inline.
> >> > >> > > > > >>> >> > > >
> >> > >> > > > > >>> >> > > > a) The official docker image uses
> 0.5.0-SNAPSHOT
> >> > and
> >> > >> not
> >> > >> > > the
> >> > >> > > > > >>> >> released
> >> > >> > > > > >>> >> > > > version.
> >> > >> > > > > >>> >> > > > - this is because we uploaded docker image
> >> before
> >> > >> apache
> >> > >> > > > > >>> release.
> >> > >> > > > > >>> >> > > actually
> >> > >> > > > > >>> >> > > > this is same codebase apache-eagle-0.5, and it
> >> can
> >> > be
> >> > >> > > fixed
> >> > >> > > > > >>> easily
> >> > >> > > > > >>> >> by
> >> > >> > > > > >>> >> > > just
> >> > >> > > > > >>> >> > > > rebuilding docker image. there should not be
> any
> >> > >> > mismatch
> >> > >> > > > due
> >> > >> > > > > to
> >> > >> > > > > >>> >> this.
> >> > >> > > > > >>> >> > > >
> >> > >> > > > > >>> >> > > > b) Aside from the above, the official docker
> >> image
> >> > >> uses
> >> > >> > a
> >> > >> > > > mix
> >> > >> > > > > >>> of "
> >> > >> > > > > >>> >> > > > server.eagle.apache.org" and "
> >> > >> sandbox.eagle.apache.org"
> >> > >> > > as
> >> > >> > > > > the
> >> > >> > > > > >>> host
> >> > >> > > > > >>> >> > > name.
> >> > >> > > > > >>> >> > > > The HBase service doesn't start by default in
> >> > Ambari
> >> > >> as
> >> > >> > a
> >> > >> > > > > >>> result.
> >> > >> > > > > >>> >> > > > - the only places it uses sandbox is in
> example
> >> > >> script
> >> > >> > > which
> >> > >> > > > > you
> >> > >> > > > > >>> >> will
> >> > >> > > > > >>> >> > > have
> >> > >> > > > > >>> >> > > > to update anyway, which i agree that it would
> be
> >> > >> good to
> >> > >> > > > keep
> >> > >> > > > > it
> >> > >> > > > > >>> >> > > > consistent.
> >> > >> > > > > >>> >> > > >
> >> > >> > > > > >>> >> > > > c) The UI seems quite buggy. On both chromium
> >> and
> >> > >> > > firefox, I
> >> > >> > > > > >>> only
> >> > >> > > > > >>> >> see
> >> > >> > > > > >>> >> > > > links to "Sandbox" and "Alert" on the left
> >> > hand-side.
> >> > >> > > Once I
> >> > >> > > > > >>> click
> >> > >> > > > > >>> >> on
> >> > >> > > > > >>> >> > > > "Alert" I have no way of going back to see the
> >> > >> > > > applications. I
> >> > >> > > > > >>> don't
> >> > >> > > > > >>> >> > see
> >> > >> > > > > >>> >> > > > the links to "integration" or "sites" as in
> the
> >> > >> picture
> >> > >> > > > here:
> >> > >> > > > > >>> >> > > > http://eagle.apache.org/docs/l
> >> > >> atest/applications/#jmx-
> >> > >> > > > monito
> >> > >> > > > > >>> ring
> >> > >> > > > > >>> >> > > > - when hbase is as deep storage is used, and
> if
> >> > eagle
> >> > >> > app
> >> > >> > > > has
> >> > >> > > > > >>> issue
> >> > >> > > > > >>> >> > > > connecting to hbase, the UI becomes
> >> unresponsive.
> >> > >> > > > > >>> >> > > >
> >> > >> > > > > >>> >> > > > d) In chromium, the button to create a new
> >> policy
> >> > >> does
> >> > >> > not
> >> > >> > > > > >>> exist - I
> >> > >> > > > > >>> >> > can
> >> > >> > > > > >>> >> > > > only see it on Firefox.
> >> > >> > > > > >>> >> > > > - i have seen when you logged in, you will see
> >> > admin
> >> > >> > > > actions.
> >> > >> > > > > >>> but if
> >> > >> > > > > >>> >> > this
> >> > >> > > > > >>> >> > > > still an issue, can you please file UI bug?
> >> > >> > > > > >>> >> > > >
> >> > >> > > > > >>> >> > > > e) I'm trying to get the "Hdfs Audit Log
> >> Monitor"
> >> > >> > use-case
> >> > >> > > > > >>> working,
> >> > >> > > > > >>> >> but
> >> > >> > > > > >>> >> > > it
> >> > >> > > > > >>> >> > > > seems to be stuck in "Initialized".
> >> > >> > > > > >>> >> > > > this eagle docs has example on how to setup
> the
> >> > app.
> >> > >> pls
> >> > >> > > let
> >> > >> > > > > us
> >> > >> > > > > >>> >> know if
> >> > >> > > > > >>> >> > > > you find any gaps.
> >> > >> > > > > >>> >> > > >
> >> > >> > > > > >>> >> > > > Thanks for trying out, and sharing your
> >> findings,
> >> > >> > > > > >>> >> > > > Jayesh
> >> > >> > > > > >>> >> > > >
> >> > >> > > > > >>> >> > > >
> >> > >> > > > > >>> >> > > > On Tue, Jan 16, 2018 at 3:34 AM, Colm O
> >> > hEigeartaigh
> >> > >> <
> >> > >> > > > > >>> >> > > coheigea@apache.org>
> >> > >> > > > > >>> >> > > > wrote:
> >> > >> > > > > >>> >> > > >
> >> > >> > > > > >>> >> > > >> Hi all,
> >> > >> > > > > >>> >> > > >>
> >> > >> > > > > >>> >> > > >> I'm trying to play around a bit with Apache
> >> Eagle
> >> > >> 0.5.0
> >> > >> > > to
> >> > >> > > > no
> >> > >> > > > > >>> >> avail.
> >> > >> > > > > >>> >> > > Here
> >> > >> > > > > >>> >> > > >> are the problems I've run into so far:
> >> > >> > > > > >>> >> > > >>
> >> > >> > > > > >>> >> > > >> a) The official docker image uses
> >> 0.5.0-SNAPSHOT
> >> > and
> >> > >> > not
> >> > >> > > > the
> >> > >> > > > > >>> >> released
> >> > >> > > > > >>> >> > > >> version.
> >> > >> > > > > >>> >> > > >>
> >> > >> > > > > >>> >> > > >> b) Aside from the above, the official docker
> >> image
> >> > >> > uses a
> >> > >> > > > mix
> >> > >> > > > > >>> of "
> >> > >> > > > > >>> >> > > >> server.eagle.apache.org" and "
> >> > >> sandbox.eagle.apache.org
> >> > >> > "
> >> > >> > > as
> >> > >> > > > > the
> >> > >> > > > > >>> >> host
> >> > >> > > > > >>> >> > > >> name. The HBase service doesn't start by
> >> default
> >> > in
> >> > >> > > Ambari
> >> > >> > > > > as a
> >> > >> > > > > >>> >> > result.
> >> > >> > > > > >>> >> > > >>
> >> > >> > > > > >>> >> > > >> c) The UI seems quite buggy. On both chromium
> >> and
> >> > >> > > firefox,
> >> > >> > > > I
> >> > >> > > > > >>> only
> >> > >> > > > > >>> >> see
> >> > >> > > > > >>> >> > > >> links to "Sandbox" and "Alert" on the left
> >> > >> hand-side.
> >> > >> > > Once
> >> > >> > > > I
> >> > >> > > > > >>> click
> >> > >> > > > > >>> >> on
> >> > >> > > > > >>> >> > > >> "Alert" I have no way of going back to see
> the
> >> > >> > > > applications.
> >> > >> > > > > I
> >> > >> > > > > >>> >> don't
> >> > >> > > > > >>> >> > see
> >> > >> > > > > >>> >> > > >> the links to "integration" or "sites" as in
> the
> >> > >> picture
> >> > >> > > > here:
> >> > >> > > > > >>> >> > > >> http://eagle.apache.org/docs/l
> >> > >> atest/applications/#jmx-
> >> > >> > > > monito
> >> > >> > > > > >>> ring
> >> > >> > > > > >>> >> > > >>
> >> > >> > > > > >>> >> > > >> d) In chromium, the button to create a new
> >> policy
> >> > >> does
> >> > >> > > not
> >> > >> > > > > >>> exist -
> >> > >> > > > > >>> >> I
> >> > >> > > > > >>> >> > can
> >> > >> > > > > >>> >> > > >> only see it on Firefox.
> >> > >> > > > > >>> >> > > >>
> >> > >> > > > > >>> >> > > >> e) I'm trying to get the "Hdfs Audit Log
> >> Monitor"
> >> > >> > > use-case
> >> > >> > > > > >>> working,
> >> > >> > > > > >>> >> > but
> >> > >> > > > > >>> >> > > >> it seems to be stuck in "Initialized".
> >> > >> > > > > >>> >> > > >>
> >> > >> > > > > >>> >> > > >> Could someone fill me in on what the
> >> "recommended"
> >> > >> way
> >> > >> > is
> >> > >> > > > to
> >> > >> > > > > >>> start
> >> > >> > > > > >>> >> > > Apache
> >> > >> > > > > >>> >> > > >> Eagle so that I can play around with the
> >> > >> functionality
> >> > >> > > that
> >> > >> > > > > it
> >> > >> > > > > >>> >> offers?
> >> > >> > > > > >>> >> > > >> Clearly the docker approach is buggy. Also,
> >> what
> >> > >> > browser
> >> > >> > > > > >>> should be
> >> > >> > > > > >>> >> > used?
> >> > >> > > > > >>> >> > > >>
> >> > >> > > > > >>> >> > > >> Thanks,
> >> > >> > > > > >>> >> > > >>
> >> > >> > > > > >>> >> > > >> Colm.
> >> > >> > > > > >>> >> > > >>
> >> > >> > > > > >>> >> > > >>
> >> > >> > > > > >>> >> > > >> --
> >> > >> > > > > >>> >> > > >> Colm O hEigeartaigh
> >> > >> > > > > >>> >> > > >>
> >> > >> > > > > >>> >> > > >> Talend Community Coder
> >> > >> > > > > >>> >> > > >> http://coders.talend.com
> >> > >> > > > > >>> >> > > >>
> >> > >> > > > > >>> >> > > >
> >> > >> > > > > >>> >> > > >
> >> > >> > > > > >>> >> > >
> >> > >> > > > > >>> >> > >
> >> > >> > > > > >>> >> > > --
> >> > >> > > > > >>> >> > > Colm O hEigeartaigh
> >> > >> > > > > >>> >> > >
> >> > >> > > > > >>> >> > > Talend Community Coder
> >> > >> > > > > >>> >> > > http://coders.talend.com
> >> > >> > > > > >>> >> > >
> >> > >> > > > > >>> >> >
> >> > >> > > > > >>> >>
> >> > >> > > > > >>> >>
> >> > >> > > > > >>> >>
> >> > >> > > > > >>> >> --
> >> > >> > > > > >>> >> Colm O hEigeartaigh
> >> > >> > > > > >>> >>
> >> > >> > > > > >>> >> Talend Community Coder
> >> > >> > > > > >>> >> http://coders.talend.com
> >> > >> > > > > >>> >>
> >> > >> > > > > >>> >>
> >> > >> > > > > >>> >>
> >> > >> > > > > >>> >
> >> > >> > > > > >>> >
> >> > >> > > > > >>> > --
> >> > >> > > > > >>> > Colm O hEigeartaigh
> >> > >> > > > > >>> >
> >> > >> > > > > >>> > Talend Community Coder
> >> > >> > > > > >>> > http://coders.talend.com
> >> > >> > > > > >>> >
> >> > >> > > > > >>>
> >> > >> > > > > >>>
> >> > >> > > > > >>>
> >> > >> > > > > >>> --
> >> > >> > > > > >>> Colm O hEigeartaigh
> >> > >> > > > > >>>
> >> > >> > > > > >>> Talend Community Coder
> >> > >> > > > > >>> http://coders.talend.com
> >> > >> > > > > >>>
> >> > >> > > > > >>
> >> > >> > > > > >>
> >> > >> > > > > >
> >> > >> > > > >
> >> > >> > > > >
> >> > >> > > > > --
> >> > >> > > > > Colm O hEigeartaigh
> >> > >> > > > >
> >> > >> > > > > Talend Community Coder
> >> > >> > > > > http://coders.talend.com
> >> > >> > > > >
> >> > >> > > >
> >> > >> > >
> >> > >> > >
> >> > >> > >
> >> > >> > > --
> >> > >> > > Colm O hEigeartaigh
> >> > >> > >
> >> > >> > > Talend Community Coder
> >> > >> > > http://coders.talend.com
> >> > >> > >
> >> > >> >
> >> > >>
> >> > >>
> >> > >>
> >> > >> --
> >> > >> Colm O hEigeartaigh
> >> > >>
> >> > >> Talend Community Coder
> >> > >> http://coders.talend.com
> >> > >>
> >> > >
> >> > >
> >> >
> >> >
> >> > --
> >> > Colm O hEigeartaigh
> >> >
> >> > Talend Community Coder
> >> > http://coders.talend.com
> >> >
> >>
> >
> >
> >
> > --
> > Colm O hEigeartaigh
> >
> > Talend Community Coder
> > http://coders.talend.com
> >
>



-- 
Colm O hEigeartaigh

Talend Community Coder
http://coders.talend.com

Re: Unable to get 0.5.0 release working

Posted by Jayesh Senjaliya <ja...@apache.org>.
Ya slackpublisher has some issue and we have fixed it and also changed the
implementation to use rest api instad of sdk.
We will push the patch soon.

Thanks
Jayesh

On Fri, Feb 2, 2018 at 9:02 AM Colm O hEigeartaigh <co...@apache.org>
wrote:

> Thanks. I've submitted an initial PR to fix the Slack
> ClassNotFoundException issue here:
>
> https://issues.apache.org/jira/browse/EAGLE-879
> https://github.com/apache/eagle/pull/984
>
> However, no actual Slack messages are sent. The problem is that
> AlertSlackPublisher only sends the message if the event contains a
> "severity" that matches the configured "severity" on the publisher. However
> the event contains neither "severity" (or "message") so the message never
> gets sent. This is for an HDFS audit log - I'm not sure if there are other
> scenarios where there is a "severity" column in the event? Either that or
> it looks like the SlackPublisher was written before the event column
> headers were changed.
>
> Colm.
>
> On Fri, Feb 2, 2018 at 12:54 AM, Jayesh Senjaliya <ja...@apache.org>
> wrote:
>
>> resolved those tickets now.
>>
>> I have asked the developer to rebase the PR #941, if he doesnt get to it
>> by
>> this week, i will take care of, its long pending one.
>> Thanks for verifying though.
>>
>> - Jayesh
>>
>> On Thu, Feb 1, 2018 at 8:56 AM, Colm O hEigeartaigh <co...@apache.org>
>> wrote:
>>
>> > Thanks Jayesh. I have two more PRs awaiting review:
>> >
>> > https://github.com/apache/eagle/pull/981
>> > https://github.com/apache/eagle/pull/982
>> >
>> > Thanks for the JIRA privileges, I can now assign issues to me + change
>> the
>> > versions. However, I can't "resolve" JIRAs that weren't reported by me
>> > which is annoying. These 3 JIRAs should be resolved as they are already
>> > merged:
>> >
>> > https://issues.apache.org/jira/browse/EAGLE-445
>> > https://issues.apache.org/jira/browse/EAGLE-476
>> > https://issues.apache.org/jira/browse/EAGLE-331
>> >
>> > In addition, I tested the fix for the Email issue and it works
>> correctly.
>> > The PR (https://github.com/apache/eagle/pull/941) just needs to have
>> the
>> > extra commits stripped away - I attached a version of the patch on the
>> > JIRA.
>> >
>> > Colm.
>> >
>> > On Wed, Jan 31, 2018 at 10:08 PM, Jayesh Senjaliya <ja...@apache.org>
>> > wrote:
>> >
>> > > Thanks for the PRs. I have merged them.
>> > >
>> > > welcome to the developer community Colm. I have also added you to jira
>> > > project so can assign the tasks to yourself.
>> > >
>> > > lets create ticket to fix the dedup functionality, I m actually
>> surprised
>> > > we havent hit this issue yet. we do use multiple publishers but
>> someone
>> > can
>> > > verify this.
>> > >
>> > > Thanks
>> > > Jayesh
>> > >
>> > >
>> > >
>> > > On Wed, Jan 31, 2018 at 9:25 AM, Colm O hEigeartaigh <
>> > coheigea@apache.org>
>> > > wrote:
>> > >
>> > >> Thanks Jayesh. I've created a JIRA here for some admin work for some
>> > >> issues
>> > >> that were incorrectly flagged as "fix for" 0.5.1/0.6.0:
>> > >>
>> > >> https://issues.apache.org/jira/browse/EAGLE-1076
>> > >>
>> > >> I've submitted the following (fairly trivial) pull requests. Could I
>> ask
>> > >> that you or one of the other committers review?
>> > >>
>> > >> https://github.com/apache/eagle/pull/978
>> > >> https://github.com/apache/eagle/pull/979
>> > >> https://github.com/apache/eagle/pull/980
>> > >>
>> > >> It would be good to try to inject some energy into the project. We
>> need
>> > >> more than one active committer though.
>> > >>
>> > >> Just in terms of the Alert Deduplication issue. The
>> DefaultDeDuplicator
>> > >> works per "output" in the policy rule. So if you have more than one
>> > >> AlertPublisher, I think it is guaranteed to only publish to one of
>> them.
>> > >> Instead, surely it would make more sense to work per publisher?
>> > >>
>> > >> Colm.
>> > >>
>> > >> On Tue, Jan 30, 2018 at 10:39 PM, Jayesh Senjaliya <
>> jaysen@apache.org>
>> > >> wrote:
>> > >>
>> > >> > Hi Colm,
>> > >> >
>> > >> > appreciate your suggestions/ efforts in looking into this project,
>> > >> > putting my comments inline...
>> > >> >
>> > >> > a) There is already a JIRA to bump the version here, although the
>> PR
>> > >> does
>> > >> > not apply as it is too old: https://issues.apache.org/
>> > >> > jira/browse/EAGLE-1025
>> > >> > .
>> > >> > I can submit a new PR, but should the version be 0.6.0 or 0.5.1?
>> > >> >
>> > >> > *since there are still minor issues, i would say, we put up 0.5.1
>> as
>> > >> next
>> > >> > version. I've updated/rebased the PR (
>> > >> > https://github.com/apache/eagle/pull/936
>> > >> > <https://github.com/apache/eagle/pull/936> )*
>> > >> >
>> > >> >
>> > >> > b) The issues that are "resolved" for the 0.5.1 release in JIRA are
>> > >> > actually already fixed in 0.5.0, so they should be updated (
>> > >> > https://issues.apache.org/jira/projects/EAGLE/versions/12341128).
>> > >> However,
>> > >> > the following two issues are resolved even though they are not
>> merged
>> > to
>> > >> > master?
>> > >> >   https://issues.apache.org/jira/browse/EAGLE-1051 .  - * this was
>> > >> pending
>> > >> > from developer;s response but i think this is reviewed, so I have
>> > merged
>> > >> > it.*
>> > >> >   https://issues.apache.org/jira/browse/EAGLE-1068 .  - * this is
>> > >> reopened
>> > >> > now. I dont think this is done yet. Also this is big change.*
>> > >> >
>> > >> >
>> > >> > Like I said I can submit PRs but I'm not convinced there is any
>> > >> activity on
>> > >> > the project. Where are the rest of the committers?
>> > >> >
>> > >> > *let me give you some ocontext on this. so there were lot of
>> > development
>> > >> > happened during last releases, and most of applications that were
>> > added
>> > >> are
>> > >> > being used in production at multiple enterprise companies, but we
>> are
>> > >> out
>> > >> > of ideas on new apps, so at this point we are only focusing on bug
>> > fixes
>> > >> > and tech upgrades until we get some new ideas to brainstorm and
>> add.*
>> > >> >
>> > >> > *I think current community's thinking is based on their own
>> industries
>> > >> > use-cases, but there is definitely room for new features and
>> > integration
>> > >> > with other monitoring and security components like grafana and
>> > rangers.*
>> > >> >
>> > >> >
>> > >> > *Thanks,*
>> > >> > *Jayesh*
>> > >> >
>> > >> >
>> > >> >
>> > >> > On Tue, Jan 30, 2018 at 8:11 AM, Colm O hEigeartaigh <
>> > >> coheigea@apache.org>
>> > >> > wrote:
>> > >> >
>> > >> > > Hi Jayesh,
>> > >> > >
>> > >> > > Dev suggestions:
>> > >> > >
>> > >> > > a) There is already a JIRA to bump the version here, although
>> the PR
>> > >> does
>> > >> > > not apply as it is too old: https://issues.apache.org/
>> > >> > > jira/browse/EAGLE-1025.
>> > >> > > I can submit a new PR, but should the version be 0.6.0 or 0.5.1?
>> > >> > > b) The issues that are "resolved" for the 0.5.1 release in JIRA
>> are
>> > >> > > actually already fixed in 0.5.0, so they should be updated (
>> > >> > > https://issues.apache.org/jira/projects/EAGLE/versions/12341128
>> ).
>> > >> > However,
>> > >> > > the following two issues are resolved even though they are not
>> > merged
>> > >> to
>> > >> > > master?
>> > >> > >   https://issues.apache.org/jira/browse/EAGLE-1051
>> > >> > >   https://issues.apache.org/jira/browse/EAGLE-1068
>> > >> > >
>> > >> > > Like I said I can submit PRs but I'm not convinced there is any
>> > >> activity
>> > >> > on
>> > >> > > the project. Where are the rest of the committers?
>> > >> > >
>> > >> > > Multiple Publisher issue:
>> > >> > >
>> > >> > > If I assign two publishers for one policy, the alert only goes to
>> > the
>> > >> > first
>> > >> > > policy. In the logs I see:
>> > >> > >
>> > >> > > 2018-01-30T15:52:45.835+0000 o.a.e.a.e.p.d.DefaultDeduplicator
>> > [INFO]
>> > >> > > Alert
>> > >> > > event is skipped because it's duplicated: Alert {site=sandbox,
>> > >> > > stream=eagle_output,timestamp=2018-01-30
>> > >> > > 00:00:11,300,data={securityZone=NA, dst=null, sensitivityType=NA,
>> > >> > > src=/apps/hbase/data/archive/data/default/ambarismoketest,
>> > >> allowed=true,
>> > >> > > host=172.22.7.129, cmd=listStatus, user=SOMETHING7.COM,
>> > >> > > timestamp=1517270411300}, policyId=test,
>> > >> > > createdBy=alertBolt3-evaluator_stage1, metaVersion=null}
>> > >> > >
>> > >> > > It looks like this deduplicator is not working properly, as I'm
>> > >> guessing
>> > >> > it
>> > >> > > should only be used to de-duplicate events for a single
>> publisher?
>> > >> > >
>> > >> > > Incognito mode: Already tried it but with the same result. Could
>> I
>> > ask
>> > >> > you
>> > >> > > to try the docker image to see if the UI is working correctly for
>> > you
>> > >> > > there?
>> > >> > >
>> > >> > > Colm.
>> > >> > >
>> > >> > > On Mon, Jan 29, 2018 at 6:46 PM, Jayesh Senjaliya <
>> > jaysen@apache.org>
>> > >> > > wrote:
>> > >> > >
>> > >> > > > Hi Colm,
>> > >> > > >
>> > >> > > > Thanks for the list of dev suggestions, I think we should take
>> > care
>> > >> of
>> > >> > > > those. even better if you can provide PR with the changes or at
>> > >> keast
>> > >> > can
>> > >> > > > you please create a ticket so we can track it?
>> > >> > > >
>> > >> > > > for other issues.
>> > >> > > >
>> > >> > > > - I dont have any issue with multiple publisher, but if there
>> is
>> > any
>> > >> > > error
>> > >> > > > updating the publisher info in storm topology, i might try
>> > >> restarting
>> > >> > the
>> > >> > > > topology and see if that works.
>> > >> > > > - for us, chrome works as fine as firefox.  can u try incognito
>> > >> mode?
>> > >> > > just
>> > >> > > > to be sure to have clean cache?
>> > >> > > >
>> > >> > > > Thanks
>> > >> > > > Jayesh
>> > >> > > >
>> > >> > > >
>> > >> > > > On Thu, Jan 25, 2018 at 4:19 AM, Colm O hEigeartaigh <
>> > >> > > coheigea@apache.org>
>> > >> > > > wrote:
>> > >> > > >
>> > >> > > > > Thanks again for your feedback. Jayesh, adding
>> > >> AlertEagleStorePlugin
>> > >> > > did
>> > >> > > > > the trick, I can now see alerts in the UI, thanks! By the
>> way, I
>> > >> > can't
>> > >> > > > > configure two Alert Publishers, or else the Alert
>> DeDuplicator
>> > >> bins
>> > >> > the
>> > >> > > > > alert. Is this a known issue?
>> > >> > > > >
>> > >> > > > > Could I ask which browser people are using with the UI? There
>> > >> appears
>> > >> > > to
>> > >> > > > be
>> > >> > > > > a  bug with Chromium where it doesn't list the pages under
>> > >> > Auth.isAdmin
>> > >> > > > > even though I am logged on as an administrator. It works OK
>> in
>> > >> > Firefox.
>> > >> > > > > Even with Firefox though, I only see a limited number of
>> links
>> > in
>> > >> the
>> > >> > > > > left-hand column - I can't get back to the "integration"
>> page.
>> > Can
>> > >> > > > someone
>> > >> > > > > else confirm this please?
>> > >> > > > >
>> > >> > > > > Could I suggest the devs do some basic house-keeping tasks:
>> > >> > > > >
>> > >> > > > > a) "Release" version 0.5.0 in JIRA (it's still listed as
>> > >> > "unreleased").
>> > >> > > > > b) Figure out whether the next version will be 0.5.1 or 0.6.0
>> > and
>> > >> > > update
>> > >> > > > > the versions on Master accordingly with 0.5.1-SNAPSHOT or
>> > >> > > 0.6.0-SNAPSHOT.
>> > >> > > > > There are some issues marked here as resolved for 0.5.1 -
>> > >> > > > >
>> https://issues.apache.org/jira/projects/EAGLE/versions/12341128
>> > ),
>> > >> > > > however
>> > >> > > > > I
>> > >> > > > > don't see a branch for 0.5.x?
>> > >> > > > >
>> > >> > > > > Colm.
>> > >> > > > >
>> > >> > > > > On Thu, Jan 25, 2018 at 8:16 AM, Jayesh Senjaliya <
>> > >> jaysen@apache.org
>> > >> > >
>> > >> > > > > wrote:
>> > >> > > > >
>> > >> > > > > > Hi,
>> > >> > > > > >
>> > >> > > > > > we do use eagle 0.5 in production although we dont use all
>> the
>> > >> > > > available
>> > >> > > > > > hadoop applications.
>> > >> > > > > >
>> > >> > > > > > EAGLE-968 <https://issues.apache.org/jira/browse/EAGLE-968
>> >
>> > is
>> > >> a
>> > >> > fix
>> > >> > > > for
>> > >> > > > > > email issue we found while our testing. should be merged
>> soon
>> > >> > after a
>> > >> > > > > > rebase.
>> > >> > > > > >
>> > >> > > > > > @Colm, did you tried adding storage publisher
>> > >> > > (AlertEagleStorePlugin)?
>> > >> > > > to
>> > >> > > > > > see alerts on UI ?
>> > >> > > > > >
>> > >> > > > > > Thanks
>> > >> > > > > > Jayesh
>> > >> > > > > >
>> > >> > > > > >
>> > >> > > > > >
>> > >> > > > > >
>> > >> > > > > >
>> > >> > > > > >
>> > >> > > > > > On Wed, Jan 24, 2018 at 7:08 PM, Edward Zhang <
>> > >> > > yonzhang2012@gmail.com>
>> > >> > > > > > wrote:
>> > >> > > > > >
>> > >> > > > > >> Eagle 0.5 was deployed in production as far as I know,
>> but it
>> > >> may
>> > >> > > not
>> > >> > > > be
>> > >> > > > > >> exact the current version in master branch.
>> > >> > > > > >>
>> > >> > > > > >> Thanks for your investigation, seems there is still some
>> bug
>> > in
>> > >> > 0.5,
>> > >> > > > but
>> > >> > > > > >> this particular issue seems is due to dependent components
>> > >> version
>> > >> > > > > conflict.
>> > >> > > > > >>
>> > >> > > > > >> @Jayesh is this Jira ready for merge to master?
>> > >> > > https://issues.apache
>> > >> > > > .
>> > >> > > > > >> org/jira/browse/EAGLE-968
>> > >> > > > > >>
>> > >> > > > > >>
>> > >> > > > > >> Thanks
>> > >> > > > > >> Edward
>> > >> > > > > >>
>> > >> > > > > >> On Tue, Jan 23, 2018 at 5:10 AM, Colm O hEigeartaigh <
>> > >> > > > > coheigea@apache.org
>> > >> > > > > >> > wrote:
>> > >> > > > > >>
>> > >> > > > > >>> OK I've made some more progress. I wasn't seeing any
>> email
>> > >> alerts
>> > >> > > due
>> > >> > > > > to
>> > >> > > > > >>> https://issues.apache.org/jira/browse/EAGLE-968. Once I
>> > >> > configure
>> > >> > > a
>> > >> > > > > >>> Kafka
>> > >> > > > > >>> alert, I can see the alerts flowing into my topic. It's
>> > still
>> > >> not
>> > >> > > > clear
>> > >> > > > > >>> to
>> > >> > > > > >>> me however where the policy "output" is going. I also
>> don't
>> > >> see
>> > >> > any
>> > >> > > > > >>> alerts
>> > >> > > > > >>> in the UI window.
>> > >> > > > > >>>
>> > >> > > > > >>> Could I ask what the status of the project is in general?
>> > >> There
>> > >> > > have
>> > >> > > > > been
>> > >> > > > > >>> no commits to master since November, so I'm not sure if
>> > there
>> > >> is
>> > >> > > any
>> > >> > > > > >>> point
>> > >> > > > > >>> in submitting Pull Requests for outstanding bugs? Are
>> recent
>> > >> > > versions
>> > >> > > > > of
>> > >> > > > > >>> Apache Eagle used in production?
>> > >> > > > > >>>
>> > >> > > > > >>> Colm.
>> > >> > > > > >>>
>> > >> > > > > >>> On Mon, Jan 22, 2018 at 1:07 PM, Colm O hEigeartaigh <
>> > >> > > > > >>> coheigea@apache.org>
>> > >> > > > > >>> wrote:
>> > >> > > > > >>>
>> > >> > > > > >>> >
>> > >> > > > > >>> > I've done that but I'm not seeing any alerts, which is
>> > why I
>> > >> > want
>> > >> > > > to
>> > >> > > > > >>> find
>> > >> > > > > >>> > out what the "output" of a policy is and where I can
>> check
>> > >> > this.
>> > >> > > > > >>> >
>> > >> > > > > >>> > Colm.
>> > >> > > > > >>> >
>> > >> > > > > >>> > On Mon, Jan 22, 2018 at 1:05 PM, SUDHA JENSLIN <
>> > >> > > sjenslin@gmail.com
>> > >> > > > >
>> > >> > > > > >>> wrote:
>> > >> > > > > >>> >
>> > >> > > > > >>> >> Create and add a publisher to see the output.
>> > >> > > > > >>> >>
>> > >> > > > > >>> >>
>> > >> > > > > >>> >>
>> > >> > > > > >>> >> Regards,
>> > >> > > > > >>> >> Sudha jenslin
>> > >> > > > > >>> >>
>> > >> > > > > >>> >> On Jan 22, 2018 6:31 PM, "Colm O hEigeartaigh" <
>> > >> > > > coheigea@apache.org
>> > >> > > > > >
>> > >> > > > > >>> >> wrote:
>> > >> > > > > >>> >>
>> > >> > > > > >>> >> Thanks - the error was due to a problem running Storm
>> > with
>> > >> > Java
>> > >> > > > 1.8.
>> > >> > > > > >>> I've
>> > >> > > > > >>> >> abandoned the docker image for now, and I'm trying to
>> get
>> > >> it
>> > >> > > > working
>> > >> > > > > >>> >> locally.
>> > >> > > > > >>> >>
>> > >> > > > > >>> >> There are two things I'm not clear on currently, if
>> > someone
>> > >> > > could
>> > >> > > > > >>> fill me
>> > >> > > > > >>> >> in:
>> > >> > > > > >>> >>
>> > >> > > > > >>> >> a) For the  'Hdfs Audit Log Monitor' application, the
>> > Kafka
>> > >> > > > Consumer
>> > >> > > > > >>> Topic
>> > >> > > > > >>> >> is 'hdfs_audit_log_sandbox'. Under 'Kafka Topic for
>> > >> Auditlog
>> > >> > > Event
>> > >> > > > > >>> Sink'
>> > >> > > > > >>> >> it
>> > >> > > > > >>> >> also specifies 'hdfs_audit_event_sandbox'. However the
>> > >> > > > documentation
>> > >> > > > > >>> for
>> > >> > > > > >>> >> the application mentions
>> 'hdfs_audit_log_enriched_sandb
>> > >> ox'?
>> > >> > > > > >>> >>
>> > >> > > > > >>> >> When I click on "STREAMS", the
>> > >> "HDFS_AUDIT_LOG_ENRICHED_STREA
>> > >> > > > > >>> M_SANDBOX"
>> > >> > > > > >>> >> uses the topic "hdfs_audit_event_sandbox". And indeed
>> > when
>> > >> I
>> > >> > run
>> > >> > > > the
>> > >> > > > > >>> >> application, I can see cleansed log data appearing in
>> > >> > > > > >>> >> "hdfs_audit_event_sandbox". So I'm thinking here that
>> > >> > > > > >>> >> 'hdfs_audit_log_enriched_sandbox' is not correct or
>> > >> > necessary?
>> > >> > > > > >>> >>
>> > >> > > > > >>> >> b) It's unclear to me where the output data goes when
>> you
>> > >> > > create a
>> > >> > > > > >>> policy.
>> > >> > > > > >>> >> E.g. say I have:
>> > >> > > > > >>> >>
>> > >> > > > > >>> >> from HDFS_AUDIT_LOG_ENRICHED_
>> > STREAM_SANDBOX[str:contains(
>> > >> > > src,'/hb
>> > >> > > > > >>> ase')]
>> > >> > > > > >>> >> select * group by user insert into
>> > hdfs_audit_log_enriched_
>> > >> > > > > stream_out
>> > >> > > > > >>> >>
>> > >> > > > > >>> >> Where is "hdfs_audit_log_enriched_stream_out" defined
>> > (is
>> > >> it
>> > >> > a
>> > >> > > > > Kafka
>> > >> > > > > >>> >> topic?). How can I check the output to make sure the
>> > >> policy is
>> > >> > > > > working
>> > >> > > > > >>> >> correctly?
>> > >> > > > > >>> >>
>> > >> > > > > >>> >> Thanks,
>> > >> > > > > >>> >>
>> > >> > > > > >>> >> Colm.
>> > >> > > > > >>> >>
>> > >> > > > > >>> >> On Wed, Jan 17, 2018 at 10:32 PM, Edward Zhang <
>> > >> > > > > >>> yonzhang2012@gmail.com>
>> > >> > > > > >>> >> wrote:
>> > >> > > > > >>> >>
>> > >> > > > > >>> >> > There is a data preparation stage between data
>> > >> source(HDFS
>> > >> > > audit
>> > >> > > > > >>> log)
>> > >> > > > > >>> >> and
>> > >> > > > > >>> >> > Alert Engine. This stage is running in Storm and
>> > >> transform
>> > >> > the
>> > >> > > > raw
>> > >> > > > > >>> HDFS
>> > >> > > > > >>> >> log
>> > >> > > > > >>> >> > into something which can be alerted.
>> > >> > > > > >>> >> >
>> > >> > > > > >>> >> > The input for data preparation is
>> > hdfs_audit_log_sandbox
>> > >> > topic
>> > >> > > > and
>> > >> > > > > >>> >> output
>> > >> > > > > >>> >> > is
>> > >> > > > > >>> >> >  hdfs_audit_log_enriched_sandbox.
>> > >> > > > > >>> >> > The input for Alert Engine is
>> hdfs_audit_log_enriched_
>> > >> > sandbox
>> > >> > > > and
>> > >> > > > > >>> >> output
>> > >> > > > > >>> >> > is
>> > >> > > > > >>> >> > hdfs_audit_log_alert_sandbox.
>> > >> > > > > >>> >> >
>> > >> > > > > >>> >> > Seems in your case, the data preparation staging is
>> not
>> > >> > > working.
>> > >> > > > > We
>> > >> > > > > >>> >> > probably need look at Storm console and figure out
>> if
>> > >> that
>> > >> > > part
>> > >> > > > is
>> > >> > > > > >>> >> working.
>> > >> > > > > >>> >> >
>> > >> > > > > >>> >> > Thanks
>> > >> > > > > >>> >> > Edward
>> > >> > > > > >>> >> >
>> > >> > > > > >>> >> > On Wed, Jan 17, 2018 at 7:19 AM, Colm O
>> hEigeartaigh <
>> > >> > > > > >>> >> coheigea@apache.org>
>> > >> > > > > >>> >> > wrote:
>> > >> > > > > >>> >> >
>> > >> > > > > >>> >> > > Hi Jayesh,
>> > >> > > > > >>> >> > >
>> > >> > > > > >>> >> > > Many thanks for your feedback! I was able to make
>> a
>> > >> little
>> > >> > > > > further
>> > >> > > > > >>> >> > headway.
>> > >> > > > > >>> >> > > There are two configuration problems with the
>> > official
>> > >> > > docker
>> > >> > > > > >>> image:
>> > >> > > > > >>> >> > >
>> > >> > > > > >>> >> > > a) A mix of "sandbox.eagle.apache.org" and "
>> > >> > > > > >>> server.eagle.apache.org"
>> > >> > > > > >>> >> > (this
>> > >> > > > > >>> >> > > only occurs in the instructions for running the
>> > docker
>> > >> > > image.
>> > >> > > > > The
>> > >> > > > > >>> >> version
>> > >> > > > > >>> >> > > that can be started via the script in the eagle
>> > source
>> > >> is
>> > >> > > OK).
>> > >> > > > > >>> I'll
>> > >> > > > > >>> >> > submit
>> > >> > > > > >>> >> > > a PR to fix this once I get a basic use-case
>> working.
>> > >> > > > > >>> >> > > b) For the audit case, it automatically logs HDFS
>> > audit
>> > >> > logs
>> > >> > > > to
>> > >> > > > > >>> the
>> > >> > > > > >>> >> KAFKA
>> > >> > > > > >>> >> > > topic sandbox_hdfs_audit_log instead of the
>> expected
>> > >> > > > > >>> >> > hdfs_audit_log_sandbox
>> > >> > > > > >>> >> > >
>> > >> > > > > >>> >> > > I've fixed these things locally and I can verify
>> that
>> > >> > > > everything
>> > >> > > > > >>> is
>> > >> > > > > >>> >> > started
>> > >> > > > > >>> >> > > correctly in Ambari. I log into the docker
>> container
>> > >> and
>> > >> > > > create
>> > >> > > > > >>> >> > > hdfs_audit_log_sandbox and
>> hdfs_audit_log_enriched_
>> > >> > sandbox
>> > >> > > > > >>> topics,
>> > >> > > > > >>> >> and
>> > >> > > > > >>> >> > > verify that the HDFS audit logs are flowing into
>> the
>> > >> first
>> > >> > > > > topic.
>> > >> > > > > >>> >> Then in
>> > >> > > > > >>> >> > > the UI I start the Alert Engine and then the HDFS
>> > Audit
>> > >> > Log
>> > >> > > > > >>> Monitor
>> > >> > > > > >>> >> > > application (changing localhost:6667 to
>> > >> > > > > >>> server.eagle.apache.org:6667
>> > >> > > > > >>> >> ).
>> > >> > > > > >>> >> > > Both
>> > >> > > > > >>> >> > > applications start up correctly and show
>> "running".
>> > >> > > > > >>> >> > >
>> > >> > > > > >>> >> > > I then create a policy with an email alert along
>> the
>> > >> lines
>> > >> > > of
>> > >> > > > > from
>> > >> > > > > >>> >> > > "HDFS_AUDIT_LOG_ENRICHED_
>> > STREAM_SANDBOX[str:contains(
>> > >> > > src,'/h
>> > >> > > > > >>> base')]
>> > >> > > > > >>> >> > select
>> > >> > > > > >>> >> > > * group by user insert into
>> hdfs_audit_log_enriched_
>> > >> > > > > stream_out".
>> > >> > > > > >>> >> However
>> > >> > > > > >>> >> > > at
>> > >> > > > > >>> >> > > this point I'm stuck - nothing appears in the
>> alert
>> > >> > window.
>> > >> > > Is
>> > >> > > > > >>> there
>> > >> > > > > >>> >> > > anything obvious I'm doing wrong, or how can I get
>> > >> access
>> > >> > to
>> > >> > > > > logs
>> > >> > > > > >>> to
>> > >> > > > > >>> >> > figure
>> > >> > > > > >>> >> > > out what the problem is? Other topics such as
>> > >> > > > > >>> >> "hdfs_audit_event_sandbox"
>> > >> > > > > >>> >> > > are mentioned in the streams window, but the
>> > >> documentation
>> > >> > > > > doesn't
>> > >> > > > > >>> >> say to
>> > >> > > > > >>> >> > > create them.
>> > >> > > > > >>> >> > >
>> > >> > > > > >>> >> > > The UI is buggy though on both Firefox and
>> Chromium
>> > on
>> > >> > > Linux.
>> > >> > > > > What
>> > >> > > > > >>> >> > > browser/platform are people using with the UI?
>> > >> > > > > >>> >> > >
>> > >> > > > > >>> >> > > Colm.
>> > >> > > > > >>> >> > >
>> > >> > > > > >>> >> > > On Wed, Jan 17, 2018 at 12:27 AM, Jayesh
>> Senjaliya <
>> > >> > > > > >>> jaysen@apache.org
>> > >> > > > > >>> >> >
>> > >> > > > > >>> >> > > wrote:
>> > >> > > > > >>> >> > >
>> > >> > > > > >>> >> > > > Hi Colm,
>> > >> > > > > >>> >> > > >
>> > >> > > > > >>> >> > > > Please find my comments inline.
>> > >> > > > > >>> >> > > >
>> > >> > > > > >>> >> > > > a) The official docker image uses 0.5.0-SNAPSHOT
>> > and
>> > >> not
>> > >> > > the
>> > >> > > > > >>> >> released
>> > >> > > > > >>> >> > > > version.
>> > >> > > > > >>> >> > > > - this is because we uploaded docker image
>> before
>> > >> apache
>> > >> > > > > >>> release.
>> > >> > > > > >>> >> > > actually
>> > >> > > > > >>> >> > > > this is same codebase apache-eagle-0.5, and it
>> can
>> > be
>> > >> > > fixed
>> > >> > > > > >>> easily
>> > >> > > > > >>> >> by
>> > >> > > > > >>> >> > > just
>> > >> > > > > >>> >> > > > rebuilding docker image. there should not be any
>> > >> > mismatch
>> > >> > > > due
>> > >> > > > > to
>> > >> > > > > >>> >> this.
>> > >> > > > > >>> >> > > >
>> > >> > > > > >>> >> > > > b) Aside from the above, the official docker
>> image
>> > >> uses
>> > >> > a
>> > >> > > > mix
>> > >> > > > > >>> of "
>> > >> > > > > >>> >> > > > server.eagle.apache.org" and "
>> > >> sandbox.eagle.apache.org"
>> > >> > > as
>> > >> > > > > the
>> > >> > > > > >>> host
>> > >> > > > > >>> >> > > name.
>> > >> > > > > >>> >> > > > The HBase service doesn't start by default in
>> > Ambari
>> > >> as
>> > >> > a
>> > >> > > > > >>> result.
>> > >> > > > > >>> >> > > > - the only places it uses sandbox is in example
>> > >> script
>> > >> > > which
>> > >> > > > > you
>> > >> > > > > >>> >> will
>> > >> > > > > >>> >> > > have
>> > >> > > > > >>> >> > > > to update anyway, which i agree that it would be
>> > >> good to
>> > >> > > > keep
>> > >> > > > > it
>> > >> > > > > >>> >> > > > consistent.
>> > >> > > > > >>> >> > > >
>> > >> > > > > >>> >> > > > c) The UI seems quite buggy. On both chromium
>> and
>> > >> > > firefox, I
>> > >> > > > > >>> only
>> > >> > > > > >>> >> see
>> > >> > > > > >>> >> > > > links to "Sandbox" and "Alert" on the left
>> > hand-side.
>> > >> > > Once I
>> > >> > > > > >>> click
>> > >> > > > > >>> >> on
>> > >> > > > > >>> >> > > > "Alert" I have no way of going back to see the
>> > >> > > > applications. I
>> > >> > > > > >>> don't
>> > >> > > > > >>> >> > see
>> > >> > > > > >>> >> > > > the links to "integration" or "sites" as in the
>> > >> picture
>> > >> > > > here:
>> > >> > > > > >>> >> > > > http://eagle.apache.org/docs/l
>> > >> atest/applications/#jmx-
>> > >> > > > monito
>> > >> > > > > >>> ring
>> > >> > > > > >>> >> > > > - when hbase is as deep storage is used, and if
>> > eagle
>> > >> > app
>> > >> > > > has
>> > >> > > > > >>> issue
>> > >> > > > > >>> >> > > > connecting to hbase, the UI becomes
>> unresponsive.
>> > >> > > > > >>> >> > > >
>> > >> > > > > >>> >> > > > d) In chromium, the button to create a new
>> policy
>> > >> does
>> > >> > not
>> > >> > > > > >>> exist - I
>> > >> > > > > >>> >> > can
>> > >> > > > > >>> >> > > > only see it on Firefox.
>> > >> > > > > >>> >> > > > - i have seen when you logged in, you will see
>> > admin
>> > >> > > > actions.
>> > >> > > > > >>> but if
>> > >> > > > > >>> >> > this
>> > >> > > > > >>> >> > > > still an issue, can you please file UI bug?
>> > >> > > > > >>> >> > > >
>> > >> > > > > >>> >> > > > e) I'm trying to get the "Hdfs Audit Log
>> Monitor"
>> > >> > use-case
>> > >> > > > > >>> working,
>> > >> > > > > >>> >> but
>> > >> > > > > >>> >> > > it
>> > >> > > > > >>> >> > > > seems to be stuck in "Initialized".
>> > >> > > > > >>> >> > > > this eagle docs has example on how to setup the
>> > app.
>> > >> pls
>> > >> > > let
>> > >> > > > > us
>> > >> > > > > >>> >> know if
>> > >> > > > > >>> >> > > > you find any gaps.
>> > >> > > > > >>> >> > > >
>> > >> > > > > >>> >> > > > Thanks for trying out, and sharing your
>> findings,
>> > >> > > > > >>> >> > > > Jayesh
>> > >> > > > > >>> >> > > >
>> > >> > > > > >>> >> > > >
>> > >> > > > > >>> >> > > > On Tue, Jan 16, 2018 at 3:34 AM, Colm O
>> > hEigeartaigh
>> > >> <
>> > >> > > > > >>> >> > > coheigea@apache.org>
>> > >> > > > > >>> >> > > > wrote:
>> > >> > > > > >>> >> > > >
>> > >> > > > > >>> >> > > >> Hi all,
>> > >> > > > > >>> >> > > >>
>> > >> > > > > >>> >> > > >> I'm trying to play around a bit with Apache
>> Eagle
>> > >> 0.5.0
>> > >> > > to
>> > >> > > > no
>> > >> > > > > >>> >> avail.
>> > >> > > > > >>> >> > > Here
>> > >> > > > > >>> >> > > >> are the problems I've run into so far:
>> > >> > > > > >>> >> > > >>
>> > >> > > > > >>> >> > > >> a) The official docker image uses
>> 0.5.0-SNAPSHOT
>> > and
>> > >> > not
>> > >> > > > the
>> > >> > > > > >>> >> released
>> > >> > > > > >>> >> > > >> version.
>> > >> > > > > >>> >> > > >>
>> > >> > > > > >>> >> > > >> b) Aside from the above, the official docker
>> image
>> > >> > uses a
>> > >> > > > mix
>> > >> > > > > >>> of "
>> > >> > > > > >>> >> > > >> server.eagle.apache.org" and "
>> > >> sandbox.eagle.apache.org
>> > >> > "
>> > >> > > as
>> > >> > > > > the
>> > >> > > > > >>> >> host
>> > >> > > > > >>> >> > > >> name. The HBase service doesn't start by
>> default
>> > in
>> > >> > > Ambari
>> > >> > > > > as a
>> > >> > > > > >>> >> > result.
>> > >> > > > > >>> >> > > >>
>> > >> > > > > >>> >> > > >> c) The UI seems quite buggy. On both chromium
>> and
>> > >> > > firefox,
>> > >> > > > I
>> > >> > > > > >>> only
>> > >> > > > > >>> >> see
>> > >> > > > > >>> >> > > >> links to "Sandbox" and "Alert" on the left
>> > >> hand-side.
>> > >> > > Once
>> > >> > > > I
>> > >> > > > > >>> click
>> > >> > > > > >>> >> on
>> > >> > > > > >>> >> > > >> "Alert" I have no way of going back to see the
>> > >> > > > applications.
>> > >> > > > > I
>> > >> > > > > >>> >> don't
>> > >> > > > > >>> >> > see
>> > >> > > > > >>> >> > > >> the links to "integration" or "sites" as in the
>> > >> picture
>> > >> > > > here:
>> > >> > > > > >>> >> > > >> http://eagle.apache.org/docs/l
>> > >> atest/applications/#jmx-
>> > >> > > > monito
>> > >> > > > > >>> ring
>> > >> > > > > >>> >> > > >>
>> > >> > > > > >>> >> > > >> d) In chromium, the button to create a new
>> policy
>> > >> does
>> > >> > > not
>> > >> > > > > >>> exist -
>> > >> > > > > >>> >> I
>> > >> > > > > >>> >> > can
>> > >> > > > > >>> >> > > >> only see it on Firefox.
>> > >> > > > > >>> >> > > >>
>> > >> > > > > >>> >> > > >> e) I'm trying to get the "Hdfs Audit Log
>> Monitor"
>> > >> > > use-case
>> > >> > > > > >>> working,
>> > >> > > > > >>> >> > but
>> > >> > > > > >>> >> > > >> it seems to be stuck in "Initialized".
>> > >> > > > > >>> >> > > >>
>> > >> > > > > >>> >> > > >> Could someone fill me in on what the
>> "recommended"
>> > >> way
>> > >> > is
>> > >> > > > to
>> > >> > > > > >>> start
>> > >> > > > > >>> >> > > Apache
>> > >> > > > > >>> >> > > >> Eagle so that I can play around with the
>> > >> functionality
>> > >> > > that
>> > >> > > > > it
>> > >> > > > > >>> >> offers?
>> > >> > > > > >>> >> > > >> Clearly the docker approach is buggy. Also,
>> what
>> > >> > browser
>> > >> > > > > >>> should be
>> > >> > > > > >>> >> > used?
>> > >> > > > > >>> >> > > >>
>> > >> > > > > >>> >> > > >> Thanks,
>> > >> > > > > >>> >> > > >>
>> > >> > > > > >>> >> > > >> Colm.
>> > >> > > > > >>> >> > > >>
>> > >> > > > > >>> >> > > >>
>> > >> > > > > >>> >> > > >> --
>> > >> > > > > >>> >> > > >> Colm O hEigeartaigh
>> > >> > > > > >>> >> > > >>
>> > >> > > > > >>> >> > > >> Talend Community Coder
>> > >> > > > > >>> >> > > >> http://coders.talend.com
>> > >> > > > > >>> >> > > >>
>> > >> > > > > >>> >> > > >
>> > >> > > > > >>> >> > > >
>> > >> > > > > >>> >> > >
>> > >> > > > > >>> >> > >
>> > >> > > > > >>> >> > > --
>> > >> > > > > >>> >> > > Colm O hEigeartaigh
>> > >> > > > > >>> >> > >
>> > >> > > > > >>> >> > > Talend Community Coder
>> > >> > > > > >>> >> > > http://coders.talend.com
>> > >> > > > > >>> >> > >
>> > >> > > > > >>> >> >
>> > >> > > > > >>> >>
>> > >> > > > > >>> >>
>> > >> > > > > >>> >>
>> > >> > > > > >>> >> --
>> > >> > > > > >>> >> Colm O hEigeartaigh
>> > >> > > > > >>> >>
>> > >> > > > > >>> >> Talend Community Coder
>> > >> > > > > >>> >> http://coders.talend.com
>> > >> > > > > >>> >>
>> > >> > > > > >>> >>
>> > >> > > > > >>> >>
>> > >> > > > > >>> >
>> > >> > > > > >>> >
>> > >> > > > > >>> > --
>> > >> > > > > >>> > Colm O hEigeartaigh
>> > >> > > > > >>> >
>> > >> > > > > >>> > Talend Community Coder
>> > >> > > > > >>> > http://coders.talend.com
>> > >> > > > > >>> >
>> > >> > > > > >>>
>> > >> > > > > >>>
>> > >> > > > > >>>
>> > >> > > > > >>> --
>> > >> > > > > >>> Colm O hEigeartaigh
>> > >> > > > > >>>
>> > >> > > > > >>> Talend Community Coder
>> > >> > > > > >>> http://coders.talend.com
>> > >> > > > > >>>
>> > >> > > > > >>
>> > >> > > > > >>
>> > >> > > > > >
>> > >> > > > >
>> > >> > > > >
>> > >> > > > > --
>> > >> > > > > Colm O hEigeartaigh
>> > >> > > > >
>> > >> > > > > Talend Community Coder
>> > >> > > > > http://coders.talend.com
>> > >> > > > >
>> > >> > > >
>> > >> > >
>> > >> > >
>> > >> > >
>> > >> > > --
>> > >> > > Colm O hEigeartaigh
>> > >> > >
>> > >> > > Talend Community Coder
>> > >> > > http://coders.talend.com
>> > >> > >
>> > >> >
>> > >>
>> > >>
>> > >>
>> > >> --
>> > >> Colm O hEigeartaigh
>> > >>
>> > >> Talend Community Coder
>> > >> http://coders.talend.com
>> > >>
>> > >
>> > >
>> >
>> >
>> > --
>> > Colm O hEigeartaigh
>> >
>> > Talend Community Coder
>> > http://coders.talend.com
>> >
>>
>
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>

Re: Unable to get 0.5.0 release working

Posted by Colm O hEigeartaigh <co...@apache.org>.
Thanks. I've submitted an initial PR to fix the Slack
ClassNotFoundException issue here:

https://issues.apache.org/jira/browse/EAGLE-879
https://github.com/apache/eagle/pull/984

However, no actual Slack messages are sent. The problem is that
AlertSlackPublisher only sends the message if the event contains a
"severity" that matches the configured "severity" on the publisher. However
the event contains neither "severity" (or "message") so the message never
gets sent. This is for an HDFS audit log - I'm not sure if there are other
scenarios where there is a "severity" column in the event? Either that or
it looks like the SlackPublisher was written before the event column
headers were changed.

Colm.

On Fri, Feb 2, 2018 at 12:54 AM, Jayesh Senjaliya <ja...@apache.org> wrote:

> resolved those tickets now.
>
> I have asked the developer to rebase the PR #941, if he doesnt get to it by
> this week, i will take care of, its long pending one.
> Thanks for verifying though.
>
> - Jayesh
>
> On Thu, Feb 1, 2018 at 8:56 AM, Colm O hEigeartaigh <co...@apache.org>
> wrote:
>
> > Thanks Jayesh. I have two more PRs awaiting review:
> >
> > https://github.com/apache/eagle/pull/981
> > https://github.com/apache/eagle/pull/982
> >
> > Thanks for the JIRA privileges, I can now assign issues to me + change
> the
> > versions. However, I can't "resolve" JIRAs that weren't reported by me
> > which is annoying. These 3 JIRAs should be resolved as they are already
> > merged:
> >
> > https://issues.apache.org/jira/browse/EAGLE-445
> > https://issues.apache.org/jira/browse/EAGLE-476
> > https://issues.apache.org/jira/browse/EAGLE-331
> >
> > In addition, I tested the fix for the Email issue and it works correctly.
> > The PR (https://github.com/apache/eagle/pull/941) just needs to have the
> > extra commits stripped away - I attached a version of the patch on the
> > JIRA.
> >
> > Colm.
> >
> > On Wed, Jan 31, 2018 at 10:08 PM, Jayesh Senjaliya <ja...@apache.org>
> > wrote:
> >
> > > Thanks for the PRs. I have merged them.
> > >
> > > welcome to the developer community Colm. I have also added you to jira
> > > project so can assign the tasks to yourself.
> > >
> > > lets create ticket to fix the dedup functionality, I m actually
> surprised
> > > we havent hit this issue yet. we do use multiple publishers but someone
> > can
> > > verify this.
> > >
> > > Thanks
> > > Jayesh
> > >
> > >
> > >
> > > On Wed, Jan 31, 2018 at 9:25 AM, Colm O hEigeartaigh <
> > coheigea@apache.org>
> > > wrote:
> > >
> > >> Thanks Jayesh. I've created a JIRA here for some admin work for some
> > >> issues
> > >> that were incorrectly flagged as "fix for" 0.5.1/0.6.0:
> > >>
> > >> https://issues.apache.org/jira/browse/EAGLE-1076
> > >>
> > >> I've submitted the following (fairly trivial) pull requests. Could I
> ask
> > >> that you or one of the other committers review?
> > >>
> > >> https://github.com/apache/eagle/pull/978
> > >> https://github.com/apache/eagle/pull/979
> > >> https://github.com/apache/eagle/pull/980
> > >>
> > >> It would be good to try to inject some energy into the project. We
> need
> > >> more than one active committer though.
> > >>
> > >> Just in terms of the Alert Deduplication issue. The
> DefaultDeDuplicator
> > >> works per "output" in the policy rule. So if you have more than one
> > >> AlertPublisher, I think it is guaranteed to only publish to one of
> them.
> > >> Instead, surely it would make more sense to work per publisher?
> > >>
> > >> Colm.
> > >>
> > >> On Tue, Jan 30, 2018 at 10:39 PM, Jayesh Senjaliya <jaysen@apache.org
> >
> > >> wrote:
> > >>
> > >> > Hi Colm,
> > >> >
> > >> > appreciate your suggestions/ efforts in looking into this project,
> > >> > putting my comments inline...
> > >> >
> > >> > a) There is already a JIRA to bump the version here, although the PR
> > >> does
> > >> > not apply as it is too old: https://issues.apache.org/
> > >> > jira/browse/EAGLE-1025
> > >> > .
> > >> > I can submit a new PR, but should the version be 0.6.0 or 0.5.1?
> > >> >
> > >> > *since there are still minor issues, i would say, we put up 0.5.1 as
> > >> next
> > >> > version. I've updated/rebased the PR (
> > >> > https://github.com/apache/eagle/pull/936
> > >> > <https://github.com/apache/eagle/pull/936> )*
> > >> >
> > >> >
> > >> > b) The issues that are "resolved" for the 0.5.1 release in JIRA are
> > >> > actually already fixed in 0.5.0, so they should be updated (
> > >> > https://issues.apache.org/jira/projects/EAGLE/versions/12341128).
> > >> However,
> > >> > the following two issues are resolved even though they are not
> merged
> > to
> > >> > master?
> > >> >   https://issues.apache.org/jira/browse/EAGLE-1051 .  - * this was
> > >> pending
> > >> > from developer;s response but i think this is reviewed, so I have
> > merged
> > >> > it.*
> > >> >   https://issues.apache.org/jira/browse/EAGLE-1068 .  - * this is
> > >> reopened
> > >> > now. I dont think this is done yet. Also this is big change.*
> > >> >
> > >> >
> > >> > Like I said I can submit PRs but I'm not convinced there is any
> > >> activity on
> > >> > the project. Where are the rest of the committers?
> > >> >
> > >> > *let me give you some ocontext on this. so there were lot of
> > development
> > >> > happened during last releases, and most of applications that were
> > added
> > >> are
> > >> > being used in production at multiple enterprise companies, but we
> are
> > >> out
> > >> > of ideas on new apps, so at this point we are only focusing on bug
> > fixes
> > >> > and tech upgrades until we get some new ideas to brainstorm and
> add.*
> > >> >
> > >> > *I think current community's thinking is based on their own
> industries
> > >> > use-cases, but there is definitely room for new features and
> > integration
> > >> > with other monitoring and security components like grafana and
> > rangers.*
> > >> >
> > >> >
> > >> > *Thanks,*
> > >> > *Jayesh*
> > >> >
> > >> >
> > >> >
> > >> > On Tue, Jan 30, 2018 at 8:11 AM, Colm O hEigeartaigh <
> > >> coheigea@apache.org>
> > >> > wrote:
> > >> >
> > >> > > Hi Jayesh,
> > >> > >
> > >> > > Dev suggestions:
> > >> > >
> > >> > > a) There is already a JIRA to bump the version here, although the
> PR
> > >> does
> > >> > > not apply as it is too old: https://issues.apache.org/
> > >> > > jira/browse/EAGLE-1025.
> > >> > > I can submit a new PR, but should the version be 0.6.0 or 0.5.1?
> > >> > > b) The issues that are "resolved" for the 0.5.1 release in JIRA
> are
> > >> > > actually already fixed in 0.5.0, so they should be updated (
> > >> > > https://issues.apache.org/jira/projects/EAGLE/versions/12341128).
> > >> > However,
> > >> > > the following two issues are resolved even though they are not
> > merged
> > >> to
> > >> > > master?
> > >> > >   https://issues.apache.org/jira/browse/EAGLE-1051
> > >> > >   https://issues.apache.org/jira/browse/EAGLE-1068
> > >> > >
> > >> > > Like I said I can submit PRs but I'm not convinced there is any
> > >> activity
> > >> > on
> > >> > > the project. Where are the rest of the committers?
> > >> > >
> > >> > > Multiple Publisher issue:
> > >> > >
> > >> > > If I assign two publishers for one policy, the alert only goes to
> > the
> > >> > first
> > >> > > policy. In the logs I see:
> > >> > >
> > >> > > 2018-01-30T15:52:45.835+0000 o.a.e.a.e.p.d.DefaultDeduplicator
> > [INFO]
> > >> > > Alert
> > >> > > event is skipped because it's duplicated: Alert {site=sandbox,
> > >> > > stream=eagle_output,timestamp=2018-01-30
> > >> > > 00:00:11,300,data={securityZone=NA, dst=null, sensitivityType=NA,
> > >> > > src=/apps/hbase/data/archive/data/default/ambarismoketest,
> > >> allowed=true,
> > >> > > host=172.22.7.129, cmd=listStatus, user=SOMETHING7.COM,
> > >> > > timestamp=1517270411300}, policyId=test,
> > >> > > createdBy=alertBolt3-evaluator_stage1, metaVersion=null}
> > >> > >
> > >> > > It looks like this deduplicator is not working properly, as I'm
> > >> guessing
> > >> > it
> > >> > > should only be used to de-duplicate events for a single publisher?
> > >> > >
> > >> > > Incognito mode: Already tried it but with the same result. Could I
> > ask
> > >> > you
> > >> > > to try the docker image to see if the UI is working correctly for
> > you
> > >> > > there?
> > >> > >
> > >> > > Colm.
> > >> > >
> > >> > > On Mon, Jan 29, 2018 at 6:46 PM, Jayesh Senjaliya <
> > jaysen@apache.org>
> > >> > > wrote:
> > >> > >
> > >> > > > Hi Colm,
> > >> > > >
> > >> > > > Thanks for the list of dev suggestions, I think we should take
> > care
> > >> of
> > >> > > > those. even better if you can provide PR with the changes or at
> > >> keast
> > >> > can
> > >> > > > you please create a ticket so we can track it?
> > >> > > >
> > >> > > > for other issues.
> > >> > > >
> > >> > > > - I dont have any issue with multiple publisher, but if there is
> > any
> > >> > > error
> > >> > > > updating the publisher info in storm topology, i might try
> > >> restarting
> > >> > the
> > >> > > > topology and see if that works.
> > >> > > > - for us, chrome works as fine as firefox.  can u try incognito
> > >> mode?
> > >> > > just
> > >> > > > to be sure to have clean cache?
> > >> > > >
> > >> > > > Thanks
> > >> > > > Jayesh
> > >> > > >
> > >> > > >
> > >> > > > On Thu, Jan 25, 2018 at 4:19 AM, Colm O hEigeartaigh <
> > >> > > coheigea@apache.org>
> > >> > > > wrote:
> > >> > > >
> > >> > > > > Thanks again for your feedback. Jayesh, adding
> > >> AlertEagleStorePlugin
> > >> > > did
> > >> > > > > the trick, I can now see alerts in the UI, thanks! By the
> way, I
> > >> > can't
> > >> > > > > configure two Alert Publishers, or else the Alert DeDuplicator
> > >> bins
> > >> > the
> > >> > > > > alert. Is this a known issue?
> > >> > > > >
> > >> > > > > Could I ask which browser people are using with the UI? There
> > >> appears
> > >> > > to
> > >> > > > be
> > >> > > > > a  bug with Chromium where it doesn't list the pages under
> > >> > Auth.isAdmin
> > >> > > > > even though I am logged on as an administrator. It works OK in
> > >> > Firefox.
> > >> > > > > Even with Firefox though, I only see a limited number of links
> > in
> > >> the
> > >> > > > > left-hand column - I can't get back to the "integration" page.
> > Can
> > >> > > > someone
> > >> > > > > else confirm this please?
> > >> > > > >
> > >> > > > > Could I suggest the devs do some basic house-keeping tasks:
> > >> > > > >
> > >> > > > > a) "Release" version 0.5.0 in JIRA (it's still listed as
> > >> > "unreleased").
> > >> > > > > b) Figure out whether the next version will be 0.5.1 or 0.6.0
> > and
> > >> > > update
> > >> > > > > the versions on Master accordingly with 0.5.1-SNAPSHOT or
> > >> > > 0.6.0-SNAPSHOT.
> > >> > > > > There are some issues marked here as resolved for 0.5.1 -
> > >> > > > > https://issues.apache.org/jira/projects/EAGLE/versions/
> 12341128
> > ),
> > >> > > > however
> > >> > > > > I
> > >> > > > > don't see a branch for 0.5.x?
> > >> > > > >
> > >> > > > > Colm.
> > >> > > > >
> > >> > > > > On Thu, Jan 25, 2018 at 8:16 AM, Jayesh Senjaliya <
> > >> jaysen@apache.org
> > >> > >
> > >> > > > > wrote:
> > >> > > > >
> > >> > > > > > Hi,
> > >> > > > > >
> > >> > > > > > we do use eagle 0.5 in production although we dont use all
> the
> > >> > > > available
> > >> > > > > > hadoop applications.
> > >> > > > > >
> > >> > > > > > EAGLE-968 <https://issues.apache.org/jira/browse/EAGLE-968>
> > is
> > >> a
> > >> > fix
> > >> > > > for
> > >> > > > > > email issue we found while our testing. should be merged
> soon
> > >> > after a
> > >> > > > > > rebase.
> > >> > > > > >
> > >> > > > > > @Colm, did you tried adding storage publisher
> > >> > > (AlertEagleStorePlugin)?
> > >> > > > to
> > >> > > > > > see alerts on UI ?
> > >> > > > > >
> > >> > > > > > Thanks
> > >> > > > > > Jayesh
> > >> > > > > >
> > >> > > > > >
> > >> > > > > >
> > >> > > > > >
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > On Wed, Jan 24, 2018 at 7:08 PM, Edward Zhang <
> > >> > > yonzhang2012@gmail.com>
> > >> > > > > > wrote:
> > >> > > > > >
> > >> > > > > >> Eagle 0.5 was deployed in production as far as I know, but
> it
> > >> may
> > >> > > not
> > >> > > > be
> > >> > > > > >> exact the current version in master branch.
> > >> > > > > >>
> > >> > > > > >> Thanks for your investigation, seems there is still some
> bug
> > in
> > >> > 0.5,
> > >> > > > but
> > >> > > > > >> this particular issue seems is due to dependent components
> > >> version
> > >> > > > > conflict.
> > >> > > > > >>
> > >> > > > > >> @Jayesh is this Jira ready for merge to master?
> > >> > > https://issues.apache
> > >> > > > .
> > >> > > > > >> org/jira/browse/EAGLE-968
> > >> > > > > >>
> > >> > > > > >>
> > >> > > > > >> Thanks
> > >> > > > > >> Edward
> > >> > > > > >>
> > >> > > > > >> On Tue, Jan 23, 2018 at 5:10 AM, Colm O hEigeartaigh <
> > >> > > > > coheigea@apache.org
> > >> > > > > >> > wrote:
> > >> > > > > >>
> > >> > > > > >>> OK I've made some more progress. I wasn't seeing any email
> > >> alerts
> > >> > > due
> > >> > > > > to
> > >> > > > > >>> https://issues.apache.org/jira/browse/EAGLE-968. Once I
> > >> > configure
> > >> > > a
> > >> > > > > >>> Kafka
> > >> > > > > >>> alert, I can see the alerts flowing into my topic. It's
> > still
> > >> not
> > >> > > > clear
> > >> > > > > >>> to
> > >> > > > > >>> me however where the policy "output" is going. I also
> don't
> > >> see
> > >> > any
> > >> > > > > >>> alerts
> > >> > > > > >>> in the UI window.
> > >> > > > > >>>
> > >> > > > > >>> Could I ask what the status of the project is in general?
> > >> There
> > >> > > have
> > >> > > > > been
> > >> > > > > >>> no commits to master since November, so I'm not sure if
> > there
> > >> is
> > >> > > any
> > >> > > > > >>> point
> > >> > > > > >>> in submitting Pull Requests for outstanding bugs? Are
> recent
> > >> > > versions
> > >> > > > > of
> > >> > > > > >>> Apache Eagle used in production?
> > >> > > > > >>>
> > >> > > > > >>> Colm.
> > >> > > > > >>>
> > >> > > > > >>> On Mon, Jan 22, 2018 at 1:07 PM, Colm O hEigeartaigh <
> > >> > > > > >>> coheigea@apache.org>
> > >> > > > > >>> wrote:
> > >> > > > > >>>
> > >> > > > > >>> >
> > >> > > > > >>> > I've done that but I'm not seeing any alerts, which is
> > why I
> > >> > want
> > >> > > > to
> > >> > > > > >>> find
> > >> > > > > >>> > out what the "output" of a policy is and where I can
> check
> > >> > this.
> > >> > > > > >>> >
> > >> > > > > >>> > Colm.
> > >> > > > > >>> >
> > >> > > > > >>> > On Mon, Jan 22, 2018 at 1:05 PM, SUDHA JENSLIN <
> > >> > > sjenslin@gmail.com
> > >> > > > >
> > >> > > > > >>> wrote:
> > >> > > > > >>> >
> > >> > > > > >>> >> Create and add a publisher to see the output.
> > >> > > > > >>> >>
> > >> > > > > >>> >>
> > >> > > > > >>> >>
> > >> > > > > >>> >> Regards,
> > >> > > > > >>> >> Sudha jenslin
> > >> > > > > >>> >>
> > >> > > > > >>> >> On Jan 22, 2018 6:31 PM, "Colm O hEigeartaigh" <
> > >> > > > coheigea@apache.org
> > >> > > > > >
> > >> > > > > >>> >> wrote:
> > >> > > > > >>> >>
> > >> > > > > >>> >> Thanks - the error was due to a problem running Storm
> > with
> > >> > Java
> > >> > > > 1.8.
> > >> > > > > >>> I've
> > >> > > > > >>> >> abandoned the docker image for now, and I'm trying to
> get
> > >> it
> > >> > > > working
> > >> > > > > >>> >> locally.
> > >> > > > > >>> >>
> > >> > > > > >>> >> There are two things I'm not clear on currently, if
> > someone
> > >> > > could
> > >> > > > > >>> fill me
> > >> > > > > >>> >> in:
> > >> > > > > >>> >>
> > >> > > > > >>> >> a) For the  'Hdfs Audit Log Monitor' application, the
> > Kafka
> > >> > > > Consumer
> > >> > > > > >>> Topic
> > >> > > > > >>> >> is 'hdfs_audit_log_sandbox'. Under 'Kafka Topic for
> > >> Auditlog
> > >> > > Event
> > >> > > > > >>> Sink'
> > >> > > > > >>> >> it
> > >> > > > > >>> >> also specifies 'hdfs_audit_event_sandbox'. However the
> > >> > > > documentation
> > >> > > > > >>> for
> > >> > > > > >>> >> the application mentions 'hdfs_audit_log_enriched_sandb
> > >> ox'?
> > >> > > > > >>> >>
> > >> > > > > >>> >> When I click on "STREAMS", the
> > >> "HDFS_AUDIT_LOG_ENRICHED_STREA
> > >> > > > > >>> M_SANDBOX"
> > >> > > > > >>> >> uses the topic "hdfs_audit_event_sandbox". And indeed
> > when
> > >> I
> > >> > run
> > >> > > > the
> > >> > > > > >>> >> application, I can see cleansed log data appearing in
> > >> > > > > >>> >> "hdfs_audit_event_sandbox". So I'm thinking here that
> > >> > > > > >>> >> 'hdfs_audit_log_enriched_sandbox' is not correct or
> > >> > necessary?
> > >> > > > > >>> >>
> > >> > > > > >>> >> b) It's unclear to me where the output data goes when
> you
> > >> > > create a
> > >> > > > > >>> policy.
> > >> > > > > >>> >> E.g. say I have:
> > >> > > > > >>> >>
> > >> > > > > >>> >> from HDFS_AUDIT_LOG_ENRICHED_
> > STREAM_SANDBOX[str:contains(
> > >> > > src,'/hb
> > >> > > > > >>> ase')]
> > >> > > > > >>> >> select * group by user insert into
> > hdfs_audit_log_enriched_
> > >> > > > > stream_out
> > >> > > > > >>> >>
> > >> > > > > >>> >> Where is "hdfs_audit_log_enriched_stream_out" defined
> > (is
> > >> it
> > >> > a
> > >> > > > > Kafka
> > >> > > > > >>> >> topic?). How can I check the output to make sure the
> > >> policy is
> > >> > > > > working
> > >> > > > > >>> >> correctly?
> > >> > > > > >>> >>
> > >> > > > > >>> >> Thanks,
> > >> > > > > >>> >>
> > >> > > > > >>> >> Colm.
> > >> > > > > >>> >>
> > >> > > > > >>> >> On Wed, Jan 17, 2018 at 10:32 PM, Edward Zhang <
> > >> > > > > >>> yonzhang2012@gmail.com>
> > >> > > > > >>> >> wrote:
> > >> > > > > >>> >>
> > >> > > > > >>> >> > There is a data preparation stage between data
> > >> source(HDFS
> > >> > > audit
> > >> > > > > >>> log)
> > >> > > > > >>> >> and
> > >> > > > > >>> >> > Alert Engine. This stage is running in Storm and
> > >> transform
> > >> > the
> > >> > > > raw
> > >> > > > > >>> HDFS
> > >> > > > > >>> >> log
> > >> > > > > >>> >> > into something which can be alerted.
> > >> > > > > >>> >> >
> > >> > > > > >>> >> > The input for data preparation is
> > hdfs_audit_log_sandbox
> > >> > topic
> > >> > > > and
> > >> > > > > >>> >> output
> > >> > > > > >>> >> > is
> > >> > > > > >>> >> >  hdfs_audit_log_enriched_sandbox.
> > >> > > > > >>> >> > The input for Alert Engine is
> hdfs_audit_log_enriched_
> > >> > sandbox
> > >> > > > and
> > >> > > > > >>> >> output
> > >> > > > > >>> >> > is
> > >> > > > > >>> >> > hdfs_audit_log_alert_sandbox.
> > >> > > > > >>> >> >
> > >> > > > > >>> >> > Seems in your case, the data preparation staging is
> not
> > >> > > working.
> > >> > > > > We
> > >> > > > > >>> >> > probably need look at Storm console and figure out if
> > >> that
> > >> > > part
> > >> > > > is
> > >> > > > > >>> >> working.
> > >> > > > > >>> >> >
> > >> > > > > >>> >> > Thanks
> > >> > > > > >>> >> > Edward
> > >> > > > > >>> >> >
> > >> > > > > >>> >> > On Wed, Jan 17, 2018 at 7:19 AM, Colm O hEigeartaigh
> <
> > >> > > > > >>> >> coheigea@apache.org>
> > >> > > > > >>> >> > wrote:
> > >> > > > > >>> >> >
> > >> > > > > >>> >> > > Hi Jayesh,
> > >> > > > > >>> >> > >
> > >> > > > > >>> >> > > Many thanks for your feedback! I was able to make a
> > >> little
> > >> > > > > further
> > >> > > > > >>> >> > headway.
> > >> > > > > >>> >> > > There are two configuration problems with the
> > official
> > >> > > docker
> > >> > > > > >>> image:
> > >> > > > > >>> >> > >
> > >> > > > > >>> >> > > a) A mix of "sandbox.eagle.apache.org" and "
> > >> > > > > >>> server.eagle.apache.org"
> > >> > > > > >>> >> > (this
> > >> > > > > >>> >> > > only occurs in the instructions for running the
> > docker
> > >> > > image.
> > >> > > > > The
> > >> > > > > >>> >> version
> > >> > > > > >>> >> > > that can be started via the script in the eagle
> > source
> > >> is
> > >> > > OK).
> > >> > > > > >>> I'll
> > >> > > > > >>> >> > submit
> > >> > > > > >>> >> > > a PR to fix this once I get a basic use-case
> working.
> > >> > > > > >>> >> > > b) For the audit case, it automatically logs HDFS
> > audit
> > >> > logs
> > >> > > > to
> > >> > > > > >>> the
> > >> > > > > >>> >> KAFKA
> > >> > > > > >>> >> > > topic sandbox_hdfs_audit_log instead of the
> expected
> > >> > > > > >>> >> > hdfs_audit_log_sandbox
> > >> > > > > >>> >> > >
> > >> > > > > >>> >> > > I've fixed these things locally and I can verify
> that
> > >> > > > everything
> > >> > > > > >>> is
> > >> > > > > >>> >> > started
> > >> > > > > >>> >> > > correctly in Ambari. I log into the docker
> container
> > >> and
> > >> > > > create
> > >> > > > > >>> >> > > hdfs_audit_log_sandbox and hdfs_audit_log_enriched_
> > >> > sandbox
> > >> > > > > >>> topics,
> > >> > > > > >>> >> and
> > >> > > > > >>> >> > > verify that the HDFS audit logs are flowing into
> the
> > >> first
> > >> > > > > topic.
> > >> > > > > >>> >> Then in
> > >> > > > > >>> >> > > the UI I start the Alert Engine and then the HDFS
> > Audit
> > >> > Log
> > >> > > > > >>> Monitor
> > >> > > > > >>> >> > > application (changing localhost:6667 to
> > >> > > > > >>> server.eagle.apache.org:6667
> > >> > > > > >>> >> ).
> > >> > > > > >>> >> > > Both
> > >> > > > > >>> >> > > applications start up correctly and show "running".
> > >> > > > > >>> >> > >
> > >> > > > > >>> >> > > I then create a policy with an email alert along
> the
> > >> lines
> > >> > > of
> > >> > > > > from
> > >> > > > > >>> >> > > "HDFS_AUDIT_LOG_ENRICHED_
> > STREAM_SANDBOX[str:contains(
> > >> > > src,'/h
> > >> > > > > >>> base')]
> > >> > > > > >>> >> > select
> > >> > > > > >>> >> > > * group by user insert into
> hdfs_audit_log_enriched_
> > >> > > > > stream_out".
> > >> > > > > >>> >> However
> > >> > > > > >>> >> > > at
> > >> > > > > >>> >> > > this point I'm stuck - nothing appears in the alert
> > >> > window.
> > >> > > Is
> > >> > > > > >>> there
> > >> > > > > >>> >> > > anything obvious I'm doing wrong, or how can I get
> > >> access
> > >> > to
> > >> > > > > logs
> > >> > > > > >>> to
> > >> > > > > >>> >> > figure
> > >> > > > > >>> >> > > out what the problem is? Other topics such as
> > >> > > > > >>> >> "hdfs_audit_event_sandbox"
> > >> > > > > >>> >> > > are mentioned in the streams window, but the
> > >> documentation
> > >> > > > > doesn't
> > >> > > > > >>> >> say to
> > >> > > > > >>> >> > > create them.
> > >> > > > > >>> >> > >
> > >> > > > > >>> >> > > The UI is buggy though on both Firefox and Chromium
> > on
> > >> > > Linux.
> > >> > > > > What
> > >> > > > > >>> >> > > browser/platform are people using with the UI?
> > >> > > > > >>> >> > >
> > >> > > > > >>> >> > > Colm.
> > >> > > > > >>> >> > >
> > >> > > > > >>> >> > > On Wed, Jan 17, 2018 at 12:27 AM, Jayesh Senjaliya
> <
> > >> > > > > >>> jaysen@apache.org
> > >> > > > > >>> >> >
> > >> > > > > >>> >> > > wrote:
> > >> > > > > >>> >> > >
> > >> > > > > >>> >> > > > Hi Colm,
> > >> > > > > >>> >> > > >
> > >> > > > > >>> >> > > > Please find my comments inline.
> > >> > > > > >>> >> > > >
> > >> > > > > >>> >> > > > a) The official docker image uses 0.5.0-SNAPSHOT
> > and
> > >> not
> > >> > > the
> > >> > > > > >>> >> released
> > >> > > > > >>> >> > > > version.
> > >> > > > > >>> >> > > > - this is because we uploaded docker image before
> > >> apache
> > >> > > > > >>> release.
> > >> > > > > >>> >> > > actually
> > >> > > > > >>> >> > > > this is same codebase apache-eagle-0.5, and it
> can
> > be
> > >> > > fixed
> > >> > > > > >>> easily
> > >> > > > > >>> >> by
> > >> > > > > >>> >> > > just
> > >> > > > > >>> >> > > > rebuilding docker image. there should not be any
> > >> > mismatch
> > >> > > > due
> > >> > > > > to
> > >> > > > > >>> >> this.
> > >> > > > > >>> >> > > >
> > >> > > > > >>> >> > > > b) Aside from the above, the official docker
> image
> > >> uses
> > >> > a
> > >> > > > mix
> > >> > > > > >>> of "
> > >> > > > > >>> >> > > > server.eagle.apache.org" and "
> > >> sandbox.eagle.apache.org"
> > >> > > as
> > >> > > > > the
> > >> > > > > >>> host
> > >> > > > > >>> >> > > name.
> > >> > > > > >>> >> > > > The HBase service doesn't start by default in
> > Ambari
> > >> as
> > >> > a
> > >> > > > > >>> result.
> > >> > > > > >>> >> > > > - the only places it uses sandbox is in example
> > >> script
> > >> > > which
> > >> > > > > you
> > >> > > > > >>> >> will
> > >> > > > > >>> >> > > have
> > >> > > > > >>> >> > > > to update anyway, which i agree that it would be
> > >> good to
> > >> > > > keep
> > >> > > > > it
> > >> > > > > >>> >> > > > consistent.
> > >> > > > > >>> >> > > >
> > >> > > > > >>> >> > > > c) The UI seems quite buggy. On both chromium and
> > >> > > firefox, I
> > >> > > > > >>> only
> > >> > > > > >>> >> see
> > >> > > > > >>> >> > > > links to "Sandbox" and "Alert" on the left
> > hand-side.
> > >> > > Once I
> > >> > > > > >>> click
> > >> > > > > >>> >> on
> > >> > > > > >>> >> > > > "Alert" I have no way of going back to see the
> > >> > > > applications. I
> > >> > > > > >>> don't
> > >> > > > > >>> >> > see
> > >> > > > > >>> >> > > > the links to "integration" or "sites" as in the
> > >> picture
> > >> > > > here:
> > >> > > > > >>> >> > > > http://eagle.apache.org/docs/l
> > >> atest/applications/#jmx-
> > >> > > > monito
> > >> > > > > >>> ring
> > >> > > > > >>> >> > > > - when hbase is as deep storage is used, and if
> > eagle
> > >> > app
> > >> > > > has
> > >> > > > > >>> issue
> > >> > > > > >>> >> > > > connecting to hbase, the UI becomes unresponsive.
> > >> > > > > >>> >> > > >
> > >> > > > > >>> >> > > > d) In chromium, the button to create a new policy
> > >> does
> > >> > not
> > >> > > > > >>> exist - I
> > >> > > > > >>> >> > can
> > >> > > > > >>> >> > > > only see it on Firefox.
> > >> > > > > >>> >> > > > - i have seen when you logged in, you will see
> > admin
> > >> > > > actions.
> > >> > > > > >>> but if
> > >> > > > > >>> >> > this
> > >> > > > > >>> >> > > > still an issue, can you please file UI bug?
> > >> > > > > >>> >> > > >
> > >> > > > > >>> >> > > > e) I'm trying to get the "Hdfs Audit Log Monitor"
> > >> > use-case
> > >> > > > > >>> working,
> > >> > > > > >>> >> but
> > >> > > > > >>> >> > > it
> > >> > > > > >>> >> > > > seems to be stuck in "Initialized".
> > >> > > > > >>> >> > > > this eagle docs has example on how to setup the
> > app.
> > >> pls
> > >> > > let
> > >> > > > > us
> > >> > > > > >>> >> know if
> > >> > > > > >>> >> > > > you find any gaps.
> > >> > > > > >>> >> > > >
> > >> > > > > >>> >> > > > Thanks for trying out, and sharing your findings,
> > >> > > > > >>> >> > > > Jayesh
> > >> > > > > >>> >> > > >
> > >> > > > > >>> >> > > >
> > >> > > > > >>> >> > > > On Tue, Jan 16, 2018 at 3:34 AM, Colm O
> > hEigeartaigh
> > >> <
> > >> > > > > >>> >> > > coheigea@apache.org>
> > >> > > > > >>> >> > > > wrote:
> > >> > > > > >>> >> > > >
> > >> > > > > >>> >> > > >> Hi all,
> > >> > > > > >>> >> > > >>
> > >> > > > > >>> >> > > >> I'm trying to play around a bit with Apache
> Eagle
> > >> 0.5.0
> > >> > > to
> > >> > > > no
> > >> > > > > >>> >> avail.
> > >> > > > > >>> >> > > Here
> > >> > > > > >>> >> > > >> are the problems I've run into so far:
> > >> > > > > >>> >> > > >>
> > >> > > > > >>> >> > > >> a) The official docker image uses 0.5.0-SNAPSHOT
> > and
> > >> > not
> > >> > > > the
> > >> > > > > >>> >> released
> > >> > > > > >>> >> > > >> version.
> > >> > > > > >>> >> > > >>
> > >> > > > > >>> >> > > >> b) Aside from the above, the official docker
> image
> > >> > uses a
> > >> > > > mix
> > >> > > > > >>> of "
> > >> > > > > >>> >> > > >> server.eagle.apache.org" and "
> > >> sandbox.eagle.apache.org
> > >> > "
> > >> > > as
> > >> > > > > the
> > >> > > > > >>> >> host
> > >> > > > > >>> >> > > >> name. The HBase service doesn't start by default
> > in
> > >> > > Ambari
> > >> > > > > as a
> > >> > > > > >>> >> > result.
> > >> > > > > >>> >> > > >>
> > >> > > > > >>> >> > > >> c) The UI seems quite buggy. On both chromium
> and
> > >> > > firefox,
> > >> > > > I
> > >> > > > > >>> only
> > >> > > > > >>> >> see
> > >> > > > > >>> >> > > >> links to "Sandbox" and "Alert" on the left
> > >> hand-side.
> > >> > > Once
> > >> > > > I
> > >> > > > > >>> click
> > >> > > > > >>> >> on
> > >> > > > > >>> >> > > >> "Alert" I have no way of going back to see the
> > >> > > > applications.
> > >> > > > > I
> > >> > > > > >>> >> don't
> > >> > > > > >>> >> > see
> > >> > > > > >>> >> > > >> the links to "integration" or "sites" as in the
> > >> picture
> > >> > > > here:
> > >> > > > > >>> >> > > >> http://eagle.apache.org/docs/l
> > >> atest/applications/#jmx-
> > >> > > > monito
> > >> > > > > >>> ring
> > >> > > > > >>> >> > > >>
> > >> > > > > >>> >> > > >> d) In chromium, the button to create a new
> policy
> > >> does
> > >> > > not
> > >> > > > > >>> exist -
> > >> > > > > >>> >> I
> > >> > > > > >>> >> > can
> > >> > > > > >>> >> > > >> only see it on Firefox.
> > >> > > > > >>> >> > > >>
> > >> > > > > >>> >> > > >> e) I'm trying to get the "Hdfs Audit Log
> Monitor"
> > >> > > use-case
> > >> > > > > >>> working,
> > >> > > > > >>> >> > but
> > >> > > > > >>> >> > > >> it seems to be stuck in "Initialized".
> > >> > > > > >>> >> > > >>
> > >> > > > > >>> >> > > >> Could someone fill me in on what the
> "recommended"
> > >> way
> > >> > is
> > >> > > > to
> > >> > > > > >>> start
> > >> > > > > >>> >> > > Apache
> > >> > > > > >>> >> > > >> Eagle so that I can play around with the
> > >> functionality
> > >> > > that
> > >> > > > > it
> > >> > > > > >>> >> offers?
> > >> > > > > >>> >> > > >> Clearly the docker approach is buggy. Also, what
> > >> > browser
> > >> > > > > >>> should be
> > >> > > > > >>> >> > used?
> > >> > > > > >>> >> > > >>
> > >> > > > > >>> >> > > >> Thanks,
> > >> > > > > >>> >> > > >>
> > >> > > > > >>> >> > > >> Colm.
> > >> > > > > >>> >> > > >>
> > >> > > > > >>> >> > > >>
> > >> > > > > >>> >> > > >> --
> > >> > > > > >>> >> > > >> Colm O hEigeartaigh
> > >> > > > > >>> >> > > >>
> > >> > > > > >>> >> > > >> Talend Community Coder
> > >> > > > > >>> >> > > >> http://coders.talend.com
> > >> > > > > >>> >> > > >>
> > >> > > > > >>> >> > > >
> > >> > > > > >>> >> > > >
> > >> > > > > >>> >> > >
> > >> > > > > >>> >> > >
> > >> > > > > >>> >> > > --
> > >> > > > > >>> >> > > Colm O hEigeartaigh
> > >> > > > > >>> >> > >
> > >> > > > > >>> >> > > Talend Community Coder
> > >> > > > > >>> >> > > http://coders.talend.com
> > >> > > > > >>> >> > >
> > >> > > > > >>> >> >
> > >> > > > > >>> >>
> > >> > > > > >>> >>
> > >> > > > > >>> >>
> > >> > > > > >>> >> --
> > >> > > > > >>> >> Colm O hEigeartaigh
> > >> > > > > >>> >>
> > >> > > > > >>> >> Talend Community Coder
> > >> > > > > >>> >> http://coders.talend.com
> > >> > > > > >>> >>
> > >> > > > > >>> >>
> > >> > > > > >>> >>
> > >> > > > > >>> >
> > >> > > > > >>> >
> > >> > > > > >>> > --
> > >> > > > > >>> > Colm O hEigeartaigh
> > >> > > > > >>> >
> > >> > > > > >>> > Talend Community Coder
> > >> > > > > >>> > http://coders.talend.com
> > >> > > > > >>> >
> > >> > > > > >>>
> > >> > > > > >>>
> > >> > > > > >>>
> > >> > > > > >>> --
> > >> > > > > >>> Colm O hEigeartaigh
> > >> > > > > >>>
> > >> > > > > >>> Talend Community Coder
> > >> > > > > >>> http://coders.talend.com
> > >> > > > > >>>
> > >> > > > > >>
> > >> > > > > >>
> > >> > > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > > --
> > >> > > > > Colm O hEigeartaigh
> > >> > > > >
> > >> > > > > Talend Community Coder
> > >> > > > > http://coders.talend.com
> > >> > > > >
> > >> > > >
> > >> > >
> > >> > >
> > >> > >
> > >> > > --
> > >> > > Colm O hEigeartaigh
> > >> > >
> > >> > > Talend Community Coder
> > >> > > http://coders.talend.com
> > >> > >
> > >> >
> > >>
> > >>
> > >>
> > >> --
> > >> Colm O hEigeartaigh
> > >>
> > >> Talend Community Coder
> > >> http://coders.talend.com
> > >>
> > >
> > >
> >
> >
> > --
> > Colm O hEigeartaigh
> >
> > Talend Community Coder
> > http://coders.talend.com
> >
>



-- 
Colm O hEigeartaigh

Talend Community Coder
http://coders.talend.com

Re: Unable to get 0.5.0 release working

Posted by Jayesh Senjaliya <ja...@apache.org>.
resolved those tickets now.

I have asked the developer to rebase the PR #941, if he doesnt get to it by
this week, i will take care of, its long pending one.
Thanks for verifying though.

- Jayesh

On Thu, Feb 1, 2018 at 8:56 AM, Colm O hEigeartaigh <co...@apache.org>
wrote:

> Thanks Jayesh. I have two more PRs awaiting review:
>
> https://github.com/apache/eagle/pull/981
> https://github.com/apache/eagle/pull/982
>
> Thanks for the JIRA privileges, I can now assign issues to me + change the
> versions. However, I can't "resolve" JIRAs that weren't reported by me
> which is annoying. These 3 JIRAs should be resolved as they are already
> merged:
>
> https://issues.apache.org/jira/browse/EAGLE-445
> https://issues.apache.org/jira/browse/EAGLE-476
> https://issues.apache.org/jira/browse/EAGLE-331
>
> In addition, I tested the fix for the Email issue and it works correctly.
> The PR (https://github.com/apache/eagle/pull/941) just needs to have the
> extra commits stripped away - I attached a version of the patch on the
> JIRA.
>
> Colm.
>
> On Wed, Jan 31, 2018 at 10:08 PM, Jayesh Senjaliya <ja...@apache.org>
> wrote:
>
> > Thanks for the PRs. I have merged them.
> >
> > welcome to the developer community Colm. I have also added you to jira
> > project so can assign the tasks to yourself.
> >
> > lets create ticket to fix the dedup functionality, I m actually surprised
> > we havent hit this issue yet. we do use multiple publishers but someone
> can
> > verify this.
> >
> > Thanks
> > Jayesh
> >
> >
> >
> > On Wed, Jan 31, 2018 at 9:25 AM, Colm O hEigeartaigh <
> coheigea@apache.org>
> > wrote:
> >
> >> Thanks Jayesh. I've created a JIRA here for some admin work for some
> >> issues
> >> that were incorrectly flagged as "fix for" 0.5.1/0.6.0:
> >>
> >> https://issues.apache.org/jira/browse/EAGLE-1076
> >>
> >> I've submitted the following (fairly trivial) pull requests. Could I ask
> >> that you or one of the other committers review?
> >>
> >> https://github.com/apache/eagle/pull/978
> >> https://github.com/apache/eagle/pull/979
> >> https://github.com/apache/eagle/pull/980
> >>
> >> It would be good to try to inject some energy into the project. We need
> >> more than one active committer though.
> >>
> >> Just in terms of the Alert Deduplication issue. The DefaultDeDuplicator
> >> works per "output" in the policy rule. So if you have more than one
> >> AlertPublisher, I think it is guaranteed to only publish to one of them.
> >> Instead, surely it would make more sense to work per publisher?
> >>
> >> Colm.
> >>
> >> On Tue, Jan 30, 2018 at 10:39 PM, Jayesh Senjaliya <ja...@apache.org>
> >> wrote:
> >>
> >> > Hi Colm,
> >> >
> >> > appreciate your suggestions/ efforts in looking into this project,
> >> > putting my comments inline...
> >> >
> >> > a) There is already a JIRA to bump the version here, although the PR
> >> does
> >> > not apply as it is too old: https://issues.apache.org/
> >> > jira/browse/EAGLE-1025
> >> > .
> >> > I can submit a new PR, but should the version be 0.6.0 or 0.5.1?
> >> >
> >> > *since there are still minor issues, i would say, we put up 0.5.1 as
> >> next
> >> > version. I've updated/rebased the PR (
> >> > https://github.com/apache/eagle/pull/936
> >> > <https://github.com/apache/eagle/pull/936> )*
> >> >
> >> >
> >> > b) The issues that are "resolved" for the 0.5.1 release in JIRA are
> >> > actually already fixed in 0.5.0, so they should be updated (
> >> > https://issues.apache.org/jira/projects/EAGLE/versions/12341128).
> >> However,
> >> > the following two issues are resolved even though they are not merged
> to
> >> > master?
> >> >   https://issues.apache.org/jira/browse/EAGLE-1051 .  - * this was
> >> pending
> >> > from developer;s response but i think this is reviewed, so I have
> merged
> >> > it.*
> >> >   https://issues.apache.org/jira/browse/EAGLE-1068 .  - * this is
> >> reopened
> >> > now. I dont think this is done yet. Also this is big change.*
> >> >
> >> >
> >> > Like I said I can submit PRs but I'm not convinced there is any
> >> activity on
> >> > the project. Where are the rest of the committers?
> >> >
> >> > *let me give you some ocontext on this. so there were lot of
> development
> >> > happened during last releases, and most of applications that were
> added
> >> are
> >> > being used in production at multiple enterprise companies, but we are
> >> out
> >> > of ideas on new apps, so at this point we are only focusing on bug
> fixes
> >> > and tech upgrades until we get some new ideas to brainstorm and add.*
> >> >
> >> > *I think current community's thinking is based on their own industries
> >> > use-cases, but there is definitely room for new features and
> integration
> >> > with other monitoring and security components like grafana and
> rangers.*
> >> >
> >> >
> >> > *Thanks,*
> >> > *Jayesh*
> >> >
> >> >
> >> >
> >> > On Tue, Jan 30, 2018 at 8:11 AM, Colm O hEigeartaigh <
> >> coheigea@apache.org>
> >> > wrote:
> >> >
> >> > > Hi Jayesh,
> >> > >
> >> > > Dev suggestions:
> >> > >
> >> > > a) There is already a JIRA to bump the version here, although the PR
> >> does
> >> > > not apply as it is too old: https://issues.apache.org/
> >> > > jira/browse/EAGLE-1025.
> >> > > I can submit a new PR, but should the version be 0.6.0 or 0.5.1?
> >> > > b) The issues that are "resolved" for the 0.5.1 release in JIRA are
> >> > > actually already fixed in 0.5.0, so they should be updated (
> >> > > https://issues.apache.org/jira/projects/EAGLE/versions/12341128).
> >> > However,
> >> > > the following two issues are resolved even though they are not
> merged
> >> to
> >> > > master?
> >> > >   https://issues.apache.org/jira/browse/EAGLE-1051
> >> > >   https://issues.apache.org/jira/browse/EAGLE-1068
> >> > >
> >> > > Like I said I can submit PRs but I'm not convinced there is any
> >> activity
> >> > on
> >> > > the project. Where are the rest of the committers?
> >> > >
> >> > > Multiple Publisher issue:
> >> > >
> >> > > If I assign two publishers for one policy, the alert only goes to
> the
> >> > first
> >> > > policy. In the logs I see:
> >> > >
> >> > > 2018-01-30T15:52:45.835+0000 o.a.e.a.e.p.d.DefaultDeduplicator
> [INFO]
> >> > > Alert
> >> > > event is skipped because it's duplicated: Alert {site=sandbox,
> >> > > stream=eagle_output,timestamp=2018-01-30
> >> > > 00:00:11,300,data={securityZone=NA, dst=null, sensitivityType=NA,
> >> > > src=/apps/hbase/data/archive/data/default/ambarismoketest,
> >> allowed=true,
> >> > > host=172.22.7.129, cmd=listStatus, user=SOMETHING7.COM,
> >> > > timestamp=1517270411300}, policyId=test,
> >> > > createdBy=alertBolt3-evaluator_stage1, metaVersion=null}
> >> > >
> >> > > It looks like this deduplicator is not working properly, as I'm
> >> guessing
> >> > it
> >> > > should only be used to de-duplicate events for a single publisher?
> >> > >
> >> > > Incognito mode: Already tried it but with the same result. Could I
> ask
> >> > you
> >> > > to try the docker image to see if the UI is working correctly for
> you
> >> > > there?
> >> > >
> >> > > Colm.
> >> > >
> >> > > On Mon, Jan 29, 2018 at 6:46 PM, Jayesh Senjaliya <
> jaysen@apache.org>
> >> > > wrote:
> >> > >
> >> > > > Hi Colm,
> >> > > >
> >> > > > Thanks for the list of dev suggestions, I think we should take
> care
> >> of
> >> > > > those. even better if you can provide PR with the changes or at
> >> keast
> >> > can
> >> > > > you please create a ticket so we can track it?
> >> > > >
> >> > > > for other issues.
> >> > > >
> >> > > > - I dont have any issue with multiple publisher, but if there is
> any
> >> > > error
> >> > > > updating the publisher info in storm topology, i might try
> >> restarting
> >> > the
> >> > > > topology and see if that works.
> >> > > > - for us, chrome works as fine as firefox.  can u try incognito
> >> mode?
> >> > > just
> >> > > > to be sure to have clean cache?
> >> > > >
> >> > > > Thanks
> >> > > > Jayesh
> >> > > >
> >> > > >
> >> > > > On Thu, Jan 25, 2018 at 4:19 AM, Colm O hEigeartaigh <
> >> > > coheigea@apache.org>
> >> > > > wrote:
> >> > > >
> >> > > > > Thanks again for your feedback. Jayesh, adding
> >> AlertEagleStorePlugin
> >> > > did
> >> > > > > the trick, I can now see alerts in the UI, thanks! By the way, I
> >> > can't
> >> > > > > configure two Alert Publishers, or else the Alert DeDuplicator
> >> bins
> >> > the
> >> > > > > alert. Is this a known issue?
> >> > > > >
> >> > > > > Could I ask which browser people are using with the UI? There
> >> appears
> >> > > to
> >> > > > be
> >> > > > > a  bug with Chromium where it doesn't list the pages under
> >> > Auth.isAdmin
> >> > > > > even though I am logged on as an administrator. It works OK in
> >> > Firefox.
> >> > > > > Even with Firefox though, I only see a limited number of links
> in
> >> the
> >> > > > > left-hand column - I can't get back to the "integration" page.
> Can
> >> > > > someone
> >> > > > > else confirm this please?
> >> > > > >
> >> > > > > Could I suggest the devs do some basic house-keeping tasks:
> >> > > > >
> >> > > > > a) "Release" version 0.5.0 in JIRA (it's still listed as
> >> > "unreleased").
> >> > > > > b) Figure out whether the next version will be 0.5.1 or 0.6.0
> and
> >> > > update
> >> > > > > the versions on Master accordingly with 0.5.1-SNAPSHOT or
> >> > > 0.6.0-SNAPSHOT.
> >> > > > > There are some issues marked here as resolved for 0.5.1 -
> >> > > > > https://issues.apache.org/jira/projects/EAGLE/versions/12341128
> ),
> >> > > > however
> >> > > > > I
> >> > > > > don't see a branch for 0.5.x?
> >> > > > >
> >> > > > > Colm.
> >> > > > >
> >> > > > > On Thu, Jan 25, 2018 at 8:16 AM, Jayesh Senjaliya <
> >> jaysen@apache.org
> >> > >
> >> > > > > wrote:
> >> > > > >
> >> > > > > > Hi,
> >> > > > > >
> >> > > > > > we do use eagle 0.5 in production although we dont use all the
> >> > > > available
> >> > > > > > hadoop applications.
> >> > > > > >
> >> > > > > > EAGLE-968 <https://issues.apache.org/jira/browse/EAGLE-968>
> is
> >> a
> >> > fix
> >> > > > for
> >> > > > > > email issue we found while our testing. should be merged soon
> >> > after a
> >> > > > > > rebase.
> >> > > > > >
> >> > > > > > @Colm, did you tried adding storage publisher
> >> > > (AlertEagleStorePlugin)?
> >> > > > to
> >> > > > > > see alerts on UI ?
> >> > > > > >
> >> > > > > > Thanks
> >> > > > > > Jayesh
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > > On Wed, Jan 24, 2018 at 7:08 PM, Edward Zhang <
> >> > > yonzhang2012@gmail.com>
> >> > > > > > wrote:
> >> > > > > >
> >> > > > > >> Eagle 0.5 was deployed in production as far as I know, but it
> >> may
> >> > > not
> >> > > > be
> >> > > > > >> exact the current version in master branch.
> >> > > > > >>
> >> > > > > >> Thanks for your investigation, seems there is still some bug
> in
> >> > 0.5,
> >> > > > but
> >> > > > > >> this particular issue seems is due to dependent components
> >> version
> >> > > > > conflict.
> >> > > > > >>
> >> > > > > >> @Jayesh is this Jira ready for merge to master?
> >> > > https://issues.apache
> >> > > > .
> >> > > > > >> org/jira/browse/EAGLE-968
> >> > > > > >>
> >> > > > > >>
> >> > > > > >> Thanks
> >> > > > > >> Edward
> >> > > > > >>
> >> > > > > >> On Tue, Jan 23, 2018 at 5:10 AM, Colm O hEigeartaigh <
> >> > > > > coheigea@apache.org
> >> > > > > >> > wrote:
> >> > > > > >>
> >> > > > > >>> OK I've made some more progress. I wasn't seeing any email
> >> alerts
> >> > > due
> >> > > > > to
> >> > > > > >>> https://issues.apache.org/jira/browse/EAGLE-968. Once I
> >> > configure
> >> > > a
> >> > > > > >>> Kafka
> >> > > > > >>> alert, I can see the alerts flowing into my topic. It's
> still
> >> not
> >> > > > clear
> >> > > > > >>> to
> >> > > > > >>> me however where the policy "output" is going. I also don't
> >> see
> >> > any
> >> > > > > >>> alerts
> >> > > > > >>> in the UI window.
> >> > > > > >>>
> >> > > > > >>> Could I ask what the status of the project is in general?
> >> There
> >> > > have
> >> > > > > been
> >> > > > > >>> no commits to master since November, so I'm not sure if
> there
> >> is
> >> > > any
> >> > > > > >>> point
> >> > > > > >>> in submitting Pull Requests for outstanding bugs? Are recent
> >> > > versions
> >> > > > > of
> >> > > > > >>> Apache Eagle used in production?
> >> > > > > >>>
> >> > > > > >>> Colm.
> >> > > > > >>>
> >> > > > > >>> On Mon, Jan 22, 2018 at 1:07 PM, Colm O hEigeartaigh <
> >> > > > > >>> coheigea@apache.org>
> >> > > > > >>> wrote:
> >> > > > > >>>
> >> > > > > >>> >
> >> > > > > >>> > I've done that but I'm not seeing any alerts, which is
> why I
> >> > want
> >> > > > to
> >> > > > > >>> find
> >> > > > > >>> > out what the "output" of a policy is and where I can check
> >> > this.
> >> > > > > >>> >
> >> > > > > >>> > Colm.
> >> > > > > >>> >
> >> > > > > >>> > On Mon, Jan 22, 2018 at 1:05 PM, SUDHA JENSLIN <
> >> > > sjenslin@gmail.com
> >> > > > >
> >> > > > > >>> wrote:
> >> > > > > >>> >
> >> > > > > >>> >> Create and add a publisher to see the output.
> >> > > > > >>> >>
> >> > > > > >>> >>
> >> > > > > >>> >>
> >> > > > > >>> >> Regards,
> >> > > > > >>> >> Sudha jenslin
> >> > > > > >>> >>
> >> > > > > >>> >> On Jan 22, 2018 6:31 PM, "Colm O hEigeartaigh" <
> >> > > > coheigea@apache.org
> >> > > > > >
> >> > > > > >>> >> wrote:
> >> > > > > >>> >>
> >> > > > > >>> >> Thanks - the error was due to a problem running Storm
> with
> >> > Java
> >> > > > 1.8.
> >> > > > > >>> I've
> >> > > > > >>> >> abandoned the docker image for now, and I'm trying to get
> >> it
> >> > > > working
> >> > > > > >>> >> locally.
> >> > > > > >>> >>
> >> > > > > >>> >> There are two things I'm not clear on currently, if
> someone
> >> > > could
> >> > > > > >>> fill me
> >> > > > > >>> >> in:
> >> > > > > >>> >>
> >> > > > > >>> >> a) For the  'Hdfs Audit Log Monitor' application, the
> Kafka
> >> > > > Consumer
> >> > > > > >>> Topic
> >> > > > > >>> >> is 'hdfs_audit_log_sandbox'. Under 'Kafka Topic for
> >> Auditlog
> >> > > Event
> >> > > > > >>> Sink'
> >> > > > > >>> >> it
> >> > > > > >>> >> also specifies 'hdfs_audit_event_sandbox'. However the
> >> > > > documentation
> >> > > > > >>> for
> >> > > > > >>> >> the application mentions 'hdfs_audit_log_enriched_sandb
> >> ox'?
> >> > > > > >>> >>
> >> > > > > >>> >> When I click on "STREAMS", the
> >> "HDFS_AUDIT_LOG_ENRICHED_STREA
> >> > > > > >>> M_SANDBOX"
> >> > > > > >>> >> uses the topic "hdfs_audit_event_sandbox". And indeed
> when
> >> I
> >> > run
> >> > > > the
> >> > > > > >>> >> application, I can see cleansed log data appearing in
> >> > > > > >>> >> "hdfs_audit_event_sandbox". So I'm thinking here that
> >> > > > > >>> >> 'hdfs_audit_log_enriched_sandbox' is not correct or
> >> > necessary?
> >> > > > > >>> >>
> >> > > > > >>> >> b) It's unclear to me where the output data goes when you
> >> > > create a
> >> > > > > >>> policy.
> >> > > > > >>> >> E.g. say I have:
> >> > > > > >>> >>
> >> > > > > >>> >> from HDFS_AUDIT_LOG_ENRICHED_
> STREAM_SANDBOX[str:contains(
> >> > > src,'/hb
> >> > > > > >>> ase')]
> >> > > > > >>> >> select * group by user insert into
> hdfs_audit_log_enriched_
> >> > > > > stream_out
> >> > > > > >>> >>
> >> > > > > >>> >> Where is "hdfs_audit_log_enriched_stream_out" defined
> (is
> >> it
> >> > a
> >> > > > > Kafka
> >> > > > > >>> >> topic?). How can I check the output to make sure the
> >> policy is
> >> > > > > working
> >> > > > > >>> >> correctly?
> >> > > > > >>> >>
> >> > > > > >>> >> Thanks,
> >> > > > > >>> >>
> >> > > > > >>> >> Colm.
> >> > > > > >>> >>
> >> > > > > >>> >> On Wed, Jan 17, 2018 at 10:32 PM, Edward Zhang <
> >> > > > > >>> yonzhang2012@gmail.com>
> >> > > > > >>> >> wrote:
> >> > > > > >>> >>
> >> > > > > >>> >> > There is a data preparation stage between data
> >> source(HDFS
> >> > > audit
> >> > > > > >>> log)
> >> > > > > >>> >> and
> >> > > > > >>> >> > Alert Engine. This stage is running in Storm and
> >> transform
> >> > the
> >> > > > raw
> >> > > > > >>> HDFS
> >> > > > > >>> >> log
> >> > > > > >>> >> > into something which can be alerted.
> >> > > > > >>> >> >
> >> > > > > >>> >> > The input for data preparation is
> hdfs_audit_log_sandbox
> >> > topic
> >> > > > and
> >> > > > > >>> >> output
> >> > > > > >>> >> > is
> >> > > > > >>> >> >  hdfs_audit_log_enriched_sandbox.
> >> > > > > >>> >> > The input for Alert Engine is hdfs_audit_log_enriched_
> >> > sandbox
> >> > > > and
> >> > > > > >>> >> output
> >> > > > > >>> >> > is
> >> > > > > >>> >> > hdfs_audit_log_alert_sandbox.
> >> > > > > >>> >> >
> >> > > > > >>> >> > Seems in your case, the data preparation staging is not
> >> > > working.
> >> > > > > We
> >> > > > > >>> >> > probably need look at Storm console and figure out if
> >> that
> >> > > part
> >> > > > is
> >> > > > > >>> >> working.
> >> > > > > >>> >> >
> >> > > > > >>> >> > Thanks
> >> > > > > >>> >> > Edward
> >> > > > > >>> >> >
> >> > > > > >>> >> > On Wed, Jan 17, 2018 at 7:19 AM, Colm O hEigeartaigh <
> >> > > > > >>> >> coheigea@apache.org>
> >> > > > > >>> >> > wrote:
> >> > > > > >>> >> >
> >> > > > > >>> >> > > Hi Jayesh,
> >> > > > > >>> >> > >
> >> > > > > >>> >> > > Many thanks for your feedback! I was able to make a
> >> little
> >> > > > > further
> >> > > > > >>> >> > headway.
> >> > > > > >>> >> > > There are two configuration problems with the
> official
> >> > > docker
> >> > > > > >>> image:
> >> > > > > >>> >> > >
> >> > > > > >>> >> > > a) A mix of "sandbox.eagle.apache.org" and "
> >> > > > > >>> server.eagle.apache.org"
> >> > > > > >>> >> > (this
> >> > > > > >>> >> > > only occurs in the instructions for running the
> docker
> >> > > image.
> >> > > > > The
> >> > > > > >>> >> version
> >> > > > > >>> >> > > that can be started via the script in the eagle
> source
> >> is
> >> > > OK).
> >> > > > > >>> I'll
> >> > > > > >>> >> > submit
> >> > > > > >>> >> > > a PR to fix this once I get a basic use-case working.
> >> > > > > >>> >> > > b) For the audit case, it automatically logs HDFS
> audit
> >> > logs
> >> > > > to
> >> > > > > >>> the
> >> > > > > >>> >> KAFKA
> >> > > > > >>> >> > > topic sandbox_hdfs_audit_log instead of the expected
> >> > > > > >>> >> > hdfs_audit_log_sandbox
> >> > > > > >>> >> > >
> >> > > > > >>> >> > > I've fixed these things locally and I can verify that
> >> > > > everything
> >> > > > > >>> is
> >> > > > > >>> >> > started
> >> > > > > >>> >> > > correctly in Ambari. I log into the docker container
> >> and
> >> > > > create
> >> > > > > >>> >> > > hdfs_audit_log_sandbox and hdfs_audit_log_enriched_
> >> > sandbox
> >> > > > > >>> topics,
> >> > > > > >>> >> and
> >> > > > > >>> >> > > verify that the HDFS audit logs are flowing into the
> >> first
> >> > > > > topic.
> >> > > > > >>> >> Then in
> >> > > > > >>> >> > > the UI I start the Alert Engine and then the HDFS
> Audit
> >> > Log
> >> > > > > >>> Monitor
> >> > > > > >>> >> > > application (changing localhost:6667 to
> >> > > > > >>> server.eagle.apache.org:6667
> >> > > > > >>> >> ).
> >> > > > > >>> >> > > Both
> >> > > > > >>> >> > > applications start up correctly and show "running".
> >> > > > > >>> >> > >
> >> > > > > >>> >> > > I then create a policy with an email alert along the
> >> lines
> >> > > of
> >> > > > > from
> >> > > > > >>> >> > > "HDFS_AUDIT_LOG_ENRICHED_
> STREAM_SANDBOX[str:contains(
> >> > > src,'/h
> >> > > > > >>> base')]
> >> > > > > >>> >> > select
> >> > > > > >>> >> > > * group by user insert into hdfs_audit_log_enriched_
> >> > > > > stream_out".
> >> > > > > >>> >> However
> >> > > > > >>> >> > > at
> >> > > > > >>> >> > > this point I'm stuck - nothing appears in the alert
> >> > window.
> >> > > Is
> >> > > > > >>> there
> >> > > > > >>> >> > > anything obvious I'm doing wrong, or how can I get
> >> access
> >> > to
> >> > > > > logs
> >> > > > > >>> to
> >> > > > > >>> >> > figure
> >> > > > > >>> >> > > out what the problem is? Other topics such as
> >> > > > > >>> >> "hdfs_audit_event_sandbox"
> >> > > > > >>> >> > > are mentioned in the streams window, but the
> >> documentation
> >> > > > > doesn't
> >> > > > > >>> >> say to
> >> > > > > >>> >> > > create them.
> >> > > > > >>> >> > >
> >> > > > > >>> >> > > The UI is buggy though on both Firefox and Chromium
> on
> >> > > Linux.
> >> > > > > What
> >> > > > > >>> >> > > browser/platform are people using with the UI?
> >> > > > > >>> >> > >
> >> > > > > >>> >> > > Colm.
> >> > > > > >>> >> > >
> >> > > > > >>> >> > > On Wed, Jan 17, 2018 at 12:27 AM, Jayesh Senjaliya <
> >> > > > > >>> jaysen@apache.org
> >> > > > > >>> >> >
> >> > > > > >>> >> > > wrote:
> >> > > > > >>> >> > >
> >> > > > > >>> >> > > > Hi Colm,
> >> > > > > >>> >> > > >
> >> > > > > >>> >> > > > Please find my comments inline.
> >> > > > > >>> >> > > >
> >> > > > > >>> >> > > > a) The official docker image uses 0.5.0-SNAPSHOT
> and
> >> not
> >> > > the
> >> > > > > >>> >> released
> >> > > > > >>> >> > > > version.
> >> > > > > >>> >> > > > - this is because we uploaded docker image before
> >> apache
> >> > > > > >>> release.
> >> > > > > >>> >> > > actually
> >> > > > > >>> >> > > > this is same codebase apache-eagle-0.5, and it can
> be
> >> > > fixed
> >> > > > > >>> easily
> >> > > > > >>> >> by
> >> > > > > >>> >> > > just
> >> > > > > >>> >> > > > rebuilding docker image. there should not be any
> >> > mismatch
> >> > > > due
> >> > > > > to
> >> > > > > >>> >> this.
> >> > > > > >>> >> > > >
> >> > > > > >>> >> > > > b) Aside from the above, the official docker image
> >> uses
> >> > a
> >> > > > mix
> >> > > > > >>> of "
> >> > > > > >>> >> > > > server.eagle.apache.org" and "
> >> sandbox.eagle.apache.org"
> >> > > as
> >> > > > > the
> >> > > > > >>> host
> >> > > > > >>> >> > > name.
> >> > > > > >>> >> > > > The HBase service doesn't start by default in
> Ambari
> >> as
> >> > a
> >> > > > > >>> result.
> >> > > > > >>> >> > > > - the only places it uses sandbox is in example
> >> script
> >> > > which
> >> > > > > you
> >> > > > > >>> >> will
> >> > > > > >>> >> > > have
> >> > > > > >>> >> > > > to update anyway, which i agree that it would be
> >> good to
> >> > > > keep
> >> > > > > it
> >> > > > > >>> >> > > > consistent.
> >> > > > > >>> >> > > >
> >> > > > > >>> >> > > > c) The UI seems quite buggy. On both chromium and
> >> > > firefox, I
> >> > > > > >>> only
> >> > > > > >>> >> see
> >> > > > > >>> >> > > > links to "Sandbox" and "Alert" on the left
> hand-side.
> >> > > Once I
> >> > > > > >>> click
> >> > > > > >>> >> on
> >> > > > > >>> >> > > > "Alert" I have no way of going back to see the
> >> > > > applications. I
> >> > > > > >>> don't
> >> > > > > >>> >> > see
> >> > > > > >>> >> > > > the links to "integration" or "sites" as in the
> >> picture
> >> > > > here:
> >> > > > > >>> >> > > > http://eagle.apache.org/docs/l
> >> atest/applications/#jmx-
> >> > > > monito
> >> > > > > >>> ring
> >> > > > > >>> >> > > > - when hbase is as deep storage is used, and if
> eagle
> >> > app
> >> > > > has
> >> > > > > >>> issue
> >> > > > > >>> >> > > > connecting to hbase, the UI becomes unresponsive.
> >> > > > > >>> >> > > >
> >> > > > > >>> >> > > > d) In chromium, the button to create a new policy
> >> does
> >> > not
> >> > > > > >>> exist - I
> >> > > > > >>> >> > can
> >> > > > > >>> >> > > > only see it on Firefox.
> >> > > > > >>> >> > > > - i have seen when you logged in, you will see
> admin
> >> > > > actions.
> >> > > > > >>> but if
> >> > > > > >>> >> > this
> >> > > > > >>> >> > > > still an issue, can you please file UI bug?
> >> > > > > >>> >> > > >
> >> > > > > >>> >> > > > e) I'm trying to get the "Hdfs Audit Log Monitor"
> >> > use-case
> >> > > > > >>> working,
> >> > > > > >>> >> but
> >> > > > > >>> >> > > it
> >> > > > > >>> >> > > > seems to be stuck in "Initialized".
> >> > > > > >>> >> > > > this eagle docs has example on how to setup the
> app.
> >> pls
> >> > > let
> >> > > > > us
> >> > > > > >>> >> know if
> >> > > > > >>> >> > > > you find any gaps.
> >> > > > > >>> >> > > >
> >> > > > > >>> >> > > > Thanks for trying out, and sharing your findings,
> >> > > > > >>> >> > > > Jayesh
> >> > > > > >>> >> > > >
> >> > > > > >>> >> > > >
> >> > > > > >>> >> > > > On Tue, Jan 16, 2018 at 3:34 AM, Colm O
> hEigeartaigh
> >> <
> >> > > > > >>> >> > > coheigea@apache.org>
> >> > > > > >>> >> > > > wrote:
> >> > > > > >>> >> > > >
> >> > > > > >>> >> > > >> Hi all,
> >> > > > > >>> >> > > >>
> >> > > > > >>> >> > > >> I'm trying to play around a bit with Apache Eagle
> >> 0.5.0
> >> > > to
> >> > > > no
> >> > > > > >>> >> avail.
> >> > > > > >>> >> > > Here
> >> > > > > >>> >> > > >> are the problems I've run into so far:
> >> > > > > >>> >> > > >>
> >> > > > > >>> >> > > >> a) The official docker image uses 0.5.0-SNAPSHOT
> and
> >> > not
> >> > > > the
> >> > > > > >>> >> released
> >> > > > > >>> >> > > >> version.
> >> > > > > >>> >> > > >>
> >> > > > > >>> >> > > >> b) Aside from the above, the official docker image
> >> > uses a
> >> > > > mix
> >> > > > > >>> of "
> >> > > > > >>> >> > > >> server.eagle.apache.org" and "
> >> sandbox.eagle.apache.org
> >> > "
> >> > > as
> >> > > > > the
> >> > > > > >>> >> host
> >> > > > > >>> >> > > >> name. The HBase service doesn't start by default
> in
> >> > > Ambari
> >> > > > > as a
> >> > > > > >>> >> > result.
> >> > > > > >>> >> > > >>
> >> > > > > >>> >> > > >> c) The UI seems quite buggy. On both chromium and
> >> > > firefox,
> >> > > > I
> >> > > > > >>> only
> >> > > > > >>> >> see
> >> > > > > >>> >> > > >> links to "Sandbox" and "Alert" on the left
> >> hand-side.
> >> > > Once
> >> > > > I
> >> > > > > >>> click
> >> > > > > >>> >> on
> >> > > > > >>> >> > > >> "Alert" I have no way of going back to see the
> >> > > > applications.
> >> > > > > I
> >> > > > > >>> >> don't
> >> > > > > >>> >> > see
> >> > > > > >>> >> > > >> the links to "integration" or "sites" as in the
> >> picture
> >> > > > here:
> >> > > > > >>> >> > > >> http://eagle.apache.org/docs/l
> >> atest/applications/#jmx-
> >> > > > monito
> >> > > > > >>> ring
> >> > > > > >>> >> > > >>
> >> > > > > >>> >> > > >> d) In chromium, the button to create a new policy
> >> does
> >> > > not
> >> > > > > >>> exist -
> >> > > > > >>> >> I
> >> > > > > >>> >> > can
> >> > > > > >>> >> > > >> only see it on Firefox.
> >> > > > > >>> >> > > >>
> >> > > > > >>> >> > > >> e) I'm trying to get the "Hdfs Audit Log Monitor"
> >> > > use-case
> >> > > > > >>> working,
> >> > > > > >>> >> > but
> >> > > > > >>> >> > > >> it seems to be stuck in "Initialized".
> >> > > > > >>> >> > > >>
> >> > > > > >>> >> > > >> Could someone fill me in on what the "recommended"
> >> way
> >> > is
> >> > > > to
> >> > > > > >>> start
> >> > > > > >>> >> > > Apache
> >> > > > > >>> >> > > >> Eagle so that I can play around with the
> >> functionality
> >> > > that
> >> > > > > it
> >> > > > > >>> >> offers?
> >> > > > > >>> >> > > >> Clearly the docker approach is buggy. Also, what
> >> > browser
> >> > > > > >>> should be
> >> > > > > >>> >> > used?
> >> > > > > >>> >> > > >>
> >> > > > > >>> >> > > >> Thanks,
> >> > > > > >>> >> > > >>
> >> > > > > >>> >> > > >> Colm.
> >> > > > > >>> >> > > >>
> >> > > > > >>> >> > > >>
> >> > > > > >>> >> > > >> --
> >> > > > > >>> >> > > >> Colm O hEigeartaigh
> >> > > > > >>> >> > > >>
> >> > > > > >>> >> > > >> Talend Community Coder
> >> > > > > >>> >> > > >> http://coders.talend.com
> >> > > > > >>> >> > > >>
> >> > > > > >>> >> > > >
> >> > > > > >>> >> > > >
> >> > > > > >>> >> > >
> >> > > > > >>> >> > >
> >> > > > > >>> >> > > --
> >> > > > > >>> >> > > Colm O hEigeartaigh
> >> > > > > >>> >> > >
> >> > > > > >>> >> > > Talend Community Coder
> >> > > > > >>> >> > > http://coders.talend.com
> >> > > > > >>> >> > >
> >> > > > > >>> >> >
> >> > > > > >>> >>
> >> > > > > >>> >>
> >> > > > > >>> >>
> >> > > > > >>> >> --
> >> > > > > >>> >> Colm O hEigeartaigh
> >> > > > > >>> >>
> >> > > > > >>> >> Talend Community Coder
> >> > > > > >>> >> http://coders.talend.com
> >> > > > > >>> >>
> >> > > > > >>> >>
> >> > > > > >>> >>
> >> > > > > >>> >
> >> > > > > >>> >
> >> > > > > >>> > --
> >> > > > > >>> > Colm O hEigeartaigh
> >> > > > > >>> >
> >> > > > > >>> > Talend Community Coder
> >> > > > > >>> > http://coders.talend.com
> >> > > > > >>> >
> >> > > > > >>>
> >> > > > > >>>
> >> > > > > >>>
> >> > > > > >>> --
> >> > > > > >>> Colm O hEigeartaigh
> >> > > > > >>>
> >> > > > > >>> Talend Community Coder
> >> > > > > >>> http://coders.talend.com
> >> > > > > >>>
> >> > > > > >>
> >> > > > > >>
> >> > > > > >
> >> > > > >
> >> > > > >
> >> > > > > --
> >> > > > > Colm O hEigeartaigh
> >> > > > >
> >> > > > > Talend Community Coder
> >> > > > > http://coders.talend.com
> >> > > > >
> >> > > >
> >> > >
> >> > >
> >> > >
> >> > > --
> >> > > Colm O hEigeartaigh
> >> > >
> >> > > Talend Community Coder
> >> > > http://coders.talend.com
> >> > >
> >> >
> >>
> >>
> >>
> >> --
> >> Colm O hEigeartaigh
> >>
> >> Talend Community Coder
> >> http://coders.talend.com
> >>
> >
> >
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>

Re: Unable to get 0.5.0 release working

Posted by Colm O hEigeartaigh <co...@apache.org>.
Thanks Jayesh. I have two more PRs awaiting review:

https://github.com/apache/eagle/pull/981
https://github.com/apache/eagle/pull/982

Thanks for the JIRA privileges, I can now assign issues to me + change the
versions. However, I can't "resolve" JIRAs that weren't reported by me
which is annoying. These 3 JIRAs should be resolved as they are already
merged:

https://issues.apache.org/jira/browse/EAGLE-445
https://issues.apache.org/jira/browse/EAGLE-476
https://issues.apache.org/jira/browse/EAGLE-331

In addition, I tested the fix for the Email issue and it works correctly.
The PR (https://github.com/apache/eagle/pull/941) just needs to have the
extra commits stripped away - I attached a version of the patch on the JIRA.

Colm.

On Wed, Jan 31, 2018 at 10:08 PM, Jayesh Senjaliya <ja...@apache.org>
wrote:

> Thanks for the PRs. I have merged them.
>
> welcome to the developer community Colm. I have also added you to jira
> project so can assign the tasks to yourself.
>
> lets create ticket to fix the dedup functionality, I m actually surprised
> we havent hit this issue yet. we do use multiple publishers but someone can
> verify this.
>
> Thanks
> Jayesh
>
>
>
> On Wed, Jan 31, 2018 at 9:25 AM, Colm O hEigeartaigh <co...@apache.org>
> wrote:
>
>> Thanks Jayesh. I've created a JIRA here for some admin work for some
>> issues
>> that were incorrectly flagged as "fix for" 0.5.1/0.6.0:
>>
>> https://issues.apache.org/jira/browse/EAGLE-1076
>>
>> I've submitted the following (fairly trivial) pull requests. Could I ask
>> that you or one of the other committers review?
>>
>> https://github.com/apache/eagle/pull/978
>> https://github.com/apache/eagle/pull/979
>> https://github.com/apache/eagle/pull/980
>>
>> It would be good to try to inject some energy into the project. We need
>> more than one active committer though.
>>
>> Just in terms of the Alert Deduplication issue. The DefaultDeDuplicator
>> works per "output" in the policy rule. So if you have more than one
>> AlertPublisher, I think it is guaranteed to only publish to one of them.
>> Instead, surely it would make more sense to work per publisher?
>>
>> Colm.
>>
>> On Tue, Jan 30, 2018 at 10:39 PM, Jayesh Senjaliya <ja...@apache.org>
>> wrote:
>>
>> > Hi Colm,
>> >
>> > appreciate your suggestions/ efforts in looking into this project,
>> > putting my comments inline...
>> >
>> > a) There is already a JIRA to bump the version here, although the PR
>> does
>> > not apply as it is too old: https://issues.apache.org/
>> > jira/browse/EAGLE-1025
>> > .
>> > I can submit a new PR, but should the version be 0.6.0 or 0.5.1?
>> >
>> > *since there are still minor issues, i would say, we put up 0.5.1 as
>> next
>> > version. I've updated/rebased the PR (
>> > https://github.com/apache/eagle/pull/936
>> > <https://github.com/apache/eagle/pull/936> )*
>> >
>> >
>> > b) The issues that are "resolved" for the 0.5.1 release in JIRA are
>> > actually already fixed in 0.5.0, so they should be updated (
>> > https://issues.apache.org/jira/projects/EAGLE/versions/12341128).
>> However,
>> > the following two issues are resolved even though they are not merged to
>> > master?
>> >   https://issues.apache.org/jira/browse/EAGLE-1051 .  - * this was
>> pending
>> > from developer;s response but i think this is reviewed, so I have merged
>> > it.*
>> >   https://issues.apache.org/jira/browse/EAGLE-1068 .  - * this is
>> reopened
>> > now. I dont think this is done yet. Also this is big change.*
>> >
>> >
>> > Like I said I can submit PRs but I'm not convinced there is any
>> activity on
>> > the project. Where are the rest of the committers?
>> >
>> > *let me give you some ocontext on this. so there were lot of development
>> > happened during last releases, and most of applications that were added
>> are
>> > being used in production at multiple enterprise companies, but we are
>> out
>> > of ideas on new apps, so at this point we are only focusing on bug fixes
>> > and tech upgrades until we get some new ideas to brainstorm and add.*
>> >
>> > *I think current community's thinking is based on their own industries
>> > use-cases, but there is definitely room for new features and integration
>> > with other monitoring and security components like grafana and rangers.*
>> >
>> >
>> > *Thanks,*
>> > *Jayesh*
>> >
>> >
>> >
>> > On Tue, Jan 30, 2018 at 8:11 AM, Colm O hEigeartaigh <
>> coheigea@apache.org>
>> > wrote:
>> >
>> > > Hi Jayesh,
>> > >
>> > > Dev suggestions:
>> > >
>> > > a) There is already a JIRA to bump the version here, although the PR
>> does
>> > > not apply as it is too old: https://issues.apache.org/
>> > > jira/browse/EAGLE-1025.
>> > > I can submit a new PR, but should the version be 0.6.0 or 0.5.1?
>> > > b) The issues that are "resolved" for the 0.5.1 release in JIRA are
>> > > actually already fixed in 0.5.0, so they should be updated (
>> > > https://issues.apache.org/jira/projects/EAGLE/versions/12341128).
>> > However,
>> > > the following two issues are resolved even though they are not merged
>> to
>> > > master?
>> > >   https://issues.apache.org/jira/browse/EAGLE-1051
>> > >   https://issues.apache.org/jira/browse/EAGLE-1068
>> > >
>> > > Like I said I can submit PRs but I'm not convinced there is any
>> activity
>> > on
>> > > the project. Where are the rest of the committers?
>> > >
>> > > Multiple Publisher issue:
>> > >
>> > > If I assign two publishers for one policy, the alert only goes to the
>> > first
>> > > policy. In the logs I see:
>> > >
>> > > 2018-01-30T15:52:45.835+0000 o.a.e.a.e.p.d.DefaultDeduplicator [INFO]
>> > > Alert
>> > > event is skipped because it's duplicated: Alert {site=sandbox,
>> > > stream=eagle_output,timestamp=2018-01-30
>> > > 00:00:11,300,data={securityZone=NA, dst=null, sensitivityType=NA,
>> > > src=/apps/hbase/data/archive/data/default/ambarismoketest,
>> allowed=true,
>> > > host=172.22.7.129, cmd=listStatus, user=SOMETHING7.COM,
>> > > timestamp=1517270411300}, policyId=test,
>> > > createdBy=alertBolt3-evaluator_stage1, metaVersion=null}
>> > >
>> > > It looks like this deduplicator is not working properly, as I'm
>> guessing
>> > it
>> > > should only be used to de-duplicate events for a single publisher?
>> > >
>> > > Incognito mode: Already tried it but with the same result. Could I ask
>> > you
>> > > to try the docker image to see if the UI is working correctly for you
>> > > there?
>> > >
>> > > Colm.
>> > >
>> > > On Mon, Jan 29, 2018 at 6:46 PM, Jayesh Senjaliya <ja...@apache.org>
>> > > wrote:
>> > >
>> > > > Hi Colm,
>> > > >
>> > > > Thanks for the list of dev suggestions, I think we should take care
>> of
>> > > > those. even better if you can provide PR with the changes or at
>> keast
>> > can
>> > > > you please create a ticket so we can track it?
>> > > >
>> > > > for other issues.
>> > > >
>> > > > - I dont have any issue with multiple publisher, but if there is any
>> > > error
>> > > > updating the publisher info in storm topology, i might try
>> restarting
>> > the
>> > > > topology and see if that works.
>> > > > - for us, chrome works as fine as firefox.  can u try incognito
>> mode?
>> > > just
>> > > > to be sure to have clean cache?
>> > > >
>> > > > Thanks
>> > > > Jayesh
>> > > >
>> > > >
>> > > > On Thu, Jan 25, 2018 at 4:19 AM, Colm O hEigeartaigh <
>> > > coheigea@apache.org>
>> > > > wrote:
>> > > >
>> > > > > Thanks again for your feedback. Jayesh, adding
>> AlertEagleStorePlugin
>> > > did
>> > > > > the trick, I can now see alerts in the UI, thanks! By the way, I
>> > can't
>> > > > > configure two Alert Publishers, or else the Alert DeDuplicator
>> bins
>> > the
>> > > > > alert. Is this a known issue?
>> > > > >
>> > > > > Could I ask which browser people are using with the UI? There
>> appears
>> > > to
>> > > > be
>> > > > > a  bug with Chromium where it doesn't list the pages under
>> > Auth.isAdmin
>> > > > > even though I am logged on as an administrator. It works OK in
>> > Firefox.
>> > > > > Even with Firefox though, I only see a limited number of links in
>> the
>> > > > > left-hand column - I can't get back to the "integration" page. Can
>> > > > someone
>> > > > > else confirm this please?
>> > > > >
>> > > > > Could I suggest the devs do some basic house-keeping tasks:
>> > > > >
>> > > > > a) "Release" version 0.5.0 in JIRA (it's still listed as
>> > "unreleased").
>> > > > > b) Figure out whether the next version will be 0.5.1 or 0.6.0 and
>> > > update
>> > > > > the versions on Master accordingly with 0.5.1-SNAPSHOT or
>> > > 0.6.0-SNAPSHOT.
>> > > > > There are some issues marked here as resolved for 0.5.1 -
>> > > > > https://issues.apache.org/jira/projects/EAGLE/versions/12341128),
>> > > > however
>> > > > > I
>> > > > > don't see a branch for 0.5.x?
>> > > > >
>> > > > > Colm.
>> > > > >
>> > > > > On Thu, Jan 25, 2018 at 8:16 AM, Jayesh Senjaliya <
>> jaysen@apache.org
>> > >
>> > > > > wrote:
>> > > > >
>> > > > > > Hi,
>> > > > > >
>> > > > > > we do use eagle 0.5 in production although we dont use all the
>> > > > available
>> > > > > > hadoop applications.
>> > > > > >
>> > > > > > EAGLE-968 <https://issues.apache.org/jira/browse/EAGLE-968> is
>> a
>> > fix
>> > > > for
>> > > > > > email issue we found while our testing. should be merged soon
>> > after a
>> > > > > > rebase.
>> > > > > >
>> > > > > > @Colm, did you tried adding storage publisher
>> > > (AlertEagleStorePlugin)?
>> > > > to
>> > > > > > see alerts on UI ?
>> > > > > >
>> > > > > > Thanks
>> > > > > > Jayesh
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > On Wed, Jan 24, 2018 at 7:08 PM, Edward Zhang <
>> > > yonzhang2012@gmail.com>
>> > > > > > wrote:
>> > > > > >
>> > > > > >> Eagle 0.5 was deployed in production as far as I know, but it
>> may
>> > > not
>> > > > be
>> > > > > >> exact the current version in master branch.
>> > > > > >>
>> > > > > >> Thanks for your investigation, seems there is still some bug in
>> > 0.5,
>> > > > but
>> > > > > >> this particular issue seems is due to dependent components
>> version
>> > > > > conflict.
>> > > > > >>
>> > > > > >> @Jayesh is this Jira ready for merge to master?
>> > > https://issues.apache
>> > > > .
>> > > > > >> org/jira/browse/EAGLE-968
>> > > > > >>
>> > > > > >>
>> > > > > >> Thanks
>> > > > > >> Edward
>> > > > > >>
>> > > > > >> On Tue, Jan 23, 2018 at 5:10 AM, Colm O hEigeartaigh <
>> > > > > coheigea@apache.org
>> > > > > >> > wrote:
>> > > > > >>
>> > > > > >>> OK I've made some more progress. I wasn't seeing any email
>> alerts
>> > > due
>> > > > > to
>> > > > > >>> https://issues.apache.org/jira/browse/EAGLE-968. Once I
>> > configure
>> > > a
>> > > > > >>> Kafka
>> > > > > >>> alert, I can see the alerts flowing into my topic. It's still
>> not
>> > > > clear
>> > > > > >>> to
>> > > > > >>> me however where the policy "output" is going. I also don't
>> see
>> > any
>> > > > > >>> alerts
>> > > > > >>> in the UI window.
>> > > > > >>>
>> > > > > >>> Could I ask what the status of the project is in general?
>> There
>> > > have
>> > > > > been
>> > > > > >>> no commits to master since November, so I'm not sure if there
>> is
>> > > any
>> > > > > >>> point
>> > > > > >>> in submitting Pull Requests for outstanding bugs? Are recent
>> > > versions
>> > > > > of
>> > > > > >>> Apache Eagle used in production?
>> > > > > >>>
>> > > > > >>> Colm.
>> > > > > >>>
>> > > > > >>> On Mon, Jan 22, 2018 at 1:07 PM, Colm O hEigeartaigh <
>> > > > > >>> coheigea@apache.org>
>> > > > > >>> wrote:
>> > > > > >>>
>> > > > > >>> >
>> > > > > >>> > I've done that but I'm not seeing any alerts, which is why I
>> > want
>> > > > to
>> > > > > >>> find
>> > > > > >>> > out what the "output" of a policy is and where I can check
>> > this.
>> > > > > >>> >
>> > > > > >>> > Colm.
>> > > > > >>> >
>> > > > > >>> > On Mon, Jan 22, 2018 at 1:05 PM, SUDHA JENSLIN <
>> > > sjenslin@gmail.com
>> > > > >
>> > > > > >>> wrote:
>> > > > > >>> >
>> > > > > >>> >> Create and add a publisher to see the output.
>> > > > > >>> >>
>> > > > > >>> >>
>> > > > > >>> >>
>> > > > > >>> >> Regards,
>> > > > > >>> >> Sudha jenslin
>> > > > > >>> >>
>> > > > > >>> >> On Jan 22, 2018 6:31 PM, "Colm O hEigeartaigh" <
>> > > > coheigea@apache.org
>> > > > > >
>> > > > > >>> >> wrote:
>> > > > > >>> >>
>> > > > > >>> >> Thanks - the error was due to a problem running Storm with
>> > Java
>> > > > 1.8.
>> > > > > >>> I've
>> > > > > >>> >> abandoned the docker image for now, and I'm trying to get
>> it
>> > > > working
>> > > > > >>> >> locally.
>> > > > > >>> >>
>> > > > > >>> >> There are two things I'm not clear on currently, if someone
>> > > could
>> > > > > >>> fill me
>> > > > > >>> >> in:
>> > > > > >>> >>
>> > > > > >>> >> a) For the  'Hdfs Audit Log Monitor' application, the Kafka
>> > > > Consumer
>> > > > > >>> Topic
>> > > > > >>> >> is 'hdfs_audit_log_sandbox'. Under 'Kafka Topic for
>> Auditlog
>> > > Event
>> > > > > >>> Sink'
>> > > > > >>> >> it
>> > > > > >>> >> also specifies 'hdfs_audit_event_sandbox'. However the
>> > > > documentation
>> > > > > >>> for
>> > > > > >>> >> the application mentions 'hdfs_audit_log_enriched_sandb
>> ox'?
>> > > > > >>> >>
>> > > > > >>> >> When I click on "STREAMS", the
>> "HDFS_AUDIT_LOG_ENRICHED_STREA
>> > > > > >>> M_SANDBOX"
>> > > > > >>> >> uses the topic "hdfs_audit_event_sandbox". And indeed when
>> I
>> > run
>> > > > the
>> > > > > >>> >> application, I can see cleansed log data appearing in
>> > > > > >>> >> "hdfs_audit_event_sandbox". So I'm thinking here that
>> > > > > >>> >> 'hdfs_audit_log_enriched_sandbox' is not correct or
>> > necessary?
>> > > > > >>> >>
>> > > > > >>> >> b) It's unclear to me where the output data goes when you
>> > > create a
>> > > > > >>> policy.
>> > > > > >>> >> E.g. say I have:
>> > > > > >>> >>
>> > > > > >>> >> from HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX[str:contains(
>> > > src,'/hb
>> > > > > >>> ase')]
>> > > > > >>> >> select * group by user insert into hdfs_audit_log_enriched_
>> > > > > stream_out
>> > > > > >>> >>
>> > > > > >>> >> Where is "hdfs_audit_log_enriched_stream_out" defined (is
>> it
>> > a
>> > > > > Kafka
>> > > > > >>> >> topic?). How can I check the output to make sure the
>> policy is
>> > > > > working
>> > > > > >>> >> correctly?
>> > > > > >>> >>
>> > > > > >>> >> Thanks,
>> > > > > >>> >>
>> > > > > >>> >> Colm.
>> > > > > >>> >>
>> > > > > >>> >> On Wed, Jan 17, 2018 at 10:32 PM, Edward Zhang <
>> > > > > >>> yonzhang2012@gmail.com>
>> > > > > >>> >> wrote:
>> > > > > >>> >>
>> > > > > >>> >> > There is a data preparation stage between data
>> source(HDFS
>> > > audit
>> > > > > >>> log)
>> > > > > >>> >> and
>> > > > > >>> >> > Alert Engine. This stage is running in Storm and
>> transform
>> > the
>> > > > raw
>> > > > > >>> HDFS
>> > > > > >>> >> log
>> > > > > >>> >> > into something which can be alerted.
>> > > > > >>> >> >
>> > > > > >>> >> > The input for data preparation is hdfs_audit_log_sandbox
>> > topic
>> > > > and
>> > > > > >>> >> output
>> > > > > >>> >> > is
>> > > > > >>> >> >  hdfs_audit_log_enriched_sandbox.
>> > > > > >>> >> > The input for Alert Engine is hdfs_audit_log_enriched_
>> > sandbox
>> > > > and
>> > > > > >>> >> output
>> > > > > >>> >> > is
>> > > > > >>> >> > hdfs_audit_log_alert_sandbox.
>> > > > > >>> >> >
>> > > > > >>> >> > Seems in your case, the data preparation staging is not
>> > > working.
>> > > > > We
>> > > > > >>> >> > probably need look at Storm console and figure out if
>> that
>> > > part
>> > > > is
>> > > > > >>> >> working.
>> > > > > >>> >> >
>> > > > > >>> >> > Thanks
>> > > > > >>> >> > Edward
>> > > > > >>> >> >
>> > > > > >>> >> > On Wed, Jan 17, 2018 at 7:19 AM, Colm O hEigeartaigh <
>> > > > > >>> >> coheigea@apache.org>
>> > > > > >>> >> > wrote:
>> > > > > >>> >> >
>> > > > > >>> >> > > Hi Jayesh,
>> > > > > >>> >> > >
>> > > > > >>> >> > > Many thanks for your feedback! I was able to make a
>> little
>> > > > > further
>> > > > > >>> >> > headway.
>> > > > > >>> >> > > There are two configuration problems with the official
>> > > docker
>> > > > > >>> image:
>> > > > > >>> >> > >
>> > > > > >>> >> > > a) A mix of "sandbox.eagle.apache.org" and "
>> > > > > >>> server.eagle.apache.org"
>> > > > > >>> >> > (this
>> > > > > >>> >> > > only occurs in the instructions for running the docker
>> > > image.
>> > > > > The
>> > > > > >>> >> version
>> > > > > >>> >> > > that can be started via the script in the eagle source
>> is
>> > > OK).
>> > > > > >>> I'll
>> > > > > >>> >> > submit
>> > > > > >>> >> > > a PR to fix this once I get a basic use-case working.
>> > > > > >>> >> > > b) For the audit case, it automatically logs HDFS audit
>> > logs
>> > > > to
>> > > > > >>> the
>> > > > > >>> >> KAFKA
>> > > > > >>> >> > > topic sandbox_hdfs_audit_log instead of the expected
>> > > > > >>> >> > hdfs_audit_log_sandbox
>> > > > > >>> >> > >
>> > > > > >>> >> > > I've fixed these things locally and I can verify that
>> > > > everything
>> > > > > >>> is
>> > > > > >>> >> > started
>> > > > > >>> >> > > correctly in Ambari. I log into the docker container
>> and
>> > > > create
>> > > > > >>> >> > > hdfs_audit_log_sandbox and hdfs_audit_log_enriched_
>> > sandbox
>> > > > > >>> topics,
>> > > > > >>> >> and
>> > > > > >>> >> > > verify that the HDFS audit logs are flowing into the
>> first
>> > > > > topic.
>> > > > > >>> >> Then in
>> > > > > >>> >> > > the UI I start the Alert Engine and then the HDFS Audit
>> > Log
>> > > > > >>> Monitor
>> > > > > >>> >> > > application (changing localhost:6667 to
>> > > > > >>> server.eagle.apache.org:6667
>> > > > > >>> >> ).
>> > > > > >>> >> > > Both
>> > > > > >>> >> > > applications start up correctly and show "running".
>> > > > > >>> >> > >
>> > > > > >>> >> > > I then create a policy with an email alert along the
>> lines
>> > > of
>> > > > > from
>> > > > > >>> >> > > "HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX[str:contains(
>> > > src,'/h
>> > > > > >>> base')]
>> > > > > >>> >> > select
>> > > > > >>> >> > > * group by user insert into hdfs_audit_log_enriched_
>> > > > > stream_out".
>> > > > > >>> >> However
>> > > > > >>> >> > > at
>> > > > > >>> >> > > this point I'm stuck - nothing appears in the alert
>> > window.
>> > > Is
>> > > > > >>> there
>> > > > > >>> >> > > anything obvious I'm doing wrong, or how can I get
>> access
>> > to
>> > > > > logs
>> > > > > >>> to
>> > > > > >>> >> > figure
>> > > > > >>> >> > > out what the problem is? Other topics such as
>> > > > > >>> >> "hdfs_audit_event_sandbox"
>> > > > > >>> >> > > are mentioned in the streams window, but the
>> documentation
>> > > > > doesn't
>> > > > > >>> >> say to
>> > > > > >>> >> > > create them.
>> > > > > >>> >> > >
>> > > > > >>> >> > > The UI is buggy though on both Firefox and Chromium on
>> > > Linux.
>> > > > > What
>> > > > > >>> >> > > browser/platform are people using with the UI?
>> > > > > >>> >> > >
>> > > > > >>> >> > > Colm.
>> > > > > >>> >> > >
>> > > > > >>> >> > > On Wed, Jan 17, 2018 at 12:27 AM, Jayesh Senjaliya <
>> > > > > >>> jaysen@apache.org
>> > > > > >>> >> >
>> > > > > >>> >> > > wrote:
>> > > > > >>> >> > >
>> > > > > >>> >> > > > Hi Colm,
>> > > > > >>> >> > > >
>> > > > > >>> >> > > > Please find my comments inline.
>> > > > > >>> >> > > >
>> > > > > >>> >> > > > a) The official docker image uses 0.5.0-SNAPSHOT and
>> not
>> > > the
>> > > > > >>> >> released
>> > > > > >>> >> > > > version.
>> > > > > >>> >> > > > - this is because we uploaded docker image before
>> apache
>> > > > > >>> release.
>> > > > > >>> >> > > actually
>> > > > > >>> >> > > > this is same codebase apache-eagle-0.5, and it can be
>> > > fixed
>> > > > > >>> easily
>> > > > > >>> >> by
>> > > > > >>> >> > > just
>> > > > > >>> >> > > > rebuilding docker image. there should not be any
>> > mismatch
>> > > > due
>> > > > > to
>> > > > > >>> >> this.
>> > > > > >>> >> > > >
>> > > > > >>> >> > > > b) Aside from the above, the official docker image
>> uses
>> > a
>> > > > mix
>> > > > > >>> of "
>> > > > > >>> >> > > > server.eagle.apache.org" and "
>> sandbox.eagle.apache.org"
>> > > as
>> > > > > the
>> > > > > >>> host
>> > > > > >>> >> > > name.
>> > > > > >>> >> > > > The HBase service doesn't start by default in Ambari
>> as
>> > a
>> > > > > >>> result.
>> > > > > >>> >> > > > - the only places it uses sandbox is in example
>> script
>> > > which
>> > > > > you
>> > > > > >>> >> will
>> > > > > >>> >> > > have
>> > > > > >>> >> > > > to update anyway, which i agree that it would be
>> good to
>> > > > keep
>> > > > > it
>> > > > > >>> >> > > > consistent.
>> > > > > >>> >> > > >
>> > > > > >>> >> > > > c) The UI seems quite buggy. On both chromium and
>> > > firefox, I
>> > > > > >>> only
>> > > > > >>> >> see
>> > > > > >>> >> > > > links to "Sandbox" and "Alert" on the left hand-side.
>> > > Once I
>> > > > > >>> click
>> > > > > >>> >> on
>> > > > > >>> >> > > > "Alert" I have no way of going back to see the
>> > > > applications. I
>> > > > > >>> don't
>> > > > > >>> >> > see
>> > > > > >>> >> > > > the links to "integration" or "sites" as in the
>> picture
>> > > > here:
>> > > > > >>> >> > > > http://eagle.apache.org/docs/l
>> atest/applications/#jmx-
>> > > > monito
>> > > > > >>> ring
>> > > > > >>> >> > > > - when hbase is as deep storage is used, and if eagle
>> > app
>> > > > has
>> > > > > >>> issue
>> > > > > >>> >> > > > connecting to hbase, the UI becomes unresponsive.
>> > > > > >>> >> > > >
>> > > > > >>> >> > > > d) In chromium, the button to create a new policy
>> does
>> > not
>> > > > > >>> exist - I
>> > > > > >>> >> > can
>> > > > > >>> >> > > > only see it on Firefox.
>> > > > > >>> >> > > > - i have seen when you logged in, you will see admin
>> > > > actions.
>> > > > > >>> but if
>> > > > > >>> >> > this
>> > > > > >>> >> > > > still an issue, can you please file UI bug?
>> > > > > >>> >> > > >
>> > > > > >>> >> > > > e) I'm trying to get the "Hdfs Audit Log Monitor"
>> > use-case
>> > > > > >>> working,
>> > > > > >>> >> but
>> > > > > >>> >> > > it
>> > > > > >>> >> > > > seems to be stuck in "Initialized".
>> > > > > >>> >> > > > this eagle docs has example on how to setup the app.
>> pls
>> > > let
>> > > > > us
>> > > > > >>> >> know if
>> > > > > >>> >> > > > you find any gaps.
>> > > > > >>> >> > > >
>> > > > > >>> >> > > > Thanks for trying out, and sharing your findings,
>> > > > > >>> >> > > > Jayesh
>> > > > > >>> >> > > >
>> > > > > >>> >> > > >
>> > > > > >>> >> > > > On Tue, Jan 16, 2018 at 3:34 AM, Colm O hEigeartaigh
>> <
>> > > > > >>> >> > > coheigea@apache.org>
>> > > > > >>> >> > > > wrote:
>> > > > > >>> >> > > >
>> > > > > >>> >> > > >> Hi all,
>> > > > > >>> >> > > >>
>> > > > > >>> >> > > >> I'm trying to play around a bit with Apache Eagle
>> 0.5.0
>> > > to
>> > > > no
>> > > > > >>> >> avail.
>> > > > > >>> >> > > Here
>> > > > > >>> >> > > >> are the problems I've run into so far:
>> > > > > >>> >> > > >>
>> > > > > >>> >> > > >> a) The official docker image uses 0.5.0-SNAPSHOT and
>> > not
>> > > > the
>> > > > > >>> >> released
>> > > > > >>> >> > > >> version.
>> > > > > >>> >> > > >>
>> > > > > >>> >> > > >> b) Aside from the above, the official docker image
>> > uses a
>> > > > mix
>> > > > > >>> of "
>> > > > > >>> >> > > >> server.eagle.apache.org" and "
>> sandbox.eagle.apache.org
>> > "
>> > > as
>> > > > > the
>> > > > > >>> >> host
>> > > > > >>> >> > > >> name. The HBase service doesn't start by default in
>> > > Ambari
>> > > > > as a
>> > > > > >>> >> > result.
>> > > > > >>> >> > > >>
>> > > > > >>> >> > > >> c) The UI seems quite buggy. On both chromium and
>> > > firefox,
>> > > > I
>> > > > > >>> only
>> > > > > >>> >> see
>> > > > > >>> >> > > >> links to "Sandbox" and "Alert" on the left
>> hand-side.
>> > > Once
>> > > > I
>> > > > > >>> click
>> > > > > >>> >> on
>> > > > > >>> >> > > >> "Alert" I have no way of going back to see the
>> > > > applications.
>> > > > > I
>> > > > > >>> >> don't
>> > > > > >>> >> > see
>> > > > > >>> >> > > >> the links to "integration" or "sites" as in the
>> picture
>> > > > here:
>> > > > > >>> >> > > >> http://eagle.apache.org/docs/l
>> atest/applications/#jmx-
>> > > > monito
>> > > > > >>> ring
>> > > > > >>> >> > > >>
>> > > > > >>> >> > > >> d) In chromium, the button to create a new policy
>> does
>> > > not
>> > > > > >>> exist -
>> > > > > >>> >> I
>> > > > > >>> >> > can
>> > > > > >>> >> > > >> only see it on Firefox.
>> > > > > >>> >> > > >>
>> > > > > >>> >> > > >> e) I'm trying to get the "Hdfs Audit Log Monitor"
>> > > use-case
>> > > > > >>> working,
>> > > > > >>> >> > but
>> > > > > >>> >> > > >> it seems to be stuck in "Initialized".
>> > > > > >>> >> > > >>
>> > > > > >>> >> > > >> Could someone fill me in on what the "recommended"
>> way
>> > is
>> > > > to
>> > > > > >>> start
>> > > > > >>> >> > > Apache
>> > > > > >>> >> > > >> Eagle so that I can play around with the
>> functionality
>> > > that
>> > > > > it
>> > > > > >>> >> offers?
>> > > > > >>> >> > > >> Clearly the docker approach is buggy. Also, what
>> > browser
>> > > > > >>> should be
>> > > > > >>> >> > used?
>> > > > > >>> >> > > >>
>> > > > > >>> >> > > >> Thanks,
>> > > > > >>> >> > > >>
>> > > > > >>> >> > > >> Colm.
>> > > > > >>> >> > > >>
>> > > > > >>> >> > > >>
>> > > > > >>> >> > > >> --
>> > > > > >>> >> > > >> Colm O hEigeartaigh
>> > > > > >>> >> > > >>
>> > > > > >>> >> > > >> Talend Community Coder
>> > > > > >>> >> > > >> http://coders.talend.com
>> > > > > >>> >> > > >>
>> > > > > >>> >> > > >
>> > > > > >>> >> > > >
>> > > > > >>> >> > >
>> > > > > >>> >> > >
>> > > > > >>> >> > > --
>> > > > > >>> >> > > Colm O hEigeartaigh
>> > > > > >>> >> > >
>> > > > > >>> >> > > Talend Community Coder
>> > > > > >>> >> > > http://coders.talend.com
>> > > > > >>> >> > >
>> > > > > >>> >> >
>> > > > > >>> >>
>> > > > > >>> >>
>> > > > > >>> >>
>> > > > > >>> >> --
>> > > > > >>> >> Colm O hEigeartaigh
>> > > > > >>> >>
>> > > > > >>> >> Talend Community Coder
>> > > > > >>> >> http://coders.talend.com
>> > > > > >>> >>
>> > > > > >>> >>
>> > > > > >>> >>
>> > > > > >>> >
>> > > > > >>> >
>> > > > > >>> > --
>> > > > > >>> > Colm O hEigeartaigh
>> > > > > >>> >
>> > > > > >>> > Talend Community Coder
>> > > > > >>> > http://coders.talend.com
>> > > > > >>> >
>> > > > > >>>
>> > > > > >>>
>> > > > > >>>
>> > > > > >>> --
>> > > > > >>> Colm O hEigeartaigh
>> > > > > >>>
>> > > > > >>> Talend Community Coder
>> > > > > >>> http://coders.talend.com
>> > > > > >>>
>> > > > > >>
>> > > > > >>
>> > > > > >
>> > > > >
>> > > > >
>> > > > > --
>> > > > > Colm O hEigeartaigh
>> > > > >
>> > > > > Talend Community Coder
>> > > > > http://coders.talend.com
>> > > > >
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > > Colm O hEigeartaigh
>> > >
>> > > Talend Community Coder
>> > > http://coders.talend.com
>> > >
>> >
>>
>>
>>
>> --
>> Colm O hEigeartaigh
>>
>> Talend Community Coder
>> http://coders.talend.com
>>
>
>


-- 
Colm O hEigeartaigh

Talend Community Coder
http://coders.talend.com

Re: Unable to get 0.5.0 release working

Posted by Jayesh Senjaliya <ja...@apache.org>.
Thanks for the PRs. I have merged them.

welcome to the developer community Colm. I have also added you to jira
project so can assign the tasks to yourself.

lets create ticket to fix the dedup functionality, I m actually surprised
we havent hit this issue yet. we do use multiple publishers but someone can
verify this.

Thanks
Jayesh



On Wed, Jan 31, 2018 at 9:25 AM, Colm O hEigeartaigh <co...@apache.org>
wrote:

> Thanks Jayesh. I've created a JIRA here for some admin work for some issues
> that were incorrectly flagged as "fix for" 0.5.1/0.6.0:
>
> https://issues.apache.org/jira/browse/EAGLE-1076
>
> I've submitted the following (fairly trivial) pull requests. Could I ask
> that you or one of the other committers review?
>
> https://github.com/apache/eagle/pull/978
> https://github.com/apache/eagle/pull/979
> https://github.com/apache/eagle/pull/980
>
> It would be good to try to inject some energy into the project. We need
> more than one active committer though.
>
> Just in terms of the Alert Deduplication issue. The DefaultDeDuplicator
> works per "output" in the policy rule. So if you have more than one
> AlertPublisher, I think it is guaranteed to only publish to one of them.
> Instead, surely it would make more sense to work per publisher?
>
> Colm.
>
> On Tue, Jan 30, 2018 at 10:39 PM, Jayesh Senjaliya <ja...@apache.org>
> wrote:
>
> > Hi Colm,
> >
> > appreciate your suggestions/ efforts in looking into this project,
> > putting my comments inline...
> >
> > a) There is already a JIRA to bump the version here, although the PR does
> > not apply as it is too old: https://issues.apache.org/
> > jira/browse/EAGLE-1025
> > .
> > I can submit a new PR, but should the version be 0.6.0 or 0.5.1?
> >
> > *since there are still minor issues, i would say, we put up 0.5.1 as next
> > version. I've updated/rebased the PR (
> > https://github.com/apache/eagle/pull/936
> > <https://github.com/apache/eagle/pull/936> )*
> >
> >
> > b) The issues that are "resolved" for the 0.5.1 release in JIRA are
> > actually already fixed in 0.5.0, so they should be updated (
> > https://issues.apache.org/jira/projects/EAGLE/versions/12341128).
> However,
> > the following two issues are resolved even though they are not merged to
> > master?
> >   https://issues.apache.org/jira/browse/EAGLE-1051 .  - * this was
> pending
> > from developer;s response but i think this is reviewed, so I have merged
> > it.*
> >   https://issues.apache.org/jira/browse/EAGLE-1068 .  - * this is
> reopened
> > now. I dont think this is done yet. Also this is big change.*
> >
> >
> > Like I said I can submit PRs but I'm not convinced there is any activity
> on
> > the project. Where are the rest of the committers?
> >
> > *let me give you some ocontext on this. so there were lot of development
> > happened during last releases, and most of applications that were added
> are
> > being used in production at multiple enterprise companies, but we are out
> > of ideas on new apps, so at this point we are only focusing on bug fixes
> > and tech upgrades until we get some new ideas to brainstorm and add.*
> >
> > *I think current community's thinking is based on their own industries
> > use-cases, but there is definitely room for new features and integration
> > with other monitoring and security components like grafana and rangers.*
> >
> >
> > *Thanks,*
> > *Jayesh*
> >
> >
> >
> > On Tue, Jan 30, 2018 at 8:11 AM, Colm O hEigeartaigh <
> coheigea@apache.org>
> > wrote:
> >
> > > Hi Jayesh,
> > >
> > > Dev suggestions:
> > >
> > > a) There is already a JIRA to bump the version here, although the PR
> does
> > > not apply as it is too old: https://issues.apache.org/
> > > jira/browse/EAGLE-1025.
> > > I can submit a new PR, but should the version be 0.6.0 or 0.5.1?
> > > b) The issues that are "resolved" for the 0.5.1 release in JIRA are
> > > actually already fixed in 0.5.0, so they should be updated (
> > > https://issues.apache.org/jira/projects/EAGLE/versions/12341128).
> > However,
> > > the following two issues are resolved even though they are not merged
> to
> > > master?
> > >   https://issues.apache.org/jira/browse/EAGLE-1051
> > >   https://issues.apache.org/jira/browse/EAGLE-1068
> > >
> > > Like I said I can submit PRs but I'm not convinced there is any
> activity
> > on
> > > the project. Where are the rest of the committers?
> > >
> > > Multiple Publisher issue:
> > >
> > > If I assign two publishers for one policy, the alert only goes to the
> > first
> > > policy. In the logs I see:
> > >
> > > 2018-01-30T15:52:45.835+0000 o.a.e.a.e.p.d.DefaultDeduplicator [INFO]
> > > Alert
> > > event is skipped because it's duplicated: Alert {site=sandbox,
> > > stream=eagle_output,timestamp=2018-01-30
> > > 00:00:11,300,data={securityZone=NA, dst=null, sensitivityType=NA,
> > > src=/apps/hbase/data/archive/data/default/ambarismoketest,
> allowed=true,
> > > host=172.22.7.129, cmd=listStatus, user=SOMETHING7.COM,
> > > timestamp=1517270411300}, policyId=test,
> > > createdBy=alertBolt3-evaluator_stage1, metaVersion=null}
> > >
> > > It looks like this deduplicator is not working properly, as I'm
> guessing
> > it
> > > should only be used to de-duplicate events for a single publisher?
> > >
> > > Incognito mode: Already tried it but with the same result. Could I ask
> > you
> > > to try the docker image to see if the UI is working correctly for you
> > > there?
> > >
> > > Colm.
> > >
> > > On Mon, Jan 29, 2018 at 6:46 PM, Jayesh Senjaliya <ja...@apache.org>
> > > wrote:
> > >
> > > > Hi Colm,
> > > >
> > > > Thanks for the list of dev suggestions, I think we should take care
> of
> > > > those. even better if you can provide PR with the changes or at keast
> > can
> > > > you please create a ticket so we can track it?
> > > >
> > > > for other issues.
> > > >
> > > > - I dont have any issue with multiple publisher, but if there is any
> > > error
> > > > updating the publisher info in storm topology, i might try restarting
> > the
> > > > topology and see if that works.
> > > > - for us, chrome works as fine as firefox.  can u try incognito mode?
> > > just
> > > > to be sure to have clean cache?
> > > >
> > > > Thanks
> > > > Jayesh
> > > >
> > > >
> > > > On Thu, Jan 25, 2018 at 4:19 AM, Colm O hEigeartaigh <
> > > coheigea@apache.org>
> > > > wrote:
> > > >
> > > > > Thanks again for your feedback. Jayesh, adding
> AlertEagleStorePlugin
> > > did
> > > > > the trick, I can now see alerts in the UI, thanks! By the way, I
> > can't
> > > > > configure two Alert Publishers, or else the Alert DeDuplicator bins
> > the
> > > > > alert. Is this a known issue?
> > > > >
> > > > > Could I ask which browser people are using with the UI? There
> appears
> > > to
> > > > be
> > > > > a  bug with Chromium where it doesn't list the pages under
> > Auth.isAdmin
> > > > > even though I am logged on as an administrator. It works OK in
> > Firefox.
> > > > > Even with Firefox though, I only see a limited number of links in
> the
> > > > > left-hand column - I can't get back to the "integration" page. Can
> > > > someone
> > > > > else confirm this please?
> > > > >
> > > > > Could I suggest the devs do some basic house-keeping tasks:
> > > > >
> > > > > a) "Release" version 0.5.0 in JIRA (it's still listed as
> > "unreleased").
> > > > > b) Figure out whether the next version will be 0.5.1 or 0.6.0 and
> > > update
> > > > > the versions on Master accordingly with 0.5.1-SNAPSHOT or
> > > 0.6.0-SNAPSHOT.
> > > > > There are some issues marked here as resolved for 0.5.1 -
> > > > > https://issues.apache.org/jira/projects/EAGLE/versions/12341128),
> > > > however
> > > > > I
> > > > > don't see a branch for 0.5.x?
> > > > >
> > > > > Colm.
> > > > >
> > > > > On Thu, Jan 25, 2018 at 8:16 AM, Jayesh Senjaliya <
> jaysen@apache.org
> > >
> > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > we do use eagle 0.5 in production although we dont use all the
> > > > available
> > > > > > hadoop applications.
> > > > > >
> > > > > > EAGLE-968 <https://issues.apache.org/jira/browse/EAGLE-968> is a
> > fix
> > > > for
> > > > > > email issue we found while our testing. should be merged soon
> > after a
> > > > > > rebase.
> > > > > >
> > > > > > @Colm, did you tried adding storage publisher
> > > (AlertEagleStorePlugin)?
> > > > to
> > > > > > see alerts on UI ?
> > > > > >
> > > > > > Thanks
> > > > > > Jayesh
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Wed, Jan 24, 2018 at 7:08 PM, Edward Zhang <
> > > yonzhang2012@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > >> Eagle 0.5 was deployed in production as far as I know, but it
> may
> > > not
> > > > be
> > > > > >> exact the current version in master branch.
> > > > > >>
> > > > > >> Thanks for your investigation, seems there is still some bug in
> > 0.5,
> > > > but
> > > > > >> this particular issue seems is due to dependent components
> version
> > > > > conflict.
> > > > > >>
> > > > > >> @Jayesh is this Jira ready for merge to master?
> > > https://issues.apache
> > > > .
> > > > > >> org/jira/browse/EAGLE-968
> > > > > >>
> > > > > >>
> > > > > >> Thanks
> > > > > >> Edward
> > > > > >>
> > > > > >> On Tue, Jan 23, 2018 at 5:10 AM, Colm O hEigeartaigh <
> > > > > coheigea@apache.org
> > > > > >> > wrote:
> > > > > >>
> > > > > >>> OK I've made some more progress. I wasn't seeing any email
> alerts
> > > due
> > > > > to
> > > > > >>> https://issues.apache.org/jira/browse/EAGLE-968. Once I
> > configure
> > > a
> > > > > >>> Kafka
> > > > > >>> alert, I can see the alerts flowing into my topic. It's still
> not
> > > > clear
> > > > > >>> to
> > > > > >>> me however where the policy "output" is going. I also don't see
> > any
> > > > > >>> alerts
> > > > > >>> in the UI window.
> > > > > >>>
> > > > > >>> Could I ask what the status of the project is in general? There
> > > have
> > > > > been
> > > > > >>> no commits to master since November, so I'm not sure if there
> is
> > > any
> > > > > >>> point
> > > > > >>> in submitting Pull Requests for outstanding bugs? Are recent
> > > versions
> > > > > of
> > > > > >>> Apache Eagle used in production?
> > > > > >>>
> > > > > >>> Colm.
> > > > > >>>
> > > > > >>> On Mon, Jan 22, 2018 at 1:07 PM, Colm O hEigeartaigh <
> > > > > >>> coheigea@apache.org>
> > > > > >>> wrote:
> > > > > >>>
> > > > > >>> >
> > > > > >>> > I've done that but I'm not seeing any alerts, which is why I
> > want
> > > > to
> > > > > >>> find
> > > > > >>> > out what the "output" of a policy is and where I can check
> > this.
> > > > > >>> >
> > > > > >>> > Colm.
> > > > > >>> >
> > > > > >>> > On Mon, Jan 22, 2018 at 1:05 PM, SUDHA JENSLIN <
> > > sjenslin@gmail.com
> > > > >
> > > > > >>> wrote:
> > > > > >>> >
> > > > > >>> >> Create and add a publisher to see the output.
> > > > > >>> >>
> > > > > >>> >>
> > > > > >>> >>
> > > > > >>> >> Regards,
> > > > > >>> >> Sudha jenslin
> > > > > >>> >>
> > > > > >>> >> On Jan 22, 2018 6:31 PM, "Colm O hEigeartaigh" <
> > > > coheigea@apache.org
> > > > > >
> > > > > >>> >> wrote:
> > > > > >>> >>
> > > > > >>> >> Thanks - the error was due to a problem running Storm with
> > Java
> > > > 1.8.
> > > > > >>> I've
> > > > > >>> >> abandoned the docker image for now, and I'm trying to get it
> > > > working
> > > > > >>> >> locally.
> > > > > >>> >>
> > > > > >>> >> There are two things I'm not clear on currently, if someone
> > > could
> > > > > >>> fill me
> > > > > >>> >> in:
> > > > > >>> >>
> > > > > >>> >> a) For the  'Hdfs Audit Log Monitor' application, the Kafka
> > > > Consumer
> > > > > >>> Topic
> > > > > >>> >> is 'hdfs_audit_log_sandbox'. Under 'Kafka Topic for Auditlog
> > > Event
> > > > > >>> Sink'
> > > > > >>> >> it
> > > > > >>> >> also specifies 'hdfs_audit_event_sandbox'. However the
> > > > documentation
> > > > > >>> for
> > > > > >>> >> the application mentions 'hdfs_audit_log_enriched_sandbox'?
> > > > > >>> >>
> > > > > >>> >> When I click on "STREAMS", the
> "HDFS_AUDIT_LOG_ENRICHED_STREA
> > > > > >>> M_SANDBOX"
> > > > > >>> >> uses the topic "hdfs_audit_event_sandbox". And indeed when I
> > run
> > > > the
> > > > > >>> >> application, I can see cleansed log data appearing in
> > > > > >>> >> "hdfs_audit_event_sandbox". So I'm thinking here that
> > > > > >>> >> 'hdfs_audit_log_enriched_sandbox' is not correct or
> > necessary?
> > > > > >>> >>
> > > > > >>> >> b) It's unclear to me where the output data goes when you
> > > create a
> > > > > >>> policy.
> > > > > >>> >> E.g. say I have:
> > > > > >>> >>
> > > > > >>> >> from HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX[str:contains(
> > > src,'/hb
> > > > > >>> ase')]
> > > > > >>> >> select * group by user insert into hdfs_audit_log_enriched_
> > > > > stream_out
> > > > > >>> >>
> > > > > >>> >> Where is "hdfs_audit_log_enriched_stream_out" defined (is
> it
> > a
> > > > > Kafka
> > > > > >>> >> topic?). How can I check the output to make sure the policy
> is
> > > > > working
> > > > > >>> >> correctly?
> > > > > >>> >>
> > > > > >>> >> Thanks,
> > > > > >>> >>
> > > > > >>> >> Colm.
> > > > > >>> >>
> > > > > >>> >> On Wed, Jan 17, 2018 at 10:32 PM, Edward Zhang <
> > > > > >>> yonzhang2012@gmail.com>
> > > > > >>> >> wrote:
> > > > > >>> >>
> > > > > >>> >> > There is a data preparation stage between data source(HDFS
> > > audit
> > > > > >>> log)
> > > > > >>> >> and
> > > > > >>> >> > Alert Engine. This stage is running in Storm and transform
> > the
> > > > raw
> > > > > >>> HDFS
> > > > > >>> >> log
> > > > > >>> >> > into something which can be alerted.
> > > > > >>> >> >
> > > > > >>> >> > The input for data preparation is hdfs_audit_log_sandbox
> > topic
> > > > and
> > > > > >>> >> output
> > > > > >>> >> > is
> > > > > >>> >> >  hdfs_audit_log_enriched_sandbox.
> > > > > >>> >> > The input for Alert Engine is hdfs_audit_log_enriched_
> > sandbox
> > > > and
> > > > > >>> >> output
> > > > > >>> >> > is
> > > > > >>> >> > hdfs_audit_log_alert_sandbox.
> > > > > >>> >> >
> > > > > >>> >> > Seems in your case, the data preparation staging is not
> > > working.
> > > > > We
> > > > > >>> >> > probably need look at Storm console and figure out if that
> > > part
> > > > is
> > > > > >>> >> working.
> > > > > >>> >> >
> > > > > >>> >> > Thanks
> > > > > >>> >> > Edward
> > > > > >>> >> >
> > > > > >>> >> > On Wed, Jan 17, 2018 at 7:19 AM, Colm O hEigeartaigh <
> > > > > >>> >> coheigea@apache.org>
> > > > > >>> >> > wrote:
> > > > > >>> >> >
> > > > > >>> >> > > Hi Jayesh,
> > > > > >>> >> > >
> > > > > >>> >> > > Many thanks for your feedback! I was able to make a
> little
> > > > > further
> > > > > >>> >> > headway.
> > > > > >>> >> > > There are two configuration problems with the official
> > > docker
> > > > > >>> image:
> > > > > >>> >> > >
> > > > > >>> >> > > a) A mix of "sandbox.eagle.apache.org" and "
> > > > > >>> server.eagle.apache.org"
> > > > > >>> >> > (this
> > > > > >>> >> > > only occurs in the instructions for running the docker
> > > image.
> > > > > The
> > > > > >>> >> version
> > > > > >>> >> > > that can be started via the script in the eagle source
> is
> > > OK).
> > > > > >>> I'll
> > > > > >>> >> > submit
> > > > > >>> >> > > a PR to fix this once I get a basic use-case working.
> > > > > >>> >> > > b) For the audit case, it automatically logs HDFS audit
> > logs
> > > > to
> > > > > >>> the
> > > > > >>> >> KAFKA
> > > > > >>> >> > > topic sandbox_hdfs_audit_log instead of the expected
> > > > > >>> >> > hdfs_audit_log_sandbox
> > > > > >>> >> > >
> > > > > >>> >> > > I've fixed these things locally and I can verify that
> > > > everything
> > > > > >>> is
> > > > > >>> >> > started
> > > > > >>> >> > > correctly in Ambari. I log into the docker container and
> > > > create
> > > > > >>> >> > > hdfs_audit_log_sandbox and hdfs_audit_log_enriched_
> > sandbox
> > > > > >>> topics,
> > > > > >>> >> and
> > > > > >>> >> > > verify that the HDFS audit logs are flowing into the
> first
> > > > > topic.
> > > > > >>> >> Then in
> > > > > >>> >> > > the UI I start the Alert Engine and then the HDFS Audit
> > Log
> > > > > >>> Monitor
> > > > > >>> >> > > application (changing localhost:6667 to
> > > > > >>> server.eagle.apache.org:6667
> > > > > >>> >> ).
> > > > > >>> >> > > Both
> > > > > >>> >> > > applications start up correctly and show "running".
> > > > > >>> >> > >
> > > > > >>> >> > > I then create a policy with an email alert along the
> lines
> > > of
> > > > > from
> > > > > >>> >> > > "HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX[str:contains(
> > > src,'/h
> > > > > >>> base')]
> > > > > >>> >> > select
> > > > > >>> >> > > * group by user insert into hdfs_audit_log_enriched_
> > > > > stream_out".
> > > > > >>> >> However
> > > > > >>> >> > > at
> > > > > >>> >> > > this point I'm stuck - nothing appears in the alert
> > window.
> > > Is
> > > > > >>> there
> > > > > >>> >> > > anything obvious I'm doing wrong, or how can I get
> access
> > to
> > > > > logs
> > > > > >>> to
> > > > > >>> >> > figure
> > > > > >>> >> > > out what the problem is? Other topics such as
> > > > > >>> >> "hdfs_audit_event_sandbox"
> > > > > >>> >> > > are mentioned in the streams window, but the
> documentation
> > > > > doesn't
> > > > > >>> >> say to
> > > > > >>> >> > > create them.
> > > > > >>> >> > >
> > > > > >>> >> > > The UI is buggy though on both Firefox and Chromium on
> > > Linux.
> > > > > What
> > > > > >>> >> > > browser/platform are people using with the UI?
> > > > > >>> >> > >
> > > > > >>> >> > > Colm.
> > > > > >>> >> > >
> > > > > >>> >> > > On Wed, Jan 17, 2018 at 12:27 AM, Jayesh Senjaliya <
> > > > > >>> jaysen@apache.org
> > > > > >>> >> >
> > > > > >>> >> > > wrote:
> > > > > >>> >> > >
> > > > > >>> >> > > > Hi Colm,
> > > > > >>> >> > > >
> > > > > >>> >> > > > Please find my comments inline.
> > > > > >>> >> > > >
> > > > > >>> >> > > > a) The official docker image uses 0.5.0-SNAPSHOT and
> not
> > > the
> > > > > >>> >> released
> > > > > >>> >> > > > version.
> > > > > >>> >> > > > - this is because we uploaded docker image before
> apache
> > > > > >>> release.
> > > > > >>> >> > > actually
> > > > > >>> >> > > > this is same codebase apache-eagle-0.5, and it can be
> > > fixed
> > > > > >>> easily
> > > > > >>> >> by
> > > > > >>> >> > > just
> > > > > >>> >> > > > rebuilding docker image. there should not be any
> > mismatch
> > > > due
> > > > > to
> > > > > >>> >> this.
> > > > > >>> >> > > >
> > > > > >>> >> > > > b) Aside from the above, the official docker image
> uses
> > a
> > > > mix
> > > > > >>> of "
> > > > > >>> >> > > > server.eagle.apache.org" and "
> sandbox.eagle.apache.org"
> > > as
> > > > > the
> > > > > >>> host
> > > > > >>> >> > > name.
> > > > > >>> >> > > > The HBase service doesn't start by default in Ambari
> as
> > a
> > > > > >>> result.
> > > > > >>> >> > > > - the only places it uses sandbox is in example script
> > > which
> > > > > you
> > > > > >>> >> will
> > > > > >>> >> > > have
> > > > > >>> >> > > > to update anyway, which i agree that it would be good
> to
> > > > keep
> > > > > it
> > > > > >>> >> > > > consistent.
> > > > > >>> >> > > >
> > > > > >>> >> > > > c) The UI seems quite buggy. On both chromium and
> > > firefox, I
> > > > > >>> only
> > > > > >>> >> see
> > > > > >>> >> > > > links to "Sandbox" and "Alert" on the left hand-side.
> > > Once I
> > > > > >>> click
> > > > > >>> >> on
> > > > > >>> >> > > > "Alert" I have no way of going back to see the
> > > > applications. I
> > > > > >>> don't
> > > > > >>> >> > see
> > > > > >>> >> > > > the links to "integration" or "sites" as in the
> picture
> > > > here:
> > > > > >>> >> > > > http://eagle.apache.org/docs/
> latest/applications/#jmx-
> > > > monito
> > > > > >>> ring
> > > > > >>> >> > > > - when hbase is as deep storage is used, and if eagle
> > app
> > > > has
> > > > > >>> issue
> > > > > >>> >> > > > connecting to hbase, the UI becomes unresponsive.
> > > > > >>> >> > > >
> > > > > >>> >> > > > d) In chromium, the button to create a new policy does
> > not
> > > > > >>> exist - I
> > > > > >>> >> > can
> > > > > >>> >> > > > only see it on Firefox.
> > > > > >>> >> > > > - i have seen when you logged in, you will see admin
> > > > actions.
> > > > > >>> but if
> > > > > >>> >> > this
> > > > > >>> >> > > > still an issue, can you please file UI bug?
> > > > > >>> >> > > >
> > > > > >>> >> > > > e) I'm trying to get the "Hdfs Audit Log Monitor"
> > use-case
> > > > > >>> working,
> > > > > >>> >> but
> > > > > >>> >> > > it
> > > > > >>> >> > > > seems to be stuck in "Initialized".
> > > > > >>> >> > > > this eagle docs has example on how to setup the app.
> pls
> > > let
> > > > > us
> > > > > >>> >> know if
> > > > > >>> >> > > > you find any gaps.
> > > > > >>> >> > > >
> > > > > >>> >> > > > Thanks for trying out, and sharing your findings,
> > > > > >>> >> > > > Jayesh
> > > > > >>> >> > > >
> > > > > >>> >> > > >
> > > > > >>> >> > > > On Tue, Jan 16, 2018 at 3:34 AM, Colm O hEigeartaigh <
> > > > > >>> >> > > coheigea@apache.org>
> > > > > >>> >> > > > wrote:
> > > > > >>> >> > > >
> > > > > >>> >> > > >> Hi all,
> > > > > >>> >> > > >>
> > > > > >>> >> > > >> I'm trying to play around a bit with Apache Eagle
> 0.5.0
> > > to
> > > > no
> > > > > >>> >> avail.
> > > > > >>> >> > > Here
> > > > > >>> >> > > >> are the problems I've run into so far:
> > > > > >>> >> > > >>
> > > > > >>> >> > > >> a) The official docker image uses 0.5.0-SNAPSHOT and
> > not
> > > > the
> > > > > >>> >> released
> > > > > >>> >> > > >> version.
> > > > > >>> >> > > >>
> > > > > >>> >> > > >> b) Aside from the above, the official docker image
> > uses a
> > > > mix
> > > > > >>> of "
> > > > > >>> >> > > >> server.eagle.apache.org" and "
> sandbox.eagle.apache.org
> > "
> > > as
> > > > > the
> > > > > >>> >> host
> > > > > >>> >> > > >> name. The HBase service doesn't start by default in
> > > Ambari
> > > > > as a
> > > > > >>> >> > result.
> > > > > >>> >> > > >>
> > > > > >>> >> > > >> c) The UI seems quite buggy. On both chromium and
> > > firefox,
> > > > I
> > > > > >>> only
> > > > > >>> >> see
> > > > > >>> >> > > >> links to "Sandbox" and "Alert" on the left hand-side.
> > > Once
> > > > I
> > > > > >>> click
> > > > > >>> >> on
> > > > > >>> >> > > >> "Alert" I have no way of going back to see the
> > > > applications.
> > > > > I
> > > > > >>> >> don't
> > > > > >>> >> > see
> > > > > >>> >> > > >> the links to "integration" or "sites" as in the
> picture
> > > > here:
> > > > > >>> >> > > >> http://eagle.apache.org/docs/
> latest/applications/#jmx-
> > > > monito
> > > > > >>> ring
> > > > > >>> >> > > >>
> > > > > >>> >> > > >> d) In chromium, the button to create a new policy
> does
> > > not
> > > > > >>> exist -
> > > > > >>> >> I
> > > > > >>> >> > can
> > > > > >>> >> > > >> only see it on Firefox.
> > > > > >>> >> > > >>
> > > > > >>> >> > > >> e) I'm trying to get the "Hdfs Audit Log Monitor"
> > > use-case
> > > > > >>> working,
> > > > > >>> >> > but
> > > > > >>> >> > > >> it seems to be stuck in "Initialized".
> > > > > >>> >> > > >>
> > > > > >>> >> > > >> Could someone fill me in on what the "recommended"
> way
> > is
> > > > to
> > > > > >>> start
> > > > > >>> >> > > Apache
> > > > > >>> >> > > >> Eagle so that I can play around with the
> functionality
> > > that
> > > > > it
> > > > > >>> >> offers?
> > > > > >>> >> > > >> Clearly the docker approach is buggy. Also, what
> > browser
> > > > > >>> should be
> > > > > >>> >> > used?
> > > > > >>> >> > > >>
> > > > > >>> >> > > >> Thanks,
> > > > > >>> >> > > >>
> > > > > >>> >> > > >> Colm.
> > > > > >>> >> > > >>
> > > > > >>> >> > > >>
> > > > > >>> >> > > >> --
> > > > > >>> >> > > >> Colm O hEigeartaigh
> > > > > >>> >> > > >>
> > > > > >>> >> > > >> Talend Community Coder
> > > > > >>> >> > > >> http://coders.talend.com
> > > > > >>> >> > > >>
> > > > > >>> >> > > >
> > > > > >>> >> > > >
> > > > > >>> >> > >
> > > > > >>> >> > >
> > > > > >>> >> > > --
> > > > > >>> >> > > Colm O hEigeartaigh
> > > > > >>> >> > >
> > > > > >>> >> > > Talend Community Coder
> > > > > >>> >> > > http://coders.talend.com
> > > > > >>> >> > >
> > > > > >>> >> >
> > > > > >>> >>
> > > > > >>> >>
> > > > > >>> >>
> > > > > >>> >> --
> > > > > >>> >> Colm O hEigeartaigh
> > > > > >>> >>
> > > > > >>> >> Talend Community Coder
> > > > > >>> >> http://coders.talend.com
> > > > > >>> >>
> > > > > >>> >>
> > > > > >>> >>
> > > > > >>> >
> > > > > >>> >
> > > > > >>> > --
> > > > > >>> > Colm O hEigeartaigh
> > > > > >>> >
> > > > > >>> > Talend Community Coder
> > > > > >>> > http://coders.talend.com
> > > > > >>> >
> > > > > >>>
> > > > > >>>
> > > > > >>>
> > > > > >>> --
> > > > > >>> Colm O hEigeartaigh
> > > > > >>>
> > > > > >>> Talend Community Coder
> > > > > >>> http://coders.talend.com
> > > > > >>>
> > > > > >>
> > > > > >>
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Colm O hEigeartaigh
> > > > >
> > > > > Talend Community Coder
> > > > > http://coders.talend.com
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Colm O hEigeartaigh
> > >
> > > Talend Community Coder
> > > http://coders.talend.com
> > >
> >
>
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>

Re: Unable to get 0.5.0 release working

Posted by Colm O hEigeartaigh <co...@apache.org>.
Thanks Jayesh. I've created a JIRA here for some admin work for some issues
that were incorrectly flagged as "fix for" 0.5.1/0.6.0:

https://issues.apache.org/jira/browse/EAGLE-1076

I've submitted the following (fairly trivial) pull requests. Could I ask
that you or one of the other committers review?

https://github.com/apache/eagle/pull/978
https://github.com/apache/eagle/pull/979
https://github.com/apache/eagle/pull/980

It would be good to try to inject some energy into the project. We need
more than one active committer though.

Just in terms of the Alert Deduplication issue. The DefaultDeDuplicator
works per "output" in the policy rule. So if you have more than one
AlertPublisher, I think it is guaranteed to only publish to one of them.
Instead, surely it would make more sense to work per publisher?

Colm.

On Tue, Jan 30, 2018 at 10:39 PM, Jayesh Senjaliya <ja...@apache.org>
wrote:

> Hi Colm,
>
> appreciate your suggestions/ efforts in looking into this project,
> putting my comments inline...
>
> a) There is already a JIRA to bump the version here, although the PR does
> not apply as it is too old: https://issues.apache.org/
> jira/browse/EAGLE-1025
> .
> I can submit a new PR, but should the version be 0.6.0 or 0.5.1?
>
> *since there are still minor issues, i would say, we put up 0.5.1 as next
> version. I've updated/rebased the PR (
> https://github.com/apache/eagle/pull/936
> <https://github.com/apache/eagle/pull/936> )*
>
>
> b) The issues that are "resolved" for the 0.5.1 release in JIRA are
> actually already fixed in 0.5.0, so they should be updated (
> https://issues.apache.org/jira/projects/EAGLE/versions/12341128). However,
> the following two issues are resolved even though they are not merged to
> master?
>   https://issues.apache.org/jira/browse/EAGLE-1051 .  - * this was pending
> from developer;s response but i think this is reviewed, so I have merged
> it.*
>   https://issues.apache.org/jira/browse/EAGLE-1068 .  - * this is reopened
> now. I dont think this is done yet. Also this is big change.*
>
>
> Like I said I can submit PRs but I'm not convinced there is any activity on
> the project. Where are the rest of the committers?
>
> *let me give you some ocontext on this. so there were lot of development
> happened during last releases, and most of applications that were added are
> being used in production at multiple enterprise companies, but we are out
> of ideas on new apps, so at this point we are only focusing on bug fixes
> and tech upgrades until we get some new ideas to brainstorm and add.*
>
> *I think current community's thinking is based on their own industries
> use-cases, but there is definitely room for new features and integration
> with other monitoring and security components like grafana and rangers.*
>
>
> *Thanks,*
> *Jayesh*
>
>
>
> On Tue, Jan 30, 2018 at 8:11 AM, Colm O hEigeartaigh <co...@apache.org>
> wrote:
>
> > Hi Jayesh,
> >
> > Dev suggestions:
> >
> > a) There is already a JIRA to bump the version here, although the PR does
> > not apply as it is too old: https://issues.apache.org/
> > jira/browse/EAGLE-1025.
> > I can submit a new PR, but should the version be 0.6.0 or 0.5.1?
> > b) The issues that are "resolved" for the 0.5.1 release in JIRA are
> > actually already fixed in 0.5.0, so they should be updated (
> > https://issues.apache.org/jira/projects/EAGLE/versions/12341128).
> However,
> > the following two issues are resolved even though they are not merged to
> > master?
> >   https://issues.apache.org/jira/browse/EAGLE-1051
> >   https://issues.apache.org/jira/browse/EAGLE-1068
> >
> > Like I said I can submit PRs but I'm not convinced there is any activity
> on
> > the project. Where are the rest of the committers?
> >
> > Multiple Publisher issue:
> >
> > If I assign two publishers for one policy, the alert only goes to the
> first
> > policy. In the logs I see:
> >
> > 2018-01-30T15:52:45.835+0000 o.a.e.a.e.p.d.DefaultDeduplicator [INFO]
> > Alert
> > event is skipped because it's duplicated: Alert {site=sandbox,
> > stream=eagle_output,timestamp=2018-01-30
> > 00:00:11,300,data={securityZone=NA, dst=null, sensitivityType=NA,
> > src=/apps/hbase/data/archive/data/default/ambarismoketest, allowed=true,
> > host=172.22.7.129, cmd=listStatus, user=SOMETHING7.COM,
> > timestamp=1517270411300}, policyId=test,
> > createdBy=alertBolt3-evaluator_stage1, metaVersion=null}
> >
> > It looks like this deduplicator is not working properly, as I'm guessing
> it
> > should only be used to de-duplicate events for a single publisher?
> >
> > Incognito mode: Already tried it but with the same result. Could I ask
> you
> > to try the docker image to see if the UI is working correctly for you
> > there?
> >
> > Colm.
> >
> > On Mon, Jan 29, 2018 at 6:46 PM, Jayesh Senjaliya <ja...@apache.org>
> > wrote:
> >
> > > Hi Colm,
> > >
> > > Thanks for the list of dev suggestions, I think we should take care of
> > > those. even better if you can provide PR with the changes or at keast
> can
> > > you please create a ticket so we can track it?
> > >
> > > for other issues.
> > >
> > > - I dont have any issue with multiple publisher, but if there is any
> > error
> > > updating the publisher info in storm topology, i might try restarting
> the
> > > topology and see if that works.
> > > - for us, chrome works as fine as firefox.  can u try incognito mode?
> > just
> > > to be sure to have clean cache?
> > >
> > > Thanks
> > > Jayesh
> > >
> > >
> > > On Thu, Jan 25, 2018 at 4:19 AM, Colm O hEigeartaigh <
> > coheigea@apache.org>
> > > wrote:
> > >
> > > > Thanks again for your feedback. Jayesh, adding AlertEagleStorePlugin
> > did
> > > > the trick, I can now see alerts in the UI, thanks! By the way, I
> can't
> > > > configure two Alert Publishers, or else the Alert DeDuplicator bins
> the
> > > > alert. Is this a known issue?
> > > >
> > > > Could I ask which browser people are using with the UI? There appears
> > to
> > > be
> > > > a  bug with Chromium where it doesn't list the pages under
> Auth.isAdmin
> > > > even though I am logged on as an administrator. It works OK in
> Firefox.
> > > > Even with Firefox though, I only see a limited number of links in the
> > > > left-hand column - I can't get back to the "integration" page. Can
> > > someone
> > > > else confirm this please?
> > > >
> > > > Could I suggest the devs do some basic house-keeping tasks:
> > > >
> > > > a) "Release" version 0.5.0 in JIRA (it's still listed as
> "unreleased").
> > > > b) Figure out whether the next version will be 0.5.1 or 0.6.0 and
> > update
> > > > the versions on Master accordingly with 0.5.1-SNAPSHOT or
> > 0.6.0-SNAPSHOT.
> > > > There are some issues marked here as resolved for 0.5.1 -
> > > > https://issues.apache.org/jira/projects/EAGLE/versions/12341128),
> > > however
> > > > I
> > > > don't see a branch for 0.5.x?
> > > >
> > > > Colm.
> > > >
> > > > On Thu, Jan 25, 2018 at 8:16 AM, Jayesh Senjaliya <jaysen@apache.org
> >
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > we do use eagle 0.5 in production although we dont use all the
> > > available
> > > > > hadoop applications.
> > > > >
> > > > > EAGLE-968 <https://issues.apache.org/jira/browse/EAGLE-968> is a
> fix
> > > for
> > > > > email issue we found while our testing. should be merged soon
> after a
> > > > > rebase.
> > > > >
> > > > > @Colm, did you tried adding storage publisher
> > (AlertEagleStorePlugin)?
> > > to
> > > > > see alerts on UI ?
> > > > >
> > > > > Thanks
> > > > > Jayesh
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Wed, Jan 24, 2018 at 7:08 PM, Edward Zhang <
> > yonzhang2012@gmail.com>
> > > > > wrote:
> > > > >
> > > > >> Eagle 0.5 was deployed in production as far as I know, but it may
> > not
> > > be
> > > > >> exact the current version in master branch.
> > > > >>
> > > > >> Thanks for your investigation, seems there is still some bug in
> 0.5,
> > > but
> > > > >> this particular issue seems is due to dependent components version
> > > > conflict.
> > > > >>
> > > > >> @Jayesh is this Jira ready for merge to master?
> > https://issues.apache
> > > .
> > > > >> org/jira/browse/EAGLE-968
> > > > >>
> > > > >>
> > > > >> Thanks
> > > > >> Edward
> > > > >>
> > > > >> On Tue, Jan 23, 2018 at 5:10 AM, Colm O hEigeartaigh <
> > > > coheigea@apache.org
> > > > >> > wrote:
> > > > >>
> > > > >>> OK I've made some more progress. I wasn't seeing any email alerts
> > due
> > > > to
> > > > >>> https://issues.apache.org/jira/browse/EAGLE-968. Once I
> configure
> > a
> > > > >>> Kafka
> > > > >>> alert, I can see the alerts flowing into my topic. It's still not
> > > clear
> > > > >>> to
> > > > >>> me however where the policy "output" is going. I also don't see
> any
> > > > >>> alerts
> > > > >>> in the UI window.
> > > > >>>
> > > > >>> Could I ask what the status of the project is in general? There
> > have
> > > > been
> > > > >>> no commits to master since November, so I'm not sure if there is
> > any
> > > > >>> point
> > > > >>> in submitting Pull Requests for outstanding bugs? Are recent
> > versions
> > > > of
> > > > >>> Apache Eagle used in production?
> > > > >>>
> > > > >>> Colm.
> > > > >>>
> > > > >>> On Mon, Jan 22, 2018 at 1:07 PM, Colm O hEigeartaigh <
> > > > >>> coheigea@apache.org>
> > > > >>> wrote:
> > > > >>>
> > > > >>> >
> > > > >>> > I've done that but I'm not seeing any alerts, which is why I
> want
> > > to
> > > > >>> find
> > > > >>> > out what the "output" of a policy is and where I can check
> this.
> > > > >>> >
> > > > >>> > Colm.
> > > > >>> >
> > > > >>> > On Mon, Jan 22, 2018 at 1:05 PM, SUDHA JENSLIN <
> > sjenslin@gmail.com
> > > >
> > > > >>> wrote:
> > > > >>> >
> > > > >>> >> Create and add a publisher to see the output.
> > > > >>> >>
> > > > >>> >>
> > > > >>> >>
> > > > >>> >> Regards,
> > > > >>> >> Sudha jenslin
> > > > >>> >>
> > > > >>> >> On Jan 22, 2018 6:31 PM, "Colm O hEigeartaigh" <
> > > coheigea@apache.org
> > > > >
> > > > >>> >> wrote:
> > > > >>> >>
> > > > >>> >> Thanks - the error was due to a problem running Storm with
> Java
> > > 1.8.
> > > > >>> I've
> > > > >>> >> abandoned the docker image for now, and I'm trying to get it
> > > working
> > > > >>> >> locally.
> > > > >>> >>
> > > > >>> >> There are two things I'm not clear on currently, if someone
> > could
> > > > >>> fill me
> > > > >>> >> in:
> > > > >>> >>
> > > > >>> >> a) For the  'Hdfs Audit Log Monitor' application, the Kafka
> > > Consumer
> > > > >>> Topic
> > > > >>> >> is 'hdfs_audit_log_sandbox'. Under 'Kafka Topic for Auditlog
> > Event
> > > > >>> Sink'
> > > > >>> >> it
> > > > >>> >> also specifies 'hdfs_audit_event_sandbox'. However the
> > > documentation
> > > > >>> for
> > > > >>> >> the application mentions 'hdfs_audit_log_enriched_sandbox'?
> > > > >>> >>
> > > > >>> >> When I click on "STREAMS", the "HDFS_AUDIT_LOG_ENRICHED_STREA
> > > > >>> M_SANDBOX"
> > > > >>> >> uses the topic "hdfs_audit_event_sandbox". And indeed when I
> run
> > > the
> > > > >>> >> application, I can see cleansed log data appearing in
> > > > >>> >> "hdfs_audit_event_sandbox". So I'm thinking here that
> > > > >>> >> 'hdfs_audit_log_enriched_sandbox' is not correct or
> necessary?
> > > > >>> >>
> > > > >>> >> b) It's unclear to me where the output data goes when you
> > create a
> > > > >>> policy.
> > > > >>> >> E.g. say I have:
> > > > >>> >>
> > > > >>> >> from HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX[str:contains(
> > src,'/hb
> > > > >>> ase')]
> > > > >>> >> select * group by user insert into hdfs_audit_log_enriched_
> > > > stream_out
> > > > >>> >>
> > > > >>> >> Where is "hdfs_audit_log_enriched_stream_out" defined (is it
> a
> > > > Kafka
> > > > >>> >> topic?). How can I check the output to make sure the policy is
> > > > working
> > > > >>> >> correctly?
> > > > >>> >>
> > > > >>> >> Thanks,
> > > > >>> >>
> > > > >>> >> Colm.
> > > > >>> >>
> > > > >>> >> On Wed, Jan 17, 2018 at 10:32 PM, Edward Zhang <
> > > > >>> yonzhang2012@gmail.com>
> > > > >>> >> wrote:
> > > > >>> >>
> > > > >>> >> > There is a data preparation stage between data source(HDFS
> > audit
> > > > >>> log)
> > > > >>> >> and
> > > > >>> >> > Alert Engine. This stage is running in Storm and transform
> the
> > > raw
> > > > >>> HDFS
> > > > >>> >> log
> > > > >>> >> > into something which can be alerted.
> > > > >>> >> >
> > > > >>> >> > The input for data preparation is hdfs_audit_log_sandbox
> topic
> > > and
> > > > >>> >> output
> > > > >>> >> > is
> > > > >>> >> >  hdfs_audit_log_enriched_sandbox.
> > > > >>> >> > The input for Alert Engine is hdfs_audit_log_enriched_
> sandbox
> > > and
> > > > >>> >> output
> > > > >>> >> > is
> > > > >>> >> > hdfs_audit_log_alert_sandbox.
> > > > >>> >> >
> > > > >>> >> > Seems in your case, the data preparation staging is not
> > working.
> > > > We
> > > > >>> >> > probably need look at Storm console and figure out if that
> > part
> > > is
> > > > >>> >> working.
> > > > >>> >> >
> > > > >>> >> > Thanks
> > > > >>> >> > Edward
> > > > >>> >> >
> > > > >>> >> > On Wed, Jan 17, 2018 at 7:19 AM, Colm O hEigeartaigh <
> > > > >>> >> coheigea@apache.org>
> > > > >>> >> > wrote:
> > > > >>> >> >
> > > > >>> >> > > Hi Jayesh,
> > > > >>> >> > >
> > > > >>> >> > > Many thanks for your feedback! I was able to make a little
> > > > further
> > > > >>> >> > headway.
> > > > >>> >> > > There are two configuration problems with the official
> > docker
> > > > >>> image:
> > > > >>> >> > >
> > > > >>> >> > > a) A mix of "sandbox.eagle.apache.org" and "
> > > > >>> server.eagle.apache.org"
> > > > >>> >> > (this
> > > > >>> >> > > only occurs in the instructions for running the docker
> > image.
> > > > The
> > > > >>> >> version
> > > > >>> >> > > that can be started via the script in the eagle source is
> > OK).
> > > > >>> I'll
> > > > >>> >> > submit
> > > > >>> >> > > a PR to fix this once I get a basic use-case working.
> > > > >>> >> > > b) For the audit case, it automatically logs HDFS audit
> logs
> > > to
> > > > >>> the
> > > > >>> >> KAFKA
> > > > >>> >> > > topic sandbox_hdfs_audit_log instead of the expected
> > > > >>> >> > hdfs_audit_log_sandbox
> > > > >>> >> > >
> > > > >>> >> > > I've fixed these things locally and I can verify that
> > > everything
> > > > >>> is
> > > > >>> >> > started
> > > > >>> >> > > correctly in Ambari. I log into the docker container and
> > > create
> > > > >>> >> > > hdfs_audit_log_sandbox and hdfs_audit_log_enriched_
> sandbox
> > > > >>> topics,
> > > > >>> >> and
> > > > >>> >> > > verify that the HDFS audit logs are flowing into the first
> > > > topic.
> > > > >>> >> Then in
> > > > >>> >> > > the UI I start the Alert Engine and then the HDFS Audit
> Log
> > > > >>> Monitor
> > > > >>> >> > > application (changing localhost:6667 to
> > > > >>> server.eagle.apache.org:6667
> > > > >>> >> ).
> > > > >>> >> > > Both
> > > > >>> >> > > applications start up correctly and show "running".
> > > > >>> >> > >
> > > > >>> >> > > I then create a policy with an email alert along the lines
> > of
> > > > from
> > > > >>> >> > > "HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX[str:contains(
> > src,'/h
> > > > >>> base')]
> > > > >>> >> > select
> > > > >>> >> > > * group by user insert into hdfs_audit_log_enriched_
> > > > stream_out".
> > > > >>> >> However
> > > > >>> >> > > at
> > > > >>> >> > > this point I'm stuck - nothing appears in the alert
> window.
> > Is
> > > > >>> there
> > > > >>> >> > > anything obvious I'm doing wrong, or how can I get access
> to
> > > > logs
> > > > >>> to
> > > > >>> >> > figure
> > > > >>> >> > > out what the problem is? Other topics such as
> > > > >>> >> "hdfs_audit_event_sandbox"
> > > > >>> >> > > are mentioned in the streams window, but the documentation
> > > > doesn't
> > > > >>> >> say to
> > > > >>> >> > > create them.
> > > > >>> >> > >
> > > > >>> >> > > The UI is buggy though on both Firefox and Chromium on
> > Linux.
> > > > What
> > > > >>> >> > > browser/platform are people using with the UI?
> > > > >>> >> > >
> > > > >>> >> > > Colm.
> > > > >>> >> > >
> > > > >>> >> > > On Wed, Jan 17, 2018 at 12:27 AM, Jayesh Senjaliya <
> > > > >>> jaysen@apache.org
> > > > >>> >> >
> > > > >>> >> > > wrote:
> > > > >>> >> > >
> > > > >>> >> > > > Hi Colm,
> > > > >>> >> > > >
> > > > >>> >> > > > Please find my comments inline.
> > > > >>> >> > > >
> > > > >>> >> > > > a) The official docker image uses 0.5.0-SNAPSHOT and not
> > the
> > > > >>> >> released
> > > > >>> >> > > > version.
> > > > >>> >> > > > - this is because we uploaded docker image before apache
> > > > >>> release.
> > > > >>> >> > > actually
> > > > >>> >> > > > this is same codebase apache-eagle-0.5, and it can be
> > fixed
> > > > >>> easily
> > > > >>> >> by
> > > > >>> >> > > just
> > > > >>> >> > > > rebuilding docker image. there should not be any
> mismatch
> > > due
> > > > to
> > > > >>> >> this.
> > > > >>> >> > > >
> > > > >>> >> > > > b) Aside from the above, the official docker image uses
> a
> > > mix
> > > > >>> of "
> > > > >>> >> > > > server.eagle.apache.org" and "sandbox.eagle.apache.org"
> > as
> > > > the
> > > > >>> host
> > > > >>> >> > > name.
> > > > >>> >> > > > The HBase service doesn't start by default in Ambari as
> a
> > > > >>> result.
> > > > >>> >> > > > - the only places it uses sandbox is in example script
> > which
> > > > you
> > > > >>> >> will
> > > > >>> >> > > have
> > > > >>> >> > > > to update anyway, which i agree that it would be good to
> > > keep
> > > > it
> > > > >>> >> > > > consistent.
> > > > >>> >> > > >
> > > > >>> >> > > > c) The UI seems quite buggy. On both chromium and
> > firefox, I
> > > > >>> only
> > > > >>> >> see
> > > > >>> >> > > > links to "Sandbox" and "Alert" on the left hand-side.
> > Once I
> > > > >>> click
> > > > >>> >> on
> > > > >>> >> > > > "Alert" I have no way of going back to see the
> > > applications. I
> > > > >>> don't
> > > > >>> >> > see
> > > > >>> >> > > > the links to "integration" or "sites" as in the picture
> > > here:
> > > > >>> >> > > > http://eagle.apache.org/docs/latest/applications/#jmx-
> > > monito
> > > > >>> ring
> > > > >>> >> > > > - when hbase is as deep storage is used, and if eagle
> app
> > > has
> > > > >>> issue
> > > > >>> >> > > > connecting to hbase, the UI becomes unresponsive.
> > > > >>> >> > > >
> > > > >>> >> > > > d) In chromium, the button to create a new policy does
> not
> > > > >>> exist - I
> > > > >>> >> > can
> > > > >>> >> > > > only see it on Firefox.
> > > > >>> >> > > > - i have seen when you logged in, you will see admin
> > > actions.
> > > > >>> but if
> > > > >>> >> > this
> > > > >>> >> > > > still an issue, can you please file UI bug?
> > > > >>> >> > > >
> > > > >>> >> > > > e) I'm trying to get the "Hdfs Audit Log Monitor"
> use-case
> > > > >>> working,
> > > > >>> >> but
> > > > >>> >> > > it
> > > > >>> >> > > > seems to be stuck in "Initialized".
> > > > >>> >> > > > this eagle docs has example on how to setup the app. pls
> > let
> > > > us
> > > > >>> >> know if
> > > > >>> >> > > > you find any gaps.
> > > > >>> >> > > >
> > > > >>> >> > > > Thanks for trying out, and sharing your findings,
> > > > >>> >> > > > Jayesh
> > > > >>> >> > > >
> > > > >>> >> > > >
> > > > >>> >> > > > On Tue, Jan 16, 2018 at 3:34 AM, Colm O hEigeartaigh <
> > > > >>> >> > > coheigea@apache.org>
> > > > >>> >> > > > wrote:
> > > > >>> >> > > >
> > > > >>> >> > > >> Hi all,
> > > > >>> >> > > >>
> > > > >>> >> > > >> I'm trying to play around a bit with Apache Eagle 0.5.0
> > to
> > > no
> > > > >>> >> avail.
> > > > >>> >> > > Here
> > > > >>> >> > > >> are the problems I've run into so far:
> > > > >>> >> > > >>
> > > > >>> >> > > >> a) The official docker image uses 0.5.0-SNAPSHOT and
> not
> > > the
> > > > >>> >> released
> > > > >>> >> > > >> version.
> > > > >>> >> > > >>
> > > > >>> >> > > >> b) Aside from the above, the official docker image
> uses a
> > > mix
> > > > >>> of "
> > > > >>> >> > > >> server.eagle.apache.org" and "sandbox.eagle.apache.org
> "
> > as
> > > > the
> > > > >>> >> host
> > > > >>> >> > > >> name. The HBase service doesn't start by default in
> > Ambari
> > > > as a
> > > > >>> >> > result.
> > > > >>> >> > > >>
> > > > >>> >> > > >> c) The UI seems quite buggy. On both chromium and
> > firefox,
> > > I
> > > > >>> only
> > > > >>> >> see
> > > > >>> >> > > >> links to "Sandbox" and "Alert" on the left hand-side.
> > Once
> > > I
> > > > >>> click
> > > > >>> >> on
> > > > >>> >> > > >> "Alert" I have no way of going back to see the
> > > applications.
> > > > I
> > > > >>> >> don't
> > > > >>> >> > see
> > > > >>> >> > > >> the links to "integration" or "sites" as in the picture
> > > here:
> > > > >>> >> > > >> http://eagle.apache.org/docs/latest/applications/#jmx-
> > > monito
> > > > >>> ring
> > > > >>> >> > > >>
> > > > >>> >> > > >> d) In chromium, the button to create a new policy does
> > not
> > > > >>> exist -
> > > > >>> >> I
> > > > >>> >> > can
> > > > >>> >> > > >> only see it on Firefox.
> > > > >>> >> > > >>
> > > > >>> >> > > >> e) I'm trying to get the "Hdfs Audit Log Monitor"
> > use-case
> > > > >>> working,
> > > > >>> >> > but
> > > > >>> >> > > >> it seems to be stuck in "Initialized".
> > > > >>> >> > > >>
> > > > >>> >> > > >> Could someone fill me in on what the "recommended" way
> is
> > > to
> > > > >>> start
> > > > >>> >> > > Apache
> > > > >>> >> > > >> Eagle so that I can play around with the functionality
> > that
> > > > it
> > > > >>> >> offers?
> > > > >>> >> > > >> Clearly the docker approach is buggy. Also, what
> browser
> > > > >>> should be
> > > > >>> >> > used?
> > > > >>> >> > > >>
> > > > >>> >> > > >> Thanks,
> > > > >>> >> > > >>
> > > > >>> >> > > >> Colm.
> > > > >>> >> > > >>
> > > > >>> >> > > >>
> > > > >>> >> > > >> --
> > > > >>> >> > > >> Colm O hEigeartaigh
> > > > >>> >> > > >>
> > > > >>> >> > > >> Talend Community Coder
> > > > >>> >> > > >> http://coders.talend.com
> > > > >>> >> > > >>
> > > > >>> >> > > >
> > > > >>> >> > > >
> > > > >>> >> > >
> > > > >>> >> > >
> > > > >>> >> > > --
> > > > >>> >> > > Colm O hEigeartaigh
> > > > >>> >> > >
> > > > >>> >> > > Talend Community Coder
> > > > >>> >> > > http://coders.talend.com
> > > > >>> >> > >
> > > > >>> >> >
> > > > >>> >>
> > > > >>> >>
> > > > >>> >>
> > > > >>> >> --
> > > > >>> >> Colm O hEigeartaigh
> > > > >>> >>
> > > > >>> >> Talend Community Coder
> > > > >>> >> http://coders.talend.com
> > > > >>> >>
> > > > >>> >>
> > > > >>> >>
> > > > >>> >
> > > > >>> >
> > > > >>> > --
> > > > >>> > Colm O hEigeartaigh
> > > > >>> >
> > > > >>> > Talend Community Coder
> > > > >>> > http://coders.talend.com
> > > > >>> >
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>> --
> > > > >>> Colm O hEigeartaigh
> > > > >>>
> > > > >>> Talend Community Coder
> > > > >>> http://coders.talend.com
> > > > >>>
> > > > >>
> > > > >>
> > > > >
> > > >
> > > >
> > > > --
> > > > Colm O hEigeartaigh
> > > >
> > > > Talend Community Coder
> > > > http://coders.talend.com
> > > >
> > >
> >
> >
> >
> > --
> > Colm O hEigeartaigh
> >
> > Talend Community Coder
> > http://coders.talend.com
> >
>



-- 
Colm O hEigeartaigh

Talend Community Coder
http://coders.talend.com

Re: Unable to get 0.5.0 release working

Posted by Jayesh Senjaliya <ja...@apache.org>.
Hi Colm,

appreciate your suggestions/ efforts in looking into this project,
putting my comments inline...

a) There is already a JIRA to bump the version here, although the PR does
not apply as it is too old: https://issues.apache.org/jira/browse/EAGLE-1025
.
I can submit a new PR, but should the version be 0.6.0 or 0.5.1?

*since there are still minor issues, i would say, we put up 0.5.1 as next
version. I've updated/rebased the PR (
https://github.com/apache/eagle/pull/936
<https://github.com/apache/eagle/pull/936> )*


b) The issues that are "resolved" for the 0.5.1 release in JIRA are
actually already fixed in 0.5.0, so they should be updated (
https://issues.apache.org/jira/projects/EAGLE/versions/12341128). However,
the following two issues are resolved even though they are not merged to
master?
  https://issues.apache.org/jira/browse/EAGLE-1051 .  - * this was pending
from developer;s response but i think this is reviewed, so I have merged
it.*
  https://issues.apache.org/jira/browse/EAGLE-1068 .  - * this is reopened
now. I dont think this is done yet. Also this is big change.*


Like I said I can submit PRs but I'm not convinced there is any activity on
the project. Where are the rest of the committers?

*let me give you some ocontext on this. so there were lot of development
happened during last releases, and most of applications that were added are
being used in production at multiple enterprise companies, but we are out
of ideas on new apps, so at this point we are only focusing on bug fixes
and tech upgrades until we get some new ideas to brainstorm and add.*

*I think current community's thinking is based on their own industries
use-cases, but there is definitely room for new features and integration
with other monitoring and security components like grafana and rangers.*


*Thanks,*
*Jayesh*



On Tue, Jan 30, 2018 at 8:11 AM, Colm O hEigeartaigh <co...@apache.org>
wrote:

> Hi Jayesh,
>
> Dev suggestions:
>
> a) There is already a JIRA to bump the version here, although the PR does
> not apply as it is too old: https://issues.apache.org/
> jira/browse/EAGLE-1025.
> I can submit a new PR, but should the version be 0.6.0 or 0.5.1?
> b) The issues that are "resolved" for the 0.5.1 release in JIRA are
> actually already fixed in 0.5.0, so they should be updated (
> https://issues.apache.org/jira/projects/EAGLE/versions/12341128). However,
> the following two issues are resolved even though they are not merged to
> master?
>   https://issues.apache.org/jira/browse/EAGLE-1051
>   https://issues.apache.org/jira/browse/EAGLE-1068
>
> Like I said I can submit PRs but I'm not convinced there is any activity on
> the project. Where are the rest of the committers?
>
> Multiple Publisher issue:
>
> If I assign two publishers for one policy, the alert only goes to the first
> policy. In the logs I see:
>
> 2018-01-30T15:52:45.835+0000 o.a.e.a.e.p.d.DefaultDeduplicator [INFO]
> Alert
> event is skipped because it's duplicated: Alert {site=sandbox,
> stream=eagle_output,timestamp=2018-01-30
> 00:00:11,300,data={securityZone=NA, dst=null, sensitivityType=NA,
> src=/apps/hbase/data/archive/data/default/ambarismoketest, allowed=true,
> host=172.22.7.129, cmd=listStatus, user=SOMETHING7.COM,
> timestamp=1517270411300}, policyId=test,
> createdBy=alertBolt3-evaluator_stage1, metaVersion=null}
>
> It looks like this deduplicator is not working properly, as I'm guessing it
> should only be used to de-duplicate events for a single publisher?
>
> Incognito mode: Already tried it but with the same result. Could I ask you
> to try the docker image to see if the UI is working correctly for you
> there?
>
> Colm.
>
> On Mon, Jan 29, 2018 at 6:46 PM, Jayesh Senjaliya <ja...@apache.org>
> wrote:
>
> > Hi Colm,
> >
> > Thanks for the list of dev suggestions, I think we should take care of
> > those. even better if you can provide PR with the changes or at keast can
> > you please create a ticket so we can track it?
> >
> > for other issues.
> >
> > - I dont have any issue with multiple publisher, but if there is any
> error
> > updating the publisher info in storm topology, i might try restarting the
> > topology and see if that works.
> > - for us, chrome works as fine as firefox.  can u try incognito mode?
> just
> > to be sure to have clean cache?
> >
> > Thanks
> > Jayesh
> >
> >
> > On Thu, Jan 25, 2018 at 4:19 AM, Colm O hEigeartaigh <
> coheigea@apache.org>
> > wrote:
> >
> > > Thanks again for your feedback. Jayesh, adding AlertEagleStorePlugin
> did
> > > the trick, I can now see alerts in the UI, thanks! By the way, I can't
> > > configure two Alert Publishers, or else the Alert DeDuplicator bins the
> > > alert. Is this a known issue?
> > >
> > > Could I ask which browser people are using with the UI? There appears
> to
> > be
> > > a  bug with Chromium where it doesn't list the pages under Auth.isAdmin
> > > even though I am logged on as an administrator. It works OK in Firefox.
> > > Even with Firefox though, I only see a limited number of links in the
> > > left-hand column - I can't get back to the "integration" page. Can
> > someone
> > > else confirm this please?
> > >
> > > Could I suggest the devs do some basic house-keeping tasks:
> > >
> > > a) "Release" version 0.5.0 in JIRA (it's still listed as "unreleased").
> > > b) Figure out whether the next version will be 0.5.1 or 0.6.0 and
> update
> > > the versions on Master accordingly with 0.5.1-SNAPSHOT or
> 0.6.0-SNAPSHOT.
> > > There are some issues marked here as resolved for 0.5.1 -
> > > https://issues.apache.org/jira/projects/EAGLE/versions/12341128),
> > however
> > > I
> > > don't see a branch for 0.5.x?
> > >
> > > Colm.
> > >
> > > On Thu, Jan 25, 2018 at 8:16 AM, Jayesh Senjaliya <ja...@apache.org>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > we do use eagle 0.5 in production although we dont use all the
> > available
> > > > hadoop applications.
> > > >
> > > > EAGLE-968 <https://issues.apache.org/jira/browse/EAGLE-968> is a fix
> > for
> > > > email issue we found while our testing. should be merged soon after a
> > > > rebase.
> > > >
> > > > @Colm, did you tried adding storage publisher
> (AlertEagleStorePlugin)?
> > to
> > > > see alerts on UI ?
> > > >
> > > > Thanks
> > > > Jayesh
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Wed, Jan 24, 2018 at 7:08 PM, Edward Zhang <
> yonzhang2012@gmail.com>
> > > > wrote:
> > > >
> > > >> Eagle 0.5 was deployed in production as far as I know, but it may
> not
> > be
> > > >> exact the current version in master branch.
> > > >>
> > > >> Thanks for your investigation, seems there is still some bug in 0.5,
> > but
> > > >> this particular issue seems is due to dependent components version
> > > conflict.
> > > >>
> > > >> @Jayesh is this Jira ready for merge to master?
> https://issues.apache
> > .
> > > >> org/jira/browse/EAGLE-968
> > > >>
> > > >>
> > > >> Thanks
> > > >> Edward
> > > >>
> > > >> On Tue, Jan 23, 2018 at 5:10 AM, Colm O hEigeartaigh <
> > > coheigea@apache.org
> > > >> > wrote:
> > > >>
> > > >>> OK I've made some more progress. I wasn't seeing any email alerts
> due
> > > to
> > > >>> https://issues.apache.org/jira/browse/EAGLE-968. Once I configure
> a
> > > >>> Kafka
> > > >>> alert, I can see the alerts flowing into my topic. It's still not
> > clear
> > > >>> to
> > > >>> me however where the policy "output" is going. I also don't see any
> > > >>> alerts
> > > >>> in the UI window.
> > > >>>
> > > >>> Could I ask what the status of the project is in general? There
> have
> > > been
> > > >>> no commits to master since November, so I'm not sure if there is
> any
> > > >>> point
> > > >>> in submitting Pull Requests for outstanding bugs? Are recent
> versions
> > > of
> > > >>> Apache Eagle used in production?
> > > >>>
> > > >>> Colm.
> > > >>>
> > > >>> On Mon, Jan 22, 2018 at 1:07 PM, Colm O hEigeartaigh <
> > > >>> coheigea@apache.org>
> > > >>> wrote:
> > > >>>
> > > >>> >
> > > >>> > I've done that but I'm not seeing any alerts, which is why I want
> > to
> > > >>> find
> > > >>> > out what the "output" of a policy is and where I can check this.
> > > >>> >
> > > >>> > Colm.
> > > >>> >
> > > >>> > On Mon, Jan 22, 2018 at 1:05 PM, SUDHA JENSLIN <
> sjenslin@gmail.com
> > >
> > > >>> wrote:
> > > >>> >
> > > >>> >> Create and add a publisher to see the output.
> > > >>> >>
> > > >>> >>
> > > >>> >>
> > > >>> >> Regards,
> > > >>> >> Sudha jenslin
> > > >>> >>
> > > >>> >> On Jan 22, 2018 6:31 PM, "Colm O hEigeartaigh" <
> > coheigea@apache.org
> > > >
> > > >>> >> wrote:
> > > >>> >>
> > > >>> >> Thanks - the error was due to a problem running Storm with Java
> > 1.8.
> > > >>> I've
> > > >>> >> abandoned the docker image for now, and I'm trying to get it
> > working
> > > >>> >> locally.
> > > >>> >>
> > > >>> >> There are two things I'm not clear on currently, if someone
> could
> > > >>> fill me
> > > >>> >> in:
> > > >>> >>
> > > >>> >> a) For the  'Hdfs Audit Log Monitor' application, the Kafka
> > Consumer
> > > >>> Topic
> > > >>> >> is 'hdfs_audit_log_sandbox'. Under 'Kafka Topic for Auditlog
> Event
> > > >>> Sink'
> > > >>> >> it
> > > >>> >> also specifies 'hdfs_audit_event_sandbox'. However the
> > documentation
> > > >>> for
> > > >>> >> the application mentions 'hdfs_audit_log_enriched_sandbox'?
> > > >>> >>
> > > >>> >> When I click on "STREAMS", the "HDFS_AUDIT_LOG_ENRICHED_STREA
> > > >>> M_SANDBOX"
> > > >>> >> uses the topic "hdfs_audit_event_sandbox". And indeed when I run
> > the
> > > >>> >> application, I can see cleansed log data appearing in
> > > >>> >> "hdfs_audit_event_sandbox". So I'm thinking here that
> > > >>> >> 'hdfs_audit_log_enriched_sandbox' is not correct or necessary?
> > > >>> >>
> > > >>> >> b) It's unclear to me where the output data goes when you
> create a
> > > >>> policy.
> > > >>> >> E.g. say I have:
> > > >>> >>
> > > >>> >> from HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX[str:contains(
> src,'/hb
> > > >>> ase')]
> > > >>> >> select * group by user insert into hdfs_audit_log_enriched_
> > > stream_out
> > > >>> >>
> > > >>> >> Where is "hdfs_audit_log_enriched_stream_out" defined (is it a
> > > Kafka
> > > >>> >> topic?). How can I check the output to make sure the policy is
> > > working
> > > >>> >> correctly?
> > > >>> >>
> > > >>> >> Thanks,
> > > >>> >>
> > > >>> >> Colm.
> > > >>> >>
> > > >>> >> On Wed, Jan 17, 2018 at 10:32 PM, Edward Zhang <
> > > >>> yonzhang2012@gmail.com>
> > > >>> >> wrote:
> > > >>> >>
> > > >>> >> > There is a data preparation stage between data source(HDFS
> audit
> > > >>> log)
> > > >>> >> and
> > > >>> >> > Alert Engine. This stage is running in Storm and transform the
> > raw
> > > >>> HDFS
> > > >>> >> log
> > > >>> >> > into something which can be alerted.
> > > >>> >> >
> > > >>> >> > The input for data preparation is hdfs_audit_log_sandbox topic
> > and
> > > >>> >> output
> > > >>> >> > is
> > > >>> >> >  hdfs_audit_log_enriched_sandbox.
> > > >>> >> > The input for Alert Engine is hdfs_audit_log_enriched_sandbox
> > and
> > > >>> >> output
> > > >>> >> > is
> > > >>> >> > hdfs_audit_log_alert_sandbox.
> > > >>> >> >
> > > >>> >> > Seems in your case, the data preparation staging is not
> working.
> > > We
> > > >>> >> > probably need look at Storm console and figure out if that
> part
> > is
> > > >>> >> working.
> > > >>> >> >
> > > >>> >> > Thanks
> > > >>> >> > Edward
> > > >>> >> >
> > > >>> >> > On Wed, Jan 17, 2018 at 7:19 AM, Colm O hEigeartaigh <
> > > >>> >> coheigea@apache.org>
> > > >>> >> > wrote:
> > > >>> >> >
> > > >>> >> > > Hi Jayesh,
> > > >>> >> > >
> > > >>> >> > > Many thanks for your feedback! I was able to make a little
> > > further
> > > >>> >> > headway.
> > > >>> >> > > There are two configuration problems with the official
> docker
> > > >>> image:
> > > >>> >> > >
> > > >>> >> > > a) A mix of "sandbox.eagle.apache.org" and "
> > > >>> server.eagle.apache.org"
> > > >>> >> > (this
> > > >>> >> > > only occurs in the instructions for running the docker
> image.
> > > The
> > > >>> >> version
> > > >>> >> > > that can be started via the script in the eagle source is
> OK).
> > > >>> I'll
> > > >>> >> > submit
> > > >>> >> > > a PR to fix this once I get a basic use-case working.
> > > >>> >> > > b) For the audit case, it automatically logs HDFS audit logs
> > to
> > > >>> the
> > > >>> >> KAFKA
> > > >>> >> > > topic sandbox_hdfs_audit_log instead of the expected
> > > >>> >> > hdfs_audit_log_sandbox
> > > >>> >> > >
> > > >>> >> > > I've fixed these things locally and I can verify that
> > everything
> > > >>> is
> > > >>> >> > started
> > > >>> >> > > correctly in Ambari. I log into the docker container and
> > create
> > > >>> >> > > hdfs_audit_log_sandbox and hdfs_audit_log_enriched_sandbox
> > > >>> topics,
> > > >>> >> and
> > > >>> >> > > verify that the HDFS audit logs are flowing into the first
> > > topic.
> > > >>> >> Then in
> > > >>> >> > > the UI I start the Alert Engine and then the HDFS Audit Log
> > > >>> Monitor
> > > >>> >> > > application (changing localhost:6667 to
> > > >>> server.eagle.apache.org:6667
> > > >>> >> ).
> > > >>> >> > > Both
> > > >>> >> > > applications start up correctly and show "running".
> > > >>> >> > >
> > > >>> >> > > I then create a policy with an email alert along the lines
> of
> > > from
> > > >>> >> > > "HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX[str:contains(
> src,'/h
> > > >>> base')]
> > > >>> >> > select
> > > >>> >> > > * group by user insert into hdfs_audit_log_enriched_
> > > stream_out".
> > > >>> >> However
> > > >>> >> > > at
> > > >>> >> > > this point I'm stuck - nothing appears in the alert window.
> Is
> > > >>> there
> > > >>> >> > > anything obvious I'm doing wrong, or how can I get access to
> > > logs
> > > >>> to
> > > >>> >> > figure
> > > >>> >> > > out what the problem is? Other topics such as
> > > >>> >> "hdfs_audit_event_sandbox"
> > > >>> >> > > are mentioned in the streams window, but the documentation
> > > doesn't
> > > >>> >> say to
> > > >>> >> > > create them.
> > > >>> >> > >
> > > >>> >> > > The UI is buggy though on both Firefox and Chromium on
> Linux.
> > > What
> > > >>> >> > > browser/platform are people using with the UI?
> > > >>> >> > >
> > > >>> >> > > Colm.
> > > >>> >> > >
> > > >>> >> > > On Wed, Jan 17, 2018 at 12:27 AM, Jayesh Senjaliya <
> > > >>> jaysen@apache.org
> > > >>> >> >
> > > >>> >> > > wrote:
> > > >>> >> > >
> > > >>> >> > > > Hi Colm,
> > > >>> >> > > >
> > > >>> >> > > > Please find my comments inline.
> > > >>> >> > > >
> > > >>> >> > > > a) The official docker image uses 0.5.0-SNAPSHOT and not
> the
> > > >>> >> released
> > > >>> >> > > > version.
> > > >>> >> > > > - this is because we uploaded docker image before apache
> > > >>> release.
> > > >>> >> > > actually
> > > >>> >> > > > this is same codebase apache-eagle-0.5, and it can be
> fixed
> > > >>> easily
> > > >>> >> by
> > > >>> >> > > just
> > > >>> >> > > > rebuilding docker image. there should not be any mismatch
> > due
> > > to
> > > >>> >> this.
> > > >>> >> > > >
> > > >>> >> > > > b) Aside from the above, the official docker image uses a
> > mix
> > > >>> of "
> > > >>> >> > > > server.eagle.apache.org" and "sandbox.eagle.apache.org"
> as
> > > the
> > > >>> host
> > > >>> >> > > name.
> > > >>> >> > > > The HBase service doesn't start by default in Ambari as a
> > > >>> result.
> > > >>> >> > > > - the only places it uses sandbox is in example script
> which
> > > you
> > > >>> >> will
> > > >>> >> > > have
> > > >>> >> > > > to update anyway, which i agree that it would be good to
> > keep
> > > it
> > > >>> >> > > > consistent.
> > > >>> >> > > >
> > > >>> >> > > > c) The UI seems quite buggy. On both chromium and
> firefox, I
> > > >>> only
> > > >>> >> see
> > > >>> >> > > > links to "Sandbox" and "Alert" on the left hand-side.
> Once I
> > > >>> click
> > > >>> >> on
> > > >>> >> > > > "Alert" I have no way of going back to see the
> > applications. I
> > > >>> don't
> > > >>> >> > see
> > > >>> >> > > > the links to "integration" or "sites" as in the picture
> > here:
> > > >>> >> > > > http://eagle.apache.org/docs/latest/applications/#jmx-
> > monito
> > > >>> ring
> > > >>> >> > > > - when hbase is as deep storage is used, and if eagle app
> > has
> > > >>> issue
> > > >>> >> > > > connecting to hbase, the UI becomes unresponsive.
> > > >>> >> > > >
> > > >>> >> > > > d) In chromium, the button to create a new policy does not
> > > >>> exist - I
> > > >>> >> > can
> > > >>> >> > > > only see it on Firefox.
> > > >>> >> > > > - i have seen when you logged in, you will see admin
> > actions.
> > > >>> but if
> > > >>> >> > this
> > > >>> >> > > > still an issue, can you please file UI bug?
> > > >>> >> > > >
> > > >>> >> > > > e) I'm trying to get the "Hdfs Audit Log Monitor" use-case
> > > >>> working,
> > > >>> >> but
> > > >>> >> > > it
> > > >>> >> > > > seems to be stuck in "Initialized".
> > > >>> >> > > > this eagle docs has example on how to setup the app. pls
> let
> > > us
> > > >>> >> know if
> > > >>> >> > > > you find any gaps.
> > > >>> >> > > >
> > > >>> >> > > > Thanks for trying out, and sharing your findings,
> > > >>> >> > > > Jayesh
> > > >>> >> > > >
> > > >>> >> > > >
> > > >>> >> > > > On Tue, Jan 16, 2018 at 3:34 AM, Colm O hEigeartaigh <
> > > >>> >> > > coheigea@apache.org>
> > > >>> >> > > > wrote:
> > > >>> >> > > >
> > > >>> >> > > >> Hi all,
> > > >>> >> > > >>
> > > >>> >> > > >> I'm trying to play around a bit with Apache Eagle 0.5.0
> to
> > no
> > > >>> >> avail.
> > > >>> >> > > Here
> > > >>> >> > > >> are the problems I've run into so far:
> > > >>> >> > > >>
> > > >>> >> > > >> a) The official docker image uses 0.5.0-SNAPSHOT and not
> > the
> > > >>> >> released
> > > >>> >> > > >> version.
> > > >>> >> > > >>
> > > >>> >> > > >> b) Aside from the above, the official docker image uses a
> > mix
> > > >>> of "
> > > >>> >> > > >> server.eagle.apache.org" and "sandbox.eagle.apache.org"
> as
> > > the
> > > >>> >> host
> > > >>> >> > > >> name. The HBase service doesn't start by default in
> Ambari
> > > as a
> > > >>> >> > result.
> > > >>> >> > > >>
> > > >>> >> > > >> c) The UI seems quite buggy. On both chromium and
> firefox,
> > I
> > > >>> only
> > > >>> >> see
> > > >>> >> > > >> links to "Sandbox" and "Alert" on the left hand-side.
> Once
> > I
> > > >>> click
> > > >>> >> on
> > > >>> >> > > >> "Alert" I have no way of going back to see the
> > applications.
> > > I
> > > >>> >> don't
> > > >>> >> > see
> > > >>> >> > > >> the links to "integration" or "sites" as in the picture
> > here:
> > > >>> >> > > >> http://eagle.apache.org/docs/latest/applications/#jmx-
> > monito
> > > >>> ring
> > > >>> >> > > >>
> > > >>> >> > > >> d) In chromium, the button to create a new policy does
> not
> > > >>> exist -
> > > >>> >> I
> > > >>> >> > can
> > > >>> >> > > >> only see it on Firefox.
> > > >>> >> > > >>
> > > >>> >> > > >> e) I'm trying to get the "Hdfs Audit Log Monitor"
> use-case
> > > >>> working,
> > > >>> >> > but
> > > >>> >> > > >> it seems to be stuck in "Initialized".
> > > >>> >> > > >>
> > > >>> >> > > >> Could someone fill me in on what the "recommended" way is
> > to
> > > >>> start
> > > >>> >> > > Apache
> > > >>> >> > > >> Eagle so that I can play around with the functionality
> that
> > > it
> > > >>> >> offers?
> > > >>> >> > > >> Clearly the docker approach is buggy. Also, what browser
> > > >>> should be
> > > >>> >> > used?
> > > >>> >> > > >>
> > > >>> >> > > >> Thanks,
> > > >>> >> > > >>
> > > >>> >> > > >> Colm.
> > > >>> >> > > >>
> > > >>> >> > > >>
> > > >>> >> > > >> --
> > > >>> >> > > >> Colm O hEigeartaigh
> > > >>> >> > > >>
> > > >>> >> > > >> Talend Community Coder
> > > >>> >> > > >> http://coders.talend.com
> > > >>> >> > > >>
> > > >>> >> > > >
> > > >>> >> > > >
> > > >>> >> > >
> > > >>> >> > >
> > > >>> >> > > --
> > > >>> >> > > Colm O hEigeartaigh
> > > >>> >> > >
> > > >>> >> > > Talend Community Coder
> > > >>> >> > > http://coders.talend.com
> > > >>> >> > >
> > > >>> >> >
> > > >>> >>
> > > >>> >>
> > > >>> >>
> > > >>> >> --
> > > >>> >> Colm O hEigeartaigh
> > > >>> >>
> > > >>> >> Talend Community Coder
> > > >>> >> http://coders.talend.com
> > > >>> >>
> > > >>> >>
> > > >>> >>
> > > >>> >
> > > >>> >
> > > >>> > --
> > > >>> > Colm O hEigeartaigh
> > > >>> >
> > > >>> > Talend Community Coder
> > > >>> > http://coders.talend.com
> > > >>> >
> > > >>>
> > > >>>
> > > >>>
> > > >>> --
> > > >>> Colm O hEigeartaigh
> > > >>>
> > > >>> Talend Community Coder
> > > >>> http://coders.talend.com
> > > >>>
> > > >>
> > > >>
> > > >
> > >
> > >
> > > --
> > > Colm O hEigeartaigh
> > >
> > > Talend Community Coder
> > > http://coders.talend.com
> > >
> >
>
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>

Re: Unable to get 0.5.0 release working

Posted by Colm O hEigeartaigh <co...@apache.org>.
Hi Jayesh,

Dev suggestions:

a) There is already a JIRA to bump the version here, although the PR does
not apply as it is too old: https://issues.apache.org/jira/browse/EAGLE-1025.
I can submit a new PR, but should the version be 0.6.0 or 0.5.1?
b) The issues that are "resolved" for the 0.5.1 release in JIRA are
actually already fixed in 0.5.0, so they should be updated (
https://issues.apache.org/jira/projects/EAGLE/versions/12341128). However,
the following two issues are resolved even though they are not merged to
master?
  https://issues.apache.org/jira/browse/EAGLE-1051
  https://issues.apache.org/jira/browse/EAGLE-1068

Like I said I can submit PRs but I'm not convinced there is any activity on
the project. Where are the rest of the committers?

Multiple Publisher issue:

If I assign two publishers for one policy, the alert only goes to the first
policy. In the logs I see:

2018-01-30T15:52:45.835+0000 o.a.e.a.e.p.d.DefaultDeduplicator [INFO] Alert
event is skipped because it's duplicated: Alert {site=sandbox,
stream=eagle_output,timestamp=2018-01-30
00:00:11,300,data={securityZone=NA, dst=null, sensitivityType=NA,
src=/apps/hbase/data/archive/data/default/ambarismoketest, allowed=true,
host=172.22.7.129, cmd=listStatus, user=SOMETHING7.COM,
timestamp=1517270411300}, policyId=test,
createdBy=alertBolt3-evaluator_stage1, metaVersion=null}

It looks like this deduplicator is not working properly, as I'm guessing it
should only be used to de-duplicate events for a single publisher?

Incognito mode: Already tried it but with the same result. Could I ask you
to try the docker image to see if the UI is working correctly for you there?

Colm.

On Mon, Jan 29, 2018 at 6:46 PM, Jayesh Senjaliya <ja...@apache.org> wrote:

> Hi Colm,
>
> Thanks for the list of dev suggestions, I think we should take care of
> those. even better if you can provide PR with the changes or at keast can
> you please create a ticket so we can track it?
>
> for other issues.
>
> - I dont have any issue with multiple publisher, but if there is any error
> updating the publisher info in storm topology, i might try restarting the
> topology and see if that works.
> - for us, chrome works as fine as firefox.  can u try incognito mode? just
> to be sure to have clean cache?
>
> Thanks
> Jayesh
>
>
> On Thu, Jan 25, 2018 at 4:19 AM, Colm O hEigeartaigh <co...@apache.org>
> wrote:
>
> > Thanks again for your feedback. Jayesh, adding AlertEagleStorePlugin did
> > the trick, I can now see alerts in the UI, thanks! By the way, I can't
> > configure two Alert Publishers, or else the Alert DeDuplicator bins the
> > alert. Is this a known issue?
> >
> > Could I ask which browser people are using with the UI? There appears to
> be
> > a  bug with Chromium where it doesn't list the pages under Auth.isAdmin
> > even though I am logged on as an administrator. It works OK in Firefox.
> > Even with Firefox though, I only see a limited number of links in the
> > left-hand column - I can't get back to the "integration" page. Can
> someone
> > else confirm this please?
> >
> > Could I suggest the devs do some basic house-keeping tasks:
> >
> > a) "Release" version 0.5.0 in JIRA (it's still listed as "unreleased").
> > b) Figure out whether the next version will be 0.5.1 or 0.6.0 and update
> > the versions on Master accordingly with 0.5.1-SNAPSHOT or 0.6.0-SNAPSHOT.
> > There are some issues marked here as resolved for 0.5.1 -
> > https://issues.apache.org/jira/projects/EAGLE/versions/12341128),
> however
> > I
> > don't see a branch for 0.5.x?
> >
> > Colm.
> >
> > On Thu, Jan 25, 2018 at 8:16 AM, Jayesh Senjaliya <ja...@apache.org>
> > wrote:
> >
> > > Hi,
> > >
> > > we do use eagle 0.5 in production although we dont use all the
> available
> > > hadoop applications.
> > >
> > > EAGLE-968 <https://issues.apache.org/jira/browse/EAGLE-968> is a fix
> for
> > > email issue we found while our testing. should be merged soon after a
> > > rebase.
> > >
> > > @Colm, did you tried adding storage publisher (AlertEagleStorePlugin)?
> to
> > > see alerts on UI ?
> > >
> > > Thanks
> > > Jayesh
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Wed, Jan 24, 2018 at 7:08 PM, Edward Zhang <yo...@gmail.com>
> > > wrote:
> > >
> > >> Eagle 0.5 was deployed in production as far as I know, but it may not
> be
> > >> exact the current version in master branch.
> > >>
> > >> Thanks for your investigation, seems there is still some bug in 0.5,
> but
> > >> this particular issue seems is due to dependent components version
> > conflict.
> > >>
> > >> @Jayesh is this Jira ready for merge to master? https://issues.apache
> .
> > >> org/jira/browse/EAGLE-968
> > >>
> > >>
> > >> Thanks
> > >> Edward
> > >>
> > >> On Tue, Jan 23, 2018 at 5:10 AM, Colm O hEigeartaigh <
> > coheigea@apache.org
> > >> > wrote:
> > >>
> > >>> OK I've made some more progress. I wasn't seeing any email alerts due
> > to
> > >>> https://issues.apache.org/jira/browse/EAGLE-968. Once I configure a
> > >>> Kafka
> > >>> alert, I can see the alerts flowing into my topic. It's still not
> clear
> > >>> to
> > >>> me however where the policy "output" is going. I also don't see any
> > >>> alerts
> > >>> in the UI window.
> > >>>
> > >>> Could I ask what the status of the project is in general? There have
> > been
> > >>> no commits to master since November, so I'm not sure if there is any
> > >>> point
> > >>> in submitting Pull Requests for outstanding bugs? Are recent versions
> > of
> > >>> Apache Eagle used in production?
> > >>>
> > >>> Colm.
> > >>>
> > >>> On Mon, Jan 22, 2018 at 1:07 PM, Colm O hEigeartaigh <
> > >>> coheigea@apache.org>
> > >>> wrote:
> > >>>
> > >>> >
> > >>> > I've done that but I'm not seeing any alerts, which is why I want
> to
> > >>> find
> > >>> > out what the "output" of a policy is and where I can check this.
> > >>> >
> > >>> > Colm.
> > >>> >
> > >>> > On Mon, Jan 22, 2018 at 1:05 PM, SUDHA JENSLIN <sjenslin@gmail.com
> >
> > >>> wrote:
> > >>> >
> > >>> >> Create and add a publisher to see the output.
> > >>> >>
> > >>> >>
> > >>> >>
> > >>> >> Regards,
> > >>> >> Sudha jenslin
> > >>> >>
> > >>> >> On Jan 22, 2018 6:31 PM, "Colm O hEigeartaigh" <
> coheigea@apache.org
> > >
> > >>> >> wrote:
> > >>> >>
> > >>> >> Thanks - the error was due to a problem running Storm with Java
> 1.8.
> > >>> I've
> > >>> >> abandoned the docker image for now, and I'm trying to get it
> working
> > >>> >> locally.
> > >>> >>
> > >>> >> There are two things I'm not clear on currently, if someone could
> > >>> fill me
> > >>> >> in:
> > >>> >>
> > >>> >> a) For the  'Hdfs Audit Log Monitor' application, the Kafka
> Consumer
> > >>> Topic
> > >>> >> is 'hdfs_audit_log_sandbox'. Under 'Kafka Topic for Auditlog Event
> > >>> Sink'
> > >>> >> it
> > >>> >> also specifies 'hdfs_audit_event_sandbox'. However the
> documentation
> > >>> for
> > >>> >> the application mentions 'hdfs_audit_log_enriched_sandbox'?
> > >>> >>
> > >>> >> When I click on "STREAMS", the "HDFS_AUDIT_LOG_ENRICHED_STREA
> > >>> M_SANDBOX"
> > >>> >> uses the topic "hdfs_audit_event_sandbox". And indeed when I run
> the
> > >>> >> application, I can see cleansed log data appearing in
> > >>> >> "hdfs_audit_event_sandbox". So I'm thinking here that
> > >>> >> 'hdfs_audit_log_enriched_sandbox' is not correct or necessary?
> > >>> >>
> > >>> >> b) It's unclear to me where the output data goes when you create a
> > >>> policy.
> > >>> >> E.g. say I have:
> > >>> >>
> > >>> >> from HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX[str:contains(src,'/hb
> > >>> ase')]
> > >>> >> select * group by user insert into hdfs_audit_log_enriched_
> > stream_out
> > >>> >>
> > >>> >> Where is "hdfs_audit_log_enriched_stream_out" defined (is it a
> > Kafka
> > >>> >> topic?). How can I check the output to make sure the policy is
> > working
> > >>> >> correctly?
> > >>> >>
> > >>> >> Thanks,
> > >>> >>
> > >>> >> Colm.
> > >>> >>
> > >>> >> On Wed, Jan 17, 2018 at 10:32 PM, Edward Zhang <
> > >>> yonzhang2012@gmail.com>
> > >>> >> wrote:
> > >>> >>
> > >>> >> > There is a data preparation stage between data source(HDFS audit
> > >>> log)
> > >>> >> and
> > >>> >> > Alert Engine. This stage is running in Storm and transform the
> raw
> > >>> HDFS
> > >>> >> log
> > >>> >> > into something which can be alerted.
> > >>> >> >
> > >>> >> > The input for data preparation is hdfs_audit_log_sandbox topic
> and
> > >>> >> output
> > >>> >> > is
> > >>> >> >  hdfs_audit_log_enriched_sandbox.
> > >>> >> > The input for Alert Engine is hdfs_audit_log_enriched_sandbox
> and
> > >>> >> output
> > >>> >> > is
> > >>> >> > hdfs_audit_log_alert_sandbox.
> > >>> >> >
> > >>> >> > Seems in your case, the data preparation staging is not working.
> > We
> > >>> >> > probably need look at Storm console and figure out if that part
> is
> > >>> >> working.
> > >>> >> >
> > >>> >> > Thanks
> > >>> >> > Edward
> > >>> >> >
> > >>> >> > On Wed, Jan 17, 2018 at 7:19 AM, Colm O hEigeartaigh <
> > >>> >> coheigea@apache.org>
> > >>> >> > wrote:
> > >>> >> >
> > >>> >> > > Hi Jayesh,
> > >>> >> > >
> > >>> >> > > Many thanks for your feedback! I was able to make a little
> > further
> > >>> >> > headway.
> > >>> >> > > There are two configuration problems with the official docker
> > >>> image:
> > >>> >> > >
> > >>> >> > > a) A mix of "sandbox.eagle.apache.org" and "
> > >>> server.eagle.apache.org"
> > >>> >> > (this
> > >>> >> > > only occurs in the instructions for running the docker image.
> > The
> > >>> >> version
> > >>> >> > > that can be started via the script in the eagle source is OK).
> > >>> I'll
> > >>> >> > submit
> > >>> >> > > a PR to fix this once I get a basic use-case working.
> > >>> >> > > b) For the audit case, it automatically logs HDFS audit logs
> to
> > >>> the
> > >>> >> KAFKA
> > >>> >> > > topic sandbox_hdfs_audit_log instead of the expected
> > >>> >> > hdfs_audit_log_sandbox
> > >>> >> > >
> > >>> >> > > I've fixed these things locally and I can verify that
> everything
> > >>> is
> > >>> >> > started
> > >>> >> > > correctly in Ambari. I log into the docker container and
> create
> > >>> >> > > hdfs_audit_log_sandbox and hdfs_audit_log_enriched_sandbox
> > >>> topics,
> > >>> >> and
> > >>> >> > > verify that the HDFS audit logs are flowing into the first
> > topic.
> > >>> >> Then in
> > >>> >> > > the UI I start the Alert Engine and then the HDFS Audit Log
> > >>> Monitor
> > >>> >> > > application (changing localhost:6667 to
> > >>> server.eagle.apache.org:6667
> > >>> >> ).
> > >>> >> > > Both
> > >>> >> > > applications start up correctly and show "running".
> > >>> >> > >
> > >>> >> > > I then create a policy with an email alert along the lines of
> > from
> > >>> >> > > "HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX[str:contains(src,'/h
> > >>> base')]
> > >>> >> > select
> > >>> >> > > * group by user insert into hdfs_audit_log_enriched_
> > stream_out".
> > >>> >> However
> > >>> >> > > at
> > >>> >> > > this point I'm stuck - nothing appears in the alert window. Is
> > >>> there
> > >>> >> > > anything obvious I'm doing wrong, or how can I get access to
> > logs
> > >>> to
> > >>> >> > figure
> > >>> >> > > out what the problem is? Other topics such as
> > >>> >> "hdfs_audit_event_sandbox"
> > >>> >> > > are mentioned in the streams window, but the documentation
> > doesn't
> > >>> >> say to
> > >>> >> > > create them.
> > >>> >> > >
> > >>> >> > > The UI is buggy though on both Firefox and Chromium on Linux.
> > What
> > >>> >> > > browser/platform are people using with the UI?
> > >>> >> > >
> > >>> >> > > Colm.
> > >>> >> > >
> > >>> >> > > On Wed, Jan 17, 2018 at 12:27 AM, Jayesh Senjaliya <
> > >>> jaysen@apache.org
> > >>> >> >
> > >>> >> > > wrote:
> > >>> >> > >
> > >>> >> > > > Hi Colm,
> > >>> >> > > >
> > >>> >> > > > Please find my comments inline.
> > >>> >> > > >
> > >>> >> > > > a) The official docker image uses 0.5.0-SNAPSHOT and not the
> > >>> >> released
> > >>> >> > > > version.
> > >>> >> > > > - this is because we uploaded docker image before apache
> > >>> release.
> > >>> >> > > actually
> > >>> >> > > > this is same codebase apache-eagle-0.5, and it can be fixed
> > >>> easily
> > >>> >> by
> > >>> >> > > just
> > >>> >> > > > rebuilding docker image. there should not be any mismatch
> due
> > to
> > >>> >> this.
> > >>> >> > > >
> > >>> >> > > > b) Aside from the above, the official docker image uses a
> mix
> > >>> of "
> > >>> >> > > > server.eagle.apache.org" and "sandbox.eagle.apache.org" as
> > the
> > >>> host
> > >>> >> > > name.
> > >>> >> > > > The HBase service doesn't start by default in Ambari as a
> > >>> result.
> > >>> >> > > > - the only places it uses sandbox is in example script which
> > you
> > >>> >> will
> > >>> >> > > have
> > >>> >> > > > to update anyway, which i agree that it would be good to
> keep
> > it
> > >>> >> > > > consistent.
> > >>> >> > > >
> > >>> >> > > > c) The UI seems quite buggy. On both chromium and firefox, I
> > >>> only
> > >>> >> see
> > >>> >> > > > links to "Sandbox" and "Alert" on the left hand-side. Once I
> > >>> click
> > >>> >> on
> > >>> >> > > > "Alert" I have no way of going back to see the
> applications. I
> > >>> don't
> > >>> >> > see
> > >>> >> > > > the links to "integration" or "sites" as in the picture
> here:
> > >>> >> > > > http://eagle.apache.org/docs/latest/applications/#jmx-
> monito
> > >>> ring
> > >>> >> > > > - when hbase is as deep storage is used, and if eagle app
> has
> > >>> issue
> > >>> >> > > > connecting to hbase, the UI becomes unresponsive.
> > >>> >> > > >
> > >>> >> > > > d) In chromium, the button to create a new policy does not
> > >>> exist - I
> > >>> >> > can
> > >>> >> > > > only see it on Firefox.
> > >>> >> > > > - i have seen when you logged in, you will see admin
> actions.
> > >>> but if
> > >>> >> > this
> > >>> >> > > > still an issue, can you please file UI bug?
> > >>> >> > > >
> > >>> >> > > > e) I'm trying to get the "Hdfs Audit Log Monitor" use-case
> > >>> working,
> > >>> >> but
> > >>> >> > > it
> > >>> >> > > > seems to be stuck in "Initialized".
> > >>> >> > > > this eagle docs has example on how to setup the app. pls let
> > us
> > >>> >> know if
> > >>> >> > > > you find any gaps.
> > >>> >> > > >
> > >>> >> > > > Thanks for trying out, and sharing your findings,
> > >>> >> > > > Jayesh
> > >>> >> > > >
> > >>> >> > > >
> > >>> >> > > > On Tue, Jan 16, 2018 at 3:34 AM, Colm O hEigeartaigh <
> > >>> >> > > coheigea@apache.org>
> > >>> >> > > > wrote:
> > >>> >> > > >
> > >>> >> > > >> Hi all,
> > >>> >> > > >>
> > >>> >> > > >> I'm trying to play around a bit with Apache Eagle 0.5.0 to
> no
> > >>> >> avail.
> > >>> >> > > Here
> > >>> >> > > >> are the problems I've run into so far:
> > >>> >> > > >>
> > >>> >> > > >> a) The official docker image uses 0.5.0-SNAPSHOT and not
> the
> > >>> >> released
> > >>> >> > > >> version.
> > >>> >> > > >>
> > >>> >> > > >> b) Aside from the above, the official docker image uses a
> mix
> > >>> of "
> > >>> >> > > >> server.eagle.apache.org" and "sandbox.eagle.apache.org" as
> > the
> > >>> >> host
> > >>> >> > > >> name. The HBase service doesn't start by default in Ambari
> > as a
> > >>> >> > result.
> > >>> >> > > >>
> > >>> >> > > >> c) The UI seems quite buggy. On both chromium and firefox,
> I
> > >>> only
> > >>> >> see
> > >>> >> > > >> links to "Sandbox" and "Alert" on the left hand-side. Once
> I
> > >>> click
> > >>> >> on
> > >>> >> > > >> "Alert" I have no way of going back to see the
> applications.
> > I
> > >>> >> don't
> > >>> >> > see
> > >>> >> > > >> the links to "integration" or "sites" as in the picture
> here:
> > >>> >> > > >> http://eagle.apache.org/docs/latest/applications/#jmx-
> monito
> > >>> ring
> > >>> >> > > >>
> > >>> >> > > >> d) In chromium, the button to create a new policy does not
> > >>> exist -
> > >>> >> I
> > >>> >> > can
> > >>> >> > > >> only see it on Firefox.
> > >>> >> > > >>
> > >>> >> > > >> e) I'm trying to get the "Hdfs Audit Log Monitor" use-case
> > >>> working,
> > >>> >> > but
> > >>> >> > > >> it seems to be stuck in "Initialized".
> > >>> >> > > >>
> > >>> >> > > >> Could someone fill me in on what the "recommended" way is
> to
> > >>> start
> > >>> >> > > Apache
> > >>> >> > > >> Eagle so that I can play around with the functionality that
> > it
> > >>> >> offers?
> > >>> >> > > >> Clearly the docker approach is buggy. Also, what browser
> > >>> should be
> > >>> >> > used?
> > >>> >> > > >>
> > >>> >> > > >> Thanks,
> > >>> >> > > >>
> > >>> >> > > >> Colm.
> > >>> >> > > >>
> > >>> >> > > >>
> > >>> >> > > >> --
> > >>> >> > > >> Colm O hEigeartaigh
> > >>> >> > > >>
> > >>> >> > > >> Talend Community Coder
> > >>> >> > > >> http://coders.talend.com
> > >>> >> > > >>
> > >>> >> > > >
> > >>> >> > > >
> > >>> >> > >
> > >>> >> > >
> > >>> >> > > --
> > >>> >> > > Colm O hEigeartaigh
> > >>> >> > >
> > >>> >> > > Talend Community Coder
> > >>> >> > > http://coders.talend.com
> > >>> >> > >
> > >>> >> >
> > >>> >>
> > >>> >>
> > >>> >>
> > >>> >> --
> > >>> >> Colm O hEigeartaigh
> > >>> >>
> > >>> >> Talend Community Coder
> > >>> >> http://coders.talend.com
> > >>> >>
> > >>> >>
> > >>> >>
> > >>> >
> > >>> >
> > >>> > --
> > >>> > Colm O hEigeartaigh
> > >>> >
> > >>> > Talend Community Coder
> > >>> > http://coders.talend.com
> > >>> >
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>> Colm O hEigeartaigh
> > >>>
> > >>> Talend Community Coder
> > >>> http://coders.talend.com
> > >>>
> > >>
> > >>
> > >
> >
> >
> > --
> > Colm O hEigeartaigh
> >
> > Talend Community Coder
> > http://coders.talend.com
> >
>



-- 
Colm O hEigeartaigh

Talend Community Coder
http://coders.talend.com

Re: Unable to get 0.5.0 release working

Posted by Jayesh Senjaliya <ja...@apache.org>.
Hi Colm,

Thanks for the list of dev suggestions, I think we should take care of
those. even better if you can provide PR with the changes or at keast can
you please create a ticket so we can track it?

for other issues.

- I dont have any issue with multiple publisher, but if there is any error
updating the publisher info in storm topology, i might try restarting the
topology and see if that works.
- for us, chrome works as fine as firefox.  can u try incognito mode? just
to be sure to have clean cache?

Thanks
Jayesh


On Thu, Jan 25, 2018 at 4:19 AM, Colm O hEigeartaigh <co...@apache.org>
wrote:

> Thanks again for your feedback. Jayesh, adding AlertEagleStorePlugin did
> the trick, I can now see alerts in the UI, thanks! By the way, I can't
> configure two Alert Publishers, or else the Alert DeDuplicator bins the
> alert. Is this a known issue?
>
> Could I ask which browser people are using with the UI? There appears to be
> a  bug with Chromium where it doesn't list the pages under Auth.isAdmin
> even though I am logged on as an administrator. It works OK in Firefox.
> Even with Firefox though, I only see a limited number of links in the
> left-hand column - I can't get back to the "integration" page. Can someone
> else confirm this please?
>
> Could I suggest the devs do some basic house-keeping tasks:
>
> a) "Release" version 0.5.0 in JIRA (it's still listed as "unreleased").
> b) Figure out whether the next version will be 0.5.1 or 0.6.0 and update
> the versions on Master accordingly with 0.5.1-SNAPSHOT or 0.6.0-SNAPSHOT.
> There are some issues marked here as resolved for 0.5.1 -
> https://issues.apache.org/jira/projects/EAGLE/versions/12341128), however
> I
> don't see a branch for 0.5.x?
>
> Colm.
>
> On Thu, Jan 25, 2018 at 8:16 AM, Jayesh Senjaliya <ja...@apache.org>
> wrote:
>
> > Hi,
> >
> > we do use eagle 0.5 in production although we dont use all the available
> > hadoop applications.
> >
> > EAGLE-968 <https://issues.apache.org/jira/browse/EAGLE-968> is a fix for
> > email issue we found while our testing. should be merged soon after a
> > rebase.
> >
> > @Colm, did you tried adding storage publisher (AlertEagleStorePlugin)? to
> > see alerts on UI ?
> >
> > Thanks
> > Jayesh
> >
> >
> >
> >
> >
> >
> > On Wed, Jan 24, 2018 at 7:08 PM, Edward Zhang <yo...@gmail.com>
> > wrote:
> >
> >> Eagle 0.5 was deployed in production as far as I know, but it may not be
> >> exact the current version in master branch.
> >>
> >> Thanks for your investigation, seems there is still some bug in 0.5, but
> >> this particular issue seems is due to dependent components version
> conflict.
> >>
> >> @Jayesh is this Jira ready for merge to master? https://issues.apache.
> >> org/jira/browse/EAGLE-968
> >>
> >>
> >> Thanks
> >> Edward
> >>
> >> On Tue, Jan 23, 2018 at 5:10 AM, Colm O hEigeartaigh <
> coheigea@apache.org
> >> > wrote:
> >>
> >>> OK I've made some more progress. I wasn't seeing any email alerts due
> to
> >>> https://issues.apache.org/jira/browse/EAGLE-968. Once I configure a
> >>> Kafka
> >>> alert, I can see the alerts flowing into my topic. It's still not clear
> >>> to
> >>> me however where the policy "output" is going. I also don't see any
> >>> alerts
> >>> in the UI window.
> >>>
> >>> Could I ask what the status of the project is in general? There have
> been
> >>> no commits to master since November, so I'm not sure if there is any
> >>> point
> >>> in submitting Pull Requests for outstanding bugs? Are recent versions
> of
> >>> Apache Eagle used in production?
> >>>
> >>> Colm.
> >>>
> >>> On Mon, Jan 22, 2018 at 1:07 PM, Colm O hEigeartaigh <
> >>> coheigea@apache.org>
> >>> wrote:
> >>>
> >>> >
> >>> > I've done that but I'm not seeing any alerts, which is why I want to
> >>> find
> >>> > out what the "output" of a policy is and where I can check this.
> >>> >
> >>> > Colm.
> >>> >
> >>> > On Mon, Jan 22, 2018 at 1:05 PM, SUDHA JENSLIN <sj...@gmail.com>
> >>> wrote:
> >>> >
> >>> >> Create and add a publisher to see the output.
> >>> >>
> >>> >>
> >>> >>
> >>> >> Regards,
> >>> >> Sudha jenslin
> >>> >>
> >>> >> On Jan 22, 2018 6:31 PM, "Colm O hEigeartaigh" <coheigea@apache.org
> >
> >>> >> wrote:
> >>> >>
> >>> >> Thanks - the error was due to a problem running Storm with Java 1.8.
> >>> I've
> >>> >> abandoned the docker image for now, and I'm trying to get it working
> >>> >> locally.
> >>> >>
> >>> >> There are two things I'm not clear on currently, if someone could
> >>> fill me
> >>> >> in:
> >>> >>
> >>> >> a) For the  'Hdfs Audit Log Monitor' application, the Kafka Consumer
> >>> Topic
> >>> >> is 'hdfs_audit_log_sandbox'. Under 'Kafka Topic for Auditlog Event
> >>> Sink'
> >>> >> it
> >>> >> also specifies 'hdfs_audit_event_sandbox'. However the documentation
> >>> for
> >>> >> the application mentions 'hdfs_audit_log_enriched_sandbox'?
> >>> >>
> >>> >> When I click on "STREAMS", the "HDFS_AUDIT_LOG_ENRICHED_STREA
> >>> M_SANDBOX"
> >>> >> uses the topic "hdfs_audit_event_sandbox". And indeed when I run the
> >>> >> application, I can see cleansed log data appearing in
> >>> >> "hdfs_audit_event_sandbox". So I'm thinking here that
> >>> >> 'hdfs_audit_log_enriched_sandbox' is not correct or necessary?
> >>> >>
> >>> >> b) It's unclear to me where the output data goes when you create a
> >>> policy.
> >>> >> E.g. say I have:
> >>> >>
> >>> >> from HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX[str:contains(src,'/hb
> >>> ase')]
> >>> >> select * group by user insert into hdfs_audit_log_enriched_
> stream_out
> >>> >>
> >>> >> Where is "hdfs_audit_log_enriched_stream_out" defined (is it a
> Kafka
> >>> >> topic?). How can I check the output to make sure the policy is
> working
> >>> >> correctly?
> >>> >>
> >>> >> Thanks,
> >>> >>
> >>> >> Colm.
> >>> >>
> >>> >> On Wed, Jan 17, 2018 at 10:32 PM, Edward Zhang <
> >>> yonzhang2012@gmail.com>
> >>> >> wrote:
> >>> >>
> >>> >> > There is a data preparation stage between data source(HDFS audit
> >>> log)
> >>> >> and
> >>> >> > Alert Engine. This stage is running in Storm and transform the raw
> >>> HDFS
> >>> >> log
> >>> >> > into something which can be alerted.
> >>> >> >
> >>> >> > The input for data preparation is hdfs_audit_log_sandbox topic and
> >>> >> output
> >>> >> > is
> >>> >> >  hdfs_audit_log_enriched_sandbox.
> >>> >> > The input for Alert Engine is hdfs_audit_log_enriched_sandbox and
> >>> >> output
> >>> >> > is
> >>> >> > hdfs_audit_log_alert_sandbox.
> >>> >> >
> >>> >> > Seems in your case, the data preparation staging is not working.
> We
> >>> >> > probably need look at Storm console and figure out if that part is
> >>> >> working.
> >>> >> >
> >>> >> > Thanks
> >>> >> > Edward
> >>> >> >
> >>> >> > On Wed, Jan 17, 2018 at 7:19 AM, Colm O hEigeartaigh <
> >>> >> coheigea@apache.org>
> >>> >> > wrote:
> >>> >> >
> >>> >> > > Hi Jayesh,
> >>> >> > >
> >>> >> > > Many thanks for your feedback! I was able to make a little
> further
> >>> >> > headway.
> >>> >> > > There are two configuration problems with the official docker
> >>> image:
> >>> >> > >
> >>> >> > > a) A mix of "sandbox.eagle.apache.org" and "
> >>> server.eagle.apache.org"
> >>> >> > (this
> >>> >> > > only occurs in the instructions for running the docker image.
> The
> >>> >> version
> >>> >> > > that can be started via the script in the eagle source is OK).
> >>> I'll
> >>> >> > submit
> >>> >> > > a PR to fix this once I get a basic use-case working.
> >>> >> > > b) For the audit case, it automatically logs HDFS audit logs to
> >>> the
> >>> >> KAFKA
> >>> >> > > topic sandbox_hdfs_audit_log instead of the expected
> >>> >> > hdfs_audit_log_sandbox
> >>> >> > >
> >>> >> > > I've fixed these things locally and I can verify that everything
> >>> is
> >>> >> > started
> >>> >> > > correctly in Ambari. I log into the docker container and create
> >>> >> > > hdfs_audit_log_sandbox and hdfs_audit_log_enriched_sandbox
> >>> topics,
> >>> >> and
> >>> >> > > verify that the HDFS audit logs are flowing into the first
> topic.
> >>> >> Then in
> >>> >> > > the UI I start the Alert Engine and then the HDFS Audit Log
> >>> Monitor
> >>> >> > > application (changing localhost:6667 to
> >>> server.eagle.apache.org:6667
> >>> >> ).
> >>> >> > > Both
> >>> >> > > applications start up correctly and show "running".
> >>> >> > >
> >>> >> > > I then create a policy with an email alert along the lines of
> from
> >>> >> > > "HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX[str:contains(src,'/h
> >>> base')]
> >>> >> > select
> >>> >> > > * group by user insert into hdfs_audit_log_enriched_
> stream_out".
> >>> >> However
> >>> >> > > at
> >>> >> > > this point I'm stuck - nothing appears in the alert window. Is
> >>> there
> >>> >> > > anything obvious I'm doing wrong, or how can I get access to
> logs
> >>> to
> >>> >> > figure
> >>> >> > > out what the problem is? Other topics such as
> >>> >> "hdfs_audit_event_sandbox"
> >>> >> > > are mentioned in the streams window, but the documentation
> doesn't
> >>> >> say to
> >>> >> > > create them.
> >>> >> > >
> >>> >> > > The UI is buggy though on both Firefox and Chromium on Linux.
> What
> >>> >> > > browser/platform are people using with the UI?
> >>> >> > >
> >>> >> > > Colm.
> >>> >> > >
> >>> >> > > On Wed, Jan 17, 2018 at 12:27 AM, Jayesh Senjaliya <
> >>> jaysen@apache.org
> >>> >> >
> >>> >> > > wrote:
> >>> >> > >
> >>> >> > > > Hi Colm,
> >>> >> > > >
> >>> >> > > > Please find my comments inline.
> >>> >> > > >
> >>> >> > > > a) The official docker image uses 0.5.0-SNAPSHOT and not the
> >>> >> released
> >>> >> > > > version.
> >>> >> > > > - this is because we uploaded docker image before apache
> >>> release.
> >>> >> > > actually
> >>> >> > > > this is same codebase apache-eagle-0.5, and it can be fixed
> >>> easily
> >>> >> by
> >>> >> > > just
> >>> >> > > > rebuilding docker image. there should not be any mismatch due
> to
> >>> >> this.
> >>> >> > > >
> >>> >> > > > b) Aside from the above, the official docker image uses a mix
> >>> of "
> >>> >> > > > server.eagle.apache.org" and "sandbox.eagle.apache.org" as
> the
> >>> host
> >>> >> > > name.
> >>> >> > > > The HBase service doesn't start by default in Ambari as a
> >>> result.
> >>> >> > > > - the only places it uses sandbox is in example script which
> you
> >>> >> will
> >>> >> > > have
> >>> >> > > > to update anyway, which i agree that it would be good to keep
> it
> >>> >> > > > consistent.
> >>> >> > > >
> >>> >> > > > c) The UI seems quite buggy. On both chromium and firefox, I
> >>> only
> >>> >> see
> >>> >> > > > links to "Sandbox" and "Alert" on the left hand-side. Once I
> >>> click
> >>> >> on
> >>> >> > > > "Alert" I have no way of going back to see the applications. I
> >>> don't
> >>> >> > see
> >>> >> > > > the links to "integration" or "sites" as in the picture here:
> >>> >> > > > http://eagle.apache.org/docs/latest/applications/#jmx-monito
> >>> ring
> >>> >> > > > - when hbase is as deep storage is used, and if eagle app has
> >>> issue
> >>> >> > > > connecting to hbase, the UI becomes unresponsive.
> >>> >> > > >
> >>> >> > > > d) In chromium, the button to create a new policy does not
> >>> exist - I
> >>> >> > can
> >>> >> > > > only see it on Firefox.
> >>> >> > > > - i have seen when you logged in, you will see admin actions.
> >>> but if
> >>> >> > this
> >>> >> > > > still an issue, can you please file UI bug?
> >>> >> > > >
> >>> >> > > > e) I'm trying to get the "Hdfs Audit Log Monitor" use-case
> >>> working,
> >>> >> but
> >>> >> > > it
> >>> >> > > > seems to be stuck in "Initialized".
> >>> >> > > > this eagle docs has example on how to setup the app. pls let
> us
> >>> >> know if
> >>> >> > > > you find any gaps.
> >>> >> > > >
> >>> >> > > > Thanks for trying out, and sharing your findings,
> >>> >> > > > Jayesh
> >>> >> > > >
> >>> >> > > >
> >>> >> > > > On Tue, Jan 16, 2018 at 3:34 AM, Colm O hEigeartaigh <
> >>> >> > > coheigea@apache.org>
> >>> >> > > > wrote:
> >>> >> > > >
> >>> >> > > >> Hi all,
> >>> >> > > >>
> >>> >> > > >> I'm trying to play around a bit with Apache Eagle 0.5.0 to no
> >>> >> avail.
> >>> >> > > Here
> >>> >> > > >> are the problems I've run into so far:
> >>> >> > > >>
> >>> >> > > >> a) The official docker image uses 0.5.0-SNAPSHOT and not the
> >>> >> released
> >>> >> > > >> version.
> >>> >> > > >>
> >>> >> > > >> b) Aside from the above, the official docker image uses a mix
> >>> of "
> >>> >> > > >> server.eagle.apache.org" and "sandbox.eagle.apache.org" as
> the
> >>> >> host
> >>> >> > > >> name. The HBase service doesn't start by default in Ambari
> as a
> >>> >> > result.
> >>> >> > > >>
> >>> >> > > >> c) The UI seems quite buggy. On both chromium and firefox, I
> >>> only
> >>> >> see
> >>> >> > > >> links to "Sandbox" and "Alert" on the left hand-side. Once I
> >>> click
> >>> >> on
> >>> >> > > >> "Alert" I have no way of going back to see the applications.
> I
> >>> >> don't
> >>> >> > see
> >>> >> > > >> the links to "integration" or "sites" as in the picture here:
> >>> >> > > >> http://eagle.apache.org/docs/latest/applications/#jmx-monito
> >>> ring
> >>> >> > > >>
> >>> >> > > >> d) In chromium, the button to create a new policy does not
> >>> exist -
> >>> >> I
> >>> >> > can
> >>> >> > > >> only see it on Firefox.
> >>> >> > > >>
> >>> >> > > >> e) I'm trying to get the "Hdfs Audit Log Monitor" use-case
> >>> working,
> >>> >> > but
> >>> >> > > >> it seems to be stuck in "Initialized".
> >>> >> > > >>
> >>> >> > > >> Could someone fill me in on what the "recommended" way is to
> >>> start
> >>> >> > > Apache
> >>> >> > > >> Eagle so that I can play around with the functionality that
> it
> >>> >> offers?
> >>> >> > > >> Clearly the docker approach is buggy. Also, what browser
> >>> should be
> >>> >> > used?
> >>> >> > > >>
> >>> >> > > >> Thanks,
> >>> >> > > >>
> >>> >> > > >> Colm.
> >>> >> > > >>
> >>> >> > > >>
> >>> >> > > >> --
> >>> >> > > >> Colm O hEigeartaigh
> >>> >> > > >>
> >>> >> > > >> Talend Community Coder
> >>> >> > > >> http://coders.talend.com
> >>> >> > > >>
> >>> >> > > >
> >>> >> > > >
> >>> >> > >
> >>> >> > >
> >>> >> > > --
> >>> >> > > Colm O hEigeartaigh
> >>> >> > >
> >>> >> > > Talend Community Coder
> >>> >> > > http://coders.talend.com
> >>> >> > >
> >>> >> >
> >>> >>
> >>> >>
> >>> >>
> >>> >> --
> >>> >> Colm O hEigeartaigh
> >>> >>
> >>> >> Talend Community Coder
> >>> >> http://coders.talend.com
> >>> >>
> >>> >>
> >>> >>
> >>> >
> >>> >
> >>> > --
> >>> > Colm O hEigeartaigh
> >>> >
> >>> > Talend Community Coder
> >>> > http://coders.talend.com
> >>> >
> >>>
> >>>
> >>>
> >>> --
> >>> Colm O hEigeartaigh
> >>>
> >>> Talend Community Coder
> >>> http://coders.talend.com
> >>>
> >>
> >>
> >
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>

Re: Unable to get 0.5.0 release working

Posted by Colm O hEigeartaigh <co...@apache.org>.
Thanks again for your feedback. Jayesh, adding AlertEagleStorePlugin did
the trick, I can now see alerts in the UI, thanks! By the way, I can't
configure two Alert Publishers, or else the Alert DeDuplicator bins the
alert. Is this a known issue?

Could I ask which browser people are using with the UI? There appears to be
a  bug with Chromium where it doesn't list the pages under Auth.isAdmin
even though I am logged on as an administrator. It works OK in Firefox.
Even with Firefox though, I only see a limited number of links in the
left-hand column - I can't get back to the "integration" page. Can someone
else confirm this please?

Could I suggest the devs do some basic house-keeping tasks:

a) "Release" version 0.5.0 in JIRA (it's still listed as "unreleased").
b) Figure out whether the next version will be 0.5.1 or 0.6.0 and update
the versions on Master accordingly with 0.5.1-SNAPSHOT or 0.6.0-SNAPSHOT.
There are some issues marked here as resolved for 0.5.1 -
https://issues.apache.org/jira/projects/EAGLE/versions/12341128), however I
don't see a branch for 0.5.x?

Colm.

On Thu, Jan 25, 2018 at 8:16 AM, Jayesh Senjaliya <ja...@apache.org> wrote:

> Hi,
>
> we do use eagle 0.5 in production although we dont use all the available
> hadoop applications.
>
> EAGLE-968 <https://issues.apache.org/jira/browse/EAGLE-968> is a fix for
> email issue we found while our testing. should be merged soon after a
> rebase.
>
> @Colm, did you tried adding storage publisher (AlertEagleStorePlugin)? to
> see alerts on UI ?
>
> Thanks
> Jayesh
>
>
>
>
>
>
> On Wed, Jan 24, 2018 at 7:08 PM, Edward Zhang <yo...@gmail.com>
> wrote:
>
>> Eagle 0.5 was deployed in production as far as I know, but it may not be
>> exact the current version in master branch.
>>
>> Thanks for your investigation, seems there is still some bug in 0.5, but
>> this particular issue seems is due to dependent components version conflict.
>>
>> @Jayesh is this Jira ready for merge to master? https://issues.apache.
>> org/jira/browse/EAGLE-968
>>
>>
>> Thanks
>> Edward
>>
>> On Tue, Jan 23, 2018 at 5:10 AM, Colm O hEigeartaigh <coheigea@apache.org
>> > wrote:
>>
>>> OK I've made some more progress. I wasn't seeing any email alerts due to
>>> https://issues.apache.org/jira/browse/EAGLE-968. Once I configure a
>>> Kafka
>>> alert, I can see the alerts flowing into my topic. It's still not clear
>>> to
>>> me however where the policy "output" is going. I also don't see any
>>> alerts
>>> in the UI window.
>>>
>>> Could I ask what the status of the project is in general? There have been
>>> no commits to master since November, so I'm not sure if there is any
>>> point
>>> in submitting Pull Requests for outstanding bugs? Are recent versions of
>>> Apache Eagle used in production?
>>>
>>> Colm.
>>>
>>> On Mon, Jan 22, 2018 at 1:07 PM, Colm O hEigeartaigh <
>>> coheigea@apache.org>
>>> wrote:
>>>
>>> >
>>> > I've done that but I'm not seeing any alerts, which is why I want to
>>> find
>>> > out what the "output" of a policy is and where I can check this.
>>> >
>>> > Colm.
>>> >
>>> > On Mon, Jan 22, 2018 at 1:05 PM, SUDHA JENSLIN <sj...@gmail.com>
>>> wrote:
>>> >
>>> >> Create and add a publisher to see the output.
>>> >>
>>> >>
>>> >>
>>> >> Regards,
>>> >> Sudha jenslin
>>> >>
>>> >> On Jan 22, 2018 6:31 PM, "Colm O hEigeartaigh" <co...@apache.org>
>>> >> wrote:
>>> >>
>>> >> Thanks - the error was due to a problem running Storm with Java 1.8.
>>> I've
>>> >> abandoned the docker image for now, and I'm trying to get it working
>>> >> locally.
>>> >>
>>> >> There are two things I'm not clear on currently, if someone could
>>> fill me
>>> >> in:
>>> >>
>>> >> a) For the  'Hdfs Audit Log Monitor' application, the Kafka Consumer
>>> Topic
>>> >> is 'hdfs_audit_log_sandbox'. Under 'Kafka Topic for Auditlog Event
>>> Sink'
>>> >> it
>>> >> also specifies 'hdfs_audit_event_sandbox'. However the documentation
>>> for
>>> >> the application mentions 'hdfs_audit_log_enriched_sandbox'?
>>> >>
>>> >> When I click on "STREAMS", the "HDFS_AUDIT_LOG_ENRICHED_STREA
>>> M_SANDBOX"
>>> >> uses the topic "hdfs_audit_event_sandbox". And indeed when I run the
>>> >> application, I can see cleansed log data appearing in
>>> >> "hdfs_audit_event_sandbox". So I'm thinking here that
>>> >> 'hdfs_audit_log_enriched_sandbox' is not correct or necessary?
>>> >>
>>> >> b) It's unclear to me where the output data goes when you create a
>>> policy.
>>> >> E.g. say I have:
>>> >>
>>> >> from HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX[str:contains(src,'/hb
>>> ase')]
>>> >> select * group by user insert into hdfs_audit_log_enriched_stream_out
>>> >>
>>> >> Where is "hdfs_audit_log_enriched_stream_out" defined (is it a Kafka
>>> >> topic?). How can I check the output to make sure the policy is working
>>> >> correctly?
>>> >>
>>> >> Thanks,
>>> >>
>>> >> Colm.
>>> >>
>>> >> On Wed, Jan 17, 2018 at 10:32 PM, Edward Zhang <
>>> yonzhang2012@gmail.com>
>>> >> wrote:
>>> >>
>>> >> > There is a data preparation stage between data source(HDFS audit
>>> log)
>>> >> and
>>> >> > Alert Engine. This stage is running in Storm and transform the raw
>>> HDFS
>>> >> log
>>> >> > into something which can be alerted.
>>> >> >
>>> >> > The input for data preparation is hdfs_audit_log_sandbox topic and
>>> >> output
>>> >> > is
>>> >> >  hdfs_audit_log_enriched_sandbox.
>>> >> > The input for Alert Engine is hdfs_audit_log_enriched_sandbox and
>>> >> output
>>> >> > is
>>> >> > hdfs_audit_log_alert_sandbox.
>>> >> >
>>> >> > Seems in your case, the data preparation staging is not working. We
>>> >> > probably need look at Storm console and figure out if that part is
>>> >> working.
>>> >> >
>>> >> > Thanks
>>> >> > Edward
>>> >> >
>>> >> > On Wed, Jan 17, 2018 at 7:19 AM, Colm O hEigeartaigh <
>>> >> coheigea@apache.org>
>>> >> > wrote:
>>> >> >
>>> >> > > Hi Jayesh,
>>> >> > >
>>> >> > > Many thanks for your feedback! I was able to make a little further
>>> >> > headway.
>>> >> > > There are two configuration problems with the official docker
>>> image:
>>> >> > >
>>> >> > > a) A mix of "sandbox.eagle.apache.org" and "
>>> server.eagle.apache.org"
>>> >> > (this
>>> >> > > only occurs in the instructions for running the docker image. The
>>> >> version
>>> >> > > that can be started via the script in the eagle source is OK).
>>> I'll
>>> >> > submit
>>> >> > > a PR to fix this once I get a basic use-case working.
>>> >> > > b) For the audit case, it automatically logs HDFS audit logs to
>>> the
>>> >> KAFKA
>>> >> > > topic sandbox_hdfs_audit_log instead of the expected
>>> >> > hdfs_audit_log_sandbox
>>> >> > >
>>> >> > > I've fixed these things locally and I can verify that everything
>>> is
>>> >> > started
>>> >> > > correctly in Ambari. I log into the docker container and create
>>> >> > > hdfs_audit_log_sandbox and hdfs_audit_log_enriched_sandbox
>>> topics,
>>> >> and
>>> >> > > verify that the HDFS audit logs are flowing into the first topic.
>>> >> Then in
>>> >> > > the UI I start the Alert Engine and then the HDFS Audit Log
>>> Monitor
>>> >> > > application (changing localhost:6667 to
>>> server.eagle.apache.org:6667
>>> >> ).
>>> >> > > Both
>>> >> > > applications start up correctly and show "running".
>>> >> > >
>>> >> > > I then create a policy with an email alert along the lines of from
>>> >> > > "HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX[str:contains(src,'/h
>>> base')]
>>> >> > select
>>> >> > > * group by user insert into hdfs_audit_log_enriched_stream_out".
>>> >> However
>>> >> > > at
>>> >> > > this point I'm stuck - nothing appears in the alert window. Is
>>> there
>>> >> > > anything obvious I'm doing wrong, or how can I get access to logs
>>> to
>>> >> > figure
>>> >> > > out what the problem is? Other topics such as
>>> >> "hdfs_audit_event_sandbox"
>>> >> > > are mentioned in the streams window, but the documentation doesn't
>>> >> say to
>>> >> > > create them.
>>> >> > >
>>> >> > > The UI is buggy though on both Firefox and Chromium on Linux. What
>>> >> > > browser/platform are people using with the UI?
>>> >> > >
>>> >> > > Colm.
>>> >> > >
>>> >> > > On Wed, Jan 17, 2018 at 12:27 AM, Jayesh Senjaliya <
>>> jaysen@apache.org
>>> >> >
>>> >> > > wrote:
>>> >> > >
>>> >> > > > Hi Colm,
>>> >> > > >
>>> >> > > > Please find my comments inline.
>>> >> > > >
>>> >> > > > a) The official docker image uses 0.5.0-SNAPSHOT and not the
>>> >> released
>>> >> > > > version.
>>> >> > > > - this is because we uploaded docker image before apache
>>> release.
>>> >> > > actually
>>> >> > > > this is same codebase apache-eagle-0.5, and it can be fixed
>>> easily
>>> >> by
>>> >> > > just
>>> >> > > > rebuilding docker image. there should not be any mismatch due to
>>> >> this.
>>> >> > > >
>>> >> > > > b) Aside from the above, the official docker image uses a mix
>>> of "
>>> >> > > > server.eagle.apache.org" and "sandbox.eagle.apache.org" as the
>>> host
>>> >> > > name.
>>> >> > > > The HBase service doesn't start by default in Ambari as a
>>> result.
>>> >> > > > - the only places it uses sandbox is in example script which you
>>> >> will
>>> >> > > have
>>> >> > > > to update anyway, which i agree that it would be good to keep it
>>> >> > > > consistent.
>>> >> > > >
>>> >> > > > c) The UI seems quite buggy. On both chromium and firefox, I
>>> only
>>> >> see
>>> >> > > > links to "Sandbox" and "Alert" on the left hand-side. Once I
>>> click
>>> >> on
>>> >> > > > "Alert" I have no way of going back to see the applications. I
>>> don't
>>> >> > see
>>> >> > > > the links to "integration" or "sites" as in the picture here:
>>> >> > > > http://eagle.apache.org/docs/latest/applications/#jmx-monito
>>> ring
>>> >> > > > - when hbase is as deep storage is used, and if eagle app has
>>> issue
>>> >> > > > connecting to hbase, the UI becomes unresponsive.
>>> >> > > >
>>> >> > > > d) In chromium, the button to create a new policy does not
>>> exist - I
>>> >> > can
>>> >> > > > only see it on Firefox.
>>> >> > > > - i have seen when you logged in, you will see admin actions.
>>> but if
>>> >> > this
>>> >> > > > still an issue, can you please file UI bug?
>>> >> > > >
>>> >> > > > e) I'm trying to get the "Hdfs Audit Log Monitor" use-case
>>> working,
>>> >> but
>>> >> > > it
>>> >> > > > seems to be stuck in "Initialized".
>>> >> > > > this eagle docs has example on how to setup the app. pls let us
>>> >> know if
>>> >> > > > you find any gaps.
>>> >> > > >
>>> >> > > > Thanks for trying out, and sharing your findings,
>>> >> > > > Jayesh
>>> >> > > >
>>> >> > > >
>>> >> > > > On Tue, Jan 16, 2018 at 3:34 AM, Colm O hEigeartaigh <
>>> >> > > coheigea@apache.org>
>>> >> > > > wrote:
>>> >> > > >
>>> >> > > >> Hi all,
>>> >> > > >>
>>> >> > > >> I'm trying to play around a bit with Apache Eagle 0.5.0 to no
>>> >> avail.
>>> >> > > Here
>>> >> > > >> are the problems I've run into so far:
>>> >> > > >>
>>> >> > > >> a) The official docker image uses 0.5.0-SNAPSHOT and not the
>>> >> released
>>> >> > > >> version.
>>> >> > > >>
>>> >> > > >> b) Aside from the above, the official docker image uses a mix
>>> of "
>>> >> > > >> server.eagle.apache.org" and "sandbox.eagle.apache.org" as the
>>> >> host
>>> >> > > >> name. The HBase service doesn't start by default in Ambari as a
>>> >> > result.
>>> >> > > >>
>>> >> > > >> c) The UI seems quite buggy. On both chromium and firefox, I
>>> only
>>> >> see
>>> >> > > >> links to "Sandbox" and "Alert" on the left hand-side. Once I
>>> click
>>> >> on
>>> >> > > >> "Alert" I have no way of going back to see the applications. I
>>> >> don't
>>> >> > see
>>> >> > > >> the links to "integration" or "sites" as in the picture here:
>>> >> > > >> http://eagle.apache.org/docs/latest/applications/#jmx-monito
>>> ring
>>> >> > > >>
>>> >> > > >> d) In chromium, the button to create a new policy does not
>>> exist -
>>> >> I
>>> >> > can
>>> >> > > >> only see it on Firefox.
>>> >> > > >>
>>> >> > > >> e) I'm trying to get the "Hdfs Audit Log Monitor" use-case
>>> working,
>>> >> > but
>>> >> > > >> it seems to be stuck in "Initialized".
>>> >> > > >>
>>> >> > > >> Could someone fill me in on what the "recommended" way is to
>>> start
>>> >> > > Apache
>>> >> > > >> Eagle so that I can play around with the functionality that it
>>> >> offers?
>>> >> > > >> Clearly the docker approach is buggy. Also, what browser
>>> should be
>>> >> > used?
>>> >> > > >>
>>> >> > > >> Thanks,
>>> >> > > >>
>>> >> > > >> Colm.
>>> >> > > >>
>>> >> > > >>
>>> >> > > >> --
>>> >> > > >> Colm O hEigeartaigh
>>> >> > > >>
>>> >> > > >> Talend Community Coder
>>> >> > > >> http://coders.talend.com
>>> >> > > >>
>>> >> > > >
>>> >> > > >
>>> >> > >
>>> >> > >
>>> >> > > --
>>> >> > > Colm O hEigeartaigh
>>> >> > >
>>> >> > > Talend Community Coder
>>> >> > > http://coders.talend.com
>>> >> > >
>>> >> >
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> Colm O hEigeartaigh
>>> >>
>>> >> Talend Community Coder
>>> >> http://coders.talend.com
>>> >>
>>> >>
>>> >>
>>> >
>>> >
>>> > --
>>> > Colm O hEigeartaigh
>>> >
>>> > Talend Community Coder
>>> > http://coders.talend.com
>>> >
>>>
>>>
>>>
>>> --
>>> Colm O hEigeartaigh
>>>
>>> Talend Community Coder
>>> http://coders.talend.com
>>>
>>
>>
>


-- 
Colm O hEigeartaigh

Talend Community Coder
http://coders.talend.com

Re: Unable to get 0.5.0 release working

Posted by Colm O hEigeartaigh <co...@apache.org>.
Thanks again for your feedback. Jayesh, adding AlertEagleStorePlugin did
the trick, I can now see alerts in the UI, thanks! By the way, I can't
configure two Alert Publishers, or else the Alert DeDuplicator bins the
alert. Is this a known issue?

Could I ask which browser people are using with the UI? There appears to be
a  bug with Chromium where it doesn't list the pages under Auth.isAdmin
even though I am logged on as an administrator. It works OK in Firefox.
Even with Firefox though, I only see a limited number of links in the
left-hand column - I can't get back to the "integration" page. Can someone
else confirm this please?

Could I suggest the devs do some basic house-keeping tasks:

a) "Release" version 0.5.0 in JIRA (it's still listed as "unreleased").
b) Figure out whether the next version will be 0.5.1 or 0.6.0 and update
the versions on Master accordingly with 0.5.1-SNAPSHOT or 0.6.0-SNAPSHOT.
There are some issues marked here as resolved for 0.5.1 -
https://issues.apache.org/jira/projects/EAGLE/versions/12341128), however I
don't see a branch for 0.5.x?

Colm.

On Thu, Jan 25, 2018 at 8:16 AM, Jayesh Senjaliya <ja...@apache.org> wrote:

> Hi,
>
> we do use eagle 0.5 in production although we dont use all the available
> hadoop applications.
>
> EAGLE-968 <https://issues.apache.org/jira/browse/EAGLE-968> is a fix for
> email issue we found while our testing. should be merged soon after a
> rebase.
>
> @Colm, did you tried adding storage publisher (AlertEagleStorePlugin)? to
> see alerts on UI ?
>
> Thanks
> Jayesh
>
>
>
>
>
>
> On Wed, Jan 24, 2018 at 7:08 PM, Edward Zhang <yo...@gmail.com>
> wrote:
>
>> Eagle 0.5 was deployed in production as far as I know, but it may not be
>> exact the current version in master branch.
>>
>> Thanks for your investigation, seems there is still some bug in 0.5, but
>> this particular issue seems is due to dependent components version conflict.
>>
>> @Jayesh is this Jira ready for merge to master? https://issues.apache.
>> org/jira/browse/EAGLE-968
>>
>>
>> Thanks
>> Edward
>>
>> On Tue, Jan 23, 2018 at 5:10 AM, Colm O hEigeartaigh <coheigea@apache.org
>> > wrote:
>>
>>> OK I've made some more progress. I wasn't seeing any email alerts due to
>>> https://issues.apache.org/jira/browse/EAGLE-968. Once I configure a
>>> Kafka
>>> alert, I can see the alerts flowing into my topic. It's still not clear
>>> to
>>> me however where the policy "output" is going. I also don't see any
>>> alerts
>>> in the UI window.
>>>
>>> Could I ask what the status of the project is in general? There have been
>>> no commits to master since November, so I'm not sure if there is any
>>> point
>>> in submitting Pull Requests for outstanding bugs? Are recent versions of
>>> Apache Eagle used in production?
>>>
>>> Colm.
>>>
>>> On Mon, Jan 22, 2018 at 1:07 PM, Colm O hEigeartaigh <
>>> coheigea@apache.org>
>>> wrote:
>>>
>>> >
>>> > I've done that but I'm not seeing any alerts, which is why I want to
>>> find
>>> > out what the "output" of a policy is and where I can check this.
>>> >
>>> > Colm.
>>> >
>>> > On Mon, Jan 22, 2018 at 1:05 PM, SUDHA JENSLIN <sj...@gmail.com>
>>> wrote:
>>> >
>>> >> Create and add a publisher to see the output.
>>> >>
>>> >>
>>> >>
>>> >> Regards,
>>> >> Sudha jenslin
>>> >>
>>> >> On Jan 22, 2018 6:31 PM, "Colm O hEigeartaigh" <co...@apache.org>
>>> >> wrote:
>>> >>
>>> >> Thanks - the error was due to a problem running Storm with Java 1.8.
>>> I've
>>> >> abandoned the docker image for now, and I'm trying to get it working
>>> >> locally.
>>> >>
>>> >> There are two things I'm not clear on currently, if someone could
>>> fill me
>>> >> in:
>>> >>
>>> >> a) For the  'Hdfs Audit Log Monitor' application, the Kafka Consumer
>>> Topic
>>> >> is 'hdfs_audit_log_sandbox'. Under 'Kafka Topic for Auditlog Event
>>> Sink'
>>> >> it
>>> >> also specifies 'hdfs_audit_event_sandbox'. However the documentation
>>> for
>>> >> the application mentions 'hdfs_audit_log_enriched_sandbox'?
>>> >>
>>> >> When I click on "STREAMS", the "HDFS_AUDIT_LOG_ENRICHED_STREA
>>> M_SANDBOX"
>>> >> uses the topic "hdfs_audit_event_sandbox". And indeed when I run the
>>> >> application, I can see cleansed log data appearing in
>>> >> "hdfs_audit_event_sandbox". So I'm thinking here that
>>> >> 'hdfs_audit_log_enriched_sandbox' is not correct or necessary?
>>> >>
>>> >> b) It's unclear to me where the output data goes when you create a
>>> policy.
>>> >> E.g. say I have:
>>> >>
>>> >> from HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX[str:contains(src,'/hb
>>> ase')]
>>> >> select * group by user insert into hdfs_audit_log_enriched_stream_out
>>> >>
>>> >> Where is "hdfs_audit_log_enriched_stream_out" defined (is it a Kafka
>>> >> topic?). How can I check the output to make sure the policy is working
>>> >> correctly?
>>> >>
>>> >> Thanks,
>>> >>
>>> >> Colm.
>>> >>
>>> >> On Wed, Jan 17, 2018 at 10:32 PM, Edward Zhang <
>>> yonzhang2012@gmail.com>
>>> >> wrote:
>>> >>
>>> >> > There is a data preparation stage between data source(HDFS audit
>>> log)
>>> >> and
>>> >> > Alert Engine. This stage is running in Storm and transform the raw
>>> HDFS
>>> >> log
>>> >> > into something which can be alerted.
>>> >> >
>>> >> > The input for data preparation is hdfs_audit_log_sandbox topic and
>>> >> output
>>> >> > is
>>> >> >  hdfs_audit_log_enriched_sandbox.
>>> >> > The input for Alert Engine is hdfs_audit_log_enriched_sandbox and
>>> >> output
>>> >> > is
>>> >> > hdfs_audit_log_alert_sandbox.
>>> >> >
>>> >> > Seems in your case, the data preparation staging is not working. We
>>> >> > probably need look at Storm console and figure out if that part is
>>> >> working.
>>> >> >
>>> >> > Thanks
>>> >> > Edward
>>> >> >
>>> >> > On Wed, Jan 17, 2018 at 7:19 AM, Colm O hEigeartaigh <
>>> >> coheigea@apache.org>
>>> >> > wrote:
>>> >> >
>>> >> > > Hi Jayesh,
>>> >> > >
>>> >> > > Many thanks for your feedback! I was able to make a little further
>>> >> > headway.
>>> >> > > There are two configuration problems with the official docker
>>> image:
>>> >> > >
>>> >> > > a) A mix of "sandbox.eagle.apache.org" and "
>>> server.eagle.apache.org"
>>> >> > (this
>>> >> > > only occurs in the instructions for running the docker image. The
>>> >> version
>>> >> > > that can be started via the script in the eagle source is OK).
>>> I'll
>>> >> > submit
>>> >> > > a PR to fix this once I get a basic use-case working.
>>> >> > > b) For the audit case, it automatically logs HDFS audit logs to
>>> the
>>> >> KAFKA
>>> >> > > topic sandbox_hdfs_audit_log instead of the expected
>>> >> > hdfs_audit_log_sandbox
>>> >> > >
>>> >> > > I've fixed these things locally and I can verify that everything
>>> is
>>> >> > started
>>> >> > > correctly in Ambari. I log into the docker container and create
>>> >> > > hdfs_audit_log_sandbox and hdfs_audit_log_enriched_sandbox
>>> topics,
>>> >> and
>>> >> > > verify that the HDFS audit logs are flowing into the first topic.
>>> >> Then in
>>> >> > > the UI I start the Alert Engine and then the HDFS Audit Log
>>> Monitor
>>> >> > > application (changing localhost:6667 to
>>> server.eagle.apache.org:6667
>>> >> ).
>>> >> > > Both
>>> >> > > applications start up correctly and show "running".
>>> >> > >
>>> >> > > I then create a policy with an email alert along the lines of from
>>> >> > > "HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX[str:contains(src,'/h
>>> base')]
>>> >> > select
>>> >> > > * group by user insert into hdfs_audit_log_enriched_stream_out".
>>> >> However
>>> >> > > at
>>> >> > > this point I'm stuck - nothing appears in the alert window. Is
>>> there
>>> >> > > anything obvious I'm doing wrong, or how can I get access to logs
>>> to
>>> >> > figure
>>> >> > > out what the problem is? Other topics such as
>>> >> "hdfs_audit_event_sandbox"
>>> >> > > are mentioned in the streams window, but the documentation doesn't
>>> >> say to
>>> >> > > create them.
>>> >> > >
>>> >> > > The UI is buggy though on both Firefox and Chromium on Linux. What
>>> >> > > browser/platform are people using with the UI?
>>> >> > >
>>> >> > > Colm.
>>> >> > >
>>> >> > > On Wed, Jan 17, 2018 at 12:27 AM, Jayesh Senjaliya <
>>> jaysen@apache.org
>>> >> >
>>> >> > > wrote:
>>> >> > >
>>> >> > > > Hi Colm,
>>> >> > > >
>>> >> > > > Please find my comments inline.
>>> >> > > >
>>> >> > > > a) The official docker image uses 0.5.0-SNAPSHOT and not the
>>> >> released
>>> >> > > > version.
>>> >> > > > - this is because we uploaded docker image before apache
>>> release.
>>> >> > > actually
>>> >> > > > this is same codebase apache-eagle-0.5, and it can be fixed
>>> easily
>>> >> by
>>> >> > > just
>>> >> > > > rebuilding docker image. there should not be any mismatch due to
>>> >> this.
>>> >> > > >
>>> >> > > > b) Aside from the above, the official docker image uses a mix
>>> of "
>>> >> > > > server.eagle.apache.org" and "sandbox.eagle.apache.org" as the
>>> host
>>> >> > > name.
>>> >> > > > The HBase service doesn't start by default in Ambari as a
>>> result.
>>> >> > > > - the only places it uses sandbox is in example script which you
>>> >> will
>>> >> > > have
>>> >> > > > to update anyway, which i agree that it would be good to keep it
>>> >> > > > consistent.
>>> >> > > >
>>> >> > > > c) The UI seems quite buggy. On both chromium and firefox, I
>>> only
>>> >> see
>>> >> > > > links to "Sandbox" and "Alert" on the left hand-side. Once I
>>> click
>>> >> on
>>> >> > > > "Alert" I have no way of going back to see the applications. I
>>> don't
>>> >> > see
>>> >> > > > the links to "integration" or "sites" as in the picture here:
>>> >> > > > http://eagle.apache.org/docs/latest/applications/#jmx-monito
>>> ring
>>> >> > > > - when hbase is as deep storage is used, and if eagle app has
>>> issue
>>> >> > > > connecting to hbase, the UI becomes unresponsive.
>>> >> > > >
>>> >> > > > d) In chromium, the button to create a new policy does not
>>> exist - I
>>> >> > can
>>> >> > > > only see it on Firefox.
>>> >> > > > - i have seen when you logged in, you will see admin actions.
>>> but if
>>> >> > this
>>> >> > > > still an issue, can you please file UI bug?
>>> >> > > >
>>> >> > > > e) I'm trying to get the "Hdfs Audit Log Monitor" use-case
>>> working,
>>> >> but
>>> >> > > it
>>> >> > > > seems to be stuck in "Initialized".
>>> >> > > > this eagle docs has example on how to setup the app. pls let us
>>> >> know if
>>> >> > > > you find any gaps.
>>> >> > > >
>>> >> > > > Thanks for trying out, and sharing your findings,
>>> >> > > > Jayesh
>>> >> > > >
>>> >> > > >
>>> >> > > > On Tue, Jan 16, 2018 at 3:34 AM, Colm O hEigeartaigh <
>>> >> > > coheigea@apache.org>
>>> >> > > > wrote:
>>> >> > > >
>>> >> > > >> Hi all,
>>> >> > > >>
>>> >> > > >> I'm trying to play around a bit with Apache Eagle 0.5.0 to no
>>> >> avail.
>>> >> > > Here
>>> >> > > >> are the problems I've run into so far:
>>> >> > > >>
>>> >> > > >> a) The official docker image uses 0.5.0-SNAPSHOT and not the
>>> >> released
>>> >> > > >> version.
>>> >> > > >>
>>> >> > > >> b) Aside from the above, the official docker image uses a mix
>>> of "
>>> >> > > >> server.eagle.apache.org" and "sandbox.eagle.apache.org" as the
>>> >> host
>>> >> > > >> name. The HBase service doesn't start by default in Ambari as a
>>> >> > result.
>>> >> > > >>
>>> >> > > >> c) The UI seems quite buggy. On both chromium and firefox, I
>>> only
>>> >> see
>>> >> > > >> links to "Sandbox" and "Alert" on the left hand-side. Once I
>>> click
>>> >> on
>>> >> > > >> "Alert" I have no way of going back to see the applications. I
>>> >> don't
>>> >> > see
>>> >> > > >> the links to "integration" or "sites" as in the picture here:
>>> >> > > >> http://eagle.apache.org/docs/latest/applications/#jmx-monito
>>> ring
>>> >> > > >>
>>> >> > > >> d) In chromium, the button to create a new policy does not
>>> exist -
>>> >> I
>>> >> > can
>>> >> > > >> only see it on Firefox.
>>> >> > > >>
>>> >> > > >> e) I'm trying to get the "Hdfs Audit Log Monitor" use-case
>>> working,
>>> >> > but
>>> >> > > >> it seems to be stuck in "Initialized".
>>> >> > > >>
>>> >> > > >> Could someone fill me in on what the "recommended" way is to
>>> start
>>> >> > > Apache
>>> >> > > >> Eagle so that I can play around with the functionality that it
>>> >> offers?
>>> >> > > >> Clearly the docker approach is buggy. Also, what browser
>>> should be
>>> >> > used?
>>> >> > > >>
>>> >> > > >> Thanks,
>>> >> > > >>
>>> >> > > >> Colm.
>>> >> > > >>
>>> >> > > >>
>>> >> > > >> --
>>> >> > > >> Colm O hEigeartaigh
>>> >> > > >>
>>> >> > > >> Talend Community Coder
>>> >> > > >> http://coders.talend.com
>>> >> > > >>
>>> >> > > >
>>> >> > > >
>>> >> > >
>>> >> > >
>>> >> > > --
>>> >> > > Colm O hEigeartaigh
>>> >> > >
>>> >> > > Talend Community Coder
>>> >> > > http://coders.talend.com
>>> >> > >
>>> >> >
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> Colm O hEigeartaigh
>>> >>
>>> >> Talend Community Coder
>>> >> http://coders.talend.com
>>> >>
>>> >>
>>> >>
>>> >
>>> >
>>> > --
>>> > Colm O hEigeartaigh
>>> >
>>> > Talend Community Coder
>>> > http://coders.talend.com
>>> >
>>>
>>>
>>>
>>> --
>>> Colm O hEigeartaigh
>>>
>>> Talend Community Coder
>>> http://coders.talend.com
>>>
>>
>>
>


-- 
Colm O hEigeartaigh

Talend Community Coder
http://coders.talend.com

Re: Unable to get 0.5.0 release working

Posted by Jayesh Senjaliya <ja...@apache.org>.
Hi,

we do use eagle 0.5 in production although we dont use all the available
hadoop applications.

EAGLE-968 <https://issues.apache.org/jira/browse/EAGLE-968> is a fix for
email issue we found while our testing. should be merged soon after a
rebase.

@Colm, did you tried adding storage publisher (AlertEagleStorePlugin)? to
see alerts on UI ?

Thanks
Jayesh






On Wed, Jan 24, 2018 at 7:08 PM, Edward Zhang <yo...@gmail.com>
wrote:

> Eagle 0.5 was deployed in production as far as I know, but it may not be
> exact the current version in master branch.
>
> Thanks for your investigation, seems there is still some bug in 0.5, but
> this particular issue seems is due to dependent components version conflict.
>
> @Jayesh is this Jira ready for merge to master? https://issues.apache.
> org/jira/browse/EAGLE-968
>
>
> Thanks
> Edward
>
> On Tue, Jan 23, 2018 at 5:10 AM, Colm O hEigeartaigh <co...@apache.org>
> wrote:
>
>> OK I've made some more progress. I wasn't seeing any email alerts due to
>> https://issues.apache.org/jira/browse/EAGLE-968. Once I configure a Kafka
>> alert, I can see the alerts flowing into my topic. It's still not clear to
>> me however where the policy "output" is going. I also don't see any alerts
>> in the UI window.
>>
>> Could I ask what the status of the project is in general? There have been
>> no commits to master since November, so I'm not sure if there is any point
>> in submitting Pull Requests for outstanding bugs? Are recent versions of
>> Apache Eagle used in production?
>>
>> Colm.
>>
>> On Mon, Jan 22, 2018 at 1:07 PM, Colm O hEigeartaigh <coheigea@apache.org
>> >
>> wrote:
>>
>> >
>> > I've done that but I'm not seeing any alerts, which is why I want to
>> find
>> > out what the "output" of a policy is and where I can check this.
>> >
>> > Colm.
>> >
>> > On Mon, Jan 22, 2018 at 1:05 PM, SUDHA JENSLIN <sj...@gmail.com>
>> wrote:
>> >
>> >> Create and add a publisher to see the output.
>> >>
>> >>
>> >>
>> >> Regards,
>> >> Sudha jenslin
>> >>
>> >> On Jan 22, 2018 6:31 PM, "Colm O hEigeartaigh" <co...@apache.org>
>> >> wrote:
>> >>
>> >> Thanks - the error was due to a problem running Storm with Java 1.8.
>> I've
>> >> abandoned the docker image for now, and I'm trying to get it working
>> >> locally.
>> >>
>> >> There are two things I'm not clear on currently, if someone could fill
>> me
>> >> in:
>> >>
>> >> a) For the  'Hdfs Audit Log Monitor' application, the Kafka Consumer
>> Topic
>> >> is 'hdfs_audit_log_sandbox'. Under 'Kafka Topic for Auditlog Event
>> Sink'
>> >> it
>> >> also specifies 'hdfs_audit_event_sandbox'. However the documentation
>> for
>> >> the application mentions 'hdfs_audit_log_enriched_sandbox'?
>> >>
>> >> When I click on "STREAMS", the "HDFS_AUDIT_LOG_ENRICHED_STREA
>> M_SANDBOX"
>> >> uses the topic "hdfs_audit_event_sandbox". And indeed when I run the
>> >> application, I can see cleansed log data appearing in
>> >> "hdfs_audit_event_sandbox". So I'm thinking here that
>> >> 'hdfs_audit_log_enriched_sandbox' is not correct or necessary?
>> >>
>> >> b) It's unclear to me where the output data goes when you create a
>> policy.
>> >> E.g. say I have:
>> >>
>> >> from HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX[str:contains(src,'/
>> hbase')]
>> >> select * group by user insert into hdfs_audit_log_enriched_stream_out
>> >>
>> >> Where is "hdfs_audit_log_enriched_stream_out" defined (is it a Kafka
>> >> topic?). How can I check the output to make sure the policy is working
>> >> correctly?
>> >>
>> >> Thanks,
>> >>
>> >> Colm.
>> >>
>> >> On Wed, Jan 17, 2018 at 10:32 PM, Edward Zhang <yonzhang2012@gmail.com
>> >
>> >> wrote:
>> >>
>> >> > There is a data preparation stage between data source(HDFS audit log)
>> >> and
>> >> > Alert Engine. This stage is running in Storm and transform the raw
>> HDFS
>> >> log
>> >> > into something which can be alerted.
>> >> >
>> >> > The input for data preparation is hdfs_audit_log_sandbox topic and
>> >> output
>> >> > is
>> >> >  hdfs_audit_log_enriched_sandbox.
>> >> > The input for Alert Engine is hdfs_audit_log_enriched_sandbox and
>> >> output
>> >> > is
>> >> > hdfs_audit_log_alert_sandbox.
>> >> >
>> >> > Seems in your case, the data preparation staging is not working. We
>> >> > probably need look at Storm console and figure out if that part is
>> >> working.
>> >> >
>> >> > Thanks
>> >> > Edward
>> >> >
>> >> > On Wed, Jan 17, 2018 at 7:19 AM, Colm O hEigeartaigh <
>> >> coheigea@apache.org>
>> >> > wrote:
>> >> >
>> >> > > Hi Jayesh,
>> >> > >
>> >> > > Many thanks for your feedback! I was able to make a little further
>> >> > headway.
>> >> > > There are two configuration problems with the official docker
>> image:
>> >> > >
>> >> > > a) A mix of "sandbox.eagle.apache.org" and "
>> server.eagle.apache.org"
>> >> > (this
>> >> > > only occurs in the instructions for running the docker image. The
>> >> version
>> >> > > that can be started via the script in the eagle source is OK). I'll
>> >> > submit
>> >> > > a PR to fix this once I get a basic use-case working.
>> >> > > b) For the audit case, it automatically logs HDFS audit logs to the
>> >> KAFKA
>> >> > > topic sandbox_hdfs_audit_log instead of the expected
>> >> > hdfs_audit_log_sandbox
>> >> > >
>> >> > > I've fixed these things locally and I can verify that everything is
>> >> > started
>> >> > > correctly in Ambari. I log into the docker container and create
>> >> > > hdfs_audit_log_sandbox and hdfs_audit_log_enriched_sandbox topics,
>> >> and
>> >> > > verify that the HDFS audit logs are flowing into the first topic.
>> >> Then in
>> >> > > the UI I start the Alert Engine and then the HDFS Audit Log Monitor
>> >> > > application (changing localhost:6667 to
>> server.eagle.apache.org:6667
>> >> ).
>> >> > > Both
>> >> > > applications start up correctly and show "running".
>> >> > >
>> >> > > I then create a policy with an email alert along the lines of from
>> >> > > "HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX[str:contains(src,'/
>> hbase')]
>> >> > select
>> >> > > * group by user insert into hdfs_audit_log_enriched_stream_out".
>> >> However
>> >> > > at
>> >> > > this point I'm stuck - nothing appears in the alert window. Is
>> there
>> >> > > anything obvious I'm doing wrong, or how can I get access to logs
>> to
>> >> > figure
>> >> > > out what the problem is? Other topics such as
>> >> "hdfs_audit_event_sandbox"
>> >> > > are mentioned in the streams window, but the documentation doesn't
>> >> say to
>> >> > > create them.
>> >> > >
>> >> > > The UI is buggy though on both Firefox and Chromium on Linux. What
>> >> > > browser/platform are people using with the UI?
>> >> > >
>> >> > > Colm.
>> >> > >
>> >> > > On Wed, Jan 17, 2018 at 12:27 AM, Jayesh Senjaliya <
>> jaysen@apache.org
>> >> >
>> >> > > wrote:
>> >> > >
>> >> > > > Hi Colm,
>> >> > > >
>> >> > > > Please find my comments inline.
>> >> > > >
>> >> > > > a) The official docker image uses 0.5.0-SNAPSHOT and not the
>> >> released
>> >> > > > version.
>> >> > > > - this is because we uploaded docker image before apache release.
>> >> > > actually
>> >> > > > this is same codebase apache-eagle-0.5, and it can be fixed
>> easily
>> >> by
>> >> > > just
>> >> > > > rebuilding docker image. there should not be any mismatch due to
>> >> this.
>> >> > > >
>> >> > > > b) Aside from the above, the official docker image uses a mix of
>> "
>> >> > > > server.eagle.apache.org" and "sandbox.eagle.apache.org" as the
>> host
>> >> > > name.
>> >> > > > The HBase service doesn't start by default in Ambari as a result.
>> >> > > > - the only places it uses sandbox is in example script which you
>> >> will
>> >> > > have
>> >> > > > to update anyway, which i agree that it would be good to keep it
>> >> > > > consistent.
>> >> > > >
>> >> > > > c) The UI seems quite buggy. On both chromium and firefox, I only
>> >> see
>> >> > > > links to "Sandbox" and "Alert" on the left hand-side. Once I
>> click
>> >> on
>> >> > > > "Alert" I have no way of going back to see the applications. I
>> don't
>> >> > see
>> >> > > > the links to "integration" or "sites" as in the picture here:
>> >> > > > http://eagle.apache.org/docs/latest/applications/#jmx-monitoring
>> >> > > > - when hbase is as deep storage is used, and if eagle app has
>> issue
>> >> > > > connecting to hbase, the UI becomes unresponsive.
>> >> > > >
>> >> > > > d) In chromium, the button to create a new policy does not exist
>> - I
>> >> > can
>> >> > > > only see it on Firefox.
>> >> > > > - i have seen when you logged in, you will see admin actions.
>> but if
>> >> > this
>> >> > > > still an issue, can you please file UI bug?
>> >> > > >
>> >> > > > e) I'm trying to get the "Hdfs Audit Log Monitor" use-case
>> working,
>> >> but
>> >> > > it
>> >> > > > seems to be stuck in "Initialized".
>> >> > > > this eagle docs has example on how to setup the app. pls let us
>> >> know if
>> >> > > > you find any gaps.
>> >> > > >
>> >> > > > Thanks for trying out, and sharing your findings,
>> >> > > > Jayesh
>> >> > > >
>> >> > > >
>> >> > > > On Tue, Jan 16, 2018 at 3:34 AM, Colm O hEigeartaigh <
>> >> > > coheigea@apache.org>
>> >> > > > wrote:
>> >> > > >
>> >> > > >> Hi all,
>> >> > > >>
>> >> > > >> I'm trying to play around a bit with Apache Eagle 0.5.0 to no
>> >> avail.
>> >> > > Here
>> >> > > >> are the problems I've run into so far:
>> >> > > >>
>> >> > > >> a) The official docker image uses 0.5.0-SNAPSHOT and not the
>> >> released
>> >> > > >> version.
>> >> > > >>
>> >> > > >> b) Aside from the above, the official docker image uses a mix
>> of "
>> >> > > >> server.eagle.apache.org" and "sandbox.eagle.apache.org" as the
>> >> host
>> >> > > >> name. The HBase service doesn't start by default in Ambari as a
>> >> > result.
>> >> > > >>
>> >> > > >> c) The UI seems quite buggy. On both chromium and firefox, I
>> only
>> >> see
>> >> > > >> links to "Sandbox" and "Alert" on the left hand-side. Once I
>> click
>> >> on
>> >> > > >> "Alert" I have no way of going back to see the applications. I
>> >> don't
>> >> > see
>> >> > > >> the links to "integration" or "sites" as in the picture here:
>> >> > > >> http://eagle.apache.org/docs/latest/applications/#jmx-monito
>> ring
>> >> > > >>
>> >> > > >> d) In chromium, the button to create a new policy does not
>> exist -
>> >> I
>> >> > can
>> >> > > >> only see it on Firefox.
>> >> > > >>
>> >> > > >> e) I'm trying to get the "Hdfs Audit Log Monitor" use-case
>> working,
>> >> > but
>> >> > > >> it seems to be stuck in "Initialized".
>> >> > > >>
>> >> > > >> Could someone fill me in on what the "recommended" way is to
>> start
>> >> > > Apache
>> >> > > >> Eagle so that I can play around with the functionality that it
>> >> offers?
>> >> > > >> Clearly the docker approach is buggy. Also, what browser should
>> be
>> >> > used?
>> >> > > >>
>> >> > > >> Thanks,
>> >> > > >>
>> >> > > >> Colm.
>> >> > > >>
>> >> > > >>
>> >> > > >> --
>> >> > > >> Colm O hEigeartaigh
>> >> > > >>
>> >> > > >> Talend Community Coder
>> >> > > >> http://coders.talend.com
>> >> > > >>
>> >> > > >
>> >> > > >
>> >> > >
>> >> > >
>> >> > > --
>> >> > > Colm O hEigeartaigh
>> >> > >
>> >> > > Talend Community Coder
>> >> > > http://coders.talend.com
>> >> > >
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Colm O hEigeartaigh
>> >>
>> >> Talend Community Coder
>> >> http://coders.talend.com
>> >>
>> >>
>> >>
>> >
>> >
>> > --
>> > Colm O hEigeartaigh
>> >
>> > Talend Community Coder
>> > http://coders.talend.com
>> >
>>
>>
>>
>> --
>> Colm O hEigeartaigh
>>
>> Talend Community Coder
>> http://coders.talend.com
>>
>
>

Re: Unable to get 0.5.0 release working

Posted by Jayesh Senjaliya <ja...@apache.org>.
Hi,

we do use eagle 0.5 in production although we dont use all the available
hadoop applications.

EAGLE-968 <https://issues.apache.org/jira/browse/EAGLE-968> is a fix for
email issue we found while our testing. should be merged soon after a
rebase.

@Colm, did you tried adding storage publisher (AlertEagleStorePlugin)? to
see alerts on UI ?

Thanks
Jayesh






On Wed, Jan 24, 2018 at 7:08 PM, Edward Zhang <yo...@gmail.com>
wrote:

> Eagle 0.5 was deployed in production as far as I know, but it may not be
> exact the current version in master branch.
>
> Thanks for your investigation, seems there is still some bug in 0.5, but
> this particular issue seems is due to dependent components version conflict.
>
> @Jayesh is this Jira ready for merge to master? https://issues.apache.
> org/jira/browse/EAGLE-968
>
>
> Thanks
> Edward
>
> On Tue, Jan 23, 2018 at 5:10 AM, Colm O hEigeartaigh <co...@apache.org>
> wrote:
>
>> OK I've made some more progress. I wasn't seeing any email alerts due to
>> https://issues.apache.org/jira/browse/EAGLE-968. Once I configure a Kafka
>> alert, I can see the alerts flowing into my topic. It's still not clear to
>> me however where the policy "output" is going. I also don't see any alerts
>> in the UI window.
>>
>> Could I ask what the status of the project is in general? There have been
>> no commits to master since November, so I'm not sure if there is any point
>> in submitting Pull Requests for outstanding bugs? Are recent versions of
>> Apache Eagle used in production?
>>
>> Colm.
>>
>> On Mon, Jan 22, 2018 at 1:07 PM, Colm O hEigeartaigh <coheigea@apache.org
>> >
>> wrote:
>>
>> >
>> > I've done that but I'm not seeing any alerts, which is why I want to
>> find
>> > out what the "output" of a policy is and where I can check this.
>> >
>> > Colm.
>> >
>> > On Mon, Jan 22, 2018 at 1:05 PM, SUDHA JENSLIN <sj...@gmail.com>
>> wrote:
>> >
>> >> Create and add a publisher to see the output.
>> >>
>> >>
>> >>
>> >> Regards,
>> >> Sudha jenslin
>> >>
>> >> On Jan 22, 2018 6:31 PM, "Colm O hEigeartaigh" <co...@apache.org>
>> >> wrote:
>> >>
>> >> Thanks - the error was due to a problem running Storm with Java 1.8.
>> I've
>> >> abandoned the docker image for now, and I'm trying to get it working
>> >> locally.
>> >>
>> >> There are two things I'm not clear on currently, if someone could fill
>> me
>> >> in:
>> >>
>> >> a) For the  'Hdfs Audit Log Monitor' application, the Kafka Consumer
>> Topic
>> >> is 'hdfs_audit_log_sandbox'. Under 'Kafka Topic for Auditlog Event
>> Sink'
>> >> it
>> >> also specifies 'hdfs_audit_event_sandbox'. However the documentation
>> for
>> >> the application mentions 'hdfs_audit_log_enriched_sandbox'?
>> >>
>> >> When I click on "STREAMS", the "HDFS_AUDIT_LOG_ENRICHED_STREA
>> M_SANDBOX"
>> >> uses the topic "hdfs_audit_event_sandbox". And indeed when I run the
>> >> application, I can see cleansed log data appearing in
>> >> "hdfs_audit_event_sandbox". So I'm thinking here that
>> >> 'hdfs_audit_log_enriched_sandbox' is not correct or necessary?
>> >>
>> >> b) It's unclear to me where the output data goes when you create a
>> policy.
>> >> E.g. say I have:
>> >>
>> >> from HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX[str:contains(src,'/
>> hbase')]
>> >> select * group by user insert into hdfs_audit_log_enriched_stream_out
>> >>
>> >> Where is "hdfs_audit_log_enriched_stream_out" defined (is it a Kafka
>> >> topic?). How can I check the output to make sure the policy is working
>> >> correctly?
>> >>
>> >> Thanks,
>> >>
>> >> Colm.
>> >>
>> >> On Wed, Jan 17, 2018 at 10:32 PM, Edward Zhang <yonzhang2012@gmail.com
>> >
>> >> wrote:
>> >>
>> >> > There is a data preparation stage between data source(HDFS audit log)
>> >> and
>> >> > Alert Engine. This stage is running in Storm and transform the raw
>> HDFS
>> >> log
>> >> > into something which can be alerted.
>> >> >
>> >> > The input for data preparation is hdfs_audit_log_sandbox topic and
>> >> output
>> >> > is
>> >> >  hdfs_audit_log_enriched_sandbox.
>> >> > The input for Alert Engine is hdfs_audit_log_enriched_sandbox and
>> >> output
>> >> > is
>> >> > hdfs_audit_log_alert_sandbox.
>> >> >
>> >> > Seems in your case, the data preparation staging is not working. We
>> >> > probably need look at Storm console and figure out if that part is
>> >> working.
>> >> >
>> >> > Thanks
>> >> > Edward
>> >> >
>> >> > On Wed, Jan 17, 2018 at 7:19 AM, Colm O hEigeartaigh <
>> >> coheigea@apache.org>
>> >> > wrote:
>> >> >
>> >> > > Hi Jayesh,
>> >> > >
>> >> > > Many thanks for your feedback! I was able to make a little further
>> >> > headway.
>> >> > > There are two configuration problems with the official docker
>> image:
>> >> > >
>> >> > > a) A mix of "sandbox.eagle.apache.org" and "
>> server.eagle.apache.org"
>> >> > (this
>> >> > > only occurs in the instructions for running the docker image. The
>> >> version
>> >> > > that can be started via the script in the eagle source is OK). I'll
>> >> > submit
>> >> > > a PR to fix this once I get a basic use-case working.
>> >> > > b) For the audit case, it automatically logs HDFS audit logs to the
>> >> KAFKA
>> >> > > topic sandbox_hdfs_audit_log instead of the expected
>> >> > hdfs_audit_log_sandbox
>> >> > >
>> >> > > I've fixed these things locally and I can verify that everything is
>> >> > started
>> >> > > correctly in Ambari. I log into the docker container and create
>> >> > > hdfs_audit_log_sandbox and hdfs_audit_log_enriched_sandbox topics,
>> >> and
>> >> > > verify that the HDFS audit logs are flowing into the first topic.
>> >> Then in
>> >> > > the UI I start the Alert Engine and then the HDFS Audit Log Monitor
>> >> > > application (changing localhost:6667 to
>> server.eagle.apache.org:6667
>> >> ).
>> >> > > Both
>> >> > > applications start up correctly and show "running".
>> >> > >
>> >> > > I then create a policy with an email alert along the lines of from
>> >> > > "HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX[str:contains(src,'/
>> hbase')]
>> >> > select
>> >> > > * group by user insert into hdfs_audit_log_enriched_stream_out".
>> >> However
>> >> > > at
>> >> > > this point I'm stuck - nothing appears in the alert window. Is
>> there
>> >> > > anything obvious I'm doing wrong, or how can I get access to logs
>> to
>> >> > figure
>> >> > > out what the problem is? Other topics such as
>> >> "hdfs_audit_event_sandbox"
>> >> > > are mentioned in the streams window, but the documentation doesn't
>> >> say to
>> >> > > create them.
>> >> > >
>> >> > > The UI is buggy though on both Firefox and Chromium on Linux. What
>> >> > > browser/platform are people using with the UI?
>> >> > >
>> >> > > Colm.
>> >> > >
>> >> > > On Wed, Jan 17, 2018 at 12:27 AM, Jayesh Senjaliya <
>> jaysen@apache.org
>> >> >
>> >> > > wrote:
>> >> > >
>> >> > > > Hi Colm,
>> >> > > >
>> >> > > > Please find my comments inline.
>> >> > > >
>> >> > > > a) The official docker image uses 0.5.0-SNAPSHOT and not the
>> >> released
>> >> > > > version.
>> >> > > > - this is because we uploaded docker image before apache release.
>> >> > > actually
>> >> > > > this is same codebase apache-eagle-0.5, and it can be fixed
>> easily
>> >> by
>> >> > > just
>> >> > > > rebuilding docker image. there should not be any mismatch due to
>> >> this.
>> >> > > >
>> >> > > > b) Aside from the above, the official docker image uses a mix of
>> "
>> >> > > > server.eagle.apache.org" and "sandbox.eagle.apache.org" as the
>> host
>> >> > > name.
>> >> > > > The HBase service doesn't start by default in Ambari as a result.
>> >> > > > - the only places it uses sandbox is in example script which you
>> >> will
>> >> > > have
>> >> > > > to update anyway, which i agree that it would be good to keep it
>> >> > > > consistent.
>> >> > > >
>> >> > > > c) The UI seems quite buggy. On both chromium and firefox, I only
>> >> see
>> >> > > > links to "Sandbox" and "Alert" on the left hand-side. Once I
>> click
>> >> on
>> >> > > > "Alert" I have no way of going back to see the applications. I
>> don't
>> >> > see
>> >> > > > the links to "integration" or "sites" as in the picture here:
>> >> > > > http://eagle.apache.org/docs/latest/applications/#jmx-monitoring
>> >> > > > - when hbase is as deep storage is used, and if eagle app has
>> issue
>> >> > > > connecting to hbase, the UI becomes unresponsive.
>> >> > > >
>> >> > > > d) In chromium, the button to create a new policy does not exist
>> - I
>> >> > can
>> >> > > > only see it on Firefox.
>> >> > > > - i have seen when you logged in, you will see admin actions.
>> but if
>> >> > this
>> >> > > > still an issue, can you please file UI bug?
>> >> > > >
>> >> > > > e) I'm trying to get the "Hdfs Audit Log Monitor" use-case
>> working,
>> >> but
>> >> > > it
>> >> > > > seems to be stuck in "Initialized".
>> >> > > > this eagle docs has example on how to setup the app. pls let us
>> >> know if
>> >> > > > you find any gaps.
>> >> > > >
>> >> > > > Thanks for trying out, and sharing your findings,
>> >> > > > Jayesh
>> >> > > >
>> >> > > >
>> >> > > > On Tue, Jan 16, 2018 at 3:34 AM, Colm O hEigeartaigh <
>> >> > > coheigea@apache.org>
>> >> > > > wrote:
>> >> > > >
>> >> > > >> Hi all,
>> >> > > >>
>> >> > > >> I'm trying to play around a bit with Apache Eagle 0.5.0 to no
>> >> avail.
>> >> > > Here
>> >> > > >> are the problems I've run into so far:
>> >> > > >>
>> >> > > >> a) The official docker image uses 0.5.0-SNAPSHOT and not the
>> >> released
>> >> > > >> version.
>> >> > > >>
>> >> > > >> b) Aside from the above, the official docker image uses a mix
>> of "
>> >> > > >> server.eagle.apache.org" and "sandbox.eagle.apache.org" as the
>> >> host
>> >> > > >> name. The HBase service doesn't start by default in Ambari as a
>> >> > result.
>> >> > > >>
>> >> > > >> c) The UI seems quite buggy. On both chromium and firefox, I
>> only
>> >> see
>> >> > > >> links to "Sandbox" and "Alert" on the left hand-side. Once I
>> click
>> >> on
>> >> > > >> "Alert" I have no way of going back to see the applications. I
>> >> don't
>> >> > see
>> >> > > >> the links to "integration" or "sites" as in the picture here:
>> >> > > >> http://eagle.apache.org/docs/latest/applications/#jmx-monito
>> ring
>> >> > > >>
>> >> > > >> d) In chromium, the button to create a new policy does not
>> exist -
>> >> I
>> >> > can
>> >> > > >> only see it on Firefox.
>> >> > > >>
>> >> > > >> e) I'm trying to get the "Hdfs Audit Log Monitor" use-case
>> working,
>> >> > but
>> >> > > >> it seems to be stuck in "Initialized".
>> >> > > >>
>> >> > > >> Could someone fill me in on what the "recommended" way is to
>> start
>> >> > > Apache
>> >> > > >> Eagle so that I can play around with the functionality that it
>> >> offers?
>> >> > > >> Clearly the docker approach is buggy. Also, what browser should
>> be
>> >> > used?
>> >> > > >>
>> >> > > >> Thanks,
>> >> > > >>
>> >> > > >> Colm.
>> >> > > >>
>> >> > > >>
>> >> > > >> --
>> >> > > >> Colm O hEigeartaigh
>> >> > > >>
>> >> > > >> Talend Community Coder
>> >> > > >> http://coders.talend.com
>> >> > > >>
>> >> > > >
>> >> > > >
>> >> > >
>> >> > >
>> >> > > --
>> >> > > Colm O hEigeartaigh
>> >> > >
>> >> > > Talend Community Coder
>> >> > > http://coders.talend.com
>> >> > >
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Colm O hEigeartaigh
>> >>
>> >> Talend Community Coder
>> >> http://coders.talend.com
>> >>
>> >>
>> >>
>> >
>> >
>> > --
>> > Colm O hEigeartaigh
>> >
>> > Talend Community Coder
>> > http://coders.talend.com
>> >
>>
>>
>>
>> --
>> Colm O hEigeartaigh
>>
>> Talend Community Coder
>> http://coders.talend.com
>>
>
>

Re: Unable to get 0.5.0 release working

Posted by Edward Zhang <yo...@gmail.com>.
Eagle 0.5 was deployed in production as far as I know, but it may not be
exact the current version in master branch.

Thanks for your investigation, seems there is still some bug in 0.5, but
this particular issue seems is due to dependent components version conflict.

@Jayesh is this Jira ready for merge to master?
https://issues.apache.org/jira/browse/EAGLE-968


Thanks
Edward

On Tue, Jan 23, 2018 at 5:10 AM, Colm O hEigeartaigh <co...@apache.org>
wrote:

> OK I've made some more progress. I wasn't seeing any email alerts due to
> https://issues.apache.org/jira/browse/EAGLE-968. Once I configure a Kafka
> alert, I can see the alerts flowing into my topic. It's still not clear to
> me however where the policy "output" is going. I also don't see any alerts
> in the UI window.
>
> Could I ask what the status of the project is in general? There have been
> no commits to master since November, so I'm not sure if there is any point
> in submitting Pull Requests for outstanding bugs? Are recent versions of
> Apache Eagle used in production?
>
> Colm.
>
> On Mon, Jan 22, 2018 at 1:07 PM, Colm O hEigeartaigh <co...@apache.org>
> wrote:
>
> >
> > I've done that but I'm not seeing any alerts, which is why I want to find
> > out what the "output" of a policy is and where I can check this.
> >
> > Colm.
> >
> > On Mon, Jan 22, 2018 at 1:05 PM, SUDHA JENSLIN <sj...@gmail.com>
> wrote:
> >
> >> Create and add a publisher to see the output.
> >>
> >>
> >>
> >> Regards,
> >> Sudha jenslin
> >>
> >> On Jan 22, 2018 6:31 PM, "Colm O hEigeartaigh" <co...@apache.org>
> >> wrote:
> >>
> >> Thanks - the error was due to a problem running Storm with Java 1.8.
> I've
> >> abandoned the docker image for now, and I'm trying to get it working
> >> locally.
> >>
> >> There are two things I'm not clear on currently, if someone could fill
> me
> >> in:
> >>
> >> a) For the  'Hdfs Audit Log Monitor' application, the Kafka Consumer
> Topic
> >> is 'hdfs_audit_log_sandbox'. Under 'Kafka Topic for Auditlog Event Sink'
> >> it
> >> also specifies 'hdfs_audit_event_sandbox'. However the documentation for
> >> the application mentions 'hdfs_audit_log_enriched_sandbox'?
> >>
> >> When I click on "STREAMS", the "HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX"
> >> uses the topic "hdfs_audit_event_sandbox". And indeed when I run the
> >> application, I can see cleansed log data appearing in
> >> "hdfs_audit_event_sandbox". So I'm thinking here that
> >> 'hdfs_audit_log_enriched_sandbox' is not correct or necessary?
> >>
> >> b) It's unclear to me where the output data goes when you create a
> policy.
> >> E.g. say I have:
> >>
> >> from HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX[str:contains(src,'/hbase')]
> >> select * group by user insert into hdfs_audit_log_enriched_stream_out
> >>
> >> Where is "hdfs_audit_log_enriched_stream_out" defined (is it a Kafka
> >> topic?). How can I check the output to make sure the policy is working
> >> correctly?
> >>
> >> Thanks,
> >>
> >> Colm.
> >>
> >> On Wed, Jan 17, 2018 at 10:32 PM, Edward Zhang <yo...@gmail.com>
> >> wrote:
> >>
> >> > There is a data preparation stage between data source(HDFS audit log)
> >> and
> >> > Alert Engine. This stage is running in Storm and transform the raw
> HDFS
> >> log
> >> > into something which can be alerted.
> >> >
> >> > The input for data preparation is hdfs_audit_log_sandbox topic and
> >> output
> >> > is
> >> >  hdfs_audit_log_enriched_sandbox.
> >> > The input for Alert Engine is hdfs_audit_log_enriched_sandbox and
> >> output
> >> > is
> >> > hdfs_audit_log_alert_sandbox.
> >> >
> >> > Seems in your case, the data preparation staging is not working. We
> >> > probably need look at Storm console and figure out if that part is
> >> working.
> >> >
> >> > Thanks
> >> > Edward
> >> >
> >> > On Wed, Jan 17, 2018 at 7:19 AM, Colm O hEigeartaigh <
> >> coheigea@apache.org>
> >> > wrote:
> >> >
> >> > > Hi Jayesh,
> >> > >
> >> > > Many thanks for your feedback! I was able to make a little further
> >> > headway.
> >> > > There are two configuration problems with the official docker image:
> >> > >
> >> > > a) A mix of "sandbox.eagle.apache.org" and "server.eagle.apache.org
> "
> >> > (this
> >> > > only occurs in the instructions for running the docker image. The
> >> version
> >> > > that can be started via the script in the eagle source is OK). I'll
> >> > submit
> >> > > a PR to fix this once I get a basic use-case working.
> >> > > b) For the audit case, it automatically logs HDFS audit logs to the
> >> KAFKA
> >> > > topic sandbox_hdfs_audit_log instead of the expected
> >> > hdfs_audit_log_sandbox
> >> > >
> >> > > I've fixed these things locally and I can verify that everything is
> >> > started
> >> > > correctly in Ambari. I log into the docker container and create
> >> > > hdfs_audit_log_sandbox and hdfs_audit_log_enriched_sandbox topics,
> >> and
> >> > > verify that the HDFS audit logs are flowing into the first topic.
> >> Then in
> >> > > the UI I start the Alert Engine and then the HDFS Audit Log Monitor
> >> > > application (changing localhost:6667 to
> server.eagle.apache.org:6667
> >> ).
> >> > > Both
> >> > > applications start up correctly and show "running".
> >> > >
> >> > > I then create a policy with an email alert along the lines of from
> >> > > "HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX[str:contains(src,'/hbase')]
> >> > select
> >> > > * group by user insert into hdfs_audit_log_enriched_stream_out".
> >> However
> >> > > at
> >> > > this point I'm stuck - nothing appears in the alert window. Is there
> >> > > anything obvious I'm doing wrong, or how can I get access to logs to
> >> > figure
> >> > > out what the problem is? Other topics such as
> >> "hdfs_audit_event_sandbox"
> >> > > are mentioned in the streams window, but the documentation doesn't
> >> say to
> >> > > create them.
> >> > >
> >> > > The UI is buggy though on both Firefox and Chromium on Linux. What
> >> > > browser/platform are people using with the UI?
> >> > >
> >> > > Colm.
> >> > >
> >> > > On Wed, Jan 17, 2018 at 12:27 AM, Jayesh Senjaliya <
> jaysen@apache.org
> >> >
> >> > > wrote:
> >> > >
> >> > > > Hi Colm,
> >> > > >
> >> > > > Please find my comments inline.
> >> > > >
> >> > > > a) The official docker image uses 0.5.0-SNAPSHOT and not the
> >> released
> >> > > > version.
> >> > > > - this is because we uploaded docker image before apache release.
> >> > > actually
> >> > > > this is same codebase apache-eagle-0.5, and it can be fixed easily
> >> by
> >> > > just
> >> > > > rebuilding docker image. there should not be any mismatch due to
> >> this.
> >> > > >
> >> > > > b) Aside from the above, the official docker image uses a mix of "
> >> > > > server.eagle.apache.org" and "sandbox.eagle.apache.org" as the
> host
> >> > > name.
> >> > > > The HBase service doesn't start by default in Ambari as a result.
> >> > > > - the only places it uses sandbox is in example script which you
> >> will
> >> > > have
> >> > > > to update anyway, which i agree that it would be good to keep it
> >> > > > consistent.
> >> > > >
> >> > > > c) The UI seems quite buggy. On both chromium and firefox, I only
> >> see
> >> > > > links to "Sandbox" and "Alert" on the left hand-side. Once I click
> >> on
> >> > > > "Alert" I have no way of going back to see the applications. I
> don't
> >> > see
> >> > > > the links to "integration" or "sites" as in the picture here:
> >> > > > http://eagle.apache.org/docs/latest/applications/#jmx-monitoring
> >> > > > - when hbase is as deep storage is used, and if eagle app has
> issue
> >> > > > connecting to hbase, the UI becomes unresponsive.
> >> > > >
> >> > > > d) In chromium, the button to create a new policy does not exist
> - I
> >> > can
> >> > > > only see it on Firefox.
> >> > > > - i have seen when you logged in, you will see admin actions. but
> if
> >> > this
> >> > > > still an issue, can you please file UI bug?
> >> > > >
> >> > > > e) I'm trying to get the "Hdfs Audit Log Monitor" use-case
> working,
> >> but
> >> > > it
> >> > > > seems to be stuck in "Initialized".
> >> > > > this eagle docs has example on how to setup the app. pls let us
> >> know if
> >> > > > you find any gaps.
> >> > > >
> >> > > > Thanks for trying out, and sharing your findings,
> >> > > > Jayesh
> >> > > >
> >> > > >
> >> > > > On Tue, Jan 16, 2018 at 3:34 AM, Colm O hEigeartaigh <
> >> > > coheigea@apache.org>
> >> > > > wrote:
> >> > > >
> >> > > >> Hi all,
> >> > > >>
> >> > > >> I'm trying to play around a bit with Apache Eagle 0.5.0 to no
> >> avail.
> >> > > Here
> >> > > >> are the problems I've run into so far:
> >> > > >>
> >> > > >> a) The official docker image uses 0.5.0-SNAPSHOT and not the
> >> released
> >> > > >> version.
> >> > > >>
> >> > > >> b) Aside from the above, the official docker image uses a mix of
> "
> >> > > >> server.eagle.apache.org" and "sandbox.eagle.apache.org" as the
> >> host
> >> > > >> name. The HBase service doesn't start by default in Ambari as a
> >> > result.
> >> > > >>
> >> > > >> c) The UI seems quite buggy. On both chromium and firefox, I only
> >> see
> >> > > >> links to "Sandbox" and "Alert" on the left hand-side. Once I
> click
> >> on
> >> > > >> "Alert" I have no way of going back to see the applications. I
> >> don't
> >> > see
> >> > > >> the links to "integration" or "sites" as in the picture here:
> >> > > >> http://eagle.apache.org/docs/latest/applications/#jmx-monitoring
> >> > > >>
> >> > > >> d) In chromium, the button to create a new policy does not exist
> -
> >> I
> >> > can
> >> > > >> only see it on Firefox.
> >> > > >>
> >> > > >> e) I'm trying to get the "Hdfs Audit Log Monitor" use-case
> working,
> >> > but
> >> > > >> it seems to be stuck in "Initialized".
> >> > > >>
> >> > > >> Could someone fill me in on what the "recommended" way is to
> start
> >> > > Apache
> >> > > >> Eagle so that I can play around with the functionality that it
> >> offers?
> >> > > >> Clearly the docker approach is buggy. Also, what browser should
> be
> >> > used?
> >> > > >>
> >> > > >> Thanks,
> >> > > >>
> >> > > >> Colm.
> >> > > >>
> >> > > >>
> >> > > >> --
> >> > > >> Colm O hEigeartaigh
> >> > > >>
> >> > > >> Talend Community Coder
> >> > > >> http://coders.talend.com
> >> > > >>
> >> > > >
> >> > > >
> >> > >
> >> > >
> >> > > --
> >> > > Colm O hEigeartaigh
> >> > >
> >> > > Talend Community Coder
> >> > > http://coders.talend.com
> >> > >
> >> >
> >>
> >>
> >>
> >> --
> >> Colm O hEigeartaigh
> >>
> >> Talend Community Coder
> >> http://coders.talend.com
> >>
> >>
> >>
> >
> >
> > --
> > Colm O hEigeartaigh
> >
> > Talend Community Coder
> > http://coders.talend.com
> >
>
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>

Re: Unable to get 0.5.0 release working

Posted by Edward Zhang <yo...@gmail.com>.
Eagle 0.5 was deployed in production as far as I know, but it may not be
exact the current version in master branch.

Thanks for your investigation, seems there is still some bug in 0.5, but
this particular issue seems is due to dependent components version conflict.

@Jayesh is this Jira ready for merge to master?
https://issues.apache.org/jira/browse/EAGLE-968


Thanks
Edward

On Tue, Jan 23, 2018 at 5:10 AM, Colm O hEigeartaigh <co...@apache.org>
wrote:

> OK I've made some more progress. I wasn't seeing any email alerts due to
> https://issues.apache.org/jira/browse/EAGLE-968. Once I configure a Kafka
> alert, I can see the alerts flowing into my topic. It's still not clear to
> me however where the policy "output" is going. I also don't see any alerts
> in the UI window.
>
> Could I ask what the status of the project is in general? There have been
> no commits to master since November, so I'm not sure if there is any point
> in submitting Pull Requests for outstanding bugs? Are recent versions of
> Apache Eagle used in production?
>
> Colm.
>
> On Mon, Jan 22, 2018 at 1:07 PM, Colm O hEigeartaigh <co...@apache.org>
> wrote:
>
> >
> > I've done that but I'm not seeing any alerts, which is why I want to find
> > out what the "output" of a policy is and where I can check this.
> >
> > Colm.
> >
> > On Mon, Jan 22, 2018 at 1:05 PM, SUDHA JENSLIN <sj...@gmail.com>
> wrote:
> >
> >> Create and add a publisher to see the output.
> >>
> >>
> >>
> >> Regards,
> >> Sudha jenslin
> >>
> >> On Jan 22, 2018 6:31 PM, "Colm O hEigeartaigh" <co...@apache.org>
> >> wrote:
> >>
> >> Thanks - the error was due to a problem running Storm with Java 1.8.
> I've
> >> abandoned the docker image for now, and I'm trying to get it working
> >> locally.
> >>
> >> There are two things I'm not clear on currently, if someone could fill
> me
> >> in:
> >>
> >> a) For the  'Hdfs Audit Log Monitor' application, the Kafka Consumer
> Topic
> >> is 'hdfs_audit_log_sandbox'. Under 'Kafka Topic for Auditlog Event Sink'
> >> it
> >> also specifies 'hdfs_audit_event_sandbox'. However the documentation for
> >> the application mentions 'hdfs_audit_log_enriched_sandbox'?
> >>
> >> When I click on "STREAMS", the "HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX"
> >> uses the topic "hdfs_audit_event_sandbox". And indeed when I run the
> >> application, I can see cleansed log data appearing in
> >> "hdfs_audit_event_sandbox". So I'm thinking here that
> >> 'hdfs_audit_log_enriched_sandbox' is not correct or necessary?
> >>
> >> b) It's unclear to me where the output data goes when you create a
> policy.
> >> E.g. say I have:
> >>
> >> from HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX[str:contains(src,'/hbase')]
> >> select * group by user insert into hdfs_audit_log_enriched_stream_out
> >>
> >> Where is "hdfs_audit_log_enriched_stream_out" defined (is it a Kafka
> >> topic?). How can I check the output to make sure the policy is working
> >> correctly?
> >>
> >> Thanks,
> >>
> >> Colm.
> >>
> >> On Wed, Jan 17, 2018 at 10:32 PM, Edward Zhang <yo...@gmail.com>
> >> wrote:
> >>
> >> > There is a data preparation stage between data source(HDFS audit log)
> >> and
> >> > Alert Engine. This stage is running in Storm and transform the raw
> HDFS
> >> log
> >> > into something which can be alerted.
> >> >
> >> > The input for data preparation is hdfs_audit_log_sandbox topic and
> >> output
> >> > is
> >> >  hdfs_audit_log_enriched_sandbox.
> >> > The input for Alert Engine is hdfs_audit_log_enriched_sandbox and
> >> output
> >> > is
> >> > hdfs_audit_log_alert_sandbox.
> >> >
> >> > Seems in your case, the data preparation staging is not working. We
> >> > probably need look at Storm console and figure out if that part is
> >> working.
> >> >
> >> > Thanks
> >> > Edward
> >> >
> >> > On Wed, Jan 17, 2018 at 7:19 AM, Colm O hEigeartaigh <
> >> coheigea@apache.org>
> >> > wrote:
> >> >
> >> > > Hi Jayesh,
> >> > >
> >> > > Many thanks for your feedback! I was able to make a little further
> >> > headway.
> >> > > There are two configuration problems with the official docker image:
> >> > >
> >> > > a) A mix of "sandbox.eagle.apache.org" and "server.eagle.apache.org
> "
> >> > (this
> >> > > only occurs in the instructions for running the docker image. The
> >> version
> >> > > that can be started via the script in the eagle source is OK). I'll
> >> > submit
> >> > > a PR to fix this once I get a basic use-case working.
> >> > > b) For the audit case, it automatically logs HDFS audit logs to the
> >> KAFKA
> >> > > topic sandbox_hdfs_audit_log instead of the expected
> >> > hdfs_audit_log_sandbox
> >> > >
> >> > > I've fixed these things locally and I can verify that everything is
> >> > started
> >> > > correctly in Ambari. I log into the docker container and create
> >> > > hdfs_audit_log_sandbox and hdfs_audit_log_enriched_sandbox topics,
> >> and
> >> > > verify that the HDFS audit logs are flowing into the first topic.
> >> Then in
> >> > > the UI I start the Alert Engine and then the HDFS Audit Log Monitor
> >> > > application (changing localhost:6667 to
> server.eagle.apache.org:6667
> >> ).
> >> > > Both
> >> > > applications start up correctly and show "running".
> >> > >
> >> > > I then create a policy with an email alert along the lines of from
> >> > > "HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX[str:contains(src,'/hbase')]
> >> > select
> >> > > * group by user insert into hdfs_audit_log_enriched_stream_out".
> >> However
> >> > > at
> >> > > this point I'm stuck - nothing appears in the alert window. Is there
> >> > > anything obvious I'm doing wrong, or how can I get access to logs to
> >> > figure
> >> > > out what the problem is? Other topics such as
> >> "hdfs_audit_event_sandbox"
> >> > > are mentioned in the streams window, but the documentation doesn't
> >> say to
> >> > > create them.
> >> > >
> >> > > The UI is buggy though on both Firefox and Chromium on Linux. What
> >> > > browser/platform are people using with the UI?
> >> > >
> >> > > Colm.
> >> > >
> >> > > On Wed, Jan 17, 2018 at 12:27 AM, Jayesh Senjaliya <
> jaysen@apache.org
> >> >
> >> > > wrote:
> >> > >
> >> > > > Hi Colm,
> >> > > >
> >> > > > Please find my comments inline.
> >> > > >
> >> > > > a) The official docker image uses 0.5.0-SNAPSHOT and not the
> >> released
> >> > > > version.
> >> > > > - this is because we uploaded docker image before apache release.
> >> > > actually
> >> > > > this is same codebase apache-eagle-0.5, and it can be fixed easily
> >> by
> >> > > just
> >> > > > rebuilding docker image. there should not be any mismatch due to
> >> this.
> >> > > >
> >> > > > b) Aside from the above, the official docker image uses a mix of "
> >> > > > server.eagle.apache.org" and "sandbox.eagle.apache.org" as the
> host
> >> > > name.
> >> > > > The HBase service doesn't start by default in Ambari as a result.
> >> > > > - the only places it uses sandbox is in example script which you
> >> will
> >> > > have
> >> > > > to update anyway, which i agree that it would be good to keep it
> >> > > > consistent.
> >> > > >
> >> > > > c) The UI seems quite buggy. On both chromium and firefox, I only
> >> see
> >> > > > links to "Sandbox" and "Alert" on the left hand-side. Once I click
> >> on
> >> > > > "Alert" I have no way of going back to see the applications. I
> don't
> >> > see
> >> > > > the links to "integration" or "sites" as in the picture here:
> >> > > > http://eagle.apache.org/docs/latest/applications/#jmx-monitoring
> >> > > > - when hbase is as deep storage is used, and if eagle app has
> issue
> >> > > > connecting to hbase, the UI becomes unresponsive.
> >> > > >
> >> > > > d) In chromium, the button to create a new policy does not exist
> - I
> >> > can
> >> > > > only see it on Firefox.
> >> > > > - i have seen when you logged in, you will see admin actions. but
> if
> >> > this
> >> > > > still an issue, can you please file UI bug?
> >> > > >
> >> > > > e) I'm trying to get the "Hdfs Audit Log Monitor" use-case
> working,
> >> but
> >> > > it
> >> > > > seems to be stuck in "Initialized".
> >> > > > this eagle docs has example on how to setup the app. pls let us
> >> know if
> >> > > > you find any gaps.
> >> > > >
> >> > > > Thanks for trying out, and sharing your findings,
> >> > > > Jayesh
> >> > > >
> >> > > >
> >> > > > On Tue, Jan 16, 2018 at 3:34 AM, Colm O hEigeartaigh <
> >> > > coheigea@apache.org>
> >> > > > wrote:
> >> > > >
> >> > > >> Hi all,
> >> > > >>
> >> > > >> I'm trying to play around a bit with Apache Eagle 0.5.0 to no
> >> avail.
> >> > > Here
> >> > > >> are the problems I've run into so far:
> >> > > >>
> >> > > >> a) The official docker image uses 0.5.0-SNAPSHOT and not the
> >> released
> >> > > >> version.
> >> > > >>
> >> > > >> b) Aside from the above, the official docker image uses a mix of
> "
> >> > > >> server.eagle.apache.org" and "sandbox.eagle.apache.org" as the
> >> host
> >> > > >> name. The HBase service doesn't start by default in Ambari as a
> >> > result.
> >> > > >>
> >> > > >> c) The UI seems quite buggy. On both chromium and firefox, I only
> >> see
> >> > > >> links to "Sandbox" and "Alert" on the left hand-side. Once I
> click
> >> on
> >> > > >> "Alert" I have no way of going back to see the applications. I
> >> don't
> >> > see
> >> > > >> the links to "integration" or "sites" as in the picture here:
> >> > > >> http://eagle.apache.org/docs/latest/applications/#jmx-monitoring
> >> > > >>
> >> > > >> d) In chromium, the button to create a new policy does not exist
> -
> >> I
> >> > can
> >> > > >> only see it on Firefox.
> >> > > >>
> >> > > >> e) I'm trying to get the "Hdfs Audit Log Monitor" use-case
> working,
> >> > but
> >> > > >> it seems to be stuck in "Initialized".
> >> > > >>
> >> > > >> Could someone fill me in on what the "recommended" way is to
> start
> >> > > Apache
> >> > > >> Eagle so that I can play around with the functionality that it
> >> offers?
> >> > > >> Clearly the docker approach is buggy. Also, what browser should
> be
> >> > used?
> >> > > >>
> >> > > >> Thanks,
> >> > > >>
> >> > > >> Colm.
> >> > > >>
> >> > > >>
> >> > > >> --
> >> > > >> Colm O hEigeartaigh
> >> > > >>
> >> > > >> Talend Community Coder
> >> > > >> http://coders.talend.com
> >> > > >>
> >> > > >
> >> > > >
> >> > >
> >> > >
> >> > > --
> >> > > Colm O hEigeartaigh
> >> > >
> >> > > Talend Community Coder
> >> > > http://coders.talend.com
> >> > >
> >> >
> >>
> >>
> >>
> >> --
> >> Colm O hEigeartaigh
> >>
> >> Talend Community Coder
> >> http://coders.talend.com
> >>
> >>
> >>
> >
> >
> > --
> > Colm O hEigeartaigh
> >
> > Talend Community Coder
> > http://coders.talend.com
> >
>
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>

Re: Unable to get 0.5.0 release working

Posted by Colm O hEigeartaigh <co...@apache.org>.
OK I've made some more progress. I wasn't seeing any email alerts due to
https://issues.apache.org/jira/browse/EAGLE-968. Once I configure a Kafka
alert, I can see the alerts flowing into my topic. It's still not clear to
me however where the policy "output" is going. I also don't see any alerts
in the UI window.

Could I ask what the status of the project is in general? There have been
no commits to master since November, so I'm not sure if there is any point
in submitting Pull Requests for outstanding bugs? Are recent versions of
Apache Eagle used in production?

Colm.

On Mon, Jan 22, 2018 at 1:07 PM, Colm O hEigeartaigh <co...@apache.org>
wrote:

>
> I've done that but I'm not seeing any alerts, which is why I want to find
> out what the "output" of a policy is and where I can check this.
>
> Colm.
>
> On Mon, Jan 22, 2018 at 1:05 PM, SUDHA JENSLIN <sj...@gmail.com> wrote:
>
>> Create and add a publisher to see the output.
>>
>>
>>
>> Regards,
>> Sudha jenslin
>>
>> On Jan 22, 2018 6:31 PM, "Colm O hEigeartaigh" <co...@apache.org>
>> wrote:
>>
>> Thanks - the error was due to a problem running Storm with Java 1.8. I've
>> abandoned the docker image for now, and I'm trying to get it working
>> locally.
>>
>> There are two things I'm not clear on currently, if someone could fill me
>> in:
>>
>> a) For the  'Hdfs Audit Log Monitor' application, the Kafka Consumer Topic
>> is 'hdfs_audit_log_sandbox'. Under 'Kafka Topic for Auditlog Event Sink'
>> it
>> also specifies 'hdfs_audit_event_sandbox'. However the documentation for
>> the application mentions 'hdfs_audit_log_enriched_sandbox'?
>>
>> When I click on "STREAMS", the "HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX"
>> uses the topic "hdfs_audit_event_sandbox". And indeed when I run the
>> application, I can see cleansed log data appearing in
>> "hdfs_audit_event_sandbox". So I'm thinking here that
>> 'hdfs_audit_log_enriched_sandbox' is not correct or necessary?
>>
>> b) It's unclear to me where the output data goes when you create a policy.
>> E.g. say I have:
>>
>> from HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX[str:contains(src,'/hbase')]
>> select * group by user insert into hdfs_audit_log_enriched_stream_out
>>
>> Where is "hdfs_audit_log_enriched_stream_out" defined (is it a Kafka
>> topic?). How can I check the output to make sure the policy is working
>> correctly?
>>
>> Thanks,
>>
>> Colm.
>>
>> On Wed, Jan 17, 2018 at 10:32 PM, Edward Zhang <yo...@gmail.com>
>> wrote:
>>
>> > There is a data preparation stage between data source(HDFS audit log)
>> and
>> > Alert Engine. This stage is running in Storm and transform the raw HDFS
>> log
>> > into something which can be alerted.
>> >
>> > The input for data preparation is hdfs_audit_log_sandbox topic and
>> output
>> > is
>> >  hdfs_audit_log_enriched_sandbox.
>> > The input for Alert Engine is hdfs_audit_log_enriched_sandbox and
>> output
>> > is
>> > hdfs_audit_log_alert_sandbox.
>> >
>> > Seems in your case, the data preparation staging is not working. We
>> > probably need look at Storm console and figure out if that part is
>> working.
>> >
>> > Thanks
>> > Edward
>> >
>> > On Wed, Jan 17, 2018 at 7:19 AM, Colm O hEigeartaigh <
>> coheigea@apache.org>
>> > wrote:
>> >
>> > > Hi Jayesh,
>> > >
>> > > Many thanks for your feedback! I was able to make a little further
>> > headway.
>> > > There are two configuration problems with the official docker image:
>> > >
>> > > a) A mix of "sandbox.eagle.apache.org" and "server.eagle.apache.org"
>> > (this
>> > > only occurs in the instructions for running the docker image. The
>> version
>> > > that can be started via the script in the eagle source is OK). I'll
>> > submit
>> > > a PR to fix this once I get a basic use-case working.
>> > > b) For the audit case, it automatically logs HDFS audit logs to the
>> KAFKA
>> > > topic sandbox_hdfs_audit_log instead of the expected
>> > hdfs_audit_log_sandbox
>> > >
>> > > I've fixed these things locally and I can verify that everything is
>> > started
>> > > correctly in Ambari. I log into the docker container and create
>> > > hdfs_audit_log_sandbox and hdfs_audit_log_enriched_sandbox topics,
>> and
>> > > verify that the HDFS audit logs are flowing into the first topic.
>> Then in
>> > > the UI I start the Alert Engine and then the HDFS Audit Log Monitor
>> > > application (changing localhost:6667 to server.eagle.apache.org:6667
>> ).
>> > > Both
>> > > applications start up correctly and show "running".
>> > >
>> > > I then create a policy with an email alert along the lines of from
>> > > "HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX[str:contains(src,'/hbase')]
>> > select
>> > > * group by user insert into hdfs_audit_log_enriched_stream_out".
>> However
>> > > at
>> > > this point I'm stuck - nothing appears in the alert window. Is there
>> > > anything obvious I'm doing wrong, or how can I get access to logs to
>> > figure
>> > > out what the problem is? Other topics such as
>> "hdfs_audit_event_sandbox"
>> > > are mentioned in the streams window, but the documentation doesn't
>> say to
>> > > create them.
>> > >
>> > > The UI is buggy though on both Firefox and Chromium on Linux. What
>> > > browser/platform are people using with the UI?
>> > >
>> > > Colm.
>> > >
>> > > On Wed, Jan 17, 2018 at 12:27 AM, Jayesh Senjaliya <jaysen@apache.org
>> >
>> > > wrote:
>> > >
>> > > > Hi Colm,
>> > > >
>> > > > Please find my comments inline.
>> > > >
>> > > > a) The official docker image uses 0.5.0-SNAPSHOT and not the
>> released
>> > > > version.
>> > > > - this is because we uploaded docker image before apache release.
>> > > actually
>> > > > this is same codebase apache-eagle-0.5, and it can be fixed easily
>> by
>> > > just
>> > > > rebuilding docker image. there should not be any mismatch due to
>> this.
>> > > >
>> > > > b) Aside from the above, the official docker image uses a mix of "
>> > > > server.eagle.apache.org" and "sandbox.eagle.apache.org" as the host
>> > > name.
>> > > > The HBase service doesn't start by default in Ambari as a result.
>> > > > - the only places it uses sandbox is in example script which you
>> will
>> > > have
>> > > > to update anyway, which i agree that it would be good to keep it
>> > > > consistent.
>> > > >
>> > > > c) The UI seems quite buggy. On both chromium and firefox, I only
>> see
>> > > > links to "Sandbox" and "Alert" on the left hand-side. Once I click
>> on
>> > > > "Alert" I have no way of going back to see the applications. I don't
>> > see
>> > > > the links to "integration" or "sites" as in the picture here:
>> > > > http://eagle.apache.org/docs/latest/applications/#jmx-monitoring
>> > > > - when hbase is as deep storage is used, and if eagle app has issue
>> > > > connecting to hbase, the UI becomes unresponsive.
>> > > >
>> > > > d) In chromium, the button to create a new policy does not exist - I
>> > can
>> > > > only see it on Firefox.
>> > > > - i have seen when you logged in, you will see admin actions. but if
>> > this
>> > > > still an issue, can you please file UI bug?
>> > > >
>> > > > e) I'm trying to get the "Hdfs Audit Log Monitor" use-case working,
>> but
>> > > it
>> > > > seems to be stuck in "Initialized".
>> > > > this eagle docs has example on how to setup the app. pls let us
>> know if
>> > > > you find any gaps.
>> > > >
>> > > > Thanks for trying out, and sharing your findings,
>> > > > Jayesh
>> > > >
>> > > >
>> > > > On Tue, Jan 16, 2018 at 3:34 AM, Colm O hEigeartaigh <
>> > > coheigea@apache.org>
>> > > > wrote:
>> > > >
>> > > >> Hi all,
>> > > >>
>> > > >> I'm trying to play around a bit with Apache Eagle 0.5.0 to no
>> avail.
>> > > Here
>> > > >> are the problems I've run into so far:
>> > > >>
>> > > >> a) The official docker image uses 0.5.0-SNAPSHOT and not the
>> released
>> > > >> version.
>> > > >>
>> > > >> b) Aside from the above, the official docker image uses a mix of "
>> > > >> server.eagle.apache.org" and "sandbox.eagle.apache.org" as the
>> host
>> > > >> name. The HBase service doesn't start by default in Ambari as a
>> > result.
>> > > >>
>> > > >> c) The UI seems quite buggy. On both chromium and firefox, I only
>> see
>> > > >> links to "Sandbox" and "Alert" on the left hand-side. Once I click
>> on
>> > > >> "Alert" I have no way of going back to see the applications. I
>> don't
>> > see
>> > > >> the links to "integration" or "sites" as in the picture here:
>> > > >> http://eagle.apache.org/docs/latest/applications/#jmx-monitoring
>> > > >>
>> > > >> d) In chromium, the button to create a new policy does not exist -
>> I
>> > can
>> > > >> only see it on Firefox.
>> > > >>
>> > > >> e) I'm trying to get the "Hdfs Audit Log Monitor" use-case working,
>> > but
>> > > >> it seems to be stuck in "Initialized".
>> > > >>
>> > > >> Could someone fill me in on what the "recommended" way is to start
>> > > Apache
>> > > >> Eagle so that I can play around with the functionality that it
>> offers?
>> > > >> Clearly the docker approach is buggy. Also, what browser should be
>> > used?
>> > > >>
>> > > >> Thanks,
>> > > >>
>> > > >> Colm.
>> > > >>
>> > > >>
>> > > >> --
>> > > >> Colm O hEigeartaigh
>> > > >>
>> > > >> Talend Community Coder
>> > > >> http://coders.talend.com
>> > > >>
>> > > >
>> > > >
>> > >
>> > >
>> > > --
>> > > Colm O hEigeartaigh
>> > >
>> > > Talend Community Coder
>> > > http://coders.talend.com
>> > >
>> >
>>
>>
>>
>> --
>> Colm O hEigeartaigh
>>
>> Talend Community Coder
>> http://coders.talend.com
>>
>>
>>
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>



-- 
Colm O hEigeartaigh

Talend Community Coder
http://coders.talend.com

Re: Unable to get 0.5.0 release working

Posted by Colm O hEigeartaigh <co...@apache.org>.
OK I've made some more progress. I wasn't seeing any email alerts due to
https://issues.apache.org/jira/browse/EAGLE-968. Once I configure a Kafka
alert, I can see the alerts flowing into my topic. It's still not clear to
me however where the policy "output" is going. I also don't see any alerts
in the UI window.

Could I ask what the status of the project is in general? There have been
no commits to master since November, so I'm not sure if there is any point
in submitting Pull Requests for outstanding bugs? Are recent versions of
Apache Eagle used in production?

Colm.

On Mon, Jan 22, 2018 at 1:07 PM, Colm O hEigeartaigh <co...@apache.org>
wrote:

>
> I've done that but I'm not seeing any alerts, which is why I want to find
> out what the "output" of a policy is and where I can check this.
>
> Colm.
>
> On Mon, Jan 22, 2018 at 1:05 PM, SUDHA JENSLIN <sj...@gmail.com> wrote:
>
>> Create and add a publisher to see the output.
>>
>>
>>
>> Regards,
>> Sudha jenslin
>>
>> On Jan 22, 2018 6:31 PM, "Colm O hEigeartaigh" <co...@apache.org>
>> wrote:
>>
>> Thanks - the error was due to a problem running Storm with Java 1.8. I've
>> abandoned the docker image for now, and I'm trying to get it working
>> locally.
>>
>> There are two things I'm not clear on currently, if someone could fill me
>> in:
>>
>> a) For the  'Hdfs Audit Log Monitor' application, the Kafka Consumer Topic
>> is 'hdfs_audit_log_sandbox'. Under 'Kafka Topic for Auditlog Event Sink'
>> it
>> also specifies 'hdfs_audit_event_sandbox'. However the documentation for
>> the application mentions 'hdfs_audit_log_enriched_sandbox'?
>>
>> When I click on "STREAMS", the "HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX"
>> uses the topic "hdfs_audit_event_sandbox". And indeed when I run the
>> application, I can see cleansed log data appearing in
>> "hdfs_audit_event_sandbox". So I'm thinking here that
>> 'hdfs_audit_log_enriched_sandbox' is not correct or necessary?
>>
>> b) It's unclear to me where the output data goes when you create a policy.
>> E.g. say I have:
>>
>> from HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX[str:contains(src,'/hbase')]
>> select * group by user insert into hdfs_audit_log_enriched_stream_out
>>
>> Where is "hdfs_audit_log_enriched_stream_out" defined (is it a Kafka
>> topic?). How can I check the output to make sure the policy is working
>> correctly?
>>
>> Thanks,
>>
>> Colm.
>>
>> On Wed, Jan 17, 2018 at 10:32 PM, Edward Zhang <yo...@gmail.com>
>> wrote:
>>
>> > There is a data preparation stage between data source(HDFS audit log)
>> and
>> > Alert Engine. This stage is running in Storm and transform the raw HDFS
>> log
>> > into something which can be alerted.
>> >
>> > The input for data preparation is hdfs_audit_log_sandbox topic and
>> output
>> > is
>> >  hdfs_audit_log_enriched_sandbox.
>> > The input for Alert Engine is hdfs_audit_log_enriched_sandbox and
>> output
>> > is
>> > hdfs_audit_log_alert_sandbox.
>> >
>> > Seems in your case, the data preparation staging is not working. We
>> > probably need look at Storm console and figure out if that part is
>> working.
>> >
>> > Thanks
>> > Edward
>> >
>> > On Wed, Jan 17, 2018 at 7:19 AM, Colm O hEigeartaigh <
>> coheigea@apache.org>
>> > wrote:
>> >
>> > > Hi Jayesh,
>> > >
>> > > Many thanks for your feedback! I was able to make a little further
>> > headway.
>> > > There are two configuration problems with the official docker image:
>> > >
>> > > a) A mix of "sandbox.eagle.apache.org" and "server.eagle.apache.org"
>> > (this
>> > > only occurs in the instructions for running the docker image. The
>> version
>> > > that can be started via the script in the eagle source is OK). I'll
>> > submit
>> > > a PR to fix this once I get a basic use-case working.
>> > > b) For the audit case, it automatically logs HDFS audit logs to the
>> KAFKA
>> > > topic sandbox_hdfs_audit_log instead of the expected
>> > hdfs_audit_log_sandbox
>> > >
>> > > I've fixed these things locally and I can verify that everything is
>> > started
>> > > correctly in Ambari. I log into the docker container and create
>> > > hdfs_audit_log_sandbox and hdfs_audit_log_enriched_sandbox topics,
>> and
>> > > verify that the HDFS audit logs are flowing into the first topic.
>> Then in
>> > > the UI I start the Alert Engine and then the HDFS Audit Log Monitor
>> > > application (changing localhost:6667 to server.eagle.apache.org:6667
>> ).
>> > > Both
>> > > applications start up correctly and show "running".
>> > >
>> > > I then create a policy with an email alert along the lines of from
>> > > "HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX[str:contains(src,'/hbase')]
>> > select
>> > > * group by user insert into hdfs_audit_log_enriched_stream_out".
>> However
>> > > at
>> > > this point I'm stuck - nothing appears in the alert window. Is there
>> > > anything obvious I'm doing wrong, or how can I get access to logs to
>> > figure
>> > > out what the problem is? Other topics such as
>> "hdfs_audit_event_sandbox"
>> > > are mentioned in the streams window, but the documentation doesn't
>> say to
>> > > create them.
>> > >
>> > > The UI is buggy though on both Firefox and Chromium on Linux. What
>> > > browser/platform are people using with the UI?
>> > >
>> > > Colm.
>> > >
>> > > On Wed, Jan 17, 2018 at 12:27 AM, Jayesh Senjaliya <jaysen@apache.org
>> >
>> > > wrote:
>> > >
>> > > > Hi Colm,
>> > > >
>> > > > Please find my comments inline.
>> > > >
>> > > > a) The official docker image uses 0.5.0-SNAPSHOT and not the
>> released
>> > > > version.
>> > > > - this is because we uploaded docker image before apache release.
>> > > actually
>> > > > this is same codebase apache-eagle-0.5, and it can be fixed easily
>> by
>> > > just
>> > > > rebuilding docker image. there should not be any mismatch due to
>> this.
>> > > >
>> > > > b) Aside from the above, the official docker image uses a mix of "
>> > > > server.eagle.apache.org" and "sandbox.eagle.apache.org" as the host
>> > > name.
>> > > > The HBase service doesn't start by default in Ambari as a result.
>> > > > - the only places it uses sandbox is in example script which you
>> will
>> > > have
>> > > > to update anyway, which i agree that it would be good to keep it
>> > > > consistent.
>> > > >
>> > > > c) The UI seems quite buggy. On both chromium and firefox, I only
>> see
>> > > > links to "Sandbox" and "Alert" on the left hand-side. Once I click
>> on
>> > > > "Alert" I have no way of going back to see the applications. I don't
>> > see
>> > > > the links to "integration" or "sites" as in the picture here:
>> > > > http://eagle.apache.org/docs/latest/applications/#jmx-monitoring
>> > > > - when hbase is as deep storage is used, and if eagle app has issue
>> > > > connecting to hbase, the UI becomes unresponsive.
>> > > >
>> > > > d) In chromium, the button to create a new policy does not exist - I
>> > can
>> > > > only see it on Firefox.
>> > > > - i have seen when you logged in, you will see admin actions. but if
>> > this
>> > > > still an issue, can you please file UI bug?
>> > > >
>> > > > e) I'm trying to get the "Hdfs Audit Log Monitor" use-case working,
>> but
>> > > it
>> > > > seems to be stuck in "Initialized".
>> > > > this eagle docs has example on how to setup the app. pls let us
>> know if
>> > > > you find any gaps.
>> > > >
>> > > > Thanks for trying out, and sharing your findings,
>> > > > Jayesh
>> > > >
>> > > >
>> > > > On Tue, Jan 16, 2018 at 3:34 AM, Colm O hEigeartaigh <
>> > > coheigea@apache.org>
>> > > > wrote:
>> > > >
>> > > >> Hi all,
>> > > >>
>> > > >> I'm trying to play around a bit with Apache Eagle 0.5.0 to no
>> avail.
>> > > Here
>> > > >> are the problems I've run into so far:
>> > > >>
>> > > >> a) The official docker image uses 0.5.0-SNAPSHOT and not the
>> released
>> > > >> version.
>> > > >>
>> > > >> b) Aside from the above, the official docker image uses a mix of "
>> > > >> server.eagle.apache.org" and "sandbox.eagle.apache.org" as the
>> host
>> > > >> name. The HBase service doesn't start by default in Ambari as a
>> > result.
>> > > >>
>> > > >> c) The UI seems quite buggy. On both chromium and firefox, I only
>> see
>> > > >> links to "Sandbox" and "Alert" on the left hand-side. Once I click
>> on
>> > > >> "Alert" I have no way of going back to see the applications. I
>> don't
>> > see
>> > > >> the links to "integration" or "sites" as in the picture here:
>> > > >> http://eagle.apache.org/docs/latest/applications/#jmx-monitoring
>> > > >>
>> > > >> d) In chromium, the button to create a new policy does not exist -
>> I
>> > can
>> > > >> only see it on Firefox.
>> > > >>
>> > > >> e) I'm trying to get the "Hdfs Audit Log Monitor" use-case working,
>> > but
>> > > >> it seems to be stuck in "Initialized".
>> > > >>
>> > > >> Could someone fill me in on what the "recommended" way is to start
>> > > Apache
>> > > >> Eagle so that I can play around with the functionality that it
>> offers?
>> > > >> Clearly the docker approach is buggy. Also, what browser should be
>> > used?
>> > > >>
>> > > >> Thanks,
>> > > >>
>> > > >> Colm.
>> > > >>
>> > > >>
>> > > >> --
>> > > >> Colm O hEigeartaigh
>> > > >>
>> > > >> Talend Community Coder
>> > > >> http://coders.talend.com
>> > > >>
>> > > >
>> > > >
>> > >
>> > >
>> > > --
>> > > Colm O hEigeartaigh
>> > >
>> > > Talend Community Coder
>> > > http://coders.talend.com
>> > >
>> >
>>
>>
>>
>> --
>> Colm O hEigeartaigh
>>
>> Talend Community Coder
>> http://coders.talend.com
>>
>>
>>
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>



-- 
Colm O hEigeartaigh

Talend Community Coder
http://coders.talend.com

Re: Unable to get 0.5.0 release working

Posted by Colm O hEigeartaigh <co...@apache.org>.
I've done that but I'm not seeing any alerts, which is why I want to find
out what the "output" of a policy is and where I can check this.

Colm.

On Mon, Jan 22, 2018 at 1:05 PM, SUDHA JENSLIN <sj...@gmail.com> wrote:

> Create and add a publisher to see the output.
>
>
>
> Regards,
> Sudha jenslin
>
> On Jan 22, 2018 6:31 PM, "Colm O hEigeartaigh" <co...@apache.org>
> wrote:
>
> Thanks - the error was due to a problem running Storm with Java 1.8. I've
> abandoned the docker image for now, and I'm trying to get it working
> locally.
>
> There are two things I'm not clear on currently, if someone could fill me
> in:
>
> a) For the  'Hdfs Audit Log Monitor' application, the Kafka Consumer Topic
> is 'hdfs_audit_log_sandbox'. Under 'Kafka Topic for Auditlog Event Sink' it
> also specifies 'hdfs_audit_event_sandbox'. However the documentation for
> the application mentions 'hdfs_audit_log_enriched_sandbox'?
>
> When I click on "STREAMS", the "HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX"
> uses the topic "hdfs_audit_event_sandbox". And indeed when I run the
> application, I can see cleansed log data appearing in
> "hdfs_audit_event_sandbox". So I'm thinking here that
> 'hdfs_audit_log_enriched_sandbox' is not correct or necessary?
>
> b) It's unclear to me where the output data goes when you create a policy.
> E.g. say I have:
>
> from HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX[str:contains(src,'/hbase')]
> select * group by user insert into hdfs_audit_log_enriched_stream_out
>
> Where is "hdfs_audit_log_enriched_stream_out" defined (is it a Kafka
> topic?). How can I check the output to make sure the policy is working
> correctly?
>
> Thanks,
>
> Colm.
>
> On Wed, Jan 17, 2018 at 10:32 PM, Edward Zhang <yo...@gmail.com>
> wrote:
>
> > There is a data preparation stage between data source(HDFS audit log) and
> > Alert Engine. This stage is running in Storm and transform the raw HDFS
> log
> > into something which can be alerted.
> >
> > The input for data preparation is hdfs_audit_log_sandbox topic and output
> > is
> >  hdfs_audit_log_enriched_sandbox.
> > The input for Alert Engine is hdfs_audit_log_enriched_sandbox and output
> > is
> > hdfs_audit_log_alert_sandbox.
> >
> > Seems in your case, the data preparation staging is not working. We
> > probably need look at Storm console and figure out if that part is
> working.
> >
> > Thanks
> > Edward
> >
> > On Wed, Jan 17, 2018 at 7:19 AM, Colm O hEigeartaigh <
> coheigea@apache.org>
> > wrote:
> >
> > > Hi Jayesh,
> > >
> > > Many thanks for your feedback! I was able to make a little further
> > headway.
> > > There are two configuration problems with the official docker image:
> > >
> > > a) A mix of "sandbox.eagle.apache.org" and "server.eagle.apache.org"
> > (this
> > > only occurs in the instructions for running the docker image. The
> version
> > > that can be started via the script in the eagle source is OK). I'll
> > submit
> > > a PR to fix this once I get a basic use-case working.
> > > b) For the audit case, it automatically logs HDFS audit logs to the
> KAFKA
> > > topic sandbox_hdfs_audit_log instead of the expected
> > hdfs_audit_log_sandbox
> > >
> > > I've fixed these things locally and I can verify that everything is
> > started
> > > correctly in Ambari. I log into the docker container and create
> > > hdfs_audit_log_sandbox and hdfs_audit_log_enriched_sandbox topics, and
> > > verify that the HDFS audit logs are flowing into the first topic. Then
> in
> > > the UI I start the Alert Engine and then the HDFS Audit Log Monitor
> > > application (changing localhost:6667 to server.eagle.apache.org:6667).
> > > Both
> > > applications start up correctly and show "running".
> > >
> > > I then create a policy with an email alert along the lines of from
> > > "HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX[str:contains(src,'/hbase')]
> > select
> > > * group by user insert into hdfs_audit_log_enriched_stream_out".
> However
> > > at
> > > this point I'm stuck - nothing appears in the alert window. Is there
> > > anything obvious I'm doing wrong, or how can I get access to logs to
> > figure
> > > out what the problem is? Other topics such as
> "hdfs_audit_event_sandbox"
> > > are mentioned in the streams window, but the documentation doesn't say
> to
> > > create them.
> > >
> > > The UI is buggy though on both Firefox and Chromium on Linux. What
> > > browser/platform are people using with the UI?
> > >
> > > Colm.
> > >
> > > On Wed, Jan 17, 2018 at 12:27 AM, Jayesh Senjaliya <ja...@apache.org>
> > > wrote:
> > >
> > > > Hi Colm,
> > > >
> > > > Please find my comments inline.
> > > >
> > > > a) The official docker image uses 0.5.0-SNAPSHOT and not the released
> > > > version.
> > > > - this is because we uploaded docker image before apache release.
> > > actually
> > > > this is same codebase apache-eagle-0.5, and it can be fixed easily by
> > > just
> > > > rebuilding docker image. there should not be any mismatch due to
> this.
> > > >
> > > > b) Aside from the above, the official docker image uses a mix of "
> > > > server.eagle.apache.org" and "sandbox.eagle.apache.org" as the host
> > > name.
> > > > The HBase service doesn't start by default in Ambari as a result.
> > > > - the only places it uses sandbox is in example script which you will
> > > have
> > > > to update anyway, which i agree that it would be good to keep it
> > > > consistent.
> > > >
> > > > c) The UI seems quite buggy. On both chromium and firefox, I only see
> > > > links to "Sandbox" and "Alert" on the left hand-side. Once I click on
> > > > "Alert" I have no way of going back to see the applications. I don't
> > see
> > > > the links to "integration" or "sites" as in the picture here:
> > > > http://eagle.apache.org/docs/latest/applications/#jmx-monitoring
> > > > - when hbase is as deep storage is used, and if eagle app has issue
> > > > connecting to hbase, the UI becomes unresponsive.
> > > >
> > > > d) In chromium, the button to create a new policy does not exist - I
> > can
> > > > only see it on Firefox.
> > > > - i have seen when you logged in, you will see admin actions. but if
> > this
> > > > still an issue, can you please file UI bug?
> > > >
> > > > e) I'm trying to get the "Hdfs Audit Log Monitor" use-case working,
> but
> > > it
> > > > seems to be stuck in "Initialized".
> > > > this eagle docs has example on how to setup the app. pls let us know
> if
> > > > you find any gaps.
> > > >
> > > > Thanks for trying out, and sharing your findings,
> > > > Jayesh
> > > >
> > > >
> > > > On Tue, Jan 16, 2018 at 3:34 AM, Colm O hEigeartaigh <
> > > coheigea@apache.org>
> > > > wrote:
> > > >
> > > >> Hi all,
> > > >>
> > > >> I'm trying to play around a bit with Apache Eagle 0.5.0 to no avail.
> > > Here
> > > >> are the problems I've run into so far:
> > > >>
> > > >> a) The official docker image uses 0.5.0-SNAPSHOT and not the
> released
> > > >> version.
> > > >>
> > > >> b) Aside from the above, the official docker image uses a mix of "
> > > >> server.eagle.apache.org" and "sandbox.eagle.apache.org" as the host
> > > >> name. The HBase service doesn't start by default in Ambari as a
> > result.
> > > >>
> > > >> c) The UI seems quite buggy. On both chromium and firefox, I only
> see
> > > >> links to "Sandbox" and "Alert" on the left hand-side. Once I click
> on
> > > >> "Alert" I have no way of going back to see the applications. I don't
> > see
> > > >> the links to "integration" or "sites" as in the picture here:
> > > >> http://eagle.apache.org/docs/latest/applications/#jmx-monitoring
> > > >>
> > > >> d) In chromium, the button to create a new policy does not exist - I
> > can
> > > >> only see it on Firefox.
> > > >>
> > > >> e) I'm trying to get the "Hdfs Audit Log Monitor" use-case working,
> > but
> > > >> it seems to be stuck in "Initialized".
> > > >>
> > > >> Could someone fill me in on what the "recommended" way is to start
> > > Apache
> > > >> Eagle so that I can play around with the functionality that it
> offers?
> > > >> Clearly the docker approach is buggy. Also, what browser should be
> > used?
> > > >>
> > > >> Thanks,
> > > >>
> > > >> Colm.
> > > >>
> > > >>
> > > >> --
> > > >> Colm O hEigeartaigh
> > > >>
> > > >> Talend Community Coder
> > > >> http://coders.talend.com
> > > >>
> > > >
> > > >
> > >
> > >
> > > --
> > > Colm O hEigeartaigh
> > >
> > > Talend Community Coder
> > > http://coders.talend.com
> > >
> >
>
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
>
>


-- 
Colm O hEigeartaigh

Talend Community Coder
http://coders.talend.com

Re: Unable to get 0.5.0 release working

Posted by Colm O hEigeartaigh <co...@apache.org>.
I've done that but I'm not seeing any alerts, which is why I want to find
out what the "output" of a policy is and where I can check this.

Colm.

On Mon, Jan 22, 2018 at 1:05 PM, SUDHA JENSLIN <sj...@gmail.com> wrote:

> Create and add a publisher to see the output.
>
>
>
> Regards,
> Sudha jenslin
>
> On Jan 22, 2018 6:31 PM, "Colm O hEigeartaigh" <co...@apache.org>
> wrote:
>
> Thanks - the error was due to a problem running Storm with Java 1.8. I've
> abandoned the docker image for now, and I'm trying to get it working
> locally.
>
> There are two things I'm not clear on currently, if someone could fill me
> in:
>
> a) For the  'Hdfs Audit Log Monitor' application, the Kafka Consumer Topic
> is 'hdfs_audit_log_sandbox'. Under 'Kafka Topic for Auditlog Event Sink' it
> also specifies 'hdfs_audit_event_sandbox'. However the documentation for
> the application mentions 'hdfs_audit_log_enriched_sandbox'?
>
> When I click on "STREAMS", the "HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX"
> uses the topic "hdfs_audit_event_sandbox". And indeed when I run the
> application, I can see cleansed log data appearing in
> "hdfs_audit_event_sandbox". So I'm thinking here that
> 'hdfs_audit_log_enriched_sandbox' is not correct or necessary?
>
> b) It's unclear to me where the output data goes when you create a policy.
> E.g. say I have:
>
> from HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX[str:contains(src,'/hbase')]
> select * group by user insert into hdfs_audit_log_enriched_stream_out
>
> Where is "hdfs_audit_log_enriched_stream_out" defined (is it a Kafka
> topic?). How can I check the output to make sure the policy is working
> correctly?
>
> Thanks,
>
> Colm.
>
> On Wed, Jan 17, 2018 at 10:32 PM, Edward Zhang <yo...@gmail.com>
> wrote:
>
> > There is a data preparation stage between data source(HDFS audit log) and
> > Alert Engine. This stage is running in Storm and transform the raw HDFS
> log
> > into something which can be alerted.
> >
> > The input for data preparation is hdfs_audit_log_sandbox topic and output
> > is
> >  hdfs_audit_log_enriched_sandbox.
> > The input for Alert Engine is hdfs_audit_log_enriched_sandbox and output
> > is
> > hdfs_audit_log_alert_sandbox.
> >
> > Seems in your case, the data preparation staging is not working. We
> > probably need look at Storm console and figure out if that part is
> working.
> >
> > Thanks
> > Edward
> >
> > On Wed, Jan 17, 2018 at 7:19 AM, Colm O hEigeartaigh <
> coheigea@apache.org>
> > wrote:
> >
> > > Hi Jayesh,
> > >
> > > Many thanks for your feedback! I was able to make a little further
> > headway.
> > > There are two configuration problems with the official docker image:
> > >
> > > a) A mix of "sandbox.eagle.apache.org" and "server.eagle.apache.org"
> > (this
> > > only occurs in the instructions for running the docker image. The
> version
> > > that can be started via the script in the eagle source is OK). I'll
> > submit
> > > a PR to fix this once I get a basic use-case working.
> > > b) For the audit case, it automatically logs HDFS audit logs to the
> KAFKA
> > > topic sandbox_hdfs_audit_log instead of the expected
> > hdfs_audit_log_sandbox
> > >
> > > I've fixed these things locally and I can verify that everything is
> > started
> > > correctly in Ambari. I log into the docker container and create
> > > hdfs_audit_log_sandbox and hdfs_audit_log_enriched_sandbox topics, and
> > > verify that the HDFS audit logs are flowing into the first topic. Then
> in
> > > the UI I start the Alert Engine and then the HDFS Audit Log Monitor
> > > application (changing localhost:6667 to server.eagle.apache.org:6667).
> > > Both
> > > applications start up correctly and show "running".
> > >
> > > I then create a policy with an email alert along the lines of from
> > > "HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX[str:contains(src,'/hbase')]
> > select
> > > * group by user insert into hdfs_audit_log_enriched_stream_out".
> However
> > > at
> > > this point I'm stuck - nothing appears in the alert window. Is there
> > > anything obvious I'm doing wrong, or how can I get access to logs to
> > figure
> > > out what the problem is? Other topics such as
> "hdfs_audit_event_sandbox"
> > > are mentioned in the streams window, but the documentation doesn't say
> to
> > > create them.
> > >
> > > The UI is buggy though on both Firefox and Chromium on Linux. What
> > > browser/platform are people using with the UI?
> > >
> > > Colm.
> > >
> > > On Wed, Jan 17, 2018 at 12:27 AM, Jayesh Senjaliya <ja...@apache.org>
> > > wrote:
> > >
> > > > Hi Colm,
> > > >
> > > > Please find my comments inline.
> > > >
> > > > a) The official docker image uses 0.5.0-SNAPSHOT and not the released
> > > > version.
> > > > - this is because we uploaded docker image before apache release.
> > > actually
> > > > this is same codebase apache-eagle-0.5, and it can be fixed easily by
> > > just
> > > > rebuilding docker image. there should not be any mismatch due to
> this.
> > > >
> > > > b) Aside from the above, the official docker image uses a mix of "
> > > > server.eagle.apache.org" and "sandbox.eagle.apache.org" as the host
> > > name.
> > > > The HBase service doesn't start by default in Ambari as a result.
> > > > - the only places it uses sandbox is in example script which you will
> > > have
> > > > to update anyway, which i agree that it would be good to keep it
> > > > consistent.
> > > >
> > > > c) The UI seems quite buggy. On both chromium and firefox, I only see
> > > > links to "Sandbox" and "Alert" on the left hand-side. Once I click on
> > > > "Alert" I have no way of going back to see the applications. I don't
> > see
> > > > the links to "integration" or "sites" as in the picture here:
> > > > http://eagle.apache.org/docs/latest/applications/#jmx-monitoring
> > > > - when hbase is as deep storage is used, and if eagle app has issue
> > > > connecting to hbase, the UI becomes unresponsive.
> > > >
> > > > d) In chromium, the button to create a new policy does not exist - I
> > can
> > > > only see it on Firefox.
> > > > - i have seen when you logged in, you will see admin actions. but if
> > this
> > > > still an issue, can you please file UI bug?
> > > >
> > > > e) I'm trying to get the "Hdfs Audit Log Monitor" use-case working,
> but
> > > it
> > > > seems to be stuck in "Initialized".
> > > > this eagle docs has example on how to setup the app. pls let us know
> if
> > > > you find any gaps.
> > > >
> > > > Thanks for trying out, and sharing your findings,
> > > > Jayesh
> > > >
> > > >
> > > > On Tue, Jan 16, 2018 at 3:34 AM, Colm O hEigeartaigh <
> > > coheigea@apache.org>
> > > > wrote:
> > > >
> > > >> Hi all,
> > > >>
> > > >> I'm trying to play around a bit with Apache Eagle 0.5.0 to no avail.
> > > Here
> > > >> are the problems I've run into so far:
> > > >>
> > > >> a) The official docker image uses 0.5.0-SNAPSHOT and not the
> released
> > > >> version.
> > > >>
> > > >> b) Aside from the above, the official docker image uses a mix of "
> > > >> server.eagle.apache.org" and "sandbox.eagle.apache.org" as the host
> > > >> name. The HBase service doesn't start by default in Ambari as a
> > result.
> > > >>
> > > >> c) The UI seems quite buggy. On both chromium and firefox, I only
> see
> > > >> links to "Sandbox" and "Alert" on the left hand-side. Once I click
> on
> > > >> "Alert" I have no way of going back to see the applications. I don't
> > see
> > > >> the links to "integration" or "sites" as in the picture here:
> > > >> http://eagle.apache.org/docs/latest/applications/#jmx-monitoring
> > > >>
> > > >> d) In chromium, the button to create a new policy does not exist - I
> > can
> > > >> only see it on Firefox.
> > > >>
> > > >> e) I'm trying to get the "Hdfs Audit Log Monitor" use-case working,
> > but
> > > >> it seems to be stuck in "Initialized".
> > > >>
> > > >> Could someone fill me in on what the "recommended" way is to start
> > > Apache
> > > >> Eagle so that I can play around with the functionality that it
> offers?
> > > >> Clearly the docker approach is buggy. Also, what browser should be
> > used?
> > > >>
> > > >> Thanks,
> > > >>
> > > >> Colm.
> > > >>
> > > >>
> > > >> --
> > > >> Colm O hEigeartaigh
> > > >>
> > > >> Talend Community Coder
> > > >> http://coders.talend.com
> > > >>
> > > >
> > > >
> > >
> > >
> > > --
> > > Colm O hEigeartaigh
> > >
> > > Talend Community Coder
> > > http://coders.talend.com
> > >
> >
>
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>
>
>


-- 
Colm O hEigeartaigh

Talend Community Coder
http://coders.talend.com

Re: Unable to get 0.5.0 release working

Posted by SUDHA JENSLIN <sj...@gmail.com>.
Create and add a publisher to see the output.



Regards,
Sudha jenslin

On Jan 22, 2018 6:31 PM, "Colm O hEigeartaigh" <co...@apache.org> wrote:

Thanks - the error was due to a problem running Storm with Java 1.8. I've
abandoned the docker image for now, and I'm trying to get it working
locally.

There are two things I'm not clear on currently, if someone could fill me
in:

a) For the  'Hdfs Audit Log Monitor' application, the Kafka Consumer Topic
is 'hdfs_audit_log_sandbox'. Under 'Kafka Topic for Auditlog Event Sink' it
also specifies 'hdfs_audit_event_sandbox'. However the documentation for
the application mentions 'hdfs_audit_log_enriched_sandbox'?

When I click on "STREAMS", the "HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX"
uses the topic "hdfs_audit_event_sandbox". And indeed when I run the
application, I can see cleansed log data appearing in
"hdfs_audit_event_sandbox". So I'm thinking here that
'hdfs_audit_log_enriched_sandbox' is not correct or necessary?

b) It's unclear to me where the output data goes when you create a policy.
E.g. say I have:

from HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX[str:contains(src,'/hbase')]
select * group by user insert into hdfs_audit_log_enriched_stream_out

Where is "hdfs_audit_log_enriched_stream_out" defined (is it a Kafka
topic?). How can I check the output to make sure the policy is working
correctly?

Thanks,

Colm.

On Wed, Jan 17, 2018 at 10:32 PM, Edward Zhang <yo...@gmail.com>
wrote:

> There is a data preparation stage between data source(HDFS audit log) and
> Alert Engine. This stage is running in Storm and transform the raw HDFS
log
> into something which can be alerted.
>
> The input for data preparation is hdfs_audit_log_sandbox topic and output
> is
>  hdfs_audit_log_enriched_sandbox.
> The input for Alert Engine is hdfs_audit_log_enriched_sandbox and output
> is
> hdfs_audit_log_alert_sandbox.
>
> Seems in your case, the data preparation staging is not working. We
> probably need look at Storm console and figure out if that part is
working.
>
> Thanks
> Edward
>
> On Wed, Jan 17, 2018 at 7:19 AM, Colm O hEigeartaigh <co...@apache.org>
> wrote:
>
> > Hi Jayesh,
> >
> > Many thanks for your feedback! I was able to make a little further
> headway.
> > There are two configuration problems with the official docker image:
> >
> > a) A mix of "sandbox.eagle.apache.org" and "server.eagle.apache.org"
> (this
> > only occurs in the instructions for running the docker image. The
version
> > that can be started via the script in the eagle source is OK). I'll
> submit
> > a PR to fix this once I get a basic use-case working.
> > b) For the audit case, it automatically logs HDFS audit logs to the
KAFKA
> > topic sandbox_hdfs_audit_log instead of the expected
> hdfs_audit_log_sandbox
> >
> > I've fixed these things locally and I can verify that everything is
> started
> > correctly in Ambari. I log into the docker container and create
> > hdfs_audit_log_sandbox and hdfs_audit_log_enriched_sandbox topics, and
> > verify that the HDFS audit logs are flowing into the first topic. Then
in
> > the UI I start the Alert Engine and then the HDFS Audit Log Monitor
> > application (changing localhost:6667 to server.eagle.apache.org:6667).
> > Both
> > applications start up correctly and show "running".
> >
> > I then create a policy with an email alert along the lines of from
> > "HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX[str:contains(src,'/hbase')]
> select
> > * group by user insert into hdfs_audit_log_enriched_stream_out". However
> > at
> > this point I'm stuck - nothing appears in the alert window. Is there
> > anything obvious I'm doing wrong, or how can I get access to logs to
> figure
> > out what the problem is? Other topics such as "hdfs_audit_event_sandbox"
> > are mentioned in the streams window, but the documentation doesn't say
to
> > create them.
> >
> > The UI is buggy though on both Firefox and Chromium on Linux. What
> > browser/platform are people using with the UI?
> >
> > Colm.
> >
> > On Wed, Jan 17, 2018 at 12:27 AM, Jayesh Senjaliya <ja...@apache.org>
> > wrote:
> >
> > > Hi Colm,
> > >
> > > Please find my comments inline.
> > >
> > > a) The official docker image uses 0.5.0-SNAPSHOT and not the released
> > > version.
> > > - this is because we uploaded docker image before apache release.
> > actually
> > > this is same codebase apache-eagle-0.5, and it can be fixed easily by
> > just
> > > rebuilding docker image. there should not be any mismatch due to this.
> > >
> > > b) Aside from the above, the official docker image uses a mix of "
> > > server.eagle.apache.org" and "sandbox.eagle.apache.org" as the host
> > name.
> > > The HBase service doesn't start by default in Ambari as a result.
> > > - the only places it uses sandbox is in example script which you will
> > have
> > > to update anyway, which i agree that it would be good to keep it
> > > consistent.
> > >
> > > c) The UI seems quite buggy. On both chromium and firefox, I only see
> > > links to "Sandbox" and "Alert" on the left hand-side. Once I click on
> > > "Alert" I have no way of going back to see the applications. I don't
> see
> > > the links to "integration" or "sites" as in the picture here:
> > > http://eagle.apache.org/docs/latest/applications/#jmx-monitoring
> > > - when hbase is as deep storage is used, and if eagle app has issue
> > > connecting to hbase, the UI becomes unresponsive.
> > >
> > > d) In chromium, the button to create a new policy does not exist - I
> can
> > > only see it on Firefox.
> > > - i have seen when you logged in, you will see admin actions. but if
> this
> > > still an issue, can you please file UI bug?
> > >
> > > e) I'm trying to get the "Hdfs Audit Log Monitor" use-case working,
but
> > it
> > > seems to be stuck in "Initialized".
> > > this eagle docs has example on how to setup the app. pls let us know
if
> > > you find any gaps.
> > >
> > > Thanks for trying out, and sharing your findings,
> > > Jayesh
> > >
> > >
> > > On Tue, Jan 16, 2018 at 3:34 AM, Colm O hEigeartaigh <
> > coheigea@apache.org>
> > > wrote:
> > >
> > >> Hi all,
> > >>
> > >> I'm trying to play around a bit with Apache Eagle 0.5.0 to no avail.
> > Here
> > >> are the problems I've run into so far:
> > >>
> > >> a) The official docker image uses 0.5.0-SNAPSHOT and not the released
> > >> version.
> > >>
> > >> b) Aside from the above, the official docker image uses a mix of "
> > >> server.eagle.apache.org" and "sandbox.eagle.apache.org" as the host
> > >> name. The HBase service doesn't start by default in Ambari as a
> result.
> > >>
> > >> c) The UI seems quite buggy. On both chromium and firefox, I only see
> > >> links to "Sandbox" and "Alert" on the left hand-side. Once I click on
> > >> "Alert" I have no way of going back to see the applications. I don't
> see
> > >> the links to "integration" or "sites" as in the picture here:
> > >> http://eagle.apache.org/docs/latest/applications/#jmx-monitoring
> > >>
> > >> d) In chromium, the button to create a new policy does not exist - I
> can
> > >> only see it on Firefox.
> > >>
> > >> e) I'm trying to get the "Hdfs Audit Log Monitor" use-case working,
> but
> > >> it seems to be stuck in "Initialized".
> > >>
> > >> Could someone fill me in on what the "recommended" way is to start
> > Apache
> > >> Eagle so that I can play around with the functionality that it
offers?
> > >> Clearly the docker approach is buggy. Also, what browser should be
> used?
> > >>
> > >> Thanks,
> > >>
> > >> Colm.
> > >>
> > >>
> > >> --
> > >> Colm O hEigeartaigh
> > >>
> > >> Talend Community Coder
> > >> http://coders.talend.com
> > >>
> > >
> > >
> >
> >
> > --
> > Colm O hEigeartaigh
> >
> > Talend Community Coder
> > http://coders.talend.com
> >
>



--
Colm O hEigeartaigh

Talend Community Coder
http://coders.talend.com

Re: Unable to get 0.5.0 release working

Posted by SUDHA JENSLIN <sj...@gmail.com>.
Create and add a publisher to see the output.



Regards,
Sudha jenslin

On Jan 22, 2018 6:31 PM, "Colm O hEigeartaigh" <co...@apache.org> wrote:

Thanks - the error was due to a problem running Storm with Java 1.8. I've
abandoned the docker image for now, and I'm trying to get it working
locally.

There are two things I'm not clear on currently, if someone could fill me
in:

a) For the  'Hdfs Audit Log Monitor' application, the Kafka Consumer Topic
is 'hdfs_audit_log_sandbox'. Under 'Kafka Topic for Auditlog Event Sink' it
also specifies 'hdfs_audit_event_sandbox'. However the documentation for
the application mentions 'hdfs_audit_log_enriched_sandbox'?

When I click on "STREAMS", the "HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX"
uses the topic "hdfs_audit_event_sandbox". And indeed when I run the
application, I can see cleansed log data appearing in
"hdfs_audit_event_sandbox". So I'm thinking here that
'hdfs_audit_log_enriched_sandbox' is not correct or necessary?

b) It's unclear to me where the output data goes when you create a policy.
E.g. say I have:

from HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX[str:contains(src,'/hbase')]
select * group by user insert into hdfs_audit_log_enriched_stream_out

Where is "hdfs_audit_log_enriched_stream_out" defined (is it a Kafka
topic?). How can I check the output to make sure the policy is working
correctly?

Thanks,

Colm.

On Wed, Jan 17, 2018 at 10:32 PM, Edward Zhang <yo...@gmail.com>
wrote:

> There is a data preparation stage between data source(HDFS audit log) and
> Alert Engine. This stage is running in Storm and transform the raw HDFS
log
> into something which can be alerted.
>
> The input for data preparation is hdfs_audit_log_sandbox topic and output
> is
>  hdfs_audit_log_enriched_sandbox.
> The input for Alert Engine is hdfs_audit_log_enriched_sandbox and output
> is
> hdfs_audit_log_alert_sandbox.
>
> Seems in your case, the data preparation staging is not working. We
> probably need look at Storm console and figure out if that part is
working.
>
> Thanks
> Edward
>
> On Wed, Jan 17, 2018 at 7:19 AM, Colm O hEigeartaigh <co...@apache.org>
> wrote:
>
> > Hi Jayesh,
> >
> > Many thanks for your feedback! I was able to make a little further
> headway.
> > There are two configuration problems with the official docker image:
> >
> > a) A mix of "sandbox.eagle.apache.org" and "server.eagle.apache.org"
> (this
> > only occurs in the instructions for running the docker image. The
version
> > that can be started via the script in the eagle source is OK). I'll
> submit
> > a PR to fix this once I get a basic use-case working.
> > b) For the audit case, it automatically logs HDFS audit logs to the
KAFKA
> > topic sandbox_hdfs_audit_log instead of the expected
> hdfs_audit_log_sandbox
> >
> > I've fixed these things locally and I can verify that everything is
> started
> > correctly in Ambari. I log into the docker container and create
> > hdfs_audit_log_sandbox and hdfs_audit_log_enriched_sandbox topics, and
> > verify that the HDFS audit logs are flowing into the first topic. Then
in
> > the UI I start the Alert Engine and then the HDFS Audit Log Monitor
> > application (changing localhost:6667 to server.eagle.apache.org:6667).
> > Both
> > applications start up correctly and show "running".
> >
> > I then create a policy with an email alert along the lines of from
> > "HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX[str:contains(src,'/hbase')]
> select
> > * group by user insert into hdfs_audit_log_enriched_stream_out". However
> > at
> > this point I'm stuck - nothing appears in the alert window. Is there
> > anything obvious I'm doing wrong, or how can I get access to logs to
> figure
> > out what the problem is? Other topics such as "hdfs_audit_event_sandbox"
> > are mentioned in the streams window, but the documentation doesn't say
to
> > create them.
> >
> > The UI is buggy though on both Firefox and Chromium on Linux. What
> > browser/platform are people using with the UI?
> >
> > Colm.
> >
> > On Wed, Jan 17, 2018 at 12:27 AM, Jayesh Senjaliya <ja...@apache.org>
> > wrote:
> >
> > > Hi Colm,
> > >
> > > Please find my comments inline.
> > >
> > > a) The official docker image uses 0.5.0-SNAPSHOT and not the released
> > > version.
> > > - this is because we uploaded docker image before apache release.
> > actually
> > > this is same codebase apache-eagle-0.5, and it can be fixed easily by
> > just
> > > rebuilding docker image. there should not be any mismatch due to this.
> > >
> > > b) Aside from the above, the official docker image uses a mix of "
> > > server.eagle.apache.org" and "sandbox.eagle.apache.org" as the host
> > name.
> > > The HBase service doesn't start by default in Ambari as a result.
> > > - the only places it uses sandbox is in example script which you will
> > have
> > > to update anyway, which i agree that it would be good to keep it
> > > consistent.
> > >
> > > c) The UI seems quite buggy. On both chromium and firefox, I only see
> > > links to "Sandbox" and "Alert" on the left hand-side. Once I click on
> > > "Alert" I have no way of going back to see the applications. I don't
> see
> > > the links to "integration" or "sites" as in the picture here:
> > > http://eagle.apache.org/docs/latest/applications/#jmx-monitoring
> > > - when hbase is as deep storage is used, and if eagle app has issue
> > > connecting to hbase, the UI becomes unresponsive.
> > >
> > > d) In chromium, the button to create a new policy does not exist - I
> can
> > > only see it on Firefox.
> > > - i have seen when you logged in, you will see admin actions. but if
> this
> > > still an issue, can you please file UI bug?
> > >
> > > e) I'm trying to get the "Hdfs Audit Log Monitor" use-case working,
but
> > it
> > > seems to be stuck in "Initialized".
> > > this eagle docs has example on how to setup the app. pls let us know
if
> > > you find any gaps.
> > >
> > > Thanks for trying out, and sharing your findings,
> > > Jayesh
> > >
> > >
> > > On Tue, Jan 16, 2018 at 3:34 AM, Colm O hEigeartaigh <
> > coheigea@apache.org>
> > > wrote:
> > >
> > >> Hi all,
> > >>
> > >> I'm trying to play around a bit with Apache Eagle 0.5.0 to no avail.
> > Here
> > >> are the problems I've run into so far:
> > >>
> > >> a) The official docker image uses 0.5.0-SNAPSHOT and not the released
> > >> version.
> > >>
> > >> b) Aside from the above, the official docker image uses a mix of "
> > >> server.eagle.apache.org" and "sandbox.eagle.apache.org" as the host
> > >> name. The HBase service doesn't start by default in Ambari as a
> result.
> > >>
> > >> c) The UI seems quite buggy. On both chromium and firefox, I only see
> > >> links to "Sandbox" and "Alert" on the left hand-side. Once I click on
> > >> "Alert" I have no way of going back to see the applications. I don't
> see
> > >> the links to "integration" or "sites" as in the picture here:
> > >> http://eagle.apache.org/docs/latest/applications/#jmx-monitoring
> > >>
> > >> d) In chromium, the button to create a new policy does not exist - I
> can
> > >> only see it on Firefox.
> > >>
> > >> e) I'm trying to get the "Hdfs Audit Log Monitor" use-case working,
> but
> > >> it seems to be stuck in "Initialized".
> > >>
> > >> Could someone fill me in on what the "recommended" way is to start
> > Apache
> > >> Eagle so that I can play around with the functionality that it
offers?
> > >> Clearly the docker approach is buggy. Also, what browser should be
> used?
> > >>
> > >> Thanks,
> > >>
> > >> Colm.
> > >>
> > >>
> > >> --
> > >> Colm O hEigeartaigh
> > >>
> > >> Talend Community Coder
> > >> http://coders.talend.com
> > >>
> > >
> > >
> >
> >
> > --
> > Colm O hEigeartaigh
> >
> > Talend Community Coder
> > http://coders.talend.com
> >
>



--
Colm O hEigeartaigh

Talend Community Coder
http://coders.talend.com

Re: Unable to get 0.5.0 release working

Posted by Colm O hEigeartaigh <co...@apache.org>.
Thanks - the error was due to a problem running Storm with Java 1.8. I've
abandoned the docker image for now, and I'm trying to get it working
locally.

There are two things I'm not clear on currently, if someone could fill me
in:

a) For the  'Hdfs Audit Log Monitor' application, the Kafka Consumer Topic
is 'hdfs_audit_log_sandbox'. Under 'Kafka Topic for Auditlog Event Sink' it
also specifies 'hdfs_audit_event_sandbox'. However the documentation for
the application mentions 'hdfs_audit_log_enriched_sandbox'?

When I click on "STREAMS", the "HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX"
uses the topic "hdfs_audit_event_sandbox". And indeed when I run the
application, I can see cleansed log data appearing in
"hdfs_audit_event_sandbox". So I'm thinking here that
'hdfs_audit_log_enriched_sandbox' is not correct or necessary?

b) It's unclear to me where the output data goes when you create a policy.
E.g. say I have:

from HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX[str:contains(src,'/hbase')]
select * group by user insert into hdfs_audit_log_enriched_stream_out

Where is "hdfs_audit_log_enriched_stream_out" defined (is it a Kafka
topic?). How can I check the output to make sure the policy is working
correctly?

Thanks,

Colm.

On Wed, Jan 17, 2018 at 10:32 PM, Edward Zhang <yo...@gmail.com>
wrote:

> There is a data preparation stage between data source(HDFS audit log) and
> Alert Engine. This stage is running in Storm and transform the raw HDFS log
> into something which can be alerted.
>
> The input for data preparation is hdfs_audit_log_sandbox topic and output
> is
>  hdfs_audit_log_enriched_sandbox.
> The input for Alert Engine is hdfs_audit_log_enriched_sandbox and output
> is
> hdfs_audit_log_alert_sandbox.
>
> Seems in your case, the data preparation staging is not working. We
> probably need look at Storm console and figure out if that part is working.
>
> Thanks
> Edward
>
> On Wed, Jan 17, 2018 at 7:19 AM, Colm O hEigeartaigh <co...@apache.org>
> wrote:
>
> > Hi Jayesh,
> >
> > Many thanks for your feedback! I was able to make a little further
> headway.
> > There are two configuration problems with the official docker image:
> >
> > a) A mix of "sandbox.eagle.apache.org" and "server.eagle.apache.org"
> (this
> > only occurs in the instructions for running the docker image. The version
> > that can be started via the script in the eagle source is OK). I'll
> submit
> > a PR to fix this once I get a basic use-case working.
> > b) For the audit case, it automatically logs HDFS audit logs to the KAFKA
> > topic sandbox_hdfs_audit_log instead of the expected
> hdfs_audit_log_sandbox
> >
> > I've fixed these things locally and I can verify that everything is
> started
> > correctly in Ambari. I log into the docker container and create
> > hdfs_audit_log_sandbox and hdfs_audit_log_enriched_sandbox topics, and
> > verify that the HDFS audit logs are flowing into the first topic. Then in
> > the UI I start the Alert Engine and then the HDFS Audit Log Monitor
> > application (changing localhost:6667 to server.eagle.apache.org:6667).
> > Both
> > applications start up correctly and show "running".
> >
> > I then create a policy with an email alert along the lines of from
> > "HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX[str:contains(src,'/hbase')]
> select
> > * group by user insert into hdfs_audit_log_enriched_stream_out". However
> > at
> > this point I'm stuck - nothing appears in the alert window. Is there
> > anything obvious I'm doing wrong, or how can I get access to logs to
> figure
> > out what the problem is? Other topics such as "hdfs_audit_event_sandbox"
> > are mentioned in the streams window, but the documentation doesn't say to
> > create them.
> >
> > The UI is buggy though on both Firefox and Chromium on Linux. What
> > browser/platform are people using with the UI?
> >
> > Colm.
> >
> > On Wed, Jan 17, 2018 at 12:27 AM, Jayesh Senjaliya <ja...@apache.org>
> > wrote:
> >
> > > Hi Colm,
> > >
> > > Please find my comments inline.
> > >
> > > a) The official docker image uses 0.5.0-SNAPSHOT and not the released
> > > version.
> > > - this is because we uploaded docker image before apache release.
> > actually
> > > this is same codebase apache-eagle-0.5, and it can be fixed easily by
> > just
> > > rebuilding docker image. there should not be any mismatch due to this.
> > >
> > > b) Aside from the above, the official docker image uses a mix of "
> > > server.eagle.apache.org" and "sandbox.eagle.apache.org" as the host
> > name.
> > > The HBase service doesn't start by default in Ambari as a result.
> > > - the only places it uses sandbox is in example script which you will
> > have
> > > to update anyway, which i agree that it would be good to keep it
> > > consistent.
> > >
> > > c) The UI seems quite buggy. On both chromium and firefox, I only see
> > > links to "Sandbox" and "Alert" on the left hand-side. Once I click on
> > > "Alert" I have no way of going back to see the applications. I don't
> see
> > > the links to "integration" or "sites" as in the picture here:
> > > http://eagle.apache.org/docs/latest/applications/#jmx-monitoring
> > > - when hbase is as deep storage is used, and if eagle app has issue
> > > connecting to hbase, the UI becomes unresponsive.
> > >
> > > d) In chromium, the button to create a new policy does not exist - I
> can
> > > only see it on Firefox.
> > > - i have seen when you logged in, you will see admin actions. but if
> this
> > > still an issue, can you please file UI bug?
> > >
> > > e) I'm trying to get the "Hdfs Audit Log Monitor" use-case working, but
> > it
> > > seems to be stuck in "Initialized".
> > > this eagle docs has example on how to setup the app. pls let us know if
> > > you find any gaps.
> > >
> > > Thanks for trying out, and sharing your findings,
> > > Jayesh
> > >
> > >
> > > On Tue, Jan 16, 2018 at 3:34 AM, Colm O hEigeartaigh <
> > coheigea@apache.org>
> > > wrote:
> > >
> > >> Hi all,
> > >>
> > >> I'm trying to play around a bit with Apache Eagle 0.5.0 to no avail.
> > Here
> > >> are the problems I've run into so far:
> > >>
> > >> a) The official docker image uses 0.5.0-SNAPSHOT and not the released
> > >> version.
> > >>
> > >> b) Aside from the above, the official docker image uses a mix of "
> > >> server.eagle.apache.org" and "sandbox.eagle.apache.org" as the host
> > >> name. The HBase service doesn't start by default in Ambari as a
> result.
> > >>
> > >> c) The UI seems quite buggy. On both chromium and firefox, I only see
> > >> links to "Sandbox" and "Alert" on the left hand-side. Once I click on
> > >> "Alert" I have no way of going back to see the applications. I don't
> see
> > >> the links to "integration" or "sites" as in the picture here:
> > >> http://eagle.apache.org/docs/latest/applications/#jmx-monitoring
> > >>
> > >> d) In chromium, the button to create a new policy does not exist - I
> can
> > >> only see it on Firefox.
> > >>
> > >> e) I'm trying to get the "Hdfs Audit Log Monitor" use-case working,
> but
> > >> it seems to be stuck in "Initialized".
> > >>
> > >> Could someone fill me in on what the "recommended" way is to start
> > Apache
> > >> Eagle so that I can play around with the functionality that it offers?
> > >> Clearly the docker approach is buggy. Also, what browser should be
> used?
> > >>
> > >> Thanks,
> > >>
> > >> Colm.
> > >>
> > >>
> > >> --
> > >> Colm O hEigeartaigh
> > >>
> > >> Talend Community Coder
> > >> http://coders.talend.com
> > >>
> > >
> > >
> >
> >
> > --
> > Colm O hEigeartaigh
> >
> > Talend Community Coder
> > http://coders.talend.com
> >
>



-- 
Colm O hEigeartaigh

Talend Community Coder
http://coders.talend.com

Re: Unable to get 0.5.0 release working

Posted by Colm O hEigeartaigh <co...@apache.org>.
Thanks - the error was due to a problem running Storm with Java 1.8. I've
abandoned the docker image for now, and I'm trying to get it working
locally.

There are two things I'm not clear on currently, if someone could fill me
in:

a) For the  'Hdfs Audit Log Monitor' application, the Kafka Consumer Topic
is 'hdfs_audit_log_sandbox'. Under 'Kafka Topic for Auditlog Event Sink' it
also specifies 'hdfs_audit_event_sandbox'. However the documentation for
the application mentions 'hdfs_audit_log_enriched_sandbox'?

When I click on "STREAMS", the "HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX"
uses the topic "hdfs_audit_event_sandbox". And indeed when I run the
application, I can see cleansed log data appearing in
"hdfs_audit_event_sandbox". So I'm thinking here that
'hdfs_audit_log_enriched_sandbox' is not correct or necessary?

b) It's unclear to me where the output data goes when you create a policy.
E.g. say I have:

from HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX[str:contains(src,'/hbase')]
select * group by user insert into hdfs_audit_log_enriched_stream_out

Where is "hdfs_audit_log_enriched_stream_out" defined (is it a Kafka
topic?). How can I check the output to make sure the policy is working
correctly?

Thanks,

Colm.

On Wed, Jan 17, 2018 at 10:32 PM, Edward Zhang <yo...@gmail.com>
wrote:

> There is a data preparation stage between data source(HDFS audit log) and
> Alert Engine. This stage is running in Storm and transform the raw HDFS log
> into something which can be alerted.
>
> The input for data preparation is hdfs_audit_log_sandbox topic and output
> is
>  hdfs_audit_log_enriched_sandbox.
> The input for Alert Engine is hdfs_audit_log_enriched_sandbox and output
> is
> hdfs_audit_log_alert_sandbox.
>
> Seems in your case, the data preparation staging is not working. We
> probably need look at Storm console and figure out if that part is working.
>
> Thanks
> Edward
>
> On Wed, Jan 17, 2018 at 7:19 AM, Colm O hEigeartaigh <co...@apache.org>
> wrote:
>
> > Hi Jayesh,
> >
> > Many thanks for your feedback! I was able to make a little further
> headway.
> > There are two configuration problems with the official docker image:
> >
> > a) A mix of "sandbox.eagle.apache.org" and "server.eagle.apache.org"
> (this
> > only occurs in the instructions for running the docker image. The version
> > that can be started via the script in the eagle source is OK). I'll
> submit
> > a PR to fix this once I get a basic use-case working.
> > b) For the audit case, it automatically logs HDFS audit logs to the KAFKA
> > topic sandbox_hdfs_audit_log instead of the expected
> hdfs_audit_log_sandbox
> >
> > I've fixed these things locally and I can verify that everything is
> started
> > correctly in Ambari. I log into the docker container and create
> > hdfs_audit_log_sandbox and hdfs_audit_log_enriched_sandbox topics, and
> > verify that the HDFS audit logs are flowing into the first topic. Then in
> > the UI I start the Alert Engine and then the HDFS Audit Log Monitor
> > application (changing localhost:6667 to server.eagle.apache.org:6667).
> > Both
> > applications start up correctly and show "running".
> >
> > I then create a policy with an email alert along the lines of from
> > "HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX[str:contains(src,'/hbase')]
> select
> > * group by user insert into hdfs_audit_log_enriched_stream_out". However
> > at
> > this point I'm stuck - nothing appears in the alert window. Is there
> > anything obvious I'm doing wrong, or how can I get access to logs to
> figure
> > out what the problem is? Other topics such as "hdfs_audit_event_sandbox"
> > are mentioned in the streams window, but the documentation doesn't say to
> > create them.
> >
> > The UI is buggy though on both Firefox and Chromium on Linux. What
> > browser/platform are people using with the UI?
> >
> > Colm.
> >
> > On Wed, Jan 17, 2018 at 12:27 AM, Jayesh Senjaliya <ja...@apache.org>
> > wrote:
> >
> > > Hi Colm,
> > >
> > > Please find my comments inline.
> > >
> > > a) The official docker image uses 0.5.0-SNAPSHOT and not the released
> > > version.
> > > - this is because we uploaded docker image before apache release.
> > actually
> > > this is same codebase apache-eagle-0.5, and it can be fixed easily by
> > just
> > > rebuilding docker image. there should not be any mismatch due to this.
> > >
> > > b) Aside from the above, the official docker image uses a mix of "
> > > server.eagle.apache.org" and "sandbox.eagle.apache.org" as the host
> > name.
> > > The HBase service doesn't start by default in Ambari as a result.
> > > - the only places it uses sandbox is in example script which you will
> > have
> > > to update anyway, which i agree that it would be good to keep it
> > > consistent.
> > >
> > > c) The UI seems quite buggy. On both chromium and firefox, I only see
> > > links to "Sandbox" and "Alert" on the left hand-side. Once I click on
> > > "Alert" I have no way of going back to see the applications. I don't
> see
> > > the links to "integration" or "sites" as in the picture here:
> > > http://eagle.apache.org/docs/latest/applications/#jmx-monitoring
> > > - when hbase is as deep storage is used, and if eagle app has issue
> > > connecting to hbase, the UI becomes unresponsive.
> > >
> > > d) In chromium, the button to create a new policy does not exist - I
> can
> > > only see it on Firefox.
> > > - i have seen when you logged in, you will see admin actions. but if
> this
> > > still an issue, can you please file UI bug?
> > >
> > > e) I'm trying to get the "Hdfs Audit Log Monitor" use-case working, but
> > it
> > > seems to be stuck in "Initialized".
> > > this eagle docs has example on how to setup the app. pls let us know if
> > > you find any gaps.
> > >
> > > Thanks for trying out, and sharing your findings,
> > > Jayesh
> > >
> > >
> > > On Tue, Jan 16, 2018 at 3:34 AM, Colm O hEigeartaigh <
> > coheigea@apache.org>
> > > wrote:
> > >
> > >> Hi all,
> > >>
> > >> I'm trying to play around a bit with Apache Eagle 0.5.0 to no avail.
> > Here
> > >> are the problems I've run into so far:
> > >>
> > >> a) The official docker image uses 0.5.0-SNAPSHOT and not the released
> > >> version.
> > >>
> > >> b) Aside from the above, the official docker image uses a mix of "
> > >> server.eagle.apache.org" and "sandbox.eagle.apache.org" as the host
> > >> name. The HBase service doesn't start by default in Ambari as a
> result.
> > >>
> > >> c) The UI seems quite buggy. On both chromium and firefox, I only see
> > >> links to "Sandbox" and "Alert" on the left hand-side. Once I click on
> > >> "Alert" I have no way of going back to see the applications. I don't
> see
> > >> the links to "integration" or "sites" as in the picture here:
> > >> http://eagle.apache.org/docs/latest/applications/#jmx-monitoring
> > >>
> > >> d) In chromium, the button to create a new policy does not exist - I
> can
> > >> only see it on Firefox.
> > >>
> > >> e) I'm trying to get the "Hdfs Audit Log Monitor" use-case working,
> but
> > >> it seems to be stuck in "Initialized".
> > >>
> > >> Could someone fill me in on what the "recommended" way is to start
> > Apache
> > >> Eagle so that I can play around with the functionality that it offers?
> > >> Clearly the docker approach is buggy. Also, what browser should be
> used?
> > >>
> > >> Thanks,
> > >>
> > >> Colm.
> > >>
> > >>
> > >> --
> > >> Colm O hEigeartaigh
> > >>
> > >> Talend Community Coder
> > >> http://coders.talend.com
> > >>
> > >
> > >
> >
> >
> > --
> > Colm O hEigeartaigh
> >
> > Talend Community Coder
> > http://coders.talend.com
> >
>



-- 
Colm O hEigeartaigh

Talend Community Coder
http://coders.talend.com

Re: Unable to get 0.5.0 release working

Posted by Edward Zhang <yo...@gmail.com>.
There is a data preparation stage between data source(HDFS audit log) and
Alert Engine. This stage is running in Storm and transform the raw HDFS log
into something which can be alerted.

The input for data preparation is hdfs_audit_log_sandbox topic and output is
 hdfs_audit_log_enriched_sandbox.
The input for Alert Engine is hdfs_audit_log_enriched_sandbox and output is
hdfs_audit_log_alert_sandbox.

Seems in your case, the data preparation staging is not working. We
probably need look at Storm console and figure out if that part is working.

Thanks
Edward

On Wed, Jan 17, 2018 at 7:19 AM, Colm O hEigeartaigh <co...@apache.org>
wrote:

> Hi Jayesh,
>
> Many thanks for your feedback! I was able to make a little further headway.
> There are two configuration problems with the official docker image:
>
> a) A mix of "sandbox.eagle.apache.org" and "server.eagle.apache.org" (this
> only occurs in the instructions for running the docker image. The version
> that can be started via the script in the eagle source is OK). I'll submit
> a PR to fix this once I get a basic use-case working.
> b) For the audit case, it automatically logs HDFS audit logs to the KAFKA
> topic sandbox_hdfs_audit_log instead of the expected hdfs_audit_log_sandbox
>
> I've fixed these things locally and I can verify that everything is started
> correctly in Ambari. I log into the docker container and create
> hdfs_audit_log_sandbox and hdfs_audit_log_enriched_sandbox topics, and
> verify that the HDFS audit logs are flowing into the first topic. Then in
> the UI I start the Alert Engine and then the HDFS Audit Log Monitor
> application (changing localhost:6667 to server.eagle.apache.org:6667).
> Both
> applications start up correctly and show "running".
>
> I then create a policy with an email alert along the lines of from
> "HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX[str:contains(src,'/hbase')] select
> * group by user insert into hdfs_audit_log_enriched_stream_out". However
> at
> this point I'm stuck - nothing appears in the alert window. Is there
> anything obvious I'm doing wrong, or how can I get access to logs to figure
> out what the problem is? Other topics such as "hdfs_audit_event_sandbox"
> are mentioned in the streams window, but the documentation doesn't say to
> create them.
>
> The UI is buggy though on both Firefox and Chromium on Linux. What
> browser/platform are people using with the UI?
>
> Colm.
>
> On Wed, Jan 17, 2018 at 12:27 AM, Jayesh Senjaliya <ja...@apache.org>
> wrote:
>
> > Hi Colm,
> >
> > Please find my comments inline.
> >
> > a) The official docker image uses 0.5.0-SNAPSHOT and not the released
> > version.
> > - this is because we uploaded docker image before apache release.
> actually
> > this is same codebase apache-eagle-0.5, and it can be fixed easily by
> just
> > rebuilding docker image. there should not be any mismatch due to this.
> >
> > b) Aside from the above, the official docker image uses a mix of "
> > server.eagle.apache.org" and "sandbox.eagle.apache.org" as the host
> name.
> > The HBase service doesn't start by default in Ambari as a result.
> > - the only places it uses sandbox is in example script which you will
> have
> > to update anyway, which i agree that it would be good to keep it
> > consistent.
> >
> > c) The UI seems quite buggy. On both chromium and firefox, I only see
> > links to "Sandbox" and "Alert" on the left hand-side. Once I click on
> > "Alert" I have no way of going back to see the applications. I don't see
> > the links to "integration" or "sites" as in the picture here:
> > http://eagle.apache.org/docs/latest/applications/#jmx-monitoring
> > - when hbase is as deep storage is used, and if eagle app has issue
> > connecting to hbase, the UI becomes unresponsive.
> >
> > d) In chromium, the button to create a new policy does not exist - I can
> > only see it on Firefox.
> > - i have seen when you logged in, you will see admin actions. but if this
> > still an issue, can you please file UI bug?
> >
> > e) I'm trying to get the "Hdfs Audit Log Monitor" use-case working, but
> it
> > seems to be stuck in "Initialized".
> > this eagle docs has example on how to setup the app. pls let us know if
> > you find any gaps.
> >
> > Thanks for trying out, and sharing your findings,
> > Jayesh
> >
> >
> > On Tue, Jan 16, 2018 at 3:34 AM, Colm O hEigeartaigh <
> coheigea@apache.org>
> > wrote:
> >
> >> Hi all,
> >>
> >> I'm trying to play around a bit with Apache Eagle 0.5.0 to no avail.
> Here
> >> are the problems I've run into so far:
> >>
> >> a) The official docker image uses 0.5.0-SNAPSHOT and not the released
> >> version.
> >>
> >> b) Aside from the above, the official docker image uses a mix of "
> >> server.eagle.apache.org" and "sandbox.eagle.apache.org" as the host
> >> name. The HBase service doesn't start by default in Ambari as a result.
> >>
> >> c) The UI seems quite buggy. On both chromium and firefox, I only see
> >> links to "Sandbox" and "Alert" on the left hand-side. Once I click on
> >> "Alert" I have no way of going back to see the applications. I don't see
> >> the links to "integration" or "sites" as in the picture here:
> >> http://eagle.apache.org/docs/latest/applications/#jmx-monitoring
> >>
> >> d) In chromium, the button to create a new policy does not exist - I can
> >> only see it on Firefox.
> >>
> >> e) I'm trying to get the "Hdfs Audit Log Monitor" use-case working, but
> >> it seems to be stuck in "Initialized".
> >>
> >> Could someone fill me in on what the "recommended" way is to start
> Apache
> >> Eagle so that I can play around with the functionality that it offers?
> >> Clearly the docker approach is buggy. Also, what browser should be used?
> >>
> >> Thanks,
> >>
> >> Colm.
> >>
> >>
> >> --
> >> Colm O hEigeartaigh
> >>
> >> Talend Community Coder
> >> http://coders.talend.com
> >>
> >
> >
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>

Re: Unable to get 0.5.0 release working

Posted by Edward Zhang <yo...@gmail.com>.
There is a data preparation stage between data source(HDFS audit log) and
Alert Engine. This stage is running in Storm and transform the raw HDFS log
into something which can be alerted.

The input for data preparation is hdfs_audit_log_sandbox topic and output is
 hdfs_audit_log_enriched_sandbox.
The input for Alert Engine is hdfs_audit_log_enriched_sandbox and output is
hdfs_audit_log_alert_sandbox.

Seems in your case, the data preparation staging is not working. We
probably need look at Storm console and figure out if that part is working.

Thanks
Edward

On Wed, Jan 17, 2018 at 7:19 AM, Colm O hEigeartaigh <co...@apache.org>
wrote:

> Hi Jayesh,
>
> Many thanks for your feedback! I was able to make a little further headway.
> There are two configuration problems with the official docker image:
>
> a) A mix of "sandbox.eagle.apache.org" and "server.eagle.apache.org" (this
> only occurs in the instructions for running the docker image. The version
> that can be started via the script in the eagle source is OK). I'll submit
> a PR to fix this once I get a basic use-case working.
> b) For the audit case, it automatically logs HDFS audit logs to the KAFKA
> topic sandbox_hdfs_audit_log instead of the expected hdfs_audit_log_sandbox
>
> I've fixed these things locally and I can verify that everything is started
> correctly in Ambari. I log into the docker container and create
> hdfs_audit_log_sandbox and hdfs_audit_log_enriched_sandbox topics, and
> verify that the HDFS audit logs are flowing into the first topic. Then in
> the UI I start the Alert Engine and then the HDFS Audit Log Monitor
> application (changing localhost:6667 to server.eagle.apache.org:6667).
> Both
> applications start up correctly and show "running".
>
> I then create a policy with an email alert along the lines of from
> "HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX[str:contains(src,'/hbase')] select
> * group by user insert into hdfs_audit_log_enriched_stream_out". However
> at
> this point I'm stuck - nothing appears in the alert window. Is there
> anything obvious I'm doing wrong, or how can I get access to logs to figure
> out what the problem is? Other topics such as "hdfs_audit_event_sandbox"
> are mentioned in the streams window, but the documentation doesn't say to
> create them.
>
> The UI is buggy though on both Firefox and Chromium on Linux. What
> browser/platform are people using with the UI?
>
> Colm.
>
> On Wed, Jan 17, 2018 at 12:27 AM, Jayesh Senjaliya <ja...@apache.org>
> wrote:
>
> > Hi Colm,
> >
> > Please find my comments inline.
> >
> > a) The official docker image uses 0.5.0-SNAPSHOT and not the released
> > version.
> > - this is because we uploaded docker image before apache release.
> actually
> > this is same codebase apache-eagle-0.5, and it can be fixed easily by
> just
> > rebuilding docker image. there should not be any mismatch due to this.
> >
> > b) Aside from the above, the official docker image uses a mix of "
> > server.eagle.apache.org" and "sandbox.eagle.apache.org" as the host
> name.
> > The HBase service doesn't start by default in Ambari as a result.
> > - the only places it uses sandbox is in example script which you will
> have
> > to update anyway, which i agree that it would be good to keep it
> > consistent.
> >
> > c) The UI seems quite buggy. On both chromium and firefox, I only see
> > links to "Sandbox" and "Alert" on the left hand-side. Once I click on
> > "Alert" I have no way of going back to see the applications. I don't see
> > the links to "integration" or "sites" as in the picture here:
> > http://eagle.apache.org/docs/latest/applications/#jmx-monitoring
> > - when hbase is as deep storage is used, and if eagle app has issue
> > connecting to hbase, the UI becomes unresponsive.
> >
> > d) In chromium, the button to create a new policy does not exist - I can
> > only see it on Firefox.
> > - i have seen when you logged in, you will see admin actions. but if this
> > still an issue, can you please file UI bug?
> >
> > e) I'm trying to get the "Hdfs Audit Log Monitor" use-case working, but
> it
> > seems to be stuck in "Initialized".
> > this eagle docs has example on how to setup the app. pls let us know if
> > you find any gaps.
> >
> > Thanks for trying out, and sharing your findings,
> > Jayesh
> >
> >
> > On Tue, Jan 16, 2018 at 3:34 AM, Colm O hEigeartaigh <
> coheigea@apache.org>
> > wrote:
> >
> >> Hi all,
> >>
> >> I'm trying to play around a bit with Apache Eagle 0.5.0 to no avail.
> Here
> >> are the problems I've run into so far:
> >>
> >> a) The official docker image uses 0.5.0-SNAPSHOT and not the released
> >> version.
> >>
> >> b) Aside from the above, the official docker image uses a mix of "
> >> server.eagle.apache.org" and "sandbox.eagle.apache.org" as the host
> >> name. The HBase service doesn't start by default in Ambari as a result.
> >>
> >> c) The UI seems quite buggy. On both chromium and firefox, I only see
> >> links to "Sandbox" and "Alert" on the left hand-side. Once I click on
> >> "Alert" I have no way of going back to see the applications. I don't see
> >> the links to "integration" or "sites" as in the picture here:
> >> http://eagle.apache.org/docs/latest/applications/#jmx-monitoring
> >>
> >> d) In chromium, the button to create a new policy does not exist - I can
> >> only see it on Firefox.
> >>
> >> e) I'm trying to get the "Hdfs Audit Log Monitor" use-case working, but
> >> it seems to be stuck in "Initialized".
> >>
> >> Could someone fill me in on what the "recommended" way is to start
> Apache
> >> Eagle so that I can play around with the functionality that it offers?
> >> Clearly the docker approach is buggy. Also, what browser should be used?
> >>
> >> Thanks,
> >>
> >> Colm.
> >>
> >>
> >> --
> >> Colm O hEigeartaigh
> >>
> >> Talend Community Coder
> >> http://coders.talend.com
> >>
> >
> >
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>

Re: Unable to get 0.5.0 release working

Posted by Colm O hEigeartaigh <co...@apache.org>.
Hi Jayesh,

Many thanks for your feedback! I was able to make a little further headway.
There are two configuration problems with the official docker image:

a) A mix of "sandbox.eagle.apache.org" and "server.eagle.apache.org" (this
only occurs in the instructions for running the docker image. The version
that can be started via the script in the eagle source is OK). I'll submit
a PR to fix this once I get a basic use-case working.
b) For the audit case, it automatically logs HDFS audit logs to the KAFKA
topic sandbox_hdfs_audit_log instead of the expected hdfs_audit_log_sandbox

I've fixed these things locally and I can verify that everything is started
correctly in Ambari. I log into the docker container and create
hdfs_audit_log_sandbox and hdfs_audit_log_enriched_sandbox topics, and
verify that the HDFS audit logs are flowing into the first topic. Then in
the UI I start the Alert Engine and then the HDFS Audit Log Monitor
application (changing localhost:6667 to server.eagle.apache.org:6667). Both
applications start up correctly and show "running".

I then create a policy with an email alert along the lines of from
"HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX[str:contains(src,'/hbase')] select
* group by user insert into hdfs_audit_log_enriched_stream_out". However at
this point I'm stuck - nothing appears in the alert window. Is there
anything obvious I'm doing wrong, or how can I get access to logs to figure
out what the problem is? Other topics such as "hdfs_audit_event_sandbox"
are mentioned in the streams window, but the documentation doesn't say to
create them.

The UI is buggy though on both Firefox and Chromium on Linux. What
browser/platform are people using with the UI?

Colm.

On Wed, Jan 17, 2018 at 12:27 AM, Jayesh Senjaliya <ja...@apache.org>
wrote:

> Hi Colm,
>
> Please find my comments inline.
>
> a) The official docker image uses 0.5.0-SNAPSHOT and not the released
> version.
> - this is because we uploaded docker image before apache release. actually
> this is same codebase apache-eagle-0.5, and it can be fixed easily by just
> rebuilding docker image. there should not be any mismatch due to this.
>
> b) Aside from the above, the official docker image uses a mix of "
> server.eagle.apache.org" and "sandbox.eagle.apache.org" as the host name.
> The HBase service doesn't start by default in Ambari as a result.
> - the only places it uses sandbox is in example script which you will have
> to update anyway, which i agree that it would be good to keep it
> consistent.
>
> c) The UI seems quite buggy. On both chromium and firefox, I only see
> links to "Sandbox" and "Alert" on the left hand-side. Once I click on
> "Alert" I have no way of going back to see the applications. I don't see
> the links to "integration" or "sites" as in the picture here:
> http://eagle.apache.org/docs/latest/applications/#jmx-monitoring
> - when hbase is as deep storage is used, and if eagle app has issue
> connecting to hbase, the UI becomes unresponsive.
>
> d) In chromium, the button to create a new policy does not exist - I can
> only see it on Firefox.
> - i have seen when you logged in, you will see admin actions. but if this
> still an issue, can you please file UI bug?
>
> e) I'm trying to get the "Hdfs Audit Log Monitor" use-case working, but it
> seems to be stuck in "Initialized".
> this eagle docs has example on how to setup the app. pls let us know if
> you find any gaps.
>
> Thanks for trying out, and sharing your findings,
> Jayesh
>
>
> On Tue, Jan 16, 2018 at 3:34 AM, Colm O hEigeartaigh <co...@apache.org>
> wrote:
>
>> Hi all,
>>
>> I'm trying to play around a bit with Apache Eagle 0.5.0 to no avail. Here
>> are the problems I've run into so far:
>>
>> a) The official docker image uses 0.5.0-SNAPSHOT and not the released
>> version.
>>
>> b) Aside from the above, the official docker image uses a mix of "
>> server.eagle.apache.org" and "sandbox.eagle.apache.org" as the host
>> name. The HBase service doesn't start by default in Ambari as a result.
>>
>> c) The UI seems quite buggy. On both chromium and firefox, I only see
>> links to "Sandbox" and "Alert" on the left hand-side. Once I click on
>> "Alert" I have no way of going back to see the applications. I don't see
>> the links to "integration" or "sites" as in the picture here:
>> http://eagle.apache.org/docs/latest/applications/#jmx-monitoring
>>
>> d) In chromium, the button to create a new policy does not exist - I can
>> only see it on Firefox.
>>
>> e) I'm trying to get the "Hdfs Audit Log Monitor" use-case working, but
>> it seems to be stuck in "Initialized".
>>
>> Could someone fill me in on what the "recommended" way is to start Apache
>> Eagle so that I can play around with the functionality that it offers?
>> Clearly the docker approach is buggy. Also, what browser should be used?
>>
>> Thanks,
>>
>> Colm.
>>
>>
>> --
>> Colm O hEigeartaigh
>>
>> Talend Community Coder
>> http://coders.talend.com
>>
>
>


-- 
Colm O hEigeartaigh

Talend Community Coder
http://coders.talend.com

Re: Unable to get 0.5.0 release working

Posted by Colm O hEigeartaigh <co...@apache.org>.
Hi Jayesh,

Many thanks for your feedback! I was able to make a little further headway.
There are two configuration problems with the official docker image:

a) A mix of "sandbox.eagle.apache.org" and "server.eagle.apache.org" (this
only occurs in the instructions for running the docker image. The version
that can be started via the script in the eagle source is OK). I'll submit
a PR to fix this once I get a basic use-case working.
b) For the audit case, it automatically logs HDFS audit logs to the KAFKA
topic sandbox_hdfs_audit_log instead of the expected hdfs_audit_log_sandbox

I've fixed these things locally and I can verify that everything is started
correctly in Ambari. I log into the docker container and create
hdfs_audit_log_sandbox and hdfs_audit_log_enriched_sandbox topics, and
verify that the HDFS audit logs are flowing into the first topic. Then in
the UI I start the Alert Engine and then the HDFS Audit Log Monitor
application (changing localhost:6667 to server.eagle.apache.org:6667). Both
applications start up correctly and show "running".

I then create a policy with an email alert along the lines of from
"HDFS_AUDIT_LOG_ENRICHED_STREAM_SANDBOX[str:contains(src,'/hbase')] select
* group by user insert into hdfs_audit_log_enriched_stream_out". However at
this point I'm stuck - nothing appears in the alert window. Is there
anything obvious I'm doing wrong, or how can I get access to logs to figure
out what the problem is? Other topics such as "hdfs_audit_event_sandbox"
are mentioned in the streams window, but the documentation doesn't say to
create them.

The UI is buggy though on both Firefox and Chromium on Linux. What
browser/platform are people using with the UI?

Colm.

On Wed, Jan 17, 2018 at 12:27 AM, Jayesh Senjaliya <ja...@apache.org>
wrote:

> Hi Colm,
>
> Please find my comments inline.
>
> a) The official docker image uses 0.5.0-SNAPSHOT and not the released
> version.
> - this is because we uploaded docker image before apache release. actually
> this is same codebase apache-eagle-0.5, and it can be fixed easily by just
> rebuilding docker image. there should not be any mismatch due to this.
>
> b) Aside from the above, the official docker image uses a mix of "
> server.eagle.apache.org" and "sandbox.eagle.apache.org" as the host name.
> The HBase service doesn't start by default in Ambari as a result.
> - the only places it uses sandbox is in example script which you will have
> to update anyway, which i agree that it would be good to keep it
> consistent.
>
> c) The UI seems quite buggy. On both chromium and firefox, I only see
> links to "Sandbox" and "Alert" on the left hand-side. Once I click on
> "Alert" I have no way of going back to see the applications. I don't see
> the links to "integration" or "sites" as in the picture here:
> http://eagle.apache.org/docs/latest/applications/#jmx-monitoring
> - when hbase is as deep storage is used, and if eagle app has issue
> connecting to hbase, the UI becomes unresponsive.
>
> d) In chromium, the button to create a new policy does not exist - I can
> only see it on Firefox.
> - i have seen when you logged in, you will see admin actions. but if this
> still an issue, can you please file UI bug?
>
> e) I'm trying to get the "Hdfs Audit Log Monitor" use-case working, but it
> seems to be stuck in "Initialized".
> this eagle docs has example on how to setup the app. pls let us know if
> you find any gaps.
>
> Thanks for trying out, and sharing your findings,
> Jayesh
>
>
> On Tue, Jan 16, 2018 at 3:34 AM, Colm O hEigeartaigh <co...@apache.org>
> wrote:
>
>> Hi all,
>>
>> I'm trying to play around a bit with Apache Eagle 0.5.0 to no avail. Here
>> are the problems I've run into so far:
>>
>> a) The official docker image uses 0.5.0-SNAPSHOT and not the released
>> version.
>>
>> b) Aside from the above, the official docker image uses a mix of "
>> server.eagle.apache.org" and "sandbox.eagle.apache.org" as the host
>> name. The HBase service doesn't start by default in Ambari as a result.
>>
>> c) The UI seems quite buggy. On both chromium and firefox, I only see
>> links to "Sandbox" and "Alert" on the left hand-side. Once I click on
>> "Alert" I have no way of going back to see the applications. I don't see
>> the links to "integration" or "sites" as in the picture here:
>> http://eagle.apache.org/docs/latest/applications/#jmx-monitoring
>>
>> d) In chromium, the button to create a new policy does not exist - I can
>> only see it on Firefox.
>>
>> e) I'm trying to get the "Hdfs Audit Log Monitor" use-case working, but
>> it seems to be stuck in "Initialized".
>>
>> Could someone fill me in on what the "recommended" way is to start Apache
>> Eagle so that I can play around with the functionality that it offers?
>> Clearly the docker approach is buggy. Also, what browser should be used?
>>
>> Thanks,
>>
>> Colm.
>>
>>
>> --
>> Colm O hEigeartaigh
>>
>> Talend Community Coder
>> http://coders.talend.com
>>
>
>


-- 
Colm O hEigeartaigh

Talend Community Coder
http://coders.talend.com

Re: Unable to get 0.5.0 release working

Posted by Jayesh Senjaliya <ja...@apache.org>.
Hi Colm,

Please find my comments inline.

a) The official docker image uses 0.5.0-SNAPSHOT and not the released
version.
- this is because we uploaded docker image before apache release. actually
this is same codebase apache-eagle-0.5, and it can be fixed easily by just
rebuilding docker image. there should not be any mismatch due to this.

b) Aside from the above, the official docker image uses a mix of "
server.eagle.apache.org" and "sandbox.eagle.apache.org" as the host name.
The HBase service doesn't start by default in Ambari as a result.
- the only places it uses sandbox is in example script which you will have
to update anyway, which i agree that it would be good to keep it
consistent.

c) The UI seems quite buggy. On both chromium and firefox, I only see links
to "Sandbox" and "Alert" on the left hand-side. Once I click on "Alert" I
have no way of going back to see the applications. I don't see the links to
"integration" or "sites" as in the picture here:
http://eagle.apache.org/docs/latest/applications/#jmx-monitoring
- when hbase is as deep storage is used, and if eagle app has issue
connecting to hbase, the UI becomes unresponsive.

d) In chromium, the button to create a new policy does not exist - I can
only see it on Firefox.
- i have seen when you logged in, you will see admin actions. but if this
still an issue, can you please file UI bug?

e) I'm trying to get the "Hdfs Audit Log Monitor" use-case working, but it
seems to be stuck in "Initialized".
this eagle docs has example on how to setup the app. pls let us know if you
find any gaps.

Thanks for trying out, and sharing your findings,
Jayesh


On Tue, Jan 16, 2018 at 3:34 AM, Colm O hEigeartaigh <co...@apache.org>
wrote:

> Hi all,
>
> I'm trying to play around a bit with Apache Eagle 0.5.0 to no avail. Here
> are the problems I've run into so far:
>
> a) The official docker image uses 0.5.0-SNAPSHOT and not the released
> version.
>
> b) Aside from the above, the official docker image uses a mix of "
> server.eagle.apache.org" and "sandbox.eagle.apache.org" as the host name.
> The HBase service doesn't start by default in Ambari as a result.
>
> c) The UI seems quite buggy. On both chromium and firefox, I only see
> links to "Sandbox" and "Alert" on the left hand-side. Once I click on
> "Alert" I have no way of going back to see the applications. I don't see
> the links to "integration" or "sites" as in the picture here:
> http://eagle.apache.org/docs/latest/applications/#jmx-monitoring
>
> d) In chromium, the button to create a new policy does not exist - I can
> only see it on Firefox.
>
> e) I'm trying to get the "Hdfs Audit Log Monitor" use-case working, but it
> seems to be stuck in "Initialized".
>
> Could someone fill me in on what the "recommended" way is to start Apache
> Eagle so that I can play around with the functionality that it offers?
> Clearly the docker approach is buggy. Also, what browser should be used?
>
> Thanks,
>
> Colm.
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>

Re: Unable to get 0.5.0 release working

Posted by Jayesh Senjaliya <ja...@apache.org>.
Hi Colm,

Please find my comments inline.

a) The official docker image uses 0.5.0-SNAPSHOT and not the released
version.
- this is because we uploaded docker image before apache release. actually
this is same codebase apache-eagle-0.5, and it can be fixed easily by just
rebuilding docker image. there should not be any mismatch due to this.

b) Aside from the above, the official docker image uses a mix of "
server.eagle.apache.org" and "sandbox.eagle.apache.org" as the host name.
The HBase service doesn't start by default in Ambari as a result.
- the only places it uses sandbox is in example script which you will have
to update anyway, which i agree that it would be good to keep it
consistent.

c) The UI seems quite buggy. On both chromium and firefox, I only see links
to "Sandbox" and "Alert" on the left hand-side. Once I click on "Alert" I
have no way of going back to see the applications. I don't see the links to
"integration" or "sites" as in the picture here:
http://eagle.apache.org/docs/latest/applications/#jmx-monitoring
- when hbase is as deep storage is used, and if eagle app has issue
connecting to hbase, the UI becomes unresponsive.

d) In chromium, the button to create a new policy does not exist - I can
only see it on Firefox.
- i have seen when you logged in, you will see admin actions. but if this
still an issue, can you please file UI bug?

e) I'm trying to get the "Hdfs Audit Log Monitor" use-case working, but it
seems to be stuck in "Initialized".
this eagle docs has example on how to setup the app. pls let us know if you
find any gaps.

Thanks for trying out, and sharing your findings,
Jayesh


On Tue, Jan 16, 2018 at 3:34 AM, Colm O hEigeartaigh <co...@apache.org>
wrote:

> Hi all,
>
> I'm trying to play around a bit with Apache Eagle 0.5.0 to no avail. Here
> are the problems I've run into so far:
>
> a) The official docker image uses 0.5.0-SNAPSHOT and not the released
> version.
>
> b) Aside from the above, the official docker image uses a mix of "
> server.eagle.apache.org" and "sandbox.eagle.apache.org" as the host name.
> The HBase service doesn't start by default in Ambari as a result.
>
> c) The UI seems quite buggy. On both chromium and firefox, I only see
> links to "Sandbox" and "Alert" on the left hand-side. Once I click on
> "Alert" I have no way of going back to see the applications. I don't see
> the links to "integration" or "sites" as in the picture here:
> http://eagle.apache.org/docs/latest/applications/#jmx-monitoring
>
> d) In chromium, the button to create a new policy does not exist - I can
> only see it on Firefox.
>
> e) I'm trying to get the "Hdfs Audit Log Monitor" use-case working, but it
> seems to be stuck in "Initialized".
>
> Could someone fill me in on what the "recommended" way is to start Apache
> Eagle so that I can play around with the functionality that it offers?
> Clearly the docker approach is buggy. Also, what browser should be used?
>
> Thanks,
>
> Colm.
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>