You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@falcon.apache.org by Adam Kawa <ka...@gmail.com> on 2015/01/10 20:34:12 UTC
Any roadmap for Falcon Web UI enhancements ?
Hi,
I have been extensively using Falcon for several months and I found it very
useful!
Once you adopt Falcon, however, and define tens of feeds and processes, it
becomes a bit inconvenient to track down what happens with your processes
using the current Falcon Web UI (yes, you can use Oozie Web UI for that)
and corresponding feeds.
I am curious if there is any roadmap for Falcon Web UI enhancements.
Use-cases that I would find useful include:
1. Being able to search feeds/processes by name, tag and status (
https://issues.apache.org/jira/secure/attachment/12686141/landing.png
provides search-box and drop-down lists, but a whole patch seems to be too
huge to be committed soon)
2. Being able to come to the office at 8 AM and after a quick look know
what processes aren't done yet (and feeds aren't delivered).
This could be achieved e.g. by adding a date of the last
successful/failed/running instance for each process on the process list and
an intuitive icon. Being able to distinguish between daily and hourly feeds
would be useful here.
3. Being able to sorting of the list of feeds/processes in some way e.g. by
name, status, owner, type or something that makes sense
Currently, you will not find a feed/process on the long list without
pressing CTRL+F first.
4. Instead of having a flat list, it would be able to see be useful to see
groups of feeds, if you use the <groups> tags in their XML definition
5. Being able to quickly learn more about my feed from other source of
information e.g. Hive Metastore or HDFS.
A simple integration with HUE could give some benefits and wouldn't be time
consuming e.g. if a feed is backed by a Hive table, then there could be a
link to HUE webpage that shows that table i.e. the same idea as with "Logs"
link next to the process instance that points to Oozie. Obviously this
feature would be displayed only if somebody configures HUE for Falcon.
--
Probably everyone can come up with some other ideas...
Please let me know if these features are valid for Falcon. I do realize
that the core idea of Falcon is to simplify data management and scheduling,
but it would be nice if Falcon provides a bit more feature-rich Web UI
instead of delegating to other tools (such as Oozie, CLI).
Cheers!
Adam
Re: Any roadmap for Falcon Web UI enhancements ?
Posted by Adam Kawa <ka...@gmail.com>.
Hi All!
Thanks a lot for many great and prompt responses!
hi Adam, please look at some screenshots i have uploaded to Falcon-790 that
> should address #1.
>
Yes, the landing page is nice
https://issues.apache.org/jira/secure/attachment/12686141/landing.png
because it provides the search feature. It's super useful!
I wonder if this patch can be split into two smaller patches (1 for landing
page with search features and 1 for write-access to entities), if this
helps to commit a new landing page to trunk sooner.
Being able to create/edit entities in the Web UI is really great, but in
our case, it's not a priority yet. We currently use Falcon to schedule and
manage our daily/hourly ETL and KPI processes and their feeds. To do so, we
have dedicated engineers (e.g. myself), who can create these XML files and
submit them to the Falcon server with small effort. Other people just look
at Falcon Web UI.
I guess that many many people (including Hadoop admins, data engineers and
data analysts) could benefit from the features like:
> lineage (already looks nice, thanks!),
> triage (https://issues.apache.org/jira/browse/FALCON-796 as Ajay Yadav
mentioned),
> being able to quickly find any feed and process on the list
> being able to have a quickly look at Web UI to check if some KPIs failed
during a night or are still being processed....
Being able to re-run an instance of the process from the Web UI, I would
find also very useful (the CLI for that is not that friendly, because you
must provide a timestamp and so). I created a JIRA for that in case it's
useful for other people as well
https://issues.apache.org/jira/browse/FALCON-986.
> - (5) Can you please explain it in more detail? You want only size, hdfs
location etc. or something more?
I can find a couple of use-cases:
1. the list of recently created partitions of the feed (initially, it can
be just a link to HUE so one can easily navigate back and forth).
2. the hint for the retention period for a dataset
In this talk
http://www.slideshare.net/AdamKawa/hadoop-operations-powered-by-hadoop-hadoop-summit-2014-amsterdam
(slides 53 and 57), I talked about the "empirical retention policy" for a
dataset. This could be (more or less) calculated using the last registered
access time kept by NameNode for each file (with the precision
of dfs.namenode.accesstime.precision). Let's say, you have a dataset where
a new directory is created each day. If you aggregate the information about
the last modification time and last registered access time for the each
file in the each directory, then you can get some insight about what
retention for a dataset should be. So you know, how many days after the
creation, an instance of the dataset is not accessed anymore... I can
provide an exemplary Python code for that as I have it implemented
somewhere... :)
This hint could be helpful to set a better retention period for a feed. In
practice, many owner of the dataset don't really know what the retention
should be and they specify too long periods.
3. automatic detection for the hot datasets
In the same talk (slides 57-61), I talked about hot datasets. Initially, it
could be as simple as counting at the number of processes that depend on
that feed, and use some colours (e.g. red, yellow, green) or numbers, to
distinguish popular feeds in some way
If the dataset is really hot, you might want to temporarily increase its
replication factor for the most recent instances (to benefit from the
higher data locality). Falcon might also take care adjusting HDFS
replication factor since it's data management... :)
> *P.S. *To see how we are thinking about monitoring dashboard you can take
a
look at the POC that I did at https://github.com/ajayyadav/falcon-dashboard
Currently this works on mocked API calls and we are working to provide the
backend APIs in falcon as of now.
Thanks for sharing, I will have a look!
Cheers!
Adam
Re: Any roadmap for Falcon Web UI enhancements ?
Posted by Kenneth Ho <kh...@hortonworks.com>.
hi Adam, please look at some screenshots i have uploaded to Falcon-790 that should address #1.
On Jan 10, 2015, at 11:34 AM, Adam Kawa <ka...@gmail.com> wrote:
> Hi,
>
> I have been extensively using Falcon for several months and I found it very
> useful!
>
> Once you adopt Falcon, however, and define tens of feeds and processes, it
> becomes a bit inconvenient to track down what happens with your processes
> using the current Falcon Web UI (yes, you can use Oozie Web UI for that)
> and corresponding feeds.
>
> I am curious if there is any roadmap for Falcon Web UI enhancements.
> Use-cases that I would find useful include:
>
> 1. Being able to search feeds/processes by name, tag and status (
> https://issues.apache.org/jira/secure/attachment/12686141/landing.png
> provides search-box and drop-down lists, but a whole patch seems to be too
> huge to be committed soon)
>
> 2. Being able to come to the office at 8 AM and after a quick look know
> what processes aren't done yet (and feeds aren't delivered).
>
> This could be achieved e.g. by adding a date of the last
> successful/failed/running instance for each process on the process list and
> an intuitive icon. Being able to distinguish between daily and hourly feeds
> would be useful here.
>
> 3. Being able to sorting of the list of feeds/processes in some way e.g. by
> name, status, owner, type or something that makes sense
>
> Currently, you will not find a feed/process on the long list without
> pressing CTRL+F first.
>
> 4. Instead of having a flat list, it would be able to see be useful to see
> groups of feeds, if you use the <groups> tags in their XML definition
>
> 5. Being able to quickly learn more about my feed from other source of
> information e.g. Hive Metastore or HDFS.
>
> A simple integration with HUE could give some benefits and wouldn't be time
> consuming e.g. if a feed is backed by a Hive table, then there could be a
> link to HUE webpage that shows that table i.e. the same idea as with "Logs"
> link next to the process instance that points to Oozie. Obviously this
> feature would be displayed only if somebody configures HUE for Falcon.
>
> --
>
> Probably everyone can come up with some other ideas...
>
> Please let me know if these features are valid for Falcon. I do realize
> that the core idea of Falcon is to simplify data management and scheduling,
> but it would be nice if Falcon provides a bit more feature-rich Web UI
> instead of delegating to other tools (such as Oozie, CLI).
>
> Cheers!
> Adam
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Any roadmap for Falcon Web UI enhancements ?
Posted by Ajay Yadav <aj...@gmail.com>.
Here are my thoughts on Adam's list
- (1) is already patch available and has some minor code review
comments. I think it will be available very soon.
- (2) We are still working to support monitoring in backend. After that
we will add it to the UI dashboard as well.
- (3) Can be added to UI dashboard if these cases are not solved as of
now.
- (4) This is an excellent suggestion. Another way is to have a pipeline
view of your system but that gives only processes and not feeds. We need to
leverage groups feature more.
- (5) Can you please explain it in more detail? You want only size, hdfs
location etc. or something more?
There is more exciting stuff like entity lineage
<https://issues.apache.org/jira/browse/FALCON-256>(already committed)
and instance
triage <https://issues.apache.org/jira/browse/FALCON-796>(in progress)
which is under works in backend and will be available in UI after that.
*P.S. *To see how we are thinking about monitoring dashboard you can take a
look at the POC that I did at https://github.com/ajayyadav/falcon-dashboard
Currently this works on mocked API calls and we are working to provide the
backend APIs in falcon as of now.
On Sun, Jan 11, 2015 at 6:13 AM, Srikanth Sundarrajan <sr...@hotmail.com>
wrote:
> -dev@falcon.incubator.apache.org
> +dev@falcon.apache.org
>
> Thanks for the feedback, we as a community are absolutely committed to
> making a UI that greatly simplifies administration, management and
> operations through Falcon. To this effect we wanted to create Falcon
> Administration & Management console. There are bunch of jiras already filed
> to enable various use cases relating to this.
>
> Will try and spend some time to piece them together and put forth for
> consideration. I would also like to point out that I am aware of some
> efforts from @Kenneth Ho (management features), @Sanjeev Tripurari
> (operations features) and @Ajay Yadav (operations features) in this regard
> and perhaps they can chime in and provide their thoughts.
>
> Regards
> Srikanth Sundarrajan
>
> > Date: Sat, 10 Jan 2015 20:34:12 +0100
> > Subject: Any roadmap for Falcon Web UI enhancements ?
> > From: kawa.adam@gmail.com
> > To: dev@falcon.incubator.apache.org
> >
> > Hi,
> >
> > I have been extensively using Falcon for several months and I found it
> very
> > useful!
> >
> > Once you adopt Falcon, however, and define tens of feeds and processes,
> it
> > becomes a bit inconvenient to track down what happens with your processes
> > using the current Falcon Web UI (yes, you can use Oozie Web UI for that)
> > and corresponding feeds.
> >
> > I am curious if there is any roadmap for Falcon Web UI enhancements.
> > Use-cases that I would find useful include:
> >
> > 1. Being able to search feeds/processes by name, tag and status (
> > https://issues.apache.org/jira/secure/attachment/12686141/landing.png
> > provides search-box and drop-down lists, but a whole patch seems to be
> too
> > huge to be committed soon)
> >
> > 2. Being able to come to the office at 8 AM and after a quick look know
> > what processes aren't done yet (and feeds aren't delivered).
> >
> > This could be achieved e.g. by adding a date of the last
> > successful/failed/running instance for each process on the process list
> and
> > an intuitive icon. Being able to distinguish between daily and hourly
> feeds
> > would be useful here.
> >
> > 3. Being able to sorting of the list of feeds/processes in some way e.g.
> by
> > name, status, owner, type or something that makes sense
> >
> > Currently, you will not find a feed/process on the long list without
> > pressing CTRL+F first.
> >
> > 4. Instead of having a flat list, it would be able to see be useful to
> see
> > groups of feeds, if you use the <groups> tags in their XML definition
> >
> > 5. Being able to quickly learn more about my feed from other source of
> > information e.g. Hive Metastore or HDFS.
> >
> > A simple integration with HUE could give some benefits and wouldn't be
> time
> > consuming e.g. if a feed is backed by a Hive table, then there could be a
> > link to HUE webpage that shows that table i.e. the same idea as with
> "Logs"
> > link next to the process instance that points to Oozie. Obviously this
> > feature would be displayed only if somebody configures HUE for Falcon.
> >
> > --
> >
> > Probably everyone can come up with some other ideas...
> >
> > Please let me know if these features are valid for Falcon. I do realize
> > that the core idea of Falcon is to simplify data management and
> scheduling,
> > but it would be nice if Falcon provides a bit more feature-rich Web UI
> > instead of delegating to other tools (such as Oozie, CLI).
> >
> > Cheers!
> > Adam
>
>
RE: Any roadmap for Falcon Web UI enhancements ?
Posted by Srikanth Sundarrajan <sr...@hotmail.com>.
Absolutely as a community we are very open and welcome to new contributions. To help you get started will urge you to look at http://apache.org/foundation/getinvolved.html for broad guidelines on how to get involved with any apache project and more specifically https://cwiki.apache.org/confluence/display/FALCON/How+To+Contribute on how you can contribute to Falcon.
Hope that helps. Should you need further info, please do feel free to write back to the dev list.
Regards
Srikanth Sundarrajan
> Date: Sun, 11 Jan 2015 13:57:50 +0530
> Subject: Re: Any roadmap for Falcon Web UI enhancements ?
> From: gzagarwal@gmail.com
> To: dev@falcon.apache.org
>
> Hello everyone
>
> I am also interested to commit for falcon, Is there any way that i can also
> start committing
>
> Thanks
>
> On Sun, Jan 11, 2015 at 6:13 AM, Srikanth Sundarrajan <sr...@hotmail.com>
> wrote:
>
> > -dev@falcon.incubator.apache.org
> > +dev@falcon.apache.org
> >
> > Thanks for the feedback, we as a community are absolutely committed to
> > making a UI that greatly simplifies administration, management and
> > operations through Falcon. To this effect we wanted to create Falcon
> > Administration & Management console. There are bunch of jiras already filed
> > to enable various use cases relating to this.
> >
> > Will try and spend some time to piece them together and put forth for
> > consideration. I would also like to point out that I am aware of some
> > efforts from @Kenneth Ho (management features), @Sanjeev Tripurari
> > (operations features) and @Ajay Yadav (operations features) in this regard
> > and perhaps they can chime in and provide their thoughts.
> >
> > Regards
> > Srikanth Sundarrajan
> >
> > > Date: Sat, 10 Jan 2015 20:34:12 +0100
> > > Subject: Any roadmap for Falcon Web UI enhancements ?
> > > From: kawa.adam@gmail.com
> > > To: dev@falcon.incubator.apache.org
> > >
> > > Hi,
> > >
> > > I have been extensively using Falcon for several months and I found it
> > very
> > > useful!
> > >
> > > Once you adopt Falcon, however, and define tens of feeds and processes,
> > it
> > > becomes a bit inconvenient to track down what happens with your processes
> > > using the current Falcon Web UI (yes, you can use Oozie Web UI for that)
> > > and corresponding feeds.
> > >
> > > I am curious if there is any roadmap for Falcon Web UI enhancements.
> > > Use-cases that I would find useful include:
> > >
> > > 1. Being able to search feeds/processes by name, tag and status (
> > > https://issues.apache.org/jira/secure/attachment/12686141/landing.png
> > > provides search-box and drop-down lists, but a whole patch seems to be
> > too
> > > huge to be committed soon)
> > >
> > > 2. Being able to come to the office at 8 AM and after a quick look know
> > > what processes aren't done yet (and feeds aren't delivered).
> > >
> > > This could be achieved e.g. by adding a date of the last
> > > successful/failed/running instance for each process on the process list
> > and
> > > an intuitive icon. Being able to distinguish between daily and hourly
> > feeds
> > > would be useful here.
> > >
> > > 3. Being able to sorting of the list of feeds/processes in some way e.g.
> > by
> > > name, status, owner, type or something that makes sense
> > >
> > > Currently, you will not find a feed/process on the long list without
> > > pressing CTRL+F first.
> > >
> > > 4. Instead of having a flat list, it would be able to see be useful to
> > see
> > > groups of feeds, if you use the <groups> tags in their XML definition
> > >
> > > 5. Being able to quickly learn more about my feed from other source of
> > > information e.g. Hive Metastore or HDFS.
> > >
> > > A simple integration with HUE could give some benefits and wouldn't be
> > time
> > > consuming e.g. if a feed is backed by a Hive table, then there could be a
> > > link to HUE webpage that shows that table i.e. the same idea as with
> > "Logs"
> > > link next to the process instance that points to Oozie. Obviously this
> > > feature would be displayed only if somebody configures HUE for Falcon.
> > >
> > > --
> > >
> > > Probably everyone can come up with some other ideas...
> > >
> > > Please let me know if these features are valid for Falcon. I do realize
> > > that the core idea of Falcon is to simplify data management and
> > scheduling,
> > > but it would be nice if Falcon provides a bit more feature-rich Web UI
> > > instead of delegating to other tools (such as Oozie, CLI).
> > >
> > > Cheers!
> > > Adam
> >
Re: Any roadmap for Falcon Web UI enhancements ?
Posted by Gaurav Agarwal <gz...@gmail.com>.
Hello everyone
I am also interested to commit for falcon, Is there any way that i can also
start committing
Thanks
On Sun, Jan 11, 2015 at 6:13 AM, Srikanth Sundarrajan <sr...@hotmail.com>
wrote:
> -dev@falcon.incubator.apache.org
> +dev@falcon.apache.org
>
> Thanks for the feedback, we as a community are absolutely committed to
> making a UI that greatly simplifies administration, management and
> operations through Falcon. To this effect we wanted to create Falcon
> Administration & Management console. There are bunch of jiras already filed
> to enable various use cases relating to this.
>
> Will try and spend some time to piece them together and put forth for
> consideration. I would also like to point out that I am aware of some
> efforts from @Kenneth Ho (management features), @Sanjeev Tripurari
> (operations features) and @Ajay Yadav (operations features) in this regard
> and perhaps they can chime in and provide their thoughts.
>
> Regards
> Srikanth Sundarrajan
>
> > Date: Sat, 10 Jan 2015 20:34:12 +0100
> > Subject: Any roadmap for Falcon Web UI enhancements ?
> > From: kawa.adam@gmail.com
> > To: dev@falcon.incubator.apache.org
> >
> > Hi,
> >
> > I have been extensively using Falcon for several months and I found it
> very
> > useful!
> >
> > Once you adopt Falcon, however, and define tens of feeds and processes,
> it
> > becomes a bit inconvenient to track down what happens with your processes
> > using the current Falcon Web UI (yes, you can use Oozie Web UI for that)
> > and corresponding feeds.
> >
> > I am curious if there is any roadmap for Falcon Web UI enhancements.
> > Use-cases that I would find useful include:
> >
> > 1. Being able to search feeds/processes by name, tag and status (
> > https://issues.apache.org/jira/secure/attachment/12686141/landing.png
> > provides search-box and drop-down lists, but a whole patch seems to be
> too
> > huge to be committed soon)
> >
> > 2. Being able to come to the office at 8 AM and after a quick look know
> > what processes aren't done yet (and feeds aren't delivered).
> >
> > This could be achieved e.g. by adding a date of the last
> > successful/failed/running instance for each process on the process list
> and
> > an intuitive icon. Being able to distinguish between daily and hourly
> feeds
> > would be useful here.
> >
> > 3. Being able to sorting of the list of feeds/processes in some way e.g.
> by
> > name, status, owner, type or something that makes sense
> >
> > Currently, you will not find a feed/process on the long list without
> > pressing CTRL+F first.
> >
> > 4. Instead of having a flat list, it would be able to see be useful to
> see
> > groups of feeds, if you use the <groups> tags in their XML definition
> >
> > 5. Being able to quickly learn more about my feed from other source of
> > information e.g. Hive Metastore or HDFS.
> >
> > A simple integration with HUE could give some benefits and wouldn't be
> time
> > consuming e.g. if a feed is backed by a Hive table, then there could be a
> > link to HUE webpage that shows that table i.e. the same idea as with
> "Logs"
> > link next to the process instance that points to Oozie. Obviously this
> > feature would be displayed only if somebody configures HUE for Falcon.
> >
> > --
> >
> > Probably everyone can come up with some other ideas...
> >
> > Please let me know if these features are valid for Falcon. I do realize
> > that the core idea of Falcon is to simplify data management and
> scheduling,
> > but it would be nice if Falcon provides a bit more feature-rich Web UI
> > instead of delegating to other tools (such as Oozie, CLI).
> >
> > Cheers!
> > Adam
>
RE: Any roadmap for Falcon Web UI enhancements ?
Posted by Srikanth Sundarrajan <sr...@hotmail.com>.
-dev@falcon.incubator.apache.org
+dev@falcon.apache.org
Thanks for the feedback, we as a community are absolutely committed to making a UI that greatly simplifies administration, management and operations through Falcon. To this effect we wanted to create Falcon Administration & Management console. There are bunch of jiras already filed to enable various use cases relating to this.
Will try and spend some time to piece them together and put forth for consideration. I would also like to point out that I am aware of some efforts from @Kenneth Ho (management features), @Sanjeev Tripurari (operations features) and @Ajay Yadav (operations features) in this regard and perhaps they can chime in and provide their thoughts.
Regards
Srikanth Sundarrajan
> Date: Sat, 10 Jan 2015 20:34:12 +0100
> Subject: Any roadmap for Falcon Web UI enhancements ?
> From: kawa.adam@gmail.com
> To: dev@falcon.incubator.apache.org
>
> Hi,
>
> I have been extensively using Falcon for several months and I found it very
> useful!
>
> Once you adopt Falcon, however, and define tens of feeds and processes, it
> becomes a bit inconvenient to track down what happens with your processes
> using the current Falcon Web UI (yes, you can use Oozie Web UI for that)
> and corresponding feeds.
>
> I am curious if there is any roadmap for Falcon Web UI enhancements.
> Use-cases that I would find useful include:
>
> 1. Being able to search feeds/processes by name, tag and status (
> https://issues.apache.org/jira/secure/attachment/12686141/landing.png
> provides search-box and drop-down lists, but a whole patch seems to be too
> huge to be committed soon)
>
> 2. Being able to come to the office at 8 AM and after a quick look know
> what processes aren't done yet (and feeds aren't delivered).
>
> This could be achieved e.g. by adding a date of the last
> successful/failed/running instance for each process on the process list and
> an intuitive icon. Being able to distinguish between daily and hourly feeds
> would be useful here.
>
> 3. Being able to sorting of the list of feeds/processes in some way e.g. by
> name, status, owner, type or something that makes sense
>
> Currently, you will not find a feed/process on the long list without
> pressing CTRL+F first.
>
> 4. Instead of having a flat list, it would be able to see be useful to see
> groups of feeds, if you use the <groups> tags in their XML definition
>
> 5. Being able to quickly learn more about my feed from other source of
> information e.g. Hive Metastore or HDFS.
>
> A simple integration with HUE could give some benefits and wouldn't be time
> consuming e.g. if a feed is backed by a Hive table, then there could be a
> link to HUE webpage that shows that table i.e. the same idea as with "Logs"
> link next to the process instance that points to Oozie. Obviously this
> feature would be displayed only if somebody configures HUE for Falcon.
>
> --
>
> Probably everyone can come up with some other ideas...
>
> Please let me know if these features are valid for Falcon. I do realize
> that the core idea of Falcon is to simplify data management and scheduling,
> but it would be nice if Falcon provides a bit more feature-rich Web UI
> instead of delegating to other tools (such as Oozie, CLI).
>
> Cheers!
> Adam