Replacing the JSP web UIs to HTML 5 applications

[Haohui Mai <hm...@hortonworks.com>]

Hi all,


Jing Zhao and I recently have reimplemented the JSP-based web UIs in HTML 5
applications (HDFS-5333). Based on our prelimanary testing results we
believe thst the new web UIs of the namenodes and the datanode are ready
for everyday uses.

You're more than welcome to try it out on trunk by visiting http://
<namenode>/dfshealth.html

There are a number of benefits from this transition. From a developer's
prospective, the most notable one is *maintainability*:

(1) The abstractions between the UI and the core server are well-defined,
decoupling the UI and the core hadoop servers.

(2) It allows us to deprecate the logic in the JSP pages. The old web UIs
have to duplicate the logic in the JSPs. The logic is often out-of-dated
and not well-tested, which leads to broken pages and security
vulnerabilities(e.g. HDFS-5251, HDFS-5307, HDFS-5308, HDFS-5317 and
HDFS-4901). The architecture of the new UIs prevent these bugs at the very
beginning.


I propose that deprecate the old, JSP-based web UIs in 2.3. I opened
HDFS-5402 to track the relevant discussions.

Your feedbacks are highly appreciated.


Sincerely,

Haohui

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Replacing the JSP web UIs to HTML 5 applications

[Haohui Mai <hm...@hortonworks.com>]

Hi Alejandro,

Can you please elaborate on producing JSON? All information presented in
the new Web UIs directly comes from the JMX side.

I'm okay with leaving the current JSP right now, since both the old and the
new Web UI can happily coexist.

When do you think it is a good time to switch the default web UI to the new
one, and to mark the JSP pages as deprecated?

Thanks,
Haohui


On Mon, Oct 28, 2013 at 3:57 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:

> Producing JSON would be great. Agree with Colin that we should leave for
> now the current JSP based web ui.
>
> thx
>
>
> On Mon, Oct 28, 2013 at 11:16 AM, Colin McCabe <cmccabe@alumni.cmu.edu
> >wrote:
>
> > This is a really interesting project, Haohui.  I think it will make
> > our web UI much nicer.
> >
> > I have a few concerns about removing the old web UI, however:
> >
> > * If we're going to remove the old web UI, I think the new web UI has
> > to have the same level of unit testing.  We shouldn't go backwards in
> > terms of unit testing.
> >
> > * Most of the deployments of elinks and links out there don't support
> > Javascript.  This is just a reality of life when using CentOS 5 or 6,
> > which many users are still using.  I have used "links" to diagnose
> > problems through the web UI in the past, in systems where access to
> > the cluster was available only through telnet.  If we are going to
> > remove this capability, we need to add some other command-line tools
> > to get the same functionality.  These tools could use REST if we have
> > that, or JMX, but they need to exist before we can consider removing
> > the old UI.
> >
> > best,
> > Colin
> >
> > On Fri, Oct 25, 2013 at 7:31 PM, Haohui Mai <hm...@hortonworks.com>
> wrote:
> > > Thanks for the reply, Luke. Here I just echo my response from the jira:
> > >
> > > bq. this client-side js only approach, which is less secure than a
> > > progressively enhanced hybrid approach used by YARN. The recent gmail
> > > XSS fiasco highlights the issue.
> > >
> > > I'm presenting an informal security analysis to compare the security of
> > the
> > > old and the new web UIs.
> > >
> > > An attacker launches an XSS attack by injecting malicious code which
> are
> > > usually HTML or JavaScript fragments into the web page, so that the
> > > malicious code can have the same privileges of the web page.
> > >
> > > First, in the scope of XSS attacks, note that the threat models of
> > > launching XSS attacks on Internet sites Gmail/Linkedin and the one of
> the
> > > Hadoop UIs are different. They have fundamental different sets of
> > external
> > > inputs that the attackers have control to. Internet sites have little
> > > control of these inputs. In the case of Gmail / Linkedin, an attack can
> > > send you a crafted e-mail, or put malicious description in his /
> > > her Linkedin profile. The sets of external inputs are *restricted* in
> > > Hadoop UIs. The new web UIs take JMX and WebHDFS as inputs. The
> > > attacker has to launch a XSS attack by:
> > >
> > > * Compromise the jars so that the output of JMX / WebHDFS have the
> > > malicious code.
> > > * Replace the web UIs completely to include the malicious code.
> > >
> > > In either case *the attacker has to compromise the hadoop core or the
> > > namenode*. That means the new web UIs are at least as secure as the
> > hadoop
> > > core, and the namenode machine.
> > >
> > > Second, I argue that using client-side templates are more secure than
> the
> > > current JSP-based server-side templates. To defend against XSS
> > > attacks, both techniques have to filter the external inputs at *every*
> > > possible execution paths. Several facts much be taken into
> > > plays when evaluating the security of both approaches in real-world
> > > environments:
> > >
> > > * The JavaScript libraries used in the new web UIs have survived in
> > > extremely large-scale production tests. jQuery is used by Google and
> > >  Microsoft, bootstrap is used by Twitter, and dust.js is used by
> > Linkedin.
> > > All libraries survived from hundreds of thousands of
> > >  attack attempts on a daily basis. I agree that the libraries might
> still
> > > be imperfect, but there's no way that we can test the JSP web
> > >  UIs to achieve the same level of assurances given the amount of
> > resources
> > > the community has.
> > > * Client-side templates consolidate all filtering logic in one central
> > > place. Recall that the goal is to filter all external inputs at every
> > >  execution paths, this is a much more systematic approach compared to
> the
> > > server-side templates we have today. It is difficult (if not
> > >  impossible) to do it in a JSP/ASP/PHP application, since such
> filtering
> > > can be only achieved via ad-hoc approaches ([1] shows some
> > >  empirical data). Also, HDFS-4901 recently describes a XSS
> vulnerability
> > in
> > > browseDirectory.jsp.
> > >
> > > bq. You'd require proper SSL (not self signed) setup to avoid JS
> > > injection
> > >
> > > Commodity browsers enforce Same-Origin Policy to defend against code
> > > injections. It has nothing to do with what kinds of SSL certificates
> > > you hold.
> > >
> > > bq.  I also have concerns that we commit these changes without matching
> > > unit tests
> > >
> > > The JavaScript code can be automatically tested. The same code can be
> run
> > > by node.js and the test can compared with pre-defined
> > > results. It is also possible to write an adapter to use Rhino to
> > accomplish
> > > the same task. We can discuss how to integrate them into
> > > the maven test routines in a different thread.
> > >
> > > bq. Client side rendering completely breaks the workflows for ops who
> > rely
> > > on text based terminal/emacs/vim browsers (no js support) to
> > > monitor component UI.
> > >
> > > links / elinks (http://elinks.or.cz/) are text-based web browsers that
> > > support JavaScript.
> > >
> > > bq. The priority/requirements for UI in core Hadoop should be security
> > and
> > > correctness, which client side templating cannot address properly
> > > so far.
> > >
> > > I agree that we should focus on security and correctness. The
> paragraphs
> > > above explain that how the architecture of the new UIs
> > > makes the UIs more secure in real-world settings compared to the UI we
> > have
> > > today.
> > >
> > > References:
> > >
> > > 1. A. Yip et al. Improving Application Security with Data Flow
> > Assertions.
> > > In SOSP'2009.
> > >
> > >
> > > On Fri, Oct 25, 2013 at 10:02 AM, Luke Lu <ll...@apache.org> wrote:
> > >
> > >> Echoing my comments on HDFS-3555:
> > >>
> > >> I have concerns with this client-side js only approach, which is less
> > >> secure than a progressively enhanced hybrid approach used by YARN. The
> > >> recent gmail XSS fiasco highlights the issue. I also have concerns
> that
> > we
> > >> commit these changes without matching unit tests – the fact you cannot
> > >> effectively unit test these changes should tell you something about
> this
> > >> approach.
> > >>
> > >> *Requiring* JS means that an admin cannot turn off js to (partially)
> use
> > >> core Hadoop UI. You'd *require* proper SSL (not self signed) setup to
> > avoid
> > >> JS injection, even if security of js libraries used is perfect, which
> I
> > >> doubt (search gmail/linkedin XSS). Client side rendering completely
> > breaks
> > >> the workflows for ops who rely on text based terminal/emacs/vim
> browsers
> > >> (no js support) to monitor component UI.
> > >>
> > >> IMO, JS-only rendering belongs to social networking sites and/or SaaS
> > >> front-ends, where full time UI/security specialists babysits UI
> > changes. I
> > >> think eventually most users will use a self servicing UI in a SaaS
> > >> front-end that uses REST/JMX API to get data from back-end components,
> > >> besides their own app master/service UI. The priority/requirements for
> > UI
> > >> in core Hadoop should be security and correctness, which client side
> > >> templating cannot address properly so far.
> > >>
> > >>
> > >> On Tue, Oct 22, 2013 at 3:59 PM, Haohui Mai <hm...@hortonworks.com>
> > wrote:
> > >>
> > >> > Hi all,
> > >> >
> > >> >
> > >> > Jing Zhao and I recently have reimplemented the JSP-based web UIs in
> > >> HTML 5
> > >> > applications (HDFS-5333). Based on our prelimanary testing results
> we
> > >> > believe thst the new web UIs of the namenodes and the datanode are
> > ready
> > >> > for everyday uses.
> > >> >
> > >> > You're more than welcome to try it out on trunk by visiting http://
> > >> > <namenode>/dfshealth.html
> > >> >
> > >> > There are a number of benefits from this transition. From a
> > developer's
> > >> > prospective, the most notable one is *maintainability*:
> > >> >
> > >> > (1) The abstractions between the UI and the core server are
> > well-defined,
> > >> > decoupling the UI and the core hadoop servers.
> > >> >
> > >> > (2) It allows us to deprecate the logic in the JSP pages. The old
> web
> > UIs
> > >> > have to duplicate the logic in the JSPs. The logic is often
> > out-of-dated
> > >> > and not well-tested, which leads to broken pages and security
> > >> > vulnerabilities(e.g. HDFS-5251, HDFS-5307, HDFS-5308, HDFS-5317 and
> > >> > HDFS-4901). The architecture of the new UIs prevent these bugs at
> the
> > >> very
> > >> > beginning.
> > >> >
> > >> >
> > >> > I propose that deprecate the old, JSP-based web UIs in 2.3. I opened
> > >> > HDFS-5402 to track the relevant discussions.
> > >> >
> > >> > Your feedbacks are highly appreciated.
> > >> >
> > >> >
> > >> > Sincerely,
> > >> >
> > >> > Haohui
> > >> >
> > >> > --
> > >> > CONFIDENTIALITY NOTICE
> > >> > NOTICE: This message is intended for the use of the individual or
> > entity
> > >> to
> > >> > which it is addressed and may contain information that is
> > confidential,
> > >> > privileged and exempt from disclosure under applicable law. If the
> > reader
> > >> > of this message is not the intended recipient, you are hereby
> notified
> > >> that
> > >> > any printing, copying, dissemination, distribution, disclosure or
> > >> > forwarding of this communication is strictly prohibited. If you have
> > >> > received this communication in error, please contact the sender
> > >> immediately
> > >> > and delete it from your system. Thank You.
> > >> >
> > >>
> > >
> > > --
> > > CONFIDENTIALITY NOTICE
> > > NOTICE: This message is intended for the use of the individual or
> entity
> > to
> > > which it is addressed and may contain information that is confidential,
> > > privileged and exempt from disclosure under applicable law. If the
> reader
> > > of this message is not the intended recipient, you are hereby notified
> > that
> > > any printing, copying, dissemination, distribution, disclosure or
> > > forwarding of this communication is strictly prohibited. If you have
> > > received this communication in error, please contact the sender
> > immediately
> > > and delete it from your system. Thank You.
> >
>
>
>
> --
> Alejandro
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Replacing the JSP web UIs to HTML 5 applications

[Larry McCay <lm...@hortonworks.com>]

So, if I understand correctly, we are using an HTTP based API to access JMX
data. The API isn't strictly designed to REST principals but does reflect
the management API and query parameters of JMX.

While unfortunate that we don't have a better REST based design for it, it
may not add enough value to add another API that would be less familiar to
those familiar with JMX.

I believe that this is the servlet in question:
https://hadoop.apache.org/docs/r1.0.4/api/org/apache/hadoop/jmx/JMXJsonServlet.html

Now, as far as gateway deployments, if/when we are able to get access to
the Web UIs from outside the Hadoop cluster and firewalls than all AJAX
calls will need to be routed through the gateway.

I think that this seems like a great step in the right direction of using
Web APIs for the Web UIs and may provide us with interesting ways to
provide additional security and auditing, etc.




On Mon, Oct 28, 2013 at 10:28 PM, Haohui Mai <hm...@hortonworks.com> wrote:

> Just to provide some data point to make the discussion concrete. Here is a
> part of the dump of the JMX information:
>
>
> curl "
> http://localhost:50070/jmx?qry=Hadoop:service=NameNode,name=NameNodeInfo"
>
> {
>   "beans" : [ {
>     "name" : "Hadoop:service=NameNode,name=NameNodeInfo",
>     "modelerType" : "org.apache.hadoop.hdfs.server.namenode.FSNamesystem",
>     "Threads" : 32,
>     "Used" : 0,
>     "Version" : "3.0.0-SNAPSHOT,
> rd56cd7ab85de00cfda62698e66bd6f0fef00ff61",
>     "Total" : 0,
>     "ClusterId" : "CID-ddaec89d-7801-40a9-b14c-f82d225746e1",
>     "PercentUsed" : 100.0,
>     "PercentRemaining" : 0.0,
>     "Free" : 0,
>     "Safemode" : "",
>    ...
> }
>
> The new web UI only extracts the values of the fields and renders them as
> HTML.
>
> Thanks,
> Haohui
>
>
> On Mon, Oct 28, 2013 at 6:12 PM, Larry McCay <lm...@hortonworks.com>
> wrote:
>
> > I think it is important that we make provisions for all Ajax calls to be
> > able to go through gateway deployments like Knox with the cluster
> > firewalled off.
> > As I have commented on the Jira, any calls that are currently on the
> > serverside but are moving to the browser will need to either require
> > punching holes in filewalls (unfortunate) or directing through a given
> REST
> > API gateway.
> >
> >
> > On Mon, Oct 28, 2013 at 9:09 PM, Haohui Mai <hm...@hortonworks.com>
> wrote:
> >
> > > Neither of them will go through JMX.
> > >
> > > The new Web UI implements hdfs browsing through WebHDFS.
> > >
> > > The logs are available through the static servlets, which is exactly
> the
> > > same as what we have today.
> > >
> > > Thanks,
> > > Haohui
> > >
> > >
> > > On Mon, Oct 28, 2013 at 6:01 PM, Alejandro Abdelnur <tucu@cloudera.com
> > > >wrote:
> > >
> > > > are you planning to expose things like hdfs browsing and nn/dn logs
> > over
> > > > jmx?
> > > >
> > > > thx
> > > >
> > > > Alejandro
> > > > (phone typing)
> > > >
> > > > On Oct 28, 2013, at 17:48, Haohui Mai <hm...@hortonworks.com> wrote:
> > > >
> > > > > It seems more appealing to me that the UI should JMX directly,
> > because:
> > > > >
> > > > > * We're support the JMX in the long term for other management
> > software.
> > > > > * The information provided by the JMX API will be mostly identical
> of
> > > the
> > > > > JSON API. Today the Web UI covers most of the information provided
> by
> > > > JMX.
> > > > > The Web UI does some trivial work to extract the information and
> > > renders
> > > > it
> > > > > as HTML.
> > > > > * We can compatibility and unit tests for free.
> > > > >
> > > > > I do agree that the JMX APIs are imperfect and they should be
> > revisited
> > > > in
> > > > > the 3.0 timeframe. However, this is orthogonal of the discussions
> of
> > > > > transitioning from JSP-based Web UI to client-side JavaScript Web
> UI.
> > > The
> > > > > architecture of the new Web UI allows easy migration to any
> > JSON-based
> > > > APIs
> > > > > whenever they land in the trunk.
> > > > >
> > > > > Thanks,
> > > > > Haohui
> > > > >
> > > > >
> > > > > On Mon, Oct 28, 2013 at 5:13 PM, Alejandro Abdelnur <
> > tucu@cloudera.com
> > > > >wrote:
> > > > >
> > > > >> Isn't using JMX to expose JSON for the web UI misusing JMX?
> > > > >>
> > > > >> I would think a more appropriate approach would be having /JMX for
> > > > >> monitoring integration and a /JSON end point for the UI data.
> > > > >>
> > > > >> Thanks.
> > > > >>
> > > > >>
> > > > >> On Mon, Oct 28, 2013 at 4:58 PM, Haohui Mai <hmai@hortonworks.com
> >
> > > > wrote:
> > > > >>
> > > > >>> Alejandro,
> > > > >>>
> > > > >>> If I understand correctly, that is the exact approach that the
> new
> > > web
> > > > UI
> > > > >>> is taking. The new web UI takes the output from JMX and renders
> > them
> > > as
> > > > >>> HTML at the client side.
> > > > >>>
> > > > >>> ~Haohui
> > > > >>>
> > > > >>>
> > > > >>> On Mon, Oct 28, 2013 at 4:18 PM, Alejandro Abdelnur <
> > > tucu@cloudera.com
> > > > >>>> wrote:
> > > > >>>
> > > > >>>> Haohui,
> > > > >>>>
> > > > >>>> If you have NN and DNs producing JSON instead HTML, then you can
> > > build
> > > > >> JS
> > > > >>>> based web UIs. Take for example Oozie, Oozie produces JSON, it
> > has a
> > > > >>> built
> > > > >>>> in JS web ui that consumes JSON and Hue has built an external
> web
> > UI
> > > > >> that
> > > > >>>> also consumes JSON. In the case of Hue UI, Oozie didn't have to
> > > change
> > > > >>>> anything to get that UI and improvements on the Hue UI don't
> > require
> > > > >>>> changes in Oozie unless it is to produce additional information.
> > > > >>>>
> > > > >>>> hope this clarifies.
> > > > >>>>
> > > > >>>> Thx
> > > > >>>>
> > > > >>>>
> > > > >>>> On Mon, Oct 28, 2013 at 4:06 PM, Haohui Mai <
> hmai@hortonworks.com
> > >
> > > > >>> wrote:
> > > > >>>>
> > > > >>>>> Echo my comments on HDFS-5402:
> > > > >>>>>
> > > > >>>>> bq. If we're going to remove the old web UI, I think the new
> web
> > UI
> > > > >> has
> > > > >>>>> to have the same level of unit testing. We shouldn't go
> backwards
> > > in
> > > > >>>>> terms of unit testing.
> > > > >>>>>
> > > > >>>>> I take a look at TestNamenodeJspHelper / TestDatanodeJspHelper
> /
> > > > >>>>> TestClusterJspHelper. It seems to me that we can merge these
> > tests
> > > > >> with
> > > > >>>> the
> > > > >>>>> unit tests on JMX.
> > > > >>>>>
> > > > >>>>> bq. If we are going to
> > > > >>>>> remove this capability, we need to add some other command-line
> > > tools
> > > > >>>>> to get the same functionality. These tools could use REST if we
> > > have
> > > > >>>>> that, or JMX, but they need to exist before we can consider
> > > removing
> > > > >>>>> the old UI.
> > > > >>>>>
> > > > >>>>> This is a good point. Since all information are available
> through
> > > > >> JMX,
> > > > >>>> the
> > > > >>>>> easiest way to approach it is to write some scripts using
> > Node.js.
> > > > >> The
> > > > >>>>> architecture of the new Web UIs is ready for this.
> > > > >>>>>
> > > > >>>>>
> > > > >>>>> On Mon, Oct 28, 2013 at 3:57 PM, Alejandro Abdelnur <
> > > > >> tucu@cloudera.com
> > > > >>>>>> wrote:
> > > > >>>>>
> > > > >>>>>> Producing JSON would be great. Agree with Colin that we should
> > > > >> leave
> > > > >>>> for
> > > > >>>>>> now the current JSP based web ui.
> > > > >>>>>>
> > > > >>>>>> thx
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>> On Mon, Oct 28, 2013 at 11:16 AM, Colin McCabe <
> > > > >>> cmccabe@alumni.cmu.edu
> > > > >>>>>>> wrote:
> > > > >>>>>>
> > > > >>>>>>> This is a really interesting project, Haohui.  I think it
> will
> > > > >> make
> > > > >>>>>>> our web UI much nicer.
> > > > >>>>>>>
> > > > >>>>>>> I have a few concerns about removing the old web UI, however:
> > > > >>>>>>>
> > > > >>>>>>> * If we're going to remove the old web UI, I think the new
> web
> > UI
> > > > >>> has
> > > > >>>>>>> to have the same level of unit testing.  We shouldn't go
> > > > >> backwards
> > > > >>> in
> > > > >>>>>>> terms of unit testing.
> > > > >>>>>>>
> > > > >>>>>>> * Most of the deployments of elinks and links out there don't
> > > > >>> support
> > > > >>>>>>> Javascript.  This is just a reality of life when using
> CentOS 5
> > > > >> or
> > > > >>> 6,
> > > > >>>>>>> which many users are still using.  I have used "links" to
> > > > >> diagnose
> > > > >>>>>>> problems through the web UI in the past, in systems where
> > access
> > > > >> to
> > > > >>>>>>> the cluster was available only through telnet.  If we are
> going
> > > > >> to
> > > > >>>>>>> remove this capability, we need to add some other
> command-line
> > > > >>> tools
> > > > >>>>>>> to get the same functionality.  These tools could use REST if
> > we
> > > > >>> have
> > > > >>>>>>> that, or JMX, but they need to exist before we can consider
> > > > >>> removing
> > > > >>>>>>> the old UI.
> > > > >>>>>>>
> > > > >>>>>>> best,
> > > > >>>>>>> Colin
> > > > >>>>>>>
> > > > >>>>>>> On Fri, Oct 25, 2013 at 7:31 PM, Haohui Mai <
> > > > >> hmai@hortonworks.com>
> > > > >>>>>> wrote:
> > > > >>>>>>>> Thanks for the reply, Luke. Here I just echo my response
> from
> > > > >> the
> > > > >>>>> jira:
> > > > >>>>>>>>
> > > > >>>>>>>> bq. this client-side js only approach, which is less secure
> > > > >> than
> > > > >>> a
> > > > >>>>>>>> progressively enhanced hybrid approach used by YARN. The
> > recent
> > > > >>>> gmail
> > > > >>>>>>>> XSS fiasco highlights the issue.
> > > > >>>>>>>>
> > > > >>>>>>>> I'm presenting an informal security analysis to compare the
> > > > >>>> security
> > > > >>>>> of
> > > > >>>>>>> the
> > > > >>>>>>>> old and the new web UIs.
> > > > >>>>>>>>
> > > > >>>>>>>> An attacker launches an XSS attack by injecting malicious
> code
> > > > >>>> which
> > > > >>>>>> are
> > > > >>>>>>>> usually HTML or JavaScript fragments into the web page, so
> > that
> > > > >>> the
> > > > >>>>>>>> malicious code can have the same privileges of the web page.
> > > > >>>>>>>>
> > > > >>>>>>>> First, in the scope of XSS attacks, note that the threat
> > models
> > > > >>> of
> > > > >>>>>>>> launching XSS attacks on Internet sites Gmail/Linkedin and
> the
> > > > >>> one
> > > > >>>> of
> > > > >>>>>> the
> > > > >>>>>>>> Hadoop UIs are different. They have fundamental different
> sets
> > > > >> of
> > > > >>>>>>> external
> > > > >>>>>>>> inputs that the attackers have control to. Internet sites
> have
> > > > >>>> little
> > > > >>>>>>>> control of these inputs. In the case of Gmail / Linkedin, an
> > > > >>> attack
> > > > >>>>> can
> > > > >>>>>>>> send you a crafted e-mail, or put malicious description in
> his
> > > > >> /
> > > > >>>>>>>> her Linkedin profile. The sets of external inputs are
> > > > >>> *restricted*
> > > > >>>> in
> > > > >>>>>>>> Hadoop UIs. The new web UIs take JMX and WebHDFS as inputs.
> > The
> > > > >>>>>>>> attacker has to launch a XSS attack by:
> > > > >>>>>>>>
> > > > >>>>>>>> * Compromise the jars so that the output of JMX / WebHDFS
> have
> > > > >>> the
> > > > >>>>>>>> malicious code.
> > > > >>>>>>>> * Replace the web UIs completely to include the malicious
> > code.
> > > > >>>>>>>>
> > > > >>>>>>>> In either case *the attacker has to compromise the hadoop
> core
> > > > >> or
> > > > >>>> the
> > > > >>>>>>>> namenode*. That means the new web UIs are at least as secure
> > as
> > > > >>> the
> > > > >>>>>>> hadoop
> > > > >>>>>>>> core, and the namenode machine.
> > > > >>>>>>>>
> > > > >>>>>>>> Second, I argue that using client-side templates are more
> > > > >> secure
> > > > >>>> than
> > > > >>>>>> the
> > > > >>>>>>>> current JSP-based server-side templates. To defend against
> XSS
> > > > >>>>>>>> attacks, both techniques have to filter the external inputs
> at
> > > > >>>>> *every*
> > > > >>>>>>>> possible execution paths. Several facts much be taken into
> > > > >>>>>>>> plays when evaluating the security of both approaches in
> > > > >>> real-world
> > > > >>>>>>>> environments:
> > > > >>>>>>>>
> > > > >>>>>>>> * The JavaScript libraries used in the new web UIs have
> > > > >> survived
> > > > >>> in
> > > > >>>>>>>> extremely large-scale production tests. jQuery is used by
> > > > >> Google
> > > > >>>> and
> > > > >>>>>>>> Microsoft, bootstrap is used by Twitter, and dust.js is used
> > > > >> by
> > > > >>>>>>> Linkedin.
> > > > >>>>>>>> All libraries survived from hundreds of thousands of
> > > > >>>>>>>> attack attempts on a daily basis. I agree that the libraries
> > > > >>> might
> > > > >>>>>> still
> > > > >>>>>>>> be imperfect, but there's no way that we can test the JSP
> web
> > > > >>>>>>>> UIs to achieve the same level of assurances given the amount
> > > > >> of
> > > > >>>>>>> resources
> > > > >>>>>>>> the community has.
> > > > >>>>>>>> * Client-side templates consolidate all filtering logic in
> one
> > > > >>>>> central
> > > > >>>>>>>> place. Recall that the goal is to filter all external inputs
> > at
> > > > >>>> every
> > > > >>>>>>>> execution paths, this is a much more systematic approach
> > > > >>> compared
> > > > >>>> to
> > > > >>>>>> the
> > > > >>>>>>>> server-side templates we have today. It is difficult (if not
> > > > >>>>>>>> impossible) to do it in a JSP/ASP/PHP application, since
> such
> > > > >>>>>> filtering
> > > > >>>>>>>> can be only achieved via ad-hoc approaches ([1] shows some
> > > > >>>>>>>> empirical data). Also, HDFS-4901 recently describes a XSS
> > > > >>>>>> vulnerability
> > > > >>>>>>> in
> > > > >>>>>>>> browseDirectory.jsp.
> > > > >>>>>>>>
> > > > >>>>>>>> bq. You'd require proper SSL (not self signed) setup to
> avoid
> > > > >> JS
> > > > >>>>>>>> injection
> > > > >>>>>>>>
> > > > >>>>>>>> Commodity browsers enforce Same-Origin Policy to defend
> > against
> > > > >>>> code
> > > > >>>>>>>> injections. It has nothing to do with what kinds of SSL
> > > > >>>> certificates
> > > > >>>>>>>> you hold.
> > > > >>>>>>>>
> > > > >>>>>>>> bq.  I also have concerns that we commit these changes
> without
> > > > >>>>> matching
> > > > >>>>>>>> unit tests
> > > > >>>>>>>>
> > > > >>>>>>>> The JavaScript code can be automatically tested. The same
> code
> > > > >>> can
> > > > >>>> be
> > > > >>>>>> run
> > > > >>>>>>>> by node.js and the test can compared with pre-defined
> > > > >>>>>>>> results. It is also possible to write an adapter to use
> Rhino
> > > > >> to
> > > > >>>>>>> accomplish
> > > > >>>>>>>> the same task. We can discuss how to integrate them into
> > > > >>>>>>>> the maven test routines in a different thread.
> > > > >>>>>>>>
> > > > >>>>>>>> bq. Client side rendering completely breaks the workflows
> for
> > > > >> ops
> > > > >>>> who
> > > > >>>>>>> rely
> > > > >>>>>>>> on text based terminal/emacs/vim browsers (no js support) to
> > > > >>>>>>>> monitor component UI.
> > > > >>>>>>>>
> > > > >>>>>>>> links / elinks (http://elinks.or.cz/) are text-based web
> > > > >>> browsers
> > > > >>>>> that
> > > > >>>>>>>> support JavaScript.
> > > > >>>>>>>>
> > > > >>>>>>>> bq. The priority/requirements for UI in core Hadoop should
> be
> > > > >>>>> security
> > > > >>>>>>> and
> > > > >>>>>>>> correctness, which client side templating cannot address
> > > > >> properly
> > > > >>>>>>>> so far.
> > > > >>>>>>>>
> > > > >>>>>>>> I agree that we should focus on security and correctness.
> The
> > > > >>>>>> paragraphs
> > > > >>>>>>>> above explain that how the architecture of the new UIs
> > > > >>>>>>>> makes the UIs more secure in real-world settings compared to
> > > > >> the
> > > > >>> UI
> > > > >>>>> we
> > > > >>>>>>> have
> > > > >>>>>>>> today.
> > > > >>>>>>>>
> > > > >>>>>>>> References:
> > > > >>>>>>>>
> > > > >>>>>>>> 1. A. Yip et al. Improving Application Security with Data
> Flow
> > > > >>>>>>> Assertions.
> > > > >>>>>>>> In SOSP'2009.
> > > > >>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>> On Fri, Oct 25, 2013 at 10:02 AM, Luke Lu <ll...@apache.org>
> > > > >>> wrote:
> > > > >>>>>>>>
> > > > >>>>>>>>> Echoing my comments on HDFS-3555:
> > > > >>>>>>>>>
> > > > >>>>>>>>> I have concerns with this client-side js only approach,
> which
> > > > >> is
> > > > >>>>> less
> > > > >>>>>>>>> secure than a progressively enhanced hybrid approach used
> by
> > > > >>> YARN.
> > > > >>>>> The
> > > > >>>>>>>>> recent gmail XSS fiasco highlights the issue. I also have
> > > > >>> concerns
> > > > >>>>>> that
> > > > >>>>>>> we
> > > > >>>>>>>>> commit these changes without matching unit tests – the fact
> > > > >> you
> > > > >>>>> cannot
> > > > >>>>>>>>> effectively unit test these changes should tell you
> something
> > > > >>>> about
> > > > >>>>>> this
> > > > >>>>>>>>> approach.
> > > > >>>>>>>>>
> > > > >>>>>>>>> *Requiring* JS means that an admin cannot turn off js to
> > > > >>>> (partially)
> > > > >>>>>> use
> > > > >>>>>>>>> core Hadoop UI. You'd *require* proper SSL (not self
> signed)
> > > > >>> setup
> > > > >>>>> to
> > > > >>>>>>> avoid
> > > > >>>>>>>>> JS injection, even if security of js libraries used is
> > > > >> perfect,
> > > > >>>>> which
> > > > >>>>>> I
> > > > >>>>>>>>> doubt (search gmail/linkedin XSS). Client side rendering
> > > > >>>> completely
> > > > >>>>>>> breaks
> > > > >>>>>>>>> the workflows for ops who rely on text based
> > > > >> terminal/emacs/vim
> > > > >>>>>> browsers
> > > > >>>>>>>>> (no js support) to monitor component UI.
> > > > >>>>>>>>>
> > > > >>>>>>>>> IMO, JS-only rendering belongs to social networking sites
> > > > >> and/or
> > > > >>>>> SaaS
> > > > >>>>>>>>> front-ends, where full time UI/security specialists
> babysits
> > > > >> UI
> > > > >>>>>>> changes. I
> > > > >>>>>>>>> think eventually most users will use a self servicing UI
> in a
> > > > >>> SaaS
> > > > >>>>>>>>> front-end that uses REST/JMX API to get data from back-end
> > > > >>>>> components,
> > > > >>>>>>>>> besides their own app master/service UI. The
> > > > >>> priority/requirements
> > > > >>>>> for
> > > > >>>>>>> UI
> > > > >>>>>>>>> in core Hadoop should be security and correctness, which
> > > > >> client
> > > > >>>> side
> > > > >>>>>>>>> templating cannot address properly so far.
> > > > >>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>>> On Tue, Oct 22, 2013 at 3:59 PM, Haohui Mai <
> > > > >>> hmai@hortonworks.com
> > > > >>>>>
> > > > >>>>>>> wrote:
> > > > >>>>>>>>>
> > > > >>>>>>>>>> Hi all,
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Jing Zhao and I recently have reimplemented the JSP-based
> > > > >> web
> > > > >>>> UIs
> > > > >>>>> in
> > > > >>>>>>>>> HTML 5
> > > > >>>>>>>>>> applications (HDFS-5333). Based on our prelimanary testing
> > > > >>>> results
> > > > >>>>>> we
> > > > >>>>>>>>>> believe thst the new web UIs of the namenodes and the
> > > > >> datanode
> > > > >>>> are
> > > > >>>>>>> ready
> > > > >>>>>>>>>> for everyday uses.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> You're more than welcome to try it out on trunk by
> visiting
> > > > >>>>> http://
> > > > >>>>>>>>>> <namenode>/dfshealth.html
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> There are a number of benefits from this transition. From
> a
> > > > >>>>>>> developer's
> > > > >>>>>>>>>> prospective, the most notable one is *maintainability*:
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> (1) The abstractions between the UI and the core server
> are
> > > > >>>>>>> well-defined,
> > > > >>>>>>>>>> decoupling the UI and the core hadoop servers.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> (2) It allows us to deprecate the logic in the JSP pages.
> > > > >> The
> > > > >>>> old
> > > > >>>>>> web
> > > > >>>>>>> UIs
> > > > >>>>>>>>>> have to duplicate the logic in the JSPs. The logic is
> often
> > > > >>>>>>> out-of-dated
> > > > >>>>>>>>>> and not well-tested, which leads to broken pages and
> > > > >> security
> > > > >>>>>>>>>> vulnerabilities(e.g. HDFS-5251, HDFS-5307, HDFS-5308,
> > > > >>> HDFS-5317
> > > > >>>>> and
> > > > >>>>>>>>>> HDFS-4901). The architecture of the new UIs prevent these
> > > > >> bugs
> > > > >>>> at
> > > > >>>>>> the
> > > > >>>>>>>>> very
> > > > >>>>>>>>>> beginning.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> I propose that deprecate the old, JSP-based web UIs in
> 2.3.
> > > > >> I
> > > > >>>>> opened
> > > > >>>>>>>>>> HDFS-5402 to track the relevant discussions.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Your feedbacks are highly appreciated.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Sincerely,
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Haohui
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> --
> > > > >>>>>>>>>> CONFIDENTIALITY NOTICE
> > > > >>>>>>>>>> NOTICE: This message is intended for the use of the
> > > > >> individual
> > > > >>>> or
> > > > >>>>>>> entity
> > > > >>>>>>>>> to
> > > > >>>>>>>>>> which it is addressed and may contain information that is
> > > > >>>>>>> confidential,
> > > > >>>>>>>>>> privileged and exempt from disclosure under applicable
> law.
> > > > >> If
> > > > >>>> the
> > > > >>>>>>> reader
> > > > >>>>>>>>>> of this message is not the intended recipient, you are
> > > > >> hereby
> > > > >>>>>> notified
> > > > >>>>>>>>> that
> > > > >>>>>>>>>> any printing, copying, dissemination, distribution,
> > > > >> disclosure
> > > > >>>> or
> > > > >>>>>>>>>> forwarding of this communication is strictly prohibited.
> If
> > > > >>> you
> > > > >>>>> have
> > > > >>>>>>>>>> received this communication in error, please contact the
> > > > >>> sender
> > > > >>>>>>>>> immediately
> > > > >>>>>>>>>> and delete it from your system. Thank You.
> > > > >>>>>>>>
> > > > >>>>>>>> --
> > > > >>>>>>>> CONFIDENTIALITY NOTICE
> > > > >>>>>>>> NOTICE: This message is intended for the use of the
> individual
> > > > >> or
> > > > >>>>>> entity
> > > > >>>>>>> to
> > > > >>>>>>>> which it is addressed and may contain information that is
> > > > >>>>> confidential,
> > > > >>>>>>>> privileged and exempt from disclosure under applicable law.
> If
> > > > >>> the
> > > > >>>>>> reader
> > > > >>>>>>>> of this message is not the intended recipient, you are
> hereby
> > > > >>>>> notified
> > > > >>>>>>> that
> > > > >>>>>>>> any printing, copying, dissemination, distribution,
> disclosure
> > > > >> or
> > > > >>>>>>>> forwarding of this communication is strictly prohibited. If
> > you
> > > > >>>> have
> > > > >>>>>>>> received this communication in error, please contact the
> > sender
> > > > >>>>>>> immediately
> > > > >>>>>>>> and delete it from your system. Thank You.
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>> --
> > > > >>>>>> Alejandro
> > > > >>>>>
> > > > >>>>> --
> > > > >>>>> CONFIDENTIALITY NOTICE
> > > > >>>>> NOTICE: This message is intended for the use of the individual
> or
> > > > >>> entity
> > > > >>>> to
> > > > >>>>> which it is addressed and may contain information that is
> > > > >> confidential,
> > > > >>>>> privileged and exempt from disclosure under applicable law. If
> > the
> > > > >>> reader
> > > > >>>>> of this message is not the intended recipient, you are hereby
> > > > >> notified
> > > > >>>> that
> > > > >>>>> any printing, copying, dissemination, distribution, disclosure
> or
> > > > >>>>> forwarding of this communication is strictly prohibited. If you
> > > have
> > > > >>>>> received this communication in error, please contact the sender
> > > > >>>> immediately
> > > > >>>>> and delete it from your system. Thank You.
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>> --
> > > > >>>> Alejandro
> > > > >>>
> > > > >>> --
> > > > >>> CONFIDENTIALITY NOTICE
> > > > >>> NOTICE: This message is intended for the use of the individual or
> > > > entity
> > > > >> to
> > > > >>> which it is addressed and may contain information that is
> > > confidential,
> > > > >>> privileged and exempt from disclosure under applicable law. If
> the
> > > > reader
> > > > >>> of this message is not the intended recipient, you are hereby
> > > notified
> > > > >> that
> > > > >>> any printing, copying, dissemination, distribution, disclosure or
> > > > >>> forwarding of this communication is strictly prohibited. If you
> > have
> > > > >>> received this communication in error, please contact the sender
> > > > >> immediately
> > > > >>> and delete it from your system. Thank You.
> > > > >>
> > > > >>
> > > > >>
> > > > >> --
> > > > >> Alejandro
> > > > >
> > > > > --
> > > > > CONFIDENTIALITY NOTICE
> > > > > NOTICE: This message is intended for the use of the individual or
> > > entity
> > > > to
> > > > > which it is addressed and may contain information that is
> > confidential,
> > > > > privileged and exempt from disclosure under applicable law. If the
> > > reader
> > > > > of this message is not the intended recipient, you are hereby
> > notified
> > > > that
> > > > > any printing, copying, dissemination, distribution, disclosure or
> > > > > forwarding of this communication is strictly prohibited. If you
> have
> > > > > received this communication in error, please contact the sender
> > > > immediately
> > > > > and delete it from your system. Thank You.
> > > >
> > >
> > > --
> > > CONFIDENTIALITY NOTICE
> > > NOTICE: This message is intended for the use of the individual or
> entity
> > to
> > > which it is addressed and may contain information that is confidential,
> > > privileged and exempt from disclosure under applicable law. If the
> reader
> > > of this message is not the intended recipient, you are hereby notified
> > that
> > > any printing, copying, dissemination, distribution, disclosure or
> > > forwarding of this communication is strictly prohibited. If you have
> > > received this communication in error, please contact the sender
> > immediately
> > > and delete it from your system. Thank You.
> > >
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Replacing the JSP web UIs to HTML 5 applications

[Haohui Mai <hm...@hortonworks.com>]

Just to provide some data point to make the discussion concrete. Here is a
part of the dump of the JMX information:


curl "
http://localhost:50070/jmx?qry=Hadoop:service=NameNode,name=NameNodeInfo"

{
  "beans" : [ {
    "name" : "Hadoop:service=NameNode,name=NameNodeInfo",
    "modelerType" : "org.apache.hadoop.hdfs.server.namenode.FSNamesystem",
    "Threads" : 32,
    "Used" : 0,
    "Version" : "3.0.0-SNAPSHOT, rd56cd7ab85de00cfda62698e66bd6f0fef00ff61",
    "Total" : 0,
    "ClusterId" : "CID-ddaec89d-7801-40a9-b14c-f82d225746e1",
    "PercentUsed" : 100.0,
    "PercentRemaining" : 0.0,
    "Free" : 0,
    "Safemode" : "",
   ...
}

The new web UI only extracts the values of the fields and renders them as
HTML.

Thanks,
Haohui


On Mon, Oct 28, 2013 at 6:12 PM, Larry McCay <lm...@hortonworks.com> wrote:

> I think it is important that we make provisions for all Ajax calls to be
> able to go through gateway deployments like Knox with the cluster
> firewalled off.
> As I have commented on the Jira, any calls that are currently on the
> serverside but are moving to the browser will need to either require
> punching holes in filewalls (unfortunate) or directing through a given REST
> API gateway.
>
>
> On Mon, Oct 28, 2013 at 9:09 PM, Haohui Mai <hm...@hortonworks.com> wrote:
>
> > Neither of them will go through JMX.
> >
> > The new Web UI implements hdfs browsing through WebHDFS.
> >
> > The logs are available through the static servlets, which is exactly the
> > same as what we have today.
> >
> > Thanks,
> > Haohui
> >
> >
> > On Mon, Oct 28, 2013 at 6:01 PM, Alejandro Abdelnur <tucu@cloudera.com
> > >wrote:
> >
> > > are you planning to expose things like hdfs browsing and nn/dn logs
> over
> > > jmx?
> > >
> > > thx
> > >
> > > Alejandro
> > > (phone typing)
> > >
> > > On Oct 28, 2013, at 17:48, Haohui Mai <hm...@hortonworks.com> wrote:
> > >
> > > > It seems more appealing to me that the UI should JMX directly,
> because:
> > > >
> > > > * We're support the JMX in the long term for other management
> software.
> > > > * The information provided by the JMX API will be mostly identical of
> > the
> > > > JSON API. Today the Web UI covers most of the information provided by
> > > JMX.
> > > > The Web UI does some trivial work to extract the information and
> > renders
> > > it
> > > > as HTML.
> > > > * We can compatibility and unit tests for free.
> > > >
> > > > I do agree that the JMX APIs are imperfect and they should be
> revisited
> > > in
> > > > the 3.0 timeframe. However, this is orthogonal of the discussions of
> > > > transitioning from JSP-based Web UI to client-side JavaScript Web UI.
> > The
> > > > architecture of the new Web UI allows easy migration to any
> JSON-based
> > > APIs
> > > > whenever they land in the trunk.
> > > >
> > > > Thanks,
> > > > Haohui
> > > >
> > > >
> > > > On Mon, Oct 28, 2013 at 5:13 PM, Alejandro Abdelnur <
> tucu@cloudera.com
> > > >wrote:
> > > >
> > > >> Isn't using JMX to expose JSON for the web UI misusing JMX?
> > > >>
> > > >> I would think a more appropriate approach would be having /JMX for
> > > >> monitoring integration and a /JSON end point for the UI data.
> > > >>
> > > >> Thanks.
> > > >>
> > > >>
> > > >> On Mon, Oct 28, 2013 at 4:58 PM, Haohui Mai <hm...@hortonworks.com>
> > > wrote:
> > > >>
> > > >>> Alejandro,
> > > >>>
> > > >>> If I understand correctly, that is the exact approach that the new
> > web
> > > UI
> > > >>> is taking. The new web UI takes the output from JMX and renders
> them
> > as
> > > >>> HTML at the client side.
> > > >>>
> > > >>> ~Haohui
> > > >>>
> > > >>>
> > > >>> On Mon, Oct 28, 2013 at 4:18 PM, Alejandro Abdelnur <
> > tucu@cloudera.com
> > > >>>> wrote:
> > > >>>
> > > >>>> Haohui,
> > > >>>>
> > > >>>> If you have NN and DNs producing JSON instead HTML, then you can
> > build
> > > >> JS
> > > >>>> based web UIs. Take for example Oozie, Oozie produces JSON, it
> has a
> > > >>> built
> > > >>>> in JS web ui that consumes JSON and Hue has built an external web
> UI
> > > >> that
> > > >>>> also consumes JSON. In the case of Hue UI, Oozie didn't have to
> > change
> > > >>>> anything to get that UI and improvements on the Hue UI don't
> require
> > > >>>> changes in Oozie unless it is to produce additional information.
> > > >>>>
> > > >>>> hope this clarifies.
> > > >>>>
> > > >>>> Thx
> > > >>>>
> > > >>>>
> > > >>>> On Mon, Oct 28, 2013 at 4:06 PM, Haohui Mai <hmai@hortonworks.com
> >
> > > >>> wrote:
> > > >>>>
> > > >>>>> Echo my comments on HDFS-5402:
> > > >>>>>
> > > >>>>> bq. If we're going to remove the old web UI, I think the new web
> UI
> > > >> has
> > > >>>>> to have the same level of unit testing. We shouldn't go backwards
> > in
> > > >>>>> terms of unit testing.
> > > >>>>>
> > > >>>>> I take a look at TestNamenodeJspHelper / TestDatanodeJspHelper /
> > > >>>>> TestClusterJspHelper. It seems to me that we can merge these
> tests
> > > >> with
> > > >>>> the
> > > >>>>> unit tests on JMX.
> > > >>>>>
> > > >>>>> bq. If we are going to
> > > >>>>> remove this capability, we need to add some other command-line
> > tools
> > > >>>>> to get the same functionality. These tools could use REST if we
> > have
> > > >>>>> that, or JMX, but they need to exist before we can consider
> > removing
> > > >>>>> the old UI.
> > > >>>>>
> > > >>>>> This is a good point. Since all information are available through
> > > >> JMX,
> > > >>>> the
> > > >>>>> easiest way to approach it is to write some scripts using
> Node.js.
> > > >> The
> > > >>>>> architecture of the new Web UIs is ready for this.
> > > >>>>>
> > > >>>>>
> > > >>>>> On Mon, Oct 28, 2013 at 3:57 PM, Alejandro Abdelnur <
> > > >> tucu@cloudera.com
> > > >>>>>> wrote:
> > > >>>>>
> > > >>>>>> Producing JSON would be great. Agree with Colin that we should
> > > >> leave
> > > >>>> for
> > > >>>>>> now the current JSP based web ui.
> > > >>>>>>
> > > >>>>>> thx
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> On Mon, Oct 28, 2013 at 11:16 AM, Colin McCabe <
> > > >>> cmccabe@alumni.cmu.edu
> > > >>>>>>> wrote:
> > > >>>>>>
> > > >>>>>>> This is a really interesting project, Haohui.  I think it will
> > > >> make
> > > >>>>>>> our web UI much nicer.
> > > >>>>>>>
> > > >>>>>>> I have a few concerns about removing the old web UI, however:
> > > >>>>>>>
> > > >>>>>>> * If we're going to remove the old web UI, I think the new web
> UI
> > > >>> has
> > > >>>>>>> to have the same level of unit testing.  We shouldn't go
> > > >> backwards
> > > >>> in
> > > >>>>>>> terms of unit testing.
> > > >>>>>>>
> > > >>>>>>> * Most of the deployments of elinks and links out there don't
> > > >>> support
> > > >>>>>>> Javascript.  This is just a reality of life when using CentOS 5
> > > >> or
> > > >>> 6,
> > > >>>>>>> which many users are still using.  I have used "links" to
> > > >> diagnose
> > > >>>>>>> problems through the web UI in the past, in systems where
> access
> > > >> to
> > > >>>>>>> the cluster was available only through telnet.  If we are going
> > > >> to
> > > >>>>>>> remove this capability, we need to add some other command-line
> > > >>> tools
> > > >>>>>>> to get the same functionality.  These tools could use REST if
> we
> > > >>> have
> > > >>>>>>> that, or JMX, but they need to exist before we can consider
> > > >>> removing
> > > >>>>>>> the old UI.
> > > >>>>>>>
> > > >>>>>>> best,
> > > >>>>>>> Colin
> > > >>>>>>>
> > > >>>>>>> On Fri, Oct 25, 2013 at 7:31 PM, Haohui Mai <
> > > >> hmai@hortonworks.com>
> > > >>>>>> wrote:
> > > >>>>>>>> Thanks for the reply, Luke. Here I just echo my response from
> > > >> the
> > > >>>>> jira:
> > > >>>>>>>>
> > > >>>>>>>> bq. this client-side js only approach, which is less secure
> > > >> than
> > > >>> a
> > > >>>>>>>> progressively enhanced hybrid approach used by YARN. The
> recent
> > > >>>> gmail
> > > >>>>>>>> XSS fiasco highlights the issue.
> > > >>>>>>>>
> > > >>>>>>>> I'm presenting an informal security analysis to compare the
> > > >>>> security
> > > >>>>> of
> > > >>>>>>> the
> > > >>>>>>>> old and the new web UIs.
> > > >>>>>>>>
> > > >>>>>>>> An attacker launches an XSS attack by injecting malicious code
> > > >>>> which
> > > >>>>>> are
> > > >>>>>>>> usually HTML or JavaScript fragments into the web page, so
> that
> > > >>> the
> > > >>>>>>>> malicious code can have the same privileges of the web page.
> > > >>>>>>>>
> > > >>>>>>>> First, in the scope of XSS attacks, note that the threat
> models
> > > >>> of
> > > >>>>>>>> launching XSS attacks on Internet sites Gmail/Linkedin and the
> > > >>> one
> > > >>>> of
> > > >>>>>> the
> > > >>>>>>>> Hadoop UIs are different. They have fundamental different sets
> > > >> of
> > > >>>>>>> external
> > > >>>>>>>> inputs that the attackers have control to. Internet sites have
> > > >>>> little
> > > >>>>>>>> control of these inputs. In the case of Gmail / Linkedin, an
> > > >>> attack
> > > >>>>> can
> > > >>>>>>>> send you a crafted e-mail, or put malicious description in his
> > > >> /
> > > >>>>>>>> her Linkedin profile. The sets of external inputs are
> > > >>> *restricted*
> > > >>>> in
> > > >>>>>>>> Hadoop UIs. The new web UIs take JMX and WebHDFS as inputs.
> The
> > > >>>>>>>> attacker has to launch a XSS attack by:
> > > >>>>>>>>
> > > >>>>>>>> * Compromise the jars so that the output of JMX / WebHDFS have
> > > >>> the
> > > >>>>>>>> malicious code.
> > > >>>>>>>> * Replace the web UIs completely to include the malicious
> code.
> > > >>>>>>>>
> > > >>>>>>>> In either case *the attacker has to compromise the hadoop core
> > > >> or
> > > >>>> the
> > > >>>>>>>> namenode*. That means the new web UIs are at least as secure
> as
> > > >>> the
> > > >>>>>>> hadoop
> > > >>>>>>>> core, and the namenode machine.
> > > >>>>>>>>
> > > >>>>>>>> Second, I argue that using client-side templates are more
> > > >> secure
> > > >>>> than
> > > >>>>>> the
> > > >>>>>>>> current JSP-based server-side templates. To defend against XSS
> > > >>>>>>>> attacks, both techniques have to filter the external inputs at
> > > >>>>> *every*
> > > >>>>>>>> possible execution paths. Several facts much be taken into
> > > >>>>>>>> plays when evaluating the security of both approaches in
> > > >>> real-world
> > > >>>>>>>> environments:
> > > >>>>>>>>
> > > >>>>>>>> * The JavaScript libraries used in the new web UIs have
> > > >> survived
> > > >>> in
> > > >>>>>>>> extremely large-scale production tests. jQuery is used by
> > > >> Google
> > > >>>> and
> > > >>>>>>>> Microsoft, bootstrap is used by Twitter, and dust.js is used
> > > >> by
> > > >>>>>>> Linkedin.
> > > >>>>>>>> All libraries survived from hundreds of thousands of
> > > >>>>>>>> attack attempts on a daily basis. I agree that the libraries
> > > >>> might
> > > >>>>>> still
> > > >>>>>>>> be imperfect, but there's no way that we can test the JSP web
> > > >>>>>>>> UIs to achieve the same level of assurances given the amount
> > > >> of
> > > >>>>>>> resources
> > > >>>>>>>> the community has.
> > > >>>>>>>> * Client-side templates consolidate all filtering logic in one
> > > >>>>> central
> > > >>>>>>>> place. Recall that the goal is to filter all external inputs
> at
> > > >>>> every
> > > >>>>>>>> execution paths, this is a much more systematic approach
> > > >>> compared
> > > >>>> to
> > > >>>>>> the
> > > >>>>>>>> server-side templates we have today. It is difficult (if not
> > > >>>>>>>> impossible) to do it in a JSP/ASP/PHP application, since such
> > > >>>>>> filtering
> > > >>>>>>>> can be only achieved via ad-hoc approaches ([1] shows some
> > > >>>>>>>> empirical data). Also, HDFS-4901 recently describes a XSS
> > > >>>>>> vulnerability
> > > >>>>>>> in
> > > >>>>>>>> browseDirectory.jsp.
> > > >>>>>>>>
> > > >>>>>>>> bq. You'd require proper SSL (not self signed) setup to avoid
> > > >> JS
> > > >>>>>>>> injection
> > > >>>>>>>>
> > > >>>>>>>> Commodity browsers enforce Same-Origin Policy to defend
> against
> > > >>>> code
> > > >>>>>>>> injections. It has nothing to do with what kinds of SSL
> > > >>>> certificates
> > > >>>>>>>> you hold.
> > > >>>>>>>>
> > > >>>>>>>> bq.  I also have concerns that we commit these changes without
> > > >>>>> matching
> > > >>>>>>>> unit tests
> > > >>>>>>>>
> > > >>>>>>>> The JavaScript code can be automatically tested. The same code
> > > >>> can
> > > >>>> be
> > > >>>>>> run
> > > >>>>>>>> by node.js and the test can compared with pre-defined
> > > >>>>>>>> results. It is also possible to write an adapter to use Rhino
> > > >> to
> > > >>>>>>> accomplish
> > > >>>>>>>> the same task. We can discuss how to integrate them into
> > > >>>>>>>> the maven test routines in a different thread.
> > > >>>>>>>>
> > > >>>>>>>> bq. Client side rendering completely breaks the workflows for
> > > >> ops
> > > >>>> who
> > > >>>>>>> rely
> > > >>>>>>>> on text based terminal/emacs/vim browsers (no js support) to
> > > >>>>>>>> monitor component UI.
> > > >>>>>>>>
> > > >>>>>>>> links / elinks (http://elinks.or.cz/) are text-based web
> > > >>> browsers
> > > >>>>> that
> > > >>>>>>>> support JavaScript.
> > > >>>>>>>>
> > > >>>>>>>> bq. The priority/requirements for UI in core Hadoop should be
> > > >>>>> security
> > > >>>>>>> and
> > > >>>>>>>> correctness, which client side templating cannot address
> > > >> properly
> > > >>>>>>>> so far.
> > > >>>>>>>>
> > > >>>>>>>> I agree that we should focus on security and correctness. The
> > > >>>>>> paragraphs
> > > >>>>>>>> above explain that how the architecture of the new UIs
> > > >>>>>>>> makes the UIs more secure in real-world settings compared to
> > > >> the
> > > >>> UI
> > > >>>>> we
> > > >>>>>>> have
> > > >>>>>>>> today.
> > > >>>>>>>>
> > > >>>>>>>> References:
> > > >>>>>>>>
> > > >>>>>>>> 1. A. Yip et al. Improving Application Security with Data Flow
> > > >>>>>>> Assertions.
> > > >>>>>>>> In SOSP'2009.
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>> On Fri, Oct 25, 2013 at 10:02 AM, Luke Lu <ll...@apache.org>
> > > >>> wrote:
> > > >>>>>>>>
> > > >>>>>>>>> Echoing my comments on HDFS-3555:
> > > >>>>>>>>>
> > > >>>>>>>>> I have concerns with this client-side js only approach, which
> > > >> is
> > > >>>>> less
> > > >>>>>>>>> secure than a progressively enhanced hybrid approach used by
> > > >>> YARN.
> > > >>>>> The
> > > >>>>>>>>> recent gmail XSS fiasco highlights the issue. I also have
> > > >>> concerns
> > > >>>>>> that
> > > >>>>>>> we
> > > >>>>>>>>> commit these changes without matching unit tests – the fact
> > > >> you
> > > >>>>> cannot
> > > >>>>>>>>> effectively unit test these changes should tell you something
> > > >>>> about
> > > >>>>>> this
> > > >>>>>>>>> approach.
> > > >>>>>>>>>
> > > >>>>>>>>> *Requiring* JS means that an admin cannot turn off js to
> > > >>>> (partially)
> > > >>>>>> use
> > > >>>>>>>>> core Hadoop UI. You'd *require* proper SSL (not self signed)
> > > >>> setup
> > > >>>>> to
> > > >>>>>>> avoid
> > > >>>>>>>>> JS injection, even if security of js libraries used is
> > > >> perfect,
> > > >>>>> which
> > > >>>>>> I
> > > >>>>>>>>> doubt (search gmail/linkedin XSS). Client side rendering
> > > >>>> completely
> > > >>>>>>> breaks
> > > >>>>>>>>> the workflows for ops who rely on text based
> > > >> terminal/emacs/vim
> > > >>>>>> browsers
> > > >>>>>>>>> (no js support) to monitor component UI.
> > > >>>>>>>>>
> > > >>>>>>>>> IMO, JS-only rendering belongs to social networking sites
> > > >> and/or
> > > >>>>> SaaS
> > > >>>>>>>>> front-ends, where full time UI/security specialists babysits
> > > >> UI
> > > >>>>>>> changes. I
> > > >>>>>>>>> think eventually most users will use a self servicing UI in a
> > > >>> SaaS
> > > >>>>>>>>> front-end that uses REST/JMX API to get data from back-end
> > > >>>>> components,
> > > >>>>>>>>> besides their own app master/service UI. The
> > > >>> priority/requirements
> > > >>>>> for
> > > >>>>>>> UI
> > > >>>>>>>>> in core Hadoop should be security and correctness, which
> > > >> client
> > > >>>> side
> > > >>>>>>>>> templating cannot address properly so far.
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>> On Tue, Oct 22, 2013 at 3:59 PM, Haohui Mai <
> > > >>> hmai@hortonworks.com
> > > >>>>>
> > > >>>>>>> wrote:
> > > >>>>>>>>>
> > > >>>>>>>>>> Hi all,
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>> Jing Zhao and I recently have reimplemented the JSP-based
> > > >> web
> > > >>>> UIs
> > > >>>>> in
> > > >>>>>>>>> HTML 5
> > > >>>>>>>>>> applications (HDFS-5333). Based on our prelimanary testing
> > > >>>> results
> > > >>>>>> we
> > > >>>>>>>>>> believe thst the new web UIs of the namenodes and the
> > > >> datanode
> > > >>>> are
> > > >>>>>>> ready
> > > >>>>>>>>>> for everyday uses.
> > > >>>>>>>>>>
> > > >>>>>>>>>> You're more than welcome to try it out on trunk by visiting
> > > >>>>> http://
> > > >>>>>>>>>> <namenode>/dfshealth.html
> > > >>>>>>>>>>
> > > >>>>>>>>>> There are a number of benefits from this transition. From a
> > > >>>>>>> developer's
> > > >>>>>>>>>> prospective, the most notable one is *maintainability*:
> > > >>>>>>>>>>
> > > >>>>>>>>>> (1) The abstractions between the UI and the core server are
> > > >>>>>>> well-defined,
> > > >>>>>>>>>> decoupling the UI and the core hadoop servers.
> > > >>>>>>>>>>
> > > >>>>>>>>>> (2) It allows us to deprecate the logic in the JSP pages.
> > > >> The
> > > >>>> old
> > > >>>>>> web
> > > >>>>>>> UIs
> > > >>>>>>>>>> have to duplicate the logic in the JSPs. The logic is often
> > > >>>>>>> out-of-dated
> > > >>>>>>>>>> and not well-tested, which leads to broken pages and
> > > >> security
> > > >>>>>>>>>> vulnerabilities(e.g. HDFS-5251, HDFS-5307, HDFS-5308,
> > > >>> HDFS-5317
> > > >>>>> and
> > > >>>>>>>>>> HDFS-4901). The architecture of the new UIs prevent these
> > > >> bugs
> > > >>>> at
> > > >>>>>> the
> > > >>>>>>>>> very
> > > >>>>>>>>>> beginning.
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>> I propose that deprecate the old, JSP-based web UIs in 2.3.
> > > >> I
> > > >>>>> opened
> > > >>>>>>>>>> HDFS-5402 to track the relevant discussions.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Your feedbacks are highly appreciated.
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>> Sincerely,
> > > >>>>>>>>>>
> > > >>>>>>>>>> Haohui
> > > >>>>>>>>>>
> > > >>>>>>>>>> --
> > > >>>>>>>>>> CONFIDENTIALITY NOTICE
> > > >>>>>>>>>> NOTICE: This message is intended for the use of the
> > > >> individual
> > > >>>> or
> > > >>>>>>> entity
> > > >>>>>>>>> to
> > > >>>>>>>>>> which it is addressed and may contain information that is
> > > >>>>>>> confidential,
> > > >>>>>>>>>> privileged and exempt from disclosure under applicable law.
> > > >> If
> > > >>>> the
> > > >>>>>>> reader
> > > >>>>>>>>>> of this message is not the intended recipient, you are
> > > >> hereby
> > > >>>>>> notified
> > > >>>>>>>>> that
> > > >>>>>>>>>> any printing, copying, dissemination, distribution,
> > > >> disclosure
> > > >>>> or
> > > >>>>>>>>>> forwarding of this communication is strictly prohibited. If
> > > >>> you
> > > >>>>> have
> > > >>>>>>>>>> received this communication in error, please contact the
> > > >>> sender
> > > >>>>>>>>> immediately
> > > >>>>>>>>>> and delete it from your system. Thank You.
> > > >>>>>>>>
> > > >>>>>>>> --
> > > >>>>>>>> CONFIDENTIALITY NOTICE
> > > >>>>>>>> NOTICE: This message is intended for the use of the individual
> > > >> or
> > > >>>>>> entity
> > > >>>>>>> to
> > > >>>>>>>> which it is addressed and may contain information that is
> > > >>>>> confidential,
> > > >>>>>>>> privileged and exempt from disclosure under applicable law. If
> > > >>> the
> > > >>>>>> reader
> > > >>>>>>>> of this message is not the intended recipient, you are hereby
> > > >>>>> notified
> > > >>>>>>> that
> > > >>>>>>>> any printing, copying, dissemination, distribution, disclosure
> > > >> or
> > > >>>>>>>> forwarding of this communication is strictly prohibited. If
> you
> > > >>>> have
> > > >>>>>>>> received this communication in error, please contact the
> sender
> > > >>>>>>> immediately
> > > >>>>>>>> and delete it from your system. Thank You.
> > > >>>>>>
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> --
> > > >>>>>> Alejandro
> > > >>>>>
> > > >>>>> --
> > > >>>>> CONFIDENTIALITY NOTICE
> > > >>>>> NOTICE: This message is intended for the use of the individual or
> > > >>> entity
> > > >>>> to
> > > >>>>> which it is addressed and may contain information that is
> > > >> confidential,
> > > >>>>> privileged and exempt from disclosure under applicable law. If
> the
> > > >>> reader
> > > >>>>> of this message is not the intended recipient, you are hereby
> > > >> notified
> > > >>>> that
> > > >>>>> any printing, copying, dissemination, distribution, disclosure or
> > > >>>>> forwarding of this communication is strictly prohibited. If you
> > have
> > > >>>>> received this communication in error, please contact the sender
> > > >>>> immediately
> > > >>>>> and delete it from your system. Thank You.
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> --
> > > >>>> Alejandro
> > > >>>
> > > >>> --
> > > >>> CONFIDENTIALITY NOTICE
> > > >>> NOTICE: This message is intended for the use of the individual or
> > > entity
> > > >> to
> > > >>> which it is addressed and may contain information that is
> > confidential,
> > > >>> privileged and exempt from disclosure under applicable law. If the
> > > reader
> > > >>> of this message is not the intended recipient, you are hereby
> > notified
> > > >> that
> > > >>> any printing, copying, dissemination, distribution, disclosure or
> > > >>> forwarding of this communication is strictly prohibited. If you
> have
> > > >>> received this communication in error, please contact the sender
> > > >> immediately
> > > >>> and delete it from your system. Thank You.
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> Alejandro
> > > >
> > > > --
> > > > CONFIDENTIALITY NOTICE
> > > > NOTICE: This message is intended for the use of the individual or
> > entity
> > > to
> > > > which it is addressed and may contain information that is
> confidential,
> > > > privileged and exempt from disclosure under applicable law. If the
> > reader
> > > > of this message is not the intended recipient, you are hereby
> notified
> > > that
> > > > any printing, copying, dissemination, distribution, disclosure or
> > > > forwarding of this communication is strictly prohibited. If you have
> > > > received this communication in error, please contact the sender
> > > immediately
> > > > and delete it from your system. Thank You.
> > >
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Replacing the JSP web UIs to HTML 5 applications

[Larry McCay <lm...@hortonworks.com>]

I think it is important that we make provisions for all Ajax calls to be
able to go through gateway deployments like Knox with the cluster
firewalled off.
As I have commented on the Jira, any calls that are currently on the
serverside but are moving to the browser will need to either require
punching holes in filewalls (unfortunate) or directing through a given REST
API gateway.


On Mon, Oct 28, 2013 at 9:09 PM, Haohui Mai <hm...@hortonworks.com> wrote:

> Neither of them will go through JMX.
>
> The new Web UI implements hdfs browsing through WebHDFS.
>
> The logs are available through the static servlets, which is exactly the
> same as what we have today.
>
> Thanks,
> Haohui
>
>
> On Mon, Oct 28, 2013 at 6:01 PM, Alejandro Abdelnur <tucu@cloudera.com
> >wrote:
>
> > are you planning to expose things like hdfs browsing and nn/dn logs over
> > jmx?
> >
> > thx
> >
> > Alejandro
> > (phone typing)
> >
> > On Oct 28, 2013, at 17:48, Haohui Mai <hm...@hortonworks.com> wrote:
> >
> > > It seems more appealing to me that the UI should JMX directly, because:
> > >
> > > * We're support the JMX in the long term for other management software.
> > > * The information provided by the JMX API will be mostly identical of
> the
> > > JSON API. Today the Web UI covers most of the information provided by
> > JMX.
> > > The Web UI does some trivial work to extract the information and
> renders
> > it
> > > as HTML.
> > > * We can compatibility and unit tests for free.
> > >
> > > I do agree that the JMX APIs are imperfect and they should be revisited
> > in
> > > the 3.0 timeframe. However, this is orthogonal of the discussions of
> > > transitioning from JSP-based Web UI to client-side JavaScript Web UI.
> The
> > > architecture of the new Web UI allows easy migration to any JSON-based
> > APIs
> > > whenever they land in the trunk.
> > >
> > > Thanks,
> > > Haohui
> > >
> > >
> > > On Mon, Oct 28, 2013 at 5:13 PM, Alejandro Abdelnur <tucu@cloudera.com
> > >wrote:
> > >
> > >> Isn't using JMX to expose JSON for the web UI misusing JMX?
> > >>
> > >> I would think a more appropriate approach would be having /JMX for
> > >> monitoring integration and a /JSON end point for the UI data.
> > >>
> > >> Thanks.
> > >>
> > >>
> > >> On Mon, Oct 28, 2013 at 4:58 PM, Haohui Mai <hm...@hortonworks.com>
> > wrote:
> > >>
> > >>> Alejandro,
> > >>>
> > >>> If I understand correctly, that is the exact approach that the new
> web
> > UI
> > >>> is taking. The new web UI takes the output from JMX and renders them
> as
> > >>> HTML at the client side.
> > >>>
> > >>> ~Haohui
> > >>>
> > >>>
> > >>> On Mon, Oct 28, 2013 at 4:18 PM, Alejandro Abdelnur <
> tucu@cloudera.com
> > >>>> wrote:
> > >>>
> > >>>> Haohui,
> > >>>>
> > >>>> If you have NN and DNs producing JSON instead HTML, then you can
> build
> > >> JS
> > >>>> based web UIs. Take for example Oozie, Oozie produces JSON, it has a
> > >>> built
> > >>>> in JS web ui that consumes JSON and Hue has built an external web UI
> > >> that
> > >>>> also consumes JSON. In the case of Hue UI, Oozie didn't have to
> change
> > >>>> anything to get that UI and improvements on the Hue UI don't require
> > >>>> changes in Oozie unless it is to produce additional information.
> > >>>>
> > >>>> hope this clarifies.
> > >>>>
> > >>>> Thx
> > >>>>
> > >>>>
> > >>>> On Mon, Oct 28, 2013 at 4:06 PM, Haohui Mai <hm...@hortonworks.com>
> > >>> wrote:
> > >>>>
> > >>>>> Echo my comments on HDFS-5402:
> > >>>>>
> > >>>>> bq. If we're going to remove the old web UI, I think the new web UI
> > >> has
> > >>>>> to have the same level of unit testing. We shouldn't go backwards
> in
> > >>>>> terms of unit testing.
> > >>>>>
> > >>>>> I take a look at TestNamenodeJspHelper / TestDatanodeJspHelper /
> > >>>>> TestClusterJspHelper. It seems to me that we can merge these tests
> > >> with
> > >>>> the
> > >>>>> unit tests on JMX.
> > >>>>>
> > >>>>> bq. If we are going to
> > >>>>> remove this capability, we need to add some other command-line
> tools
> > >>>>> to get the same functionality. These tools could use REST if we
> have
> > >>>>> that, or JMX, but they need to exist before we can consider
> removing
> > >>>>> the old UI.
> > >>>>>
> > >>>>> This is a good point. Since all information are available through
> > >> JMX,
> > >>>> the
> > >>>>> easiest way to approach it is to write some scripts using Node.js.
> > >> The
> > >>>>> architecture of the new Web UIs is ready for this.
> > >>>>>
> > >>>>>
> > >>>>> On Mon, Oct 28, 2013 at 3:57 PM, Alejandro Abdelnur <
> > >> tucu@cloudera.com
> > >>>>>> wrote:
> > >>>>>
> > >>>>>> Producing JSON would be great. Agree with Colin that we should
> > >> leave
> > >>>> for
> > >>>>>> now the current JSP based web ui.
> > >>>>>>
> > >>>>>> thx
> > >>>>>>
> > >>>>>>
> > >>>>>> On Mon, Oct 28, 2013 at 11:16 AM, Colin McCabe <
> > >>> cmccabe@alumni.cmu.edu
> > >>>>>>> wrote:
> > >>>>>>
> > >>>>>>> This is a really interesting project, Haohui.  I think it will
> > >> make
> > >>>>>>> our web UI much nicer.
> > >>>>>>>
> > >>>>>>> I have a few concerns about removing the old web UI, however:
> > >>>>>>>
> > >>>>>>> * If we're going to remove the old web UI, I think the new web UI
> > >>> has
> > >>>>>>> to have the same level of unit testing.  We shouldn't go
> > >> backwards
> > >>> in
> > >>>>>>> terms of unit testing.
> > >>>>>>>
> > >>>>>>> * Most of the deployments of elinks and links out there don't
> > >>> support
> > >>>>>>> Javascript.  This is just a reality of life when using CentOS 5
> > >> or
> > >>> 6,
> > >>>>>>> which many users are still using.  I have used "links" to
> > >> diagnose
> > >>>>>>> problems through the web UI in the past, in systems where access
> > >> to
> > >>>>>>> the cluster was available only through telnet.  If we are going
> > >> to
> > >>>>>>> remove this capability, we need to add some other command-line
> > >>> tools
> > >>>>>>> to get the same functionality.  These tools could use REST if we
> > >>> have
> > >>>>>>> that, or JMX, but they need to exist before we can consider
> > >>> removing
> > >>>>>>> the old UI.
> > >>>>>>>
> > >>>>>>> best,
> > >>>>>>> Colin
> > >>>>>>>
> > >>>>>>> On Fri, Oct 25, 2013 at 7:31 PM, Haohui Mai <
> > >> hmai@hortonworks.com>
> > >>>>>> wrote:
> > >>>>>>>> Thanks for the reply, Luke. Here I just echo my response from
> > >> the
> > >>>>> jira:
> > >>>>>>>>
> > >>>>>>>> bq. this client-side js only approach, which is less secure
> > >> than
> > >>> a
> > >>>>>>>> progressively enhanced hybrid approach used by YARN. The recent
> > >>>> gmail
> > >>>>>>>> XSS fiasco highlights the issue.
> > >>>>>>>>
> > >>>>>>>> I'm presenting an informal security analysis to compare the
> > >>>> security
> > >>>>> of
> > >>>>>>> the
> > >>>>>>>> old and the new web UIs.
> > >>>>>>>>
> > >>>>>>>> An attacker launches an XSS attack by injecting malicious code
> > >>>> which
> > >>>>>> are
> > >>>>>>>> usually HTML or JavaScript fragments into the web page, so that
> > >>> the
> > >>>>>>>> malicious code can have the same privileges of the web page.
> > >>>>>>>>
> > >>>>>>>> First, in the scope of XSS attacks, note that the threat models
> > >>> of
> > >>>>>>>> launching XSS attacks on Internet sites Gmail/Linkedin and the
> > >>> one
> > >>>> of
> > >>>>>> the
> > >>>>>>>> Hadoop UIs are different. They have fundamental different sets
> > >> of
> > >>>>>>> external
> > >>>>>>>> inputs that the attackers have control to. Internet sites have
> > >>>> little
> > >>>>>>>> control of these inputs. In the case of Gmail / Linkedin, an
> > >>> attack
> > >>>>> can
> > >>>>>>>> send you a crafted e-mail, or put malicious description in his
> > >> /
> > >>>>>>>> her Linkedin profile. The sets of external inputs are
> > >>> *restricted*
> > >>>> in
> > >>>>>>>> Hadoop UIs. The new web UIs take JMX and WebHDFS as inputs. The
> > >>>>>>>> attacker has to launch a XSS attack by:
> > >>>>>>>>
> > >>>>>>>> * Compromise the jars so that the output of JMX / WebHDFS have
> > >>> the
> > >>>>>>>> malicious code.
> > >>>>>>>> * Replace the web UIs completely to include the malicious code.
> > >>>>>>>>
> > >>>>>>>> In either case *the attacker has to compromise the hadoop core
> > >> or
> > >>>> the
> > >>>>>>>> namenode*. That means the new web UIs are at least as secure as
> > >>> the
> > >>>>>>> hadoop
> > >>>>>>>> core, and the namenode machine.
> > >>>>>>>>
> > >>>>>>>> Second, I argue that using client-side templates are more
> > >> secure
> > >>>> than
> > >>>>>> the
> > >>>>>>>> current JSP-based server-side templates. To defend against XSS
> > >>>>>>>> attacks, both techniques have to filter the external inputs at
> > >>>>> *every*
> > >>>>>>>> possible execution paths. Several facts much be taken into
> > >>>>>>>> plays when evaluating the security of both approaches in
> > >>> real-world
> > >>>>>>>> environments:
> > >>>>>>>>
> > >>>>>>>> * The JavaScript libraries used in the new web UIs have
> > >> survived
> > >>> in
> > >>>>>>>> extremely large-scale production tests. jQuery is used by
> > >> Google
> > >>>> and
> > >>>>>>>> Microsoft, bootstrap is used by Twitter, and dust.js is used
> > >> by
> > >>>>>>> Linkedin.
> > >>>>>>>> All libraries survived from hundreds of thousands of
> > >>>>>>>> attack attempts on a daily basis. I agree that the libraries
> > >>> might
> > >>>>>> still
> > >>>>>>>> be imperfect, but there's no way that we can test the JSP web
> > >>>>>>>> UIs to achieve the same level of assurances given the amount
> > >> of
> > >>>>>>> resources
> > >>>>>>>> the community has.
> > >>>>>>>> * Client-side templates consolidate all filtering logic in one
> > >>>>> central
> > >>>>>>>> place. Recall that the goal is to filter all external inputs at
> > >>>> every
> > >>>>>>>> execution paths, this is a much more systematic approach
> > >>> compared
> > >>>> to
> > >>>>>> the
> > >>>>>>>> server-side templates we have today. It is difficult (if not
> > >>>>>>>> impossible) to do it in a JSP/ASP/PHP application, since such
> > >>>>>> filtering
> > >>>>>>>> can be only achieved via ad-hoc approaches ([1] shows some
> > >>>>>>>> empirical data). Also, HDFS-4901 recently describes a XSS
> > >>>>>> vulnerability
> > >>>>>>> in
> > >>>>>>>> browseDirectory.jsp.
> > >>>>>>>>
> > >>>>>>>> bq. You'd require proper SSL (not self signed) setup to avoid
> > >> JS
> > >>>>>>>> injection
> > >>>>>>>>
> > >>>>>>>> Commodity browsers enforce Same-Origin Policy to defend against
> > >>>> code
> > >>>>>>>> injections. It has nothing to do with what kinds of SSL
> > >>>> certificates
> > >>>>>>>> you hold.
> > >>>>>>>>
> > >>>>>>>> bq.  I also have concerns that we commit these changes without
> > >>>>> matching
> > >>>>>>>> unit tests
> > >>>>>>>>
> > >>>>>>>> The JavaScript code can be automatically tested. The same code
> > >>> can
> > >>>> be
> > >>>>>> run
> > >>>>>>>> by node.js and the test can compared with pre-defined
> > >>>>>>>> results. It is also possible to write an adapter to use Rhino
> > >> to
> > >>>>>>> accomplish
> > >>>>>>>> the same task. We can discuss how to integrate them into
> > >>>>>>>> the maven test routines in a different thread.
> > >>>>>>>>
> > >>>>>>>> bq. Client side rendering completely breaks the workflows for
> > >> ops
> > >>>> who
> > >>>>>>> rely
> > >>>>>>>> on text based terminal/emacs/vim browsers (no js support) to
> > >>>>>>>> monitor component UI.
> > >>>>>>>>
> > >>>>>>>> links / elinks (http://elinks.or.cz/) are text-based web
> > >>> browsers
> > >>>>> that
> > >>>>>>>> support JavaScript.
> > >>>>>>>>
> > >>>>>>>> bq. The priority/requirements for UI in core Hadoop should be
> > >>>>> security
> > >>>>>>> and
> > >>>>>>>> correctness, which client side templating cannot address
> > >> properly
> > >>>>>>>> so far.
> > >>>>>>>>
> > >>>>>>>> I agree that we should focus on security and correctness. The
> > >>>>>> paragraphs
> > >>>>>>>> above explain that how the architecture of the new UIs
> > >>>>>>>> makes the UIs more secure in real-world settings compared to
> > >> the
> > >>> UI
> > >>>>> we
> > >>>>>>> have
> > >>>>>>>> today.
> > >>>>>>>>
> > >>>>>>>> References:
> > >>>>>>>>
> > >>>>>>>> 1. A. Yip et al. Improving Application Security with Data Flow
> > >>>>>>> Assertions.
> > >>>>>>>> In SOSP'2009.
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> On Fri, Oct 25, 2013 at 10:02 AM, Luke Lu <ll...@apache.org>
> > >>> wrote:
> > >>>>>>>>
> > >>>>>>>>> Echoing my comments on HDFS-3555:
> > >>>>>>>>>
> > >>>>>>>>> I have concerns with this client-side js only approach, which
> > >> is
> > >>>>> less
> > >>>>>>>>> secure than a progressively enhanced hybrid approach used by
> > >>> YARN.
> > >>>>> The
> > >>>>>>>>> recent gmail XSS fiasco highlights the issue. I also have
> > >>> concerns
> > >>>>>> that
> > >>>>>>> we
> > >>>>>>>>> commit these changes without matching unit tests – the fact
> > >> you
> > >>>>> cannot
> > >>>>>>>>> effectively unit test these changes should tell you something
> > >>>> about
> > >>>>>> this
> > >>>>>>>>> approach.
> > >>>>>>>>>
> > >>>>>>>>> *Requiring* JS means that an admin cannot turn off js to
> > >>>> (partially)
> > >>>>>> use
> > >>>>>>>>> core Hadoop UI. You'd *require* proper SSL (not self signed)
> > >>> setup
> > >>>>> to
> > >>>>>>> avoid
> > >>>>>>>>> JS injection, even if security of js libraries used is
> > >> perfect,
> > >>>>> which
> > >>>>>> I
> > >>>>>>>>> doubt (search gmail/linkedin XSS). Client side rendering
> > >>>> completely
> > >>>>>>> breaks
> > >>>>>>>>> the workflows for ops who rely on text based
> > >> terminal/emacs/vim
> > >>>>>> browsers
> > >>>>>>>>> (no js support) to monitor component UI.
> > >>>>>>>>>
> > >>>>>>>>> IMO, JS-only rendering belongs to social networking sites
> > >> and/or
> > >>>>> SaaS
> > >>>>>>>>> front-ends, where full time UI/security specialists babysits
> > >> UI
> > >>>>>>> changes. I
> > >>>>>>>>> think eventually most users will use a self servicing UI in a
> > >>> SaaS
> > >>>>>>>>> front-end that uses REST/JMX API to get data from back-end
> > >>>>> components,
> > >>>>>>>>> besides their own app master/service UI. The
> > >>> priority/requirements
> > >>>>> for
> > >>>>>>> UI
> > >>>>>>>>> in core Hadoop should be security and correctness, which
> > >> client
> > >>>> side
> > >>>>>>>>> templating cannot address properly so far.
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> On Tue, Oct 22, 2013 at 3:59 PM, Haohui Mai <
> > >>> hmai@hortonworks.com
> > >>>>>
> > >>>>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> Hi all,
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> Jing Zhao and I recently have reimplemented the JSP-based
> > >> web
> > >>>> UIs
> > >>>>> in
> > >>>>>>>>> HTML 5
> > >>>>>>>>>> applications (HDFS-5333). Based on our prelimanary testing
> > >>>> results
> > >>>>>> we
> > >>>>>>>>>> believe thst the new web UIs of the namenodes and the
> > >> datanode
> > >>>> are
> > >>>>>>> ready
> > >>>>>>>>>> for everyday uses.
> > >>>>>>>>>>
> > >>>>>>>>>> You're more than welcome to try it out on trunk by visiting
> > >>>>> http://
> > >>>>>>>>>> <namenode>/dfshealth.html
> > >>>>>>>>>>
> > >>>>>>>>>> There are a number of benefits from this transition. From a
> > >>>>>>> developer's
> > >>>>>>>>>> prospective, the most notable one is *maintainability*:
> > >>>>>>>>>>
> > >>>>>>>>>> (1) The abstractions between the UI and the core server are
> > >>>>>>> well-defined,
> > >>>>>>>>>> decoupling the UI and the core hadoop servers.
> > >>>>>>>>>>
> > >>>>>>>>>> (2) It allows us to deprecate the logic in the JSP pages.
> > >> The
> > >>>> old
> > >>>>>> web
> > >>>>>>> UIs
> > >>>>>>>>>> have to duplicate the logic in the JSPs. The logic is often
> > >>>>>>> out-of-dated
> > >>>>>>>>>> and not well-tested, which leads to broken pages and
> > >> security
> > >>>>>>>>>> vulnerabilities(e.g. HDFS-5251, HDFS-5307, HDFS-5308,
> > >>> HDFS-5317
> > >>>>> and
> > >>>>>>>>>> HDFS-4901). The architecture of the new UIs prevent these
> > >> bugs
> > >>>> at
> > >>>>>> the
> > >>>>>>>>> very
> > >>>>>>>>>> beginning.
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> I propose that deprecate the old, JSP-based web UIs in 2.3.
> > >> I
> > >>>>> opened
> > >>>>>>>>>> HDFS-5402 to track the relevant discussions.
> > >>>>>>>>>>
> > >>>>>>>>>> Your feedbacks are highly appreciated.
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> Sincerely,
> > >>>>>>>>>>
> > >>>>>>>>>> Haohui
> > >>>>>>>>>>
> > >>>>>>>>>> --
> > >>>>>>>>>> CONFIDENTIALITY NOTICE
> > >>>>>>>>>> NOTICE: This message is intended for the use of the
> > >> individual
> > >>>> or
> > >>>>>>> entity
> > >>>>>>>>> to
> > >>>>>>>>>> which it is addressed and may contain information that is
> > >>>>>>> confidential,
> > >>>>>>>>>> privileged and exempt from disclosure under applicable law.
> > >> If
> > >>>> the
> > >>>>>>> reader
> > >>>>>>>>>> of this message is not the intended recipient, you are
> > >> hereby
> > >>>>>> notified
> > >>>>>>>>> that
> > >>>>>>>>>> any printing, copying, dissemination, distribution,
> > >> disclosure
> > >>>> or
> > >>>>>>>>>> forwarding of this communication is strictly prohibited. If
> > >>> you
> > >>>>> have
> > >>>>>>>>>> received this communication in error, please contact the
> > >>> sender
> > >>>>>>>>> immediately
> > >>>>>>>>>> and delete it from your system. Thank You.
> > >>>>>>>>
> > >>>>>>>> --
> > >>>>>>>> CONFIDENTIALITY NOTICE
> > >>>>>>>> NOTICE: This message is intended for the use of the individual
> > >> or
> > >>>>>> entity
> > >>>>>>> to
> > >>>>>>>> which it is addressed and may contain information that is
> > >>>>> confidential,
> > >>>>>>>> privileged and exempt from disclosure under applicable law. If
> > >>> the
> > >>>>>> reader
> > >>>>>>>> of this message is not the intended recipient, you are hereby
> > >>>>> notified
> > >>>>>>> that
> > >>>>>>>> any printing, copying, dissemination, distribution, disclosure
> > >> or
> > >>>>>>>> forwarding of this communication is strictly prohibited. If you
> > >>>> have
> > >>>>>>>> received this communication in error, please contact the sender
> > >>>>>>> immediately
> > >>>>>>>> and delete it from your system. Thank You.
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> --
> > >>>>>> Alejandro
> > >>>>>
> > >>>>> --
> > >>>>> CONFIDENTIALITY NOTICE
> > >>>>> NOTICE: This message is intended for the use of the individual or
> > >>> entity
> > >>>> to
> > >>>>> which it is addressed and may contain information that is
> > >> confidential,
> > >>>>> privileged and exempt from disclosure under applicable law. If the
> > >>> reader
> > >>>>> of this message is not the intended recipient, you are hereby
> > >> notified
> > >>>> that
> > >>>>> any printing, copying, dissemination, distribution, disclosure or
> > >>>>> forwarding of this communication is strictly prohibited. If you
> have
> > >>>>> received this communication in error, please contact the sender
> > >>>> immediately
> > >>>>> and delete it from your system. Thank You.
> > >>>>
> > >>>>
> > >>>>
> > >>>> --
> > >>>> Alejandro
> > >>>
> > >>> --
> > >>> CONFIDENTIALITY NOTICE
> > >>> NOTICE: This message is intended for the use of the individual or
> > entity
> > >> to
> > >>> which it is addressed and may contain information that is
> confidential,
> > >>> privileged and exempt from disclosure under applicable law. If the
> > reader
> > >>> of this message is not the intended recipient, you are hereby
> notified
> > >> that
> > >>> any printing, copying, dissemination, distribution, disclosure or
> > >>> forwarding of this communication is strictly prohibited. If you have
> > >>> received this communication in error, please contact the sender
> > >> immediately
> > >>> and delete it from your system. Thank You.
> > >>
> > >>
> > >>
> > >> --
> > >> Alejandro
> > >
> > > --
> > > CONFIDENTIALITY NOTICE
> > > NOTICE: This message is intended for the use of the individual or
> entity
> > to
> > > which it is addressed and may contain information that is confidential,
> > > privileged and exempt from disclosure under applicable law. If the
> reader
> > > of this message is not the intended recipient, you are hereby notified
> > that
> > > any printing, copying, dissemination, distribution, disclosure or
> > > forwarding of this communication is strictly prohibited. If you have
> > > received this communication in error, please contact the sender
> > immediately
> > > and delete it from your system. Thank You.
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Replacing the JSP web UIs to HTML 5 applications

[Haohui Mai <hm...@hortonworks.com>]

Neither of them will go through JMX.

The new Web UI implements hdfs browsing through WebHDFS.

The logs are available through the static servlets, which is exactly the
same as what we have today.

Thanks,
Haohui


On Mon, Oct 28, 2013 at 6:01 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:

> are you planning to expose things like hdfs browsing and nn/dn logs over
> jmx?
>
> thx
>
> Alejandro
> (phone typing)
>
> On Oct 28, 2013, at 17:48, Haohui Mai <hm...@hortonworks.com> wrote:
>
> > It seems more appealing to me that the UI should JMX directly, because:
> >
> > * We're support the JMX in the long term for other management software.
> > * The information provided by the JMX API will be mostly identical of the
> > JSON API. Today the Web UI covers most of the information provided by
> JMX.
> > The Web UI does some trivial work to extract the information and renders
> it
> > as HTML.
> > * We can compatibility and unit tests for free.
> >
> > I do agree that the JMX APIs are imperfect and they should be revisited
> in
> > the 3.0 timeframe. However, this is orthogonal of the discussions of
> > transitioning from JSP-based Web UI to client-side JavaScript Web UI. The
> > architecture of the new Web UI allows easy migration to any JSON-based
> APIs
> > whenever they land in the trunk.
> >
> > Thanks,
> > Haohui
> >
> >
> > On Mon, Oct 28, 2013 at 5:13 PM, Alejandro Abdelnur <tucu@cloudera.com
> >wrote:
> >
> >> Isn't using JMX to expose JSON for the web UI misusing JMX?
> >>
> >> I would think a more appropriate approach would be having /JMX for
> >> monitoring integration and a /JSON end point for the UI data.
> >>
> >> Thanks.
> >>
> >>
> >> On Mon, Oct 28, 2013 at 4:58 PM, Haohui Mai <hm...@hortonworks.com>
> wrote:
> >>
> >>> Alejandro,
> >>>
> >>> If I understand correctly, that is the exact approach that the new web
> UI
> >>> is taking. The new web UI takes the output from JMX and renders them as
> >>> HTML at the client side.
> >>>
> >>> ~Haohui
> >>>
> >>>
> >>> On Mon, Oct 28, 2013 at 4:18 PM, Alejandro Abdelnur <tucu@cloudera.com
> >>>> wrote:
> >>>
> >>>> Haohui,
> >>>>
> >>>> If you have NN and DNs producing JSON instead HTML, then you can build
> >> JS
> >>>> based web UIs. Take for example Oozie, Oozie produces JSON, it has a
> >>> built
> >>>> in JS web ui that consumes JSON and Hue has built an external web UI
> >> that
> >>>> also consumes JSON. In the case of Hue UI, Oozie didn't have to change
> >>>> anything to get that UI and improvements on the Hue UI don't require
> >>>> changes in Oozie unless it is to produce additional information.
> >>>>
> >>>> hope this clarifies.
> >>>>
> >>>> Thx
> >>>>
> >>>>
> >>>> On Mon, Oct 28, 2013 at 4:06 PM, Haohui Mai <hm...@hortonworks.com>
> >>> wrote:
> >>>>
> >>>>> Echo my comments on HDFS-5402:
> >>>>>
> >>>>> bq. If we're going to remove the old web UI, I think the new web UI
> >> has
> >>>>> to have the same level of unit testing. We shouldn't go backwards in
> >>>>> terms of unit testing.
> >>>>>
> >>>>> I take a look at TestNamenodeJspHelper / TestDatanodeJspHelper /
> >>>>> TestClusterJspHelper. It seems to me that we can merge these tests
> >> with
> >>>> the
> >>>>> unit tests on JMX.
> >>>>>
> >>>>> bq. If we are going to
> >>>>> remove this capability, we need to add some other command-line tools
> >>>>> to get the same functionality. These tools could use REST if we have
> >>>>> that, or JMX, but they need to exist before we can consider removing
> >>>>> the old UI.
> >>>>>
> >>>>> This is a good point. Since all information are available through
> >> JMX,
> >>>> the
> >>>>> easiest way to approach it is to write some scripts using Node.js.
> >> The
> >>>>> architecture of the new Web UIs is ready for this.
> >>>>>
> >>>>>
> >>>>> On Mon, Oct 28, 2013 at 3:57 PM, Alejandro Abdelnur <
> >> tucu@cloudera.com
> >>>>>> wrote:
> >>>>>
> >>>>>> Producing JSON would be great. Agree with Colin that we should
> >> leave
> >>>> for
> >>>>>> now the current JSP based web ui.
> >>>>>>
> >>>>>> thx
> >>>>>>
> >>>>>>
> >>>>>> On Mon, Oct 28, 2013 at 11:16 AM, Colin McCabe <
> >>> cmccabe@alumni.cmu.edu
> >>>>>>> wrote:
> >>>>>>
> >>>>>>> This is a really interesting project, Haohui.  I think it will
> >> make
> >>>>>>> our web UI much nicer.
> >>>>>>>
> >>>>>>> I have a few concerns about removing the old web UI, however:
> >>>>>>>
> >>>>>>> * If we're going to remove the old web UI, I think the new web UI
> >>> has
> >>>>>>> to have the same level of unit testing.  We shouldn't go
> >> backwards
> >>> in
> >>>>>>> terms of unit testing.
> >>>>>>>
> >>>>>>> * Most of the deployments of elinks and links out there don't
> >>> support
> >>>>>>> Javascript.  This is just a reality of life when using CentOS 5
> >> or
> >>> 6,
> >>>>>>> which many users are still using.  I have used "links" to
> >> diagnose
> >>>>>>> problems through the web UI in the past, in systems where access
> >> to
> >>>>>>> the cluster was available only through telnet.  If we are going
> >> to
> >>>>>>> remove this capability, we need to add some other command-line
> >>> tools
> >>>>>>> to get the same functionality.  These tools could use REST if we
> >>> have
> >>>>>>> that, or JMX, but they need to exist before we can consider
> >>> removing
> >>>>>>> the old UI.
> >>>>>>>
> >>>>>>> best,
> >>>>>>> Colin
> >>>>>>>
> >>>>>>> On Fri, Oct 25, 2013 at 7:31 PM, Haohui Mai <
> >> hmai@hortonworks.com>
> >>>>>> wrote:
> >>>>>>>> Thanks for the reply, Luke. Here I just echo my response from
> >> the
> >>>>> jira:
> >>>>>>>>
> >>>>>>>> bq. this client-side js only approach, which is less secure
> >> than
> >>> a
> >>>>>>>> progressively enhanced hybrid approach used by YARN. The recent
> >>>> gmail
> >>>>>>>> XSS fiasco highlights the issue.
> >>>>>>>>
> >>>>>>>> I'm presenting an informal security analysis to compare the
> >>>> security
> >>>>> of
> >>>>>>> the
> >>>>>>>> old and the new web UIs.
> >>>>>>>>
> >>>>>>>> An attacker launches an XSS attack by injecting malicious code
> >>>> which
> >>>>>> are
> >>>>>>>> usually HTML or JavaScript fragments into the web page, so that
> >>> the
> >>>>>>>> malicious code can have the same privileges of the web page.
> >>>>>>>>
> >>>>>>>> First, in the scope of XSS attacks, note that the threat models
> >>> of
> >>>>>>>> launching XSS attacks on Internet sites Gmail/Linkedin and the
> >>> one
> >>>> of
> >>>>>> the
> >>>>>>>> Hadoop UIs are different. They have fundamental different sets
> >> of
> >>>>>>> external
> >>>>>>>> inputs that the attackers have control to. Internet sites have
> >>>> little
> >>>>>>>> control of these inputs. In the case of Gmail / Linkedin, an
> >>> attack
> >>>>> can
> >>>>>>>> send you a crafted e-mail, or put malicious description in his
> >> /
> >>>>>>>> her Linkedin profile. The sets of external inputs are
> >>> *restricted*
> >>>> in
> >>>>>>>> Hadoop UIs. The new web UIs take JMX and WebHDFS as inputs. The
> >>>>>>>> attacker has to launch a XSS attack by:
> >>>>>>>>
> >>>>>>>> * Compromise the jars so that the output of JMX / WebHDFS have
> >>> the
> >>>>>>>> malicious code.
> >>>>>>>> * Replace the web UIs completely to include the malicious code.
> >>>>>>>>
> >>>>>>>> In either case *the attacker has to compromise the hadoop core
> >> or
> >>>> the
> >>>>>>>> namenode*. That means the new web UIs are at least as secure as
> >>> the
> >>>>>>> hadoop
> >>>>>>>> core, and the namenode machine.
> >>>>>>>>
> >>>>>>>> Second, I argue that using client-side templates are more
> >> secure
> >>>> than
> >>>>>> the
> >>>>>>>> current JSP-based server-side templates. To defend against XSS
> >>>>>>>> attacks, both techniques have to filter the external inputs at
> >>>>> *every*
> >>>>>>>> possible execution paths. Several facts much be taken into
> >>>>>>>> plays when evaluating the security of both approaches in
> >>> real-world
> >>>>>>>> environments:
> >>>>>>>>
> >>>>>>>> * The JavaScript libraries used in the new web UIs have
> >> survived
> >>> in
> >>>>>>>> extremely large-scale production tests. jQuery is used by
> >> Google
> >>>> and
> >>>>>>>> Microsoft, bootstrap is used by Twitter, and dust.js is used
> >> by
> >>>>>>> Linkedin.
> >>>>>>>> All libraries survived from hundreds of thousands of
> >>>>>>>> attack attempts on a daily basis. I agree that the libraries
> >>> might
> >>>>>> still
> >>>>>>>> be imperfect, but there's no way that we can test the JSP web
> >>>>>>>> UIs to achieve the same level of assurances given the amount
> >> of
> >>>>>>> resources
> >>>>>>>> the community has.
> >>>>>>>> * Client-side templates consolidate all filtering logic in one
> >>>>> central
> >>>>>>>> place. Recall that the goal is to filter all external inputs at
> >>>> every
> >>>>>>>> execution paths, this is a much more systematic approach
> >>> compared
> >>>> to
> >>>>>> the
> >>>>>>>> server-side templates we have today. It is difficult (if not
> >>>>>>>> impossible) to do it in a JSP/ASP/PHP application, since such
> >>>>>> filtering
> >>>>>>>> can be only achieved via ad-hoc approaches ([1] shows some
> >>>>>>>> empirical data). Also, HDFS-4901 recently describes a XSS
> >>>>>> vulnerability
> >>>>>>> in
> >>>>>>>> browseDirectory.jsp.
> >>>>>>>>
> >>>>>>>> bq. You'd require proper SSL (not self signed) setup to avoid
> >> JS
> >>>>>>>> injection
> >>>>>>>>
> >>>>>>>> Commodity browsers enforce Same-Origin Policy to defend against
> >>>> code
> >>>>>>>> injections. It has nothing to do with what kinds of SSL
> >>>> certificates
> >>>>>>>> you hold.
> >>>>>>>>
> >>>>>>>> bq.  I also have concerns that we commit these changes without
> >>>>> matching
> >>>>>>>> unit tests
> >>>>>>>>
> >>>>>>>> The JavaScript code can be automatically tested. The same code
> >>> can
> >>>> be
> >>>>>> run
> >>>>>>>> by node.js and the test can compared with pre-defined
> >>>>>>>> results. It is also possible to write an adapter to use Rhino
> >> to
> >>>>>>> accomplish
> >>>>>>>> the same task. We can discuss how to integrate them into
> >>>>>>>> the maven test routines in a different thread.
> >>>>>>>>
> >>>>>>>> bq. Client side rendering completely breaks the workflows for
> >> ops
> >>>> who
> >>>>>>> rely
> >>>>>>>> on text based terminal/emacs/vim browsers (no js support) to
> >>>>>>>> monitor component UI.
> >>>>>>>>
> >>>>>>>> links / elinks (http://elinks.or.cz/) are text-based web
> >>> browsers
> >>>>> that
> >>>>>>>> support JavaScript.
> >>>>>>>>
> >>>>>>>> bq. The priority/requirements for UI in core Hadoop should be
> >>>>> security
> >>>>>>> and
> >>>>>>>> correctness, which client side templating cannot address
> >> properly
> >>>>>>>> so far.
> >>>>>>>>
> >>>>>>>> I agree that we should focus on security and correctness. The
> >>>>>> paragraphs
> >>>>>>>> above explain that how the architecture of the new UIs
> >>>>>>>> makes the UIs more secure in real-world settings compared to
> >> the
> >>> UI
> >>>>> we
> >>>>>>> have
> >>>>>>>> today.
> >>>>>>>>
> >>>>>>>> References:
> >>>>>>>>
> >>>>>>>> 1. A. Yip et al. Improving Application Security with Data Flow
> >>>>>>> Assertions.
> >>>>>>>> In SOSP'2009.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Fri, Oct 25, 2013 at 10:02 AM, Luke Lu <ll...@apache.org>
> >>> wrote:
> >>>>>>>>
> >>>>>>>>> Echoing my comments on HDFS-3555:
> >>>>>>>>>
> >>>>>>>>> I have concerns with this client-side js only approach, which
> >> is
> >>>>> less
> >>>>>>>>> secure than a progressively enhanced hybrid approach used by
> >>> YARN.
> >>>>> The
> >>>>>>>>> recent gmail XSS fiasco highlights the issue. I also have
> >>> concerns
> >>>>>> that
> >>>>>>> we
> >>>>>>>>> commit these changes without matching unit tests – the fact
> >> you
> >>>>> cannot
> >>>>>>>>> effectively unit test these changes should tell you something
> >>>> about
> >>>>>> this
> >>>>>>>>> approach.
> >>>>>>>>>
> >>>>>>>>> *Requiring* JS means that an admin cannot turn off js to
> >>>> (partially)
> >>>>>> use
> >>>>>>>>> core Hadoop UI. You'd *require* proper SSL (not self signed)
> >>> setup
> >>>>> to
> >>>>>>> avoid
> >>>>>>>>> JS injection, even if security of js libraries used is
> >> perfect,
> >>>>> which
> >>>>>> I
> >>>>>>>>> doubt (search gmail/linkedin XSS). Client side rendering
> >>>> completely
> >>>>>>> breaks
> >>>>>>>>> the workflows for ops who rely on text based
> >> terminal/emacs/vim
> >>>>>> browsers
> >>>>>>>>> (no js support) to monitor component UI.
> >>>>>>>>>
> >>>>>>>>> IMO, JS-only rendering belongs to social networking sites
> >> and/or
> >>>>> SaaS
> >>>>>>>>> front-ends, where full time UI/security specialists babysits
> >> UI
> >>>>>>> changes. I
> >>>>>>>>> think eventually most users will use a self servicing UI in a
> >>> SaaS
> >>>>>>>>> front-end that uses REST/JMX API to get data from back-end
> >>>>> components,
> >>>>>>>>> besides their own app master/service UI. The
> >>> priority/requirements
> >>>>> for
> >>>>>>> UI
> >>>>>>>>> in core Hadoop should be security and correctness, which
> >> client
> >>>> side
> >>>>>>>>> templating cannot address properly so far.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Tue, Oct 22, 2013 at 3:59 PM, Haohui Mai <
> >>> hmai@hortonworks.com
> >>>>>
> >>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Hi all,
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Jing Zhao and I recently have reimplemented the JSP-based
> >> web
> >>>> UIs
> >>>>> in
> >>>>>>>>> HTML 5
> >>>>>>>>>> applications (HDFS-5333). Based on our prelimanary testing
> >>>> results
> >>>>>> we
> >>>>>>>>>> believe thst the new web UIs of the namenodes and the
> >> datanode
> >>>> are
> >>>>>>> ready
> >>>>>>>>>> for everyday uses.
> >>>>>>>>>>
> >>>>>>>>>> You're more than welcome to try it out on trunk by visiting
> >>>>> http://
> >>>>>>>>>> <namenode>/dfshealth.html
> >>>>>>>>>>
> >>>>>>>>>> There are a number of benefits from this transition. From a
> >>>>>>> developer's
> >>>>>>>>>> prospective, the most notable one is *maintainability*:
> >>>>>>>>>>
> >>>>>>>>>> (1) The abstractions between the UI and the core server are
> >>>>>>> well-defined,
> >>>>>>>>>> decoupling the UI and the core hadoop servers.
> >>>>>>>>>>
> >>>>>>>>>> (2) It allows us to deprecate the logic in the JSP pages.
> >> The
> >>>> old
> >>>>>> web
> >>>>>>> UIs
> >>>>>>>>>> have to duplicate the logic in the JSPs. The logic is often
> >>>>>>> out-of-dated
> >>>>>>>>>> and not well-tested, which leads to broken pages and
> >> security
> >>>>>>>>>> vulnerabilities(e.g. HDFS-5251, HDFS-5307, HDFS-5308,
> >>> HDFS-5317
> >>>>> and
> >>>>>>>>>> HDFS-4901). The architecture of the new UIs prevent these
> >> bugs
> >>>> at
> >>>>>> the
> >>>>>>>>> very
> >>>>>>>>>> beginning.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> I propose that deprecate the old, JSP-based web UIs in 2.3.
> >> I
> >>>>> opened
> >>>>>>>>>> HDFS-5402 to track the relevant discussions.
> >>>>>>>>>>
> >>>>>>>>>> Your feedbacks are highly appreciated.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Sincerely,
> >>>>>>>>>>
> >>>>>>>>>> Haohui
> >>>>>>>>>>
> >>>>>>>>>> --
> >>>>>>>>>> CONFIDENTIALITY NOTICE
> >>>>>>>>>> NOTICE: This message is intended for the use of the
> >> individual
> >>>> or
> >>>>>>> entity
> >>>>>>>>> to
> >>>>>>>>>> which it is addressed and may contain information that is
> >>>>>>> confidential,
> >>>>>>>>>> privileged and exempt from disclosure under applicable law.
> >> If
> >>>> the
> >>>>>>> reader
> >>>>>>>>>> of this message is not the intended recipient, you are
> >> hereby
> >>>>>> notified
> >>>>>>>>> that
> >>>>>>>>>> any printing, copying, dissemination, distribution,
> >> disclosure
> >>>> or
> >>>>>>>>>> forwarding of this communication is strictly prohibited. If
> >>> you
> >>>>> have
> >>>>>>>>>> received this communication in error, please contact the
> >>> sender
> >>>>>>>>> immediately
> >>>>>>>>>> and delete it from your system. Thank You.
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>> CONFIDENTIALITY NOTICE
> >>>>>>>> NOTICE: This message is intended for the use of the individual
> >> or
> >>>>>> entity
> >>>>>>> to
> >>>>>>>> which it is addressed and may contain information that is
> >>>>> confidential,
> >>>>>>>> privileged and exempt from disclosure under applicable law. If
> >>> the
> >>>>>> reader
> >>>>>>>> of this message is not the intended recipient, you are hereby
> >>>>> notified
> >>>>>>> that
> >>>>>>>> any printing, copying, dissemination, distribution, disclosure
> >> or
> >>>>>>>> forwarding of this communication is strictly prohibited. If you
> >>>> have
> >>>>>>>> received this communication in error, please contact the sender
> >>>>>>> immediately
> >>>>>>>> and delete it from your system. Thank You.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> Alejandro
> >>>>>
> >>>>> --
> >>>>> CONFIDENTIALITY NOTICE
> >>>>> NOTICE: This message is intended for the use of the individual or
> >>> entity
> >>>> to
> >>>>> which it is addressed and may contain information that is
> >> confidential,
> >>>>> privileged and exempt from disclosure under applicable law. If the
> >>> reader
> >>>>> of this message is not the intended recipient, you are hereby
> >> notified
> >>>> that
> >>>>> any printing, copying, dissemination, distribution, disclosure or
> >>>>> forwarding of this communication is strictly prohibited. If you have
> >>>>> received this communication in error, please contact the sender
> >>>> immediately
> >>>>> and delete it from your system. Thank You.
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Alejandro
> >>>
> >>> --
> >>> CONFIDENTIALITY NOTICE
> >>> NOTICE: This message is intended for the use of the individual or
> entity
> >> to
> >>> which it is addressed and may contain information that is confidential,
> >>> privileged and exempt from disclosure under applicable law. If the
> reader
> >>> of this message is not the intended recipient, you are hereby notified
> >> that
> >>> any printing, copying, dissemination, distribution, disclosure or
> >>> forwarding of this communication is strictly prohibited. If you have
> >>> received this communication in error, please contact the sender
> >> immediately
> >>> and delete it from your system. Thank You.
> >>
> >>
> >>
> >> --
> >> Alejandro
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Replacing the JSP web UIs to HTML 5 applications

[Alejandro Abdelnur <tu...@cloudera.com>]

are you planning to expose things like hdfs browsing and nn/dn logs over jmx?

thx

Alejandro
(phone typing)

On Oct 28, 2013, at 17:48, Haohui Mai <hm...@hortonworks.com> wrote:

> It seems more appealing to me that the UI should JMX directly, because:
> 
> * We're support the JMX in the long term for other management software.
> * The information provided by the JMX API will be mostly identical of the
> JSON API. Today the Web UI covers most of the information provided by JMX.
> The Web UI does some trivial work to extract the information and renders it
> as HTML.
> * We can compatibility and unit tests for free.
> 
> I do agree that the JMX APIs are imperfect and they should be revisited in
> the 3.0 timeframe. However, this is orthogonal of the discussions of
> transitioning from JSP-based Web UI to client-side JavaScript Web UI. The
> architecture of the new Web UI allows easy migration to any JSON-based APIs
> whenever they land in the trunk.
> 
> Thanks,
> Haohui
> 
> 
> On Mon, Oct 28, 2013 at 5:13 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
> 
>> Isn't using JMX to expose JSON for the web UI misusing JMX?
>> 
>> I would think a more appropriate approach would be having /JMX for
>> monitoring integration and a /JSON end point for the UI data.
>> 
>> Thanks.
>> 
>> 
>> On Mon, Oct 28, 2013 at 4:58 PM, Haohui Mai <hm...@hortonworks.com> wrote:
>> 
>>> Alejandro,
>>> 
>>> If I understand correctly, that is the exact approach that the new web UI
>>> is taking. The new web UI takes the output from JMX and renders them as
>>> HTML at the client side.
>>> 
>>> ~Haohui
>>> 
>>> 
>>> On Mon, Oct 28, 2013 at 4:18 PM, Alejandro Abdelnur <tucu@cloudera.com
>>>> wrote:
>>> 
>>>> Haohui,
>>>> 
>>>> If you have NN and DNs producing JSON instead HTML, then you can build
>> JS
>>>> based web UIs. Take for example Oozie, Oozie produces JSON, it has a
>>> built
>>>> in JS web ui that consumes JSON and Hue has built an external web UI
>> that
>>>> also consumes JSON. In the case of Hue UI, Oozie didn't have to change
>>>> anything to get that UI and improvements on the Hue UI don't require
>>>> changes in Oozie unless it is to produce additional information.
>>>> 
>>>> hope this clarifies.
>>>> 
>>>> Thx
>>>> 
>>>> 
>>>> On Mon, Oct 28, 2013 at 4:06 PM, Haohui Mai <hm...@hortonworks.com>
>>> wrote:
>>>> 
>>>>> Echo my comments on HDFS-5402:
>>>>> 
>>>>> bq. If we're going to remove the old web UI, I think the new web UI
>> has
>>>>> to have the same level of unit testing. We shouldn't go backwards in
>>>>> terms of unit testing.
>>>>> 
>>>>> I take a look at TestNamenodeJspHelper / TestDatanodeJspHelper /
>>>>> TestClusterJspHelper. It seems to me that we can merge these tests
>> with
>>>> the
>>>>> unit tests on JMX.
>>>>> 
>>>>> bq. If we are going to
>>>>> remove this capability, we need to add some other command-line tools
>>>>> to get the same functionality. These tools could use REST if we have
>>>>> that, or JMX, but they need to exist before we can consider removing
>>>>> the old UI.
>>>>> 
>>>>> This is a good point. Since all information are available through
>> JMX,
>>>> the
>>>>> easiest way to approach it is to write some scripts using Node.js.
>> The
>>>>> architecture of the new Web UIs is ready for this.
>>>>> 
>>>>> 
>>>>> On Mon, Oct 28, 2013 at 3:57 PM, Alejandro Abdelnur <
>> tucu@cloudera.com
>>>>>> wrote:
>>>>> 
>>>>>> Producing JSON would be great. Agree with Colin that we should
>> leave
>>>> for
>>>>>> now the current JSP based web ui.
>>>>>> 
>>>>>> thx
>>>>>> 
>>>>>> 
>>>>>> On Mon, Oct 28, 2013 at 11:16 AM, Colin McCabe <
>>> cmccabe@alumni.cmu.edu
>>>>>>> wrote:
>>>>>> 
>>>>>>> This is a really interesting project, Haohui.  I think it will
>> make
>>>>>>> our web UI much nicer.
>>>>>>> 
>>>>>>> I have a few concerns about removing the old web UI, however:
>>>>>>> 
>>>>>>> * If we're going to remove the old web UI, I think the new web UI
>>> has
>>>>>>> to have the same level of unit testing.  We shouldn't go
>> backwards
>>> in
>>>>>>> terms of unit testing.
>>>>>>> 
>>>>>>> * Most of the deployments of elinks and links out there don't
>>> support
>>>>>>> Javascript.  This is just a reality of life when using CentOS 5
>> or
>>> 6,
>>>>>>> which many users are still using.  I have used "links" to
>> diagnose
>>>>>>> problems through the web UI in the past, in systems where access
>> to
>>>>>>> the cluster was available only through telnet.  If we are going
>> to
>>>>>>> remove this capability, we need to add some other command-line
>>> tools
>>>>>>> to get the same functionality.  These tools could use REST if we
>>> have
>>>>>>> that, or JMX, but they need to exist before we can consider
>>> removing
>>>>>>> the old UI.
>>>>>>> 
>>>>>>> best,
>>>>>>> Colin
>>>>>>> 
>>>>>>> On Fri, Oct 25, 2013 at 7:31 PM, Haohui Mai <
>> hmai@hortonworks.com>
>>>>>> wrote:
>>>>>>>> Thanks for the reply, Luke. Here I just echo my response from
>> the
>>>>> jira:
>>>>>>>> 
>>>>>>>> bq. this client-side js only approach, which is less secure
>> than
>>> a
>>>>>>>> progressively enhanced hybrid approach used by YARN. The recent
>>>> gmail
>>>>>>>> XSS fiasco highlights the issue.
>>>>>>>> 
>>>>>>>> I'm presenting an informal security analysis to compare the
>>>> security
>>>>> of
>>>>>>> the
>>>>>>>> old and the new web UIs.
>>>>>>>> 
>>>>>>>> An attacker launches an XSS attack by injecting malicious code
>>>> which
>>>>>> are
>>>>>>>> usually HTML or JavaScript fragments into the web page, so that
>>> the
>>>>>>>> malicious code can have the same privileges of the web page.
>>>>>>>> 
>>>>>>>> First, in the scope of XSS attacks, note that the threat models
>>> of
>>>>>>>> launching XSS attacks on Internet sites Gmail/Linkedin and the
>>> one
>>>> of
>>>>>> the
>>>>>>>> Hadoop UIs are different. They have fundamental different sets
>> of
>>>>>>> external
>>>>>>>> inputs that the attackers have control to. Internet sites have
>>>> little
>>>>>>>> control of these inputs. In the case of Gmail / Linkedin, an
>>> attack
>>>>> can
>>>>>>>> send you a crafted e-mail, or put malicious description in his
>> /
>>>>>>>> her Linkedin profile. The sets of external inputs are
>>> *restricted*
>>>> in
>>>>>>>> Hadoop UIs. The new web UIs take JMX and WebHDFS as inputs. The
>>>>>>>> attacker has to launch a XSS attack by:
>>>>>>>> 
>>>>>>>> * Compromise the jars so that the output of JMX / WebHDFS have
>>> the
>>>>>>>> malicious code.
>>>>>>>> * Replace the web UIs completely to include the malicious code.
>>>>>>>> 
>>>>>>>> In either case *the attacker has to compromise the hadoop core
>> or
>>>> the
>>>>>>>> namenode*. That means the new web UIs are at least as secure as
>>> the
>>>>>>> hadoop
>>>>>>>> core, and the namenode machine.
>>>>>>>> 
>>>>>>>> Second, I argue that using client-side templates are more
>> secure
>>>> than
>>>>>> the
>>>>>>>> current JSP-based server-side templates. To defend against XSS
>>>>>>>> attacks, both techniques have to filter the external inputs at
>>>>> *every*
>>>>>>>> possible execution paths. Several facts much be taken into
>>>>>>>> plays when evaluating the security of both approaches in
>>> real-world
>>>>>>>> environments:
>>>>>>>> 
>>>>>>>> * The JavaScript libraries used in the new web UIs have
>> survived
>>> in
>>>>>>>> extremely large-scale production tests. jQuery is used by
>> Google
>>>> and
>>>>>>>> Microsoft, bootstrap is used by Twitter, and dust.js is used
>> by
>>>>>>> Linkedin.
>>>>>>>> All libraries survived from hundreds of thousands of
>>>>>>>> attack attempts on a daily basis. I agree that the libraries
>>> might
>>>>>> still
>>>>>>>> be imperfect, but there's no way that we can test the JSP web
>>>>>>>> UIs to achieve the same level of assurances given the amount
>> of
>>>>>>> resources
>>>>>>>> the community has.
>>>>>>>> * Client-side templates consolidate all filtering logic in one
>>>>> central
>>>>>>>> place. Recall that the goal is to filter all external inputs at
>>>> every
>>>>>>>> execution paths, this is a much more systematic approach
>>> compared
>>>> to
>>>>>> the
>>>>>>>> server-side templates we have today. It is difficult (if not
>>>>>>>> impossible) to do it in a JSP/ASP/PHP application, since such
>>>>>> filtering
>>>>>>>> can be only achieved via ad-hoc approaches ([1] shows some
>>>>>>>> empirical data). Also, HDFS-4901 recently describes a XSS
>>>>>> vulnerability
>>>>>>> in
>>>>>>>> browseDirectory.jsp.
>>>>>>>> 
>>>>>>>> bq. You'd require proper SSL (not self signed) setup to avoid
>> JS
>>>>>>>> injection
>>>>>>>> 
>>>>>>>> Commodity browsers enforce Same-Origin Policy to defend against
>>>> code
>>>>>>>> injections. It has nothing to do with what kinds of SSL
>>>> certificates
>>>>>>>> you hold.
>>>>>>>> 
>>>>>>>> bq.  I also have concerns that we commit these changes without
>>>>> matching
>>>>>>>> unit tests
>>>>>>>> 
>>>>>>>> The JavaScript code can be automatically tested. The same code
>>> can
>>>> be
>>>>>> run
>>>>>>>> by node.js and the test can compared with pre-defined
>>>>>>>> results. It is also possible to write an adapter to use Rhino
>> to
>>>>>>> accomplish
>>>>>>>> the same task. We can discuss how to integrate them into
>>>>>>>> the maven test routines in a different thread.
>>>>>>>> 
>>>>>>>> bq. Client side rendering completely breaks the workflows for
>> ops
>>>> who
>>>>>>> rely
>>>>>>>> on text based terminal/emacs/vim browsers (no js support) to
>>>>>>>> monitor component UI.
>>>>>>>> 
>>>>>>>> links / elinks (http://elinks.or.cz/) are text-based web
>>> browsers
>>>>> that
>>>>>>>> support JavaScript.
>>>>>>>> 
>>>>>>>> bq. The priority/requirements for UI in core Hadoop should be
>>>>> security
>>>>>>> and
>>>>>>>> correctness, which client side templating cannot address
>> properly
>>>>>>>> so far.
>>>>>>>> 
>>>>>>>> I agree that we should focus on security and correctness. The
>>>>>> paragraphs
>>>>>>>> above explain that how the architecture of the new UIs
>>>>>>>> makes the UIs more secure in real-world settings compared to
>> the
>>> UI
>>>>> we
>>>>>>> have
>>>>>>>> today.
>>>>>>>> 
>>>>>>>> References:
>>>>>>>> 
>>>>>>>> 1. A. Yip et al. Improving Application Security with Data Flow
>>>>>>> Assertions.
>>>>>>>> In SOSP'2009.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On Fri, Oct 25, 2013 at 10:02 AM, Luke Lu <ll...@apache.org>
>>> wrote:
>>>>>>>> 
>>>>>>>>> Echoing my comments on HDFS-3555:
>>>>>>>>> 
>>>>>>>>> I have concerns with this client-side js only approach, which
>> is
>>>>> less
>>>>>>>>> secure than a progressively enhanced hybrid approach used by
>>> YARN.
>>>>> The
>>>>>>>>> recent gmail XSS fiasco highlights the issue. I also have
>>> concerns
>>>>>> that
>>>>>>> we
>>>>>>>>> commit these changes without matching unit tests – the fact
>> you
>>>>> cannot
>>>>>>>>> effectively unit test these changes should tell you something
>>>> about
>>>>>> this
>>>>>>>>> approach.
>>>>>>>>> 
>>>>>>>>> *Requiring* JS means that an admin cannot turn off js to
>>>> (partially)
>>>>>> use
>>>>>>>>> core Hadoop UI. You'd *require* proper SSL (not self signed)
>>> setup
>>>>> to
>>>>>>> avoid
>>>>>>>>> JS injection, even if security of js libraries used is
>> perfect,
>>>>> which
>>>>>> I
>>>>>>>>> doubt (search gmail/linkedin XSS). Client side rendering
>>>> completely
>>>>>>> breaks
>>>>>>>>> the workflows for ops who rely on text based
>> terminal/emacs/vim
>>>>>> browsers
>>>>>>>>> (no js support) to monitor component UI.
>>>>>>>>> 
>>>>>>>>> IMO, JS-only rendering belongs to social networking sites
>> and/or
>>>>> SaaS
>>>>>>>>> front-ends, where full time UI/security specialists babysits
>> UI
>>>>>>> changes. I
>>>>>>>>> think eventually most users will use a self servicing UI in a
>>> SaaS
>>>>>>>>> front-end that uses REST/JMX API to get data from back-end
>>>>> components,
>>>>>>>>> besides their own app master/service UI. The
>>> priority/requirements
>>>>> for
>>>>>>> UI
>>>>>>>>> in core Hadoop should be security and correctness, which
>> client
>>>> side
>>>>>>>>> templating cannot address properly so far.
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Tue, Oct 22, 2013 at 3:59 PM, Haohui Mai <
>>> hmai@hortonworks.com
>>>>> 
>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> Hi all,
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> Jing Zhao and I recently have reimplemented the JSP-based
>> web
>>>> UIs
>>>>> in
>>>>>>>>> HTML 5
>>>>>>>>>> applications (HDFS-5333). Based on our prelimanary testing
>>>> results
>>>>>> we
>>>>>>>>>> believe thst the new web UIs of the namenodes and the
>> datanode
>>>> are
>>>>>>> ready
>>>>>>>>>> for everyday uses.
>>>>>>>>>> 
>>>>>>>>>> You're more than welcome to try it out on trunk by visiting
>>>>> http://
>>>>>>>>>> <namenode>/dfshealth.html
>>>>>>>>>> 
>>>>>>>>>> There are a number of benefits from this transition. From a
>>>>>>> developer's
>>>>>>>>>> prospective, the most notable one is *maintainability*:
>>>>>>>>>> 
>>>>>>>>>> (1) The abstractions between the UI and the core server are
>>>>>>> well-defined,
>>>>>>>>>> decoupling the UI and the core hadoop servers.
>>>>>>>>>> 
>>>>>>>>>> (2) It allows us to deprecate the logic in the JSP pages.
>> The
>>>> old
>>>>>> web
>>>>>>> UIs
>>>>>>>>>> have to duplicate the logic in the JSPs. The logic is often
>>>>>>> out-of-dated
>>>>>>>>>> and not well-tested, which leads to broken pages and
>> security
>>>>>>>>>> vulnerabilities(e.g. HDFS-5251, HDFS-5307, HDFS-5308,
>>> HDFS-5317
>>>>> and
>>>>>>>>>> HDFS-4901). The architecture of the new UIs prevent these
>> bugs
>>>> at
>>>>>> the
>>>>>>>>> very
>>>>>>>>>> beginning.
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> I propose that deprecate the old, JSP-based web UIs in 2.3.
>> I
>>>>> opened
>>>>>>>>>> HDFS-5402 to track the relevant discussions.
>>>>>>>>>> 
>>>>>>>>>> Your feedbacks are highly appreciated.
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> Sincerely,
>>>>>>>>>> 
>>>>>>>>>> Haohui
>>>>>>>>>> 
>>>>>>>>>> --
>>>>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>>>>> NOTICE: This message is intended for the use of the
>> individual
>>>> or
>>>>>>> entity
>>>>>>>>> to
>>>>>>>>>> which it is addressed and may contain information that is
>>>>>>> confidential,
>>>>>>>>>> privileged and exempt from disclosure under applicable law.
>> If
>>>> the
>>>>>>> reader
>>>>>>>>>> of this message is not the intended recipient, you are
>> hereby
>>>>>> notified
>>>>>>>>> that
>>>>>>>>>> any printing, copying, dissemination, distribution,
>> disclosure
>>>> or
>>>>>>>>>> forwarding of this communication is strictly prohibited. If
>>> you
>>>>> have
>>>>>>>>>> received this communication in error, please contact the
>>> sender
>>>>>>>>> immediately
>>>>>>>>>> and delete it from your system. Thank You.
>>>>>>>> 
>>>>>>>> --
>>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>>> NOTICE: This message is intended for the use of the individual
>> or
>>>>>> entity
>>>>>>> to
>>>>>>>> which it is addressed and may contain information that is
>>>>> confidential,
>>>>>>>> privileged and exempt from disclosure under applicable law. If
>>> the
>>>>>> reader
>>>>>>>> of this message is not the intended recipient, you are hereby
>>>>> notified
>>>>>>> that
>>>>>>>> any printing, copying, dissemination, distribution, disclosure
>> or
>>>>>>>> forwarding of this communication is strictly prohibited. If you
>>>> have
>>>>>>>> received this communication in error, please contact the sender
>>>>>>> immediately
>>>>>>>> and delete it from your system. Thank You.
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> --
>>>>>> Alejandro
>>>>> 
>>>>> --
>>>>> CONFIDENTIALITY NOTICE
>>>>> NOTICE: This message is intended for the use of the individual or
>>> entity
>>>> to
>>>>> which it is addressed and may contain information that is
>> confidential,
>>>>> privileged and exempt from disclosure under applicable law. If the
>>> reader
>>>>> of this message is not the intended recipient, you are hereby
>> notified
>>>> that
>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>> forwarding of this communication is strictly prohibited. If you have
>>>>> received this communication in error, please contact the sender
>>>> immediately
>>>>> and delete it from your system. Thank You.
>>>> 
>>>> 
>>>> 
>>>> --
>>>> Alejandro
>>> 
>>> --
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>> to
>>> which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified
>> that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender
>> immediately
>>> and delete it from your system. Thank You.
>> 
>> 
>> 
>> --
>> Alejandro
> 
> -- 
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to 
> which it is addressed and may contain information that is confidential, 
> privileged and exempt from disclosure under applicable law. If the reader 
> of this message is not the intended recipient, you are hereby notified that 
> any printing, copying, dissemination, distribution, disclosure or 
> forwarding of this communication is strictly prohibited. If you have 
> received this communication in error, please contact the sender immediately 
> and delete it from your system. Thank You.

Re: Replacing the JSP web UIs to HTML 5 applications

[Haohui Mai <hm...@hortonworks.com>]

It seems more appealing to me that the UI should JMX directly, because:

* We're support the JMX in the long term for other management software.
* The information provided by the JMX API will be mostly identical of the
JSON API. Today the Web UI covers most of the information provided by JMX.
The Web UI does some trivial work to extract the information and renders it
as HTML.
* We can compatibility and unit tests for free.

I do agree that the JMX APIs are imperfect and they should be revisited in
the 3.0 timeframe. However, this is orthogonal of the discussions of
transitioning from JSP-based Web UI to client-side JavaScript Web UI. The
architecture of the new Web UI allows easy migration to any JSON-based APIs
whenever they land in the trunk.

Thanks,
Haohui


On Mon, Oct 28, 2013 at 5:13 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:

> Isn't using JMX to expose JSON for the web UI misusing JMX?
>
> I would think a more appropriate approach would be having /JMX for
> monitoring integration and a /JSON end point for the UI data.
>
> Thanks.
>
>
> On Mon, Oct 28, 2013 at 4:58 PM, Haohui Mai <hm...@hortonworks.com> wrote:
>
> > Alejandro,
> >
> > If I understand correctly, that is the exact approach that the new web UI
> > is taking. The new web UI takes the output from JMX and renders them as
> > HTML at the client side.
> >
> > ~Haohui
> >
> >
> > On Mon, Oct 28, 2013 at 4:18 PM, Alejandro Abdelnur <tucu@cloudera.com
> > >wrote:
> >
> > > Haohui,
> > >
> > > If you have NN and DNs producing JSON instead HTML, then you can build
> JS
> > > based web UIs. Take for example Oozie, Oozie produces JSON, it has a
> > built
> > > in JS web ui that consumes JSON and Hue has built an external web UI
> that
> > > also consumes JSON. In the case of Hue UI, Oozie didn't have to change
> > > anything to get that UI and improvements on the Hue UI don't require
> > > changes in Oozie unless it is to produce additional information.
> > >
> > > hope this clarifies.
> > >
> > > Thx
> > >
> > >
> > > On Mon, Oct 28, 2013 at 4:06 PM, Haohui Mai <hm...@hortonworks.com>
> > wrote:
> > >
> > > > Echo my comments on HDFS-5402:
> > > >
> > > > bq. If we're going to remove the old web UI, I think the new web UI
> has
> > > > to have the same level of unit testing. We shouldn't go backwards in
> > > > terms of unit testing.
> > > >
> > > > I take a look at TestNamenodeJspHelper / TestDatanodeJspHelper /
> > > > TestClusterJspHelper. It seems to me that we can merge these tests
> with
> > > the
> > > > unit tests on JMX.
> > > >
> > > > bq. If we are going to
> > > > remove this capability, we need to add some other command-line tools
> > > > to get the same functionality. These tools could use REST if we have
> > > > that, or JMX, but they need to exist before we can consider removing
> > > > the old UI.
> > > >
> > > > This is a good point. Since all information are available through
> JMX,
> > > the
> > > > easiest way to approach it is to write some scripts using Node.js.
> The
> > > > architecture of the new Web UIs is ready for this.
> > > >
> > > >
> > > > On Mon, Oct 28, 2013 at 3:57 PM, Alejandro Abdelnur <
> tucu@cloudera.com
> > > > >wrote:
> > > >
> > > > > Producing JSON would be great. Agree with Colin that we should
> leave
> > > for
> > > > > now the current JSP based web ui.
> > > > >
> > > > > thx
> > > > >
> > > > >
> > > > > On Mon, Oct 28, 2013 at 11:16 AM, Colin McCabe <
> > cmccabe@alumni.cmu.edu
> > > > > >wrote:
> > > > >
> > > > > > This is a really interesting project, Haohui.  I think it will
> make
> > > > > > our web UI much nicer.
> > > > > >
> > > > > > I have a few concerns about removing the old web UI, however:
> > > > > >
> > > > > > * If we're going to remove the old web UI, I think the new web UI
> > has
> > > > > > to have the same level of unit testing.  We shouldn't go
> backwards
> > in
> > > > > > terms of unit testing.
> > > > > >
> > > > > > * Most of the deployments of elinks and links out there don't
> > support
> > > > > > Javascript.  This is just a reality of life when using CentOS 5
> or
> > 6,
> > > > > > which many users are still using.  I have used "links" to
> diagnose
> > > > > > problems through the web UI in the past, in systems where access
> to
> > > > > > the cluster was available only through telnet.  If we are going
> to
> > > > > > remove this capability, we need to add some other command-line
> > tools
> > > > > > to get the same functionality.  These tools could use REST if we
> > have
> > > > > > that, or JMX, but they need to exist before we can consider
> > removing
> > > > > > the old UI.
> > > > > >
> > > > > > best,
> > > > > > Colin
> > > > > >
> > > > > > On Fri, Oct 25, 2013 at 7:31 PM, Haohui Mai <
> hmai@hortonworks.com>
> > > > > wrote:
> > > > > > > Thanks for the reply, Luke. Here I just echo my response from
> the
> > > > jira:
> > > > > > >
> > > > > > > bq. this client-side js only approach, which is less secure
> than
> > a
> > > > > > > progressively enhanced hybrid approach used by YARN. The recent
> > > gmail
> > > > > > > XSS fiasco highlights the issue.
> > > > > > >
> > > > > > > I'm presenting an informal security analysis to compare the
> > > security
> > > > of
> > > > > > the
> > > > > > > old and the new web UIs.
> > > > > > >
> > > > > > > An attacker launches an XSS attack by injecting malicious code
> > > which
> > > > > are
> > > > > > > usually HTML or JavaScript fragments into the web page, so that
> > the
> > > > > > > malicious code can have the same privileges of the web page.
> > > > > > >
> > > > > > > First, in the scope of XSS attacks, note that the threat models
> > of
> > > > > > > launching XSS attacks on Internet sites Gmail/Linkedin and the
> > one
> > > of
> > > > > the
> > > > > > > Hadoop UIs are different. They have fundamental different sets
> of
> > > > > > external
> > > > > > > inputs that the attackers have control to. Internet sites have
> > > little
> > > > > > > control of these inputs. In the case of Gmail / Linkedin, an
> > attack
> > > > can
> > > > > > > send you a crafted e-mail, or put malicious description in his
> /
> > > > > > > her Linkedin profile. The sets of external inputs are
> > *restricted*
> > > in
> > > > > > > Hadoop UIs. The new web UIs take JMX and WebHDFS as inputs. The
> > > > > > > attacker has to launch a XSS attack by:
> > > > > > >
> > > > > > > * Compromise the jars so that the output of JMX / WebHDFS have
> > the
> > > > > > > malicious code.
> > > > > > > * Replace the web UIs completely to include the malicious code.
> > > > > > >
> > > > > > > In either case *the attacker has to compromise the hadoop core
> or
> > > the
> > > > > > > namenode*. That means the new web UIs are at least as secure as
> > the
> > > > > > hadoop
> > > > > > > core, and the namenode machine.
> > > > > > >
> > > > > > > Second, I argue that using client-side templates are more
> secure
> > > than
> > > > > the
> > > > > > > current JSP-based server-side templates. To defend against XSS
> > > > > > > attacks, both techniques have to filter the external inputs at
> > > > *every*
> > > > > > > possible execution paths. Several facts much be taken into
> > > > > > > plays when evaluating the security of both approaches in
> > real-world
> > > > > > > environments:
> > > > > > >
> > > > > > > * The JavaScript libraries used in the new web UIs have
> survived
> > in
> > > > > > > extremely large-scale production tests. jQuery is used by
> Google
> > > and
> > > > > > >  Microsoft, bootstrap is used by Twitter, and dust.js is used
> by
> > > > > > Linkedin.
> > > > > > > All libraries survived from hundreds of thousands of
> > > > > > >  attack attempts on a daily basis. I agree that the libraries
> > might
> > > > > still
> > > > > > > be imperfect, but there's no way that we can test the JSP web
> > > > > > >  UIs to achieve the same level of assurances given the amount
> of
> > > > > > resources
> > > > > > > the community has.
> > > > > > > * Client-side templates consolidate all filtering logic in one
> > > > central
> > > > > > > place. Recall that the goal is to filter all external inputs at
> > > every
> > > > > > >  execution paths, this is a much more systematic approach
> > compared
> > > to
> > > > > the
> > > > > > > server-side templates we have today. It is difficult (if not
> > > > > > >  impossible) to do it in a JSP/ASP/PHP application, since such
> > > > > filtering
> > > > > > > can be only achieved via ad-hoc approaches ([1] shows some
> > > > > > >  empirical data). Also, HDFS-4901 recently describes a XSS
> > > > > vulnerability
> > > > > > in
> > > > > > > browseDirectory.jsp.
> > > > > > >
> > > > > > > bq. You'd require proper SSL (not self signed) setup to avoid
> JS
> > > > > > > injection
> > > > > > >
> > > > > > > Commodity browsers enforce Same-Origin Policy to defend against
> > > code
> > > > > > > injections. It has nothing to do with what kinds of SSL
> > > certificates
> > > > > > > you hold.
> > > > > > >
> > > > > > > bq.  I also have concerns that we commit these changes without
> > > > matching
> > > > > > > unit tests
> > > > > > >
> > > > > > > The JavaScript code can be automatically tested. The same code
> > can
> > > be
> > > > > run
> > > > > > > by node.js and the test can compared with pre-defined
> > > > > > > results. It is also possible to write an adapter to use Rhino
> to
> > > > > > accomplish
> > > > > > > the same task. We can discuss how to integrate them into
> > > > > > > the maven test routines in a different thread.
> > > > > > >
> > > > > > > bq. Client side rendering completely breaks the workflows for
> ops
> > > who
> > > > > > rely
> > > > > > > on text based terminal/emacs/vim browsers (no js support) to
> > > > > > > monitor component UI.
> > > > > > >
> > > > > > > links / elinks (http://elinks.or.cz/) are text-based web
> > browsers
> > > > that
> > > > > > > support JavaScript.
> > > > > > >
> > > > > > > bq. The priority/requirements for UI in core Hadoop should be
> > > > security
> > > > > > and
> > > > > > > correctness, which client side templating cannot address
> properly
> > > > > > > so far.
> > > > > > >
> > > > > > > I agree that we should focus on security and correctness. The
> > > > > paragraphs
> > > > > > > above explain that how the architecture of the new UIs
> > > > > > > makes the UIs more secure in real-world settings compared to
> the
> > UI
> > > > we
> > > > > > have
> > > > > > > today.
> > > > > > >
> > > > > > > References:
> > > > > > >
> > > > > > > 1. A. Yip et al. Improving Application Security with Data Flow
> > > > > > Assertions.
> > > > > > > In SOSP'2009.
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Oct 25, 2013 at 10:02 AM, Luke Lu <ll...@apache.org>
> > wrote:
> > > > > > >
> > > > > > >> Echoing my comments on HDFS-3555:
> > > > > > >>
> > > > > > >> I have concerns with this client-side js only approach, which
> is
> > > > less
> > > > > > >> secure than a progressively enhanced hybrid approach used by
> > YARN.
> > > > The
> > > > > > >> recent gmail XSS fiasco highlights the issue. I also have
> > concerns
> > > > > that
> > > > > > we
> > > > > > >> commit these changes without matching unit tests – the fact
> you
> > > > cannot
> > > > > > >> effectively unit test these changes should tell you something
> > > about
> > > > > this
> > > > > > >> approach.
> > > > > > >>
> > > > > > >> *Requiring* JS means that an admin cannot turn off js to
> > > (partially)
> > > > > use
> > > > > > >> core Hadoop UI. You'd *require* proper SSL (not self signed)
> > setup
> > > > to
> > > > > > avoid
> > > > > > >> JS injection, even if security of js libraries used is
> perfect,
> > > > which
> > > > > I
> > > > > > >> doubt (search gmail/linkedin XSS). Client side rendering
> > > completely
> > > > > > breaks
> > > > > > >> the workflows for ops who rely on text based
> terminal/emacs/vim
> > > > > browsers
> > > > > > >> (no js support) to monitor component UI.
> > > > > > >>
> > > > > > >> IMO, JS-only rendering belongs to social networking sites
> and/or
> > > > SaaS
> > > > > > >> front-ends, where full time UI/security specialists babysits
> UI
> > > > > > changes. I
> > > > > > >> think eventually most users will use a self servicing UI in a
> > SaaS
> > > > > > >> front-end that uses REST/JMX API to get data from back-end
> > > > components,
> > > > > > >> besides their own app master/service UI. The
> > priority/requirements
> > > > for
> > > > > > UI
> > > > > > >> in core Hadoop should be security and correctness, which
> client
> > > side
> > > > > > >> templating cannot address properly so far.
> > > > > > >>
> > > > > > >>
> > > > > > >> On Tue, Oct 22, 2013 at 3:59 PM, Haohui Mai <
> > hmai@hortonworks.com
> > > >
> > > > > > wrote:
> > > > > > >>
> > > > > > >> > Hi all,
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > Jing Zhao and I recently have reimplemented the JSP-based
> web
> > > UIs
> > > > in
> > > > > > >> HTML 5
> > > > > > >> > applications (HDFS-5333). Based on our prelimanary testing
> > > results
> > > > > we
> > > > > > >> > believe thst the new web UIs of the namenodes and the
> datanode
> > > are
> > > > > > ready
> > > > > > >> > for everyday uses.
> > > > > > >> >
> > > > > > >> > You're more than welcome to try it out on trunk by visiting
> > > > http://
> > > > > > >> > <namenode>/dfshealth.html
> > > > > > >> >
> > > > > > >> > There are a number of benefits from this transition. From a
> > > > > > developer's
> > > > > > >> > prospective, the most notable one is *maintainability*:
> > > > > > >> >
> > > > > > >> > (1) The abstractions between the UI and the core server are
> > > > > > well-defined,
> > > > > > >> > decoupling the UI and the core hadoop servers.
> > > > > > >> >
> > > > > > >> > (2) It allows us to deprecate the logic in the JSP pages.
> The
> > > old
> > > > > web
> > > > > > UIs
> > > > > > >> > have to duplicate the logic in the JSPs. The logic is often
> > > > > > out-of-dated
> > > > > > >> > and not well-tested, which leads to broken pages and
> security
> > > > > > >> > vulnerabilities(e.g. HDFS-5251, HDFS-5307, HDFS-5308,
> > HDFS-5317
> > > > and
> > > > > > >> > HDFS-4901). The architecture of the new UIs prevent these
> bugs
> > > at
> > > > > the
> > > > > > >> very
> > > > > > >> > beginning.
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > I propose that deprecate the old, JSP-based web UIs in 2.3.
> I
> > > > opened
> > > > > > >> > HDFS-5402 to track the relevant discussions.
> > > > > > >> >
> > > > > > >> > Your feedbacks are highly appreciated.
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > Sincerely,
> > > > > > >> >
> > > > > > >> > Haohui
> > > > > > >> >
> > > > > > >> > --
> > > > > > >> > CONFIDENTIALITY NOTICE
> > > > > > >> > NOTICE: This message is intended for the use of the
> individual
> > > or
> > > > > > entity
> > > > > > >> to
> > > > > > >> > which it is addressed and may contain information that is
> > > > > > confidential,
> > > > > > >> > privileged and exempt from disclosure under applicable law.
> If
> > > the
> > > > > > reader
> > > > > > >> > of this message is not the intended recipient, you are
> hereby
> > > > > notified
> > > > > > >> that
> > > > > > >> > any printing, copying, dissemination, distribution,
> disclosure
> > > or
> > > > > > >> > forwarding of this communication is strictly prohibited. If
> > you
> > > > have
> > > > > > >> > received this communication in error, please contact the
> > sender
> > > > > > >> immediately
> > > > > > >> > and delete it from your system. Thank You.
> > > > > > >> >
> > > > > > >>
> > > > > > >
> > > > > > > --
> > > > > > > CONFIDENTIALITY NOTICE
> > > > > > > NOTICE: This message is intended for the use of the individual
> or
> > > > > entity
> > > > > > to
> > > > > > > which it is addressed and may contain information that is
> > > > confidential,
> > > > > > > privileged and exempt from disclosure under applicable law. If
> > the
> > > > > reader
> > > > > > > of this message is not the intended recipient, you are hereby
> > > > notified
> > > > > > that
> > > > > > > any printing, copying, dissemination, distribution, disclosure
> or
> > > > > > > forwarding of this communication is strictly prohibited. If you
> > > have
> > > > > > > received this communication in error, please contact the sender
> > > > > > immediately
> > > > > > > and delete it from your system. Thank You.
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Alejandro
> > > > >
> > > >
> > > > --
> > > > CONFIDENTIALITY NOTICE
> > > > NOTICE: This message is intended for the use of the individual or
> > entity
> > > to
> > > > which it is addressed and may contain information that is
> confidential,
> > > > privileged and exempt from disclosure under applicable law. If the
> > reader
> > > > of this message is not the intended recipient, you are hereby
> notified
> > > that
> > > > any printing, copying, dissemination, distribution, disclosure or
> > > > forwarding of this communication is strictly prohibited. If you have
> > > > received this communication in error, please contact the sender
> > > immediately
> > > > and delete it from your system. Thank You.
> > > >
> > >
> > >
> > >
> > > --
> > > Alejandro
> > >
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> >
>
>
>
> --
> Alejandro
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Replacing the JSP web UIs to HTML 5 applications

[Haohui Mai <hm...@hortonworks.com>]

Hi Colin,

I put up a command line tool today to query the information of the HDFS
through JMX. The code is available at
https://github.com/haohui/dfshealth-cli

I did it within a day. Obviously the tool can be more polished, but what
I'm trying to demonstrate is that it is feasible to reuse some of the code
of new web UI to build such a tool with minimal effort.

Right now the tool is based on node.js. However, it is possible to use
java+rhino to run the same code, so that it can be integrated in the main
repository.

Thanks,
Haohui


On Fri, Nov 1, 2013 at 10:56 AM, Colin McCabe <cm...@alumni.cmu.edu>wrote:

> Right now, if you do a search on Google for dfshealth.jsp, you will
> find many unsecured NameNode web UIs which were accidentally exposed
> to the internet.  If the UIs were client-side, accessing these pages
> would not work, since the client-side Javascript would fail to make
> the JMX access.  Nearly all firewalls block JMX.  So in that sense,
> the new UI is more secure, not less.
>
> My understanding is that most cross-site scripting vulnerabilities
> arise when the server uses data from the client in page display
> without properly escaping it.  According to wikipedia, a persistent
> XSS vulnerability arises "when the data provided by the attacker is
> saved by the server, and then permanently displayed on 'normal' pages
> returned to other users in the course of regular browsing, without
> proper HTML escaping."  Nobody has proposed doing this on the
> NameNode.
>
> It seems to me that we are already exposed to any theoretical JMX
> security vulnerabilities already, since we support JMX.  The
> client-side UI just adds a convenient way of calling into the
> pre-existing hooks.
>
> My big concern, like I said earlier, is that we have reasonable
> command-line tools.  I think this is a solvable problem.  Running
> links in a terminal is not exactly the gold standard of command-line
> excellence.  It would be a shame to block progress just to support
> that.
>
> Colin
>
>
> On Wed, Oct 30, 2013 at 12:34 AM, Luke Lu <ll...@vicaya.com> wrote:
> > I don't think that we have reached a consensus that the new javascript
> only
> > UI is the right direction to go. Most people considered it
> "interesting". I
> > personally think it's inappropriate for core Hadoop UI, as it increases
> > attack surface of the UI and taking away existing mitigation options from
> > users unnecessarily. See my latest comments on HDFS-5333 for "concrete"
> > examples.
> >
> > __Luke
> >
> >
> > On Tue, Oct 29, 2013 at 11:28 AM, Haohui Mai <hm...@hortonworks.com>
> wrote:
> >
> >> I would like to summarize the discussions so far. It seems that we have
> >> reached two consensus:
> >>
> >> 1. The new JavaScript-based UI is the right direction to go.
> >> 2. For now we should keep the old JSP pages around for compatibility
> >> reasons.
> >>
> >> There're some debates on the usages of the JMX / JSON APIs, but this is
> >> orthogonal to switching the UI, thus I consider it as a technical
> detail.
> >> We can continue the discussions in the public jira.
> >>
> >> The new UI has already landed in the trunk, based on the consensus it
> seems
> >> that we can switch the default UI to the new one shortly. The user can
> >> still access the old web UI using the same URLs.
> >>
> >> The only question remain is that who is going to maintain the old web
> UI.
> >> My answer is that we should leave them as deprecated and focus the
> effort
> >> on the new web UI.
> >>
> >> Thanks,
> >> Haohui
> >>
> >>
> >>
> >> On Tue, Oct 29, 2013 at 5:22 AM, Zheng, Kai <ka...@intel.com>
> wrote:
> >>
> >> > > having /JMX for monitoring integration and a /JSON end point for
> the UI
> >> > IMHO, this makes sense, especially for the long term. JMX interface
> >> serves
> >> > as management console in admin perspective, WebUI serves as end user
> >> > interface. Both might share same functionality codes, but that does
> not
> >> > validate we couple them together.
> >> >
> >> > Thanks & regards,
> >> > Kai
> >> >
> >> > -----Original Message-----
> >> > From: Alejandro Abdelnur [mailto:tucu@cloudera.com]
> >> > Sent: Tuesday, October 29, 2013 8:14 AM
> >> > To: hdfs-dev@hadoop.apache.org
> >> > Subject: Re: Replacing the JSP web UIs to HTML 5 applications
> >> >
> >> > Isn't using JMX to expose JSON for the web UI misusing JMX?
> >> >
> >> > I would think a more appropriate approach would be having /JMX for
> >> > monitoring integration and a /JSON end point for the UI data.
> >> >
> >> > Thanks.
> >> >
> >> >
> >> > On Mon, Oct 28, 2013 at 4:58 PM, Haohui Mai <hm...@hortonworks.com>
> >> wrote:
> >> >
> >> > > Alejandro,
> >> > >
> >> > > If I understand correctly, that is the exact approach that the new
> web
> >> > > UI is taking. The new web UI takes the output from JMX and renders
> >> > > them as HTML at the client side.
> >> > >
> >> > > ~Haohui
> >> > >
> >> > >
> >> > > On Mon, Oct 28, 2013 at 4:18 PM, Alejandro Abdelnur <
> tucu@cloudera.com
> >> > > >wrote:
> >> > >
> >> > > > Haohui,
> >> > > >
> >> > > > If you have NN and DNs producing JSON instead HTML, then you can
> >> > > > build JS based web UIs. Take for example Oozie, Oozie produces
> JSON,
> >> > > > it has a
> >> > > built
> >> > > > in JS web ui that consumes JSON and Hue has built an external web
> UI
> >> > > > that also consumes JSON. In the case of Hue UI, Oozie didn't have
> to
> >> > > > change anything to get that UI and improvements on the Hue UI
> don't
> >> > > > require changes in Oozie unless it is to produce additional
> >> > information.
> >> > > >
> >> > > > hope this clarifies.
> >> > > >
> >> > > > Thx
> >> > > >
> >> > > >
> >> > > > On Mon, Oct 28, 2013 at 4:06 PM, Haohui Mai <hmai@hortonworks.com
> >
> >> > > wrote:
> >> > > >
> >> > > > > Echo my comments on HDFS-5402:
> >> > > > >
> >> > > > > bq. If we're going to remove the old web UI, I think the new web
> >> > > > > UI has to have the same level of unit testing. We shouldn't go
> >> > > > > backwards in terms of unit testing.
> >> > > > >
> >> > > > > I take a look at TestNamenodeJspHelper / TestDatanodeJspHelper /
> >> > > > > TestClusterJspHelper. It seems to me that we can merge these
> tests
> >> > > > > with
> >> > > > the
> >> > > > > unit tests on JMX.
> >> > > > >
> >> > > > > bq. If we are going to
> >> > > > > remove this capability, we need to add some other command-line
> >> > > > > tools to get the same functionality. These tools could use REST
> if
> >> > > > > we have that, or JMX, but they need to exist before we can
> >> > > > > consider removing the old UI.
> >> > > > >
> >> > > > > This is a good point. Since all information are available
> through
> >> > > > > JMX,
> >> > > > the
> >> > > > > easiest way to approach it is to write some scripts using
> Node.js.
> >> > > > > The architecture of the new Web UIs is ready for this.
> >> > > > >
> >> > > > >
> >> > > > > On Mon, Oct 28, 2013 at 3:57 PM, Alejandro Abdelnur
> >> > > > > <tucu@cloudera.com
> >> > > > > >wrote:
> >> > > > >
> >> > > > > > Producing JSON would be great. Agree with Colin that we should
> >> > > > > > leave
> >> > > > for
> >> > > > > > now the current JSP based web ui.
> >> > > > > >
> >> > > > > > thx
> >> > > > > >
> >> > > > > >
> >> > > > > > On Mon, Oct 28, 2013 at 11:16 AM, Colin McCabe <
> >> > > cmccabe@alumni.cmu.edu
> >> > > > > > >wrote:
> >> > > > > >
> >> > > > > > > This is a really interesting project, Haohui.  I think it
> will
> >> > > > > > > make our web UI much nicer.
> >> > > > > > >
> >> > > > > > > I have a few concerns about removing the old web UI,
> however:
> >> > > > > > >
> >> > > > > > > * If we're going to remove the old web UI, I think the new
> web
> >> > > > > > > UI
> >> > > has
> >> > > > > > > to have the same level of unit testing.  We shouldn't go
> >> > > > > > > backwards
> >> > > in
> >> > > > > > > terms of unit testing.
> >> > > > > > >
> >> > > > > > > * Most of the deployments of elinks and links out there
> don't
> >> > > support
> >> > > > > > > Javascript.  This is just a reality of life when using
> CentOS
> >> > > > > > > 5 or
> >> > > 6,
> >> > > > > > > which many users are still using.  I have used "links" to
> >> > > > > > > diagnose problems through the web UI in the past, in systems
> >> > > > > > > where access to the cluster was available only through
> telnet.
> >> > > > > > > If we are going to remove this capability, we need to add
> some
> >> > > > > > > other command-line
> >> > > tools
> >> > > > > > > to get the same functionality.  These tools could use REST
> if
> >> > > > > > > we
> >> > > have
> >> > > > > > > that, or JMX, but they need to exist before we can consider
> >> > > removing
> >> > > > > > > the old UI.
> >> > > > > > >
> >> > > > > > > best,
> >> > > > > > > Colin
> >> > > > > > >
> >> > > > > > > On Fri, Oct 25, 2013 at 7:31 PM, Haohui Mai
> >> > > > > > > <hm...@hortonworks.com>
> >> > > > > > wrote:
> >> > > > > > > > Thanks for the reply, Luke. Here I just echo my response
> >> > > > > > > > from the
> >> > > > > jira:
> >> > > > > > > >
> >> > > > > > > > bq. this client-side js only approach, which is less
> secure
> >> > > > > > > > than
> >> > > a
> >> > > > > > > > progressively enhanced hybrid approach used by YARN. The
> >> > > > > > > > recent
> >> > > > gmail
> >> > > > > > > > XSS fiasco highlights the issue.
> >> > > > > > > >
> >> > > > > > > > I'm presenting an informal security analysis to compare
> the
> >> > > > security
> >> > > > > of
> >> > > > > > > the
> >> > > > > > > > old and the new web UIs.
> >> > > > > > > >
> >> > > > > > > > An attacker launches an XSS attack by injecting malicious
> >> > > > > > > > code
> >> > > > which
> >> > > > > > are
> >> > > > > > > > usually HTML or JavaScript fragments into the web page, so
> >> > > > > > > > that
> >> > > the
> >> > > > > > > > malicious code can have the same privileges of the web
> page.
> >> > > > > > > >
> >> > > > > > > > First, in the scope of XSS attacks, note that the threat
> >> > > > > > > > models
> >> > > of
> >> > > > > > > > launching XSS attacks on Internet sites Gmail/Linkedin and
> >> > > > > > > > the
> >> > > one
> >> > > > of
> >> > > > > > the
> >> > > > > > > > Hadoop UIs are different. They have fundamental different
> >> > > > > > > > sets of
> >> > > > > > > external
> >> > > > > > > > inputs that the attackers have control to. Internet sites
> >> > > > > > > > have
> >> > > > little
> >> > > > > > > > control of these inputs. In the case of Gmail / Linkedin,
> an
> >> > > attack
> >> > > > > can
> >> > > > > > > > send you a crafted e-mail, or put malicious description in
> >> > > > > > > > his / her Linkedin profile. The sets of external inputs
> are
> >> > > *restricted*
> >> > > > in
> >> > > > > > > > Hadoop UIs. The new web UIs take JMX and WebHDFS as
> inputs.
> >> > > > > > > > The attacker has to launch a XSS attack by:
> >> > > > > > > >
> >> > > > > > > > * Compromise the jars so that the output of JMX / WebHDFS
> >> > > > > > > > have
> >> > > the
> >> > > > > > > > malicious code.
> >> > > > > > > > * Replace the web UIs completely to include the malicious
> >> code.
> >> > > > > > > >
> >> > > > > > > > In either case *the attacker has to compromise the hadoop
> >> > > > > > > > core or
> >> > > > the
> >> > > > > > > > namenode*. That means the new web UIs are at least as
> secure
> >> > > > > > > > as
> >> > > the
> >> > > > > > > hadoop
> >> > > > > > > > core, and the namenode machine.
> >> > > > > > > >
> >> > > > > > > > Second, I argue that using client-side templates are more
> >> > > > > > > > secure
> >> > > > than
> >> > > > > > the
> >> > > > > > > > current JSP-based server-side templates. To defend against
> >> > > > > > > > XSS attacks, both techniques have to filter the external
> >> > > > > > > > inputs at
> >> > > > > *every*
> >> > > > > > > > possible execution paths. Several facts much be taken into
> >> > > > > > > > plays when evaluating the security of both approaches in
> >> > > real-world
> >> > > > > > > > environments:
> >> > > > > > > >
> >> > > > > > > > * The JavaScript libraries used in the new web UIs have
> >> > > > > > > > survived
> >> > > in
> >> > > > > > > > extremely large-scale production tests. jQuery is used by
> >> > > > > > > > Google
> >> > > > and
> >> > > > > > > >  Microsoft, bootstrap is used by Twitter, and dust.js is
> >> > > > > > > > used by
> >> > > > > > > Linkedin.
> >> > > > > > > > All libraries survived from hundreds of thousands of
>  attack
> >> > > > > > > > attempts on a daily basis. I agree that the libraries
> >> > > might
> >> > > > > > still
> >> > > > > > > > be imperfect, but there's no way that we can test the JSP
> >> > > > > > > > web  UIs to achieve the same level of assurances given the
> >> > > > > > > > amount of
> >> > > > > > > resources
> >> > > > > > > > the community has.
> >> > > > > > > > * Client-side templates consolidate all filtering logic in
> >> > > > > > > > one
> >> > > > > central
> >> > > > > > > > place. Recall that the goal is to filter all external
> inputs
> >> > > > > > > > at
> >> > > > every
> >> > > > > > > >  execution paths, this is a much more systematic approach
> >> > > compared
> >> > > > to
> >> > > > > > the
> >> > > > > > > > server-side templates we have today. It is difficult (if
> not
> >> > > > > > > >  impossible) to do it in a JSP/ASP/PHP application, since
> >> > > > > > > > such
> >> > > > > > filtering
> >> > > > > > > > can be only achieved via ad-hoc approaches ([1] shows some
> >> > > > > > > > empirical data). Also, HDFS-4901 recently describes a XSS
> >> > > > > > vulnerability
> >> > > > > > > in
> >> > > > > > > > browseDirectory.jsp.
> >> > > > > > > >
> >> > > > > > > > bq. You'd require proper SSL (not self signed) setup to
> >> > > > > > > > avoid JS injection
> >> > > > > > > >
> >> > > > > > > > Commodity browsers enforce Same-Origin Policy to defend
> >> > > > > > > > against
> >> > > > code
> >> > > > > > > > injections. It has nothing to do with what kinds of SSL
> >> > > > certificates
> >> > > > > > > > you hold.
> >> > > > > > > >
> >> > > > > > > > bq.  I also have concerns that we commit these changes
> >> > > > > > > > without
> >> > > > > matching
> >> > > > > > > > unit tests
> >> > > > > > > >
> >> > > > > > > > The JavaScript code can be automatically tested. The same
> >> > > > > > > > code
> >> > > can
> >> > > > be
> >> > > > > > run
> >> > > > > > > > by node.js and the test can compared with pre-defined
> >> > > > > > > > results. It is also possible to write an adapter to use
> >> > > > > > > > Rhino to
> >> > > > > > > accomplish
> >> > > > > > > > the same task. We can discuss how to integrate them into
> the
> >> > > > > > > > maven test routines in a different thread.
> >> > > > > > > >
> >> > > > > > > > bq. Client side rendering completely breaks the workflows
> >> > > > > > > > for ops
> >> > > > who
> >> > > > > > > rely
> >> > > > > > > > on text based terminal/emacs/vim browsers (no js support)
> to
> >> > > > > > > > monitor component UI.
> >> > > > > > > >
> >> > > > > > > > links / elinks (http://elinks.or.cz/) are text-based web
> >> > > browsers
> >> > > > > that
> >> > > > > > > > support JavaScript.
> >> > > > > > > >
> >> > > > > > > > bq. The priority/requirements for UI in core Hadoop should
> >> > > > > > > > be
> >> > > > > security
> >> > > > > > > and
> >> > > > > > > > correctness, which client side templating cannot address
> >> > > > > > > > properly so far.
> >> > > > > > > >
> >> > > > > > > > I agree that we should focus on security and correctness.
> >> > > > > > > > The
> >> > > > > > paragraphs
> >> > > > > > > > above explain that how the architecture of the new UIs
> makes
> >> > > > > > > > the UIs more secure in real-world settings compared to the
> >> > > UI
> >> > > > > we
> >> > > > > > > have
> >> > > > > > > > today.
> >> > > > > > > >
> >> > > > > > > > References:
> >> > > > > > > >
> >> > > > > > > > 1. A. Yip et al. Improving Application Security with Data
> >> > > > > > > > Flow
> >> > > > > > > Assertions.
> >> > > > > > > > In SOSP'2009.
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > On Fri, Oct 25, 2013 at 10:02 AM, Luke Lu <llu@apache.org
> >
> >> > > wrote:
> >> > > > > > > >
> >> > > > > > > >> Echoing my comments on HDFS-3555:
> >> > > > > > > >>
> >> > > > > > > >> I have concerns with this client-side js only approach,
> >> > > > > > > >> which is
> >> > > > > less
> >> > > > > > > >> secure than a progressively enhanced hybrid approach used
> >> > > > > > > >> by
> >> > > YARN.
> >> > > > > The
> >> > > > > > > >> recent gmail XSS fiasco highlights the issue. I also have
> >> > > concerns
> >> > > > > > that
> >> > > > > > > we
> >> > > > > > > >> commit these changes without matching unit tests - the
> fact
> >> > > > > > > >> you
> >> > > > > cannot
> >> > > > > > > >> effectively unit test these changes should tell you
> >> > > > > > > >> something
> >> > > > about
> >> > > > > > this
> >> > > > > > > >> approach.
> >> > > > > > > >>
> >> > > > > > > >> *Requiring* JS means that an admin cannot turn off js to
> >> > > > (partially)
> >> > > > > > use
> >> > > > > > > >> core Hadoop UI. You'd *require* proper SSL (not self
> >> > > > > > > >> signed)
> >> > > setup
> >> > > > > to
> >> > > > > > > avoid
> >> > > > > > > >> JS injection, even if security of js libraries used is
> >> > > > > > > >> perfect,
> >> > > > > which
> >> > > > > > I
> >> > > > > > > >> doubt (search gmail/linkedin XSS). Client side rendering
> >> > > > completely
> >> > > > > > > breaks
> >> > > > > > > >> the workflows for ops who rely on text based
> >> > > > > > > >> terminal/emacs/vim
> >> > > > > > browsers
> >> > > > > > > >> (no js support) to monitor component UI.
> >> > > > > > > >>
> >> > > > > > > >> IMO, JS-only rendering belongs to social networking sites
> >> > > > > > > >> and/or
> >> > > > > SaaS
> >> > > > > > > >> front-ends, where full time UI/security specialists
> >> > > > > > > >> babysits UI
> >> > > > > > > changes. I
> >> > > > > > > >> think eventually most users will use a self servicing UI
> in
> >> > > > > > > >> a
> >> > > SaaS
> >> > > > > > > >> front-end that uses REST/JMX API to get data from
> back-end
> >> > > > > components,
> >> > > > > > > >> besides their own app master/service UI. The
> >> > > priority/requirements
> >> > > > > for
> >> > > > > > > UI
> >> > > > > > > >> in core Hadoop should be security and correctness, which
> >> > > > > > > >> client
> >> > > > side
> >> > > > > > > >> templating cannot address properly so far.
> >> > > > > > > >>
> >> > > > > > > >>
> >> > > > > > > >> On Tue, Oct 22, 2013 at 3:59 PM, Haohui Mai <
> >> > > hmai@hortonworks.com
> >> > > > >
> >> > > > > > > wrote:
> >> > > > > > > >>
> >> > > > > > > >> > Hi all,
> >> > > > > > > >> >
> >> > > > > > > >> >
> >> > > > > > > >> > Jing Zhao and I recently have reimplemented the
> JSP-based
> >> > > > > > > >> > web
> >> > > > UIs
> >> > > > > in
> >> > > > > > > >> HTML 5
> >> > > > > > > >> > applications (HDFS-5333). Based on our prelimanary
> >> > > > > > > >> > testing
> >> > > > results
> >> > > > > > we
> >> > > > > > > >> > believe thst the new web UIs of the namenodes and the
> >> > > > > > > >> > datanode
> >> > > > are
> >> > > > > > > ready
> >> > > > > > > >> > for everyday uses.
> >> > > > > > > >> >
> >> > > > > > > >> > You're more than welcome to try it out on trunk by
> >> > > > > > > >> > visiting
> >> > > > > http://
> >> > > > > > > >> > <namenode>/dfshealth.html
> >> > > > > > > >> >
> >> > > > > > > >> > There are a number of benefits from this transition.
> From
> >> > > > > > > >> > a
> >> > > > > > > developer's
> >> > > > > > > >> > prospective, the most notable one is *maintainability*:
> >> > > > > > > >> >
> >> > > > > > > >> > (1) The abstractions between the UI and the core server
> >> > > > > > > >> > are
> >> > > > > > > well-defined,
> >> > > > > > > >> > decoupling the UI and the core hadoop servers.
> >> > > > > > > >> >
> >> > > > > > > >> > (2) It allows us to deprecate the logic in the JSP
> pages.
> >> > > > > > > >> > The
> >> > > > old
> >> > > > > > web
> >> > > > > > > UIs
> >> > > > > > > >> > have to duplicate the logic in the JSPs. The logic is
> >> > > > > > > >> > often
> >> > > > > > > out-of-dated
> >> > > > > > > >> > and not well-tested, which leads to broken pages and
> >> > > > > > > >> > security vulnerabilities(e.g. HDFS-5251, HDFS-5307,
> >> > > > > > > >> > HDFS-5308,
> >> > > HDFS-5317
> >> > > > > and
> >> > > > > > > >> > HDFS-4901). The architecture of the new UIs prevent
> these
> >> > > > > > > >> > bugs
> >> > > > at
> >> > > > > > the
> >> > > > > > > >> very
> >> > > > > > > >> > beginning.
> >> > > > > > > >> >
> >> > > > > > > >> >
> >> > > > > > > >> > I propose that deprecate the old, JSP-based web UIs in
> >> > > > > > > >> > 2.3. I
> >> > > > > opened
> >> > > > > > > >> > HDFS-5402 to track the relevant discussions.
> >> > > > > > > >> >
> >> > > > > > > >> > Your feedbacks are highly appreciated.
> >> > > > > > > >> >
> >> > > > > > > >> >
> >> > > > > > > >> > Sincerely,
> >> > > > > > > >> >
> >> > > > > > > >> > Haohui
> >> > > > > > > >> >
> >> > > > > > > >> > --
> >> > > > > > > >> > CONFIDENTIALITY NOTICE
> >> > > > > > > >> > NOTICE: This message is intended for the use of the
> >> > > > > > > >> > individual
> >> > > > or
> >> > > > > > > entity
> >> > > > > > > >> to
> >> > > > > > > >> > which it is addressed and may contain information that
> is
> >> > > > > > > confidential,
> >> > > > > > > >> > privileged and exempt from disclosure under applicable
> >> > > > > > > >> > law. If
> >> > > > the
> >> > > > > > > reader
> >> > > > > > > >> > of this message is not the intended recipient, you are
> >> > > > > > > >> > hereby
> >> > > > > > notified
> >> > > > > > > >> that
> >> > > > > > > >> > any printing, copying, dissemination, distribution,
> >> > > > > > > >> > disclosure
> >> > > > or
> >> > > > > > > >> > forwarding of this communication is strictly
> prohibited.
> >> > > > > > > >> > If
> >> > > you
> >> > > > > have
> >> > > > > > > >> > received this communication in error, please contact
> the
> >> > > sender
> >> > > > > > > >> immediately
> >> > > > > > > >> > and delete it from your system. Thank You.
> >> > > > > > > >> >
> >> > > > > > > >>
> >> > > > > > > >
> >> > > > > > > > --
> >> > > > > > > > CONFIDENTIALITY NOTICE
> >> > > > > > > > NOTICE: This message is intended for the use of the
> >> > > > > > > > individual or
> >> > > > > > entity
> >> > > > > > > to
> >> > > > > > > > which it is addressed and may contain information that is
> >> > > > > confidential,
> >> > > > > > > > privileged and exempt from disclosure under applicable
> law.
> >> > > > > > > > If
> >> > > the
> >> > > > > > reader
> >> > > > > > > > of this message is not the intended recipient, you are
> >> > > > > > > > hereby
> >> > > > > notified
> >> > > > > > > that
> >> > > > > > > > any printing, copying, dissemination, distribution,
> >> > > > > > > > disclosure or forwarding of this communication is strictly
> >> > > > > > > > prohibited. If you
> >> > > > have
> >> > > > > > > > received this communication in error, please contact the
> >> > > > > > > > sender
> >> > > > > > > immediately
> >> > > > > > > > and delete it from your system. Thank You.
> >> > > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > > --
> >> > > > > > Alejandro
> >> > > > > >
> >> > > > >
> >> > > > > --
> >> > > > > CONFIDENTIALITY NOTICE
> >> > > > > NOTICE: This message is intended for the use of the individual
> or
> >> > > entity
> >> > > > to
> >> > > > > which it is addressed and may contain information that is
> >> > > > > confidential, privileged and exempt from disclosure under
> >> > > > > applicable law. If the
> >> > > reader
> >> > > > > of this message is not the intended recipient, you are hereby
> >> > > > > notified
> >> > > > that
> >> > > > > any printing, copying, dissemination, distribution, disclosure
> or
> >> > > > > forwarding of this communication is strictly prohibited. If you
> >> > > > > have received this communication in error, please contact the
> >> > > > > sender
> >> > > > immediately
> >> > > > > and delete it from your system. Thank You.
> >> > > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > > --
> >> > > > Alejandro
> >> > > >
> >> > >
> >> > > --
> >> > > CONFIDENTIALITY NOTICE
> >> > > NOTICE: This message is intended for the use of the individual or
> >> > > entity to which it is addressed and may contain information that is
> >> > > confidential, privileged and exempt from disclosure under applicable
> >> > > law. If the reader of this message is not the intended recipient,
> you
> >> > > are hereby notified that any printing, copying, dissemination,
> >> > > distribution, disclosure or forwarding of this communication is
> >> > > strictly prohibited. If you have received this communication in
> error,
> >> > > please contact the sender immediately and delete it from your
> system.
> >> > Thank You.
> >> > >
> >> >
> >> >
> >> >
> >> > --
> >> > Alejandro
> >> >
> >>
> >> --
> >> CONFIDENTIALITY NOTICE
> >> NOTICE: This message is intended for the use of the individual or
> entity to
> >> which it is addressed and may contain information that is confidential,
> >> privileged and exempt from disclosure under applicable law. If the
> reader
> >> of this message is not the intended recipient, you are hereby notified
> that
> >> any printing, copying, dissemination, distribution, disclosure or
> >> forwarding of this communication is strictly prohibited. If you have
> >> received this communication in error, please contact the sender
> immediately
> >> and delete it from your system. Thank You.
> >>
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Replacing the JSP web UIs to HTML 5 applications

[Colin McCabe <cm...@alumni.cmu.edu>]

Right now, if you do a search on Google for dfshealth.jsp, you will
find many unsecured NameNode web UIs which were accidentally exposed
to the internet.  If the UIs were client-side, accessing these pages
would not work, since the client-side Javascript would fail to make
the JMX access.  Nearly all firewalls block JMX.  So in that sense,
the new UI is more secure, not less.

My understanding is that most cross-site scripting vulnerabilities
arise when the server uses data from the client in page display
without properly escaping it.  According to wikipedia, a persistent
XSS vulnerability arises "when the data provided by the attacker is
saved by the server, and then permanently displayed on 'normal' pages
returned to other users in the course of regular browsing, without
proper HTML escaping."  Nobody has proposed doing this on the
NameNode.

It seems to me that we are already exposed to any theoretical JMX
security vulnerabilities already, since we support JMX.  The
client-side UI just adds a convenient way of calling into the
pre-existing hooks.

My big concern, like I said earlier, is that we have reasonable
command-line tools.  I think this is a solvable problem.  Running
links in a terminal is not exactly the gold standard of command-line
excellence.  It would be a shame to block progress just to support
that.

Colin


On Wed, Oct 30, 2013 at 12:34 AM, Luke Lu <ll...@vicaya.com> wrote:
> I don't think that we have reached a consensus that the new javascript only
> UI is the right direction to go. Most people considered it "interesting". I
> personally think it's inappropriate for core Hadoop UI, as it increases
> attack surface of the UI and taking away existing mitigation options from
> users unnecessarily. See my latest comments on HDFS-5333 for "concrete"
> examples.
>
> __Luke
>
>
> On Tue, Oct 29, 2013 at 11:28 AM, Haohui Mai <hm...@hortonworks.com> wrote:
>
>> I would like to summarize the discussions so far. It seems that we have
>> reached two consensus:
>>
>> 1. The new JavaScript-based UI is the right direction to go.
>> 2. For now we should keep the old JSP pages around for compatibility
>> reasons.
>>
>> There're some debates on the usages of the JMX / JSON APIs, but this is
>> orthogonal to switching the UI, thus I consider it as a technical detail.
>> We can continue the discussions in the public jira.
>>
>> The new UI has already landed in the trunk, based on the consensus it seems
>> that we can switch the default UI to the new one shortly. The user can
>> still access the old web UI using the same URLs.
>>
>> The only question remain is that who is going to maintain the old web UI.
>> My answer is that we should leave them as deprecated and focus the effort
>> on the new web UI.
>>
>> Thanks,
>> Haohui
>>
>>
>>
>> On Tue, Oct 29, 2013 at 5:22 AM, Zheng, Kai <ka...@intel.com> wrote:
>>
>> > > having /JMX for monitoring integration and a /JSON end point for the UI
>> > IMHO, this makes sense, especially for the long term. JMX interface
>> serves
>> > as management console in admin perspective, WebUI serves as end user
>> > interface. Both might share same functionality codes, but that does not
>> > validate we couple them together.
>> >
>> > Thanks & regards,
>> > Kai
>> >
>> > -----Original Message-----
>> > From: Alejandro Abdelnur [mailto:tucu@cloudera.com]
>> > Sent: Tuesday, October 29, 2013 8:14 AM
>> > To: hdfs-dev@hadoop.apache.org
>> > Subject: Re: Replacing the JSP web UIs to HTML 5 applications
>> >
>> > Isn't using JMX to expose JSON for the web UI misusing JMX?
>> >
>> > I would think a more appropriate approach would be having /JMX for
>> > monitoring integration and a /JSON end point for the UI data.
>> >
>> > Thanks.
>> >
>> >
>> > On Mon, Oct 28, 2013 at 4:58 PM, Haohui Mai <hm...@hortonworks.com>
>> wrote:
>> >
>> > > Alejandro,
>> > >
>> > > If I understand correctly, that is the exact approach that the new web
>> > > UI is taking. The new web UI takes the output from JMX and renders
>> > > them as HTML at the client side.
>> > >
>> > > ~Haohui
>> > >
>> > >
>> > > On Mon, Oct 28, 2013 at 4:18 PM, Alejandro Abdelnur <tucu@cloudera.com
>> > > >wrote:
>> > >
>> > > > Haohui,
>> > > >
>> > > > If you have NN and DNs producing JSON instead HTML, then you can
>> > > > build JS based web UIs. Take for example Oozie, Oozie produces JSON,
>> > > > it has a
>> > > built
>> > > > in JS web ui that consumes JSON and Hue has built an external web UI
>> > > > that also consumes JSON. In the case of Hue UI, Oozie didn't have to
>> > > > change anything to get that UI and improvements on the Hue UI don't
>> > > > require changes in Oozie unless it is to produce additional
>> > information.
>> > > >
>> > > > hope this clarifies.
>> > > >
>> > > > Thx
>> > > >
>> > > >
>> > > > On Mon, Oct 28, 2013 at 4:06 PM, Haohui Mai <hm...@hortonworks.com>
>> > > wrote:
>> > > >
>> > > > > Echo my comments on HDFS-5402:
>> > > > >
>> > > > > bq. If we're going to remove the old web UI, I think the new web
>> > > > > UI has to have the same level of unit testing. We shouldn't go
>> > > > > backwards in terms of unit testing.
>> > > > >
>> > > > > I take a look at TestNamenodeJspHelper / TestDatanodeJspHelper /
>> > > > > TestClusterJspHelper. It seems to me that we can merge these tests
>> > > > > with
>> > > > the
>> > > > > unit tests on JMX.
>> > > > >
>> > > > > bq. If we are going to
>> > > > > remove this capability, we need to add some other command-line
>> > > > > tools to get the same functionality. These tools could use REST if
>> > > > > we have that, or JMX, but they need to exist before we can
>> > > > > consider removing the old UI.
>> > > > >
>> > > > > This is a good point. Since all information are available through
>> > > > > JMX,
>> > > > the
>> > > > > easiest way to approach it is to write some scripts using Node.js.
>> > > > > The architecture of the new Web UIs is ready for this.
>> > > > >
>> > > > >
>> > > > > On Mon, Oct 28, 2013 at 3:57 PM, Alejandro Abdelnur
>> > > > > <tucu@cloudera.com
>> > > > > >wrote:
>> > > > >
>> > > > > > Producing JSON would be great. Agree with Colin that we should
>> > > > > > leave
>> > > > for
>> > > > > > now the current JSP based web ui.
>> > > > > >
>> > > > > > thx
>> > > > > >
>> > > > > >
>> > > > > > On Mon, Oct 28, 2013 at 11:16 AM, Colin McCabe <
>> > > cmccabe@alumni.cmu.edu
>> > > > > > >wrote:
>> > > > > >
>> > > > > > > This is a really interesting project, Haohui.  I think it will
>> > > > > > > make our web UI much nicer.
>> > > > > > >
>> > > > > > > I have a few concerns about removing the old web UI, however:
>> > > > > > >
>> > > > > > > * If we're going to remove the old web UI, I think the new web
>> > > > > > > UI
>> > > has
>> > > > > > > to have the same level of unit testing.  We shouldn't go
>> > > > > > > backwards
>> > > in
>> > > > > > > terms of unit testing.
>> > > > > > >
>> > > > > > > * Most of the deployments of elinks and links out there don't
>> > > support
>> > > > > > > Javascript.  This is just a reality of life when using CentOS
>> > > > > > > 5 or
>> > > 6,
>> > > > > > > which many users are still using.  I have used "links" to
>> > > > > > > diagnose problems through the web UI in the past, in systems
>> > > > > > > where access to the cluster was available only through telnet.
>> > > > > > > If we are going to remove this capability, we need to add some
>> > > > > > > other command-line
>> > > tools
>> > > > > > > to get the same functionality.  These tools could use REST if
>> > > > > > > we
>> > > have
>> > > > > > > that, or JMX, but they need to exist before we can consider
>> > > removing
>> > > > > > > the old UI.
>> > > > > > >
>> > > > > > > best,
>> > > > > > > Colin
>> > > > > > >
>> > > > > > > On Fri, Oct 25, 2013 at 7:31 PM, Haohui Mai
>> > > > > > > <hm...@hortonworks.com>
>> > > > > > wrote:
>> > > > > > > > Thanks for the reply, Luke. Here I just echo my response
>> > > > > > > > from the
>> > > > > jira:
>> > > > > > > >
>> > > > > > > > bq. this client-side js only approach, which is less secure
>> > > > > > > > than
>> > > a
>> > > > > > > > progressively enhanced hybrid approach used by YARN. The
>> > > > > > > > recent
>> > > > gmail
>> > > > > > > > XSS fiasco highlights the issue.
>> > > > > > > >
>> > > > > > > > I'm presenting an informal security analysis to compare the
>> > > > security
>> > > > > of
>> > > > > > > the
>> > > > > > > > old and the new web UIs.
>> > > > > > > >
>> > > > > > > > An attacker launches an XSS attack by injecting malicious
>> > > > > > > > code
>> > > > which
>> > > > > > are
>> > > > > > > > usually HTML or JavaScript fragments into the web page, so
>> > > > > > > > that
>> > > the
>> > > > > > > > malicious code can have the same privileges of the web page.
>> > > > > > > >
>> > > > > > > > First, in the scope of XSS attacks, note that the threat
>> > > > > > > > models
>> > > of
>> > > > > > > > launching XSS attacks on Internet sites Gmail/Linkedin and
>> > > > > > > > the
>> > > one
>> > > > of
>> > > > > > the
>> > > > > > > > Hadoop UIs are different. They have fundamental different
>> > > > > > > > sets of
>> > > > > > > external
>> > > > > > > > inputs that the attackers have control to. Internet sites
>> > > > > > > > have
>> > > > little
>> > > > > > > > control of these inputs. In the case of Gmail / Linkedin, an
>> > > attack
>> > > > > can
>> > > > > > > > send you a crafted e-mail, or put malicious description in
>> > > > > > > > his / her Linkedin profile. The sets of external inputs are
>> > > *restricted*
>> > > > in
>> > > > > > > > Hadoop UIs. The new web UIs take JMX and WebHDFS as inputs.
>> > > > > > > > The attacker has to launch a XSS attack by:
>> > > > > > > >
>> > > > > > > > * Compromise the jars so that the output of JMX / WebHDFS
>> > > > > > > > have
>> > > the
>> > > > > > > > malicious code.
>> > > > > > > > * Replace the web UIs completely to include the malicious
>> code.
>> > > > > > > >
>> > > > > > > > In either case *the attacker has to compromise the hadoop
>> > > > > > > > core or
>> > > > the
>> > > > > > > > namenode*. That means the new web UIs are at least as secure
>> > > > > > > > as
>> > > the
>> > > > > > > hadoop
>> > > > > > > > core, and the namenode machine.
>> > > > > > > >
>> > > > > > > > Second, I argue that using client-side templates are more
>> > > > > > > > secure
>> > > > than
>> > > > > > the
>> > > > > > > > current JSP-based server-side templates. To defend against
>> > > > > > > > XSS attacks, both techniques have to filter the external
>> > > > > > > > inputs at
>> > > > > *every*
>> > > > > > > > possible execution paths. Several facts much be taken into
>> > > > > > > > plays when evaluating the security of both approaches in
>> > > real-world
>> > > > > > > > environments:
>> > > > > > > >
>> > > > > > > > * The JavaScript libraries used in the new web UIs have
>> > > > > > > > survived
>> > > in
>> > > > > > > > extremely large-scale production tests. jQuery is used by
>> > > > > > > > Google
>> > > > and
>> > > > > > > >  Microsoft, bootstrap is used by Twitter, and dust.js is
>> > > > > > > > used by
>> > > > > > > Linkedin.
>> > > > > > > > All libraries survived from hundreds of thousands of  attack
>> > > > > > > > attempts on a daily basis. I agree that the libraries
>> > > might
>> > > > > > still
>> > > > > > > > be imperfect, but there's no way that we can test the JSP
>> > > > > > > > web  UIs to achieve the same level of assurances given the
>> > > > > > > > amount of
>> > > > > > > resources
>> > > > > > > > the community has.
>> > > > > > > > * Client-side templates consolidate all filtering logic in
>> > > > > > > > one
>> > > > > central
>> > > > > > > > place. Recall that the goal is to filter all external inputs
>> > > > > > > > at
>> > > > every
>> > > > > > > >  execution paths, this is a much more systematic approach
>> > > compared
>> > > > to
>> > > > > > the
>> > > > > > > > server-side templates we have today. It is difficult (if not
>> > > > > > > >  impossible) to do it in a JSP/ASP/PHP application, since
>> > > > > > > > such
>> > > > > > filtering
>> > > > > > > > can be only achieved via ad-hoc approaches ([1] shows some
>> > > > > > > > empirical data). Also, HDFS-4901 recently describes a XSS
>> > > > > > vulnerability
>> > > > > > > in
>> > > > > > > > browseDirectory.jsp.
>> > > > > > > >
>> > > > > > > > bq. You'd require proper SSL (not self signed) setup to
>> > > > > > > > avoid JS injection
>> > > > > > > >
>> > > > > > > > Commodity browsers enforce Same-Origin Policy to defend
>> > > > > > > > against
>> > > > code
>> > > > > > > > injections. It has nothing to do with what kinds of SSL
>> > > > certificates
>> > > > > > > > you hold.
>> > > > > > > >
>> > > > > > > > bq.  I also have concerns that we commit these changes
>> > > > > > > > without
>> > > > > matching
>> > > > > > > > unit tests
>> > > > > > > >
>> > > > > > > > The JavaScript code can be automatically tested. The same
>> > > > > > > > code
>> > > can
>> > > > be
>> > > > > > run
>> > > > > > > > by node.js and the test can compared with pre-defined
>> > > > > > > > results. It is also possible to write an adapter to use
>> > > > > > > > Rhino to
>> > > > > > > accomplish
>> > > > > > > > the same task. We can discuss how to integrate them into the
>> > > > > > > > maven test routines in a different thread.
>> > > > > > > >
>> > > > > > > > bq. Client side rendering completely breaks the workflows
>> > > > > > > > for ops
>> > > > who
>> > > > > > > rely
>> > > > > > > > on text based terminal/emacs/vim browsers (no js support) to
>> > > > > > > > monitor component UI.
>> > > > > > > >
>> > > > > > > > links / elinks (http://elinks.or.cz/) are text-based web
>> > > browsers
>> > > > > that
>> > > > > > > > support JavaScript.
>> > > > > > > >
>> > > > > > > > bq. The priority/requirements for UI in core Hadoop should
>> > > > > > > > be
>> > > > > security
>> > > > > > > and
>> > > > > > > > correctness, which client side templating cannot address
>> > > > > > > > properly so far.
>> > > > > > > >
>> > > > > > > > I agree that we should focus on security and correctness.
>> > > > > > > > The
>> > > > > > paragraphs
>> > > > > > > > above explain that how the architecture of the new UIs makes
>> > > > > > > > the UIs more secure in real-world settings compared to the
>> > > UI
>> > > > > we
>> > > > > > > have
>> > > > > > > > today.
>> > > > > > > >
>> > > > > > > > References:
>> > > > > > > >
>> > > > > > > > 1. A. Yip et al. Improving Application Security with Data
>> > > > > > > > Flow
>> > > > > > > Assertions.
>> > > > > > > > In SOSP'2009.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > On Fri, Oct 25, 2013 at 10:02 AM, Luke Lu <ll...@apache.org>
>> > > wrote:
>> > > > > > > >
>> > > > > > > >> Echoing my comments on HDFS-3555:
>> > > > > > > >>
>> > > > > > > >> I have concerns with this client-side js only approach,
>> > > > > > > >> which is
>> > > > > less
>> > > > > > > >> secure than a progressively enhanced hybrid approach used
>> > > > > > > >> by
>> > > YARN.
>> > > > > The
>> > > > > > > >> recent gmail XSS fiasco highlights the issue. I also have
>> > > concerns
>> > > > > > that
>> > > > > > > we
>> > > > > > > >> commit these changes without matching unit tests - the fact
>> > > > > > > >> you
>> > > > > cannot
>> > > > > > > >> effectively unit test these changes should tell you
>> > > > > > > >> something
>> > > > about
>> > > > > > this
>> > > > > > > >> approach.
>> > > > > > > >>
>> > > > > > > >> *Requiring* JS means that an admin cannot turn off js to
>> > > > (partially)
>> > > > > > use
>> > > > > > > >> core Hadoop UI. You'd *require* proper SSL (not self
>> > > > > > > >> signed)
>> > > setup
>> > > > > to
>> > > > > > > avoid
>> > > > > > > >> JS injection, even if security of js libraries used is
>> > > > > > > >> perfect,
>> > > > > which
>> > > > > > I
>> > > > > > > >> doubt (search gmail/linkedin XSS). Client side rendering
>> > > > completely
>> > > > > > > breaks
>> > > > > > > >> the workflows for ops who rely on text based
>> > > > > > > >> terminal/emacs/vim
>> > > > > > browsers
>> > > > > > > >> (no js support) to monitor component UI.
>> > > > > > > >>
>> > > > > > > >> IMO, JS-only rendering belongs to social networking sites
>> > > > > > > >> and/or
>> > > > > SaaS
>> > > > > > > >> front-ends, where full time UI/security specialists
>> > > > > > > >> babysits UI
>> > > > > > > changes. I
>> > > > > > > >> think eventually most users will use a self servicing UI in
>> > > > > > > >> a
>> > > SaaS
>> > > > > > > >> front-end that uses REST/JMX API to get data from back-end
>> > > > > components,
>> > > > > > > >> besides their own app master/service UI. The
>> > > priority/requirements
>> > > > > for
>> > > > > > > UI
>> > > > > > > >> in core Hadoop should be security and correctness, which
>> > > > > > > >> client
>> > > > side
>> > > > > > > >> templating cannot address properly so far.
>> > > > > > > >>
>> > > > > > > >>
>> > > > > > > >> On Tue, Oct 22, 2013 at 3:59 PM, Haohui Mai <
>> > > hmai@hortonworks.com
>> > > > >
>> > > > > > > wrote:
>> > > > > > > >>
>> > > > > > > >> > Hi all,
>> > > > > > > >> >
>> > > > > > > >> >
>> > > > > > > >> > Jing Zhao and I recently have reimplemented the JSP-based
>> > > > > > > >> > web
>> > > > UIs
>> > > > > in
>> > > > > > > >> HTML 5
>> > > > > > > >> > applications (HDFS-5333). Based on our prelimanary
>> > > > > > > >> > testing
>> > > > results
>> > > > > > we
>> > > > > > > >> > believe thst the new web UIs of the namenodes and the
>> > > > > > > >> > datanode
>> > > > are
>> > > > > > > ready
>> > > > > > > >> > for everyday uses.
>> > > > > > > >> >
>> > > > > > > >> > You're more than welcome to try it out on trunk by
>> > > > > > > >> > visiting
>> > > > > http://
>> > > > > > > >> > <namenode>/dfshealth.html
>> > > > > > > >> >
>> > > > > > > >> > There are a number of benefits from this transition. From
>> > > > > > > >> > a
>> > > > > > > developer's
>> > > > > > > >> > prospective, the most notable one is *maintainability*:
>> > > > > > > >> >
>> > > > > > > >> > (1) The abstractions between the UI and the core server
>> > > > > > > >> > are
>> > > > > > > well-defined,
>> > > > > > > >> > decoupling the UI and the core hadoop servers.
>> > > > > > > >> >
>> > > > > > > >> > (2) It allows us to deprecate the logic in the JSP pages.
>> > > > > > > >> > The
>> > > > old
>> > > > > > web
>> > > > > > > UIs
>> > > > > > > >> > have to duplicate the logic in the JSPs. The logic is
>> > > > > > > >> > often
>> > > > > > > out-of-dated
>> > > > > > > >> > and not well-tested, which leads to broken pages and
>> > > > > > > >> > security vulnerabilities(e.g. HDFS-5251, HDFS-5307,
>> > > > > > > >> > HDFS-5308,
>> > > HDFS-5317
>> > > > > and
>> > > > > > > >> > HDFS-4901). The architecture of the new UIs prevent these
>> > > > > > > >> > bugs
>> > > > at
>> > > > > > the
>> > > > > > > >> very
>> > > > > > > >> > beginning.
>> > > > > > > >> >
>> > > > > > > >> >
>> > > > > > > >> > I propose that deprecate the old, JSP-based web UIs in
>> > > > > > > >> > 2.3. I
>> > > > > opened
>> > > > > > > >> > HDFS-5402 to track the relevant discussions.
>> > > > > > > >> >
>> > > > > > > >> > Your feedbacks are highly appreciated.
>> > > > > > > >> >
>> > > > > > > >> >
>> > > > > > > >> > Sincerely,
>> > > > > > > >> >
>> > > > > > > >> > Haohui
>> > > > > > > >> >
>> > > > > > > >> > --
>> > > > > > > >> > CONFIDENTIALITY NOTICE
>> > > > > > > >> > NOTICE: This message is intended for the use of the
>> > > > > > > >> > individual
>> > > > or
>> > > > > > > entity
>> > > > > > > >> to
>> > > > > > > >> > which it is addressed and may contain information that is
>> > > > > > > confidential,
>> > > > > > > >> > privileged and exempt from disclosure under applicable
>> > > > > > > >> > law. If
>> > > > the
>> > > > > > > reader
>> > > > > > > >> > of this message is not the intended recipient, you are
>> > > > > > > >> > hereby
>> > > > > > notified
>> > > > > > > >> that
>> > > > > > > >> > any printing, copying, dissemination, distribution,
>> > > > > > > >> > disclosure
>> > > > or
>> > > > > > > >> > forwarding of this communication is strictly prohibited.
>> > > > > > > >> > If
>> > > you
>> > > > > have
>> > > > > > > >> > received this communication in error, please contact the
>> > > sender
>> > > > > > > >> immediately
>> > > > > > > >> > and delete it from your system. Thank You.
>> > > > > > > >> >
>> > > > > > > >>
>> > > > > > > >
>> > > > > > > > --
>> > > > > > > > CONFIDENTIALITY NOTICE
>> > > > > > > > NOTICE: This message is intended for the use of the
>> > > > > > > > individual or
>> > > > > > entity
>> > > > > > > to
>> > > > > > > > which it is addressed and may contain information that is
>> > > > > confidential,
>> > > > > > > > privileged and exempt from disclosure under applicable law.
>> > > > > > > > If
>> > > the
>> > > > > > reader
>> > > > > > > > of this message is not the intended recipient, you are
>> > > > > > > > hereby
>> > > > > notified
>> > > > > > > that
>> > > > > > > > any printing, copying, dissemination, distribution,
>> > > > > > > > disclosure or forwarding of this communication is strictly
>> > > > > > > > prohibited. If you
>> > > > have
>> > > > > > > > received this communication in error, please contact the
>> > > > > > > > sender
>> > > > > > > immediately
>> > > > > > > > and delete it from your system. Thank You.
>> > > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > --
>> > > > > > Alejandro
>> > > > > >
>> > > > >
>> > > > > --
>> > > > > CONFIDENTIALITY NOTICE
>> > > > > NOTICE: This message is intended for the use of the individual or
>> > > entity
>> > > > to
>> > > > > which it is addressed and may contain information that is
>> > > > > confidential, privileged and exempt from disclosure under
>> > > > > applicable law. If the
>> > > reader
>> > > > > of this message is not the intended recipient, you are hereby
>> > > > > notified
>> > > > that
>> > > > > any printing, copying, dissemination, distribution, disclosure or
>> > > > > forwarding of this communication is strictly prohibited. If you
>> > > > > have received this communication in error, please contact the
>> > > > > sender
>> > > > immediately
>> > > > > and delete it from your system. Thank You.
>> > > > >
>> > > >
>> > > >
>> > > >
>> > > > --
>> > > > Alejandro
>> > > >
>> > >
>> > > --
>> > > CONFIDENTIALITY NOTICE
>> > > NOTICE: This message is intended for the use of the individual or
>> > > entity to which it is addressed and may contain information that is
>> > > confidential, privileged and exempt from disclosure under applicable
>> > > law. If the reader of this message is not the intended recipient, you
>> > > are hereby notified that any printing, copying, dissemination,
>> > > distribution, disclosure or forwarding of this communication is
>> > > strictly prohibited. If you have received this communication in error,
>> > > please contact the sender immediately and delete it from your system.
>> > Thank You.
>> > >
>> >
>> >
>> >
>> > --
>> > Alejandro
>> >
>>
>> --
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity to
>> which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>>

Re: Replacing the JSP web UIs to HTML 5 applications

[Haohui Mai <hm...@hortonworks.com>]

Hi Andrew,

You always have the option to access the old JSP UI using the same URLs, if
you want to access the UI with a text-based browser.

~Haohui


On Wed, Oct 30, 2013 at 12:11 PM, Andrew Wang <an...@cloudera.com>wrote:

> I'm also not convinced that a Javascript-based approach is the way to go.
> We shouldn't switch the default UI until (at a minimum) we have the
> command-line tools that Colin requested, and even then I'd still want to
> retain support for text-based browsers like elinks unless there are
> compelling technical reasons not to.
>
> Haohui, I'm sympathetic since you've already done all this work on a
> pure-JS version, but it's also true that the existing JSP pages could be
> cleaned up to achieve basically the same visual effect while also still
> working in text-only browsers.
>
> Thanks,
> Andrew
>
>
> On Wed, Oct 30, 2013 at 12:34 AM, Luke Lu <ll...@vicaya.com> wrote:
>
> > I don't think that we have reached a consensus that the new javascript
> only
> > UI is the right direction to go. Most people considered it
> "interesting". I
> > personally think it's inappropriate for core Hadoop UI, as it increases
> > attack surface of the UI and taking away existing mitigation options from
> > users unnecessarily. See my latest comments on HDFS-5333 for "concrete"
> > examples.
> >
> > __Luke
> >
> >
> > On Tue, Oct 29, 2013 at 11:28 AM, Haohui Mai <hm...@hortonworks.com>
> wrote:
> >
> > > I would like to summarize the discussions so far. It seems that we have
> > > reached two consensus:
> > >
> > > 1. The new JavaScript-based UI is the right direction to go.
> > > 2. For now we should keep the old JSP pages around for compatibility
> > > reasons.
> > >
> > > There're some debates on the usages of the JMX / JSON APIs, but this is
> > > orthogonal to switching the UI, thus I consider it as a technical
> detail.
> > > We can continue the discussions in the public jira.
> > >
> > > The new UI has already landed in the trunk, based on the consensus it
> > seems
> > > that we can switch the default UI to the new one shortly. The user can
> > > still access the old web UI using the same URLs.
> > >
> > > The only question remain is that who is going to maintain the old web
> UI.
> > > My answer is that we should leave them as deprecated and focus the
> effort
> > > on the new web UI.
> > >
> > > Thanks,
> > > Haohui
> > >
> > >
> > >
> > > On Tue, Oct 29, 2013 at 5:22 AM, Zheng, Kai <ka...@intel.com>
> wrote:
> > >
> > > > > having /JMX for monitoring integration and a /JSON end point for
> the
> > UI
> > > > IMHO, this makes sense, especially for the long term. JMX interface
> > > serves
> > > > as management console in admin perspective, WebUI serves as end user
> > > > interface. Both might share same functionality codes, but that does
> not
> > > > validate we couple them together.
> > > >
> > > > Thanks & regards,
> > > > Kai
> > > >
> > > > -----Original Message-----
> > > > From: Alejandro Abdelnur [mailto:tucu@cloudera.com]
> > > > Sent: Tuesday, October 29, 2013 8:14 AM
> > > > To: hdfs-dev@hadoop.apache.org
> > > > Subject: Re: Replacing the JSP web UIs to HTML 5 applications
> > > >
> > > > Isn't using JMX to expose JSON for the web UI misusing JMX?
> > > >
> > > > I would think a more appropriate approach would be having /JMX for
> > > > monitoring integration and a /JSON end point for the UI data.
> > > >
> > > > Thanks.
> > > >
> > > >
> > > > On Mon, Oct 28, 2013 at 4:58 PM, Haohui Mai <hm...@hortonworks.com>
> > > wrote:
> > > >
> > > > > Alejandro,
> > > > >
> > > > > If I understand correctly, that is the exact approach that the new
> > web
> > > > > UI is taking. The new web UI takes the output from JMX and renders
> > > > > them as HTML at the client side.
> > > > >
> > > > > ~Haohui
> > > > >
> > > > >
> > > > > On Mon, Oct 28, 2013 at 4:18 PM, Alejandro Abdelnur <
> > tucu@cloudera.com
> > > > > >wrote:
> > > > >
> > > > > > Haohui,
> > > > > >
> > > > > > If you have NN and DNs producing JSON instead HTML, then you can
> > > > > > build JS based web UIs. Take for example Oozie, Oozie produces
> > JSON,
> > > > > > it has a
> > > > > built
> > > > > > in JS web ui that consumes JSON and Hue has built an external web
> > UI
> > > > > > that also consumes JSON. In the case of Hue UI, Oozie didn't have
> > to
> > > > > > change anything to get that UI and improvements on the Hue UI
> don't
> > > > > > require changes in Oozie unless it is to produce additional
> > > > information.
> > > > > >
> > > > > > hope this clarifies.
> > > > > >
> > > > > > Thx
> > > > > >
> > > > > >
> > > > > > On Mon, Oct 28, 2013 at 4:06 PM, Haohui Mai <
> hmai@hortonworks.com>
> > > > > wrote:
> > > > > >
> > > > > > > Echo my comments on HDFS-5402:
> > > > > > >
> > > > > > > bq. If we're going to remove the old web UI, I think the new
> web
> > > > > > > UI has to have the same level of unit testing. We shouldn't go
> > > > > > > backwards in terms of unit testing.
> > > > > > >
> > > > > > > I take a look at TestNamenodeJspHelper / TestDatanodeJspHelper
> /
> > > > > > > TestClusterJspHelper. It seems to me that we can merge these
> > tests
> > > > > > > with
> > > > > > the
> > > > > > > unit tests on JMX.
> > > > > > >
> > > > > > > bq. If we are going to
> > > > > > > remove this capability, we need to add some other command-line
> > > > > > > tools to get the same functionality. These tools could use REST
> > if
> > > > > > > we have that, or JMX, but they need to exist before we can
> > > > > > > consider removing the old UI.
> > > > > > >
> > > > > > > This is a good point. Since all information are available
> through
> > > > > > > JMX,
> > > > > > the
> > > > > > > easiest way to approach it is to write some scripts using
> > Node.js.
> > > > > > > The architecture of the new Web UIs is ready for this.
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Oct 28, 2013 at 3:57 PM, Alejandro Abdelnur
> > > > > > > <tucu@cloudera.com
> > > > > > > >wrote:
> > > > > > >
> > > > > > > > Producing JSON would be great. Agree with Colin that we
> should
> > > > > > > > leave
> > > > > > for
> > > > > > > > now the current JSP based web ui.
> > > > > > > >
> > > > > > > > thx
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, Oct 28, 2013 at 11:16 AM, Colin McCabe <
> > > > > cmccabe@alumni.cmu.edu
> > > > > > > > >wrote:
> > > > > > > >
> > > > > > > > > This is a really interesting project, Haohui.  I think it
> > will
> > > > > > > > > make our web UI much nicer.
> > > > > > > > >
> > > > > > > > > I have a few concerns about removing the old web UI,
> however:
> > > > > > > > >
> > > > > > > > > * If we're going to remove the old web UI, I think the new
> > web
> > > > > > > > > UI
> > > > > has
> > > > > > > > > to have the same level of unit testing.  We shouldn't go
> > > > > > > > > backwards
> > > > > in
> > > > > > > > > terms of unit testing.
> > > > > > > > >
> > > > > > > > > * Most of the deployments of elinks and links out there
> don't
> > > > > support
> > > > > > > > > Javascript.  This is just a reality of life when using
> CentOS
> > > > > > > > > 5 or
> > > > > 6,
> > > > > > > > > which many users are still using.  I have used "links" to
> > > > > > > > > diagnose problems through the web UI in the past, in
> systems
> > > > > > > > > where access to the cluster was available only through
> > telnet.
> > > > > > > > > If we are going to remove this capability, we need to add
> > some
> > > > > > > > > other command-line
> > > > > tools
> > > > > > > > > to get the same functionality.  These tools could use REST
> if
> > > > > > > > > we
> > > > > have
> > > > > > > > > that, or JMX, but they need to exist before we can consider
> > > > > removing
> > > > > > > > > the old UI.
> > > > > > > > >
> > > > > > > > > best,
> > > > > > > > > Colin
> > > > > > > > >
> > > > > > > > > On Fri, Oct 25, 2013 at 7:31 PM, Haohui Mai
> > > > > > > > > <hm...@hortonworks.com>
> > > > > > > > wrote:
> > > > > > > > > > Thanks for the reply, Luke. Here I just echo my response
> > > > > > > > > > from the
> > > > > > > jira:
> > > > > > > > > >
> > > > > > > > > > bq. this client-side js only approach, which is less
> secure
> > > > > > > > > > than
> > > > > a
> > > > > > > > > > progressively enhanced hybrid approach used by YARN. The
> > > > > > > > > > recent
> > > > > > gmail
> > > > > > > > > > XSS fiasco highlights the issue.
> > > > > > > > > >
> > > > > > > > > > I'm presenting an informal security analysis to compare
> the
> > > > > > security
> > > > > > > of
> > > > > > > > > the
> > > > > > > > > > old and the new web UIs.
> > > > > > > > > >
> > > > > > > > > > An attacker launches an XSS attack by injecting malicious
> > > > > > > > > > code
> > > > > > which
> > > > > > > > are
> > > > > > > > > > usually HTML or JavaScript fragments into the web page,
> so
> > > > > > > > > > that
> > > > > the
> > > > > > > > > > malicious code can have the same privileges of the web
> > page.
> > > > > > > > > >
> > > > > > > > > > First, in the scope of XSS attacks, note that the threat
> > > > > > > > > > models
> > > > > of
> > > > > > > > > > launching XSS attacks on Internet sites Gmail/Linkedin
> and
> > > > > > > > > > the
> > > > > one
> > > > > > of
> > > > > > > > the
> > > > > > > > > > Hadoop UIs are different. They have fundamental different
> > > > > > > > > > sets of
> > > > > > > > > external
> > > > > > > > > > inputs that the attackers have control to. Internet sites
> > > > > > > > > > have
> > > > > > little
> > > > > > > > > > control of these inputs. In the case of Gmail / Linkedin,
> > an
> > > > > attack
> > > > > > > can
> > > > > > > > > > send you a crafted e-mail, or put malicious description
> in
> > > > > > > > > > his / her Linkedin profile. The sets of external inputs
> are
> > > > > *restricted*
> > > > > > in
> > > > > > > > > > Hadoop UIs. The new web UIs take JMX and WebHDFS as
> inputs.
> > > > > > > > > > The attacker has to launch a XSS attack by:
> > > > > > > > > >
> > > > > > > > > > * Compromise the jars so that the output of JMX / WebHDFS
> > > > > > > > > > have
> > > > > the
> > > > > > > > > > malicious code.
> > > > > > > > > > * Replace the web UIs completely to include the malicious
> > > code.
> > > > > > > > > >
> > > > > > > > > > In either case *the attacker has to compromise the hadoop
> > > > > > > > > > core or
> > > > > > the
> > > > > > > > > > namenode*. That means the new web UIs are at least as
> > secure
> > > > > > > > > > as
> > > > > the
> > > > > > > > > hadoop
> > > > > > > > > > core, and the namenode machine.
> > > > > > > > > >
> > > > > > > > > > Second, I argue that using client-side templates are more
> > > > > > > > > > secure
> > > > > > than
> > > > > > > > the
> > > > > > > > > > current JSP-based server-side templates. To defend
> against
> > > > > > > > > > XSS attacks, both techniques have to filter the external
> > > > > > > > > > inputs at
> > > > > > > *every*
> > > > > > > > > > possible execution paths. Several facts much be taken
> into
> > > > > > > > > > plays when evaluating the security of both approaches in
> > > > > real-world
> > > > > > > > > > environments:
> > > > > > > > > >
> > > > > > > > > > * The JavaScript libraries used in the new web UIs have
> > > > > > > > > > survived
> > > > > in
> > > > > > > > > > extremely large-scale production tests. jQuery is used by
> > > > > > > > > > Google
> > > > > > and
> > > > > > > > > >  Microsoft, bootstrap is used by Twitter, and dust.js is
> > > > > > > > > > used by
> > > > > > > > > Linkedin.
> > > > > > > > > > All libraries survived from hundreds of thousands of
> >  attack
> > > > > > > > > > attempts on a daily basis. I agree that the libraries
> > > > > might
> > > > > > > > still
> > > > > > > > > > be imperfect, but there's no way that we can test the JSP
> > > > > > > > > > web  UIs to achieve the same level of assurances given
> the
> > > > > > > > > > amount of
> > > > > > > > > resources
> > > > > > > > > > the community has.
> > > > > > > > > > * Client-side templates consolidate all filtering logic
> in
> > > > > > > > > > one
> > > > > > > central
> > > > > > > > > > place. Recall that the goal is to filter all external
> > inputs
> > > > > > > > > > at
> > > > > > every
> > > > > > > > > >  execution paths, this is a much more systematic approach
> > > > > compared
> > > > > > to
> > > > > > > > the
> > > > > > > > > > server-side templates we have today. It is difficult (if
> > not
> > > > > > > > > >  impossible) to do it in a JSP/ASP/PHP application, since
> > > > > > > > > > such
> > > > > > > > filtering
> > > > > > > > > > can be only achieved via ad-hoc approaches ([1] shows
> some
> > > > > > > > > > empirical data). Also, HDFS-4901 recently describes a XSS
> > > > > > > > vulnerability
> > > > > > > > > in
> > > > > > > > > > browseDirectory.jsp.
> > > > > > > > > >
> > > > > > > > > > bq. You'd require proper SSL (not self signed) setup to
> > > > > > > > > > avoid JS injection
> > > > > > > > > >
> > > > > > > > > > Commodity browsers enforce Same-Origin Policy to defend
> > > > > > > > > > against
> > > > > > code
> > > > > > > > > > injections. It has nothing to do with what kinds of SSL
> > > > > > certificates
> > > > > > > > > > you hold.
> > > > > > > > > >
> > > > > > > > > > bq.  I also have concerns that we commit these changes
> > > > > > > > > > without
> > > > > > > matching
> > > > > > > > > > unit tests
> > > > > > > > > >
> > > > > > > > > > The JavaScript code can be automatically tested. The same
> > > > > > > > > > code
> > > > > can
> > > > > > be
> > > > > > > > run
> > > > > > > > > > by node.js and the test can compared with pre-defined
> > > > > > > > > > results. It is also possible to write an adapter to use
> > > > > > > > > > Rhino to
> > > > > > > > > accomplish
> > > > > > > > > > the same task. We can discuss how to integrate them into
> > the
> > > > > > > > > > maven test routines in a different thread.
> > > > > > > > > >
> > > > > > > > > > bq. Client side rendering completely breaks the workflows
> > > > > > > > > > for ops
> > > > > > who
> > > > > > > > > rely
> > > > > > > > > > on text based terminal/emacs/vim browsers (no js support)
> > to
> > > > > > > > > > monitor component UI.
> > > > > > > > > >
> > > > > > > > > > links / elinks (http://elinks.or.cz/) are text-based web
> > > > > browsers
> > > > > > > that
> > > > > > > > > > support JavaScript.
> > > > > > > > > >
> > > > > > > > > > bq. The priority/requirements for UI in core Hadoop
> should
> > > > > > > > > > be
> > > > > > > security
> > > > > > > > > and
> > > > > > > > > > correctness, which client side templating cannot address
> > > > > > > > > > properly so far.
> > > > > > > > > >
> > > > > > > > > > I agree that we should focus on security and correctness.
> > > > > > > > > > The
> > > > > > > > paragraphs
> > > > > > > > > > above explain that how the architecture of the new UIs
> > makes
> > > > > > > > > > the UIs more secure in real-world settings compared to
> the
> > > > > UI
> > > > > > > we
> > > > > > > > > have
> > > > > > > > > > today.
> > > > > > > > > >
> > > > > > > > > > References:
> > > > > > > > > >
> > > > > > > > > > 1. A. Yip et al. Improving Application Security with Data
> > > > > > > > > > Flow
> > > > > > > > > Assertions.
> > > > > > > > > > In SOSP'2009.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Fri, Oct 25, 2013 at 10:02 AM, Luke Lu <
> llu@apache.org>
> > > > > wrote:
> > > > > > > > > >
> > > > > > > > > >> Echoing my comments on HDFS-3555:
> > > > > > > > > >>
> > > > > > > > > >> I have concerns with this client-side js only approach,
> > > > > > > > > >> which is
> > > > > > > less
> > > > > > > > > >> secure than a progressively enhanced hybrid approach
> used
> > > > > > > > > >> by
> > > > > YARN.
> > > > > > > The
> > > > > > > > > >> recent gmail XSS fiasco highlights the issue. I also
> have
> > > > > concerns
> > > > > > > > that
> > > > > > > > > we
> > > > > > > > > >> commit these changes without matching unit tests - the
> > fact
> > > > > > > > > >> you
> > > > > > > cannot
> > > > > > > > > >> effectively unit test these changes should tell you
> > > > > > > > > >> something
> > > > > > about
> > > > > > > > this
> > > > > > > > > >> approach.
> > > > > > > > > >>
> > > > > > > > > >> *Requiring* JS means that an admin cannot turn off js to
> > > > > > (partially)
> > > > > > > > use
> > > > > > > > > >> core Hadoop UI. You'd *require* proper SSL (not self
> > > > > > > > > >> signed)
> > > > > setup
> > > > > > > to
> > > > > > > > > avoid
> > > > > > > > > >> JS injection, even if security of js libraries used is
> > > > > > > > > >> perfect,
> > > > > > > which
> > > > > > > > I
> > > > > > > > > >> doubt (search gmail/linkedin XSS). Client side rendering
> > > > > > completely
> > > > > > > > > breaks
> > > > > > > > > >> the workflows for ops who rely on text based
> > > > > > > > > >> terminal/emacs/vim
> > > > > > > > browsers
> > > > > > > > > >> (no js support) to monitor component UI.
> > > > > > > > > >>
> > > > > > > > > >> IMO, JS-only rendering belongs to social networking
> sites
> > > > > > > > > >> and/or
> > > > > > > SaaS
> > > > > > > > > >> front-ends, where full time UI/security specialists
> > > > > > > > > >> babysits UI
> > > > > > > > > changes. I
> > > > > > > > > >> think eventually most users will use a self servicing UI
> > in
> > > > > > > > > >> a
> > > > > SaaS
> > > > > > > > > >> front-end that uses REST/JMX API to get data from
> back-end
> > > > > > > components,
> > > > > > > > > >> besides their own app master/service UI. The
> > > > > priority/requirements
> > > > > > > for
> > > > > > > > > UI
> > > > > > > > > >> in core Hadoop should be security and correctness, which
> > > > > > > > > >> client
> > > > > > side
> > > > > > > > > >> templating cannot address properly so far.
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >> On Tue, Oct 22, 2013 at 3:59 PM, Haohui Mai <
> > > > > hmai@hortonworks.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > >>
> > > > > > > > > >> > Hi all,
> > > > > > > > > >> >
> > > > > > > > > >> >
> > > > > > > > > >> > Jing Zhao and I recently have reimplemented the
> > JSP-based
> > > > > > > > > >> > web
> > > > > > UIs
> > > > > > > in
> > > > > > > > > >> HTML 5
> > > > > > > > > >> > applications (HDFS-5333). Based on our prelimanary
> > > > > > > > > >> > testing
> > > > > > results
> > > > > > > > we
> > > > > > > > > >> > believe thst the new web UIs of the namenodes and the
> > > > > > > > > >> > datanode
> > > > > > are
> > > > > > > > > ready
> > > > > > > > > >> > for everyday uses.
> > > > > > > > > >> >
> > > > > > > > > >> > You're more than welcome to try it out on trunk by
> > > > > > > > > >> > visiting
> > > > > > > http://
> > > > > > > > > >> > <namenode>/dfshealth.html
> > > > > > > > > >> >
> > > > > > > > > >> > There are a number of benefits from this transition.
> > From
> > > > > > > > > >> > a
> > > > > > > > > developer's
> > > > > > > > > >> > prospective, the most notable one is
> *maintainability*:
> > > > > > > > > >> >
> > > > > > > > > >> > (1) The abstractions between the UI and the core
> server
> > > > > > > > > >> > are
> > > > > > > > > well-defined,
> > > > > > > > > >> > decoupling the UI and the core hadoop servers.
> > > > > > > > > >> >
> > > > > > > > > >> > (2) It allows us to deprecate the logic in the JSP
> > pages.
> > > > > > > > > >> > The
> > > > > > old
> > > > > > > > web
> > > > > > > > > UIs
> > > > > > > > > >> > have to duplicate the logic in the JSPs. The logic is
> > > > > > > > > >> > often
> > > > > > > > > out-of-dated
> > > > > > > > > >> > and not well-tested, which leads to broken pages and
> > > > > > > > > >> > security vulnerabilities(e.g. HDFS-5251, HDFS-5307,
> > > > > > > > > >> > HDFS-5308,
> > > > > HDFS-5317
> > > > > > > and
> > > > > > > > > >> > HDFS-4901). The architecture of the new UIs prevent
> > these
> > > > > > > > > >> > bugs
> > > > > > at
> > > > > > > > the
> > > > > > > > > >> very
> > > > > > > > > >> > beginning.
> > > > > > > > > >> >
> > > > > > > > > >> >
> > > > > > > > > >> > I propose that deprecate the old, JSP-based web UIs in
> > > > > > > > > >> > 2.3. I
> > > > > > > opened
> > > > > > > > > >> > HDFS-5402 to track the relevant discussions.
> > > > > > > > > >> >
> > > > > > > > > >> > Your feedbacks are highly appreciated.
> > > > > > > > > >> >
> > > > > > > > > >> >
> > > > > > > > > >> > Sincerely,
> > > > > > > > > >> >
> > > > > > > > > >> > Haohui
> > > > > > > > > >> >
> > > > > > > > > >> > --
> > > > > > > > > >> > CONFIDENTIALITY NOTICE
> > > > > > > > > >> > NOTICE: This message is intended for the use of the
> > > > > > > > > >> > individual
> > > > > > or
> > > > > > > > > entity
> > > > > > > > > >> to
> > > > > > > > > >> > which it is addressed and may contain information that
> > is
> > > > > > > > > confidential,
> > > > > > > > > >> > privileged and exempt from disclosure under applicable
> > > > > > > > > >> > law. If
> > > > > > the
> > > > > > > > > reader
> > > > > > > > > >> > of this message is not the intended recipient, you are
> > > > > > > > > >> > hereby
> > > > > > > > notified
> > > > > > > > > >> that
> > > > > > > > > >> > any printing, copying, dissemination, distribution,
> > > > > > > > > >> > disclosure
> > > > > > or
> > > > > > > > > >> > forwarding of this communication is strictly
> prohibited.
> > > > > > > > > >> > If
> > > > > you
> > > > > > > have
> > > > > > > > > >> > received this communication in error, please contact
> the
> > > > > sender
> > > > > > > > > >> immediately
> > > > > > > > > >> > and delete it from your system. Thank You.
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > CONFIDENTIALITY NOTICE
> > > > > > > > > > NOTICE: This message is intended for the use of the
> > > > > > > > > > individual or
> > > > > > > > entity
> > > > > > > > > to
> > > > > > > > > > which it is addressed and may contain information that is
> > > > > > > confidential,
> > > > > > > > > > privileged and exempt from disclosure under applicable
> law.
> > > > > > > > > > If
> > > > > the
> > > > > > > > reader
> > > > > > > > > > of this message is not the intended recipient, you are
> > > > > > > > > > hereby
> > > > > > > notified
> > > > > > > > > that
> > > > > > > > > > any printing, copying, dissemination, distribution,
> > > > > > > > > > disclosure or forwarding of this communication is
> strictly
> > > > > > > > > > prohibited. If you
> > > > > > have
> > > > > > > > > > received this communication in error, please contact the
> > > > > > > > > > sender
> > > > > > > > > immediately
> > > > > > > > > > and delete it from your system. Thank You.
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Alejandro
> > > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > CONFIDENTIALITY NOTICE
> > > > > > > NOTICE: This message is intended for the use of the individual
> or
> > > > > entity
> > > > > > to
> > > > > > > which it is addressed and may contain information that is
> > > > > > > confidential, privileged and exempt from disclosure under
> > > > > > > applicable law. If the
> > > > > reader
> > > > > > > of this message is not the intended recipient, you are hereby
> > > > > > > notified
> > > > > > that
> > > > > > > any printing, copying, dissemination, distribution, disclosure
> or
> > > > > > > forwarding of this communication is strictly prohibited. If you
> > > > > > > have received this communication in error, please contact the
> > > > > > > sender
> > > > > > immediately
> > > > > > > and delete it from your system. Thank You.
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Alejandro
> > > > > >
> > > > >
> > > > > --
> > > > > CONFIDENTIALITY NOTICE
> > > > > NOTICE: This message is intended for the use of the individual or
> > > > > entity to which it is addressed and may contain information that is
> > > > > confidential, privileged and exempt from disclosure under
> applicable
> > > > > law. If the reader of this message is not the intended recipient,
> you
> > > > > are hereby notified that any printing, copying, dissemination,
> > > > > distribution, disclosure or forwarding of this communication is
> > > > > strictly prohibited. If you have received this communication in
> > error,
> > > > > please contact the sender immediately and delete it from your
> system.
> > > > Thank You.
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Alejandro
> > > >
> > >
> > > --
> > > CONFIDENTIALITY NOTICE
> > > NOTICE: This message is intended for the use of the individual or
> entity
> > to
> > > which it is addressed and may contain information that is confidential,
> > > privileged and exempt from disclosure under applicable law. If the
> reader
> > > of this message is not the intended recipient, you are hereby notified
> > that
> > > any printing, copying, dissemination, distribution, disclosure or
> > > forwarding of this communication is strictly prohibited. If you have
> > > received this communication in error, please contact the sender
> > immediately
> > > and delete it from your system. Thank You.
> > >
> >
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Replacing the JSP web UIs to HTML 5 applications

[Andrew Wang <an...@cloudera.com>]

I'm also not convinced that a Javascript-based approach is the way to go.
We shouldn't switch the default UI until (at a minimum) we have the
command-line tools that Colin requested, and even then I'd still want to
retain support for text-based browsers like elinks unless there are
compelling technical reasons not to.

Haohui, I'm sympathetic since you've already done all this work on a
pure-JS version, but it's also true that the existing JSP pages could be
cleaned up to achieve basically the same visual effect while also still
working in text-only browsers.

Thanks,
Andrew


On Wed, Oct 30, 2013 at 12:34 AM, Luke Lu <ll...@vicaya.com> wrote:

> I don't think that we have reached a consensus that the new javascript only
> UI is the right direction to go. Most people considered it "interesting". I
> personally think it's inappropriate for core Hadoop UI, as it increases
> attack surface of the UI and taking away existing mitigation options from
> users unnecessarily. See my latest comments on HDFS-5333 for "concrete"
> examples.
>
> __Luke
>
>
> On Tue, Oct 29, 2013 at 11:28 AM, Haohui Mai <hm...@hortonworks.com> wrote:
>
> > I would like to summarize the discussions so far. It seems that we have
> > reached two consensus:
> >
> > 1. The new JavaScript-based UI is the right direction to go.
> > 2. For now we should keep the old JSP pages around for compatibility
> > reasons.
> >
> > There're some debates on the usages of the JMX / JSON APIs, but this is
> > orthogonal to switching the UI, thus I consider it as a technical detail.
> > We can continue the discussions in the public jira.
> >
> > The new UI has already landed in the trunk, based on the consensus it
> seems
> > that we can switch the default UI to the new one shortly. The user can
> > still access the old web UI using the same URLs.
> >
> > The only question remain is that who is going to maintain the old web UI.
> > My answer is that we should leave them as deprecated and focus the effort
> > on the new web UI.
> >
> > Thanks,
> > Haohui
> >
> >
> >
> > On Tue, Oct 29, 2013 at 5:22 AM, Zheng, Kai <ka...@intel.com> wrote:
> >
> > > > having /JMX for monitoring integration and a /JSON end point for the
> UI
> > > IMHO, this makes sense, especially for the long term. JMX interface
> > serves
> > > as management console in admin perspective, WebUI serves as end user
> > > interface. Both might share same functionality codes, but that does not
> > > validate we couple them together.
> > >
> > > Thanks & regards,
> > > Kai
> > >
> > > -----Original Message-----
> > > From: Alejandro Abdelnur [mailto:tucu@cloudera.com]
> > > Sent: Tuesday, October 29, 2013 8:14 AM
> > > To: hdfs-dev@hadoop.apache.org
> > > Subject: Re: Replacing the JSP web UIs to HTML 5 applications
> > >
> > > Isn't using JMX to expose JSON for the web UI misusing JMX?
> > >
> > > I would think a more appropriate approach would be having /JMX for
> > > monitoring integration and a /JSON end point for the UI data.
> > >
> > > Thanks.
> > >
> > >
> > > On Mon, Oct 28, 2013 at 4:58 PM, Haohui Mai <hm...@hortonworks.com>
> > wrote:
> > >
> > > > Alejandro,
> > > >
> > > > If I understand correctly, that is the exact approach that the new
> web
> > > > UI is taking. The new web UI takes the output from JMX and renders
> > > > them as HTML at the client side.
> > > >
> > > > ~Haohui
> > > >
> > > >
> > > > On Mon, Oct 28, 2013 at 4:18 PM, Alejandro Abdelnur <
> tucu@cloudera.com
> > > > >wrote:
> > > >
> > > > > Haohui,
> > > > >
> > > > > If you have NN and DNs producing JSON instead HTML, then you can
> > > > > build JS based web UIs. Take for example Oozie, Oozie produces
> JSON,
> > > > > it has a
> > > > built
> > > > > in JS web ui that consumes JSON and Hue has built an external web
> UI
> > > > > that also consumes JSON. In the case of Hue UI, Oozie didn't have
> to
> > > > > change anything to get that UI and improvements on the Hue UI don't
> > > > > require changes in Oozie unless it is to produce additional
> > > information.
> > > > >
> > > > > hope this clarifies.
> > > > >
> > > > > Thx
> > > > >
> > > > >
> > > > > On Mon, Oct 28, 2013 at 4:06 PM, Haohui Mai <hm...@hortonworks.com>
> > > > wrote:
> > > > >
> > > > > > Echo my comments on HDFS-5402:
> > > > > >
> > > > > > bq. If we're going to remove the old web UI, I think the new web
> > > > > > UI has to have the same level of unit testing. We shouldn't go
> > > > > > backwards in terms of unit testing.
> > > > > >
> > > > > > I take a look at TestNamenodeJspHelper / TestDatanodeJspHelper /
> > > > > > TestClusterJspHelper. It seems to me that we can merge these
> tests
> > > > > > with
> > > > > the
> > > > > > unit tests on JMX.
> > > > > >
> > > > > > bq. If we are going to
> > > > > > remove this capability, we need to add some other command-line
> > > > > > tools to get the same functionality. These tools could use REST
> if
> > > > > > we have that, or JMX, but they need to exist before we can
> > > > > > consider removing the old UI.
> > > > > >
> > > > > > This is a good point. Since all information are available through
> > > > > > JMX,
> > > > > the
> > > > > > easiest way to approach it is to write some scripts using
> Node.js.
> > > > > > The architecture of the new Web UIs is ready for this.
> > > > > >
> > > > > >
> > > > > > On Mon, Oct 28, 2013 at 3:57 PM, Alejandro Abdelnur
> > > > > > <tucu@cloudera.com
> > > > > > >wrote:
> > > > > >
> > > > > > > Producing JSON would be great. Agree with Colin that we should
> > > > > > > leave
> > > > > for
> > > > > > > now the current JSP based web ui.
> > > > > > >
> > > > > > > thx
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Oct 28, 2013 at 11:16 AM, Colin McCabe <
> > > > cmccabe@alumni.cmu.edu
> > > > > > > >wrote:
> > > > > > >
> > > > > > > > This is a really interesting project, Haohui.  I think it
> will
> > > > > > > > make our web UI much nicer.
> > > > > > > >
> > > > > > > > I have a few concerns about removing the old web UI, however:
> > > > > > > >
> > > > > > > > * If we're going to remove the old web UI, I think the new
> web
> > > > > > > > UI
> > > > has
> > > > > > > > to have the same level of unit testing.  We shouldn't go
> > > > > > > > backwards
> > > > in
> > > > > > > > terms of unit testing.
> > > > > > > >
> > > > > > > > * Most of the deployments of elinks and links out there don't
> > > > support
> > > > > > > > Javascript.  This is just a reality of life when using CentOS
> > > > > > > > 5 or
> > > > 6,
> > > > > > > > which many users are still using.  I have used "links" to
> > > > > > > > diagnose problems through the web UI in the past, in systems
> > > > > > > > where access to the cluster was available only through
> telnet.
> > > > > > > > If we are going to remove this capability, we need to add
> some
> > > > > > > > other command-line
> > > > tools
> > > > > > > > to get the same functionality.  These tools could use REST if
> > > > > > > > we
> > > > have
> > > > > > > > that, or JMX, but they need to exist before we can consider
> > > > removing
> > > > > > > > the old UI.
> > > > > > > >
> > > > > > > > best,
> > > > > > > > Colin
> > > > > > > >
> > > > > > > > On Fri, Oct 25, 2013 at 7:31 PM, Haohui Mai
> > > > > > > > <hm...@hortonworks.com>
> > > > > > > wrote:
> > > > > > > > > Thanks for the reply, Luke. Here I just echo my response
> > > > > > > > > from the
> > > > > > jira:
> > > > > > > > >
> > > > > > > > > bq. this client-side js only approach, which is less secure
> > > > > > > > > than
> > > > a
> > > > > > > > > progressively enhanced hybrid approach used by YARN. The
> > > > > > > > > recent
> > > > > gmail
> > > > > > > > > XSS fiasco highlights the issue.
> > > > > > > > >
> > > > > > > > > I'm presenting an informal security analysis to compare the
> > > > > security
> > > > > > of
> > > > > > > > the
> > > > > > > > > old and the new web UIs.
> > > > > > > > >
> > > > > > > > > An attacker launches an XSS attack by injecting malicious
> > > > > > > > > code
> > > > > which
> > > > > > > are
> > > > > > > > > usually HTML or JavaScript fragments into the web page, so
> > > > > > > > > that
> > > > the
> > > > > > > > > malicious code can have the same privileges of the web
> page.
> > > > > > > > >
> > > > > > > > > First, in the scope of XSS attacks, note that the threat
> > > > > > > > > models
> > > > of
> > > > > > > > > launching XSS attacks on Internet sites Gmail/Linkedin and
> > > > > > > > > the
> > > > one
> > > > > of
> > > > > > > the
> > > > > > > > > Hadoop UIs are different. They have fundamental different
> > > > > > > > > sets of
> > > > > > > > external
> > > > > > > > > inputs that the attackers have control to. Internet sites
> > > > > > > > > have
> > > > > little
> > > > > > > > > control of these inputs. In the case of Gmail / Linkedin,
> an
> > > > attack
> > > > > > can
> > > > > > > > > send you a crafted e-mail, or put malicious description in
> > > > > > > > > his / her Linkedin profile. The sets of external inputs are
> > > > *restricted*
> > > > > in
> > > > > > > > > Hadoop UIs. The new web UIs take JMX and WebHDFS as inputs.
> > > > > > > > > The attacker has to launch a XSS attack by:
> > > > > > > > >
> > > > > > > > > * Compromise the jars so that the output of JMX / WebHDFS
> > > > > > > > > have
> > > > the
> > > > > > > > > malicious code.
> > > > > > > > > * Replace the web UIs completely to include the malicious
> > code.
> > > > > > > > >
> > > > > > > > > In either case *the attacker has to compromise the hadoop
> > > > > > > > > core or
> > > > > the
> > > > > > > > > namenode*. That means the new web UIs are at least as
> secure
> > > > > > > > > as
> > > > the
> > > > > > > > hadoop
> > > > > > > > > core, and the namenode machine.
> > > > > > > > >
> > > > > > > > > Second, I argue that using client-side templates are more
> > > > > > > > > secure
> > > > > than
> > > > > > > the
> > > > > > > > > current JSP-based server-side templates. To defend against
> > > > > > > > > XSS attacks, both techniques have to filter the external
> > > > > > > > > inputs at
> > > > > > *every*
> > > > > > > > > possible execution paths. Several facts much be taken into
> > > > > > > > > plays when evaluating the security of both approaches in
> > > > real-world
> > > > > > > > > environments:
> > > > > > > > >
> > > > > > > > > * The JavaScript libraries used in the new web UIs have
> > > > > > > > > survived
> > > > in
> > > > > > > > > extremely large-scale production tests. jQuery is used by
> > > > > > > > > Google
> > > > > and
> > > > > > > > >  Microsoft, bootstrap is used by Twitter, and dust.js is
> > > > > > > > > used by
> > > > > > > > Linkedin.
> > > > > > > > > All libraries survived from hundreds of thousands of
>  attack
> > > > > > > > > attempts on a daily basis. I agree that the libraries
> > > > might
> > > > > > > still
> > > > > > > > > be imperfect, but there's no way that we can test the JSP
> > > > > > > > > web  UIs to achieve the same level of assurances given the
> > > > > > > > > amount of
> > > > > > > > resources
> > > > > > > > > the community has.
> > > > > > > > > * Client-side templates consolidate all filtering logic in
> > > > > > > > > one
> > > > > > central
> > > > > > > > > place. Recall that the goal is to filter all external
> inputs
> > > > > > > > > at
> > > > > every
> > > > > > > > >  execution paths, this is a much more systematic approach
> > > > compared
> > > > > to
> > > > > > > the
> > > > > > > > > server-side templates we have today. It is difficult (if
> not
> > > > > > > > >  impossible) to do it in a JSP/ASP/PHP application, since
> > > > > > > > > such
> > > > > > > filtering
> > > > > > > > > can be only achieved via ad-hoc approaches ([1] shows some
> > > > > > > > > empirical data). Also, HDFS-4901 recently describes a XSS
> > > > > > > vulnerability
> > > > > > > > in
> > > > > > > > > browseDirectory.jsp.
> > > > > > > > >
> > > > > > > > > bq. You'd require proper SSL (not self signed) setup to
> > > > > > > > > avoid JS injection
> > > > > > > > >
> > > > > > > > > Commodity browsers enforce Same-Origin Policy to defend
> > > > > > > > > against
> > > > > code
> > > > > > > > > injections. It has nothing to do with what kinds of SSL
> > > > > certificates
> > > > > > > > > you hold.
> > > > > > > > >
> > > > > > > > > bq.  I also have concerns that we commit these changes
> > > > > > > > > without
> > > > > > matching
> > > > > > > > > unit tests
> > > > > > > > >
> > > > > > > > > The JavaScript code can be automatically tested. The same
> > > > > > > > > code
> > > > can
> > > > > be
> > > > > > > run
> > > > > > > > > by node.js and the test can compared with pre-defined
> > > > > > > > > results. It is also possible to write an adapter to use
> > > > > > > > > Rhino to
> > > > > > > > accomplish
> > > > > > > > > the same task. We can discuss how to integrate them into
> the
> > > > > > > > > maven test routines in a different thread.
> > > > > > > > >
> > > > > > > > > bq. Client side rendering completely breaks the workflows
> > > > > > > > > for ops
> > > > > who
> > > > > > > > rely
> > > > > > > > > on text based terminal/emacs/vim browsers (no js support)
> to
> > > > > > > > > monitor component UI.
> > > > > > > > >
> > > > > > > > > links / elinks (http://elinks.or.cz/) are text-based web
> > > > browsers
> > > > > > that
> > > > > > > > > support JavaScript.
> > > > > > > > >
> > > > > > > > > bq. The priority/requirements for UI in core Hadoop should
> > > > > > > > > be
> > > > > > security
> > > > > > > > and
> > > > > > > > > correctness, which client side templating cannot address
> > > > > > > > > properly so far.
> > > > > > > > >
> > > > > > > > > I agree that we should focus on security and correctness.
> > > > > > > > > The
> > > > > > > paragraphs
> > > > > > > > > above explain that how the architecture of the new UIs
> makes
> > > > > > > > > the UIs more secure in real-world settings compared to the
> > > > UI
> > > > > > we
> > > > > > > > have
> > > > > > > > > today.
> > > > > > > > >
> > > > > > > > > References:
> > > > > > > > >
> > > > > > > > > 1. A. Yip et al. Improving Application Security with Data
> > > > > > > > > Flow
> > > > > > > > Assertions.
> > > > > > > > > In SOSP'2009.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Fri, Oct 25, 2013 at 10:02 AM, Luke Lu <ll...@apache.org>
> > > > wrote:
> > > > > > > > >
> > > > > > > > >> Echoing my comments on HDFS-3555:
> > > > > > > > >>
> > > > > > > > >> I have concerns with this client-side js only approach,
> > > > > > > > >> which is
> > > > > > less
> > > > > > > > >> secure than a progressively enhanced hybrid approach used
> > > > > > > > >> by
> > > > YARN.
> > > > > > The
> > > > > > > > >> recent gmail XSS fiasco highlights the issue. I also have
> > > > concerns
> > > > > > > that
> > > > > > > > we
> > > > > > > > >> commit these changes without matching unit tests - the
> fact
> > > > > > > > >> you
> > > > > > cannot
> > > > > > > > >> effectively unit test these changes should tell you
> > > > > > > > >> something
> > > > > about
> > > > > > > this
> > > > > > > > >> approach.
> > > > > > > > >>
> > > > > > > > >> *Requiring* JS means that an admin cannot turn off js to
> > > > > (partially)
> > > > > > > use
> > > > > > > > >> core Hadoop UI. You'd *require* proper SSL (not self
> > > > > > > > >> signed)
> > > > setup
> > > > > > to
> > > > > > > > avoid
> > > > > > > > >> JS injection, even if security of js libraries used is
> > > > > > > > >> perfect,
> > > > > > which
> > > > > > > I
> > > > > > > > >> doubt (search gmail/linkedin XSS). Client side rendering
> > > > > completely
> > > > > > > > breaks
> > > > > > > > >> the workflows for ops who rely on text based
> > > > > > > > >> terminal/emacs/vim
> > > > > > > browsers
> > > > > > > > >> (no js support) to monitor component UI.
> > > > > > > > >>
> > > > > > > > >> IMO, JS-only rendering belongs to social networking sites
> > > > > > > > >> and/or
> > > > > > SaaS
> > > > > > > > >> front-ends, where full time UI/security specialists
> > > > > > > > >> babysits UI
> > > > > > > > changes. I
> > > > > > > > >> think eventually most users will use a self servicing UI
> in
> > > > > > > > >> a
> > > > SaaS
> > > > > > > > >> front-end that uses REST/JMX API to get data from back-end
> > > > > > components,
> > > > > > > > >> besides their own app master/service UI. The
> > > > priority/requirements
> > > > > > for
> > > > > > > > UI
> > > > > > > > >> in core Hadoop should be security and correctness, which
> > > > > > > > >> client
> > > > > side
> > > > > > > > >> templating cannot address properly so far.
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> On Tue, Oct 22, 2013 at 3:59 PM, Haohui Mai <
> > > > hmai@hortonworks.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > >>
> > > > > > > > >> > Hi all,
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> > Jing Zhao and I recently have reimplemented the
> JSP-based
> > > > > > > > >> > web
> > > > > UIs
> > > > > > in
> > > > > > > > >> HTML 5
> > > > > > > > >> > applications (HDFS-5333). Based on our prelimanary
> > > > > > > > >> > testing
> > > > > results
> > > > > > > we
> > > > > > > > >> > believe thst the new web UIs of the namenodes and the
> > > > > > > > >> > datanode
> > > > > are
> > > > > > > > ready
> > > > > > > > >> > for everyday uses.
> > > > > > > > >> >
> > > > > > > > >> > You're more than welcome to try it out on trunk by
> > > > > > > > >> > visiting
> > > > > > http://
> > > > > > > > >> > <namenode>/dfshealth.html
> > > > > > > > >> >
> > > > > > > > >> > There are a number of benefits from this transition.
> From
> > > > > > > > >> > a
> > > > > > > > developer's
> > > > > > > > >> > prospective, the most notable one is *maintainability*:
> > > > > > > > >> >
> > > > > > > > >> > (1) The abstractions between the UI and the core server
> > > > > > > > >> > are
> > > > > > > > well-defined,
> > > > > > > > >> > decoupling the UI and the core hadoop servers.
> > > > > > > > >> >
> > > > > > > > >> > (2) It allows us to deprecate the logic in the JSP
> pages.
> > > > > > > > >> > The
> > > > > old
> > > > > > > web
> > > > > > > > UIs
> > > > > > > > >> > have to duplicate the logic in the JSPs. The logic is
> > > > > > > > >> > often
> > > > > > > > out-of-dated
> > > > > > > > >> > and not well-tested, which leads to broken pages and
> > > > > > > > >> > security vulnerabilities(e.g. HDFS-5251, HDFS-5307,
> > > > > > > > >> > HDFS-5308,
> > > > HDFS-5317
> > > > > > and
> > > > > > > > >> > HDFS-4901). The architecture of the new UIs prevent
> these
> > > > > > > > >> > bugs
> > > > > at
> > > > > > > the
> > > > > > > > >> very
> > > > > > > > >> > beginning.
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> > I propose that deprecate the old, JSP-based web UIs in
> > > > > > > > >> > 2.3. I
> > > > > > opened
> > > > > > > > >> > HDFS-5402 to track the relevant discussions.
> > > > > > > > >> >
> > > > > > > > >> > Your feedbacks are highly appreciated.
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> > Sincerely,
> > > > > > > > >> >
> > > > > > > > >> > Haohui
> > > > > > > > >> >
> > > > > > > > >> > --
> > > > > > > > >> > CONFIDENTIALITY NOTICE
> > > > > > > > >> > NOTICE: This message is intended for the use of the
> > > > > > > > >> > individual
> > > > > or
> > > > > > > > entity
> > > > > > > > >> to
> > > > > > > > >> > which it is addressed and may contain information that
> is
> > > > > > > > confidential,
> > > > > > > > >> > privileged and exempt from disclosure under applicable
> > > > > > > > >> > law. If
> > > > > the
> > > > > > > > reader
> > > > > > > > >> > of this message is not the intended recipient, you are
> > > > > > > > >> > hereby
> > > > > > > notified
> > > > > > > > >> that
> > > > > > > > >> > any printing, copying, dissemination, distribution,
> > > > > > > > >> > disclosure
> > > > > or
> > > > > > > > >> > forwarding of this communication is strictly prohibited.
> > > > > > > > >> > If
> > > > you
> > > > > > have
> > > > > > > > >> > received this communication in error, please contact the
> > > > sender
> > > > > > > > >> immediately
> > > > > > > > >> > and delete it from your system. Thank You.
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > CONFIDENTIALITY NOTICE
> > > > > > > > > NOTICE: This message is intended for the use of the
> > > > > > > > > individual or
> > > > > > > entity
> > > > > > > > to
> > > > > > > > > which it is addressed and may contain information that is
> > > > > > confidential,
> > > > > > > > > privileged and exempt from disclosure under applicable law.
> > > > > > > > > If
> > > > the
> > > > > > > reader
> > > > > > > > > of this message is not the intended recipient, you are
> > > > > > > > > hereby
> > > > > > notified
> > > > > > > > that
> > > > > > > > > any printing, copying, dissemination, distribution,
> > > > > > > > > disclosure or forwarding of this communication is strictly
> > > > > > > > > prohibited. If you
> > > > > have
> > > > > > > > > received this communication in error, please contact the
> > > > > > > > > sender
> > > > > > > > immediately
> > > > > > > > > and delete it from your system. Thank You.
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Alejandro
> > > > > > >
> > > > > >
> > > > > > --
> > > > > > CONFIDENTIALITY NOTICE
> > > > > > NOTICE: This message is intended for the use of the individual or
> > > > entity
> > > > > to
> > > > > > which it is addressed and may contain information that is
> > > > > > confidential, privileged and exempt from disclosure under
> > > > > > applicable law. If the
> > > > reader
> > > > > > of this message is not the intended recipient, you are hereby
> > > > > > notified
> > > > > that
> > > > > > any printing, copying, dissemination, distribution, disclosure or
> > > > > > forwarding of this communication is strictly prohibited. If you
> > > > > > have received this communication in error, please contact the
> > > > > > sender
> > > > > immediately
> > > > > > and delete it from your system. Thank You.
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Alejandro
> > > > >
> > > >
> > > > --
> > > > CONFIDENTIALITY NOTICE
> > > > NOTICE: This message is intended for the use of the individual or
> > > > entity to which it is addressed and may contain information that is
> > > > confidential, privileged and exempt from disclosure under applicable
> > > > law. If the reader of this message is not the intended recipient, you
> > > > are hereby notified that any printing, copying, dissemination,
> > > > distribution, disclosure or forwarding of this communication is
> > > > strictly prohibited. If you have received this communication in
> error,
> > > > please contact the sender immediately and delete it from your system.
> > > Thank You.
> > > >
> > >
> > >
> > >
> > > --
> > > Alejandro
> > >
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> >
>

Re: Replacing the JSP web UIs to HTML 5 applications

[Luke Lu <ll...@vicaya.com>]

I don't think that we have reached a consensus that the new javascript only
UI is the right direction to go. Most people considered it "interesting". I
personally think it's inappropriate for core Hadoop UI, as it increases
attack surface of the UI and taking away existing mitigation options from
users unnecessarily. See my latest comments on HDFS-5333 for "concrete"
examples.

__Luke


On Tue, Oct 29, 2013 at 11:28 AM, Haohui Mai <hm...@hortonworks.com> wrote:

> I would like to summarize the discussions so far. It seems that we have
> reached two consensus:
>
> 1. The new JavaScript-based UI is the right direction to go.
> 2. For now we should keep the old JSP pages around for compatibility
> reasons.
>
> There're some debates on the usages of the JMX / JSON APIs, but this is
> orthogonal to switching the UI, thus I consider it as a technical detail.
> We can continue the discussions in the public jira.
>
> The new UI has already landed in the trunk, based on the consensus it seems
> that we can switch the default UI to the new one shortly. The user can
> still access the old web UI using the same URLs.
>
> The only question remain is that who is going to maintain the old web UI.
> My answer is that we should leave them as deprecated and focus the effort
> on the new web UI.
>
> Thanks,
> Haohui
>
>
>
> On Tue, Oct 29, 2013 at 5:22 AM, Zheng, Kai <ka...@intel.com> wrote:
>
> > > having /JMX for monitoring integration and a /JSON end point for the UI
> > IMHO, this makes sense, especially for the long term. JMX interface
> serves
> > as management console in admin perspective, WebUI serves as end user
> > interface. Both might share same functionality codes, but that does not
> > validate we couple them together.
> >
> > Thanks & regards,
> > Kai
> >
> > -----Original Message-----
> > From: Alejandro Abdelnur [mailto:tucu@cloudera.com]
> > Sent: Tuesday, October 29, 2013 8:14 AM
> > To: hdfs-dev@hadoop.apache.org
> > Subject: Re: Replacing the JSP web UIs to HTML 5 applications
> >
> > Isn't using JMX to expose JSON for the web UI misusing JMX?
> >
> > I would think a more appropriate approach would be having /JMX for
> > monitoring integration and a /JSON end point for the UI data.
> >
> > Thanks.
> >
> >
> > On Mon, Oct 28, 2013 at 4:58 PM, Haohui Mai <hm...@hortonworks.com>
> wrote:
> >
> > > Alejandro,
> > >
> > > If I understand correctly, that is the exact approach that the new web
> > > UI is taking. The new web UI takes the output from JMX and renders
> > > them as HTML at the client side.
> > >
> > > ~Haohui
> > >
> > >
> > > On Mon, Oct 28, 2013 at 4:18 PM, Alejandro Abdelnur <tucu@cloudera.com
> > > >wrote:
> > >
> > > > Haohui,
> > > >
> > > > If you have NN and DNs producing JSON instead HTML, then you can
> > > > build JS based web UIs. Take for example Oozie, Oozie produces JSON,
> > > > it has a
> > > built
> > > > in JS web ui that consumes JSON and Hue has built an external web UI
> > > > that also consumes JSON. In the case of Hue UI, Oozie didn't have to
> > > > change anything to get that UI and improvements on the Hue UI don't
> > > > require changes in Oozie unless it is to produce additional
> > information.
> > > >
> > > > hope this clarifies.
> > > >
> > > > Thx
> > > >
> > > >
> > > > On Mon, Oct 28, 2013 at 4:06 PM, Haohui Mai <hm...@hortonworks.com>
> > > wrote:
> > > >
> > > > > Echo my comments on HDFS-5402:
> > > > >
> > > > > bq. If we're going to remove the old web UI, I think the new web
> > > > > UI has to have the same level of unit testing. We shouldn't go
> > > > > backwards in terms of unit testing.
> > > > >
> > > > > I take a look at TestNamenodeJspHelper / TestDatanodeJspHelper /
> > > > > TestClusterJspHelper. It seems to me that we can merge these tests
> > > > > with
> > > > the
> > > > > unit tests on JMX.
> > > > >
> > > > > bq. If we are going to
> > > > > remove this capability, we need to add some other command-line
> > > > > tools to get the same functionality. These tools could use REST if
> > > > > we have that, or JMX, but they need to exist before we can
> > > > > consider removing the old UI.
> > > > >
> > > > > This is a good point. Since all information are available through
> > > > > JMX,
> > > > the
> > > > > easiest way to approach it is to write some scripts using Node.js.
> > > > > The architecture of the new Web UIs is ready for this.
> > > > >
> > > > >
> > > > > On Mon, Oct 28, 2013 at 3:57 PM, Alejandro Abdelnur
> > > > > <tucu@cloudera.com
> > > > > >wrote:
> > > > >
> > > > > > Producing JSON would be great. Agree with Colin that we should
> > > > > > leave
> > > > for
> > > > > > now the current JSP based web ui.
> > > > > >
> > > > > > thx
> > > > > >
> > > > > >
> > > > > > On Mon, Oct 28, 2013 at 11:16 AM, Colin McCabe <
> > > cmccabe@alumni.cmu.edu
> > > > > > >wrote:
> > > > > >
> > > > > > > This is a really interesting project, Haohui.  I think it will
> > > > > > > make our web UI much nicer.
> > > > > > >
> > > > > > > I have a few concerns about removing the old web UI, however:
> > > > > > >
> > > > > > > * If we're going to remove the old web UI, I think the new web
> > > > > > > UI
> > > has
> > > > > > > to have the same level of unit testing.  We shouldn't go
> > > > > > > backwards
> > > in
> > > > > > > terms of unit testing.
> > > > > > >
> > > > > > > * Most of the deployments of elinks and links out there don't
> > > support
> > > > > > > Javascript.  This is just a reality of life when using CentOS
> > > > > > > 5 or
> > > 6,
> > > > > > > which many users are still using.  I have used "links" to
> > > > > > > diagnose problems through the web UI in the past, in systems
> > > > > > > where access to the cluster was available only through telnet.
> > > > > > > If we are going to remove this capability, we need to add some
> > > > > > > other command-line
> > > tools
> > > > > > > to get the same functionality.  These tools could use REST if
> > > > > > > we
> > > have
> > > > > > > that, or JMX, but they need to exist before we can consider
> > > removing
> > > > > > > the old UI.
> > > > > > >
> > > > > > > best,
> > > > > > > Colin
> > > > > > >
> > > > > > > On Fri, Oct 25, 2013 at 7:31 PM, Haohui Mai
> > > > > > > <hm...@hortonworks.com>
> > > > > > wrote:
> > > > > > > > Thanks for the reply, Luke. Here I just echo my response
> > > > > > > > from the
> > > > > jira:
> > > > > > > >
> > > > > > > > bq. this client-side js only approach, which is less secure
> > > > > > > > than
> > > a
> > > > > > > > progressively enhanced hybrid approach used by YARN. The
> > > > > > > > recent
> > > > gmail
> > > > > > > > XSS fiasco highlights the issue.
> > > > > > > >
> > > > > > > > I'm presenting an informal security analysis to compare the
> > > > security
> > > > > of
> > > > > > > the
> > > > > > > > old and the new web UIs.
> > > > > > > >
> > > > > > > > An attacker launches an XSS attack by injecting malicious
> > > > > > > > code
> > > > which
> > > > > > are
> > > > > > > > usually HTML or JavaScript fragments into the web page, so
> > > > > > > > that
> > > the
> > > > > > > > malicious code can have the same privileges of the web page.
> > > > > > > >
> > > > > > > > First, in the scope of XSS attacks, note that the threat
> > > > > > > > models
> > > of
> > > > > > > > launching XSS attacks on Internet sites Gmail/Linkedin and
> > > > > > > > the
> > > one
> > > > of
> > > > > > the
> > > > > > > > Hadoop UIs are different. They have fundamental different
> > > > > > > > sets of
> > > > > > > external
> > > > > > > > inputs that the attackers have control to. Internet sites
> > > > > > > > have
> > > > little
> > > > > > > > control of these inputs. In the case of Gmail / Linkedin, an
> > > attack
> > > > > can
> > > > > > > > send you a crafted e-mail, or put malicious description in
> > > > > > > > his / her Linkedin profile. The sets of external inputs are
> > > *restricted*
> > > > in
> > > > > > > > Hadoop UIs. The new web UIs take JMX and WebHDFS as inputs.
> > > > > > > > The attacker has to launch a XSS attack by:
> > > > > > > >
> > > > > > > > * Compromise the jars so that the output of JMX / WebHDFS
> > > > > > > > have
> > > the
> > > > > > > > malicious code.
> > > > > > > > * Replace the web UIs completely to include the malicious
> code.
> > > > > > > >
> > > > > > > > In either case *the attacker has to compromise the hadoop
> > > > > > > > core or
> > > > the
> > > > > > > > namenode*. That means the new web UIs are at least as secure
> > > > > > > > as
> > > the
> > > > > > > hadoop
> > > > > > > > core, and the namenode machine.
> > > > > > > >
> > > > > > > > Second, I argue that using client-side templates are more
> > > > > > > > secure
> > > > than
> > > > > > the
> > > > > > > > current JSP-based server-side templates. To defend against
> > > > > > > > XSS attacks, both techniques have to filter the external
> > > > > > > > inputs at
> > > > > *every*
> > > > > > > > possible execution paths. Several facts much be taken into
> > > > > > > > plays when evaluating the security of both approaches in
> > > real-world
> > > > > > > > environments:
> > > > > > > >
> > > > > > > > * The JavaScript libraries used in the new web UIs have
> > > > > > > > survived
> > > in
> > > > > > > > extremely large-scale production tests. jQuery is used by
> > > > > > > > Google
> > > > and
> > > > > > > >  Microsoft, bootstrap is used by Twitter, and dust.js is
> > > > > > > > used by
> > > > > > > Linkedin.
> > > > > > > > All libraries survived from hundreds of thousands of  attack
> > > > > > > > attempts on a daily basis. I agree that the libraries
> > > might
> > > > > > still
> > > > > > > > be imperfect, but there's no way that we can test the JSP
> > > > > > > > web  UIs to achieve the same level of assurances given the
> > > > > > > > amount of
> > > > > > > resources
> > > > > > > > the community has.
> > > > > > > > * Client-side templates consolidate all filtering logic in
> > > > > > > > one
> > > > > central
> > > > > > > > place. Recall that the goal is to filter all external inputs
> > > > > > > > at
> > > > every
> > > > > > > >  execution paths, this is a much more systematic approach
> > > compared
> > > > to
> > > > > > the
> > > > > > > > server-side templates we have today. It is difficult (if not
> > > > > > > >  impossible) to do it in a JSP/ASP/PHP application, since
> > > > > > > > such
> > > > > > filtering
> > > > > > > > can be only achieved via ad-hoc approaches ([1] shows some
> > > > > > > > empirical data). Also, HDFS-4901 recently describes a XSS
> > > > > > vulnerability
> > > > > > > in
> > > > > > > > browseDirectory.jsp.
> > > > > > > >
> > > > > > > > bq. You'd require proper SSL (not self signed) setup to
> > > > > > > > avoid JS injection
> > > > > > > >
> > > > > > > > Commodity browsers enforce Same-Origin Policy to defend
> > > > > > > > against
> > > > code
> > > > > > > > injections. It has nothing to do with what kinds of SSL
> > > > certificates
> > > > > > > > you hold.
> > > > > > > >
> > > > > > > > bq.  I also have concerns that we commit these changes
> > > > > > > > without
> > > > > matching
> > > > > > > > unit tests
> > > > > > > >
> > > > > > > > The JavaScript code can be automatically tested. The same
> > > > > > > > code
> > > can
> > > > be
> > > > > > run
> > > > > > > > by node.js and the test can compared with pre-defined
> > > > > > > > results. It is also possible to write an adapter to use
> > > > > > > > Rhino to
> > > > > > > accomplish
> > > > > > > > the same task. We can discuss how to integrate them into the
> > > > > > > > maven test routines in a different thread.
> > > > > > > >
> > > > > > > > bq. Client side rendering completely breaks the workflows
> > > > > > > > for ops
> > > > who
> > > > > > > rely
> > > > > > > > on text based terminal/emacs/vim browsers (no js support) to
> > > > > > > > monitor component UI.
> > > > > > > >
> > > > > > > > links / elinks (http://elinks.or.cz/) are text-based web
> > > browsers
> > > > > that
> > > > > > > > support JavaScript.
> > > > > > > >
> > > > > > > > bq. The priority/requirements for UI in core Hadoop should
> > > > > > > > be
> > > > > security
> > > > > > > and
> > > > > > > > correctness, which client side templating cannot address
> > > > > > > > properly so far.
> > > > > > > >
> > > > > > > > I agree that we should focus on security and correctness.
> > > > > > > > The
> > > > > > paragraphs
> > > > > > > > above explain that how the architecture of the new UIs makes
> > > > > > > > the UIs more secure in real-world settings compared to the
> > > UI
> > > > > we
> > > > > > > have
> > > > > > > > today.
> > > > > > > >
> > > > > > > > References:
> > > > > > > >
> > > > > > > > 1. A. Yip et al. Improving Application Security with Data
> > > > > > > > Flow
> > > > > > > Assertions.
> > > > > > > > In SOSP'2009.
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Oct 25, 2013 at 10:02 AM, Luke Lu <ll...@apache.org>
> > > wrote:
> > > > > > > >
> > > > > > > >> Echoing my comments on HDFS-3555:
> > > > > > > >>
> > > > > > > >> I have concerns with this client-side js only approach,
> > > > > > > >> which is
> > > > > less
> > > > > > > >> secure than a progressively enhanced hybrid approach used
> > > > > > > >> by
> > > YARN.
> > > > > The
> > > > > > > >> recent gmail XSS fiasco highlights the issue. I also have
> > > concerns
> > > > > > that
> > > > > > > we
> > > > > > > >> commit these changes without matching unit tests - the fact
> > > > > > > >> you
> > > > > cannot
> > > > > > > >> effectively unit test these changes should tell you
> > > > > > > >> something
> > > > about
> > > > > > this
> > > > > > > >> approach.
> > > > > > > >>
> > > > > > > >> *Requiring* JS means that an admin cannot turn off js to
> > > > (partially)
> > > > > > use
> > > > > > > >> core Hadoop UI. You'd *require* proper SSL (not self
> > > > > > > >> signed)
> > > setup
> > > > > to
> > > > > > > avoid
> > > > > > > >> JS injection, even if security of js libraries used is
> > > > > > > >> perfect,
> > > > > which
> > > > > > I
> > > > > > > >> doubt (search gmail/linkedin XSS). Client side rendering
> > > > completely
> > > > > > > breaks
> > > > > > > >> the workflows for ops who rely on text based
> > > > > > > >> terminal/emacs/vim
> > > > > > browsers
> > > > > > > >> (no js support) to monitor component UI.
> > > > > > > >>
> > > > > > > >> IMO, JS-only rendering belongs to social networking sites
> > > > > > > >> and/or
> > > > > SaaS
> > > > > > > >> front-ends, where full time UI/security specialists
> > > > > > > >> babysits UI
> > > > > > > changes. I
> > > > > > > >> think eventually most users will use a self servicing UI in
> > > > > > > >> a
> > > SaaS
> > > > > > > >> front-end that uses REST/JMX API to get data from back-end
> > > > > components,
> > > > > > > >> besides their own app master/service UI. The
> > > priority/requirements
> > > > > for
> > > > > > > UI
> > > > > > > >> in core Hadoop should be security and correctness, which
> > > > > > > >> client
> > > > side
> > > > > > > >> templating cannot address properly so far.
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> On Tue, Oct 22, 2013 at 3:59 PM, Haohui Mai <
> > > hmai@hortonworks.com
> > > > >
> > > > > > > wrote:
> > > > > > > >>
> > > > > > > >> > Hi all,
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > Jing Zhao and I recently have reimplemented the JSP-based
> > > > > > > >> > web
> > > > UIs
> > > > > in
> > > > > > > >> HTML 5
> > > > > > > >> > applications (HDFS-5333). Based on our prelimanary
> > > > > > > >> > testing
> > > > results
> > > > > > we
> > > > > > > >> > believe thst the new web UIs of the namenodes and the
> > > > > > > >> > datanode
> > > > are
> > > > > > > ready
> > > > > > > >> > for everyday uses.
> > > > > > > >> >
> > > > > > > >> > You're more than welcome to try it out on trunk by
> > > > > > > >> > visiting
> > > > > http://
> > > > > > > >> > <namenode>/dfshealth.html
> > > > > > > >> >
> > > > > > > >> > There are a number of benefits from this transition. From
> > > > > > > >> > a
> > > > > > > developer's
> > > > > > > >> > prospective, the most notable one is *maintainability*:
> > > > > > > >> >
> > > > > > > >> > (1) The abstractions between the UI and the core server
> > > > > > > >> > are
> > > > > > > well-defined,
> > > > > > > >> > decoupling the UI and the core hadoop servers.
> > > > > > > >> >
> > > > > > > >> > (2) It allows us to deprecate the logic in the JSP pages.
> > > > > > > >> > The
> > > > old
> > > > > > web
> > > > > > > UIs
> > > > > > > >> > have to duplicate the logic in the JSPs. The logic is
> > > > > > > >> > often
> > > > > > > out-of-dated
> > > > > > > >> > and not well-tested, which leads to broken pages and
> > > > > > > >> > security vulnerabilities(e.g. HDFS-5251, HDFS-5307,
> > > > > > > >> > HDFS-5308,
> > > HDFS-5317
> > > > > and
> > > > > > > >> > HDFS-4901). The architecture of the new UIs prevent these
> > > > > > > >> > bugs
> > > > at
> > > > > > the
> > > > > > > >> very
> > > > > > > >> > beginning.
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > I propose that deprecate the old, JSP-based web UIs in
> > > > > > > >> > 2.3. I
> > > > > opened
> > > > > > > >> > HDFS-5402 to track the relevant discussions.
> > > > > > > >> >
> > > > > > > >> > Your feedbacks are highly appreciated.
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > Sincerely,
> > > > > > > >> >
> > > > > > > >> > Haohui
> > > > > > > >> >
> > > > > > > >> > --
> > > > > > > >> > CONFIDENTIALITY NOTICE
> > > > > > > >> > NOTICE: This message is intended for the use of the
> > > > > > > >> > individual
> > > > or
> > > > > > > entity
> > > > > > > >> to
> > > > > > > >> > which it is addressed and may contain information that is
> > > > > > > confidential,
> > > > > > > >> > privileged and exempt from disclosure under applicable
> > > > > > > >> > law. If
> > > > the
> > > > > > > reader
> > > > > > > >> > of this message is not the intended recipient, you are
> > > > > > > >> > hereby
> > > > > > notified
> > > > > > > >> that
> > > > > > > >> > any printing, copying, dissemination, distribution,
> > > > > > > >> > disclosure
> > > > or
> > > > > > > >> > forwarding of this communication is strictly prohibited.
> > > > > > > >> > If
> > > you
> > > > > have
> > > > > > > >> > received this communication in error, please contact the
> > > sender
> > > > > > > >> immediately
> > > > > > > >> > and delete it from your system. Thank You.
> > > > > > > >> >
> > > > > > > >>
> > > > > > > >
> > > > > > > > --
> > > > > > > > CONFIDENTIALITY NOTICE
> > > > > > > > NOTICE: This message is intended for the use of the
> > > > > > > > individual or
> > > > > > entity
> > > > > > > to
> > > > > > > > which it is addressed and may contain information that is
> > > > > confidential,
> > > > > > > > privileged and exempt from disclosure under applicable law.
> > > > > > > > If
> > > the
> > > > > > reader
> > > > > > > > of this message is not the intended recipient, you are
> > > > > > > > hereby
> > > > > notified
> > > > > > > that
> > > > > > > > any printing, copying, dissemination, distribution,
> > > > > > > > disclosure or forwarding of this communication is strictly
> > > > > > > > prohibited. If you
> > > > have
> > > > > > > > received this communication in error, please contact the
> > > > > > > > sender
> > > > > > > immediately
> > > > > > > > and delete it from your system. Thank You.
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Alejandro
> > > > > >
> > > > >
> > > > > --
> > > > > CONFIDENTIALITY NOTICE
> > > > > NOTICE: This message is intended for the use of the individual or
> > > entity
> > > > to
> > > > > which it is addressed and may contain information that is
> > > > > confidential, privileged and exempt from disclosure under
> > > > > applicable law. If the
> > > reader
> > > > > of this message is not the intended recipient, you are hereby
> > > > > notified
> > > > that
> > > > > any printing, copying, dissemination, distribution, disclosure or
> > > > > forwarding of this communication is strictly prohibited. If you
> > > > > have received this communication in error, please contact the
> > > > > sender
> > > > immediately
> > > > > and delete it from your system. Thank You.
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Alejandro
> > > >
> > >
> > > --
> > > CONFIDENTIALITY NOTICE
> > > NOTICE: This message is intended for the use of the individual or
> > > entity to which it is addressed and may contain information that is
> > > confidential, privileged and exempt from disclosure under applicable
> > > law. If the reader of this message is not the intended recipient, you
> > > are hereby notified that any printing, copying, dissemination,
> > > distribution, disclosure or forwarding of this communication is
> > > strictly prohibited. If you have received this communication in error,
> > > please contact the sender immediately and delete it from your system.
> > Thank You.
> > >
> >
> >
> >
> > --
> > Alejandro
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

Re: Replacing the JSP web UIs to HTML 5 applications

[Haohui Mai <hm...@hortonworks.com>]

I would like to summarize the discussions so far. It seems that we have
reached two consensus:

1. The new JavaScript-based UI is the right direction to go.
2. For now we should keep the old JSP pages around for compatibility
reasons.

There're some debates on the usages of the JMX / JSON APIs, but this is
orthogonal to switching the UI, thus I consider it as a technical detail.
We can continue the discussions in the public jira.

The new UI has already landed in the trunk, based on the consensus it seems
that we can switch the default UI to the new one shortly. The user can
still access the old web UI using the same URLs.

The only question remain is that who is going to maintain the old web UI.
My answer is that we should leave them as deprecated and focus the effort
on the new web UI.

Thanks,
Haohui



On Tue, Oct 29, 2013 at 5:22 AM, Zheng, Kai <ka...@intel.com> wrote:

> > having /JMX for monitoring integration and a /JSON end point for the UI
> IMHO, this makes sense, especially for the long term. JMX interface serves
> as management console in admin perspective, WebUI serves as end user
> interface. Both might share same functionality codes, but that does not
> validate we couple them together.
>
> Thanks & regards,
> Kai
>
> -----Original Message-----
> From: Alejandro Abdelnur [mailto:tucu@cloudera.com]
> Sent: Tuesday, October 29, 2013 8:14 AM
> To: hdfs-dev@hadoop.apache.org
> Subject: Re: Replacing the JSP web UIs to HTML 5 applications
>
> Isn't using JMX to expose JSON for the web UI misusing JMX?
>
> I would think a more appropriate approach would be having /JMX for
> monitoring integration and a /JSON end point for the UI data.
>
> Thanks.
>
>
> On Mon, Oct 28, 2013 at 4:58 PM, Haohui Mai <hm...@hortonworks.com> wrote:
>
> > Alejandro,
> >
> > If I understand correctly, that is the exact approach that the new web
> > UI is taking. The new web UI takes the output from JMX and renders
> > them as HTML at the client side.
> >
> > ~Haohui
> >
> >
> > On Mon, Oct 28, 2013 at 4:18 PM, Alejandro Abdelnur <tucu@cloudera.com
> > >wrote:
> >
> > > Haohui,
> > >
> > > If you have NN and DNs producing JSON instead HTML, then you can
> > > build JS based web UIs. Take for example Oozie, Oozie produces JSON,
> > > it has a
> > built
> > > in JS web ui that consumes JSON and Hue has built an external web UI
> > > that also consumes JSON. In the case of Hue UI, Oozie didn't have to
> > > change anything to get that UI and improvements on the Hue UI don't
> > > require changes in Oozie unless it is to produce additional
> information.
> > >
> > > hope this clarifies.
> > >
> > > Thx
> > >
> > >
> > > On Mon, Oct 28, 2013 at 4:06 PM, Haohui Mai <hm...@hortonworks.com>
> > wrote:
> > >
> > > > Echo my comments on HDFS-5402:
> > > >
> > > > bq. If we're going to remove the old web UI, I think the new web
> > > > UI has to have the same level of unit testing. We shouldn't go
> > > > backwards in terms of unit testing.
> > > >
> > > > I take a look at TestNamenodeJspHelper / TestDatanodeJspHelper /
> > > > TestClusterJspHelper. It seems to me that we can merge these tests
> > > > with
> > > the
> > > > unit tests on JMX.
> > > >
> > > > bq. If we are going to
> > > > remove this capability, we need to add some other command-line
> > > > tools to get the same functionality. These tools could use REST if
> > > > we have that, or JMX, but they need to exist before we can
> > > > consider removing the old UI.
> > > >
> > > > This is a good point. Since all information are available through
> > > > JMX,
> > > the
> > > > easiest way to approach it is to write some scripts using Node.js.
> > > > The architecture of the new Web UIs is ready for this.
> > > >
> > > >
> > > > On Mon, Oct 28, 2013 at 3:57 PM, Alejandro Abdelnur
> > > > <tucu@cloudera.com
> > > > >wrote:
> > > >
> > > > > Producing JSON would be great. Agree with Colin that we should
> > > > > leave
> > > for
> > > > > now the current JSP based web ui.
> > > > >
> > > > > thx
> > > > >
> > > > >
> > > > > On Mon, Oct 28, 2013 at 11:16 AM, Colin McCabe <
> > cmccabe@alumni.cmu.edu
> > > > > >wrote:
> > > > >
> > > > > > This is a really interesting project, Haohui.  I think it will
> > > > > > make our web UI much nicer.
> > > > > >
> > > > > > I have a few concerns about removing the old web UI, however:
> > > > > >
> > > > > > * If we're going to remove the old web UI, I think the new web
> > > > > > UI
> > has
> > > > > > to have the same level of unit testing.  We shouldn't go
> > > > > > backwards
> > in
> > > > > > terms of unit testing.
> > > > > >
> > > > > > * Most of the deployments of elinks and links out there don't
> > support
> > > > > > Javascript.  This is just a reality of life when using CentOS
> > > > > > 5 or
> > 6,
> > > > > > which many users are still using.  I have used "links" to
> > > > > > diagnose problems through the web UI in the past, in systems
> > > > > > where access to the cluster was available only through telnet.
> > > > > > If we are going to remove this capability, we need to add some
> > > > > > other command-line
> > tools
> > > > > > to get the same functionality.  These tools could use REST if
> > > > > > we
> > have
> > > > > > that, or JMX, but they need to exist before we can consider
> > removing
> > > > > > the old UI.
> > > > > >
> > > > > > best,
> > > > > > Colin
> > > > > >
> > > > > > On Fri, Oct 25, 2013 at 7:31 PM, Haohui Mai
> > > > > > <hm...@hortonworks.com>
> > > > > wrote:
> > > > > > > Thanks for the reply, Luke. Here I just echo my response
> > > > > > > from the
> > > > jira:
> > > > > > >
> > > > > > > bq. this client-side js only approach, which is less secure
> > > > > > > than
> > a
> > > > > > > progressively enhanced hybrid approach used by YARN. The
> > > > > > > recent
> > > gmail
> > > > > > > XSS fiasco highlights the issue.
> > > > > > >
> > > > > > > I'm presenting an informal security analysis to compare the
> > > security
> > > > of
> > > > > > the
> > > > > > > old and the new web UIs.
> > > > > > >
> > > > > > > An attacker launches an XSS attack by injecting malicious
> > > > > > > code
> > > which
> > > > > are
> > > > > > > usually HTML or JavaScript fragments into the web page, so
> > > > > > > that
> > the
> > > > > > > malicious code can have the same privileges of the web page.
> > > > > > >
> > > > > > > First, in the scope of XSS attacks, note that the threat
> > > > > > > models
> > of
> > > > > > > launching XSS attacks on Internet sites Gmail/Linkedin and
> > > > > > > the
> > one
> > > of
> > > > > the
> > > > > > > Hadoop UIs are different. They have fundamental different
> > > > > > > sets of
> > > > > > external
> > > > > > > inputs that the attackers have control to. Internet sites
> > > > > > > have
> > > little
> > > > > > > control of these inputs. In the case of Gmail / Linkedin, an
> > attack
> > > > can
> > > > > > > send you a crafted e-mail, or put malicious description in
> > > > > > > his / her Linkedin profile. The sets of external inputs are
> > *restricted*
> > > in
> > > > > > > Hadoop UIs. The new web UIs take JMX and WebHDFS as inputs.
> > > > > > > The attacker has to launch a XSS attack by:
> > > > > > >
> > > > > > > * Compromise the jars so that the output of JMX / WebHDFS
> > > > > > > have
> > the
> > > > > > > malicious code.
> > > > > > > * Replace the web UIs completely to include the malicious code.
> > > > > > >
> > > > > > > In either case *the attacker has to compromise the hadoop
> > > > > > > core or
> > > the
> > > > > > > namenode*. That means the new web UIs are at least as secure
> > > > > > > as
> > the
> > > > > > hadoop
> > > > > > > core, and the namenode machine.
> > > > > > >
> > > > > > > Second, I argue that using client-side templates are more
> > > > > > > secure
> > > than
> > > > > the
> > > > > > > current JSP-based server-side templates. To defend against
> > > > > > > XSS attacks, both techniques have to filter the external
> > > > > > > inputs at
> > > > *every*
> > > > > > > possible execution paths. Several facts much be taken into
> > > > > > > plays when evaluating the security of both approaches in
> > real-world
> > > > > > > environments:
> > > > > > >
> > > > > > > * The JavaScript libraries used in the new web UIs have
> > > > > > > survived
> > in
> > > > > > > extremely large-scale production tests. jQuery is used by
> > > > > > > Google
> > > and
> > > > > > >  Microsoft, bootstrap is used by Twitter, and dust.js is
> > > > > > > used by
> > > > > > Linkedin.
> > > > > > > All libraries survived from hundreds of thousands of  attack
> > > > > > > attempts on a daily basis. I agree that the libraries
> > might
> > > > > still
> > > > > > > be imperfect, but there's no way that we can test the JSP
> > > > > > > web  UIs to achieve the same level of assurances given the
> > > > > > > amount of
> > > > > > resources
> > > > > > > the community has.
> > > > > > > * Client-side templates consolidate all filtering logic in
> > > > > > > one
> > > > central
> > > > > > > place. Recall that the goal is to filter all external inputs
> > > > > > > at
> > > every
> > > > > > >  execution paths, this is a much more systematic approach
> > compared
> > > to
> > > > > the
> > > > > > > server-side templates we have today. It is difficult (if not
> > > > > > >  impossible) to do it in a JSP/ASP/PHP application, since
> > > > > > > such
> > > > > filtering
> > > > > > > can be only achieved via ad-hoc approaches ([1] shows some
> > > > > > > empirical data). Also, HDFS-4901 recently describes a XSS
> > > > > vulnerability
> > > > > > in
> > > > > > > browseDirectory.jsp.
> > > > > > >
> > > > > > > bq. You'd require proper SSL (not self signed) setup to
> > > > > > > avoid JS injection
> > > > > > >
> > > > > > > Commodity browsers enforce Same-Origin Policy to defend
> > > > > > > against
> > > code
> > > > > > > injections. It has nothing to do with what kinds of SSL
> > > certificates
> > > > > > > you hold.
> > > > > > >
> > > > > > > bq.  I also have concerns that we commit these changes
> > > > > > > without
> > > > matching
> > > > > > > unit tests
> > > > > > >
> > > > > > > The JavaScript code can be automatically tested. The same
> > > > > > > code
> > can
> > > be
> > > > > run
> > > > > > > by node.js and the test can compared with pre-defined
> > > > > > > results. It is also possible to write an adapter to use
> > > > > > > Rhino to
> > > > > > accomplish
> > > > > > > the same task. We can discuss how to integrate them into the
> > > > > > > maven test routines in a different thread.
> > > > > > >
> > > > > > > bq. Client side rendering completely breaks the workflows
> > > > > > > for ops
> > > who
> > > > > > rely
> > > > > > > on text based terminal/emacs/vim browsers (no js support) to
> > > > > > > monitor component UI.
> > > > > > >
> > > > > > > links / elinks (http://elinks.or.cz/) are text-based web
> > browsers
> > > > that
> > > > > > > support JavaScript.
> > > > > > >
> > > > > > > bq. The priority/requirements for UI in core Hadoop should
> > > > > > > be
> > > > security
> > > > > > and
> > > > > > > correctness, which client side templating cannot address
> > > > > > > properly so far.
> > > > > > >
> > > > > > > I agree that we should focus on security and correctness.
> > > > > > > The
> > > > > paragraphs
> > > > > > > above explain that how the architecture of the new UIs makes
> > > > > > > the UIs more secure in real-world settings compared to the
> > UI
> > > > we
> > > > > > have
> > > > > > > today.
> > > > > > >
> > > > > > > References:
> > > > > > >
> > > > > > > 1. A. Yip et al. Improving Application Security with Data
> > > > > > > Flow
> > > > > > Assertions.
> > > > > > > In SOSP'2009.
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Oct 25, 2013 at 10:02 AM, Luke Lu <ll...@apache.org>
> > wrote:
> > > > > > >
> > > > > > >> Echoing my comments on HDFS-3555:
> > > > > > >>
> > > > > > >> I have concerns with this client-side js only approach,
> > > > > > >> which is
> > > > less
> > > > > > >> secure than a progressively enhanced hybrid approach used
> > > > > > >> by
> > YARN.
> > > > The
> > > > > > >> recent gmail XSS fiasco highlights the issue. I also have
> > concerns
> > > > > that
> > > > > > we
> > > > > > >> commit these changes without matching unit tests - the fact
> > > > > > >> you
> > > > cannot
> > > > > > >> effectively unit test these changes should tell you
> > > > > > >> something
> > > about
> > > > > this
> > > > > > >> approach.
> > > > > > >>
> > > > > > >> *Requiring* JS means that an admin cannot turn off js to
> > > (partially)
> > > > > use
> > > > > > >> core Hadoop UI. You'd *require* proper SSL (not self
> > > > > > >> signed)
> > setup
> > > > to
> > > > > > avoid
> > > > > > >> JS injection, even if security of js libraries used is
> > > > > > >> perfect,
> > > > which
> > > > > I
> > > > > > >> doubt (search gmail/linkedin XSS). Client side rendering
> > > completely
> > > > > > breaks
> > > > > > >> the workflows for ops who rely on text based
> > > > > > >> terminal/emacs/vim
> > > > > browsers
> > > > > > >> (no js support) to monitor component UI.
> > > > > > >>
> > > > > > >> IMO, JS-only rendering belongs to social networking sites
> > > > > > >> and/or
> > > > SaaS
> > > > > > >> front-ends, where full time UI/security specialists
> > > > > > >> babysits UI
> > > > > > changes. I
> > > > > > >> think eventually most users will use a self servicing UI in
> > > > > > >> a
> > SaaS
> > > > > > >> front-end that uses REST/JMX API to get data from back-end
> > > > components,
> > > > > > >> besides their own app master/service UI. The
> > priority/requirements
> > > > for
> > > > > > UI
> > > > > > >> in core Hadoop should be security and correctness, which
> > > > > > >> client
> > > side
> > > > > > >> templating cannot address properly so far.
> > > > > > >>
> > > > > > >>
> > > > > > >> On Tue, Oct 22, 2013 at 3:59 PM, Haohui Mai <
> > hmai@hortonworks.com
> > > >
> > > > > > wrote:
> > > > > > >>
> > > > > > >> > Hi all,
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > Jing Zhao and I recently have reimplemented the JSP-based
> > > > > > >> > web
> > > UIs
> > > > in
> > > > > > >> HTML 5
> > > > > > >> > applications (HDFS-5333). Based on our prelimanary
> > > > > > >> > testing
> > > results
> > > > > we
> > > > > > >> > believe thst the new web UIs of the namenodes and the
> > > > > > >> > datanode
> > > are
> > > > > > ready
> > > > > > >> > for everyday uses.
> > > > > > >> >
> > > > > > >> > You're more than welcome to try it out on trunk by
> > > > > > >> > visiting
> > > > http://
> > > > > > >> > <namenode>/dfshealth.html
> > > > > > >> >
> > > > > > >> > There are a number of benefits from this transition. From
> > > > > > >> > a
> > > > > > developer's
> > > > > > >> > prospective, the most notable one is *maintainability*:
> > > > > > >> >
> > > > > > >> > (1) The abstractions between the UI and the core server
> > > > > > >> > are
> > > > > > well-defined,
> > > > > > >> > decoupling the UI and the core hadoop servers.
> > > > > > >> >
> > > > > > >> > (2) It allows us to deprecate the logic in the JSP pages.
> > > > > > >> > The
> > > old
> > > > > web
> > > > > > UIs
> > > > > > >> > have to duplicate the logic in the JSPs. The logic is
> > > > > > >> > often
> > > > > > out-of-dated
> > > > > > >> > and not well-tested, which leads to broken pages and
> > > > > > >> > security vulnerabilities(e.g. HDFS-5251, HDFS-5307,
> > > > > > >> > HDFS-5308,
> > HDFS-5317
> > > > and
> > > > > > >> > HDFS-4901). The architecture of the new UIs prevent these
> > > > > > >> > bugs
> > > at
> > > > > the
> > > > > > >> very
> > > > > > >> > beginning.
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > I propose that deprecate the old, JSP-based web UIs in
> > > > > > >> > 2.3. I
> > > > opened
> > > > > > >> > HDFS-5402 to track the relevant discussions.
> > > > > > >> >
> > > > > > >> > Your feedbacks are highly appreciated.
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > Sincerely,
> > > > > > >> >
> > > > > > >> > Haohui
> > > > > > >> >
> > > > > > >> > --
> > > > > > >> > CONFIDENTIALITY NOTICE
> > > > > > >> > NOTICE: This message is intended for the use of the
> > > > > > >> > individual
> > > or
> > > > > > entity
> > > > > > >> to
> > > > > > >> > which it is addressed and may contain information that is
> > > > > > confidential,
> > > > > > >> > privileged and exempt from disclosure under applicable
> > > > > > >> > law. If
> > > the
> > > > > > reader
> > > > > > >> > of this message is not the intended recipient, you are
> > > > > > >> > hereby
> > > > > notified
> > > > > > >> that
> > > > > > >> > any printing, copying, dissemination, distribution,
> > > > > > >> > disclosure
> > > or
> > > > > > >> > forwarding of this communication is strictly prohibited.
> > > > > > >> > If
> > you
> > > > have
> > > > > > >> > received this communication in error, please contact the
> > sender
> > > > > > >> immediately
> > > > > > >> > and delete it from your system. Thank You.
> > > > > > >> >
> > > > > > >>
> > > > > > >
> > > > > > > --
> > > > > > > CONFIDENTIALITY NOTICE
> > > > > > > NOTICE: This message is intended for the use of the
> > > > > > > individual or
> > > > > entity
> > > > > > to
> > > > > > > which it is addressed and may contain information that is
> > > > confidential,
> > > > > > > privileged and exempt from disclosure under applicable law.
> > > > > > > If
> > the
> > > > > reader
> > > > > > > of this message is not the intended recipient, you are
> > > > > > > hereby
> > > > notified
> > > > > > that
> > > > > > > any printing, copying, dissemination, distribution,
> > > > > > > disclosure or forwarding of this communication is strictly
> > > > > > > prohibited. If you
> > > have
> > > > > > > received this communication in error, please contact the
> > > > > > > sender
> > > > > > immediately
> > > > > > > and delete it from your system. Thank You.
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Alejandro
> > > > >
> > > >
> > > > --
> > > > CONFIDENTIALITY NOTICE
> > > > NOTICE: This message is intended for the use of the individual or
> > entity
> > > to
> > > > which it is addressed and may contain information that is
> > > > confidential, privileged and exempt from disclosure under
> > > > applicable law. If the
> > reader
> > > > of this message is not the intended recipient, you are hereby
> > > > notified
> > > that
> > > > any printing, copying, dissemination, distribution, disclosure or
> > > > forwarding of this communication is strictly prohibited. If you
> > > > have received this communication in error, please contact the
> > > > sender
> > > immediately
> > > > and delete it from your system. Thank You.
> > > >
> > >
> > >
> > >
> > > --
> > > Alejandro
> > >
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or
> > entity to which it is addressed and may contain information that is
> > confidential, privileged and exempt from disclosure under applicable
> > law. If the reader of this message is not the intended recipient, you
> > are hereby notified that any printing, copying, dissemination,
> > distribution, disclosure or forwarding of this communication is
> > strictly prohibited. If you have received this communication in error,
> > please contact the sender immediately and delete it from your system.
> Thank You.
> >
>
>
>
> --
> Alejandro
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

RE: Replacing the JSP web UIs to HTML 5 applications

["Zheng, Kai" <ka...@intel.com>]

> having /JMX for monitoring integration and a /JSON end point for the UI
IMHO, this makes sense, especially for the long term. JMX interface serves as management console in admin perspective, WebUI serves as end user interface. Both might share same functionality codes, but that does not validate we couple them together.

Thanks & regards,
Kai

-----Original Message-----
From: Alejandro Abdelnur [mailto:tucu@cloudera.com] 
Sent: Tuesday, October 29, 2013 8:14 AM
To: hdfs-dev@hadoop.apache.org
Subject: Re: Replacing the JSP web UIs to HTML 5 applications

Isn't using JMX to expose JSON for the web UI misusing JMX?

I would think a more appropriate approach would be having /JMX for monitoring integration and a /JSON end point for the UI data.

Thanks.


On Mon, Oct 28, 2013 at 4:58 PM, Haohui Mai <hm...@hortonworks.com> wrote:

> Alejandro,
>
> If I understand correctly, that is the exact approach that the new web 
> UI is taking. The new web UI takes the output from JMX and renders 
> them as HTML at the client side.
>
> ~Haohui
>
>
> On Mon, Oct 28, 2013 at 4:18 PM, Alejandro Abdelnur <tucu@cloudera.com
> >wrote:
>
> > Haohui,
> >
> > If you have NN and DNs producing JSON instead HTML, then you can 
> > build JS based web UIs. Take for example Oozie, Oozie produces JSON, 
> > it has a
> built
> > in JS web ui that consumes JSON and Hue has built an external web UI 
> > that also consumes JSON. In the case of Hue UI, Oozie didn't have to 
> > change anything to get that UI and improvements on the Hue UI don't 
> > require changes in Oozie unless it is to produce additional information.
> >
> > hope this clarifies.
> >
> > Thx
> >
> >
> > On Mon, Oct 28, 2013 at 4:06 PM, Haohui Mai <hm...@hortonworks.com>
> wrote:
> >
> > > Echo my comments on HDFS-5402:
> > >
> > > bq. If we're going to remove the old web UI, I think the new web 
> > > UI has to have the same level of unit testing. We shouldn't go 
> > > backwards in terms of unit testing.
> > >
> > > I take a look at TestNamenodeJspHelper / TestDatanodeJspHelper / 
> > > TestClusterJspHelper. It seems to me that we can merge these tests 
> > > with
> > the
> > > unit tests on JMX.
> > >
> > > bq. If we are going to
> > > remove this capability, we need to add some other command-line 
> > > tools to get the same functionality. These tools could use REST if 
> > > we have that, or JMX, but they need to exist before we can 
> > > consider removing the old UI.
> > >
> > > This is a good point. Since all information are available through 
> > > JMX,
> > the
> > > easiest way to approach it is to write some scripts using Node.js. 
> > > The architecture of the new Web UIs is ready for this.
> > >
> > >
> > > On Mon, Oct 28, 2013 at 3:57 PM, Alejandro Abdelnur 
> > > <tucu@cloudera.com
> > > >wrote:
> > >
> > > > Producing JSON would be great. Agree with Colin that we should 
> > > > leave
> > for
> > > > now the current JSP based web ui.
> > > >
> > > > thx
> > > >
> > > >
> > > > On Mon, Oct 28, 2013 at 11:16 AM, Colin McCabe <
> cmccabe@alumni.cmu.edu
> > > > >wrote:
> > > >
> > > > > This is a really interesting project, Haohui.  I think it will 
> > > > > make our web UI much nicer.
> > > > >
> > > > > I have a few concerns about removing the old web UI, however:
> > > > >
> > > > > * If we're going to remove the old web UI, I think the new web 
> > > > > UI
> has
> > > > > to have the same level of unit testing.  We shouldn't go 
> > > > > backwards
> in
> > > > > terms of unit testing.
> > > > >
> > > > > * Most of the deployments of elinks and links out there don't
> support
> > > > > Javascript.  This is just a reality of life when using CentOS 
> > > > > 5 or
> 6,
> > > > > which many users are still using.  I have used "links" to 
> > > > > diagnose problems through the web UI in the past, in systems 
> > > > > where access to the cluster was available only through telnet.  
> > > > > If we are going to remove this capability, we need to add some 
> > > > > other command-line
> tools
> > > > > to get the same functionality.  These tools could use REST if 
> > > > > we
> have
> > > > > that, or JMX, but they need to exist before we can consider
> removing
> > > > > the old UI.
> > > > >
> > > > > best,
> > > > > Colin
> > > > >
> > > > > On Fri, Oct 25, 2013 at 7:31 PM, Haohui Mai 
> > > > > <hm...@hortonworks.com>
> > > > wrote:
> > > > > > Thanks for the reply, Luke. Here I just echo my response 
> > > > > > from the
> > > jira:
> > > > > >
> > > > > > bq. this client-side js only approach, which is less secure 
> > > > > > than
> a
> > > > > > progressively enhanced hybrid approach used by YARN. The 
> > > > > > recent
> > gmail
> > > > > > XSS fiasco highlights the issue.
> > > > > >
> > > > > > I'm presenting an informal security analysis to compare the
> > security
> > > of
> > > > > the
> > > > > > old and the new web UIs.
> > > > > >
> > > > > > An attacker launches an XSS attack by injecting malicious 
> > > > > > code
> > which
> > > > are
> > > > > > usually HTML or JavaScript fragments into the web page, so 
> > > > > > that
> the
> > > > > > malicious code can have the same privileges of the web page.
> > > > > >
> > > > > > First, in the scope of XSS attacks, note that the threat 
> > > > > > models
> of
> > > > > > launching XSS attacks on Internet sites Gmail/Linkedin and 
> > > > > > the
> one
> > of
> > > > the
> > > > > > Hadoop UIs are different. They have fundamental different 
> > > > > > sets of
> > > > > external
> > > > > > inputs that the attackers have control to. Internet sites 
> > > > > > have
> > little
> > > > > > control of these inputs. In the case of Gmail / Linkedin, an
> attack
> > > can
> > > > > > send you a crafted e-mail, or put malicious description in 
> > > > > > his / her Linkedin profile. The sets of external inputs are
> *restricted*
> > in
> > > > > > Hadoop UIs. The new web UIs take JMX and WebHDFS as inputs. 
> > > > > > The attacker has to launch a XSS attack by:
> > > > > >
> > > > > > * Compromise the jars so that the output of JMX / WebHDFS 
> > > > > > have
> the
> > > > > > malicious code.
> > > > > > * Replace the web UIs completely to include the malicious code.
> > > > > >
> > > > > > In either case *the attacker has to compromise the hadoop 
> > > > > > core or
> > the
> > > > > > namenode*. That means the new web UIs are at least as secure 
> > > > > > as
> the
> > > > > hadoop
> > > > > > core, and the namenode machine.
> > > > > >
> > > > > > Second, I argue that using client-side templates are more 
> > > > > > secure
> > than
> > > > the
> > > > > > current JSP-based server-side templates. To defend against 
> > > > > > XSS attacks, both techniques have to filter the external 
> > > > > > inputs at
> > > *every*
> > > > > > possible execution paths. Several facts much be taken into 
> > > > > > plays when evaluating the security of both approaches in
> real-world
> > > > > > environments:
> > > > > >
> > > > > > * The JavaScript libraries used in the new web UIs have 
> > > > > > survived
> in
> > > > > > extremely large-scale production tests. jQuery is used by 
> > > > > > Google
> > and
> > > > > >  Microsoft, bootstrap is used by Twitter, and dust.js is 
> > > > > > used by
> > > > > Linkedin.
> > > > > > All libraries survived from hundreds of thousands of  attack 
> > > > > > attempts on a daily basis. I agree that the libraries
> might
> > > > still
> > > > > > be imperfect, but there's no way that we can test the JSP 
> > > > > > web  UIs to achieve the same level of assurances given the 
> > > > > > amount of
> > > > > resources
> > > > > > the community has.
> > > > > > * Client-side templates consolidate all filtering logic in 
> > > > > > one
> > > central
> > > > > > place. Recall that the goal is to filter all external inputs 
> > > > > > at
> > every
> > > > > >  execution paths, this is a much more systematic approach
> compared
> > to
> > > > the
> > > > > > server-side templates we have today. It is difficult (if not
> > > > > >  impossible) to do it in a JSP/ASP/PHP application, since 
> > > > > > such
> > > > filtering
> > > > > > can be only achieved via ad-hoc approaches ([1] shows some  
> > > > > > empirical data). Also, HDFS-4901 recently describes a XSS
> > > > vulnerability
> > > > > in
> > > > > > browseDirectory.jsp.
> > > > > >
> > > > > > bq. You'd require proper SSL (not self signed) setup to 
> > > > > > avoid JS injection
> > > > > >
> > > > > > Commodity browsers enforce Same-Origin Policy to defend 
> > > > > > against
> > code
> > > > > > injections. It has nothing to do with what kinds of SSL
> > certificates
> > > > > > you hold.
> > > > > >
> > > > > > bq.  I also have concerns that we commit these changes 
> > > > > > without
> > > matching
> > > > > > unit tests
> > > > > >
> > > > > > The JavaScript code can be automatically tested. The same 
> > > > > > code
> can
> > be
> > > > run
> > > > > > by node.js and the test can compared with pre-defined 
> > > > > > results. It is also possible to write an adapter to use 
> > > > > > Rhino to
> > > > > accomplish
> > > > > > the same task. We can discuss how to integrate them into the 
> > > > > > maven test routines in a different thread.
> > > > > >
> > > > > > bq. Client side rendering completely breaks the workflows 
> > > > > > for ops
> > who
> > > > > rely
> > > > > > on text based terminal/emacs/vim browsers (no js support) to 
> > > > > > monitor component UI.
> > > > > >
> > > > > > links / elinks (http://elinks.or.cz/) are text-based web
> browsers
> > > that
> > > > > > support JavaScript.
> > > > > >
> > > > > > bq. The priority/requirements for UI in core Hadoop should 
> > > > > > be
> > > security
> > > > > and
> > > > > > correctness, which client side templating cannot address 
> > > > > > properly so far.
> > > > > >
> > > > > > I agree that we should focus on security and correctness. 
> > > > > > The
> > > > paragraphs
> > > > > > above explain that how the architecture of the new UIs makes 
> > > > > > the UIs more secure in real-world settings compared to the
> UI
> > > we
> > > > > have
> > > > > > today.
> > > > > >
> > > > > > References:
> > > > > >
> > > > > > 1. A. Yip et al. Improving Application Security with Data 
> > > > > > Flow
> > > > > Assertions.
> > > > > > In SOSP'2009.
> > > > > >
> > > > > >
> > > > > > On Fri, Oct 25, 2013 at 10:02 AM, Luke Lu <ll...@apache.org>
> wrote:
> > > > > >
> > > > > >> Echoing my comments on HDFS-3555:
> > > > > >>
> > > > > >> I have concerns with this client-side js only approach, 
> > > > > >> which is
> > > less
> > > > > >> secure than a progressively enhanced hybrid approach used 
> > > > > >> by
> YARN.
> > > The
> > > > > >> recent gmail XSS fiasco highlights the issue. I also have
> concerns
> > > > that
> > > > > we
> > > > > >> commit these changes without matching unit tests - the fact 
> > > > > >> you
> > > cannot
> > > > > >> effectively unit test these changes should tell you 
> > > > > >> something
> > about
> > > > this
> > > > > >> approach.
> > > > > >>
> > > > > >> *Requiring* JS means that an admin cannot turn off js to
> > (partially)
> > > > use
> > > > > >> core Hadoop UI. You'd *require* proper SSL (not self 
> > > > > >> signed)
> setup
> > > to
> > > > > avoid
> > > > > >> JS injection, even if security of js libraries used is 
> > > > > >> perfect,
> > > which
> > > > I
> > > > > >> doubt (search gmail/linkedin XSS). Client side rendering
> > completely
> > > > > breaks
> > > > > >> the workflows for ops who rely on text based 
> > > > > >> terminal/emacs/vim
> > > > browsers
> > > > > >> (no js support) to monitor component UI.
> > > > > >>
> > > > > >> IMO, JS-only rendering belongs to social networking sites 
> > > > > >> and/or
> > > SaaS
> > > > > >> front-ends, where full time UI/security specialists 
> > > > > >> babysits UI
> > > > > changes. I
> > > > > >> think eventually most users will use a self servicing UI in 
> > > > > >> a
> SaaS
> > > > > >> front-end that uses REST/JMX API to get data from back-end
> > > components,
> > > > > >> besides their own app master/service UI. The
> priority/requirements
> > > for
> > > > > UI
> > > > > >> in core Hadoop should be security and correctness, which 
> > > > > >> client
> > side
> > > > > >> templating cannot address properly so far.
> > > > > >>
> > > > > >>
> > > > > >> On Tue, Oct 22, 2013 at 3:59 PM, Haohui Mai <
> hmai@hortonworks.com
> > >
> > > > > wrote:
> > > > > >>
> > > > > >> > Hi all,
> > > > > >> >
> > > > > >> >
> > > > > >> > Jing Zhao and I recently have reimplemented the JSP-based 
> > > > > >> > web
> > UIs
> > > in
> > > > > >> HTML 5
> > > > > >> > applications (HDFS-5333). Based on our prelimanary 
> > > > > >> > testing
> > results
> > > > we
> > > > > >> > believe thst the new web UIs of the namenodes and the 
> > > > > >> > datanode
> > are
> > > > > ready
> > > > > >> > for everyday uses.
> > > > > >> >
> > > > > >> > You're more than welcome to try it out on trunk by 
> > > > > >> > visiting
> > > http://
> > > > > >> > <namenode>/dfshealth.html
> > > > > >> >
> > > > > >> > There are a number of benefits from this transition. From 
> > > > > >> > a
> > > > > developer's
> > > > > >> > prospective, the most notable one is *maintainability*:
> > > > > >> >
> > > > > >> > (1) The abstractions between the UI and the core server 
> > > > > >> > are
> > > > > well-defined,
> > > > > >> > decoupling the UI and the core hadoop servers.
> > > > > >> >
> > > > > >> > (2) It allows us to deprecate the logic in the JSP pages. 
> > > > > >> > The
> > old
> > > > web
> > > > > UIs
> > > > > >> > have to duplicate the logic in the JSPs. The logic is 
> > > > > >> > often
> > > > > out-of-dated
> > > > > >> > and not well-tested, which leads to broken pages and 
> > > > > >> > security vulnerabilities(e.g. HDFS-5251, HDFS-5307, 
> > > > > >> > HDFS-5308,
> HDFS-5317
> > > and
> > > > > >> > HDFS-4901). The architecture of the new UIs prevent these 
> > > > > >> > bugs
> > at
> > > > the
> > > > > >> very
> > > > > >> > beginning.
> > > > > >> >
> > > > > >> >
> > > > > >> > I propose that deprecate the old, JSP-based web UIs in 
> > > > > >> > 2.3. I
> > > opened
> > > > > >> > HDFS-5402 to track the relevant discussions.
> > > > > >> >
> > > > > >> > Your feedbacks are highly appreciated.
> > > > > >> >
> > > > > >> >
> > > > > >> > Sincerely,
> > > > > >> >
> > > > > >> > Haohui
> > > > > >> >
> > > > > >> > --
> > > > > >> > CONFIDENTIALITY NOTICE
> > > > > >> > NOTICE: This message is intended for the use of the 
> > > > > >> > individual
> > or
> > > > > entity
> > > > > >> to
> > > > > >> > which it is addressed and may contain information that is
> > > > > confidential,
> > > > > >> > privileged and exempt from disclosure under applicable 
> > > > > >> > law. If
> > the
> > > > > reader
> > > > > >> > of this message is not the intended recipient, you are 
> > > > > >> > hereby
> > > > notified
> > > > > >> that
> > > > > >> > any printing, copying, dissemination, distribution, 
> > > > > >> > disclosure
> > or
> > > > > >> > forwarding of this communication is strictly prohibited. 
> > > > > >> > If
> you
> > > have
> > > > > >> > received this communication in error, please contact the
> sender
> > > > > >> immediately
> > > > > >> > and delete it from your system. Thank You.
> > > > > >> >
> > > > > >>
> > > > > >
> > > > > > --
> > > > > > CONFIDENTIALITY NOTICE
> > > > > > NOTICE: This message is intended for the use of the 
> > > > > > individual or
> > > > entity
> > > > > to
> > > > > > which it is addressed and may contain information that is
> > > confidential,
> > > > > > privileged and exempt from disclosure under applicable law. 
> > > > > > If
> the
> > > > reader
> > > > > > of this message is not the intended recipient, you are 
> > > > > > hereby
> > > notified
> > > > > that
> > > > > > any printing, copying, dissemination, distribution, 
> > > > > > disclosure or forwarding of this communication is strictly 
> > > > > > prohibited. If you
> > have
> > > > > > received this communication in error, please contact the 
> > > > > > sender
> > > > > immediately
> > > > > > and delete it from your system. Thank You.
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Alejandro
> > > >
> > >
> > > --
> > > CONFIDENTIALITY NOTICE
> > > NOTICE: This message is intended for the use of the individual or
> entity
> > to
> > > which it is addressed and may contain information that is 
> > > confidential, privileged and exempt from disclosure under 
> > > applicable law. If the
> reader
> > > of this message is not the intended recipient, you are hereby 
> > > notified
> > that
> > > any printing, copying, dissemination, distribution, disclosure or 
> > > forwarding of this communication is strictly prohibited. If you 
> > > have received this communication in error, please contact the 
> > > sender
> > immediately
> > > and delete it from your system. Thank You.
> > >
> >
> >
> >
> > --
> > Alejandro
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or 
> entity to which it is addressed and may contain information that is 
> confidential, privileged and exempt from disclosure under applicable 
> law. If the reader of this message is not the intended recipient, you 
> are hereby notified that any printing, copying, dissemination, 
> distribution, disclosure or forwarding of this communication is 
> strictly prohibited. If you have received this communication in error, 
> please contact the sender immediately and delete it from your system. Thank You.
>



--
Alejandro

Re: Replacing the JSP web UIs to HTML 5 applications

[Alejandro Abdelnur <tu...@cloudera.com>]

Isn't using JMX to expose JSON for the web UI misusing JMX?

I would think a more appropriate approach would be having /JMX for
monitoring integration and a /JSON end point for the UI data.

Thanks.


On Mon, Oct 28, 2013 at 4:58 PM, Haohui Mai <hm...@hortonworks.com> wrote:

> Alejandro,
>
> If I understand correctly, that is the exact approach that the new web UI
> is taking. The new web UI takes the output from JMX and renders them as
> HTML at the client side.
>
> ~Haohui
>
>
> On Mon, Oct 28, 2013 at 4:18 PM, Alejandro Abdelnur <tucu@cloudera.com
> >wrote:
>
> > Haohui,
> >
> > If you have NN and DNs producing JSON instead HTML, then you can build JS
> > based web UIs. Take for example Oozie, Oozie produces JSON, it has a
> built
> > in JS web ui that consumes JSON and Hue has built an external web UI that
> > also consumes JSON. In the case of Hue UI, Oozie didn't have to change
> > anything to get that UI and improvements on the Hue UI don't require
> > changes in Oozie unless it is to produce additional information.
> >
> > hope this clarifies.
> >
> > Thx
> >
> >
> > On Mon, Oct 28, 2013 at 4:06 PM, Haohui Mai <hm...@hortonworks.com>
> wrote:
> >
> > > Echo my comments on HDFS-5402:
> > >
> > > bq. If we're going to remove the old web UI, I think the new web UI has
> > > to have the same level of unit testing. We shouldn't go backwards in
> > > terms of unit testing.
> > >
> > > I take a look at TestNamenodeJspHelper / TestDatanodeJspHelper /
> > > TestClusterJspHelper. It seems to me that we can merge these tests with
> > the
> > > unit tests on JMX.
> > >
> > > bq. If we are going to
> > > remove this capability, we need to add some other command-line tools
> > > to get the same functionality. These tools could use REST if we have
> > > that, or JMX, but they need to exist before we can consider removing
> > > the old UI.
> > >
> > > This is a good point. Since all information are available through JMX,
> > the
> > > easiest way to approach it is to write some scripts using Node.js. The
> > > architecture of the new Web UIs is ready for this.
> > >
> > >
> > > On Mon, Oct 28, 2013 at 3:57 PM, Alejandro Abdelnur <tucu@cloudera.com
> > > >wrote:
> > >
> > > > Producing JSON would be great. Agree with Colin that we should leave
> > for
> > > > now the current JSP based web ui.
> > > >
> > > > thx
> > > >
> > > >
> > > > On Mon, Oct 28, 2013 at 11:16 AM, Colin McCabe <
> cmccabe@alumni.cmu.edu
> > > > >wrote:
> > > >
> > > > > This is a really interesting project, Haohui.  I think it will make
> > > > > our web UI much nicer.
> > > > >
> > > > > I have a few concerns about removing the old web UI, however:
> > > > >
> > > > > * If we're going to remove the old web UI, I think the new web UI
> has
> > > > > to have the same level of unit testing.  We shouldn't go backwards
> in
> > > > > terms of unit testing.
> > > > >
> > > > > * Most of the deployments of elinks and links out there don't
> support
> > > > > Javascript.  This is just a reality of life when using CentOS 5 or
> 6,
> > > > > which many users are still using.  I have used "links" to diagnose
> > > > > problems through the web UI in the past, in systems where access to
> > > > > the cluster was available only through telnet.  If we are going to
> > > > > remove this capability, we need to add some other command-line
> tools
> > > > > to get the same functionality.  These tools could use REST if we
> have
> > > > > that, or JMX, but they need to exist before we can consider
> removing
> > > > > the old UI.
> > > > >
> > > > > best,
> > > > > Colin
> > > > >
> > > > > On Fri, Oct 25, 2013 at 7:31 PM, Haohui Mai <hm...@hortonworks.com>
> > > > wrote:
> > > > > > Thanks for the reply, Luke. Here I just echo my response from the
> > > jira:
> > > > > >
> > > > > > bq. this client-side js only approach, which is less secure than
> a
> > > > > > progressively enhanced hybrid approach used by YARN. The recent
> > gmail
> > > > > > XSS fiasco highlights the issue.
> > > > > >
> > > > > > I'm presenting an informal security analysis to compare the
> > security
> > > of
> > > > > the
> > > > > > old and the new web UIs.
> > > > > >
> > > > > > An attacker launches an XSS attack by injecting malicious code
> > which
> > > > are
> > > > > > usually HTML or JavaScript fragments into the web page, so that
> the
> > > > > > malicious code can have the same privileges of the web page.
> > > > > >
> > > > > > First, in the scope of XSS attacks, note that the threat models
> of
> > > > > > launching XSS attacks on Internet sites Gmail/Linkedin and the
> one
> > of
> > > > the
> > > > > > Hadoop UIs are different. They have fundamental different sets of
> > > > > external
> > > > > > inputs that the attackers have control to. Internet sites have
> > little
> > > > > > control of these inputs. In the case of Gmail / Linkedin, an
> attack
> > > can
> > > > > > send you a crafted e-mail, or put malicious description in his /
> > > > > > her Linkedin profile. The sets of external inputs are
> *restricted*
> > in
> > > > > > Hadoop UIs. The new web UIs take JMX and WebHDFS as inputs. The
> > > > > > attacker has to launch a XSS attack by:
> > > > > >
> > > > > > * Compromise the jars so that the output of JMX / WebHDFS have
> the
> > > > > > malicious code.
> > > > > > * Replace the web UIs completely to include the malicious code.
> > > > > >
> > > > > > In either case *the attacker has to compromise the hadoop core or
> > the
> > > > > > namenode*. That means the new web UIs are at least as secure as
> the
> > > > > hadoop
> > > > > > core, and the namenode machine.
> > > > > >
> > > > > > Second, I argue that using client-side templates are more secure
> > than
> > > > the
> > > > > > current JSP-based server-side templates. To defend against XSS
> > > > > > attacks, both techniques have to filter the external inputs at
> > > *every*
> > > > > > possible execution paths. Several facts much be taken into
> > > > > > plays when evaluating the security of both approaches in
> real-world
> > > > > > environments:
> > > > > >
> > > > > > * The JavaScript libraries used in the new web UIs have survived
> in
> > > > > > extremely large-scale production tests. jQuery is used by Google
> > and
> > > > > >  Microsoft, bootstrap is used by Twitter, and dust.js is used by
> > > > > Linkedin.
> > > > > > All libraries survived from hundreds of thousands of
> > > > > >  attack attempts on a daily basis. I agree that the libraries
> might
> > > > still
> > > > > > be imperfect, but there's no way that we can test the JSP web
> > > > > >  UIs to achieve the same level of assurances given the amount of
> > > > > resources
> > > > > > the community has.
> > > > > > * Client-side templates consolidate all filtering logic in one
> > > central
> > > > > > place. Recall that the goal is to filter all external inputs at
> > every
> > > > > >  execution paths, this is a much more systematic approach
> compared
> > to
> > > > the
> > > > > > server-side templates we have today. It is difficult (if not
> > > > > >  impossible) to do it in a JSP/ASP/PHP application, since such
> > > > filtering
> > > > > > can be only achieved via ad-hoc approaches ([1] shows some
> > > > > >  empirical data). Also, HDFS-4901 recently describes a XSS
> > > > vulnerability
> > > > > in
> > > > > > browseDirectory.jsp.
> > > > > >
> > > > > > bq. You'd require proper SSL (not self signed) setup to avoid JS
> > > > > > injection
> > > > > >
> > > > > > Commodity browsers enforce Same-Origin Policy to defend against
> > code
> > > > > > injections. It has nothing to do with what kinds of SSL
> > certificates
> > > > > > you hold.
> > > > > >
> > > > > > bq.  I also have concerns that we commit these changes without
> > > matching
> > > > > > unit tests
> > > > > >
> > > > > > The JavaScript code can be automatically tested. The same code
> can
> > be
> > > > run
> > > > > > by node.js and the test can compared with pre-defined
> > > > > > results. It is also possible to write an adapter to use Rhino to
> > > > > accomplish
> > > > > > the same task. We can discuss how to integrate them into
> > > > > > the maven test routines in a different thread.
> > > > > >
> > > > > > bq. Client side rendering completely breaks the workflows for ops
> > who
> > > > > rely
> > > > > > on text based terminal/emacs/vim browsers (no js support) to
> > > > > > monitor component UI.
> > > > > >
> > > > > > links / elinks (http://elinks.or.cz/) are text-based web
> browsers
> > > that
> > > > > > support JavaScript.
> > > > > >
> > > > > > bq. The priority/requirements for UI in core Hadoop should be
> > > security
> > > > > and
> > > > > > correctness, which client side templating cannot address properly
> > > > > > so far.
> > > > > >
> > > > > > I agree that we should focus on security and correctness. The
> > > > paragraphs
> > > > > > above explain that how the architecture of the new UIs
> > > > > > makes the UIs more secure in real-world settings compared to the
> UI
> > > we
> > > > > have
> > > > > > today.
> > > > > >
> > > > > > References:
> > > > > >
> > > > > > 1. A. Yip et al. Improving Application Security with Data Flow
> > > > > Assertions.
> > > > > > In SOSP'2009.
> > > > > >
> > > > > >
> > > > > > On Fri, Oct 25, 2013 at 10:02 AM, Luke Lu <ll...@apache.org>
> wrote:
> > > > > >
> > > > > >> Echoing my comments on HDFS-3555:
> > > > > >>
> > > > > >> I have concerns with this client-side js only approach, which is
> > > less
> > > > > >> secure than a progressively enhanced hybrid approach used by
> YARN.
> > > The
> > > > > >> recent gmail XSS fiasco highlights the issue. I also have
> concerns
> > > > that
> > > > > we
> > > > > >> commit these changes without matching unit tests – the fact you
> > > cannot
> > > > > >> effectively unit test these changes should tell you something
> > about
> > > > this
> > > > > >> approach.
> > > > > >>
> > > > > >> *Requiring* JS means that an admin cannot turn off js to
> > (partially)
> > > > use
> > > > > >> core Hadoop UI. You'd *require* proper SSL (not self signed)
> setup
> > > to
> > > > > avoid
> > > > > >> JS injection, even if security of js libraries used is perfect,
> > > which
> > > > I
> > > > > >> doubt (search gmail/linkedin XSS). Client side rendering
> > completely
> > > > > breaks
> > > > > >> the workflows for ops who rely on text based terminal/emacs/vim
> > > > browsers
> > > > > >> (no js support) to monitor component UI.
> > > > > >>
> > > > > >> IMO, JS-only rendering belongs to social networking sites and/or
> > > SaaS
> > > > > >> front-ends, where full time UI/security specialists babysits UI
> > > > > changes. I
> > > > > >> think eventually most users will use a self servicing UI in a
> SaaS
> > > > > >> front-end that uses REST/JMX API to get data from back-end
> > > components,
> > > > > >> besides their own app master/service UI. The
> priority/requirements
> > > for
> > > > > UI
> > > > > >> in core Hadoop should be security and correctness, which client
> > side
> > > > > >> templating cannot address properly so far.
> > > > > >>
> > > > > >>
> > > > > >> On Tue, Oct 22, 2013 at 3:59 PM, Haohui Mai <
> hmai@hortonworks.com
> > >
> > > > > wrote:
> > > > > >>
> > > > > >> > Hi all,
> > > > > >> >
> > > > > >> >
> > > > > >> > Jing Zhao and I recently have reimplemented the JSP-based web
> > UIs
> > > in
> > > > > >> HTML 5
> > > > > >> > applications (HDFS-5333). Based on our prelimanary testing
> > results
> > > > we
> > > > > >> > believe thst the new web UIs of the namenodes and the datanode
> > are
> > > > > ready
> > > > > >> > for everyday uses.
> > > > > >> >
> > > > > >> > You're more than welcome to try it out on trunk by visiting
> > > http://
> > > > > >> > <namenode>/dfshealth.html
> > > > > >> >
> > > > > >> > There are a number of benefits from this transition. From a
> > > > > developer's
> > > > > >> > prospective, the most notable one is *maintainability*:
> > > > > >> >
> > > > > >> > (1) The abstractions between the UI and the core server are
> > > > > well-defined,
> > > > > >> > decoupling the UI and the core hadoop servers.
> > > > > >> >
> > > > > >> > (2) It allows us to deprecate the logic in the JSP pages. The
> > old
> > > > web
> > > > > UIs
> > > > > >> > have to duplicate the logic in the JSPs. The logic is often
> > > > > out-of-dated
> > > > > >> > and not well-tested, which leads to broken pages and security
> > > > > >> > vulnerabilities(e.g. HDFS-5251, HDFS-5307, HDFS-5308,
> HDFS-5317
> > > and
> > > > > >> > HDFS-4901). The architecture of the new UIs prevent these bugs
> > at
> > > > the
> > > > > >> very
> > > > > >> > beginning.
> > > > > >> >
> > > > > >> >
> > > > > >> > I propose that deprecate the old, JSP-based web UIs in 2.3. I
> > > opened
> > > > > >> > HDFS-5402 to track the relevant discussions.
> > > > > >> >
> > > > > >> > Your feedbacks are highly appreciated.
> > > > > >> >
> > > > > >> >
> > > > > >> > Sincerely,
> > > > > >> >
> > > > > >> > Haohui
> > > > > >> >
> > > > > >> > --
> > > > > >> > CONFIDENTIALITY NOTICE
> > > > > >> > NOTICE: This message is intended for the use of the individual
> > or
> > > > > entity
> > > > > >> to
> > > > > >> > which it is addressed and may contain information that is
> > > > > confidential,
> > > > > >> > privileged and exempt from disclosure under applicable law. If
> > the
> > > > > reader
> > > > > >> > of this message is not the intended recipient, you are hereby
> > > > notified
> > > > > >> that
> > > > > >> > any printing, copying, dissemination, distribution, disclosure
> > or
> > > > > >> > forwarding of this communication is strictly prohibited. If
> you
> > > have
> > > > > >> > received this communication in error, please contact the
> sender
> > > > > >> immediately
> > > > > >> > and delete it from your system. Thank You.
> > > > > >> >
> > > > > >>
> > > > > >
> > > > > > --
> > > > > > CONFIDENTIALITY NOTICE
> > > > > > NOTICE: This message is intended for the use of the individual or
> > > > entity
> > > > > to
> > > > > > which it is addressed and may contain information that is
> > > confidential,
> > > > > > privileged and exempt from disclosure under applicable law. If
> the
> > > > reader
> > > > > > of this message is not the intended recipient, you are hereby
> > > notified
> > > > > that
> > > > > > any printing, copying, dissemination, distribution, disclosure or
> > > > > > forwarding of this communication is strictly prohibited. If you
> > have
> > > > > > received this communication in error, please contact the sender
> > > > > immediately
> > > > > > and delete it from your system. Thank You.
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Alejandro
> > > >
> > >
> > > --
> > > CONFIDENTIALITY NOTICE
> > > NOTICE: This message is intended for the use of the individual or
> entity
> > to
> > > which it is addressed and may contain information that is confidential,
> > > privileged and exempt from disclosure under applicable law. If the
> reader
> > > of this message is not the intended recipient, you are hereby notified
> > that
> > > any printing, copying, dissemination, distribution, disclosure or
> > > forwarding of this communication is strictly prohibited. If you have
> > > received this communication in error, please contact the sender
> > immediately
> > > and delete it from your system. Thank You.
> > >
> >
> >
> >
> > --
> > Alejandro
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>



-- 
Alejandro

Re: Replacing the JSP web UIs to HTML 5 applications

[Haohui Mai <hm...@hortonworks.com>]

Alejandro,

If I understand correctly, that is the exact approach that the new web UI
is taking. The new web UI takes the output from JMX and renders them as
HTML at the client side.

~Haohui


On Mon, Oct 28, 2013 at 4:18 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:

> Haohui,
>
> If you have NN and DNs producing JSON instead HTML, then you can build JS
> based web UIs. Take for example Oozie, Oozie produces JSON, it has a built
> in JS web ui that consumes JSON and Hue has built an external web UI that
> also consumes JSON. In the case of Hue UI, Oozie didn't have to change
> anything to get that UI and improvements on the Hue UI don't require
> changes in Oozie unless it is to produce additional information.
>
> hope this clarifies.
>
> Thx
>
>
> On Mon, Oct 28, 2013 at 4:06 PM, Haohui Mai <hm...@hortonworks.com> wrote:
>
> > Echo my comments on HDFS-5402:
> >
> > bq. If we're going to remove the old web UI, I think the new web UI has
> > to have the same level of unit testing. We shouldn't go backwards in
> > terms of unit testing.
> >
> > I take a look at TestNamenodeJspHelper / TestDatanodeJspHelper /
> > TestClusterJspHelper. It seems to me that we can merge these tests with
> the
> > unit tests on JMX.
> >
> > bq. If we are going to
> > remove this capability, we need to add some other command-line tools
> > to get the same functionality. These tools could use REST if we have
> > that, or JMX, but they need to exist before we can consider removing
> > the old UI.
> >
> > This is a good point. Since all information are available through JMX,
> the
> > easiest way to approach it is to write some scripts using Node.js. The
> > architecture of the new Web UIs is ready for this.
> >
> >
> > On Mon, Oct 28, 2013 at 3:57 PM, Alejandro Abdelnur <tucu@cloudera.com
> > >wrote:
> >
> > > Producing JSON would be great. Agree with Colin that we should leave
> for
> > > now the current JSP based web ui.
> > >
> > > thx
> > >
> > >
> > > On Mon, Oct 28, 2013 at 11:16 AM, Colin McCabe <cmccabe@alumni.cmu.edu
> > > >wrote:
> > >
> > > > This is a really interesting project, Haohui.  I think it will make
> > > > our web UI much nicer.
> > > >
> > > > I have a few concerns about removing the old web UI, however:
> > > >
> > > > * If we're going to remove the old web UI, I think the new web UI has
> > > > to have the same level of unit testing.  We shouldn't go backwards in
> > > > terms of unit testing.
> > > >
> > > > * Most of the deployments of elinks and links out there don't support
> > > > Javascript.  This is just a reality of life when using CentOS 5 or 6,
> > > > which many users are still using.  I have used "links" to diagnose
> > > > problems through the web UI in the past, in systems where access to
> > > > the cluster was available only through telnet.  If we are going to
> > > > remove this capability, we need to add some other command-line tools
> > > > to get the same functionality.  These tools could use REST if we have
> > > > that, or JMX, but they need to exist before we can consider removing
> > > > the old UI.
> > > >
> > > > best,
> > > > Colin
> > > >
> > > > On Fri, Oct 25, 2013 at 7:31 PM, Haohui Mai <hm...@hortonworks.com>
> > > wrote:
> > > > > Thanks for the reply, Luke. Here I just echo my response from the
> > jira:
> > > > >
> > > > > bq. this client-side js only approach, which is less secure than a
> > > > > progressively enhanced hybrid approach used by YARN. The recent
> gmail
> > > > > XSS fiasco highlights the issue.
> > > > >
> > > > > I'm presenting an informal security analysis to compare the
> security
> > of
> > > > the
> > > > > old and the new web UIs.
> > > > >
> > > > > An attacker launches an XSS attack by injecting malicious code
> which
> > > are
> > > > > usually HTML or JavaScript fragments into the web page, so that the
> > > > > malicious code can have the same privileges of the web page.
> > > > >
> > > > > First, in the scope of XSS attacks, note that the threat models of
> > > > > launching XSS attacks on Internet sites Gmail/Linkedin and the one
> of
> > > the
> > > > > Hadoop UIs are different. They have fundamental different sets of
> > > > external
> > > > > inputs that the attackers have control to. Internet sites have
> little
> > > > > control of these inputs. In the case of Gmail / Linkedin, an attack
> > can
> > > > > send you a crafted e-mail, or put malicious description in his /
> > > > > her Linkedin profile. The sets of external inputs are *restricted*
> in
> > > > > Hadoop UIs. The new web UIs take JMX and WebHDFS as inputs. The
> > > > > attacker has to launch a XSS attack by:
> > > > >
> > > > > * Compromise the jars so that the output of JMX / WebHDFS have the
> > > > > malicious code.
> > > > > * Replace the web UIs completely to include the malicious code.
> > > > >
> > > > > In either case *the attacker has to compromise the hadoop core or
> the
> > > > > namenode*. That means the new web UIs are at least as secure as the
> > > > hadoop
> > > > > core, and the namenode machine.
> > > > >
> > > > > Second, I argue that using client-side templates are more secure
> than
> > > the
> > > > > current JSP-based server-side templates. To defend against XSS
> > > > > attacks, both techniques have to filter the external inputs at
> > *every*
> > > > > possible execution paths. Several facts much be taken into
> > > > > plays when evaluating the security of both approaches in real-world
> > > > > environments:
> > > > >
> > > > > * The JavaScript libraries used in the new web UIs have survived in
> > > > > extremely large-scale production tests. jQuery is used by Google
> and
> > > > >  Microsoft, bootstrap is used by Twitter, and dust.js is used by
> > > > Linkedin.
> > > > > All libraries survived from hundreds of thousands of
> > > > >  attack attempts on a daily basis. I agree that the libraries might
> > > still
> > > > > be imperfect, but there's no way that we can test the JSP web
> > > > >  UIs to achieve the same level of assurances given the amount of
> > > > resources
> > > > > the community has.
> > > > > * Client-side templates consolidate all filtering logic in one
> > central
> > > > > place. Recall that the goal is to filter all external inputs at
> every
> > > > >  execution paths, this is a much more systematic approach compared
> to
> > > the
> > > > > server-side templates we have today. It is difficult (if not
> > > > >  impossible) to do it in a JSP/ASP/PHP application, since such
> > > filtering
> > > > > can be only achieved via ad-hoc approaches ([1] shows some
> > > > >  empirical data). Also, HDFS-4901 recently describes a XSS
> > > vulnerability
> > > > in
> > > > > browseDirectory.jsp.
> > > > >
> > > > > bq. You'd require proper SSL (not self signed) setup to avoid JS
> > > > > injection
> > > > >
> > > > > Commodity browsers enforce Same-Origin Policy to defend against
> code
> > > > > injections. It has nothing to do with what kinds of SSL
> certificates
> > > > > you hold.
> > > > >
> > > > > bq.  I also have concerns that we commit these changes without
> > matching
> > > > > unit tests
> > > > >
> > > > > The JavaScript code can be automatically tested. The same code can
> be
> > > run
> > > > > by node.js and the test can compared with pre-defined
> > > > > results. It is also possible to write an adapter to use Rhino to
> > > > accomplish
> > > > > the same task. We can discuss how to integrate them into
> > > > > the maven test routines in a different thread.
> > > > >
> > > > > bq. Client side rendering completely breaks the workflows for ops
> who
> > > > rely
> > > > > on text based terminal/emacs/vim browsers (no js support) to
> > > > > monitor component UI.
> > > > >
> > > > > links / elinks (http://elinks.or.cz/) are text-based web browsers
> > that
> > > > > support JavaScript.
> > > > >
> > > > > bq. The priority/requirements for UI in core Hadoop should be
> > security
> > > > and
> > > > > correctness, which client side templating cannot address properly
> > > > > so far.
> > > > >
> > > > > I agree that we should focus on security and correctness. The
> > > paragraphs
> > > > > above explain that how the architecture of the new UIs
> > > > > makes the UIs more secure in real-world settings compared to the UI
> > we
> > > > have
> > > > > today.
> > > > >
> > > > > References:
> > > > >
> > > > > 1. A. Yip et al. Improving Application Security with Data Flow
> > > > Assertions.
> > > > > In SOSP'2009.
> > > > >
> > > > >
> > > > > On Fri, Oct 25, 2013 at 10:02 AM, Luke Lu <ll...@apache.org> wrote:
> > > > >
> > > > >> Echoing my comments on HDFS-3555:
> > > > >>
> > > > >> I have concerns with this client-side js only approach, which is
> > less
> > > > >> secure than a progressively enhanced hybrid approach used by YARN.
> > The
> > > > >> recent gmail XSS fiasco highlights the issue. I also have concerns
> > > that
> > > > we
> > > > >> commit these changes without matching unit tests – the fact you
> > cannot
> > > > >> effectively unit test these changes should tell you something
> about
> > > this
> > > > >> approach.
> > > > >>
> > > > >> *Requiring* JS means that an admin cannot turn off js to
> (partially)
> > > use
> > > > >> core Hadoop UI. You'd *require* proper SSL (not self signed) setup
> > to
> > > > avoid
> > > > >> JS injection, even if security of js libraries used is perfect,
> > which
> > > I
> > > > >> doubt (search gmail/linkedin XSS). Client side rendering
> completely
> > > > breaks
> > > > >> the workflows for ops who rely on text based terminal/emacs/vim
> > > browsers
> > > > >> (no js support) to monitor component UI.
> > > > >>
> > > > >> IMO, JS-only rendering belongs to social networking sites and/or
> > SaaS
> > > > >> front-ends, where full time UI/security specialists babysits UI
> > > > changes. I
> > > > >> think eventually most users will use a self servicing UI in a SaaS
> > > > >> front-end that uses REST/JMX API to get data from back-end
> > components,
> > > > >> besides their own app master/service UI. The priority/requirements
> > for
> > > > UI
> > > > >> in core Hadoop should be security and correctness, which client
> side
> > > > >> templating cannot address properly so far.
> > > > >>
> > > > >>
> > > > >> On Tue, Oct 22, 2013 at 3:59 PM, Haohui Mai <hmai@hortonworks.com
> >
> > > > wrote:
> > > > >>
> > > > >> > Hi all,
> > > > >> >
> > > > >> >
> > > > >> > Jing Zhao and I recently have reimplemented the JSP-based web
> UIs
> > in
> > > > >> HTML 5
> > > > >> > applications (HDFS-5333). Based on our prelimanary testing
> results
> > > we
> > > > >> > believe thst the new web UIs of the namenodes and the datanode
> are
> > > > ready
> > > > >> > for everyday uses.
> > > > >> >
> > > > >> > You're more than welcome to try it out on trunk by visiting
> > http://
> > > > >> > <namenode>/dfshealth.html
> > > > >> >
> > > > >> > There are a number of benefits from this transition. From a
> > > > developer's
> > > > >> > prospective, the most notable one is *maintainability*:
> > > > >> >
> > > > >> > (1) The abstractions between the UI and the core server are
> > > > well-defined,
> > > > >> > decoupling the UI and the core hadoop servers.
> > > > >> >
> > > > >> > (2) It allows us to deprecate the logic in the JSP pages. The
> old
> > > web
> > > > UIs
> > > > >> > have to duplicate the logic in the JSPs. The logic is often
> > > > out-of-dated
> > > > >> > and not well-tested, which leads to broken pages and security
> > > > >> > vulnerabilities(e.g. HDFS-5251, HDFS-5307, HDFS-5308, HDFS-5317
> > and
> > > > >> > HDFS-4901). The architecture of the new UIs prevent these bugs
> at
> > > the
> > > > >> very
> > > > >> > beginning.
> > > > >> >
> > > > >> >
> > > > >> > I propose that deprecate the old, JSP-based web UIs in 2.3. I
> > opened
> > > > >> > HDFS-5402 to track the relevant discussions.
> > > > >> >
> > > > >> > Your feedbacks are highly appreciated.
> > > > >> >
> > > > >> >
> > > > >> > Sincerely,
> > > > >> >
> > > > >> > Haohui
> > > > >> >
> > > > >> > --
> > > > >> > CONFIDENTIALITY NOTICE
> > > > >> > NOTICE: This message is intended for the use of the individual
> or
> > > > entity
> > > > >> to
> > > > >> > which it is addressed and may contain information that is
> > > > confidential,
> > > > >> > privileged and exempt from disclosure under applicable law. If
> the
> > > > reader
> > > > >> > of this message is not the intended recipient, you are hereby
> > > notified
> > > > >> that
> > > > >> > any printing, copying, dissemination, distribution, disclosure
> or
> > > > >> > forwarding of this communication is strictly prohibited. If you
> > have
> > > > >> > received this communication in error, please contact the sender
> > > > >> immediately
> > > > >> > and delete it from your system. Thank You.
> > > > >> >
> > > > >>
> > > > >
> > > > > --
> > > > > CONFIDENTIALITY NOTICE
> > > > > NOTICE: This message is intended for the use of the individual or
> > > entity
> > > > to
> > > > > which it is addressed and may contain information that is
> > confidential,
> > > > > privileged and exempt from disclosure under applicable law. If the
> > > reader
> > > > > of this message is not the intended recipient, you are hereby
> > notified
> > > > that
> > > > > any printing, copying, dissemination, distribution, disclosure or
> > > > > forwarding of this communication is strictly prohibited. If you
> have
> > > > > received this communication in error, please contact the sender
> > > > immediately
> > > > > and delete it from your system. Thank You.
> > > >
> > >
> > >
> > >
> > > --
> > > Alejandro
> > >
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> >
>
>
>
> --
> Alejandro
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Replacing the JSP web UIs to HTML 5 applications

[Alejandro Abdelnur <tu...@cloudera.com>]

Haohui,

If you have NN and DNs producing JSON instead HTML, then you can build JS
based web UIs. Take for example Oozie, Oozie produces JSON, it has a built
in JS web ui that consumes JSON and Hue has built an external web UI that
also consumes JSON. In the case of Hue UI, Oozie didn't have to change
anything to get that UI and improvements on the Hue UI don't require
changes in Oozie unless it is to produce additional information.

hope this clarifies.

Thx


On Mon, Oct 28, 2013 at 4:06 PM, Haohui Mai <hm...@hortonworks.com> wrote:

> Echo my comments on HDFS-5402:
>
> bq. If we're going to remove the old web UI, I think the new web UI has
> to have the same level of unit testing. We shouldn't go backwards in
> terms of unit testing.
>
> I take a look at TestNamenodeJspHelper / TestDatanodeJspHelper /
> TestClusterJspHelper. It seems to me that we can merge these tests with the
> unit tests on JMX.
>
> bq. If we are going to
> remove this capability, we need to add some other command-line tools
> to get the same functionality. These tools could use REST if we have
> that, or JMX, but they need to exist before we can consider removing
> the old UI.
>
> This is a good point. Since all information are available through JMX, the
> easiest way to approach it is to write some scripts using Node.js. The
> architecture of the new Web UIs is ready for this.
>
>
> On Mon, Oct 28, 2013 at 3:57 PM, Alejandro Abdelnur <tucu@cloudera.com
> >wrote:
>
> > Producing JSON would be great. Agree with Colin that we should leave for
> > now the current JSP based web ui.
> >
> > thx
> >
> >
> > On Mon, Oct 28, 2013 at 11:16 AM, Colin McCabe <cmccabe@alumni.cmu.edu
> > >wrote:
> >
> > > This is a really interesting project, Haohui.  I think it will make
> > > our web UI much nicer.
> > >
> > > I have a few concerns about removing the old web UI, however:
> > >
> > > * If we're going to remove the old web UI, I think the new web UI has
> > > to have the same level of unit testing.  We shouldn't go backwards in
> > > terms of unit testing.
> > >
> > > * Most of the deployments of elinks and links out there don't support
> > > Javascript.  This is just a reality of life when using CentOS 5 or 6,
> > > which many users are still using.  I have used "links" to diagnose
> > > problems through the web UI in the past, in systems where access to
> > > the cluster was available only through telnet.  If we are going to
> > > remove this capability, we need to add some other command-line tools
> > > to get the same functionality.  These tools could use REST if we have
> > > that, or JMX, but they need to exist before we can consider removing
> > > the old UI.
> > >
> > > best,
> > > Colin
> > >
> > > On Fri, Oct 25, 2013 at 7:31 PM, Haohui Mai <hm...@hortonworks.com>
> > wrote:
> > > > Thanks for the reply, Luke. Here I just echo my response from the
> jira:
> > > >
> > > > bq. this client-side js only approach, which is less secure than a
> > > > progressively enhanced hybrid approach used by YARN. The recent gmail
> > > > XSS fiasco highlights the issue.
> > > >
> > > > I'm presenting an informal security analysis to compare the security
> of
> > > the
> > > > old and the new web UIs.
> > > >
> > > > An attacker launches an XSS attack by injecting malicious code which
> > are
> > > > usually HTML or JavaScript fragments into the web page, so that the
> > > > malicious code can have the same privileges of the web page.
> > > >
> > > > First, in the scope of XSS attacks, note that the threat models of
> > > > launching XSS attacks on Internet sites Gmail/Linkedin and the one of
> > the
> > > > Hadoop UIs are different. They have fundamental different sets of
> > > external
> > > > inputs that the attackers have control to. Internet sites have little
> > > > control of these inputs. In the case of Gmail / Linkedin, an attack
> can
> > > > send you a crafted e-mail, or put malicious description in his /
> > > > her Linkedin profile. The sets of external inputs are *restricted* in
> > > > Hadoop UIs. The new web UIs take JMX and WebHDFS as inputs. The
> > > > attacker has to launch a XSS attack by:
> > > >
> > > > * Compromise the jars so that the output of JMX / WebHDFS have the
> > > > malicious code.
> > > > * Replace the web UIs completely to include the malicious code.
> > > >
> > > > In either case *the attacker has to compromise the hadoop core or the
> > > > namenode*. That means the new web UIs are at least as secure as the
> > > hadoop
> > > > core, and the namenode machine.
> > > >
> > > > Second, I argue that using client-side templates are more secure than
> > the
> > > > current JSP-based server-side templates. To defend against XSS
> > > > attacks, both techniques have to filter the external inputs at
> *every*
> > > > possible execution paths. Several facts much be taken into
> > > > plays when evaluating the security of both approaches in real-world
> > > > environments:
> > > >
> > > > * The JavaScript libraries used in the new web UIs have survived in
> > > > extremely large-scale production tests. jQuery is used by Google and
> > > >  Microsoft, bootstrap is used by Twitter, and dust.js is used by
> > > Linkedin.
> > > > All libraries survived from hundreds of thousands of
> > > >  attack attempts on a daily basis. I agree that the libraries might
> > still
> > > > be imperfect, but there's no way that we can test the JSP web
> > > >  UIs to achieve the same level of assurances given the amount of
> > > resources
> > > > the community has.
> > > > * Client-side templates consolidate all filtering logic in one
> central
> > > > place. Recall that the goal is to filter all external inputs at every
> > > >  execution paths, this is a much more systematic approach compared to
> > the
> > > > server-side templates we have today. It is difficult (if not
> > > >  impossible) to do it in a JSP/ASP/PHP application, since such
> > filtering
> > > > can be only achieved via ad-hoc approaches ([1] shows some
> > > >  empirical data). Also, HDFS-4901 recently describes a XSS
> > vulnerability
> > > in
> > > > browseDirectory.jsp.
> > > >
> > > > bq. You'd require proper SSL (not self signed) setup to avoid JS
> > > > injection
> > > >
> > > > Commodity browsers enforce Same-Origin Policy to defend against code
> > > > injections. It has nothing to do with what kinds of SSL certificates
> > > > you hold.
> > > >
> > > > bq.  I also have concerns that we commit these changes without
> matching
> > > > unit tests
> > > >
> > > > The JavaScript code can be automatically tested. The same code can be
> > run
> > > > by node.js and the test can compared with pre-defined
> > > > results. It is also possible to write an adapter to use Rhino to
> > > accomplish
> > > > the same task. We can discuss how to integrate them into
> > > > the maven test routines in a different thread.
> > > >
> > > > bq. Client side rendering completely breaks the workflows for ops who
> > > rely
> > > > on text based terminal/emacs/vim browsers (no js support) to
> > > > monitor component UI.
> > > >
> > > > links / elinks (http://elinks.or.cz/) are text-based web browsers
> that
> > > > support JavaScript.
> > > >
> > > > bq. The priority/requirements for UI in core Hadoop should be
> security
> > > and
> > > > correctness, which client side templating cannot address properly
> > > > so far.
> > > >
> > > > I agree that we should focus on security and correctness. The
> > paragraphs
> > > > above explain that how the architecture of the new UIs
> > > > makes the UIs more secure in real-world settings compared to the UI
> we
> > > have
> > > > today.
> > > >
> > > > References:
> > > >
> > > > 1. A. Yip et al. Improving Application Security with Data Flow
> > > Assertions.
> > > > In SOSP'2009.
> > > >
> > > >
> > > > On Fri, Oct 25, 2013 at 10:02 AM, Luke Lu <ll...@apache.org> wrote:
> > > >
> > > >> Echoing my comments on HDFS-3555:
> > > >>
> > > >> I have concerns with this client-side js only approach, which is
> less
> > > >> secure than a progressively enhanced hybrid approach used by YARN.
> The
> > > >> recent gmail XSS fiasco highlights the issue. I also have concerns
> > that
> > > we
> > > >> commit these changes without matching unit tests – the fact you
> cannot
> > > >> effectively unit test these changes should tell you something about
> > this
> > > >> approach.
> > > >>
> > > >> *Requiring* JS means that an admin cannot turn off js to (partially)
> > use
> > > >> core Hadoop UI. You'd *require* proper SSL (not self signed) setup
> to
> > > avoid
> > > >> JS injection, even if security of js libraries used is perfect,
> which
> > I
> > > >> doubt (search gmail/linkedin XSS). Client side rendering completely
> > > breaks
> > > >> the workflows for ops who rely on text based terminal/emacs/vim
> > browsers
> > > >> (no js support) to monitor component UI.
> > > >>
> > > >> IMO, JS-only rendering belongs to social networking sites and/or
> SaaS
> > > >> front-ends, where full time UI/security specialists babysits UI
> > > changes. I
> > > >> think eventually most users will use a self servicing UI in a SaaS
> > > >> front-end that uses REST/JMX API to get data from back-end
> components,
> > > >> besides their own app master/service UI. The priority/requirements
> for
> > > UI
> > > >> in core Hadoop should be security and correctness, which client side
> > > >> templating cannot address properly so far.
> > > >>
> > > >>
> > > >> On Tue, Oct 22, 2013 at 3:59 PM, Haohui Mai <hm...@hortonworks.com>
> > > wrote:
> > > >>
> > > >> > Hi all,
> > > >> >
> > > >> >
> > > >> > Jing Zhao and I recently have reimplemented the JSP-based web UIs
> in
> > > >> HTML 5
> > > >> > applications (HDFS-5333). Based on our prelimanary testing results
> > we
> > > >> > believe thst the new web UIs of the namenodes and the datanode are
> > > ready
> > > >> > for everyday uses.
> > > >> >
> > > >> > You're more than welcome to try it out on trunk by visiting
> http://
> > > >> > <namenode>/dfshealth.html
> > > >> >
> > > >> > There are a number of benefits from this transition. From a
> > > developer's
> > > >> > prospective, the most notable one is *maintainability*:
> > > >> >
> > > >> > (1) The abstractions between the UI and the core server are
> > > well-defined,
> > > >> > decoupling the UI and the core hadoop servers.
> > > >> >
> > > >> > (2) It allows us to deprecate the logic in the JSP pages. The old
> > web
> > > UIs
> > > >> > have to duplicate the logic in the JSPs. The logic is often
> > > out-of-dated
> > > >> > and not well-tested, which leads to broken pages and security
> > > >> > vulnerabilities(e.g. HDFS-5251, HDFS-5307, HDFS-5308, HDFS-5317
> and
> > > >> > HDFS-4901). The architecture of the new UIs prevent these bugs at
> > the
> > > >> very
> > > >> > beginning.
> > > >> >
> > > >> >
> > > >> > I propose that deprecate the old, JSP-based web UIs in 2.3. I
> opened
> > > >> > HDFS-5402 to track the relevant discussions.
> > > >> >
> > > >> > Your feedbacks are highly appreciated.
> > > >> >
> > > >> >
> > > >> > Sincerely,
> > > >> >
> > > >> > Haohui
> > > >> >
> > > >> > --
> > > >> > CONFIDENTIALITY NOTICE
> > > >> > NOTICE: This message is intended for the use of the individual or
> > > entity
> > > >> to
> > > >> > which it is addressed and may contain information that is
> > > confidential,
> > > >> > privileged and exempt from disclosure under applicable law. If the
> > > reader
> > > >> > of this message is not the intended recipient, you are hereby
> > notified
> > > >> that
> > > >> > any printing, copying, dissemination, distribution, disclosure or
> > > >> > forwarding of this communication is strictly prohibited. If you
> have
> > > >> > received this communication in error, please contact the sender
> > > >> immediately
> > > >> > and delete it from your system. Thank You.
> > > >> >
> > > >>
> > > >
> > > > --
> > > > CONFIDENTIALITY NOTICE
> > > > NOTICE: This message is intended for the use of the individual or
> > entity
> > > to
> > > > which it is addressed and may contain information that is
> confidential,
> > > > privileged and exempt from disclosure under applicable law. If the
> > reader
> > > > of this message is not the intended recipient, you are hereby
> notified
> > > that
> > > > any printing, copying, dissemination, distribution, disclosure or
> > > > forwarding of this communication is strictly prohibited. If you have
> > > > received this communication in error, please contact the sender
> > > immediately
> > > > and delete it from your system. Thank You.
> > >
> >
> >
> >
> > --
> > Alejandro
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>



-- 
Alejandro

Re: Replacing the JSP web UIs to HTML 5 applications

[Haohui Mai <hm...@hortonworks.com>]

Echo my comments on HDFS-5402:

bq. If we're going to remove the old web UI, I think the new web UI has
to have the same level of unit testing. We shouldn't go backwards in
terms of unit testing.

I take a look at TestNamenodeJspHelper / TestDatanodeJspHelper /
TestClusterJspHelper. It seems to me that we can merge these tests with the
unit tests on JMX.

bq. If we are going to
remove this capability, we need to add some other command-line tools
to get the same functionality. These tools could use REST if we have
that, or JMX, but they need to exist before we can consider removing
the old UI.

This is a good point. Since all information are available through JMX, the
easiest way to approach it is to write some scripts using Node.js. The
architecture of the new Web UIs is ready for this.


On Mon, Oct 28, 2013 at 3:57 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:

> Producing JSON would be great. Agree with Colin that we should leave for
> now the current JSP based web ui.
>
> thx
>
>
> On Mon, Oct 28, 2013 at 11:16 AM, Colin McCabe <cmccabe@alumni.cmu.edu
> >wrote:
>
> > This is a really interesting project, Haohui.  I think it will make
> > our web UI much nicer.
> >
> > I have a few concerns about removing the old web UI, however:
> >
> > * If we're going to remove the old web UI, I think the new web UI has
> > to have the same level of unit testing.  We shouldn't go backwards in
> > terms of unit testing.
> >
> > * Most of the deployments of elinks and links out there don't support
> > Javascript.  This is just a reality of life when using CentOS 5 or 6,
> > which many users are still using.  I have used "links" to diagnose
> > problems through the web UI in the past, in systems where access to
> > the cluster was available only through telnet.  If we are going to
> > remove this capability, we need to add some other command-line tools
> > to get the same functionality.  These tools could use REST if we have
> > that, or JMX, but they need to exist before we can consider removing
> > the old UI.
> >
> > best,
> > Colin
> >
> > On Fri, Oct 25, 2013 at 7:31 PM, Haohui Mai <hm...@hortonworks.com>
> wrote:
> > > Thanks for the reply, Luke. Here I just echo my response from the jira:
> > >
> > > bq. this client-side js only approach, which is less secure than a
> > > progressively enhanced hybrid approach used by YARN. The recent gmail
> > > XSS fiasco highlights the issue.
> > >
> > > I'm presenting an informal security analysis to compare the security of
> > the
> > > old and the new web UIs.
> > >
> > > An attacker launches an XSS attack by injecting malicious code which
> are
> > > usually HTML or JavaScript fragments into the web page, so that the
> > > malicious code can have the same privileges of the web page.
> > >
> > > First, in the scope of XSS attacks, note that the threat models of
> > > launching XSS attacks on Internet sites Gmail/Linkedin and the one of
> the
> > > Hadoop UIs are different. They have fundamental different sets of
> > external
> > > inputs that the attackers have control to. Internet sites have little
> > > control of these inputs. In the case of Gmail / Linkedin, an attack can
> > > send you a crafted e-mail, or put malicious description in his /
> > > her Linkedin profile. The sets of external inputs are *restricted* in
> > > Hadoop UIs. The new web UIs take JMX and WebHDFS as inputs. The
> > > attacker has to launch a XSS attack by:
> > >
> > > * Compromise the jars so that the output of JMX / WebHDFS have the
> > > malicious code.
> > > * Replace the web UIs completely to include the malicious code.
> > >
> > > In either case *the attacker has to compromise the hadoop core or the
> > > namenode*. That means the new web UIs are at least as secure as the
> > hadoop
> > > core, and the namenode machine.
> > >
> > > Second, I argue that using client-side templates are more secure than
> the
> > > current JSP-based server-side templates. To defend against XSS
> > > attacks, both techniques have to filter the external inputs at *every*
> > > possible execution paths. Several facts much be taken into
> > > plays when evaluating the security of both approaches in real-world
> > > environments:
> > >
> > > * The JavaScript libraries used in the new web UIs have survived in
> > > extremely large-scale production tests. jQuery is used by Google and
> > >  Microsoft, bootstrap is used by Twitter, and dust.js is used by
> > Linkedin.
> > > All libraries survived from hundreds of thousands of
> > >  attack attempts on a daily basis. I agree that the libraries might
> still
> > > be imperfect, but there's no way that we can test the JSP web
> > >  UIs to achieve the same level of assurances given the amount of
> > resources
> > > the community has.
> > > * Client-side templates consolidate all filtering logic in one central
> > > place. Recall that the goal is to filter all external inputs at every
> > >  execution paths, this is a much more systematic approach compared to
> the
> > > server-side templates we have today. It is difficult (if not
> > >  impossible) to do it in a JSP/ASP/PHP application, since such
> filtering
> > > can be only achieved via ad-hoc approaches ([1] shows some
> > >  empirical data). Also, HDFS-4901 recently describes a XSS
> vulnerability
> > in
> > > browseDirectory.jsp.
> > >
> > > bq. You'd require proper SSL (not self signed) setup to avoid JS
> > > injection
> > >
> > > Commodity browsers enforce Same-Origin Policy to defend against code
> > > injections. It has nothing to do with what kinds of SSL certificates
> > > you hold.
> > >
> > > bq.  I also have concerns that we commit these changes without matching
> > > unit tests
> > >
> > > The JavaScript code can be automatically tested. The same code can be
> run
> > > by node.js and the test can compared with pre-defined
> > > results. It is also possible to write an adapter to use Rhino to
> > accomplish
> > > the same task. We can discuss how to integrate them into
> > > the maven test routines in a different thread.
> > >
> > > bq. Client side rendering completely breaks the workflows for ops who
> > rely
> > > on text based terminal/emacs/vim browsers (no js support) to
> > > monitor component UI.
> > >
> > > links / elinks (http://elinks.or.cz/) are text-based web browsers that
> > > support JavaScript.
> > >
> > > bq. The priority/requirements for UI in core Hadoop should be security
> > and
> > > correctness, which client side templating cannot address properly
> > > so far.
> > >
> > > I agree that we should focus on security and correctness. The
> paragraphs
> > > above explain that how the architecture of the new UIs
> > > makes the UIs more secure in real-world settings compared to the UI we
> > have
> > > today.
> > >
> > > References:
> > >
> > > 1. A. Yip et al. Improving Application Security with Data Flow
> > Assertions.
> > > In SOSP'2009.
> > >
> > >
> > > On Fri, Oct 25, 2013 at 10:02 AM, Luke Lu <ll...@apache.org> wrote:
> > >
> > >> Echoing my comments on HDFS-3555:
> > >>
> > >> I have concerns with this client-side js only approach, which is less
> > >> secure than a progressively enhanced hybrid approach used by YARN. The
> > >> recent gmail XSS fiasco highlights the issue. I also have concerns
> that
> > we
> > >> commit these changes without matching unit tests – the fact you cannot
> > >> effectively unit test these changes should tell you something about
> this
> > >> approach.
> > >>
> > >> *Requiring* JS means that an admin cannot turn off js to (partially)
> use
> > >> core Hadoop UI. You'd *require* proper SSL (not self signed) setup to
> > avoid
> > >> JS injection, even if security of js libraries used is perfect, which
> I
> > >> doubt (search gmail/linkedin XSS). Client side rendering completely
> > breaks
> > >> the workflows for ops who rely on text based terminal/emacs/vim
> browsers
> > >> (no js support) to monitor component UI.
> > >>
> > >> IMO, JS-only rendering belongs to social networking sites and/or SaaS
> > >> front-ends, where full time UI/security specialists babysits UI
> > changes. I
> > >> think eventually most users will use a self servicing UI in a SaaS
> > >> front-end that uses REST/JMX API to get data from back-end components,
> > >> besides their own app master/service UI. The priority/requirements for
> > UI
> > >> in core Hadoop should be security and correctness, which client side
> > >> templating cannot address properly so far.
> > >>
> > >>
> > >> On Tue, Oct 22, 2013 at 3:59 PM, Haohui Mai <hm...@hortonworks.com>
> > wrote:
> > >>
> > >> > Hi all,
> > >> >
> > >> >
> > >> > Jing Zhao and I recently have reimplemented the JSP-based web UIs in
> > >> HTML 5
> > >> > applications (HDFS-5333). Based on our prelimanary testing results
> we
> > >> > believe thst the new web UIs of the namenodes and the datanode are
> > ready
> > >> > for everyday uses.
> > >> >
> > >> > You're more than welcome to try it out on trunk by visiting http://
> > >> > <namenode>/dfshealth.html
> > >> >
> > >> > There are a number of benefits from this transition. From a
> > developer's
> > >> > prospective, the most notable one is *maintainability*:
> > >> >
> > >> > (1) The abstractions between the UI and the core server are
> > well-defined,
> > >> > decoupling the UI and the core hadoop servers.
> > >> >
> > >> > (2) It allows us to deprecate the logic in the JSP pages. The old
> web
> > UIs
> > >> > have to duplicate the logic in the JSPs. The logic is often
> > out-of-dated
> > >> > and not well-tested, which leads to broken pages and security
> > >> > vulnerabilities(e.g. HDFS-5251, HDFS-5307, HDFS-5308, HDFS-5317 and
> > >> > HDFS-4901). The architecture of the new UIs prevent these bugs at
> the
> > >> very
> > >> > beginning.
> > >> >
> > >> >
> > >> > I propose that deprecate the old, JSP-based web UIs in 2.3. I opened
> > >> > HDFS-5402 to track the relevant discussions.
> > >> >
> > >> > Your feedbacks are highly appreciated.
> > >> >
> > >> >
> > >> > Sincerely,
> > >> >
> > >> > Haohui
> > >> >
> > >> > --
> > >> > CONFIDENTIALITY NOTICE
> > >> > NOTICE: This message is intended for the use of the individual or
> > entity
> > >> to
> > >> > which it is addressed and may contain information that is
> > confidential,
> > >> > privileged and exempt from disclosure under applicable law. If the
> > reader
> > >> > of this message is not the intended recipient, you are hereby
> notified
> > >> that
> > >> > any printing, copying, dissemination, distribution, disclosure or
> > >> > forwarding of this communication is strictly prohibited. If you have
> > >> > received this communication in error, please contact the sender
> > >> immediately
> > >> > and delete it from your system. Thank You.
> > >> >
> > >>
> > >
> > > --
> > > CONFIDENTIALITY NOTICE
> > > NOTICE: This message is intended for the use of the individual or
> entity
> > to
> > > which it is addressed and may contain information that is confidential,
> > > privileged and exempt from disclosure under applicable law. If the
> reader
> > > of this message is not the intended recipient, you are hereby notified
> > that
> > > any printing, copying, dissemination, distribution, disclosure or
> > > forwarding of this communication is strictly prohibited. If you have
> > > received this communication in error, please contact the sender
> > immediately
> > > and delete it from your system. Thank You.
> >
>
>
>
> --
> Alejandro
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Replacing the JSP web UIs to HTML 5 applications

[Alejandro Abdelnur <tu...@cloudera.com>]

Producing JSON would be great. Agree with Colin that we should leave for
now the current JSP based web ui.

thx


On Mon, Oct 28, 2013 at 11:16 AM, Colin McCabe <cm...@alumni.cmu.edu>wrote:

> This is a really interesting project, Haohui.  I think it will make
> our web UI much nicer.
>
> I have a few concerns about removing the old web UI, however:
>
> * If we're going to remove the old web UI, I think the new web UI has
> to have the same level of unit testing.  We shouldn't go backwards in
> terms of unit testing.
>
> * Most of the deployments of elinks and links out there don't support
> Javascript.  This is just a reality of life when using CentOS 5 or 6,
> which many users are still using.  I have used "links" to diagnose
> problems through the web UI in the past, in systems where access to
> the cluster was available only through telnet.  If we are going to
> remove this capability, we need to add some other command-line tools
> to get the same functionality.  These tools could use REST if we have
> that, or JMX, but they need to exist before we can consider removing
> the old UI.
>
> best,
> Colin
>
> On Fri, Oct 25, 2013 at 7:31 PM, Haohui Mai <hm...@hortonworks.com> wrote:
> > Thanks for the reply, Luke. Here I just echo my response from the jira:
> >
> > bq. this client-side js only approach, which is less secure than a
> > progressively enhanced hybrid approach used by YARN. The recent gmail
> > XSS fiasco highlights the issue.
> >
> > I'm presenting an informal security analysis to compare the security of
> the
> > old and the new web UIs.
> >
> > An attacker launches an XSS attack by injecting malicious code which are
> > usually HTML or JavaScript fragments into the web page, so that the
> > malicious code can have the same privileges of the web page.
> >
> > First, in the scope of XSS attacks, note that the threat models of
> > launching XSS attacks on Internet sites Gmail/Linkedin and the one of the
> > Hadoop UIs are different. They have fundamental different sets of
> external
> > inputs that the attackers have control to. Internet sites have little
> > control of these inputs. In the case of Gmail / Linkedin, an attack can
> > send you a crafted e-mail, or put malicious description in his /
> > her Linkedin profile. The sets of external inputs are *restricted* in
> > Hadoop UIs. The new web UIs take JMX and WebHDFS as inputs. The
> > attacker has to launch a XSS attack by:
> >
> > * Compromise the jars so that the output of JMX / WebHDFS have the
> > malicious code.
> > * Replace the web UIs completely to include the malicious code.
> >
> > In either case *the attacker has to compromise the hadoop core or the
> > namenode*. That means the new web UIs are at least as secure as the
> hadoop
> > core, and the namenode machine.
> >
> > Second, I argue that using client-side templates are more secure than the
> > current JSP-based server-side templates. To defend against XSS
> > attacks, both techniques have to filter the external inputs at *every*
> > possible execution paths. Several facts much be taken into
> > plays when evaluating the security of both approaches in real-world
> > environments:
> >
> > * The JavaScript libraries used in the new web UIs have survived in
> > extremely large-scale production tests. jQuery is used by Google and
> >  Microsoft, bootstrap is used by Twitter, and dust.js is used by
> Linkedin.
> > All libraries survived from hundreds of thousands of
> >  attack attempts on a daily basis. I agree that the libraries might still
> > be imperfect, but there's no way that we can test the JSP web
> >  UIs to achieve the same level of assurances given the amount of
> resources
> > the community has.
> > * Client-side templates consolidate all filtering logic in one central
> > place. Recall that the goal is to filter all external inputs at every
> >  execution paths, this is a much more systematic approach compared to the
> > server-side templates we have today. It is difficult (if not
> >  impossible) to do it in a JSP/ASP/PHP application, since such filtering
> > can be only achieved via ad-hoc approaches ([1] shows some
> >  empirical data). Also, HDFS-4901 recently describes a XSS vulnerability
> in
> > browseDirectory.jsp.
> >
> > bq. You'd require proper SSL (not self signed) setup to avoid JS
> > injection
> >
> > Commodity browsers enforce Same-Origin Policy to defend against code
> > injections. It has nothing to do with what kinds of SSL certificates
> > you hold.
> >
> > bq.  I also have concerns that we commit these changes without matching
> > unit tests
> >
> > The JavaScript code can be automatically tested. The same code can be run
> > by node.js and the test can compared with pre-defined
> > results. It is also possible to write an adapter to use Rhino to
> accomplish
> > the same task. We can discuss how to integrate them into
> > the maven test routines in a different thread.
> >
> > bq. Client side rendering completely breaks the workflows for ops who
> rely
> > on text based terminal/emacs/vim browsers (no js support) to
> > monitor component UI.
> >
> > links / elinks (http://elinks.or.cz/) are text-based web browsers that
> > support JavaScript.
> >
> > bq. The priority/requirements for UI in core Hadoop should be security
> and
> > correctness, which client side templating cannot address properly
> > so far.
> >
> > I agree that we should focus on security and correctness. The paragraphs
> > above explain that how the architecture of the new UIs
> > makes the UIs more secure in real-world settings compared to the UI we
> have
> > today.
> >
> > References:
> >
> > 1. A. Yip et al. Improving Application Security with Data Flow
> Assertions.
> > In SOSP'2009.
> >
> >
> > On Fri, Oct 25, 2013 at 10:02 AM, Luke Lu <ll...@apache.org> wrote:
> >
> >> Echoing my comments on HDFS-3555:
> >>
> >> I have concerns with this client-side js only approach, which is less
> >> secure than a progressively enhanced hybrid approach used by YARN. The
> >> recent gmail XSS fiasco highlights the issue. I also have concerns that
> we
> >> commit these changes without matching unit tests – the fact you cannot
> >> effectively unit test these changes should tell you something about this
> >> approach.
> >>
> >> *Requiring* JS means that an admin cannot turn off js to (partially) use
> >> core Hadoop UI. You'd *require* proper SSL (not self signed) setup to
> avoid
> >> JS injection, even if security of js libraries used is perfect, which I
> >> doubt (search gmail/linkedin XSS). Client side rendering completely
> breaks
> >> the workflows for ops who rely on text based terminal/emacs/vim browsers
> >> (no js support) to monitor component UI.
> >>
> >> IMO, JS-only rendering belongs to social networking sites and/or SaaS
> >> front-ends, where full time UI/security specialists babysits UI
> changes. I
> >> think eventually most users will use a self servicing UI in a SaaS
> >> front-end that uses REST/JMX API to get data from back-end components,
> >> besides their own app master/service UI. The priority/requirements for
> UI
> >> in core Hadoop should be security and correctness, which client side
> >> templating cannot address properly so far.
> >>
> >>
> >> On Tue, Oct 22, 2013 at 3:59 PM, Haohui Mai <hm...@hortonworks.com>
> wrote:
> >>
> >> > Hi all,
> >> >
> >> >
> >> > Jing Zhao and I recently have reimplemented the JSP-based web UIs in
> >> HTML 5
> >> > applications (HDFS-5333). Based on our prelimanary testing results we
> >> > believe thst the new web UIs of the namenodes and the datanode are
> ready
> >> > for everyday uses.
> >> >
> >> > You're more than welcome to try it out on trunk by visiting http://
> >> > <namenode>/dfshealth.html
> >> >
> >> > There are a number of benefits from this transition. From a
> developer's
> >> > prospective, the most notable one is *maintainability*:
> >> >
> >> > (1) The abstractions between the UI and the core server are
> well-defined,
> >> > decoupling the UI and the core hadoop servers.
> >> >
> >> > (2) It allows us to deprecate the logic in the JSP pages. The old web
> UIs
> >> > have to duplicate the logic in the JSPs. The logic is often
> out-of-dated
> >> > and not well-tested, which leads to broken pages and security
> >> > vulnerabilities(e.g. HDFS-5251, HDFS-5307, HDFS-5308, HDFS-5317 and
> >> > HDFS-4901). The architecture of the new UIs prevent these bugs at the
> >> very
> >> > beginning.
> >> >
> >> >
> >> > I propose that deprecate the old, JSP-based web UIs in 2.3. I opened
> >> > HDFS-5402 to track the relevant discussions.
> >> >
> >> > Your feedbacks are highly appreciated.
> >> >
> >> >
> >> > Sincerely,
> >> >
> >> > Haohui
> >> >
> >> > --
> >> > CONFIDENTIALITY NOTICE
> >> > NOTICE: This message is intended for the use of the individual or
> entity
> >> to
> >> > which it is addressed and may contain information that is
> confidential,
> >> > privileged and exempt from disclosure under applicable law. If the
> reader
> >> > of this message is not the intended recipient, you are hereby notified
> >> that
> >> > any printing, copying, dissemination, distribution, disclosure or
> >> > forwarding of this communication is strictly prohibited. If you have
> >> > received this communication in error, please contact the sender
> >> immediately
> >> > and delete it from your system. Thank You.
> >> >
> >>
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
>



-- 
Alejandro

Re: Replacing the JSP web UIs to HTML 5 applications

[Colin McCabe <cm...@alumni.cmu.edu>]

This is a really interesting project, Haohui.  I think it will make
our web UI much nicer.

I have a few concerns about removing the old web UI, however:

* If we're going to remove the old web UI, I think the new web UI has
to have the same level of unit testing.  We shouldn't go backwards in
terms of unit testing.

* Most of the deployments of elinks and links out there don't support
Javascript.  This is just a reality of life when using CentOS 5 or 6,
which many users are still using.  I have used "links" to diagnose
problems through the web UI in the past, in systems where access to
the cluster was available only through telnet.  If we are going to
remove this capability, we need to add some other command-line tools
to get the same functionality.  These tools could use REST if we have
that, or JMX, but they need to exist before we can consider removing
the old UI.

best,
Colin

On Fri, Oct 25, 2013 at 7:31 PM, Haohui Mai <hm...@hortonworks.com> wrote:
> Thanks for the reply, Luke. Here I just echo my response from the jira:
>
> bq. this client-side js only approach, which is less secure than a
> progressively enhanced hybrid approach used by YARN. The recent gmail
> XSS fiasco highlights the issue.
>
> I'm presenting an informal security analysis to compare the security of the
> old and the new web UIs.
>
> An attacker launches an XSS attack by injecting malicious code which are
> usually HTML or JavaScript fragments into the web page, so that the
> malicious code can have the same privileges of the web page.
>
> First, in the scope of XSS attacks, note that the threat models of
> launching XSS attacks on Internet sites Gmail/Linkedin and the one of the
> Hadoop UIs are different. They have fundamental different sets of external
> inputs that the attackers have control to. Internet sites have little
> control of these inputs. In the case of Gmail / Linkedin, an attack can
> send you a crafted e-mail, or put malicious description in his /
> her Linkedin profile. The sets of external inputs are *restricted* in
> Hadoop UIs. The new web UIs take JMX and WebHDFS as inputs. The
> attacker has to launch a XSS attack by:
>
> * Compromise the jars so that the output of JMX / WebHDFS have the
> malicious code.
> * Replace the web UIs completely to include the malicious code.
>
> In either case *the attacker has to compromise the hadoop core or the
> namenode*. That means the new web UIs are at least as secure as the hadoop
> core, and the namenode machine.
>
> Second, I argue that using client-side templates are more secure than the
> current JSP-based server-side templates. To defend against XSS
> attacks, both techniques have to filter the external inputs at *every*
> possible execution paths. Several facts much be taken into
> plays when evaluating the security of both approaches in real-world
> environments:
>
> * The JavaScript libraries used in the new web UIs have survived in
> extremely large-scale production tests. jQuery is used by Google and
>  Microsoft, bootstrap is used by Twitter, and dust.js is used by Linkedin.
> All libraries survived from hundreds of thousands of
>  attack attempts on a daily basis. I agree that the libraries might still
> be imperfect, but there's no way that we can test the JSP web
>  UIs to achieve the same level of assurances given the amount of resources
> the community has.
> * Client-side templates consolidate all filtering logic in one central
> place. Recall that the goal is to filter all external inputs at every
>  execution paths, this is a much more systematic approach compared to the
> server-side templates we have today. It is difficult (if not
>  impossible) to do it in a JSP/ASP/PHP application, since such filtering
> can be only achieved via ad-hoc approaches ([1] shows some
>  empirical data). Also, HDFS-4901 recently describes a XSS vulnerability in
> browseDirectory.jsp.
>
> bq. You'd require proper SSL (not self signed) setup to avoid JS
> injection
>
> Commodity browsers enforce Same-Origin Policy to defend against code
> injections. It has nothing to do with what kinds of SSL certificates
> you hold.
>
> bq.  I also have concerns that we commit these changes without matching
> unit tests
>
> The JavaScript code can be automatically tested. The same code can be run
> by node.js and the test can compared with pre-defined
> results. It is also possible to write an adapter to use Rhino to accomplish
> the same task. We can discuss how to integrate them into
> the maven test routines in a different thread.
>
> bq. Client side rendering completely breaks the workflows for ops who rely
> on text based terminal/emacs/vim browsers (no js support) to
> monitor component UI.
>
> links / elinks (http://elinks.or.cz/) are text-based web browsers that
> support JavaScript.
>
> bq. The priority/requirements for UI in core Hadoop should be security and
> correctness, which client side templating cannot address properly
> so far.
>
> I agree that we should focus on security and correctness. The paragraphs
> above explain that how the architecture of the new UIs
> makes the UIs more secure in real-world settings compared to the UI we have
> today.
>
> References:
>
> 1. A. Yip et al. Improving Application Security with Data Flow Assertions.
> In SOSP'2009.
>
>
> On Fri, Oct 25, 2013 at 10:02 AM, Luke Lu <ll...@apache.org> wrote:
>
>> Echoing my comments on HDFS-3555:
>>
>> I have concerns with this client-side js only approach, which is less
>> secure than a progressively enhanced hybrid approach used by YARN. The
>> recent gmail XSS fiasco highlights the issue. I also have concerns that we
>> commit these changes without matching unit tests – the fact you cannot
>> effectively unit test these changes should tell you something about this
>> approach.
>>
>> *Requiring* JS means that an admin cannot turn off js to (partially) use
>> core Hadoop UI. You'd *require* proper SSL (not self signed) setup to avoid
>> JS injection, even if security of js libraries used is perfect, which I
>> doubt (search gmail/linkedin XSS). Client side rendering completely breaks
>> the workflows for ops who rely on text based terminal/emacs/vim browsers
>> (no js support) to monitor component UI.
>>
>> IMO, JS-only rendering belongs to social networking sites and/or SaaS
>> front-ends, where full time UI/security specialists babysits UI changes. I
>> think eventually most users will use a self servicing UI in a SaaS
>> front-end that uses REST/JMX API to get data from back-end components,
>> besides their own app master/service UI. The priority/requirements for UI
>> in core Hadoop should be security and correctness, which client side
>> templating cannot address properly so far.
>>
>>
>> On Tue, Oct 22, 2013 at 3:59 PM, Haohui Mai <hm...@hortonworks.com> wrote:
>>
>> > Hi all,
>> >
>> >
>> > Jing Zhao and I recently have reimplemented the JSP-based web UIs in
>> HTML 5
>> > applications (HDFS-5333). Based on our prelimanary testing results we
>> > believe thst the new web UIs of the namenodes and the datanode are ready
>> > for everyday uses.
>> >
>> > You're more than welcome to try it out on trunk by visiting http://
>> > <namenode>/dfshealth.html
>> >
>> > There are a number of benefits from this transition. From a developer's
>> > prospective, the most notable one is *maintainability*:
>> >
>> > (1) The abstractions between the UI and the core server are well-defined,
>> > decoupling the UI and the core hadoop servers.
>> >
>> > (2) It allows us to deprecate the logic in the JSP pages. The old web UIs
>> > have to duplicate the logic in the JSPs. The logic is often out-of-dated
>> > and not well-tested, which leads to broken pages and security
>> > vulnerabilities(e.g. HDFS-5251, HDFS-5307, HDFS-5308, HDFS-5317 and
>> > HDFS-4901). The architecture of the new UIs prevent these bugs at the
>> very
>> > beginning.
>> >
>> >
>> > I propose that deprecate the old, JSP-based web UIs in 2.3. I opened
>> > HDFS-5402 to track the relevant discussions.
>> >
>> > Your feedbacks are highly appreciated.
>> >
>> >
>> > Sincerely,
>> >
>> > Haohui
>> >
>> > --
>> > CONFIDENTIALITY NOTICE
>> > NOTICE: This message is intended for the use of the individual or entity
>> to
>> > which it is addressed and may contain information that is confidential,
>> > privileged and exempt from disclosure under applicable law. If the reader
>> > of this message is not the intended recipient, you are hereby notified
>> that
>> > any printing, copying, dissemination, distribution, disclosure or
>> > forwarding of this communication is strictly prohibited. If you have
>> > received this communication in error, please contact the sender
>> immediately
>> > and delete it from your system. Thank You.
>> >
>>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.

Re: Replacing the JSP web UIs to HTML 5 applications

[Haohui Mai <hm...@hortonworks.com>]

Thanks for the reply, Luke. Here I just echo my response from the jira:

bq. this client-side js only approach, which is less secure than a
progressively enhanced hybrid approach used by YARN. The recent gmail
XSS fiasco highlights the issue.

I'm presenting an informal security analysis to compare the security of the
old and the new web UIs.

An attacker launches an XSS attack by injecting malicious code which are
usually HTML or JavaScript fragments into the web page, so that the
malicious code can have the same privileges of the web page.

First, in the scope of XSS attacks, note that the threat models of
launching XSS attacks on Internet sites Gmail/Linkedin and the one of the
Hadoop UIs are different. They have fundamental different sets of external
inputs that the attackers have control to. Internet sites have little
control of these inputs. In the case of Gmail / Linkedin, an attack can
send you a crafted e-mail, or put malicious description in his /
her Linkedin profile. The sets of external inputs are *restricted* in
Hadoop UIs. The new web UIs take JMX and WebHDFS as inputs. The
attacker has to launch a XSS attack by:

* Compromise the jars so that the output of JMX / WebHDFS have the
malicious code.
* Replace the web UIs completely to include the malicious code.

In either case *the attacker has to compromise the hadoop core or the
namenode*. That means the new web UIs are at least as secure as the hadoop
core, and the namenode machine.

Second, I argue that using client-side templates are more secure than the
current JSP-based server-side templates. To defend against XSS
attacks, both techniques have to filter the external inputs at *every*
possible execution paths. Several facts much be taken into
plays when evaluating the security of both approaches in real-world
environments:

* The JavaScript libraries used in the new web UIs have survived in
extremely large-scale production tests. jQuery is used by Google and
 Microsoft, bootstrap is used by Twitter, and dust.js is used by Linkedin.
All libraries survived from hundreds of thousands of
 attack attempts on a daily basis. I agree that the libraries might still
be imperfect, but there's no way that we can test the JSP web
 UIs to achieve the same level of assurances given the amount of resources
the community has.
* Client-side templates consolidate all filtering logic in one central
place. Recall that the goal is to filter all external inputs at every
 execution paths, this is a much more systematic approach compared to the
server-side templates we have today. It is difficult (if not
 impossible) to do it in a JSP/ASP/PHP application, since such filtering
can be only achieved via ad-hoc approaches ([1] shows some
 empirical data). Also, HDFS-4901 recently describes a XSS vulnerability in
browseDirectory.jsp.

bq. You'd require proper SSL (not self signed) setup to avoid JS
injection

Commodity browsers enforce Same-Origin Policy to defend against code
injections. It has nothing to do with what kinds of SSL certificates
you hold.

bq.  I also have concerns that we commit these changes without matching
unit tests

The JavaScript code can be automatically tested. The same code can be run
by node.js and the test can compared with pre-defined
results. It is also possible to write an adapter to use Rhino to accomplish
the same task. We can discuss how to integrate them into
the maven test routines in a different thread.

bq. Client side rendering completely breaks the workflows for ops who rely
on text based terminal/emacs/vim browsers (no js support) to
monitor component UI.

links / elinks (http://elinks.or.cz/) are text-based web browsers that
support JavaScript.

bq. The priority/requirements for UI in core Hadoop should be security and
correctness, which client side templating cannot address properly
so far.

I agree that we should focus on security and correctness. The paragraphs
above explain that how the architecture of the new UIs
makes the UIs more secure in real-world settings compared to the UI we have
today.

References:

1. A. Yip et al. Improving Application Security with Data Flow Assertions.
In SOSP'2009.


On Fri, Oct 25, 2013 at 10:02 AM, Luke Lu <ll...@apache.org> wrote:

> Echoing my comments on HDFS-3555:
>
> I have concerns with this client-side js only approach, which is less
> secure than a progressively enhanced hybrid approach used by YARN. The
> recent gmail XSS fiasco highlights the issue. I also have concerns that we
> commit these changes without matching unit tests – the fact you cannot
> effectively unit test these changes should tell you something about this
> approach.
>
> *Requiring* JS means that an admin cannot turn off js to (partially) use
> core Hadoop UI. You'd *require* proper SSL (not self signed) setup to avoid
> JS injection, even if security of js libraries used is perfect, which I
> doubt (search gmail/linkedin XSS). Client side rendering completely breaks
> the workflows for ops who rely on text based terminal/emacs/vim browsers
> (no js support) to monitor component UI.
>
> IMO, JS-only rendering belongs to social networking sites and/or SaaS
> front-ends, where full time UI/security specialists babysits UI changes. I
> think eventually most users will use a self servicing UI in a SaaS
> front-end that uses REST/JMX API to get data from back-end components,
> besides their own app master/service UI. The priority/requirements for UI
> in core Hadoop should be security and correctness, which client side
> templating cannot address properly so far.
>
>
> On Tue, Oct 22, 2013 at 3:59 PM, Haohui Mai <hm...@hortonworks.com> wrote:
>
> > Hi all,
> >
> >
> > Jing Zhao and I recently have reimplemented the JSP-based web UIs in
> HTML 5
> > applications (HDFS-5333). Based on our prelimanary testing results we
> > believe thst the new web UIs of the namenodes and the datanode are ready
> > for everyday uses.
> >
> > You're more than welcome to try it out on trunk by visiting http://
> > <namenode>/dfshealth.html
> >
> > There are a number of benefits from this transition. From a developer's
> > prospective, the most notable one is *maintainability*:
> >
> > (1) The abstractions between the UI and the core server are well-defined,
> > decoupling the UI and the core hadoop servers.
> >
> > (2) It allows us to deprecate the logic in the JSP pages. The old web UIs
> > have to duplicate the logic in the JSPs. The logic is often out-of-dated
> > and not well-tested, which leads to broken pages and security
> > vulnerabilities(e.g. HDFS-5251, HDFS-5307, HDFS-5308, HDFS-5317 and
> > HDFS-4901). The architecture of the new UIs prevent these bugs at the
> very
> > beginning.
> >
> >
> > I propose that deprecate the old, JSP-based web UIs in 2.3. I opened
> > HDFS-5402 to track the relevant discussions.
> >
> > Your feedbacks are highly appreciated.
> >
> >
> > Sincerely,
> >
> > Haohui
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> >
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Replacing the JSP web UIs to HTML 5 applications

[Luke Lu <ll...@apache.org>]

Echoing my comments on HDFS-3555:

I have concerns with this client-side js only approach, which is less
secure than a progressively enhanced hybrid approach used by YARN. The
recent gmail XSS fiasco highlights the issue. I also have concerns that we
commit these changes without matching unit tests – the fact you cannot
effectively unit test these changes should tell you something about this
approach.

*Requiring* JS means that an admin cannot turn off js to (partially) use
core Hadoop UI. You'd *require* proper SSL (not self signed) setup to avoid
JS injection, even if security of js libraries used is perfect, which I
doubt (search gmail/linkedin XSS). Client side rendering completely breaks
the workflows for ops who rely on text based terminal/emacs/vim browsers
(no js support) to monitor component UI.

IMO, JS-only rendering belongs to social networking sites and/or SaaS
front-ends, where full time UI/security specialists babysits UI changes. I
think eventually most users will use a self servicing UI in a SaaS
front-end that uses REST/JMX API to get data from back-end components,
besides their own app master/service UI. The priority/requirements for UI
in core Hadoop should be security and correctness, which client side
templating cannot address properly so far.


On Tue, Oct 22, 2013 at 3:59 PM, Haohui Mai <hm...@hortonworks.com> wrote:

> Hi all,
>
>
> Jing Zhao and I recently have reimplemented the JSP-based web UIs in HTML 5
> applications (HDFS-5333). Based on our prelimanary testing results we
> believe thst the new web UIs of the namenodes and the datanode are ready
> for everyday uses.
>
> You're more than welcome to try it out on trunk by visiting http://
> <namenode>/dfshealth.html
>
> There are a number of benefits from this transition. From a developer's
> prospective, the most notable one is *maintainability*:
>
> (1) The abstractions between the UI and the core server are well-defined,
> decoupling the UI and the core hadoop servers.
>
> (2) It allows us to deprecate the logic in the JSP pages. The old web UIs
> have to duplicate the logic in the JSPs. The logic is often out-of-dated
> and not well-tested, which leads to broken pages and security
> vulnerabilities(e.g. HDFS-5251, HDFS-5307, HDFS-5308, HDFS-5317 and
> HDFS-4901). The architecture of the new UIs prevent these bugs at the very
> beginning.
>
>
> I propose that deprecate the old, JSP-based web UIs in 2.3. I opened
> HDFS-5402 to track the relevant discussions.
>
> Your feedbacks are highly appreciated.
>
>
> Sincerely,
>
> Haohui
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>