Re: Catching the webui up to features

[Benjamin Mahler <bm...@apache.org>]

What does localhost vs u64 represent for you there in terms of your actual
cluster?

Are you expecting to be able to publicly access the masters and agents? Or
are you running a proxy (like DC/OS's adminrouter)?
We recently made some changes that makes it easier to patch the UI to work
with a proxy.

Would be great to understand how you'd like it to work.

Ben

On Tue, Sep 26, 2017 at 9:15 AM, Aaron Wood <aa...@gmail.com> wrote:

> You're right about the slaves section in 0.28.x. There are no error
> messages or modals that pop up but looking at the requests after you click
> on a slave in the slaves section shows these obvious failures (u64 is the
> hostname of my VM which won't be resolved from the host):
>
> Request URL:
> http://u64:5051/slave(1)/state?jsonp=angular.callbacks._6
> Request URL:
> http://u64:5051/metrics/snapshot?jsonp=angular.callbacks._7
> Request URL:
> http://u64:5051/monitor/statistics?jsonp=angular.callbacks._8
>
> What does work in 0.28.x (and doesn't in 1.0.x) is the main page which has
> AJAX calls on an interval. These are successfully completed every 10
> seconds or so:
>
> Request URL:
> http://localhost:10050/master/state
> Request URL:
> http://localhost:10050/metrics/snapshot
>
> In the newer versions of Mesos these requests are a bit different and fail:
>
> Request URL:
> http://u64:5050/master/state?jsonp=angular.callbacks._2
> Request URL:
> http://u64:5050/metrics/snapshot?jsonp=angular.callbacks._3
>
> This is what causes the error modal to pop up on every interval tick of
> these AJAX requests.
>
> Thanks,
> Aaron
>
> On Mon, Sep 25, 2017 at 7:47 PM, Benjamin Mahler <bm...@apache.org>
> wrote:
>
> > 0.28.x UI is working for you even when you click into an agent?
> >
> > Can you show us which requests are being made in the old UI vs the new UI
> > and which ones do not work for you? (use chrome's built-in debugging
> tools)
> >
> > On Mon, Sep 25, 2017 at 3:38 PM, Aaron Wood <aa...@gmail.com>
> wrote:
> >
> > > I just built Mesos 0.28.x and tried out this scenario. It seems that
> this
> > > issue presents itself only in 1.0.x and up. I also found out that DCOS
> > > hides this issue if someone uses https://github.com/dcos/adminrouter
> > >
> > > I have not dug through the 0.28.x code enough to pinpoint the exact
> area
> > > for where the logic is different. Maybe someone that has worked on the
> UI
> > > in the past can shed more light on this...?
> > >
> > > On Wed, Sep 13, 2017 at 8:03 PM, Benjamin Mahler <bm...@apache.org>
> > > wrote:
> > >
> > > > +haosdent, vinod
> > > >
> > > > Blaming the lines you pointed to gives me:
> > > >
> > > > https://github.com/apache/mesos/commit/
> b2fc58883e2cd0ca144fd1b0e10cad
> > > > 4235a50223
> > > >
> > > > I've added vinod and haosdent since they were involved in the change,
> > it
> > > > looks like this makes the master talk directly to a different master
> > > > host:port. Is this what breaks for you?
> > > >
> > > > If so, I'm not sure why pre-1.0.x would work for you if you were to
> > click
> > > > into an agent, as that will also make a request directly to an agent
> > > > host:port.
> > > >
> > > > Ben
> > > >
> > > > On Mon, Sep 11, 2017 at 7:44 PM, Aaron Wood <aa...@gmail.com>
> > > wrote:
> > > >
> > > > > Hey Ben,
> > > > >
> > > > > There was a change sometime after Mesos 1.0.x (I think) that
> altered
> > > how
> > > > > the leader state was obtained
> > > > > https://github.com/apache/mesos/blob/master/src/webui/
> > > > > master/static/js/controllers.js#L362-L364
> > > > >
> > > > > This seems to be an intended change but this causes UI error modals
> > to
> > > > > continually pop up stating that the leader could not be reached
> when
> > > > > accessing the UI over a tunnel. The errors also show when you try
> to
> > > > access
> > > > > agent info among other things throughout the angular app.
> > > > >
> > > > > I'm curious, how many people using Mesos access the UI over a
> tunnel?
> > > Is
> > > > > there any harm in reverting this functionality to how it was in the
> > > 1.0.x
> > > > > days? I'm sure there was some important reason as to why this
> change
> > > was
> > > > > made, I'm just not aware of it. From my understanding many of the
> ops
> > > > folks
> > > > > in our org use the UI but almost always over a tunnel. Once more
> > groups
> > > > in
> > > > > the org upgrade to newer versions of Mesos they will be facing this
> > > issue
> > > > > almost on a daily basis.
> > > > >
> > > > > Thanks,
> > > > > Aaron
> > > > >
> > > > > On Mon, Sep 11, 2017 at 2:59 PM, Benjamin Mahler <
> bmahler@apache.org
> > >
> > > > > wrote:
> > > > >
> > > > > > Hi folks,
> > > > > >
> > > > > > Over time the webui has lagged behind for some of the features
> that
> > > > have
> > > > > > been added. I'm currently tracking what's required to catch it up
> > > here:
> > > > > >
> > > > > > https://issues.apache.org/jira/browse/MESOS-6440
> > > > > >
> > > > > > If you know of other features that make sense to display in the
> > > webui,
> > > > > feel
> > > > > > free to file a ticket under this epic (or link it as related if
> it
> > > > falls
> > > > > > under a different epic) and let me know. For example, I just
> filed
> > > > > another
> > > > > > one within it for displaying task health information.
> > > > > >
> > > > > > Also feel free to make contributions to the webui even if you
> don't
> > > > feel
> > > > > > that you're knowledgable on the frontend side of things. The
> > majority
> > > > of
> > > > > > webui changes are very easy and provide a lot of value to users
> who
> > > > > > interact with it on a regular basis!
> > > > > >
> > > > > > If you'd like to contribute to the webui, there are a lot of easy
> > > > tickets
> > > > > > to get started with, here is one example that I would be happy to
> > > > assist
> > > > > > with: https://issues.apache.org/jira/browse/MESOS-7962
> > > > > >
> > > > > > Thanks!
> > > > > > Ben
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Catching the webui up to features

[Timothy Anderegg <ti...@gmail.com>]

We have a similar scenario using an actual proxy (Apache in this case) in
front of a firewalled VLAN, that also does Kerberos authentication against
the endpoints.  This currently works fine for external API requests but the
WebUI is only partially functional, since I haven't looked into the newer
Mesos features to support proxying that Ben mentioned.  I will try to
implement those and see if I can get it all working as it should.  Is there
documentation on those features?  If not I will figure it out and submit
some :)

YARN handles this differently (like Aaron thought Mesos would), such that
the web UI has its own proxy that allows access to some of the services
running on it without having to do more complex standalone proxy
configurations or service discovery, but implementing that for Mesos would
certainly increase the complexity of Mesos' web UI.  This does enable some
nice features such as being able to access the Spark UI automatically via
the Yarn UI, without having to worry about where the driver is running or
what port it is on.

- Tim

On Wed, Oct 18, 2017 at 1:11 PM Aaron Wood <aa...@gmail.com> wrote:

> In this case localhost is my host (OSX) and u64 is my guest (Ubuntu VM).
> What I was doing was viewing the Mesos UI from my host by making use of
> port mapping in my VM. Port 10050 on my host maps to 5050 inside my VM. So
> in a way this is like viewing the UI with a proxy in the middle. What I was
> expecting was that I could view the UI served from the master and not have
> to worry about reaching the agents directly for their information. I'm not
> using DC/OS at all here, just my own builds of Apache Mesos.
>
> I was thinking that if the master was able to handle the communication with
> the agents (and other masters if the leader changes) that people would be
> able to use the UI in situations like mine, or more realistically, in
> situations where you have some sort of cluster that you need to tunnel into
> in order to reach Mesos. For example, Mesos is not exposed outside of our
> cluster directly since the cluster has its own private internal network.
> You have to tunnel in using our defined node(s) that are on both the
> external and internal network.
>
> I hope that helps explain the scenario!
>
> Thanks,
> Aaron
>
> On Tue, Oct 17, 2017 at 3:05 PM, Benjamin Mahler <bm...@apache.org>
> wrote:
>
> > What does localhost vs u64 represent for you there in terms of your
> actual
> > cluster?
> >
> > Are you expecting to be able to publicly access the masters and agents?
> Or
> > are you running a proxy (like DC/OS's adminrouter)?
> > We recently made some changes that makes it easier to patch the UI to
> work
> > with a proxy.
> >
> > Would be great to understand how you'd like it to work.
> >
> > Ben
> >
> > On Tue, Sep 26, 2017 at 9:15 AM, Aaron Wood <aa...@gmail.com>
> wrote:
> >
> > > You're right about the slaves section in 0.28.x. There are no error
> > > messages or modals that pop up but looking at the requests after you
> > click
> > > on a slave in the slaves section shows these obvious failures (u64 is
> the
> > > hostname of my VM which won't be resolved from the host):
> > >
> > > Request URL:
> > > http://u64:5051/slave(1)/state?jsonp=angular.callbacks._6
> > > Request URL:
> > > http://u64:5051/metrics/snapshot?jsonp=angular.callbacks._7
> > > Request URL:
> > > http://u64:5051/monitor/statistics?jsonp=angular.callbacks._8
> > >
> > > What does work in 0.28.x (and doesn't in 1.0.x) is the main page which
> > has
> > > AJAX calls on an interval. These are successfully completed every 10
> > > seconds or so:
> > >
> > > Request URL:
> > > http://localhost:10050/master/state
> > > Request URL:
> > > http://localhost:10050/metrics/snapshot
> > >
> > > In the newer versions of Mesos these requests are a bit different and
> > fail:
> > >
> > > Request URL:
> > > http://u64:5050/master/state?jsonp=angular.callbacks._2
> > > Request URL:
> > > http://u64:5050/metrics/snapshot?jsonp=angular.callbacks._3
> > >
> > > This is what causes the error modal to pop up on every interval tick of
> > > these AJAX requests.
> > >
> > > Thanks,
> > > Aaron
> > >
> > > On Mon, Sep 25, 2017 at 7:47 PM, Benjamin Mahler <bm...@apache.org>
> > > wrote:
> > >
> > > > 0.28.x UI is working for you even when you click into an agent?
> > > >
> > > > Can you show us which requests are being made in the old UI vs the
> new
> > UI
> > > > and which ones do not work for you? (use chrome's built-in debugging
> > > tools)
> > > >
> > > > On Mon, Sep 25, 2017 at 3:38 PM, Aaron Wood <aa...@gmail.com>
> > > wrote:
> > > >
> > > > > I just built Mesos 0.28.x and tried out this scenario. It seems
> that
> > > this
> > > > > issue presents itself only in 1.0.x and up. I also found out that
> > DCOS
> > > > > hides this issue if someone uses
> https://github.com/dcos/adminrouter
> > > > >
> > > > > I have not dug through the 0.28.x code enough to pinpoint the exact
> > > area
> > > > > for where the logic is different. Maybe someone that has worked on
> > the
> > > UI
> > > > > in the past can shed more light on this...?
> > > > >
> > > > > On Wed, Sep 13, 2017 at 8:03 PM, Benjamin Mahler <
> bmahler@apache.org
> > >
> > > > > wrote:
> > > > >
> > > > > > +haosdent, vinod
> > > > > >
> > > > > > Blaming the lines you pointed to gives me:
> > > > > >
> > > > > > https://github.com/apache/mesos/commit/
> > > b2fc58883e2cd0ca144fd1b0e10cad
> > > > > > 4235a50223
> > > > > >
> > > > > > I've added vinod and haosdent since they were involved in the
> > change,
> > > > it
> > > > > > looks like this makes the master talk directly to a different
> > master
> > > > > > host:port. Is this what breaks for you?
> > > > > >
> > > > > > If so, I'm not sure why pre-1.0.x would work for you if you were
> to
> > > > click
> > > > > > into an agent, as that will also make a request directly to an
> > agent
> > > > > > host:port.
> > > > > >
> > > > > > Ben
> > > > > >
> > > > > > On Mon, Sep 11, 2017 at 7:44 PM, Aaron Wood <
> aaronjwood@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > Hey Ben,
> > > > > > >
> > > > > > > There was a change sometime after Mesos 1.0.x (I think) that
> > > altered
> > > > > how
> > > > > > > the leader state was obtained
> > > > > > > https://github.com/apache/mesos/blob/master/src/webui/
> > > > > > > master/static/js/controllers.js#L362-L364
> > > > > > >
> > > > > > > This seems to be an intended change but this causes UI error
> > modals
> > > > to
> > > > > > > continually pop up stating that the leader could not be reached
> > > when
> > > > > > > accessing the UI over a tunnel. The errors also show when you
> try
> > > to
> > > > > > access
> > > > > > > agent info among other things throughout the angular app.
> > > > > > >
> > > > > > > I'm curious, how many people using Mesos access the UI over a
> > > tunnel?
> > > > > Is
> > > > > > > there any harm in reverting this functionality to how it was in
> > the
> > > > > 1.0.x
> > > > > > > days? I'm sure there was some important reason as to why this
> > > change
> > > > > was
> > > > > > > made, I'm just not aware of it. From my understanding many of
> the
> > > ops
> > > > > > folks
> > > > > > > in our org use the UI but almost always over a tunnel. Once
> more
> > > > groups
> > > > > > in
> > > > > > > the org upgrade to newer versions of Mesos they will be facing
> > this
> > > > > issue
> > > > > > > almost on a daily basis.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Aaron
> > > > > > >
> > > > > > > On Mon, Sep 11, 2017 at 2:59 PM, Benjamin Mahler <
> > > bmahler@apache.org
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi folks,
> > > > > > > >
> > > > > > > > Over time the webui has lagged behind for some of the
> features
> > > that
> > > > > > have
> > > > > > > > been added. I'm currently tracking what's required to catch
> it
> > up
> > > > > here:
> > > > > > > >
> > > > > > > > https://issues.apache.org/jira/browse/MESOS-6440
> > > > > > > >
> > > > > > > > If you know of other features that make sense to display in
> the
> > > > > webui,
> > > > > > > feel
> > > > > > > > free to file a ticket under this epic (or link it as related
> if
> > > it
> > > > > > falls
> > > > > > > > under a different epic) and let me know. For example, I just
> > > filed
> > > > > > > another
> > > > > > > > one within it for displaying task health information.
> > > > > > > >
> > > > > > > > Also feel free to make contributions to the webui even if you
> > > don't
> > > > > > feel
> > > > > > > > that you're knowledgable on the frontend side of things. The
> > > > majority
> > > > > > of
> > > > > > > > webui changes are very easy and provide a lot of value to
> users
> > > who
> > > > > > > > interact with it on a regular basis!
> > > > > > > >
> > > > > > > > If you'd like to contribute to the webui, there are a lot of
> > easy
> > > > > > tickets
> > > > > > > > to get started with, here is one example that I would be
> happy
> > to
> > > > > > assist
> > > > > > > > with: https://issues.apache.org/jira/browse/MESOS-7962
> > > > > > > >
> > > > > > > > Thanks!
> > > > > > > > Ben
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Catching the webui up to features

[Aaron Wood <aa...@gmail.com>]

In this case localhost is my host (OSX) and u64 is my guest (Ubuntu VM).
What I was doing was viewing the Mesos UI from my host by making use of
port mapping in my VM. Port 10050 on my host maps to 5050 inside my VM. So
in a way this is like viewing the UI with a proxy in the middle. What I was
expecting was that I could view the UI served from the master and not have
to worry about reaching the agents directly for their information. I'm not
using DC/OS at all here, just my own builds of Apache Mesos.

I was thinking that if the master was able to handle the communication with
the agents (and other masters if the leader changes) that people would be
able to use the UI in situations like mine, or more realistically, in
situations where you have some sort of cluster that you need to tunnel into
in order to reach Mesos. For example, Mesos is not exposed outside of our
cluster directly since the cluster has its own private internal network.
You have to tunnel in using our defined node(s) that are on both the
external and internal network.

I hope that helps explain the scenario!

Thanks,
Aaron

On Tue, Oct 17, 2017 at 3:05 PM, Benjamin Mahler <bm...@apache.org> wrote:

> What does localhost vs u64 represent for you there in terms of your actual
> cluster?
>
> Are you expecting to be able to publicly access the masters and agents? Or
> are you running a proxy (like DC/OS's adminrouter)?
> We recently made some changes that makes it easier to patch the UI to work
> with a proxy.
>
> Would be great to understand how you'd like it to work.
>
> Ben
>
> On Tue, Sep 26, 2017 at 9:15 AM, Aaron Wood <aa...@gmail.com> wrote:
>
> > You're right about the slaves section in 0.28.x. There are no error
> > messages or modals that pop up but looking at the requests after you
> click
> > on a slave in the slaves section shows these obvious failures (u64 is the
> > hostname of my VM which won't be resolved from the host):
> >
> > Request URL:
> > http://u64:5051/slave(1)/state?jsonp=angular.callbacks._6
> > Request URL:
> > http://u64:5051/metrics/snapshot?jsonp=angular.callbacks._7
> > Request URL:
> > http://u64:5051/monitor/statistics?jsonp=angular.callbacks._8
> >
> > What does work in 0.28.x (and doesn't in 1.0.x) is the main page which
> has
> > AJAX calls on an interval. These are successfully completed every 10
> > seconds or so:
> >
> > Request URL:
> > http://localhost:10050/master/state
> > Request URL:
> > http://localhost:10050/metrics/snapshot
> >
> > In the newer versions of Mesos these requests are a bit different and
> fail:
> >
> > Request URL:
> > http://u64:5050/master/state?jsonp=angular.callbacks._2
> > Request URL:
> > http://u64:5050/metrics/snapshot?jsonp=angular.callbacks._3
> >
> > This is what causes the error modal to pop up on every interval tick of
> > these AJAX requests.
> >
> > Thanks,
> > Aaron
> >
> > On Mon, Sep 25, 2017 at 7:47 PM, Benjamin Mahler <bm...@apache.org>
> > wrote:
> >
> > > 0.28.x UI is working for you even when you click into an agent?
> > >
> > > Can you show us which requests are being made in the old UI vs the new
> UI
> > > and which ones do not work for you? (use chrome's built-in debugging
> > tools)
> > >
> > > On Mon, Sep 25, 2017 at 3:38 PM, Aaron Wood <aa...@gmail.com>
> > wrote:
> > >
> > > > I just built Mesos 0.28.x and tried out this scenario. It seems that
> > this
> > > > issue presents itself only in 1.0.x and up. I also found out that
> DCOS
> > > > hides this issue if someone uses https://github.com/dcos/adminrouter
> > > >
> > > > I have not dug through the 0.28.x code enough to pinpoint the exact
> > area
> > > > for where the logic is different. Maybe someone that has worked on
> the
> > UI
> > > > in the past can shed more light on this...?
> > > >
> > > > On Wed, Sep 13, 2017 at 8:03 PM, Benjamin Mahler <bmahler@apache.org
> >
> > > > wrote:
> > > >
> > > > > +haosdent, vinod
> > > > >
> > > > > Blaming the lines you pointed to gives me:
> > > > >
> > > > > https://github.com/apache/mesos/commit/
> > b2fc58883e2cd0ca144fd1b0e10cad
> > > > > 4235a50223
> > > > >
> > > > > I've added vinod and haosdent since they were involved in the
> change,
> > > it
> > > > > looks like this makes the master talk directly to a different
> master
> > > > > host:port. Is this what breaks for you?
> > > > >
> > > > > If so, I'm not sure why pre-1.0.x would work for you if you were to
> > > click
> > > > > into an agent, as that will also make a request directly to an
> agent
> > > > > host:port.
> > > > >
> > > > > Ben
> > > > >
> > > > > On Mon, Sep 11, 2017 at 7:44 PM, Aaron Wood <aa...@gmail.com>
> > > > wrote:
> > > > >
> > > > > > Hey Ben,
> > > > > >
> > > > > > There was a change sometime after Mesos 1.0.x (I think) that
> > altered
> > > > how
> > > > > > the leader state was obtained
> > > > > > https://github.com/apache/mesos/blob/master/src/webui/
> > > > > > master/static/js/controllers.js#L362-L364
> > > > > >
> > > > > > This seems to be an intended change but this causes UI error
> modals
> > > to
> > > > > > continually pop up stating that the leader could not be reached
> > when
> > > > > > accessing the UI over a tunnel. The errors also show when you try
> > to
> > > > > access
> > > > > > agent info among other things throughout the angular app.
> > > > > >
> > > > > > I'm curious, how many people using Mesos access the UI over a
> > tunnel?
> > > > Is
> > > > > > there any harm in reverting this functionality to how it was in
> the
> > > > 1.0.x
> > > > > > days? I'm sure there was some important reason as to why this
> > change
> > > > was
> > > > > > made, I'm just not aware of it. From my understanding many of the
> > ops
> > > > > folks
> > > > > > in our org use the UI but almost always over a tunnel. Once more
> > > groups
> > > > > in
> > > > > > the org upgrade to newer versions of Mesos they will be facing
> this
> > > > issue
> > > > > > almost on a daily basis.
> > > > > >
> > > > > > Thanks,
> > > > > > Aaron
> > > > > >
> > > > > > On Mon, Sep 11, 2017 at 2:59 PM, Benjamin Mahler <
> > bmahler@apache.org
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi folks,
> > > > > > >
> > > > > > > Over time the webui has lagged behind for some of the features
> > that
> > > > > have
> > > > > > > been added. I'm currently tracking what's required to catch it
> up
> > > > here:
> > > > > > >
> > > > > > > https://issues.apache.org/jira/browse/MESOS-6440
> > > > > > >
> > > > > > > If you know of other features that make sense to display in the
> > > > webui,
> > > > > > feel
> > > > > > > free to file a ticket under this epic (or link it as related if
> > it
> > > > > falls
> > > > > > > under a different epic) and let me know. For example, I just
> > filed
> > > > > > another
> > > > > > > one within it for displaying task health information.
> > > > > > >
> > > > > > > Also feel free to make contributions to the webui even if you
> > don't
> > > > > feel
> > > > > > > that you're knowledgable on the frontend side of things. The
> > > majority
> > > > > of
> > > > > > > webui changes are very easy and provide a lot of value to users
> > who
> > > > > > > interact with it on a regular basis!
> > > > > > >
> > > > > > > If you'd like to contribute to the webui, there are a lot of
> easy
> > > > > tickets
> > > > > > > to get started with, here is one example that I would be happy
> to
> > > > > assist
> > > > > > > with: https://issues.apache.org/jira/browse/MESOS-7962
> > > > > > >
> > > > > > > Thanks!
> > > > > > > Ben
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>