Deploying Topologies problem

[Lars Francke <la...@gmail.com>]

I was facing another weird issue today that took me a while to chase down.

I didn't have the proper services directory in place when starting Knox for
the first time. This meant that a gateway.xml file was created in the
deployments dir for my topologies that was totally empty. It didn't have
any filters so all my tests returned a "Failed to match path" error.

The problem here is that Knox checks on startup what the timestamp on the
topology is and only if that has changed does it create a new deployment.
If anything else changes this is not rewritten. So changes to the services
files will not be noticed by Knox and silently ignored.

I understand that this is an edge-case but other than a slightly longer
startup would it hurt to recreate the deployments on every startup?

Cheers,
Lars

Re: Deploying Topologies problem

[Lars Francke <la...@gmail.com>]

The removal part was an accident and it took me almost a full day to debug
why things didn't work. I didn't even know about the generated stuff. I
agree that it's a fringe case though.

On Fri, Nov 30, 2018 at 1:32 AM larry mccay <lm...@apache.org> wrote:

> I don't know why you would remove the data/services directory but this is
> essentially the same as what I described.
> Whether there are no service definitions or you have changed one and try to
> get the new version uptaken it is the same problem.
>
> As I said, we could add an option to redeploy all topologies on startup but
> we would definitely not want to make this default setting.
> I know of some deployments that have LOTS of topologies and redeploying
> them all on every restart would be needlessly expensive.
>
> On Thu, Nov 29, 2018 at 6:27 PM Lars Francke <la...@gmail.com>
> wrote:
>
> > There's always a good chance that I misunderstood something as well. So
> > take it with a grain of salt.
> >
> > 1) Fresh Knox installation
> > 2) Remove data/services folder
> > 3) Start Knox
> > 4) Look
> > at /data/deployments/sandbox.topo.166f2486250/%2F/WEB-INF/gateway.xml
> > It'll contain only "<?xml version="1.0" encoding="UTF-8"?><gateway/>"
> > 5) Stop Knox
> > 6) Move data/services in place
> > 7) Start Knox again
> >
> > Knox now will _not_ update the generated gateway.xml file. To me, that is
> > unexpected. Especially because the whole generation stuff is not
> documented
> > while the services are.
> > This is because Knox looks at the timestamp of the topology file,
> generates
> > a hash out of it and if that's unchanged it keeps the old gateway.xml
> file.
> >
> >
> > On Wed, Nov 28, 2018 at 9:35 PM larry mccay <lm...@apache.org> wrote:
> >
> > > It isn't exactly clear to me what you encountered here.
> > > I think that you are saying that you made changes to "services files" -
> > > meaning the service definition.
> > >
> > > If that is the case then a simple redeploy of the topology will not be
> > > enough.
> > > You have to restart the gateway so that the service definitions are
> > > reloaded then unfortunately you have to touch the topology file/s that
> > host
> > > the service that you changed the definition for.
> > >
> > > There is an undocumented config element in gateway-site.xml to force
> auto
> > > deployment on start for specific topologies - it is a comma separated
> > list
> > > of topology name:
> > > gateway.auto.deploy.topologies
> > >
> > > It defaults to "manager,admin"
> > >
> > > You could add your topologies to that and have them redeployed at
> restart
> > > if you like - make sure to leave manager and admin in there though.
> > >
> > > If you would like a config item that redeploys each topology on startup
> > > then please feel free to file a JIRA for it and we can consider it for
> > > 1.3.0.
> > >
> > > On Wed, Nov 28, 2018 at 3:01 PM Lars Francke <la...@gmail.com>
> > > wrote:
> > >
> > > > I was facing another weird issue today that took me a while to chase
> > > down.
> > > >
> > > > I didn't have the proper services directory in place when starting
> Knox
> > > for
> > > > the first time. This meant that a gateway.xml file was created in the
> > > > deployments dir for my topologies that was totally empty. It didn't
> > have
> > > > any filters so all my tests returned a "Failed to match path" error.
> > > >
> > > > The problem here is that Knox checks on startup what the timestamp on
> > the
> > > > topology is and only if that has changed does it create a new
> > deployment.
> > > > If anything else changes this is not rewritten. So changes to the
> > > services
> > > > files will not be noticed by Knox and silently ignored.
> > > >
> > > > I understand that this is an edge-case but other than a slightly
> longer
> > > > startup would it hurt to recreate the deployments on every startup?
> > > >
> > > > Cheers,
> > > > Lars
> > > >
> > >
> >
>

Re: Deploying Topologies problem

[larry mccay <lm...@apache.org>]

I don't know why you would remove the data/services directory but this is
essentially the same as what I described.
Whether there are no service definitions or you have changed one and try to
get the new version uptaken it is the same problem.

As I said, we could add an option to redeploy all topologies on startup but
we would definitely not want to make this default setting.
I know of some deployments that have LOTS of topologies and redeploying
them all on every restart would be needlessly expensive.

On Thu, Nov 29, 2018 at 6:27 PM Lars Francke <la...@gmail.com> wrote:

> There's always a good chance that I misunderstood something as well. So
> take it with a grain of salt.
>
> 1) Fresh Knox installation
> 2) Remove data/services folder
> 3) Start Knox
> 4) Look
> at /data/deployments/sandbox.topo.166f2486250/%2F/WEB-INF/gateway.xml
> It'll contain only "<?xml version="1.0" encoding="UTF-8"?><gateway/>"
> 5) Stop Knox
> 6) Move data/services in place
> 7) Start Knox again
>
> Knox now will _not_ update the generated gateway.xml file. To me, that is
> unexpected. Especially because the whole generation stuff is not documented
> while the services are.
> This is because Knox looks at the timestamp of the topology file, generates
> a hash out of it and if that's unchanged it keeps the old gateway.xml file.
>
>
> On Wed, Nov 28, 2018 at 9:35 PM larry mccay <lm...@apache.org> wrote:
>
> > It isn't exactly clear to me what you encountered here.
> > I think that you are saying that you made changes to "services files" -
> > meaning the service definition.
> >
> > If that is the case then a simple redeploy of the topology will not be
> > enough.
> > You have to restart the gateway so that the service definitions are
> > reloaded then unfortunately you have to touch the topology file/s that
> host
> > the service that you changed the definition for.
> >
> > There is an undocumented config element in gateway-site.xml to force auto
> > deployment on start for specific topologies - it is a comma separated
> list
> > of topology name:
> > gateway.auto.deploy.topologies
> >
> > It defaults to "manager,admin"
> >
> > You could add your topologies to that and have them redeployed at restart
> > if you like - make sure to leave manager and admin in there though.
> >
> > If you would like a config item that redeploys each topology on startup
> > then please feel free to file a JIRA for it and we can consider it for
> > 1.3.0.
> >
> > On Wed, Nov 28, 2018 at 3:01 PM Lars Francke <la...@gmail.com>
> > wrote:
> >
> > > I was facing another weird issue today that took me a while to chase
> > down.
> > >
> > > I didn't have the proper services directory in place when starting Knox
> > for
> > > the first time. This meant that a gateway.xml file was created in the
> > > deployments dir for my topologies that was totally empty. It didn't
> have
> > > any filters so all my tests returned a "Failed to match path" error.
> > >
> > > The problem here is that Knox checks on startup what the timestamp on
> the
> > > topology is and only if that has changed does it create a new
> deployment.
> > > If anything else changes this is not rewritten. So changes to the
> > services
> > > files will not be noticed by Knox and silently ignored.
> > >
> > > I understand that this is an edge-case but other than a slightly longer
> > > startup would it hurt to recreate the deployments on every startup?
> > >
> > > Cheers,
> > > Lars
> > >
> >
>

Re: Deploying Topologies problem

[Lars Francke <la...@gmail.com>]

There's always a good chance that I misunderstood something as well. So
take it with a grain of salt.

1) Fresh Knox installation
2) Remove data/services folder
3) Start Knox
4) Look
at /data/deployments/sandbox.topo.166f2486250/%2F/WEB-INF/gateway.xml
It'll contain only "<?xml version="1.0" encoding="UTF-8"?><gateway/>"
5) Stop Knox
6) Move data/services in place
7) Start Knox again

Knox now will _not_ update the generated gateway.xml file. To me, that is
unexpected. Especially because the whole generation stuff is not documented
while the services are.
This is because Knox looks at the timestamp of the topology file, generates
a hash out of it and if that's unchanged it keeps the old gateway.xml file.


On Wed, Nov 28, 2018 at 9:35 PM larry mccay <lm...@apache.org> wrote:

> It isn't exactly clear to me what you encountered here.
> I think that you are saying that you made changes to "services files" -
> meaning the service definition.
>
> If that is the case then a simple redeploy of the topology will not be
> enough.
> You have to restart the gateway so that the service definitions are
> reloaded then unfortunately you have to touch the topology file/s that host
> the service that you changed the definition for.
>
> There is an undocumented config element in gateway-site.xml to force auto
> deployment on start for specific topologies - it is a comma separated list
> of topology name:
> gateway.auto.deploy.topologies
>
> It defaults to "manager,admin"
>
> You could add your topologies to that and have them redeployed at restart
> if you like - make sure to leave manager and admin in there though.
>
> If you would like a config item that redeploys each topology on startup
> then please feel free to file a JIRA for it and we can consider it for
> 1.3.0.
>
> On Wed, Nov 28, 2018 at 3:01 PM Lars Francke <la...@gmail.com>
> wrote:
>
> > I was facing another weird issue today that took me a while to chase
> down.
> >
> > I didn't have the proper services directory in place when starting Knox
> for
> > the first time. This meant that a gateway.xml file was created in the
> > deployments dir for my topologies that was totally empty. It didn't have
> > any filters so all my tests returned a "Failed to match path" error.
> >
> > The problem here is that Knox checks on startup what the timestamp on the
> > topology is and only if that has changed does it create a new deployment.
> > If anything else changes this is not rewritten. So changes to the
> services
> > files will not be noticed by Knox and silently ignored.
> >
> > I understand that this is an edge-case but other than a slightly longer
> > startup would it hurt to recreate the deployments on every startup?
> >
> > Cheers,
> > Lars
> >
>

Re: Deploying Topologies problem

[larry mccay <lm...@apache.org>]

It isn't exactly clear to me what you encountered here.
I think that you are saying that you made changes to "services files" -
meaning the service definition.

If that is the case then a simple redeploy of the topology will not be
enough.
You have to restart the gateway so that the service definitions are
reloaded then unfortunately you have to touch the topology file/s that host
the service that you changed the definition for.

There is an undocumented config element in gateway-site.xml to force auto
deployment on start for specific topologies - it is a comma separated list
of topology name:
gateway.auto.deploy.topologies

It defaults to "manager,admin"

You could add your topologies to that and have them redeployed at restart
if you like - make sure to leave manager and admin in there though.

If you would like a config item that redeploys each topology on startup
then please feel free to file a JIRA for it and we can consider it for
1.3.0.

On Wed, Nov 28, 2018 at 3:01 PM Lars Francke <la...@gmail.com> wrote:

> I was facing another weird issue today that took me a while to chase down.
>
> I didn't have the proper services directory in place when starting Knox for
> the first time. This meant that a gateway.xml file was created in the
> deployments dir for my topologies that was totally empty. It didn't have
> any filters so all my tests returned a "Failed to match path" error.
>
> The problem here is that Knox checks on startup what the timestamp on the
> topology is and only if that has changed does it create a new deployment.
> If anything else changes this is not rewritten. So changes to the services
> files will not be noticed by Knox and silently ignored.
>
> I understand that this is an edge-case but other than a slightly longer
> startup would it hurt to recreate the deployments on every startup?
>
> Cheers,
> Lars
>