You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ignite.apache.org by Denis Mekhanikov <dm...@gmail.com> on 2019/02/04 13:38:14 UTC
Services hot redeployment
Igniters,
I'd like to start a dedicated thread for discussion of the design of
services hot redeployment. The previous service design discussion can be
found in the following thread: [1]
Currently adding a new service or implementation change of an existing one
requires restarting the hosting nodes. Service instances are deserialized
using an application class loader, so the service class should be present
on the classpath of the node. The only way to change the set of available
classes is to restart the node. Potentially the whole cluster restart can
be required. This is a major drawback in the current design. This problem
should be addressed first.
At the same time this problem can be resolved by relatively simple code
changes. We need to change the way services are deserialized, and use a
mechanism, that allows dynamic class changes. Deployment SPI [2] seems to
be suitable for this. We can apply the same approach, which is used for
tasks, so services will become dynamically modifiable.
With this approach user will still need to perform a cancel-deploy routine
for the changed service. But even with that the usability improvement will
be huge. We'll think about service availability improvement after the first
part is finished.
Thoughts?
[1]
http://apache-ignite-developers.2346864.n4.nabble.com/Service-versioning-td20858.html
[2] https://apacheignite.readme.io/docs/deployment-spi#deploymentspi
Denis
Re: Services hot redeployment
Posted by Vyacheslav Daradur <da...@gmail.com>.
Denis,
I've just prepared PR [1] with a possible implementation that covers
p.1 from your list.
Please, have a look at it. Pay attention to the test, it demonstrates
manual services hot-redeployment via DeploymentSpi.
Is that what you mean?
[1] https://github.com/apache/ignite/pull/6060
On Tue, Feb 5, 2019 at 2:14 PM Denis Mekhanikov <dm...@gmail.com> wrote:
>
> In general, I think, we should work on the improvements in the following
> order:
>
> 1. Cluster availability, when services are being updated. Should be solved
> by usage of the DeploymentSpi
> 2. Service availability, when they are being updated. This one will
> probably require introduction of new API methods like
> redeploy(ServiceConfiguration).
> 3. Service versioning and packaging.
>
> I'd like to focus on the first point. Service grid will become much more
> usable and mature once we implement it.
>
> Denis
>
> вт, 5 февр. 2019 г. в 14:06, Denis Mekhanikov <dm...@gmail.com>:
>
> > Vyacheslav,
> >
> > I think, we can use IgniteConfiguration#deploymentSpi for tasks and
> > services.
> > Or we can add an analogous property.
> >
> > Nik,
> >
> > > 1. Is it possible to change the list of deployed resources in runtime
> > via built-in DeploymentSPI implementations?
> > > Can I add or remove jar to(from) class-path without node(cluster)
> > restart?
> > Yes, this is the reason why the DeploymentSpi exists. But currently only
> > compute grid can use it.
> >
> > > 2. Can we update service dependencies via DeploymentSPI? commons-lang,
> > log4j or any other common library?
> > Ideally such libraries should be loaded via app class loader at node
> > startup. Otherwise the same libraries will be loaded multiple times. It
> > will lead to a lot of memory leaks.
> > I think, we can support loading of dependencies, but discourage users from
> > doing it. The proper way should be described in the documentation, and
> > warnings could be generated, if too many classes get loaded via
> > DeploymentSpi.
> >
> > > 3. User has to execute explicit Ignite API calls(undeploy(), deploy())
> > to renew service implementation. Is it correct?
> > > I think we should develop some watcher, that will be watch for a
> > resource change and redeploy services automatically.
> > Correct. I don't like the idea to redeploy services automatically. I
> > think, redeployment should be triggered explicitly. Non-obvious actions
> > should be avoided.
> >
> > 4. Such feature would for sure improve usability of the service grid. But
> > it requires much more time and work to implement.
> > I think, it's better not to expand the scope too much. Otherwise
> > development will take another 6 moths.
> > This is a great idea, and we will keep it in mind though.
> >
> > 5. Yep, we need an extensive documentation on the service deployment
> > procedure.
> > This feature may not be perfectly clear to users, so we need some how-tos.
> >
> > Denis
> >
> > вт, 5 февр. 2019 г. в 08:19, Nikolay Izhikov <ni...@apache.org>:
> >
> >> Hello, Denis.
> >>
> >> Thank you for this discussion.
> >> I have a few notes:
> >>
> >> 1. Is it possible to change the list of deployed resources in runtime via
> >> built-in DeploymentSPI implementations?
> >> Can I add or remove jar to(from) class-path without node(cluster) restart?
> >>
> >> 2. Can we update service dependencies via DeploymentSPI? commons-lang,
> >> log4j or any other common library?
> >>
> >> 3. User has to execute explicit Ignite API calls(undeploy(), deploy()) to
> >> renew service implementation. Is it correct?
> >> I think we should develop some watcher, that will be watch for a resource
> >> change and redeploy services automatically.
> >>
> >> 4. DeploymentSPI is *node-wide* configuration. This means we change
> >> classpath for all services with this SPI.
> >> I think this is a huge limitation of the SPI.
> >> We should provide an ability to configure service-wide classpath to our
> >> users as quickly as possible.
> >> It is a common feature in modern service, task executor engines.
> >>
> >> I think the perfect world scenario would be following:
> >>
> >> 1. Start a client node or connect to a cluster with thin client.
> >>
> >> 2. Configure service classpath with some new Ignite API.
> >> The only requirement for classes - they should be available
> >> locally(on client node or thin client host).
> >>
> >> 3. User deploy the service with some Ignite API.
> >>
> >> 4. After depoyment completes successfully client node can be
> >> stopped.
> >> All required resource to run a service should be safely stored in
> >> cluster and deployed to any new node.
> >>
> >> 5. I think we should develop examples for a DeploymentSPI usage.
> >> As far as I can see, there is no such examples in our codebase for now.
> >> Is it correct? If so, I will create a ticket to create such examples.
> >>
> >> В Вт, 05/02/2019 в 01:08 +0300, Vyacheslav Daradur пишет:
> >> > Denis, thank you for driving of Service Grid's development!
> >> >
> >> > Sounds like a good plan. Does it mean that a user will have to
> >> > register a classloader for service's class explicitly in case of using
> >> > the feature?
> >> >
> >> > On Mon, Feb 4, 2019 at 4:38 PM Denis Mekhanikov <dm...@gmail.com>
> >> wrote:
> >> > >
> >> > > Igniters,
> >> > >
> >> > > I'd like to start a dedicated thread for discussion of the design of
> >> > > services hot redeployment. The previous service design discussion can
> >> be
> >> > > found in the following thread: [1]
> >> > >
> >> > > Currently adding a new service or implementation change of an
> >> existing one
> >> > > requires restarting the hosting nodes. Service instances are
> >> deserialized
> >> > > using an application class loader, so the service class should be
> >> present
> >> > > on the classpath of the node. The only way to change the set of
> >> available
> >> > > classes is to restart the node. Potentially the whole cluster restart
> >> can
> >> > > be required. This is a major drawback in the current design. This
> >> problem
> >> > > should be addressed first.
> >> > >
> >> > > At the same time this problem can be resolved by relatively simple
> >> code
> >> > > changes. We need to change the way services are deserialized, and use
> >> a
> >> > > mechanism, that allows dynamic class changes. Deployment SPI [2]
> >> seems to
> >> > > be suitable for this. We can apply the same approach, which is used
> >> for
> >> > > tasks, so services will become dynamically modifiable.
> >> > >
> >> > > With this approach user will still need to perform a cancel-deploy
> >> routine
> >> > > for the changed service. But even with that the usability improvement
> >> will
> >> > > be huge. We'll think about service availability improvement after the
> >> first
> >> > > part is finished.
> >> > >
> >> > > Thoughts?
> >> > >
> >> > > [1]
> >> > >
> >> http://apache-ignite-developers.2346864.n4.nabble.com/Service-versioning-td20858.html
> >> > > [2] https://apacheignite.readme.io/docs/deployment-spi#deploymentspi
> >> > >
> >> > > Denis
> >> >
> >> >
> >> >
> >> > --
> >> > Best Regards, Vyacheslav D.
> >>
> >
--
Best Regards, Vyacheslav D.
Re: Services hot redeployment
Posted by Denis Mekhanikov <dm...@gmail.com>.
In general, I think, we should work on the improvements in the following
order:
1. Cluster availability, when services are being updated. Should be solved
by usage of the DeploymentSpi
2. Service availability, when they are being updated. This one will
probably require introduction of new API methods like
redeploy(ServiceConfiguration).
3. Service versioning and packaging.
I'd like to focus on the first point. Service grid will become much more
usable and mature once we implement it.
Denis
вт, 5 февр. 2019 г. в 14:06, Denis Mekhanikov <dm...@gmail.com>:
> Vyacheslav,
>
> I think, we can use IgniteConfiguration#deploymentSpi for tasks and
> services.
> Or we can add an analogous property.
>
> Nik,
>
> > 1. Is it possible to change the list of deployed resources in runtime
> via built-in DeploymentSPI implementations?
> > Can I add or remove jar to(from) class-path without node(cluster)
> restart?
> Yes, this is the reason why the DeploymentSpi exists. But currently only
> compute grid can use it.
>
> > 2. Can we update service dependencies via DeploymentSPI? commons-lang,
> log4j or any other common library?
> Ideally such libraries should be loaded via app class loader at node
> startup. Otherwise the same libraries will be loaded multiple times. It
> will lead to a lot of memory leaks.
> I think, we can support loading of dependencies, but discourage users from
> doing it. The proper way should be described in the documentation, and
> warnings could be generated, if too many classes get loaded via
> DeploymentSpi.
>
> > 3. User has to execute explicit Ignite API calls(undeploy(), deploy())
> to renew service implementation. Is it correct?
> > I think we should develop some watcher, that will be watch for a
> resource change and redeploy services automatically.
> Correct. I don't like the idea to redeploy services automatically. I
> think, redeployment should be triggered explicitly. Non-obvious actions
> should be avoided.
>
> 4. Such feature would for sure improve usability of the service grid. But
> it requires much more time and work to implement.
> I think, it's better not to expand the scope too much. Otherwise
> development will take another 6 moths.
> This is a great idea, and we will keep it in mind though.
>
> 5. Yep, we need an extensive documentation on the service deployment
> procedure.
> This feature may not be perfectly clear to users, so we need some how-tos.
>
> Denis
>
> вт, 5 февр. 2019 г. в 08:19, Nikolay Izhikov <ni...@apache.org>:
>
>> Hello, Denis.
>>
>> Thank you for this discussion.
>> I have a few notes:
>>
>> 1. Is it possible to change the list of deployed resources in runtime via
>> built-in DeploymentSPI implementations?
>> Can I add or remove jar to(from) class-path without node(cluster) restart?
>>
>> 2. Can we update service dependencies via DeploymentSPI? commons-lang,
>> log4j or any other common library?
>>
>> 3. User has to execute explicit Ignite API calls(undeploy(), deploy()) to
>> renew service implementation. Is it correct?
>> I think we should develop some watcher, that will be watch for a resource
>> change and redeploy services automatically.
>>
>> 4. DeploymentSPI is *node-wide* configuration. This means we change
>> classpath for all services with this SPI.
>> I think this is a huge limitation of the SPI.
>> We should provide an ability to configure service-wide classpath to our
>> users as quickly as possible.
>> It is a common feature in modern service, task executor engines.
>>
>> I think the perfect world scenario would be following:
>>
>> 1. Start a client node or connect to a cluster with thin client.
>>
>> 2. Configure service classpath with some new Ignite API.
>> The only requirement for classes - they should be available
>> locally(on client node or thin client host).
>>
>> 3. User deploy the service with some Ignite API.
>>
>> 4. After depoyment completes successfully client node can be
>> stopped.
>> All required resource to run a service should be safely stored in
>> cluster and deployed to any new node.
>>
>> 5. I think we should develop examples for a DeploymentSPI usage.
>> As far as I can see, there is no such examples in our codebase for now.
>> Is it correct? If so, I will create a ticket to create such examples.
>>
>> В Вт, 05/02/2019 в 01:08 +0300, Vyacheslav Daradur пишет:
>> > Denis, thank you for driving of Service Grid's development!
>> >
>> > Sounds like a good plan. Does it mean that a user will have to
>> > register a classloader for service's class explicitly in case of using
>> > the feature?
>> >
>> > On Mon, Feb 4, 2019 at 4:38 PM Denis Mekhanikov <dm...@gmail.com>
>> wrote:
>> > >
>> > > Igniters,
>> > >
>> > > I'd like to start a dedicated thread for discussion of the design of
>> > > services hot redeployment. The previous service design discussion can
>> be
>> > > found in the following thread: [1]
>> > >
>> > > Currently adding a new service or implementation change of an
>> existing one
>> > > requires restarting the hosting nodes. Service instances are
>> deserialized
>> > > using an application class loader, so the service class should be
>> present
>> > > on the classpath of the node. The only way to change the set of
>> available
>> > > classes is to restart the node. Potentially the whole cluster restart
>> can
>> > > be required. This is a major drawback in the current design. This
>> problem
>> > > should be addressed first.
>> > >
>> > > At the same time this problem can be resolved by relatively simple
>> code
>> > > changes. We need to change the way services are deserialized, and use
>> a
>> > > mechanism, that allows dynamic class changes. Deployment SPI [2]
>> seems to
>> > > be suitable for this. We can apply the same approach, which is used
>> for
>> > > tasks, so services will become dynamically modifiable.
>> > >
>> > > With this approach user will still need to perform a cancel-deploy
>> routine
>> > > for the changed service. But even with that the usability improvement
>> will
>> > > be huge. We'll think about service availability improvement after the
>> first
>> > > part is finished.
>> > >
>> > > Thoughts?
>> > >
>> > > [1]
>> > >
>> http://apache-ignite-developers.2346864.n4.nabble.com/Service-versioning-td20858.html
>> > > [2] https://apacheignite.readme.io/docs/deployment-spi#deploymentspi
>> > >
>> > > Denis
>> >
>> >
>> >
>> > --
>> > Best Regards, Vyacheslav D.
>>
>
Re: Services hot redeployment
Posted by Denis Mekhanikov <dm...@gmail.com>.
Vyacheslav,
I think, we can use IgniteConfiguration#deploymentSpi for tasks and
services.
Or we can add an analogous property.
Nik,
> 1. Is it possible to change the list of deployed resources in runtime via
built-in DeploymentSPI implementations?
> Can I add or remove jar to(from) class-path without node(cluster) restart?
Yes, this is the reason why the DeploymentSpi exists. But currently only
compute grid can use it.
> 2. Can we update service dependencies via DeploymentSPI? commons-lang,
log4j or any other common library?
Ideally such libraries should be loaded via app class loader at node
startup. Otherwise the same libraries will be loaded multiple times. It
will lead to a lot of memory leaks.
I think, we can support loading of dependencies, but discourage users from
doing it. The proper way should be described in the documentation, and
warnings could be generated, if too many classes get loaded via
DeploymentSpi.
> 3. User has to execute explicit Ignite API calls(undeploy(), deploy()) to
renew service implementation. Is it correct?
> I think we should develop some watcher, that will be watch for a resource
change and redeploy services automatically.
Correct. I don't like the idea to redeploy services automatically. I think,
redeployment should be triggered explicitly. Non-obvious actions should be
avoided.
4. Such feature would for sure improve usability of the service grid. But
it requires much more time and work to implement.
I think, it's better not to expand the scope too much. Otherwise
development will take another 6 moths.
This is a great idea, and we will keep it in mind though.
5. Yep, we need an extensive documentation on the service deployment
procedure.
This feature may not be perfectly clear to users, so we need some how-tos.
Denis
вт, 5 февр. 2019 г. в 08:19, Nikolay Izhikov <ni...@apache.org>:
> Hello, Denis.
>
> Thank you for this discussion.
> I have a few notes:
>
> 1. Is it possible to change the list of deployed resources in runtime via
> built-in DeploymentSPI implementations?
> Can I add or remove jar to(from) class-path without node(cluster) restart?
>
> 2. Can we update service dependencies via DeploymentSPI? commons-lang,
> log4j or any other common library?
>
> 3. User has to execute explicit Ignite API calls(undeploy(), deploy()) to
> renew service implementation. Is it correct?
> I think we should develop some watcher, that will be watch for a resource
> change and redeploy services automatically.
>
> 4. DeploymentSPI is *node-wide* configuration. This means we change
> classpath for all services with this SPI.
> I think this is a huge limitation of the SPI.
> We should provide an ability to configure service-wide classpath to our
> users as quickly as possible.
> It is a common feature in modern service, task executor engines.
>
> I think the perfect world scenario would be following:
>
> 1. Start a client node or connect to a cluster with thin client.
>
> 2. Configure service classpath with some new Ignite API.
> The only requirement for classes - they should be available
> locally(on client node or thin client host).
>
> 3. User deploy the service with some Ignite API.
>
> 4. After depoyment completes successfully client node can be
> stopped.
> All required resource to run a service should be safely stored in
> cluster and deployed to any new node.
>
> 5. I think we should develop examples for a DeploymentSPI usage.
> As far as I can see, there is no such examples in our codebase for now.
> Is it correct? If so, I will create a ticket to create such examples.
>
> В Вт, 05/02/2019 в 01:08 +0300, Vyacheslav Daradur пишет:
> > Denis, thank you for driving of Service Grid's development!
> >
> > Sounds like a good plan. Does it mean that a user will have to
> > register a classloader for service's class explicitly in case of using
> > the feature?
> >
> > On Mon, Feb 4, 2019 at 4:38 PM Denis Mekhanikov <dm...@gmail.com>
> wrote:
> > >
> > > Igniters,
> > >
> > > I'd like to start a dedicated thread for discussion of the design of
> > > services hot redeployment. The previous service design discussion can
> be
> > > found in the following thread: [1]
> > >
> > > Currently adding a new service or implementation change of an existing
> one
> > > requires restarting the hosting nodes. Service instances are
> deserialized
> > > using an application class loader, so the service class should be
> present
> > > on the classpath of the node. The only way to change the set of
> available
> > > classes is to restart the node. Potentially the whole cluster restart
> can
> > > be required. This is a major drawback in the current design. This
> problem
> > > should be addressed first.
> > >
> > > At the same time this problem can be resolved by relatively simple code
> > > changes. We need to change the way services are deserialized, and use a
> > > mechanism, that allows dynamic class changes. Deployment SPI [2] seems
> to
> > > be suitable for this. We can apply the same approach, which is used for
> > > tasks, so services will become dynamically modifiable.
> > >
> > > With this approach user will still need to perform a cancel-deploy
> routine
> > > for the changed service. But even with that the usability improvement
> will
> > > be huge. We'll think about service availability improvement after the
> first
> > > part is finished.
> > >
> > > Thoughts?
> > >
> > > [1]
> > >
> http://apache-ignite-developers.2346864.n4.nabble.com/Service-versioning-td20858.html
> > > [2] https://apacheignite.readme.io/docs/deployment-spi#deploymentspi
> > >
> > > Denis
> >
> >
> >
> > --
> > Best Regards, Vyacheslav D.
>
Re: Services hot redeployment
Posted by Nikolay Izhikov <ni...@apache.org>.
Hello, Denis.
Thank you for this discussion.
I have a few notes:
1. Is it possible to change the list of deployed resources in runtime via built-in DeploymentSPI implementations?
Can I add or remove jar to(from) class-path without node(cluster) restart?
2. Can we update service dependencies via DeploymentSPI? commons-lang, log4j or any other common library?
3. User has to execute explicit Ignite API calls(undeploy(), deploy()) to renew service implementation. Is it correct?
I think we should develop some watcher, that will be watch for a resource change and redeploy services automatically.
4. DeploymentSPI is *node-wide* configuration. This means we change classpath for all services with this SPI.
I think this is a huge limitation of the SPI.
We should provide an ability to configure service-wide classpath to our users as quickly as possible.
It is a common feature in modern service, task executor engines.
I think the perfect world scenario would be following:
1. Start a client node or connect to a cluster with thin client.
2. Configure service classpath with some new Ignite API.
The only requirement for classes - they should be available locally(on client node or thin client host).
3. User deploy the service with some Ignite API.
4. After depoyment completes successfully client node can be stopped.
All required resource to run a service should be safely stored in cluster and deployed to any new node.
5. I think we should develop examples for a DeploymentSPI usage.
As far as I can see, there is no such examples in our codebase for now.
Is it correct? If so, I will create a ticket to create such examples.
В Вт, 05/02/2019 в 01:08 +0300, Vyacheslav Daradur пишет:
> Denis, thank you for driving of Service Grid's development!
>
> Sounds like a good plan. Does it mean that a user will have to
> register a classloader for service's class explicitly in case of using
> the feature?
>
> On Mon, Feb 4, 2019 at 4:38 PM Denis Mekhanikov <dm...@gmail.com> wrote:
> >
> > Igniters,
> >
> > I'd like to start a dedicated thread for discussion of the design of
> > services hot redeployment. The previous service design discussion can be
> > found in the following thread: [1]
> >
> > Currently adding a new service or implementation change of an existing one
> > requires restarting the hosting nodes. Service instances are deserialized
> > using an application class loader, so the service class should be present
> > on the classpath of the node. The only way to change the set of available
> > classes is to restart the node. Potentially the whole cluster restart can
> > be required. This is a major drawback in the current design. This problem
> > should be addressed first.
> >
> > At the same time this problem can be resolved by relatively simple code
> > changes. We need to change the way services are deserialized, and use a
> > mechanism, that allows dynamic class changes. Deployment SPI [2] seems to
> > be suitable for this. We can apply the same approach, which is used for
> > tasks, so services will become dynamically modifiable.
> >
> > With this approach user will still need to perform a cancel-deploy routine
> > for the changed service. But even with that the usability improvement will
> > be huge. We'll think about service availability improvement after the first
> > part is finished.
> >
> > Thoughts?
> >
> > [1]
> > http://apache-ignite-developers.2346864.n4.nabble.com/Service-versioning-td20858.html
> > [2] https://apacheignite.readme.io/docs/deployment-spi#deploymentspi
> >
> > Denis
>
>
>
> --
> Best Regards, Vyacheslav D.
Re: Services hot redeployment
Posted by Vyacheslav Daradur <da...@gmail.com>.
Denis, thank you for driving of Service Grid's development!
Sounds like a good plan. Does it mean that a user will have to
register a classloader for service's class explicitly in case of using
the feature?
On Mon, Feb 4, 2019 at 4:38 PM Denis Mekhanikov <dm...@gmail.com> wrote:
>
> Igniters,
>
> I'd like to start a dedicated thread for discussion of the design of
> services hot redeployment. The previous service design discussion can be
> found in the following thread: [1]
>
> Currently adding a new service or implementation change of an existing one
> requires restarting the hosting nodes. Service instances are deserialized
> using an application class loader, so the service class should be present
> on the classpath of the node. The only way to change the set of available
> classes is to restart the node. Potentially the whole cluster restart can
> be required. This is a major drawback in the current design. This problem
> should be addressed first.
>
> At the same time this problem can be resolved by relatively simple code
> changes. We need to change the way services are deserialized, and use a
> mechanism, that allows dynamic class changes. Deployment SPI [2] seems to
> be suitable for this. We can apply the same approach, which is used for
> tasks, so services will become dynamically modifiable.
>
> With this approach user will still need to perform a cancel-deploy routine
> for the changed service. But even with that the usability improvement will
> be huge. We'll think about service availability improvement after the first
> part is finished.
>
> Thoughts?
>
> [1]
> http://apache-ignite-developers.2346864.n4.nabble.com/Service-versioning-td20858.html
> [2] https://apacheignite.readme.io/docs/deployment-spi#deploymentspi
>
> Denis
--
Best Regards, Vyacheslav D.