You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Pascoe Scholle <pa...@gmail.com> on 2019/08/22 07:52:30 UTC

Question on submitted post

Hi there,

How long does it usually take, for a post to be seen on the forum? Maybe I
made a mistake so I will provide my question here. Excuse me if I am being
impatient:


=========================================
Good afternoon everyone,

I have recently run into an issue and I think the problem lies in the
server node configuration. I will attache the output of the stack trace at
the bottom, however I first wish to explain what the software does and how
we are using ignite.

I start multiple server nodes with peerClassEnabled set to true, using  a
TcpDiscoveryVmIpFinder and dont set anything other than a port range for
the ipFinder.

Using the REST protocol a ComputeTaskAdapter task is executed which starts
a service and this in turn starts a grpc server, I have placed some scala
code to show what I mean.

class StartService extends ComputeTaskAdapter[String, Any]{
  var ignite: Ignite = null;
  @IgniteInstanceResource
  def setIgnite(someIgnite: Ignite): Unit = {
    ignite = someIgnite
  }

 def map(...)={
...
// port is an integer
val server = new GrpcServer(ignite, port);

val service = new ServiceImpl(name, server);
/*
within the method execute of the Service interface, server.start() is called
*/

val serviceconfig = new ServiceConfiguration();
          serviceconfig.setName(name);
          serviceconfig.setTotalCount(1);
          serviceconfig.setMaxPerNodeCount(1);
          ignite.services().deploy(serviceconfig);
...
}

}

this task returns a map with some non important variables.

The grpc server takes the ignite instance created within the above
mentioned computeTask as a variable, I am not sure if this could be the
cause of the issue.

Using grpc protocol, we create a ComputeTask which is executed by the grpc
server some more code below:

class GrpcServer(val ignite:Ignite, val port:Int) extends ..Some Grpc
stuff..{

def someGrpcProtocol(request: Message):Future[String]={
val newTask = new SomeTask();

ignite.compute(ignite.cluster()).execute(newTask, someinput);
Future("Request is being processed");
}

}


If a single server node is started, the program runs without problems.
However, adding more nodes and trying to execute the new tasks on a remote
node or on a node that has a certain attribute gives me a massive stack
trace in the face.
Basically, if I want to execute a task on a node where the service and grpc
server do not reside, the exception happens.

I have placed all custom classes within a jar that lies in the libs folder
of the ignite-bin project.
We are currently on version 2.7

If you require anything else just let me know, ill be on it asap.

Thanks for any help that may come my way.

Cheers!

Here is most of the stack trace:
class org.apache.ignite.binary.BinaryObjectException: Failed to read field
[name=server]
        at
org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:192)
        at
org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:875)
        at
org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1764)
        at
org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
        at
org.apache.ignite.internal.binary.BinaryReaderExImpl.readField(BinaryReaderExImpl.java:1984)
        at
org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read0(BinaryFieldAccessor.java:703)
        at
org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:188)
        at
org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:875)
        at
org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1764)
        at
org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
        at
org.apache.ignite.internal.binary.BinaryReaderExImpl.readField(BinaryReaderExImpl.java:1984)
        at
org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read0(BinaryFieldAccessor.java:703)
        at
org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:188)
        at
org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:875)
        at
org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1764)
        at
org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
        at
org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313)
        at
org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102)
        at
org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
        at
org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140)
        at
org.apache.ignite.internal.processors.job.GridJobWorker.initialize(GridJobWorker.java:440)
        at
org.apache.ignite.internal.processors.job.GridJobProcessor.processJobExecuteRequest(GridJobProcessor.java:1119)
        at
org.apache.ignite.internal.processors.job.GridJobProcessor$JobExecutionListener.onMessage(GridJobProcessor.java:1923)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: class org.apache.ignite.binary.BinaryObjectException: Failed to
deserialize object [typeName=io.grpc.internal.ServerImpl]
        at
org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:914)
        at
org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1764)
        at
org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
        at
org.apache.ignite.internal.binary.BinaryReaderExImpl.readField(BinaryReaderExImpl.java:1984)
        at
org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read0(BinaryFieldAccessor.java:703)
        at
org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:188)
        ... 29 more
Caused by: class org.apache.ignite.binary.BinaryObjectException: Failed to
read field [name=registry]
        at
org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:192)
        at
org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:875)
        ... 34 more
Caused by: class org.apache.ignite.binary.BinaryObjectException: Failed to
deserialize object [typeName=io.grpc.internal.InternalHandlerRegistry]
        at
org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:914)
        at
org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1764)
        at
org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
        at
org.apache.ignite.internal.binary.BinaryReaderExImpl.readField(BinaryReaderExImpl.java:1984)
        at
org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read0(BinaryFieldAccessor.java:703)
        at
org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:188)
        ... 35 more

=======================================

Thanks and kind regards,
Pascoe

Re: Question on submitted post

Posted by Pascoe Scholle <pa...@gmail.com>.
Hi Stan,

Thanks for the info! Indeed this was a very big mistake and your
explanation made it clear what the error was.

I was passing an instance of the grpc server to the compute job, a part of
the code which I had neglected to add in the description above.

Cheers!

On Mon, Aug 26, 2019 at 7:21 PM Stanislav Lukyanov <st...@gmail.com>
wrote:

> The message "Failed to deserialize object
> [typeName=io.grpc.internal.InternalHandlerRegistry]"
> means InternalHandlerRegistry is being sent between the nodes - which it
> shouldn't be.
> What you need to do is to find where is it being sent. You shouldn't pass
> any gRPC to any Ignite configuration, or, I believe, to any Ignite method.
> Also, be careful with compute jobs that use lambdas and anonymous classes.
> By their nature they catch objects around them too easy, so often you'll
> see a compute job accidentally bringing some context over the network. The
> best way to avoid it is to use `public static` classes instead of anonymous
> classes and lambdas - verbose but bulletproof.
>
> If you still have an issue with this please share more complete code and
> config example. A small reproducer on github would be ideal.
>
> Stan
>
> On Mon, Aug 26, 2019 at 4:09 PM Pascoe Scholle <pa...@gmail.com>
> wrote:
>
>> Hi Stan,
>>
>> Thanks for your response. I have tried this, but it has not fixed the
>> issue.
>> The grpc server class was moved into the service where the interface
>> methods "init","execute" and "cancel" perform an initialization of the
>> serve, as well start and stopping respectively. But this was already
>> implemented.
>>
>> Does this issue not have something to do with class loading? The second
>> node that is started doesn't have any information of the custom classes for
>> serialization. I have also moved all JARS of these custom classes into the
>> libs folder in ignite but that has also not helped.
>>
>> On Fri, Aug 23, 2019 at 6:56 PM Stanislav Lukyanov <
>> stanlukyanov@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> It looks like the issue is that you're ending up sending an instance of
>>> your gRPC server inside your service. This approach is generally incorrect.
>>> What you should do is
>>> - not pass gRPC to the service instance
>>> - add an init() method implementation to your service
>>> - in your init() start your gRPC server
>>>
>>> Stan
>>>
>>> On Thu, Aug 22, 2019 at 10:52 AM Pascoe Scholle <pa...@gmail.com>
>>> wrote:
>>>
>>>> Hi there,
>>>>
>>>> How long does it usually take, for a post to be seen on the forum?
>>>> Maybe I made a mistake so I will provide my question here. Excuse me if I
>>>> am being impatient:
>>>>
>>>>
>>>> =========================================
>>>> Good afternoon everyone,
>>>>
>>>> I have recently run into an issue and I think the problem lies in the
>>>> server node configuration. I will attache the output of the stack trace at
>>>> the bottom, however I first wish to explain what the software does and how
>>>> we are using ignite.
>>>>
>>>> I start multiple server nodes with peerClassEnabled set to true, using
>>>>  a TcpDiscoveryVmIpFinder and dont set anything other than a port range for
>>>> the ipFinder.
>>>>
>>>> Using the REST protocol a ComputeTaskAdapter task is executed which
>>>> starts a service and this in turn starts a grpc server, I have placed some
>>>> scala code to show what I mean.
>>>>
>>>> class StartService extends ComputeTaskAdapter[String, Any]{
>>>>   var ignite: Ignite = null;
>>>>   @IgniteInstanceResource
>>>>   def setIgnite(someIgnite: Ignite): Unit = {
>>>>     ignite = someIgnite
>>>>   }
>>>>
>>>>  def map(...)={
>>>> ...
>>>> // port is an integer
>>>> val server = new GrpcServer(ignite, port);
>>>>
>>>> val service = new ServiceImpl(name, server);
>>>> /*
>>>> within the method execute of the Service interface, server.start() is
>>>> called
>>>> */
>>>>
>>>> val serviceconfig = new ServiceConfiguration();
>>>>           serviceconfig.setName(name);
>>>>           serviceconfig.setTotalCount(1);
>>>>           serviceconfig.setMaxPerNodeCount(1);
>>>>           ignite.services().deploy(serviceconfig);
>>>> ...
>>>> }
>>>>
>>>> }
>>>>
>>>> this task returns a map with some non important variables.
>>>>
>>>> The grpc server takes the ignite instance created within the above
>>>> mentioned computeTask as a variable, I am not sure if this could be the
>>>> cause of the issue.
>>>>
>>>> Using grpc protocol, we create a ComputeTask which is executed by the
>>>> grpc server some more code below:
>>>>
>>>> class GrpcServer(val ignite:Ignite, val port:Int) extends ..Some Grpc
>>>> stuff..{
>>>>
>>>> def someGrpcProtocol(request: Message):Future[String]={
>>>> val newTask = new SomeTask();
>>>>
>>>> ignite.compute(ignite.cluster()).execute(newTask, someinput);
>>>> Future("Request is being processed");
>>>> }
>>>>
>>>> }
>>>>
>>>>
>>>> If a single server node is started, the program runs without problems.
>>>> However, adding more nodes and trying to execute the new tasks on a remote
>>>> node or on a node that has a certain attribute gives me a massive stack
>>>> trace in the face.
>>>> Basically, if I want to execute a task on a node where the service and
>>>> grpc server do not reside, the exception happens.
>>>>
>>>> I have placed all custom classes within a jar that lies in the libs
>>>> folder of the ignite-bin project.
>>>> We are currently on version 2.7
>>>>
>>>> If you require anything else just let me know, ill be on it asap.
>>>>
>>>> Thanks for any help that may come my way.
>>>>
>>>> Cheers!
>>>>
>>>> Here is most of the stack trace:
>>>> class org.apache.ignite.binary.BinaryObjectException: Failed to read
>>>> field [name=server]
>>>>         at
>>>> org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:192)
>>>>         at
>>>> org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:875)
>>>>         at
>>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1764)
>>>>         at
>>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
>>>>         at
>>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.readField(BinaryReaderExImpl.java:1984)
>>>>         at
>>>> org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read0(BinaryFieldAccessor.java:703)
>>>>         at
>>>> org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:188)
>>>>         at
>>>> org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:875)
>>>>         at
>>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1764)
>>>>         at
>>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
>>>>         at
>>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.readField(BinaryReaderExImpl.java:1984)
>>>>         at
>>>> org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read0(BinaryFieldAccessor.java:703)
>>>>         at
>>>> org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:188)
>>>>         at
>>>> org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:875)
>>>>         at
>>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1764)
>>>>         at
>>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
>>>>         at
>>>> org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313)
>>>>         at
>>>> org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102)
>>>>         at
>>>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
>>>>         at
>>>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140)
>>>>         at
>>>> org.apache.ignite.internal.processors.job.GridJobWorker.initialize(GridJobWorker.java:440)
>>>>         at
>>>> org.apache.ignite.internal.processors.job.GridJobProcessor.processJobExecuteRequest(GridJobProcessor.java:1119)
>>>>         at
>>>> org.apache.ignite.internal.processors.job.GridJobProcessor$JobExecutionListener.onMessage(GridJobProcessor.java:1923)
>>>>         at
>>>> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
>>>>         at
>>>> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
>>>>         at
>>>> org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
>>>>         at
>>>> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
>>>>         at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>         at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>         at java.lang.Thread.run(Thread.java:748)
>>>> Caused by: class org.apache.ignite.binary.BinaryObjectException: Failed
>>>> to deserialize object [typeName=io.grpc.internal.ServerImpl]
>>>>         at
>>>> org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:914)
>>>>         at
>>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1764)
>>>>         at
>>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
>>>>         at
>>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.readField(BinaryReaderExImpl.java:1984)
>>>>         at
>>>> org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read0(BinaryFieldAccessor.java:703)
>>>>         at
>>>> org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:188)
>>>>         ... 29 more
>>>> Caused by: class org.apache.ignite.binary.BinaryObjectException: Failed
>>>> to read field [name=registry]
>>>>         at
>>>> org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:192)
>>>>         at
>>>> org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:875)
>>>>         ... 34 more
>>>> Caused by: class org.apache.ignite.binary.BinaryObjectException: Failed
>>>> to deserialize object [typeName=io.grpc.internal.InternalHandlerRegistry]
>>>>         at
>>>> org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:914)
>>>>         at
>>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1764)
>>>>         at
>>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
>>>>         at
>>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.readField(BinaryReaderExImpl.java:1984)
>>>>         at
>>>> org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read0(BinaryFieldAccessor.java:703)
>>>>         at
>>>> org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:188)
>>>>         ... 35 more
>>>>
>>>> =======================================
>>>>
>>>> Thanks and kind regards,
>>>> Pascoe
>>>>
>>>

Re: Question on submitted post

Posted by Stanislav Lukyanov <st...@gmail.com>.
The message "Failed to deserialize object
[typeName=io.grpc.internal.InternalHandlerRegistry]"
means InternalHandlerRegistry is being sent between the nodes - which it
shouldn't be.
What you need to do is to find where is it being sent. You shouldn't pass
any gRPC to any Ignite configuration, or, I believe, to any Ignite method.
Also, be careful with compute jobs that use lambdas and anonymous classes.
By their nature they catch objects around them too easy, so often you'll
see a compute job accidentally bringing some context over the network. The
best way to avoid it is to use `public static` classes instead of anonymous
classes and lambdas - verbose but bulletproof.

If you still have an issue with this please share more complete code and
config example. A small reproducer on github would be ideal.

Stan

On Mon, Aug 26, 2019 at 4:09 PM Pascoe Scholle <pa...@gmail.com>
wrote:

> Hi Stan,
>
> Thanks for your response. I have tried this, but it has not fixed the
> issue.
> The grpc server class was moved into the service where the interface
> methods "init","execute" and "cancel" perform an initialization of the
> serve, as well start and stopping respectively. But this was already
> implemented.
>
> Does this issue not have something to do with class loading? The second
> node that is started doesn't have any information of the custom classes for
> serialization. I have also moved all JARS of these custom classes into the
> libs folder in ignite but that has also not helped.
>
> On Fri, Aug 23, 2019 at 6:56 PM Stanislav Lukyanov <st...@gmail.com>
> wrote:
>
>> Hi,
>>
>> It looks like the issue is that you're ending up sending an instance of
>> your gRPC server inside your service. This approach is generally incorrect.
>> What you should do is
>> - not pass gRPC to the service instance
>> - add an init() method implementation to your service
>> - in your init() start your gRPC server
>>
>> Stan
>>
>> On Thu, Aug 22, 2019 at 10:52 AM Pascoe Scholle <pa...@gmail.com>
>> wrote:
>>
>>> Hi there,
>>>
>>> How long does it usually take, for a post to be seen on the forum? Maybe
>>> I made a mistake so I will provide my question here. Excuse me if I am
>>> being impatient:
>>>
>>>
>>> =========================================
>>> Good afternoon everyone,
>>>
>>> I have recently run into an issue and I think the problem lies in the
>>> server node configuration. I will attache the output of the stack trace at
>>> the bottom, however I first wish to explain what the software does and how
>>> we are using ignite.
>>>
>>> I start multiple server nodes with peerClassEnabled set to true, using
>>>  a TcpDiscoveryVmIpFinder and dont set anything other than a port range for
>>> the ipFinder.
>>>
>>> Using the REST protocol a ComputeTaskAdapter task is executed which
>>> starts a service and this in turn starts a grpc server, I have placed some
>>> scala code to show what I mean.
>>>
>>> class StartService extends ComputeTaskAdapter[String, Any]{
>>>   var ignite: Ignite = null;
>>>   @IgniteInstanceResource
>>>   def setIgnite(someIgnite: Ignite): Unit = {
>>>     ignite = someIgnite
>>>   }
>>>
>>>  def map(...)={
>>> ...
>>> // port is an integer
>>> val server = new GrpcServer(ignite, port);
>>>
>>> val service = new ServiceImpl(name, server);
>>> /*
>>> within the method execute of the Service interface, server.start() is
>>> called
>>> */
>>>
>>> val serviceconfig = new ServiceConfiguration();
>>>           serviceconfig.setName(name);
>>>           serviceconfig.setTotalCount(1);
>>>           serviceconfig.setMaxPerNodeCount(1);
>>>           ignite.services().deploy(serviceconfig);
>>> ...
>>> }
>>>
>>> }
>>>
>>> this task returns a map with some non important variables.
>>>
>>> The grpc server takes the ignite instance created within the above
>>> mentioned computeTask as a variable, I am not sure if this could be the
>>> cause of the issue.
>>>
>>> Using grpc protocol, we create a ComputeTask which is executed by the
>>> grpc server some more code below:
>>>
>>> class GrpcServer(val ignite:Ignite, val port:Int) extends ..Some Grpc
>>> stuff..{
>>>
>>> def someGrpcProtocol(request: Message):Future[String]={
>>> val newTask = new SomeTask();
>>>
>>> ignite.compute(ignite.cluster()).execute(newTask, someinput);
>>> Future("Request is being processed");
>>> }
>>>
>>> }
>>>
>>>
>>> If a single server node is started, the program runs without problems.
>>> However, adding more nodes and trying to execute the new tasks on a remote
>>> node or on a node that has a certain attribute gives me a massive stack
>>> trace in the face.
>>> Basically, if I want to execute a task on a node where the service and
>>> grpc server do not reside, the exception happens.
>>>
>>> I have placed all custom classes within a jar that lies in the libs
>>> folder of the ignite-bin project.
>>> We are currently on version 2.7
>>>
>>> If you require anything else just let me know, ill be on it asap.
>>>
>>> Thanks for any help that may come my way.
>>>
>>> Cheers!
>>>
>>> Here is most of the stack trace:
>>> class org.apache.ignite.binary.BinaryObjectException: Failed to read
>>> field [name=server]
>>>         at
>>> org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:192)
>>>         at
>>> org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:875)
>>>         at
>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1764)
>>>         at
>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
>>>         at
>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.readField(BinaryReaderExImpl.java:1984)
>>>         at
>>> org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read0(BinaryFieldAccessor.java:703)
>>>         at
>>> org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:188)
>>>         at
>>> org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:875)
>>>         at
>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1764)
>>>         at
>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
>>>         at
>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.readField(BinaryReaderExImpl.java:1984)
>>>         at
>>> org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read0(BinaryFieldAccessor.java:703)
>>>         at
>>> org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:188)
>>>         at
>>> org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:875)
>>>         at
>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1764)
>>>         at
>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
>>>         at
>>> org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313)
>>>         at
>>> org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102)
>>>         at
>>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
>>>         at
>>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140)
>>>         at
>>> org.apache.ignite.internal.processors.job.GridJobWorker.initialize(GridJobWorker.java:440)
>>>         at
>>> org.apache.ignite.internal.processors.job.GridJobProcessor.processJobExecuteRequest(GridJobProcessor.java:1119)
>>>         at
>>> org.apache.ignite.internal.processors.job.GridJobProcessor$JobExecutionListener.onMessage(GridJobProcessor.java:1923)
>>>         at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
>>>         at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
>>>         at
>>> org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
>>>         at
>>> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
>>>         at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>         at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>         at java.lang.Thread.run(Thread.java:748)
>>> Caused by: class org.apache.ignite.binary.BinaryObjectException: Failed
>>> to deserialize object [typeName=io.grpc.internal.ServerImpl]
>>>         at
>>> org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:914)
>>>         at
>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1764)
>>>         at
>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
>>>         at
>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.readField(BinaryReaderExImpl.java:1984)
>>>         at
>>> org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read0(BinaryFieldAccessor.java:703)
>>>         at
>>> org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:188)
>>>         ... 29 more
>>> Caused by: class org.apache.ignite.binary.BinaryObjectException: Failed
>>> to read field [name=registry]
>>>         at
>>> org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:192)
>>>         at
>>> org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:875)
>>>         ... 34 more
>>> Caused by: class org.apache.ignite.binary.BinaryObjectException: Failed
>>> to deserialize object [typeName=io.grpc.internal.InternalHandlerRegistry]
>>>         at
>>> org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:914)
>>>         at
>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1764)
>>>         at
>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
>>>         at
>>> org.apache.ignite.internal.binary.BinaryReaderExImpl.readField(BinaryReaderExImpl.java:1984)
>>>         at
>>> org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read0(BinaryFieldAccessor.java:703)
>>>         at
>>> org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:188)
>>>         ... 35 more
>>>
>>> =======================================
>>>
>>> Thanks and kind regards,
>>> Pascoe
>>>
>>

Re: Question on submitted post

Posted by Pascoe Scholle <pa...@gmail.com>.
Hi Stan,

Thanks for your response. I have tried this, but it has not fixed the
issue.
The grpc server class was moved into the service where the interface
methods "init","execute" and "cancel" perform an initialization of the
serve, as well start and stopping respectively. But this was already
implemented.

Does this issue not have something to do with class loading? The second
node that is started doesn't have any information of the custom classes for
serialization. I have also moved all JARS of these custom classes into the
libs folder in ignite but that has also not helped.

On Fri, Aug 23, 2019 at 6:56 PM Stanislav Lukyanov <st...@gmail.com>
wrote:

> Hi,
>
> It looks like the issue is that you're ending up sending an instance of
> your gRPC server inside your service. This approach is generally incorrect.
> What you should do is
> - not pass gRPC to the service instance
> - add an init() method implementation to your service
> - in your init() start your gRPC server
>
> Stan
>
> On Thu, Aug 22, 2019 at 10:52 AM Pascoe Scholle <pa...@gmail.com>
> wrote:
>
>> Hi there,
>>
>> How long does it usually take, for a post to be seen on the forum? Maybe
>> I made a mistake so I will provide my question here. Excuse me if I am
>> being impatient:
>>
>>
>> =========================================
>> Good afternoon everyone,
>>
>> I have recently run into an issue and I think the problem lies in the
>> server node configuration. I will attache the output of the stack trace at
>> the bottom, however I first wish to explain what the software does and how
>> we are using ignite.
>>
>> I start multiple server nodes with peerClassEnabled set to true, using  a
>> TcpDiscoveryVmIpFinder and dont set anything other than a port range for
>> the ipFinder.
>>
>> Using the REST protocol a ComputeTaskAdapter task is executed which
>> starts a service and this in turn starts a grpc server, I have placed some
>> scala code to show what I mean.
>>
>> class StartService extends ComputeTaskAdapter[String, Any]{
>>   var ignite: Ignite = null;
>>   @IgniteInstanceResource
>>   def setIgnite(someIgnite: Ignite): Unit = {
>>     ignite = someIgnite
>>   }
>>
>>  def map(...)={
>> ...
>> // port is an integer
>> val server = new GrpcServer(ignite, port);
>>
>> val service = new ServiceImpl(name, server);
>> /*
>> within the method execute of the Service interface, server.start() is
>> called
>> */
>>
>> val serviceconfig = new ServiceConfiguration();
>>           serviceconfig.setName(name);
>>           serviceconfig.setTotalCount(1);
>>           serviceconfig.setMaxPerNodeCount(1);
>>           ignite.services().deploy(serviceconfig);
>> ...
>> }
>>
>> }
>>
>> this task returns a map with some non important variables.
>>
>> The grpc server takes the ignite instance created within the above
>> mentioned computeTask as a variable, I am not sure if this could be the
>> cause of the issue.
>>
>> Using grpc protocol, we create a ComputeTask which is executed by the
>> grpc server some more code below:
>>
>> class GrpcServer(val ignite:Ignite, val port:Int) extends ..Some Grpc
>> stuff..{
>>
>> def someGrpcProtocol(request: Message):Future[String]={
>> val newTask = new SomeTask();
>>
>> ignite.compute(ignite.cluster()).execute(newTask, someinput);
>> Future("Request is being processed");
>> }
>>
>> }
>>
>>
>> If a single server node is started, the program runs without problems.
>> However, adding more nodes and trying to execute the new tasks on a remote
>> node or on a node that has a certain attribute gives me a massive stack
>> trace in the face.
>> Basically, if I want to execute a task on a node where the service and
>> grpc server do not reside, the exception happens.
>>
>> I have placed all custom classes within a jar that lies in the libs
>> folder of the ignite-bin project.
>> We are currently on version 2.7
>>
>> If you require anything else just let me know, ill be on it asap.
>>
>> Thanks for any help that may come my way.
>>
>> Cheers!
>>
>> Here is most of the stack trace:
>> class org.apache.ignite.binary.BinaryObjectException: Failed to read
>> field [name=server]
>>         at
>> org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:192)
>>         at
>> org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:875)
>>         at
>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1764)
>>         at
>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
>>         at
>> org.apache.ignite.internal.binary.BinaryReaderExImpl.readField(BinaryReaderExImpl.java:1984)
>>         at
>> org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read0(BinaryFieldAccessor.java:703)
>>         at
>> org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:188)
>>         at
>> org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:875)
>>         at
>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1764)
>>         at
>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
>>         at
>> org.apache.ignite.internal.binary.BinaryReaderExImpl.readField(BinaryReaderExImpl.java:1984)
>>         at
>> org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read0(BinaryFieldAccessor.java:703)
>>         at
>> org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:188)
>>         at
>> org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:875)
>>         at
>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1764)
>>         at
>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
>>         at
>> org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313)
>>         at
>> org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102)
>>         at
>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
>>         at
>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140)
>>         at
>> org.apache.ignite.internal.processors.job.GridJobWorker.initialize(GridJobWorker.java:440)
>>         at
>> org.apache.ignite.internal.processors.job.GridJobProcessor.processJobExecuteRequest(GridJobProcessor.java:1119)
>>         at
>> org.apache.ignite.internal.processors.job.GridJobProcessor$JobExecutionListener.onMessage(GridJobProcessor.java:1923)
>>         at
>> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
>>         at
>> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
>>         at
>> org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
>>         at
>> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
>>         at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>         at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>         at java.lang.Thread.run(Thread.java:748)
>> Caused by: class org.apache.ignite.binary.BinaryObjectException: Failed
>> to deserialize object [typeName=io.grpc.internal.ServerImpl]
>>         at
>> org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:914)
>>         at
>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1764)
>>         at
>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
>>         at
>> org.apache.ignite.internal.binary.BinaryReaderExImpl.readField(BinaryReaderExImpl.java:1984)
>>         at
>> org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read0(BinaryFieldAccessor.java:703)
>>         at
>> org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:188)
>>         ... 29 more
>> Caused by: class org.apache.ignite.binary.BinaryObjectException: Failed
>> to read field [name=registry]
>>         at
>> org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:192)
>>         at
>> org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:875)
>>         ... 34 more
>> Caused by: class org.apache.ignite.binary.BinaryObjectException: Failed
>> to deserialize object [typeName=io.grpc.internal.InternalHandlerRegistry]
>>         at
>> org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:914)
>>         at
>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1764)
>>         at
>> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
>>         at
>> org.apache.ignite.internal.binary.BinaryReaderExImpl.readField(BinaryReaderExImpl.java:1984)
>>         at
>> org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read0(BinaryFieldAccessor.java:703)
>>         at
>> org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:188)
>>         ... 35 more
>>
>> =======================================
>>
>> Thanks and kind regards,
>> Pascoe
>>
>

Re: Question on submitted post

Posted by Stanislav Lukyanov <st...@gmail.com>.
Hi,

It looks like the issue is that you're ending up sending an instance of
your gRPC server inside your service. This approach is generally incorrect.
What you should do is
- not pass gRPC to the service instance
- add an init() method implementation to your service
- in your init() start your gRPC server

Stan

On Thu, Aug 22, 2019 at 10:52 AM Pascoe Scholle <pa...@gmail.com>
wrote:

> Hi there,
>
> How long does it usually take, for a post to be seen on the forum? Maybe I
> made a mistake so I will provide my question here. Excuse me if I am being
> impatient:
>
>
> =========================================
> Good afternoon everyone,
>
> I have recently run into an issue and I think the problem lies in the
> server node configuration. I will attache the output of the stack trace at
> the bottom, however I first wish to explain what the software does and how
> we are using ignite.
>
> I start multiple server nodes with peerClassEnabled set to true, using  a
> TcpDiscoveryVmIpFinder and dont set anything other than a port range for
> the ipFinder.
>
> Using the REST protocol a ComputeTaskAdapter task is executed which starts
> a service and this in turn starts a grpc server, I have placed some scala
> code to show what I mean.
>
> class StartService extends ComputeTaskAdapter[String, Any]{
>   var ignite: Ignite = null;
>   @IgniteInstanceResource
>   def setIgnite(someIgnite: Ignite): Unit = {
>     ignite = someIgnite
>   }
>
>  def map(...)={
> ...
> // port is an integer
> val server = new GrpcServer(ignite, port);
>
> val service = new ServiceImpl(name, server);
> /*
> within the method execute of the Service interface, server.start() is
> called
> */
>
> val serviceconfig = new ServiceConfiguration();
>           serviceconfig.setName(name);
>           serviceconfig.setTotalCount(1);
>           serviceconfig.setMaxPerNodeCount(1);
>           ignite.services().deploy(serviceconfig);
> ...
> }
>
> }
>
> this task returns a map with some non important variables.
>
> The grpc server takes the ignite instance created within the above
> mentioned computeTask as a variable, I am not sure if this could be the
> cause of the issue.
>
> Using grpc protocol, we create a ComputeTask which is executed by the grpc
> server some more code below:
>
> class GrpcServer(val ignite:Ignite, val port:Int) extends ..Some Grpc
> stuff..{
>
> def someGrpcProtocol(request: Message):Future[String]={
> val newTask = new SomeTask();
>
> ignite.compute(ignite.cluster()).execute(newTask, someinput);
> Future("Request is being processed");
> }
>
> }
>
>
> If a single server node is started, the program runs without problems.
> However, adding more nodes and trying to execute the new tasks on a remote
> node or on a node that has a certain attribute gives me a massive stack
> trace in the face.
> Basically, if I want to execute a task on a node where the service and
> grpc server do not reside, the exception happens.
>
> I have placed all custom classes within a jar that lies in the libs folder
> of the ignite-bin project.
> We are currently on version 2.7
>
> If you require anything else just let me know, ill be on it asap.
>
> Thanks for any help that may come my way.
>
> Cheers!
>
> Here is most of the stack trace:
> class org.apache.ignite.binary.BinaryObjectException: Failed to read field
> [name=server]
>         at
> org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:192)
>         at
> org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:875)
>         at
> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1764)
>         at
> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
>         at
> org.apache.ignite.internal.binary.BinaryReaderExImpl.readField(BinaryReaderExImpl.java:1984)
>         at
> org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read0(BinaryFieldAccessor.java:703)
>         at
> org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:188)
>         at
> org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:875)
>         at
> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1764)
>         at
> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
>         at
> org.apache.ignite.internal.binary.BinaryReaderExImpl.readField(BinaryReaderExImpl.java:1984)
>         at
> org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read0(BinaryFieldAccessor.java:703)
>         at
> org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:188)
>         at
> org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:875)
>         at
> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1764)
>         at
> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
>         at
> org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313)
>         at
> org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102)
>         at
> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
>         at
> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140)
>         at
> org.apache.ignite.internal.processors.job.GridJobWorker.initialize(GridJobWorker.java:440)
>         at
> org.apache.ignite.internal.processors.job.GridJobProcessor.processJobExecuteRequest(GridJobProcessor.java:1119)
>         at
> org.apache.ignite.internal.processors.job.GridJobProcessor$JobExecutionListener.onMessage(GridJobProcessor.java:1923)
>         at
> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
>         at
> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
>         at
> org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
>         at
> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
> Caused by: class org.apache.ignite.binary.BinaryObjectException: Failed to
> deserialize object [typeName=io.grpc.internal.ServerImpl]
>         at
> org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:914)
>         at
> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1764)
>         at
> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
>         at
> org.apache.ignite.internal.binary.BinaryReaderExImpl.readField(BinaryReaderExImpl.java:1984)
>         at
> org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read0(BinaryFieldAccessor.java:703)
>         at
> org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:188)
>         ... 29 more
> Caused by: class org.apache.ignite.binary.BinaryObjectException: Failed to
> read field [name=registry]
>         at
> org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:192)
>         at
> org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:875)
>         ... 34 more
> Caused by: class org.apache.ignite.binary.BinaryObjectException: Failed to
> deserialize object [typeName=io.grpc.internal.InternalHandlerRegistry]
>         at
> org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:914)
>         at
> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1764)
>         at
> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
>         at
> org.apache.ignite.internal.binary.BinaryReaderExImpl.readField(BinaryReaderExImpl.java:1984)
>         at
> org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read0(BinaryFieldAccessor.java:703)
>         at
> org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:188)
>         ... 35 more
>
> =======================================
>
> Thanks and kind regards,
> Pascoe
>