You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@geode.apache.org by Paul Perez <pa...@pymma.com> on 2017/04/28 17:02:15 UTC
Client cache and pdf serialization
Hello all
We develop a kind of aggregation features for our monitoring system.
In our geode cache, we use the pdx serialisation for our java objets. They will not be used elsewhere.
So come two questions :
First we developed event listeners to aggregate the event of our products. Once the handler receives the asyncEvent, we have to deserialize it. Do we use pdx features here as well?
Seconds question: one of our application use a client cache for memory issues. As far as we understood, pdx serialization cannot be defined in the cache.xml.
So, if the client cache doesn't support pdx so are we forced to use the classical java serialization to send the ojects to the servers?
Thank you for your response
Paul
Re: Client cache and pdf serialization
Posted by Darrel Schneider <ds...@pivotal.io>.
What kind of event listeners are you using. Are they CacheListener
instances?
On Fri, Apr 28, 2017 at 10:02 AM, Paul Perez <pa...@pymma.com> wrote:
> Hello all
> We develop a kind of aggregation features for our monitoring system.
> In our geode cache, we use the pdx serialisation for our java objets. They
> will not be used elsewhere.
> So come two questions :
> First we developed event listeners to aggregate the event of our products.
> Once the handler receives the asyncEvent, we have to deserialize it. Do we
> use pdx features here as well?
>
> Seconds question: one of our application use a client cache for memory
> issues. As far as we understood, pdx serialization cannot be defined in
> the cache.xml.
> So, if the client cache doesn't support pdx so are we forced to use the
> classical java serialization to send the ojects to the servers?
>
> Thank you for your response
>
> Paul
>
>
RE: Client cache and pdf serialization
Posted by Paul Perez <pa...@pymma.com>.
Hello All,
First I would like to thank you for the prompt replies we received and apologise for the delay to reply to you.
I’ll to reply to everyone and provide more detail on what we do and what we want to achieve.
John wrote: Is there a specific reason why you have to deserialize it?
The question can be understood in two ways. The first is why do you want to use PXD? Our Response is simple because we read that is was faster that the classical Serialisation. The Second is why do you use serialisation? To be honest with you, I never think about that. We use serialisation because in our project, a Java class named “publisher” put messages to the cache (client cache) and serialisation support is required. (The publisher uses a client cache to access to the Geode members since we don’t want to increase the memory size of the application using the publisher. Nevertheless, if we learn that using a classical cache we will improve significantly the message insertion in the cache we can do it).
We use Geode in the following way: For each process running on our product, the publisher pushes many message type in regions (one Region per message type) then when the process is over, a last message is put in its Region.
We capture that event through an Async Queue and an Async Event Handler gather the messages related to the process from different regions and aggregate them. AFAIK, we need to deserialize the object from the event if we want to use the message in the handler.
private void aggregateEvent(AsyncEvent<String, BPELEnterpriseEvent> geodeEvent) {
BPELEnterpriseEvent bpelEventStop = geodeEvent.getDeserializedValue();
May be using pdx feature would in that case would be more Efficient ? I don’t know.
Otherwise, for the moment on low resource machines, we are close to our objectives and aggregate around 10K events per second with one server and hope processing 3 to 4 more event after optimisation and migration on decent servers.
Regards
Paul
From: Dave Barnes [mailto:dbarnes@pivotal.io]
Sent: 28 April 2017 19:53
To: user@geode.apache.org
Subject: Re: Client cache and pdf serialization
FYI, I created a ticket for the doc bug that John pointed out:
GEODE-2843 User Guide - example should specify <client-cache>
On Fri, Apr 28, 2017 at 11:13 AM, John Blum <jblum@pivotal.io <ma...@pivotal.io> > wrote:
Hi Paul-
Answers/comments inline below...
On Fri, Apr 28, 2017 at 10:02 AM, Paul Perez <paul.perez@pymma.com <ma...@pymma.com> > wrote:
Hello all
We develop a kind of aggregation features for our monitoring system.
In our geode cache, we use the pdx serialisation for our java objets. They will not be used elsewhere.
So come two questions :
First we developed event listeners to aggregate the event of our products. Once the handler receives the asyncEvent, we have to deserialize it. Do we use pdx features here as well?
Is there a specific reason why you have to deserialize it? You can work with PdxInstance(s) <http://geode.apache.org/releases/latest/javadoc/index.html> [1] directly (getting/setting fields <http://geode.apache.org/releases/latest/javadoc/org/apache/geode/pdx/PdxInstance.html#createWriter--> [2] of the object's type without deserializing). If necessary, you can always get the deserialized, object instance <http://geode.apache.org/releases/latest/javadoc/org/apache/geode/pdx/PdxInstance.html#getObject--> [3] of the data (providing the class type is on the classpath of the server where AEQ Listener is firing). It is even possible to query (with OQL) PDX serialized bytes without deserializing (unlike Java serialized bytes).
However, you should be careful in that once deserialized, Geode will retain the deserialized form of the data in the Region. This will only cause overhead when the data needs to be re-serialized, which is anytime the data is persisted/overflowed to disk, replicated between peers, sent across a WAN or sent to clients.
You can learn more here <http://geode.apache.org/docs/guide/11/developing/data_serialization/gemfire_pdx_serialization.html> [4].
[1] http://geode.apache.org/releases/latest/javadoc/index.html
[2] http://geode.apache.org/releases/latest/javadoc/org/apache/geode/pdx/PdxInstance.html#createWriter--
[3] http://geode.apache.org/releases/latest/javadoc/org/apache/geode/pdx/PdxInstance.html#getObject--
[4] http://geode.apache.org/docs/guide/11/developing/data_serialization/gemfire_pdx_serialization.html
Seconds question: one of our application use a client cache for memory issues. As far as we understood, pdx serialization cannot be defined in the cache.xml.
So, if the client cache doesn't support pdx so are we forced to use the classical java serialization to send the ojects to the servers?
Why do you think that? See here <http://geode.apache.org/docs/guide/11/reference/topics/client-cache.html#cc-pdx> [5].
Unfortunately, the example is WRONG, as it is using the "peer" cache element tag (<cache> instead of <client-cache>), but this reference ([5]) is from client-cache section and it is possible to configure PDX on the client. You must configure PDX serialization on the client to serialize bytes to the server from the client using PDX. Server-side PDX configuration maybe optional if you have nothing running in the servers directly (e.g. a Function or CacheLoader) since the server will retain the data in the form that it receives, unless, as I mention, above, something were to cause the data to be deserialized. Then, you need to specify your serialization strategy, e.g. either by your application domain objects implementing <http://geode.apache.org/docs/guide/11/developing/data_serialization/use_pdx_serializable.html> PdxSerializable [6](invasive, and not recommended) or by implementing a <http://geode.apache.org/docs/guide/11/developing/data_serialization/use_pdx_serializer.html> PdxSerializer [7] (or perhaps using a OOTB impl... ReflectionBasedAutoSerializer <http://geode.apache.org/docs/guide/11/developing/data_serialization/auto_serialization.html> [8], though personally, I think implementing your own PdxSerializer implementations for your application domain objects is better... you can even get creative using the Composite Software Design Pattern to create per domain type serializers but treat it as 1 since Geode only allows you to register 1).
Anyway, it is possible to use PDX serialization on the client.
Hope this helps!
Cheers,
John
[5] http://geode.apache.org/docs/guide/11/reference/topics/client-cache.html#cc-pdx
[6] http://geode.apache.org/docs/guide/11/developing/data_serialization/use_pdx_serializable.html
[7] http://geode.apache.org/docs/guide/11/developing/data_serialization/use_pdx_serializable.html
[8] http://geode.apache.org/docs/guide/11/developing/data_serialization/auto_serialization.html
Thank you for your response
Paul
--
-John
john.blum10101 (skype)
Re: Client cache and pdf serialization
Posted by Dave Barnes <db...@pivotal.io>.
FYI, I created a ticket for the doc bug that John pointed out:
GEODE-2843 User Guide - example should specify <client-cache>
On Fri, Apr 28, 2017 at 11:13 AM, John Blum <jb...@pivotal.io> wrote:
> Hi Paul-
>
> Answers/comments inline below...
>
>
> On Fri, Apr 28, 2017 at 10:02 AM, Paul Perez <pa...@pymma.com> wrote:
>
>> Hello all
>> We develop a kind of aggregation features for our monitoring system.
>> In our geode cache, we use the pdx serialisation for our java objets.
>> They will not be used elsewhere.
>> So come two questions :
>> First we developed event listeners to aggregate the event of our
>> products. Once the handler receives the asyncEvent, we have to deserialize
>> it. Do we use pdx features here as well?
>>
>
> Is there a specific reason why you have to deserialize it? You can work
> with PdxInstance(s)
> <http://geode.apache.org/releases/latest/javadoc/index.html> [1] directly
> (getting/setting fields
> <http://geode.apache.org/releases/latest/javadoc/org/apache/geode/pdx/PdxInstance.html#createWriter--> [2]
> of the object's type without deserializing). If necessary, you can always get
> the deserialized, object instance
> <http://geode.apache.org/releases/latest/javadoc/org/apache/geode/pdx/PdxInstance.html#getObject--> [3]
> of the data (providing the class type is on the classpath of the server
> where AEQ Listener is firing). It is even possible to query (with OQL) PDX
> serialized bytes without deserializing (unlike Java serialized bytes).
>
> However, you should be careful in that once deserialized, Geode will
> retain the deserialized form of the data in the Region. This will only
> cause overhead when the data needs to be re-serialized, which is anytime
> the data is persisted/overflowed to disk, replicated between peers, sent
> across a WAN or sent to clients.
>
> You can learn more here
> <http://geode.apache.org/docs/guide/11/developing/data_serialization/gemfire_pdx_serialization.html>
> [4].
>
> [1] http://geode.apache.org/releases/latest/javadoc/index.html
> [2] http://geode.apache.org/releases/latest/javadoc/org/
> apache/geode/pdx/PdxInstance.html#createWriter--
> [3] http://geode.apache.org/releases/latest/javadoc/org/
> apache/geode/pdx/PdxInstance.html#getObject--
> [4] http://geode.apache.org/docs/guide/11/developing/data_
> serialization/gemfire_pdx_serialization.html
>
>
>> Seconds question: one of our application use a client cache for memory
>> issues. As far as we understood, pdx serialization cannot be defined in
>> the cache.xml.
>> So, if the client cache doesn't support pdx so are we forced to use the
>> classical java serialization to send the ojects to the servers?
>>
>
> Why do you think that? See here
> <http://geode.apache.org/docs/guide/11/reference/topics/client-cache.html#cc-pdx>
> [5].
>
> Unfortunately, the example is WRONG, as it is using the "peer" cache
> element tag (<cache> instead of <client-cache>), but this reference ([5])
> is from client-cache section and it is possible to configure PDX on the
> client. You must configure PDX serialization on the client to serialize
> bytes to the server from the client using PDX. Server-side PDX
> configuration maybe optional if you have nothing running in the servers
> directly (e.g. a Function or CacheLoader) since the server will retain
> the data in the form that it receives, unless, as I mention, above,
> something were to cause the data to be deserialized. Then, you need to
> specify your serialization strategy, e.g. either by your application
> domain objects implementing PdxSerializable
> <http://geode.apache.org/docs/guide/11/developing/data_serialization/use_pdx_serializable.html> [6](invasive,
> and not recommended) or by implementing a PdxSerializer
> <http://geode.apache.org/docs/guide/11/developing/data_serialization/use_pdx_serializer.html> [7]
> (or perhaps using a OOTB impl... ReflectionBasedAutoSerializer
> <http://geode.apache.org/docs/guide/11/developing/data_serialization/auto_serialization.html>
> [8], though personally, I think implementing your own PdxSerializer
> implementations for your application domain objects is better... you can
> even get creative using the *Composite Software Design Pattern* to create
> per domain type serializers but treat it as 1 since Geode only allows you
> to register 1).
>
> Anyway, it is possible to use PDX serialization on the client.
>
> Hope this helps!
>
> Cheers,
> John
>
>
> [5] http://geode.apache.org/docs/guide/11/reference/
> topics/client-cache.html#cc-pdx
> [6] http://geode.apache.org/docs/guide/11/developing/data_
> serialization/use_pdx_serializable.html
> [7] http://geode.apache.org/docs/guide/11/developing/data_
> serialization/use_pdx_serializable.html
> [8] http://geode.apache.org/docs/guide/11/developing/data_
> serialization/auto_serialization.html
>
>
>>
>> Thank you for your response
>>
>> Paul
>>
>>
>
>
> --
> -John
> john.blum10101 (skype)
>
Re: Client cache and pdf serialization
Posted by Darrel Schneider <ds...@pivotal.io>.
One enhancement to John's excellent answers.
John wrote:
> If necessary, you can always get the deserialized, object instance
> <http://geode.apache.org/releases/latest/javadoc/org/apache/geode/pdx/PdxInstance.html#getObject--> [3]
> of the data
and:
> However, you should be careful in that once deserialized, Geode will
> retain the deserialized form of the data in the Region.
I just wanted to clarify that if you do call "getObject" on a PdxInstance
then that will not cause the deserialized form to retained. Only the caller
of "getObject" has the deserialized form object reference returned by this
method.
On Fri, Apr 28, 2017 at 11:13 AM, John Blum <jb...@pivotal.io> wrote:
> Hi Paul-
>
> Answers/comments inline below...
>
>
> On Fri, Apr 28, 2017 at 10:02 AM, Paul Perez <pa...@pymma.com> wrote:
>
>> Hello all
>> We develop a kind of aggregation features for our monitoring system.
>> In our geode cache, we use the pdx serialisation for our java objets.
>> They will not be used elsewhere.
>> So come two questions :
>> First we developed event listeners to aggregate the event of our
>> products. Once the handler receives the asyncEvent, we have to deserialize
>> it. Do we use pdx features here as well?
>>
>
> Is there a specific reason why you have to deserialize it? You can work
> with PdxInstance(s)
> <http://geode.apache.org/releases/latest/javadoc/index.html> [1] directly
> (getting/setting fields
> <http://geode.apache.org/releases/latest/javadoc/org/apache/geode/pdx/PdxInstance.html#createWriter--> [2]
> of the object's type without deserializing). If necessary, you can always get
> the deserialized, object instance
> <http://geode.apache.org/releases/latest/javadoc/org/apache/geode/pdx/PdxInstance.html#getObject--> [3]
> of the data (providing the class type is on the classpath of the server
> where AEQ Listener is firing). It is even possible to query (with OQL) PDX
> serialized bytes without deserializing (unlike Java serialized bytes).
>
> However, you should be careful in that once deserialized, Geode will
> retain the deserialized form of the data in the Region. This will only
> cause overhead when the data needs to be re-serialized, which is anytime
> the data is persisted/overflowed to disk, replicated between peers, sent
> across a WAN or sent to clients.
>
> You can learn more here
> <http://geode.apache.org/docs/guide/11/developing/data_serialization/gemfire_pdx_serialization.html>
> [4].
>
> [1] http://geode.apache.org/releases/latest/javadoc/index.html
> [2] http://geode.apache.org/releases/latest/javadoc/org/
> apache/geode/pdx/PdxInstance.html#createWriter--
> [3] http://geode.apache.org/releases/latest/javadoc/org/
> apache/geode/pdx/PdxInstance.html#getObject--
> [4] http://geode.apache.org/docs/guide/11/developing/data_
> serialization/gemfire_pdx_serialization.html
>
>
>> Seconds question: one of our application use a client cache for memory
>> issues. As far as we understood, pdx serialization cannot be defined in
>> the cache.xml.
>> So, if the client cache doesn't support pdx so are we forced to use the
>> classical java serialization to send the ojects to the servers?
>>
>
> Why do you think that? See here
> <http://geode.apache.org/docs/guide/11/reference/topics/client-cache.html#cc-pdx>
> [5].
>
> Unfortunately, the example is WRONG, as it is using the "peer" cache
> element tag (<cache> instead of <client-cache>), but this reference ([5])
> is from client-cache section and it is possible to configure PDX on the
> client. You must configure PDX serialization on the client to serialize
> bytes to the server from the client using PDX. Server-side PDX
> configuration maybe optional if you have nothing running in the servers
> directly (e.g. a Function or CacheLoader) since the server will retain
> the data in the form that it receives, unless, as I mention, above,
> something were to cause the data to be deserialized. Then, you need to
> specify your serialization strategy, e.g. either by your application
> domain objects implementing PdxSerializable
> <http://geode.apache.org/docs/guide/11/developing/data_serialization/use_pdx_serializable.html> [6](invasive,
> and not recommended) or by implementing a PdxSerializer
> <http://geode.apache.org/docs/guide/11/developing/data_serialization/use_pdx_serializer.html> [7]
> (or perhaps using a OOTB impl... ReflectionBasedAutoSerializer
> <http://geode.apache.org/docs/guide/11/developing/data_serialization/auto_serialization.html>
> [8], though personally, I think implementing your own PdxSerializer
> implementations for your application domain objects is better... you can
> even get creative using the *Composite Software Design Pattern* to create
> per domain type serializers but treat it as 1 since Geode only allows you
> to register 1).
>
> Anyway, it is possible to use PDX serialization on the client.
>
> Hope this helps!
>
> Cheers,
> John
>
>
> [5] http://geode.apache.org/docs/guide/11/reference/
> topics/client-cache.html#cc-pdx
> [6] http://geode.apache.org/docs/guide/11/developing/data_
> serialization/use_pdx_serializable.html
> [7] http://geode.apache.org/docs/guide/11/developing/data_
> serialization/use_pdx_serializable.html
> [8] http://geode.apache.org/docs/guide/11/developing/data_
> serialization/auto_serialization.html
>
>
>>
>> Thank you for your response
>>
>> Paul
>>
>>
>
>
> --
> -John
> john.blum10101 (skype)
>
Re: Client cache and pdf serialization
Posted by John Blum <jb...@pivotal.io>.
Hi Paul-
Answers/comments inline below...
On Fri, Apr 28, 2017 at 10:02 AM, Paul Perez <pa...@pymma.com> wrote:
> Hello all
> We develop a kind of aggregation features for our monitoring system.
> In our geode cache, we use the pdx serialisation for our java objets. They
> will not be used elsewhere.
> So come two questions :
> First we developed event listeners to aggregate the event of our products.
> Once the handler receives the asyncEvent, we have to deserialize it. Do we
> use pdx features here as well?
>
Is there a specific reason why you have to deserialize it? You can work
with PdxInstance(s)
<http://geode.apache.org/releases/latest/javadoc/index.html> [1]
directly (getting/setting
fields
<http://geode.apache.org/releases/latest/javadoc/org/apache/geode/pdx/PdxInstance.html#createWriter-->
[2]
of the object's type without deserializing). If necessary, you can always get
the deserialized, object instance
<http://geode.apache.org/releases/latest/javadoc/org/apache/geode/pdx/PdxInstance.html#getObject-->
[3]
of the data (providing the class type is on the classpath of the server
where AEQ Listener is firing). It is even possible to query (with OQL) PDX
serialized bytes without deserializing (unlike Java serialized bytes).
However, you should be careful in that once deserialized, Geode will retain
the deserialized form of the data in the Region. This will only cause
overhead when the data needs to be re-serialized, which is anytime the data
is persisted/overflowed to disk, replicated between peers, sent across a
WAN or sent to clients.
You can learn more here
<http://geode.apache.org/docs/guide/11/developing/data_serialization/gemfire_pdx_serialization.html>
[4].
[1] http://geode.apache.org/releases/latest/javadoc/index.html
[2]
http://geode.apache.org/releases/latest/javadoc/org/apache/geode/pdx/PdxInstance.html#createWriter--
[3]
http://geode.apache.org/releases/latest/javadoc/org/apache/geode/pdx/PdxInstance.html#getObject--
[4]
http://geode.apache.org/docs/guide/11/developing/data_serialization/gemfire_pdx_serialization.html
> Seconds question: one of our application use a client cache for memory
> issues. As far as we understood, pdx serialization cannot be defined in
> the cache.xml.
> So, if the client cache doesn't support pdx so are we forced to use the
> classical java serialization to send the ojects to the servers?
>
Why do you think that? See here
<http://geode.apache.org/docs/guide/11/reference/topics/client-cache.html#cc-pdx>
[5].
Unfortunately, the example is WRONG, as it is using the "peer" cache
element tag (<cache> instead of <client-cache>), but this reference ([5])
is from client-cache section and it is possible to configure PDX on the
client. You must configure PDX serialization on the client to serialize
bytes to the server from the client using PDX. Server-side PDX
configuration maybe optional if you have nothing running in the servers
directly (e.g. a Function or CacheLoader) since the server will retain the
data in the form that it receives, unless, as I mention, above, something
were to cause the data to be deserialized. Then, you need to specify your
serialization strategy, e.g. either by your application domain objects
implementing PdxSerializable
<http://geode.apache.org/docs/guide/11/developing/data_serialization/use_pdx_serializable.html>
[6](invasive,
and not recommended) or by implementing a PdxSerializer
<http://geode.apache.org/docs/guide/11/developing/data_serialization/use_pdx_serializer.html>
[7]
(or perhaps using a OOTB impl... ReflectionBasedAutoSerializer
<http://geode.apache.org/docs/guide/11/developing/data_serialization/auto_serialization.html>
[8],
though personally, I think implementing your own PdxSerializer
implementations for your application domain objects is better... you can
even get creative using the *Composite Software Design Pattern* to create
per domain type serializers but treat it as 1 since Geode only allows you
to register 1).
Anyway, it is possible to use PDX serialization on the client.
Hope this helps!
Cheers,
John
[5]
http://geode.apache.org/docs/guide/11/reference/topics/client-cache.html#cc-pdx
[6]
http://geode.apache.org/docs/guide/11/developing/data_serialization/use_pdx_serializable.html
[7]
http://geode.apache.org/docs/guide/11/developing/data_serialization/use_pdx_serializable.html
[8]
http://geode.apache.org/docs/guide/11/developing/data_serialization/auto_serialization.html
>
> Thank you for your response
>
> Paul
>
>
--
-John
john.blum10101 (skype)