You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pulsar.apache.org by Geoffroy Fouquier <ge...@exensa.com> on 2018/06/08 08:09:26 UTC

Non-serializable builders

We are using pulsar 1.22 within a spark framework and I am currently 
upgrading my cluster to pulsar 2.0. One of the main change concerns the 
configuration classes, replace by builders. Although ProducerBuilder and 
ConsumerBuilder interfaces implement Serializable, corresponding 
implementations aren't, because a pulsar client object is embedded. The 
schema object is also a problem for serialization.


Basically, with 1.22 my producer app. instanciate a client and a 
producer on each executor and only configurations classes need to be 
serialized

---8<---

WarcRecordExtractor
         .load(sc, input)
         .foreachPartition{ ite =>

             val client = PulsarClient.create(ServiceUrl, clientConf)
             val producer = client.createProducer(topic, producerConf)
             ite.foreach{ x => producer.sendAsync(x._2.content) }
             producer.close()

             client.close()

         }

---8<---


With 2.0 I try to replace the configuration classes with the 
corresponding builders, but this setting doesn't work since the 
producerBuilder isn't serializable :

---8<---

             [...]

             val client = clientBuilder.serviceUrl(serviceUrl).build()
             val producer = producerBuilder.topic(topicName).create

             [...]

---8<---


My actual workaround is to rewrite my own ProducerBuilderImpl without  
pulsar client or schemas (the createAsync method still exists but throw 
an exception). And then to instanciate a producer like this :

---8<---

             [...]

             val client : PulsarClientImpl = 
clientBuilder.serviceUrl(broker).build().asInstanceOf[PulsarClientImpl]
             val producer = 
client.createProducerAsync(producerBuilder.topic(topic).getConf(), 
org.apache.pulsar.client.api.Schema.BYTES).get()

             [...]

---8<---

where producerBuilder is my own implementation which implements 
ProducerBuilder interface.


I did the same thing for ConsumerBuilderImpl and also rewrite 
SparkStreamingPulsarReceiver with the same kind of workaround. I don't 
know why the configuration classes have been replaced by builders, but 
it became more difficult to use it with spark (and probably with other 
distributed framework), but maybe i did it wrong.




Re: Non-serializable builders

Posted by Sijie Guo <gu...@gmail.com>.
Hi Geoffroy,

I put up a PR https://github.com/apache/incubator-pulsar/pull/2004 for
addressing the issue. If you can take a look and see if that approach
addresses your problem.

- Sijie

On Fri, Jun 8, 2018 at 8:32 AM Sijie Guo <gu...@gmail.com> wrote:

> Hi Geoffroy,
>
> I created an issue for
> https://github.com/apache/incubator-pulsar/issues/1943.
>
> We will address it for the upcoming 2.1 release.
>
> - Sijie
>
> On Fri, Jun 8, 2018 at 6:39 AM Geoffroy Fouquier <
> geoffroy.fouquier@exensa.com> wrote:
>
>> On 08/06/2018 12:01, Sijie Guo wrote:
>> > Geoffroy,
>> >
>> > Thank you for reporting this. The change was made to evolve pulsar into
>> a
>> > type-safe client with schema and more validations. Along with the idea,
>> > we changed the client to become a more fluent style builder and hide the
>> > configuration data as an implementation detail to the pulsar client
>> > implementation.
>> >
>> > However we missed the use cases in data computing world, people tends to
>> > use java Serializable on sharing configuration.
>> >
>> > We would incorporate your use cases and try to bring the configuration
>> > object back.
>> >
>> > - Sijie
>>
>> That would be great, thanks!
>>
>> >
>> >
>> > On Fri, Jun 8, 2018 at 1:09 AM Geoffroy Fouquier <
>> > geoffroy.fouquier@exensa.com> wrote:
>> >
>> >> We are using pulsar 1.22 within a spark framework and I am currently
>> >> upgrading my cluster to pulsar 2.0. One of the main change concerns the
>> >> configuration classes, replace by builders. Although ProducerBuilder
>> and
>> >> ConsumerBuilder interfaces implement Serializable, corresponding
>> >> implementations aren't, because a pulsar client object is embedded. The
>> >> schema object is also a problem for serialization.
>> >>
>> >>
>> >> Basically, with 1.22 my producer app. instanciate a client and a
>> >> producer on each executor and only configurations classes need to be
>> >> serialized
>> >>
>> >> ---8<---
>> >>
>> >> WarcRecordExtractor
>> >>           .load(sc, input)
>> >>           .foreachPartition{ ite =>
>> >>
>> >>               val client = PulsarClient.create(ServiceUrl, clientConf)
>> >>               val producer = client.createProducer(topic, producerConf)
>> >>               ite.foreach{ x => producer.sendAsync(x._2.content) }
>> >>               producer.close()
>> >>
>> >>               client.close()
>> >>
>> >>           }
>> >>
>> >> ---8<---
>> >>
>> >>
>> >> With 2.0 I try to replace the configuration classes with the
>> >> corresponding builders, but this setting doesn't work since the
>> >> producerBuilder isn't serializable :
>> >>
>> >> ---8<---
>> >>
>> >>               [...]
>> >>
>> >>               val client = clientBuilder.serviceUrl(serviceUrl).build()
>> >>               val producer = producerBuilder.topic(topicName).create
>> >>
>> >>               [...]
>> >>
>> >> ---8<---
>> >>
>> >>
>> >> My actual workaround is to rewrite my own ProducerBuilderImpl without
>> >> pulsar client or schemas (the createAsync method still exists but throw
>> >> an exception). And then to instanciate a producer like this :
>> >>
>> >> ---8<---
>> >>
>> >>               [...]
>> >>
>> >>               val client : PulsarClientImpl =
>> >> clientBuilder.serviceUrl(broker).build().asInstanceOf[PulsarClientImpl]
>> >>               val producer =
>> >> client.createProducerAsync(producerBuilder.topic(topic).getConf(),
>> >> org.apache.pulsar.client.api.Schema.BYTES).get()
>> >>
>> >>               [...]
>> >>
>> >> ---8<---
>> >>
>> >> where producerBuilder is my own implementation which implements
>> >> ProducerBuilder interface.
>> >>
>> >>
>> >> I did the same thing for ConsumerBuilderImpl and also rewrite
>> >> SparkStreamingPulsarReceiver with the same kind of workaround. I don't
>> >> know why the configuration classes have been replaced by builders, but
>> >> it became more difficult to use it with spark (and probably with other
>> >> distributed framework), but maybe i did it wrong.
>> >>
>> >>
>> >>
>> >>
>>
>>

Re: Non-serializable builders

Posted by Sijie Guo <gu...@gmail.com>.
Hi Geoffroy,

I created an issue for
https://github.com/apache/incubator-pulsar/issues/1943.

We will address it for the upcoming 2.1 release.

- Sijie

On Fri, Jun 8, 2018 at 6:39 AM Geoffroy Fouquier <
geoffroy.fouquier@exensa.com> wrote:

> On 08/06/2018 12:01, Sijie Guo wrote:
> > Geoffroy,
> >
> > Thank you for reporting this. The change was made to evolve pulsar into a
> > type-safe client with schema and more validations. Along with the idea,
> > we changed the client to become a more fluent style builder and hide the
> > configuration data as an implementation detail to the pulsar client
> > implementation.
> >
> > However we missed the use cases in data computing world, people tends to
> > use java Serializable on sharing configuration.
> >
> > We would incorporate your use cases and try to bring the configuration
> > object back.
> >
> > - Sijie
>
> That would be great, thanks!
>
> >
> >
> > On Fri, Jun 8, 2018 at 1:09 AM Geoffroy Fouquier <
> > geoffroy.fouquier@exensa.com> wrote:
> >
> >> We are using pulsar 1.22 within a spark framework and I am currently
> >> upgrading my cluster to pulsar 2.0. One of the main change concerns the
> >> configuration classes, replace by builders. Although ProducerBuilder and
> >> ConsumerBuilder interfaces implement Serializable, corresponding
> >> implementations aren't, because a pulsar client object is embedded. The
> >> schema object is also a problem for serialization.
> >>
> >>
> >> Basically, with 1.22 my producer app. instanciate a client and a
> >> producer on each executor and only configurations classes need to be
> >> serialized
> >>
> >> ---8<---
> >>
> >> WarcRecordExtractor
> >>           .load(sc, input)
> >>           .foreachPartition{ ite =>
> >>
> >>               val client = PulsarClient.create(ServiceUrl, clientConf)
> >>               val producer = client.createProducer(topic, producerConf)
> >>               ite.foreach{ x => producer.sendAsync(x._2.content) }
> >>               producer.close()
> >>
> >>               client.close()
> >>
> >>           }
> >>
> >> ---8<---
> >>
> >>
> >> With 2.0 I try to replace the configuration classes with the
> >> corresponding builders, but this setting doesn't work since the
> >> producerBuilder isn't serializable :
> >>
> >> ---8<---
> >>
> >>               [...]
> >>
> >>               val client = clientBuilder.serviceUrl(serviceUrl).build()
> >>               val producer = producerBuilder.topic(topicName).create
> >>
> >>               [...]
> >>
> >> ---8<---
> >>
> >>
> >> My actual workaround is to rewrite my own ProducerBuilderImpl without
> >> pulsar client or schemas (the createAsync method still exists but throw
> >> an exception). And then to instanciate a producer like this :
> >>
> >> ---8<---
> >>
> >>               [...]
> >>
> >>               val client : PulsarClientImpl =
> >> clientBuilder.serviceUrl(broker).build().asInstanceOf[PulsarClientImpl]
> >>               val producer =
> >> client.createProducerAsync(producerBuilder.topic(topic).getConf(),
> >> org.apache.pulsar.client.api.Schema.BYTES).get()
> >>
> >>               [...]
> >>
> >> ---8<---
> >>
> >> where producerBuilder is my own implementation which implements
> >> ProducerBuilder interface.
> >>
> >>
> >> I did the same thing for ConsumerBuilderImpl and also rewrite
> >> SparkStreamingPulsarReceiver with the same kind of workaround. I don't
> >> know why the configuration classes have been replaced by builders, but
> >> it became more difficult to use it with spark (and probably with other
> >> distributed framework), but maybe i did it wrong.
> >>
> >>
> >>
> >>
>
>

Re: Non-serializable builders

Posted by Geoffroy Fouquier <ge...@exensa.com>.
On 08/06/2018 12:01, Sijie Guo wrote:
> Geoffroy,
>
> Thank you for reporting this. The change was made to evolve pulsar into a
> type-safe client with schema and more validations. Along with the idea,
> we changed the client to become a more fluent style builder and hide the
> configuration data as an implementation detail to the pulsar client
> implementation.
>
> However we missed the use cases in data computing world, people tends to
> use java Serializable on sharing configuration.
>
> We would incorporate your use cases and try to bring the configuration
> object back.
>
> - Sijie

That would be great, thanks!

>
>
> On Fri, Jun 8, 2018 at 1:09 AM Geoffroy Fouquier <
> geoffroy.fouquier@exensa.com> wrote:
>
>> We are using pulsar 1.22 within a spark framework and I am currently
>> upgrading my cluster to pulsar 2.0. One of the main change concerns the
>> configuration classes, replace by builders. Although ProducerBuilder and
>> ConsumerBuilder interfaces implement Serializable, corresponding
>> implementations aren't, because a pulsar client object is embedded. The
>> schema object is also a problem for serialization.
>>
>>
>> Basically, with 1.22 my producer app. instanciate a client and a
>> producer on each executor and only configurations classes need to be
>> serialized
>>
>> ---8<---
>>
>> WarcRecordExtractor
>>           .load(sc, input)
>>           .foreachPartition{ ite =>
>>
>>               val client = PulsarClient.create(ServiceUrl, clientConf)
>>               val producer = client.createProducer(topic, producerConf)
>>               ite.foreach{ x => producer.sendAsync(x._2.content) }
>>               producer.close()
>>
>>               client.close()
>>
>>           }
>>
>> ---8<---
>>
>>
>> With 2.0 I try to replace the configuration classes with the
>> corresponding builders, but this setting doesn't work since the
>> producerBuilder isn't serializable :
>>
>> ---8<---
>>
>>               [...]
>>
>>               val client = clientBuilder.serviceUrl(serviceUrl).build()
>>               val producer = producerBuilder.topic(topicName).create
>>
>>               [...]
>>
>> ---8<---
>>
>>
>> My actual workaround is to rewrite my own ProducerBuilderImpl without
>> pulsar client or schemas (the createAsync method still exists but throw
>> an exception). And then to instanciate a producer like this :
>>
>> ---8<---
>>
>>               [...]
>>
>>               val client : PulsarClientImpl =
>> clientBuilder.serviceUrl(broker).build().asInstanceOf[PulsarClientImpl]
>>               val producer =
>> client.createProducerAsync(producerBuilder.topic(topic).getConf(),
>> org.apache.pulsar.client.api.Schema.BYTES).get()
>>
>>               [...]
>>
>> ---8<---
>>
>> where producerBuilder is my own implementation which implements
>> ProducerBuilder interface.
>>
>>
>> I did the same thing for ConsumerBuilderImpl and also rewrite
>> SparkStreamingPulsarReceiver with the same kind of workaround. I don't
>> know why the configuration classes have been replaced by builders, but
>> it became more difficult to use it with spark (and probably with other
>> distributed framework), but maybe i did it wrong.
>>
>>
>>
>>


Re: Non-serializable builders

Posted by Sijie Guo <gu...@gmail.com>.
Geoffroy,

Thank you for reporting this. The change was made to evolve pulsar into a
type-safe client with schema and more validations. Along with the idea,
we changed the client to become a more fluent style builder and hide the
configuration data as an implementation detail to the pulsar client
implementation.

However we missed the use cases in data computing world, people tends to
use java Serializable on sharing configuration.

We would incorporate your use cases and try to bring the configuration
object back.

- Sijie



On Fri, Jun 8, 2018 at 1:09 AM Geoffroy Fouquier <
geoffroy.fouquier@exensa.com> wrote:

>
> We are using pulsar 1.22 within a spark framework and I am currently
> upgrading my cluster to pulsar 2.0. One of the main change concerns the
> configuration classes, replace by builders. Although ProducerBuilder and
> ConsumerBuilder interfaces implement Serializable, corresponding
> implementations aren't, because a pulsar client object is embedded. The
> schema object is also a problem for serialization.
>
>
> Basically, with 1.22 my producer app. instanciate a client and a
> producer on each executor and only configurations classes need to be
> serialized
>
> ---8<---
>
> WarcRecordExtractor
>          .load(sc, input)
>          .foreachPartition{ ite =>
>
>              val client = PulsarClient.create(ServiceUrl, clientConf)
>              val producer = client.createProducer(topic, producerConf)
>              ite.foreach{ x => producer.sendAsync(x._2.content) }
>              producer.close()
>
>              client.close()
>
>          }
>
> ---8<---
>
>
> With 2.0 I try to replace the configuration classes with the
> corresponding builders, but this setting doesn't work since the
> producerBuilder isn't serializable :
>
> ---8<---
>
>              [...]
>
>              val client = clientBuilder.serviceUrl(serviceUrl).build()
>              val producer = producerBuilder.topic(topicName).create
>
>              [...]
>
> ---8<---
>
>
> My actual workaround is to rewrite my own ProducerBuilderImpl without
> pulsar client or schemas (the createAsync method still exists but throw
> an exception). And then to instanciate a producer like this :
>
> ---8<---
>
>              [...]
>
>              val client : PulsarClientImpl =
> clientBuilder.serviceUrl(broker).build().asInstanceOf[PulsarClientImpl]
>              val producer =
> client.createProducerAsync(producerBuilder.topic(topic).getConf(),
> org.apache.pulsar.client.api.Schema.BYTES).get()
>
>              [...]
>
> ---8<---
>
> where producerBuilder is my own implementation which implements
> ProducerBuilder interface.
>
>
> I did the same thing for ConsumerBuilderImpl and also rewrite
> SparkStreamingPulsarReceiver with the same kind of workaround. I don't
> know why the configuration classes have been replaced by builders, but
> it became more difficult to use it with spark (and probably with other
> distributed framework), but maybe i did it wrong.
>
>
>
>