You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@avro.apache.org by Vadim Keylis <vk...@gmail.com> on 2014/07/11 00:28:33 UTC
Avro serialization using php
How do I just serialize single message using php? It appears php uses
Container Files serialization approach and for my project I need to
serialize single message without attaching schema as header.
Thanks so much for your help
Re: Avro serialization using php
Posted by Vadim Keylis <vk...@gmail.com>.
Thanks so much Joey. I will take a look at java version and try to
implement in php.
Here is my version of php that does avro serialization and unfortunately it
adds schema to the avro record and uses *avro.codec*
$io = new AvroStringIO();
writers_schema = AvroSchema::parse($_TAG->avroSchema[$topic]->getSchema());
$writer = new AvroIODatumWriter($writers_schema);
$data_writer = new AvroDataIOWriter($io, $writer, $writers_schema);
$data_writer->append($datum);
$data_writer->close();
Thanks,
Vadim
On Jul 10, 2014 6:26 PM, "Joey Echeverria" <jo...@cloudera.com> wrote:
> The schema won't be embedded if you use AvroIODatumWriter directly.
> You'll have to make sure that the schema is available when you later
> deserialize the record, but I assumed you have a solution for that
> already.
>
> If you want to see a Java version of this, we do this in the Kite
> SDK[1] to make sure that we're getting the same types that we'd see in
> deserialized objects:
>
>
> https://github.com/kite-sdk/kite/blob/master/kite-data/kite-data-core/src/main/java/org/kitesdk/data/spi/DataModelUtil.java#L144
>
> Again, there's not schema in the byte array, just the serialized bytes.
>
> -Joey
>
> [1] http://www.kitesdk.org
>
> On Thu, Jul 10, 2014 at 9:18 PM, Vadim Keylis <vk...@gmail.com>
> wrote:
> > Hi Joey. Thanks so much for responding. I can serialize. My problem is
> > decoupling schema that is embedded in the avro message from serialized
> > message. How would I do this in php or very least in Java? I want to
> send to
> > the Kafka node only avro serialized record without embedded schema.
> >
> > On Jul 10, 2014 5:54 PM, "Joey Echeverria" <jo...@cloudera.com> wrote:
> >>
> >> I've not used the PHP bindings, so this may not be the full procedure.
> >>
> >> You should be able to create an AvroIODatumWriter[1] to write to a
> >> string. In order to do that, you need to first create an
> >> AvroStringIO[2] and wrap that in a AvroIOBinaryEncoder[3]. It should
> >> look something like this:
> >>
> >> $io = new AvroStringIO();
> >> $encoder = new AvroIOBinaryEncoder($io);
> >> $writer = new AvroIODataumWriter($schema);
> >> $writer->write($datum, $encoder);
> >> $io->string(); // gets the serialized data as a string which in PHP is
> >> an array of bytes
> >>
> >> -Joey
> >>
> >> [1]
> >>
> https://github.com/apache/avro/blob/trunk/lang/php/lib/avro/datum.php#L73
> >> [2]
> >> https://github.com/apache/avro/blob/trunk/lang/php/lib/avro/io.php#L146
> >> [3]
> >>
> https://github.com/apache/avro/blob/trunk/lang/php/lib/avro/datum.php#L231
> >>
> >>
> >> On Thu, Jul 10, 2014 at 6:28 PM, Vadim Keylis <vk...@gmail.com>
> >> wrote:
> >> > How do I just serialize single message using php? It appears php uses
> >> > Container Files serialization approach and for my project I need to
> >> > serialize single message without attaching schema as header.
> >> >
> >> > Thanks so much for your help
> >> >
> >> >
> >>
> >>
> >>
> >> --
> >> Joey Echeverria
>
>
>
> --
> Joey Echeverria
>
Re: Avro serialization using php
Posted by Joey Echeverria <jo...@cloudera.com>.
The schema won't be embedded if you use AvroIODatumWriter directly.
You'll have to make sure that the schema is available when you later
deserialize the record, but I assumed you have a solution for that
already.
If you want to see a Java version of this, we do this in the Kite
SDK[1] to make sure that we're getting the same types that we'd see in
deserialized objects:
https://github.com/kite-sdk/kite/blob/master/kite-data/kite-data-core/src/main/java/org/kitesdk/data/spi/DataModelUtil.java#L144
Again, there's not schema in the byte array, just the serialized bytes.
-Joey
[1] http://www.kitesdk.org
On Thu, Jul 10, 2014 at 9:18 PM, Vadim Keylis <vk...@gmail.com> wrote:
> Hi Joey. Thanks so much for responding. I can serialize. My problem is
> decoupling schema that is embedded in the avro message from serialized
> message. How would I do this in php or very least in Java? I want to send to
> the Kafka node only avro serialized record without embedded schema.
>
> On Jul 10, 2014 5:54 PM, "Joey Echeverria" <jo...@cloudera.com> wrote:
>>
>> I've not used the PHP bindings, so this may not be the full procedure.
>>
>> You should be able to create an AvroIODatumWriter[1] to write to a
>> string. In order to do that, you need to first create an
>> AvroStringIO[2] and wrap that in a AvroIOBinaryEncoder[3]. It should
>> look something like this:
>>
>> $io = new AvroStringIO();
>> $encoder = new AvroIOBinaryEncoder($io);
>> $writer = new AvroIODataumWriter($schema);
>> $writer->write($datum, $encoder);
>> $io->string(); // gets the serialized data as a string which in PHP is
>> an array of bytes
>>
>> -Joey
>>
>> [1]
>> https://github.com/apache/avro/blob/trunk/lang/php/lib/avro/datum.php#L73
>> [2]
>> https://github.com/apache/avro/blob/trunk/lang/php/lib/avro/io.php#L146
>> [3]
>> https://github.com/apache/avro/blob/trunk/lang/php/lib/avro/datum.php#L231
>>
>>
>> On Thu, Jul 10, 2014 at 6:28 PM, Vadim Keylis <vk...@gmail.com>
>> wrote:
>> > How do I just serialize single message using php? It appears php uses
>> > Container Files serialization approach and for my project I need to
>> > serialize single message without attaching schema as header.
>> >
>> > Thanks so much for your help
>> >
>> >
>>
>>
>>
>> --
>> Joey Echeverria
--
Joey Echeverria
Re: Avro serialization using php
Posted by Vadim Keylis <vk...@gmail.com>.
Hi Joey. Thanks so much for responding. I can serialize. My problem is
decoupling schema that is embedded in the avro message from serialized
message. How would I do this in php or very least in Java? I want to send
to the Kafka node only avro serialized record without embedded schema.
On Jul 10, 2014 5:54 PM, "Joey Echeverria" <jo...@cloudera.com> wrote:
> I've not used the PHP bindings, so this may not be the full procedure.
>
> You should be able to create an AvroIODatumWriter[1] to write to a
> string. In order to do that, you need to first create an
> AvroStringIO[2] and wrap that in a AvroIOBinaryEncoder[3]. It should
> look something like this:
>
> $io = new AvroStringIO();
> $encoder = new AvroIOBinaryEncoder($io);
> $writer = new AvroIODataumWriter($schema);
> $writer->write($datum, $encoder);
> $io->string(); // gets the serialized data as a string which in PHP is
> an array of bytes
>
> -Joey
>
> [1]
> https://github.com/apache/avro/blob/trunk/lang/php/lib/avro/datum.php#L73
> [2]
> https://github.com/apache/avro/blob/trunk/lang/php/lib/avro/io.php#L146
> [3]
> https://github.com/apache/avro/blob/trunk/lang/php/lib/avro/datum.php#L231
>
>
> On Thu, Jul 10, 2014 at 6:28 PM, Vadim Keylis <vk...@gmail.com>
> wrote:
> > How do I just serialize single message using php? It appears php uses
> > Container Files serialization approach and for my project I need to
> > serialize single message without attaching schema as header.
> >
> > Thanks so much for your help
> >
> >
>
>
>
> --
> Joey Echeverria
>
Re: Avro serialization using php
Posted by Joey Echeverria <jo...@cloudera.com>.
I've not used the PHP bindings, so this may not be the full procedure.
You should be able to create an AvroIODatumWriter[1] to write to a
string. In order to do that, you need to first create an
AvroStringIO[2] and wrap that in a AvroIOBinaryEncoder[3]. It should
look something like this:
$io = new AvroStringIO();
$encoder = new AvroIOBinaryEncoder($io);
$writer = new AvroIODataumWriter($schema);
$writer->write($datum, $encoder);
$io->string(); // gets the serialized data as a string which in PHP is
an array of bytes
-Joey
[1] https://github.com/apache/avro/blob/trunk/lang/php/lib/avro/datum.php#L73
[2] https://github.com/apache/avro/blob/trunk/lang/php/lib/avro/io.php#L146
[3] https://github.com/apache/avro/blob/trunk/lang/php/lib/avro/datum.php#L231
On Thu, Jul 10, 2014 at 6:28 PM, Vadim Keylis <vk...@gmail.com> wrote:
> How do I just serialize single message using php? It appears php uses
> Container Files serialization approach and for my project I need to
> serialize single message without attaching schema as header.
>
> Thanks so much for your help
>
>
--
Joey Echeverria