Pushing from Pig to Kafka

[Stefan Kupstaitis-Dunkler <st...@gmail.com>]

Hi all!

I accidently posted this on the dev mailing list yesterday, when it's much
better off here. Can anybody help me?

When I run a pig script I get:

java.lang.InstantiationError: org.apache.avro.io.BinaryEncoder
at
org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
Caused by: java.lang.InstantiationError: org.apache.avro.io.BinaryEncoder
at
kafka.bridge.pig.AvroKafkaStorage.prepareToWrite(AvroKafkaStorage.java:77)

I am using a HDP 2.2 installation which comes with Kafka 0.8.1. I try to
push a single string from Pig into Kafka and follow these instructions:
https://github.com/apache/kafka/blob/43b92f8b1ce8140c432edf11b0c842f5fbe04120/contrib/hadoop-producer/README.md

The script looks like:
REGISTER
/usr/hdp/current/kafka-broker/contrib/lib/kafka-hadoop-producer-0.8.1.2.2.0.0-2041.jar;
REGISTER /usr/hdp/current/pig-client/lib/avro-1.4.1.jar;
REGISTER /usr/hdp/current/pig-client/piggybank.jar;
REGISTER
/usr/hdp/current/kafka-broker/libs/kafka_2.10-0.8.1.2.2.0.0-2041.jar;
REGISTER /usr/hdp/current/pig-client/lib/jackson-core-asl-1.8.8.jar;
REGISTER /usr/hdp/current/pig-client/lib/jackson-mapper-asl-1.8.8.jar;
REGISTER /usr/hdp/current/kafka-broker/libs/scala-library-2.10.4.jar;
STORE my_data INTO 'kafka://MY-BROKER-HOST-NAME:6667/myTopic' USING
kafka.bridge.pig.AvroKafkaStorage('"string"');

MY-BROKER-HOST-NAME is one of my Kafka brokers, myTopic is an existing
topic. The version 1.4.1 of avro is not deployed by default, but I deployed
it and registered it instead of avro 1.7.5, because I first thought this
will solve my problem, but the same problem occurs. Also, I noticed, that
the same problom still occurs, when I do not register avro at all. When I
look into the avro documentation I can see, that the BinaryEncoder class is
abstract, but nevertheless is tried to be instantiated in the
AvroKakfaStorage class. So the error is not a surprise. Is there anything I
am missing to make this work? Am I doing something wrong?

Best regards, Stefan

Re: Pushing from Pig to Kafka

[Joe Stein <jo...@stealth.ly>]

Hey Stefan, I don't think it is much a matter for which list a lot of folks
read both (depending on from where/whom you are expecting answer some).

It has been ~ 2 years since folks have been operationalizing that part of
the code (for a lot of reasons mostly map/reduce kind of having other
different options, etc).

We should maybe think about deprecating or reducing its support (which
probably is there on the latter already) such the case might be.

If you really need to go from Pig <-> Kafka and Kafka <-> Pig you may want
to find out from the Pig community their best support for integrated
systems... I suspect that integrated system will support Kafka too :)

The Hadoop community has spent A LOT of time since 0.8 making all of that
ecosystem more cohesive within that community. I urge support for it always.

~ Joe Stein
- - - - - - - - - - - - - - - - - - -
     [image: Logo-Black.jpg]
  http://www.elodina.net
    http://www.stealth.ly
- - - - - - - - - - - - - - - - - - -

On Mon, Sep 28, 2015 at 4:11 AM, Stefan Kupstaitis-Dunkler <
stefan.dun@gmail.com> wrote:

> Hi all!
>
> I accidently posted this on the dev mailing list yesterday, when it's much
> better off here. Can anybody help me?
>
> When I run a pig script I get:
>
> java.lang.InstantiationError: org.apache.avro.io.BinaryEncoder
> at
>
> org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
> at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
> Caused by: java.lang.InstantiationError: org.apache.avro.io.BinaryEncoder
> at
> kafka.bridge.pig.AvroKafkaStorage.prepareToWrite(AvroKafkaStorage.java:77)
>
> I am using a HDP 2.2 installation which comes with Kafka 0.8.1. I try to
> push a single string from Pig into Kafka and follow these instructions:
>
> https://github.com/apache/kafka/blob/43b92f8b1ce8140c432edf11b0c842f5fbe04120/contrib/hadoop-producer/README.md
>
> The script looks like:
> REGISTER
>
> /usr/hdp/current/kafka-broker/contrib/lib/kafka-hadoop-producer-0.8.1.2.2.0.0-2041.jar;
> REGISTER /usr/hdp/current/pig-client/lib/avro-1.4.1.jar;
> REGISTER /usr/hdp/current/pig-client/piggybank.jar;
> REGISTER
> /usr/hdp/current/kafka-broker/libs/kafka_2.10-0.8.1.2.2.0.0-2041.jar;
> REGISTER /usr/hdp/current/pig-client/lib/jackson-core-asl-1.8.8.jar;
> REGISTER /usr/hdp/current/pig-client/lib/jackson-mapper-asl-1.8.8.jar;
> REGISTER /usr/hdp/current/kafka-broker/libs/scala-library-2.10.4.jar;
> STORE my_data INTO 'kafka://MY-BROKER-HOST-NAME:6667/myTopic' USING
> kafka.bridge.pig.AvroKafkaStorage('"string"');
>
> MY-BROKER-HOST-NAME is one of my Kafka brokers, myTopic is an existing
> topic. The version 1.4.1 of avro is not deployed by default, but I deployed
> it and registered it instead of avro 1.7.5, because I first thought this
> will solve my problem, but the same problem occurs. Also, I noticed, that
> the same problom still occurs, when I do not register avro at all. When I
> look into the avro documentation I can see, that the BinaryEncoder class is
> abstract, but nevertheless is tried to be instantiated in the
> AvroKakfaStorage class. So the error is not a surprise. Is there anything I
> am missing to make this work? Am I doing something wrong?
>
> Best regards, Stefan
>