You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@gobblin.apache.org by archit gupta <a....@gmail.com> on 2017/12/06 11:21:27 UTC
Issue while running gobblin Kafka to HDFS example
Hi,
I am following steps at
http://gobblin.readthedocs.io/en/latest/case-studies/Kafka-HDFS-Ingestion/#job-config-properties
I have built gobblin with ./gradlew -PhadoopVersion=2.7.3
-PhiveVersion=1.0.1 -PgobblinFlavor=standard assemble -Pversion=0.11.0
Also I unpacked gobblin jars to check package, it is gobblin.* instead of
org.apache.gobblin.* so accordingly modified the example
The config file is at
https://gist.github.com/archit90/6fee7417bfe4c454fd5cc9fc924a393b#file-kafka-to-hdfs-job-pull
Running vis command bin/gobblin-standalone.sh start
I get this error
https://gist.github.com/archit90/6fee7417bfe4c454fd5cc9fc924a393b#file-gobblin-0-11-kafka-hdfs-error-log
Running via command bin/gobblin-mapreduce.sh start
I get this error
https://gist.github.com/archit90/6fee7417bfe4c454fd5cc9fc924a393b#file-kafka-to-hdfs-mapreduce-error-log
I have Hadoop version 2.8 installed and Kafka version is 0.10.2
How can this be fixed.
Thanks & Regards
Archit Gupta
Re: Issue while running gobblin Kafka to HDFS example
Posted by archit gupta <a....@gmail.com>.
Thanks Sudarshan, I have about hectic today, will try over weekend and
share my findings
Thanks & Regards
Archit Gupta
On Wed, Dec 6, 2017 at 11:24 PM, Sudarshan Vasudevan <
suvasudevan@linkedin.com> wrote:
> Hi Archit,
>
> For the standalone mode, do you have a kafka topic (e.g. “test” topic)
> with some data written to it?
>
>
>
>
>
>
> For the mapreduce mode, the version of data jar expected by
> gobblin-mapreduce.sh is currently incorrect. The gobblin-mapreduce.sh file
> needs to be changed to use data-11.0.0.jar (or the version of data jar you
> have in your lib sub-directory) instead of data-2.6.0.jar. Can you give it
> a try?
>
>
>
> Thanks,
>
> Sudarshan
>
>
>
>
>
> *From: *archit gupta <a....@gmail.com>
> *Reply-To: *"user@gobblin.incubator.apache.org" <user@gobblin.incubator.
> apache.org>
> *Date: *Wednesday, December 6, 2017 at 3:22 AM
> *To: *"user@gobblin.incubator.apache.org" <user@gobblin.incubator.
> apache.org>
> *Subject: *Issue while running gobblin Kafka to HDFS example
>
>
>
> Hi,
>
> I am following steps at
>
> http://gobblin.readthedocs.io/en/latest/case-studies/Kafka-
> HDFS-Ingestion/#job-config-properties
>
>
> I have built gobblin with ./gradlew -PhadoopVersion=2.7.3
> -PhiveVersion=1.0.1 -PgobblinFlavor=standard assemble -Pversion=0.11.0
>
>
>
> Also I unpacked gobblin jars to check package, it is gobblin.* instead of
> org.apache.gobblin.* so accordingly modified the example
>
>
>
> The config file is at https://gist.github.com/archit90/
> 6fee7417bfe4c454fd5cc9fc924a393b#file-kafka-to-hdfs-job-pull
>
>
>
> Running vis command bin/gobblin-standalone.sh start
>
> I get this error https://gist.github.com/archit90/
> 6fee7417bfe4c454fd5cc9fc924a393b#file-gobblin-0-11-kafka-hdfs-error-log
>
>
>
> Running via command bin/gobblin-mapreduce.sh start
>
> I get this error https://gist.github.com/archit90/
> 6fee7417bfe4c454fd5cc9fc924a393b#file-kafka-to-hdfs-mapreduce-error-log
>
>
>
> I have Hadoop version 2.8 installed and Kafka version is 0.10.2
>
>
>
> How can this be fixed.
>
>
>
> Thanks & Regards
>
> Archit Gupta
>
Re: Issue while running gobblin Kafka to HDFS example
Posted by Sudarshan Vasudevan <su...@linkedin.com>.
Hi Archit,
For the standalone mode, do you have a kafka topic (e.g. “test” topic) with some data written to it?
For the mapreduce mode, the version of data jar expected by gobblin-mapreduce.sh is currently incorrect. The gobblin-mapreduce.sh file needs to be changed to use data-11.0.0.jar (or the version of data jar you have in your lib sub-directory) instead of data-2.6.0.jar. Can you give it a try?
Thanks,
Sudarshan
From: archit gupta <a....@gmail.com>
Reply-To: "user@gobblin.incubator.apache.org" <us...@gobblin.incubator.apache.org>
Date: Wednesday, December 6, 2017 at 3:22 AM
To: "user@gobblin.incubator.apache.org" <us...@gobblin.incubator.apache.org>
Subject: Issue while running gobblin Kafka to HDFS example
Hi,
I am following steps at
http://gobblin.readthedocs.io/en/latest/case-studies/Kafka-HDFS-Ingestion/#job-config-properties
I have built gobblin with ./gradlew -PhadoopVersion=2.7.3 -PhiveVersion=1.0.1 -PgobblinFlavor=standard assemble -Pversion=0.11.0
Also I unpacked gobblin jars to check package, it is gobblin.* instead of org.apache.gobblin.* so accordingly modified the example
The config file is at https://gist.github.com/archit90/6fee7417bfe4c454fd5cc9fc924a393b#file-kafka-to-hdfs-job-pull
Running vis command bin/gobblin-standalone.sh start
I get this error https://gist.github.com/archit90/6fee7417bfe4c454fd5cc9fc924a393b#file-gobblin-0-11-kafka-hdfs-error-log
Running via command bin/gobblin-mapreduce.sh start
I get this error https://gist.github.com/archit90/6fee7417bfe4c454fd5cc9fc924a393b#file-kafka-to-hdfs-mapreduce-error-log
I have Hadoop version 2.8 installed and Kafka version is 0.10.2
How can this be fixed.
Thanks & Regards
Archit Gupta