You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gobblin.apache.org by "Abhishek Tiwari (JIRA)" <ji...@apache.org> on 2017/07/19 07:41:01 UTC
[jira] [Created] (GOBBLIN-104) Gobblin fails on Kafka to HDFS pull
job
Abhishek Tiwari created GOBBLIN-104:
---------------------------------------
Summary: Gobblin fails on Kafka to HDFS pull job
Key: GOBBLIN-104
URL: https://issues.apache.org/jira/browse/GOBBLIN-104
Project: Apache Gobblin
Issue Type: Bug
Reporter: Abhishek Tiwari
I'm trying to pull data from Kafka to HDFS using Gobblin.
Gobblin version (compiled from github source code with command `sudo ./gradlew clean build -PuseHadoop2 -PhadoopVersion=2.7.1 -x test`):
```
0.6.2-546-g431188b
```
Hadoop version:
```
Hadoop 2.7.1.2.4.2.0-258
Subversion git@github.com:hortonworks/hadoop.git -r 13debf893a605e8a88df18a7d8d214f571e05289
Compiled by jenkins on 2016-04-24T16:02Z
Compiled with protoc 2.5.0
From source with checksum 2a2d95f05ec6c3ac547ed58cab713ac
This command was run using /usr/hdp/2.4.2.0-258/hadoop/hadoop-common-2.7.1.2.4.2.0-258.jar
```
Gobblin job:
```
job.name=GobblinKafkaQuickStart
job.group=GobblinKafka
job.description=Gobblin quick start job for Kafka
job.lock.enabled=false
job.schedule=0 0/2 * * * ?
kafka.brokers=hd-mgt03:6667,hd-mgt02:6667,hd-mgt04:6667
source.class=gobblin.source.extractor.extract.kafka.KafkaSimpleSource
extract.namespace=gobblin.extract.kafka
writer.builder.class=gobblin.writer.AvroHdfsDataWriter
writer.file.path.type=tablename
writer.destination.type=HDFS
writer.output.format=AVRO
data.publisher.type=gobblin.publisher.BaseDataPublisher
mr.job.max.mappers=1
metrics.reporting.file.enabled=true
metrics.log.dir=/gobblin-kafka/metrics
metrics.reporting.file.suffix=txt
bootstrap.with.offset=earliest
fs.uri=hdfs://hdfs:8020
writer.fs.uri=hdfs://hdfs:8020
state.store.fs.uri=hdfs://hdfs:8020
mr.job.root.dir=/kafka/working
state.store.dir=/kafka/state-store
task.data.root.dir=/kafka/task-data
data.publisher.final.dir=/kafka/job-output
```
I'm trying to run `gobblin-mapreduce.sh` from `gobblin-dist/bin` folder, but getting error:
```
Exception in thread main gobblin.runtime.JobException: Job job_GobblinKafkaQuickStart_1464962113982 failed
at gobblin.runtime.AbstractJobLauncher.launchJob(AbstractJobLauncher.java:363)
at gobblin.runtime.mapreduce.CliMRJobLauncher.launchJob(CliMRJobLauncher.java:84)
at gobblin.runtime.mapreduce.CliMRJobLauncher.run(CliMRJobLauncher.java:61)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at gobblin.runtime.mapreduce.CliMRJobLauncher.main(CliMRJobLauncher.java:106)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
```
Log file contains error:
```
2016-06-03 16:55:17 MSK ERROR [main] gobblin.runtime.AbstractJobLauncher 321 - Failed to launch and run job job_GobblinKafkaQuickStart_1464962113982: java.lang.NoSuchFieldError: DEFAULT_MR_AM_ADMIN_USER_ENV
java.lang.NoSuchFieldError: DEFAULT_MR_AM_ADMIN_USER_ENV
at org.apache.hadoop.mapred.YARNRunner.createApplicationSubmissionContext(YARNRunner.java:470)
at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:285)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:240)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
at gobblin.runtime.mapreduce.MRJobLauncher.runWorkUnits(MRJobLauncher.java:198)
at gobblin.runtime.AbstractJobLauncher.launchJob(AbstractJobLauncher.java:296)
at gobblin.runtime.mapreduce.CliMRJobLauncher.launchJob(CliMRJobLauncher.java:84)
at gobblin.runtime.mapreduce.CliMRJobLauncher.run(CliMRJobLauncher.java:61)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at gobblin.runtime.mapreduce.CliMRJobLauncher.main(CliMRJobLauncher.java:106)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
```
What could be the reason for this error? How can I fix it?
*Github Url* : https://github.com/linkedin/gobblin/issues/1029
*Github Reporter* : *Dim0N22*
*Github Created At* : 2016-06-03T14:35:51Z
*Github Updated At* : 2016-08-02T17:50:23Z
h3. Comments
----
[~stakiar] wrote on 2016-06-09T20:14:55Z : Sometimes this occurs when Gobblin is compiled against a different version of Hadoop vs. the version your cluster is running.
It seems you are using the Hortonworks distribution, so I would recommend compiling Gobblin against the Hortonworks distribution rather than the 2.7.1 version available on Maven Central.
There are direction here for how to do this for the Cloudera distribution, it should be a similar setup for compiling against the Hortonworks distribution: http://gobblin.readthedocs.io/en/latest/user-guide/FAQs/#how-do-i-compile-gobblin-against-cdh
*Github Url* : https://github.com/linkedin/gobblin/issues/1029#issuecomment-225012843
----
[~abti] wrote on 2016-07-13T12:00:41Z : @Dim0N22 were you able to run your job successfully?
*Github Url* : https://github.com/linkedin/gobblin/issues/1029#issuecomment-232334437
----
*SamirPatni* wrote on 2016-08-02T17:50:23Z : Can you please elaborate, what you mean by Hortonworks distribution and not 2.7.1.
HDP 2.3 and 2.4 both are using 2.7.1 versions. Can you please clarify here.
*Github Url* : https://github.com/linkedin/gobblin/issues/1029#issuecomment-236985315
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)