You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@gobblin.apache.org by ap...@gmail.com on 2019/06/14 13:42:11 UTC

Apache Gobblin Gitter messages at 2019/06/14 06:42:09

####
![](https://secure.gravatar.com/avatar/edbc8c9505693916a14afcd547fee83e?s=60&d=identicon)
Julien Zouein (julien.zouein_gitlab)

**[2019-06-14T07:50:17.829Z]** hello, I followed the documentation to do the
Kafka-HDFS ingestion.

* * *

####
![](https://secure.gravatar.com/avatar/edbc8c9505693916a14afcd547fee83e?s=60&d=identicon)
Julien Zouein (julien.zouein_gitlab)

**[2019-06-14T07:50:27.643Z]** I want to run it in MapReduce mode

* * *

####
![](https://secure.gravatar.com/avatar/edbc8c9505693916a14afcd547fee83e?s=60&d=identicon)
Julien Zouein (julien.zouein_gitlab)

**[2019-06-14T07:50:31.537Z]** and got this error :

* * *

####
![](https://secure.gravatar.com/avatar/edbc8c9505693916a14afcd547fee83e?s=60&d=identicon)
Julien Zouein (julien.zouein_gitlab)

**[2019-06-14T07:50:33.318Z]** SLF4J: Class path contains multiple SLF4J
bindings.  
SLF4J: Found binding in [jar:file:/home/hadooprd/gobblin-
dist/lib/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/StaticLoggerBinder.class]  
SLF4J: Found binding in
[jar:file:/home/hadooprd/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]  
SLF4J: See <http://www.slf4j.org/codes.html#multiple_bindings> for an
explanation.  
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]  
Exception in thread "main" org.apache.gobblin.runtime.JobException: Job
job_GobblinKafkaQuickStart_1560498407992 failed  
at
org.apache.gobblin.runtime.AbstractJobLauncher.launchJob(AbstractJobLauncher.java:526)  
at
org.apache.gobblin.runtime.mapreduce.CliMRJobLauncher.launchJob(CliMRJobLauncher.java:89)  
at
org.apache.gobblin.runtime.mapreduce.CliMRJobLauncher.run(CliMRJobLauncher.java:66)  
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)  
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)  
at
org.apache.gobblin.runtime.mapreduce.CliMRJobLauncher.main(CliMRJobLauncher.java:111)  
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)  
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)  
at java.lang.reflect.Method.invoke(Method.java:498)  
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)

* * *

####
![](https://secure.gravatar.com/avatar/edbc8c9505693916a14afcd547fee83e?s=60&d=identicon)
Julien Zouein (julien.zouein_gitlab)

**[2019-06-14T07:51:03.842Z]** The log file tells me it can 't get any
workUnit

* * *

####
![](https://secure.gravatar.com/avatar/edbc8c9505693916a14afcd547fee83e?s=60&d=identicon)
Julien Zouein (julien.zouein_gitlab)

**[2019-06-14T07:51:47.254Z]** moreover when I try to use it, it tells me that
scala-library-2.11.8.jar does not exist so I changed the version in gobblin-
mapreduce.sh

* * *

####
![](https://secure.gravatar.com/avatar/edbc8c9505693916a14afcd547fee83e?s=60&d=identicon)
Julien Zouein (julien.zouein_gitlab)

**[2019-06-14T07:57:00.352Z]** I try to run it on a hadoop Cluster

* * *

####
![](https://secure.gravatar.com/avatar/edbc8c9505693916a14afcd547fee83e?s=60&d=identicon)
Julien Zouein (julien.zouein_gitlab)

**[2019-06-14T07:57:05.943Z]** all services are launched

* * *

####
![](https://secure.gravatar.com/avatar/edbc8c9505693916a14afcd547fee83e?s=60&d=identicon)
Julien Zouein (julien.zouein_gitlab)

**[2019-06-14T07:57:15.175Z]** 22033 SecondaryNameNode  
23458 GradleDaemon  
22531 HQuorumPeer  
5299 GradleDaemon  
25477 Jps  
28922 ResourceManager  
29291 NodeManager  
21563 NameNode

* * *

####
![](https://secure.gravatar.com/avatar/edbc8c9505693916a14afcd547fee83e?s=60&d=identicon)
Julien Zouein (julien.zouein_gitlab)

**[2019-06-14T07:57:34.916Z]** Hadoop version : 2.8.5

* * *

####
![](https://secure.gravatar.com/avatar/edbc8c9505693916a14afcd547fee83e?s=60&d=identicon)
Julien Zouein (julien.zouein_gitlab)

**[2019-06-14T07:57:53.935Z]** openjdk version  "1.8.0_212"

* * *

####
![](https://secure.gravatar.com/avatar/edbc8c9505693916a14afcd547fee83e?s=60&d=identicon)
Julien Zouein (julien.zouein_gitlab)

**[2019-06-14T08:02:32.946Z]** Ubuntu 18.04.2 LTS (GNU/Linux 4.15.0-50-generic
x86_64)

* * *

####
![](https://secure.gravatar.com/avatar/edbc8c9505693916a14afcd547fee83e?s=60&d=identicon)
Julien Zouein (julien.zouein_gitlab)

**[2019-06-14T08:04:04.730Z]** log file :  
019-06-14 07:47:43 UTC INFO [main] org.apache.hadoop.conf.Configuration 996 -
mapreduce.user.classpath.first is deprecated. Instead, use
mapreduce.job.user.classpath.first  
2019-06-14 07:47:44 UTC INFO [main] org.apache.gobblin.util.ClustersNames 74 -
no default cluster mapping found  
2019-06-14 07:47:44 UTC WARN [main]
org.apache.gobblin.runtime.AbstractJobLauncher 217 - Creating a job specific
SharedResourcesBroker. Objects will only be shared at the job level.  
2019-06-14 07:47:44 UTC INFO [main] org.reflections.Reflections 229 -
Reflections took 383 ms to scan 49 urls, producing 842 keys and 2961 values  
2019-06-14 07:47:44 UTC WARN [main] org.apache.gobblin.runtime.JobContext 347
- Property writer.staging.dir is deprecated. No need to use it if
task.data.root.dir is specified.  
2019-06-14 07:47:44 UTC WARN [main] org.apache.gobblin.runtime.JobContext 366
- Property writer.output.dir is deprecated. No need to use it if
task.data.root.dir is specified.  
2019-06-14 07:47:45 UTC INFO [main]
org.apache.gobblin.runtime.app.ServiceBasedAppLauncher 165 - Starting the
Gobblin application and all its associated Services  
2019-06-14 07:47:45 UTC INFO [MetricsReportingService STARTING]
org.apache.gobblin.metrics.GobblinMetrics 477 - Reporting metrics to log files  
2019-06-14 07:47:45 UTC INFO [MetricsReportingService STARTING]
org.apache.gobblin.metrics.GobblinMetrics 520 - Will start reporting metrics
to directory /gobblin-kafka/metrics/app_CliMRJob-0f1cbab1-b666-44ef-
ba6f-43e905e76af0_1560498464087  
2019-06-14 07:47:45 UTC INFO [MetricsReportingService STARTING]
org.apache.gobblin.metrics.GobblinMetrics 531 - Reporting failure to log files  
2019-06-14 07:47:45 UTC ERROR [MetricsReportingService STARTING]
org.apache.gobblin.metrics.GobblinMetrics 534 - Not reporting failure to log
files because failure.log.dir is undefined  
2019-06-14 07:47:45 UTC INFO [MetricsReportingService STARTING]
org.apache.gobblin.metrics.GobblinMetrics 428 - Metrics reporting has been
started in 237 ms: GobblinMetrics
[org.apache.gobblin.metrics.GobblinMetrics@47dcb309](mailto:org.apache.gobblin.metrics.GobblinMetrics@47dcb309)  
2019-06-14 07:47:45 UTC INFO [main] org.apache.gobblin.util.ExecutorsUtils 186
- Attempting to shutdown ExecutorService:
[java.util.concurrent.ThreadPoolExecutor@2efd2f21](mailto:java.util.concurrent.ThreadPoolExecutor@2efd2f21)[Shutting
down, pool size = 1, active threads = 0, queued tasks = 0, completed tasks =
1]  
2019-06-14 07:47:45 UTC INFO [main] org.apache.gobblin.util.ExecutorsUtils 205
- Successfully shutdown ExecutorService:
[java.util.concurrent.ThreadPoolExecutor@2efd2f21](mailto:java.util.concurrent.ThreadPoolExecutor@2efd2f21)[Terminated,
pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1]  
2019-06-14 07:47:45 UTC INFO [main]
org.apache.gobblin.runtime.AbstractJobLauncher 968 - Submitting
JobPrepareTimer  
2019-06-14 07:47:45 UTC INFO [main]
org.apache.gobblin.kafka.client.Kafka08ConsumerClient 143 - Fetching topic
metadata from broker kafkablade-group.net  
2019-06-14 07:47:45 UTC ERROR [main]
org.apache.gobblin.runtime.SourceDecorator 88 - Failed to get work units for
job job_GobblinKafkaQuickStart_1560498464274  
java.lang.IndexOutOfBoundsException: Index: 1, Size: 1  
at java.util.ArrayList.rangeCheck(ArrayList.java:657)  
at java.util.ArrayList.get(ArrayList.java:433)  
at java.util.Collections$UnmodifiableList.get(Collections.java:1309)  
at
org.apache.gobblin.kafka.client.Kafka08ConsumerClient.createSimpleConsumer(Kafka08ConsumerClient.java:182)  
at
org.apache.gobblin.kafka.client.Kafka08ConsumerClient.getSimpleConsumer(Kafka08ConsumerClient.java:171)  
at
org.apache.gobblin.kafka.client.Kafka08ConsumerClient.fetchTopicMetadataFromBroker(Kafka08ConsumerClient.java:146)  
at
org.apache.gobblin.kafka.client.Kafka08ConsumerClient.getFilteredMetadataList(Kafka08ConsumerClient.java:133)  
at
org.apache.gobblin.kafka.client.Kafka08ConsumerClient.getTopics(Kafka08ConsumerClient.java:100)  
at
org.apache.gobblin.kafka.client.AbstractBaseKafkaConsumerClient.getFilteredTopics(AbstractBaseKafkaConsumerClient.java:79)  
at org.apache.gobblin.source.extractor.extract.kafka.KafkaSour

* * *