You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@gobblin.apache.org by ap...@gmail.com on 2019/06/14 13:42:11 UTC
Apache Gobblin Gitter messages at 2019/06/14 06:42:09
####

Julien Zouein (julien.zouein_gitlab)
**[2019-06-14T07:50:17.829Z]** hello, I followed the documentation to do the
Kafka-HDFS ingestion.
* * *
####

Julien Zouein (julien.zouein_gitlab)
**[2019-06-14T07:50:27.643Z]** I want to run it in MapReduce mode
* * *
####

Julien Zouein (julien.zouein_gitlab)
**[2019-06-14T07:50:31.537Z]** and got this error :
* * *
####

Julien Zouein (julien.zouein_gitlab)
**[2019-06-14T07:50:33.318Z]** SLF4J: Class path contains multiple SLF4J
bindings.
SLF4J: Found binding in [jar:file:/home/hadooprd/gobblin-
dist/lib/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/home/hadooprd/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See <http://www.slf4j.org/codes.html#multiple_bindings> for an
explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Exception in thread "main" org.apache.gobblin.runtime.JobException: Job
job_GobblinKafkaQuickStart_1560498407992 failed
at
org.apache.gobblin.runtime.AbstractJobLauncher.launchJob(AbstractJobLauncher.java:526)
at
org.apache.gobblin.runtime.mapreduce.CliMRJobLauncher.launchJob(CliMRJobLauncher.java:89)
at
org.apache.gobblin.runtime.mapreduce.CliMRJobLauncher.run(CliMRJobLauncher.java:66)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at
org.apache.gobblin.runtime.mapreduce.CliMRJobLauncher.main(CliMRJobLauncher.java:111)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
* * *
####

Julien Zouein (julien.zouein_gitlab)
**[2019-06-14T07:51:03.842Z]** The log file tells me it can 't get any
workUnit
* * *
####

Julien Zouein (julien.zouein_gitlab)
**[2019-06-14T07:51:47.254Z]** moreover when I try to use it, it tells me that
scala-library-2.11.8.jar does not exist so I changed the version in gobblin-
mapreduce.sh
* * *
####

Julien Zouein (julien.zouein_gitlab)
**[2019-06-14T07:57:00.352Z]** I try to run it on a hadoop Cluster
* * *
####

Julien Zouein (julien.zouein_gitlab)
**[2019-06-14T07:57:05.943Z]** all services are launched
* * *
####

Julien Zouein (julien.zouein_gitlab)
**[2019-06-14T07:57:15.175Z]** 22033 SecondaryNameNode
23458 GradleDaemon
22531 HQuorumPeer
5299 GradleDaemon
25477 Jps
28922 ResourceManager
29291 NodeManager
21563 NameNode
* * *
####

Julien Zouein (julien.zouein_gitlab)
**[2019-06-14T07:57:34.916Z]** Hadoop version : 2.8.5
* * *
####

Julien Zouein (julien.zouein_gitlab)
**[2019-06-14T07:57:53.935Z]** openjdk version "1.8.0_212"
* * *
####

Julien Zouein (julien.zouein_gitlab)
**[2019-06-14T08:02:32.946Z]** Ubuntu 18.04.2 LTS (GNU/Linux 4.15.0-50-generic
x86_64)
* * *
####

Julien Zouein (julien.zouein_gitlab)
**[2019-06-14T08:04:04.730Z]** log file :
019-06-14 07:47:43 UTC INFO [main] org.apache.hadoop.conf.Configuration 996 -
mapreduce.user.classpath.first is deprecated. Instead, use
mapreduce.job.user.classpath.first
2019-06-14 07:47:44 UTC INFO [main] org.apache.gobblin.util.ClustersNames 74 -
no default cluster mapping found
2019-06-14 07:47:44 UTC WARN [main]
org.apache.gobblin.runtime.AbstractJobLauncher 217 - Creating a job specific
SharedResourcesBroker. Objects will only be shared at the job level.
2019-06-14 07:47:44 UTC INFO [main] org.reflections.Reflections 229 -
Reflections took 383 ms to scan 49 urls, producing 842 keys and 2961 values
2019-06-14 07:47:44 UTC WARN [main] org.apache.gobblin.runtime.JobContext 347
- Property writer.staging.dir is deprecated. No need to use it if
task.data.root.dir is specified.
2019-06-14 07:47:44 UTC WARN [main] org.apache.gobblin.runtime.JobContext 366
- Property writer.output.dir is deprecated. No need to use it if
task.data.root.dir is specified.
2019-06-14 07:47:45 UTC INFO [main]
org.apache.gobblin.runtime.app.ServiceBasedAppLauncher 165 - Starting the
Gobblin application and all its associated Services
2019-06-14 07:47:45 UTC INFO [MetricsReportingService STARTING]
org.apache.gobblin.metrics.GobblinMetrics 477 - Reporting metrics to log files
2019-06-14 07:47:45 UTC INFO [MetricsReportingService STARTING]
org.apache.gobblin.metrics.GobblinMetrics 520 - Will start reporting metrics
to directory /gobblin-kafka/metrics/app_CliMRJob-0f1cbab1-b666-44ef-
ba6f-43e905e76af0_1560498464087
2019-06-14 07:47:45 UTC INFO [MetricsReportingService STARTING]
org.apache.gobblin.metrics.GobblinMetrics 531 - Reporting failure to log files
2019-06-14 07:47:45 UTC ERROR [MetricsReportingService STARTING]
org.apache.gobblin.metrics.GobblinMetrics 534 - Not reporting failure to log
files because failure.log.dir is undefined
2019-06-14 07:47:45 UTC INFO [MetricsReportingService STARTING]
org.apache.gobblin.metrics.GobblinMetrics 428 - Metrics reporting has been
started in 237 ms: GobblinMetrics
[org.apache.gobblin.metrics.GobblinMetrics@47dcb309](mailto:org.apache.gobblin.metrics.GobblinMetrics@47dcb309)
2019-06-14 07:47:45 UTC INFO [main] org.apache.gobblin.util.ExecutorsUtils 186
- Attempting to shutdown ExecutorService:
[java.util.concurrent.ThreadPoolExecutor@2efd2f21](mailto:java.util.concurrent.ThreadPoolExecutor@2efd2f21)[Shutting
down, pool size = 1, active threads = 0, queued tasks = 0, completed tasks =
1]
2019-06-14 07:47:45 UTC INFO [main] org.apache.gobblin.util.ExecutorsUtils 205
- Successfully shutdown ExecutorService:
[java.util.concurrent.ThreadPoolExecutor@2efd2f21](mailto:java.util.concurrent.ThreadPoolExecutor@2efd2f21)[Terminated,
pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1]
2019-06-14 07:47:45 UTC INFO [main]
org.apache.gobblin.runtime.AbstractJobLauncher 968 - Submitting
JobPrepareTimer
2019-06-14 07:47:45 UTC INFO [main]
org.apache.gobblin.kafka.client.Kafka08ConsumerClient 143 - Fetching topic
metadata from broker kafkablade-group.net
2019-06-14 07:47:45 UTC ERROR [main]
org.apache.gobblin.runtime.SourceDecorator 88 - Failed to get work units for
job job_GobblinKafkaQuickStart_1560498464274
java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
at java.util.ArrayList.rangeCheck(ArrayList.java:657)
at java.util.ArrayList.get(ArrayList.java:433)
at java.util.Collections$UnmodifiableList.get(Collections.java:1309)
at
org.apache.gobblin.kafka.client.Kafka08ConsumerClient.createSimpleConsumer(Kafka08ConsumerClient.java:182)
at
org.apache.gobblin.kafka.client.Kafka08ConsumerClient.getSimpleConsumer(Kafka08ConsumerClient.java:171)
at
org.apache.gobblin.kafka.client.Kafka08ConsumerClient.fetchTopicMetadataFromBroker(Kafka08ConsumerClient.java:146)
at
org.apache.gobblin.kafka.client.Kafka08ConsumerClient.getFilteredMetadataList(Kafka08ConsumerClient.java:133)
at
org.apache.gobblin.kafka.client.Kafka08ConsumerClient.getTopics(Kafka08ConsumerClient.java:100)
at
org.apache.gobblin.kafka.client.AbstractBaseKafkaConsumerClient.getFilteredTopics(AbstractBaseKafkaConsumerClient.java:79)
at org.apache.gobblin.source.extractor.extract.kafka.KafkaSour
* * *