You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by "Phadnis, Varun" <ph...@sky.optymyze.com> on 2017/01/12 13:57:46 UTC
Spark and Kafka integration
Hello,
We are using Spark 2.0 with Kafka 0.10.
As I understand, much of the API packaged in the following dependency we are targeting is marked as "@Experimental"
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming-kafka-0-10_2.11</artifactId>
<version>2.0.0</version>
</dependency>
What are implications of this being marked as experimental? Are they stable enough for production?
Thanks,
Varun
Spark with oozie #Not implemented by the TFS FileSystem# issue
Posted by Rohit Mishra <ro...@mindagroup.com>.
Hello,
I am new to spark.
I need to run a spark job within oozie.
individually i am able to run the spark job but with oozie after the job is launched i am getting the following error:
017-01-12 13:51:57,696 INFO [main] org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state INITED; cause: java.lang.UnsupportedOperationException: Not implemented by the TFS FileSystem implementation
java.lang.UnsupportedOperationException: Not implemented by the TFS FileSystem implementation
at org.apache.hadoop.fs.FileSystem.getScheme(FileSystem.java:216)
at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2564)
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2574)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:169)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.getFileSystem(MRAppMaster.java:497)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:281)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster.java:1499)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1496)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1429)
Spark version: spark-1.5.2-bin-hadoop2.6
Hadoop: hadoop-2.6.2
Hbase : hbase-1.1.5
Oozie: oozie-4.2.0
snapshot of my pom.xml is:
<dependency>
<groupId>org.apache.zookeeper</groupId>
<artifactId>zookeeper</artifactId>
<version>3.4.8</version>
<type>pom</type>
</dependency>
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-common</artifactId>
<version>1.1.5</version>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-client</artifactId>
<version>1.1.5</version>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-server</artifactId>
<version>1.1.5</version>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-testing-util</artifactId>
<version>1.1.5</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>1.5.2</version>
<exclusions>
<exclusion>
<artifactId>javax.servlet</artifactId>
<groupId>org.eclipse.jetty.orbit</groupId>
</exclusion>
</exclusions>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-sql_2.10 -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.11</artifactId>
<version>1.5.2</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-yarn_2.10 -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-yarn_2.11</artifactId>
<version>1.5.2</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.mongodb.mongo-hadoop/mongo-hadoop-core -->
<dependency>
<groupId>org.mongodb.mongo-hadoop</groupId>
<artifactId>mongo-hadoop-core</artifactId>
<version>1.5.2</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-common -->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>2.6.2</version>
<exclusions>
<exclusion>
<artifactId>servlet-api</artifactId>
<groupId>javax.servlet</groupId>
</exclusion>
<exclusion>
<artifactId>jetty-util</artifactId>
<groupId>org.mortbay.jetty</groupId>
</exclusion>
<exclusion>
<artifactId>jsp-api</artifactId>
<groupId>javax.servlet.jsp</groupId>
</exclusion>
</exclusions>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-client -->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>2.6.2</version>
<exclusions>
<exclusion>
<artifactId>jetty-util</artifactId>
<groupId>org.mortbay.jetty</groupId>
</exclusion>
</exclusions>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-mapreduce-client-core -->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-core</artifactId>
<version>2.6.2</version>
</dependency>
<dependency>
<groupId>org.mongodb</groupId>
<artifactId>mongo-java-driver</artifactId>
<version>3.2.1</version>
</dependency>
<!-- hadoop dependency -->
<!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-core -->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
<version>1.2.1</version>
<exclusions>
<exclusion>
<artifactId>jetty-util</artifactId>
<groupId>org.mortbay.jetty</groupId>
</exclusion>
</exclusions>
</dependency>
Till now I have searched several blogs. What i do understand from reading those blogs iis that there is some issue with the tachyon jar which is embedded in spark-assembly-1.5.2-hadoop2.6.0.jar.
I tried removing tachyon-0.5.0.jar tachyon-client-0.5.0.jar from shared library of oozie (was present under spark library) but then i started getting error:
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, org.apache.spark.util.Utils$.DEFAULT_DRIVER_MEM_MB()I
java.lang.NoSuchMethodError: org.apache.spark.util.Utils$.DEFAULT_DRIVER_MEM_MB()I
Please help me debug and solve it.
Thanks,
Rohit
Re: Spark and Kafka integration
Posted by Jacek Laskowski <ja...@japila.pl>.
Hi Phadnis,
I found this in
http://spark.apache.org/docs/latest/streaming-kafka-0-10-integration.html:
> This version of the integration is marked as experimental, so the API is potentially subject to change.
Pozdrawiam,
Jacek Laskowski
----
https://medium.com/@jaceklaskowski/
Mastering Apache Spark 2.0 https://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski
On Thu, Jan 12, 2017 at 2:57 PM, Phadnis, Varun
<ph...@sky.optymyze.com> wrote:
> Hello,
>
>
>
> We are using Spark 2.0 with Kafka 0.10.
>
>
>
> As I understand, much of the API packaged in the following dependency we are
> targeting is marked as “@Experimental”
>
>
>
> <dependency>
>
> <groupId>org.apache.spark</groupId>
>
> <artifactId>spark-streaming-kafka-0-10_2.11</artifactId>
>
> <version>2.0.0</version>
>
> </dependency>
>
>
>
> What are implications of this being marked as experimental? Are they stable
> enough for production?
>
>
>
> Thanks,
>
> Varun
>
>
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
RE: Spark and Kafka integration
Posted by "Phadnis, Varun" <ph...@sky.optymyze.com>.
Cool! Thanks for your inputs Jacek and Mark!
From: Mark Hamstra [mailto:mark@clearstorydata.com]
Sent: 13 January 2017 12:59
To: Phadnis, Varun <ph...@sky.optymyze.com>
Cc: user@spark.apache.org
Subject: Re: Spark and Kafka integration
See "API compatibility" in http://spark.apache.org/versioning-policy.html
While code that is annotated as Experimental is still a good faith effort to provide a stable and useful API, the fact is that we're not yet confident enough that we've got the public API in exactly the form that we want to commit to maintaining until at least the next major release. That means that the API may change in the next minor/feature-level release (but it shouldn't in a patch/bugfix-level release), which would require that your source code be rewritten to use the new API. In the most extreme case, we may decide that the experimental code didn't work out the way we wanted, so it could be withdrawn entirely. Complete withdrawal of the Kafka code is unlikely, but it may well change in incompatible way with future releases even before Spark 3.0.0.
On Thu, Jan 12, 2017 at 5:57 AM, Phadnis, Varun <ph...@sky.optymyze.com>> wrote:
Hello,
We are using Spark 2.0 with Kafka 0.10.
As I understand, much of the API packaged in the following dependency we are targeting is marked as “@Experimental”
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming-kafka-0-10_2.11</artifactId>
<version>2.0.0</version>
</dependency>
What are implications of this being marked as experimental? Are they stable enough for production?
Thanks,
Varun
Re: Spark and Kafka integration
Posted by Mark Hamstra <ma...@clearstorydata.com>.
See "API compatibility" in http://spark.apache.org/versioning-policy.html
While code that is annotated as Experimental is still a good faith effort
to provide a stable and useful API, the fact is that we're not yet
confident enough that we've got the public API in exactly the form that we
want to commit to maintaining until at least the next major release. That
means that the API may change in the next minor/feature-level release (but
it shouldn't in a patch/bugfix-level release), which would require that
your source code be rewritten to use the new API. In the most extreme
case, we may decide that the experimental code didn't work out the way we
wanted, so it could be withdrawn entirely. Complete withdrawal of the
Kafka code is unlikely, but it may well change in incompatible way with
future releases even before Spark 3.0.0.
On Thu, Jan 12, 2017 at 5:57 AM, Phadnis, Varun <ph...@sky.optymyze.com>
wrote:
> Hello,
>
>
>
> We are using Spark 2.0 with Kafka 0.10.
>
>
>
> As I understand, much of the API packaged in the following dependency we
> are targeting is marked as “@Experimental”
>
>
>
> <dependency>
>
> <groupId>org.apache.spark</groupId>
>
> <artifactId>spark-streaming-kafka-0-10_2.11</artifactId>
>
> <version>2.0.0</version>
>
> </dependency>
>
>
>
> What are implications of this being marked as experimental? Are they
> stable enough for production?
>
>
>
> Thanks,
>
> Varun
>
>
>