You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Geektimus <al...@gmail.com> on 2016/10/21 14:31:07 UTC
Ignite + Spark Streaming on Amazon EMR Exception
Hello,I'm working with this set of technologies (Spark 1.6.0 and Ignite 1.6.0
) and Im having some issues, the most troublesome is this exception:
6/10/20 22:09:15 ERROR DAGSchedulerEventProcessLoop:
DAGSchedulerEventProcessLoop failed; shutting down
SparkContextjava.lang.NumberFormatException: For input string: "1%1" at
java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Integer.parseInt(Integer.java:492) at
java.lang.Integer.parseInt(Integer.java:527) at
scala.collection.immutable.StringLike$class.toInt(StringLike.scala:229) at
scala.collection.immutable.StringOps.toInt(StringOps.scala:31) at
org.apache.spark.util.Utils$.parseHostPort(Utils.scala:877) at
org.apache.spark.scheduler.cluster.YarnScheduler.getRackForHost(YarnScheduler.scala:37)
at
org.apache.spark.scheduler.TaskSetManager$$anonfun$org$apache$spark$scheduler$TaskSetManager$$addPendingTask$1.apply(TaskSetManager.scala:208)
at
org.apache.spark.scheduler.TaskSetManager$$anonfun$org$apache$spark$scheduler$TaskSetManager$$addPendingTask$1.apply(TaskSetManager.scala:187)
at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) at
org.apache.spark.scheduler.TaskSetManager.org$apache$spark$scheduler$TaskSetManager$$addPendingTask(TaskSetManager.scala:187)
at
org.apache.spark.scheduler.TaskSetManager$$anonfun$1.apply$mcVI$sp(TaskSetManager.scala:166)
at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141) at
org.apache.spark.scheduler.TaskSetManager.(TaskSetManager.scala:165) at
org.apache.spark.scheduler.TaskSchedulerImpl.createTaskSetManager(TaskSchedulerImpl.scala:200)
at
org.apache.spark.scheduler.TaskSchedulerImpl.submitTasks(TaskSchedulerImpl.scala:164)
at
org.apache.spark.scheduler.DAGScheduler.submitMissingTasks(DAGScheduler.scala:1052)
at
org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitStage(DAGScheduler.scala:921)
at
org.apache.spark.scheduler.DAGScheduler.handleJobSubmitted(DAGScheduler.scala:861)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1607)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1599)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1588)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
I'm using the XML config to use the DiscoverySPI from S3
<property name="discoverySpi"><bean
class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi"><property
name="ipFinder"><bean
class="org.apache.ignite.spi.discovery.tcp.ipfinder.s3.TcpDiscoveryS3IpFinder"><property
name="awsCredentials" ref="aws.creds"/><property
name="bucketName"
value="embedded-ignite-discovery"/></bean></property>
But that doesn't work, So I changed the scala code to do the same inside the
code:
val cfg = new IgniteConfiguration with Serializable//val clientCfg = new
ClientConfiguration()val ipFinder = new CustomTcpDiscoveryS3IpFinderval
discoverySPI = new TcpDiscoverySpival accessKey =
configHelper.envOrElseConfig("aws.security.credentials.access_key")val
secretKey =
configHelper.envOrElseConfig("aws.security.credentials.secret_key")val
awsCredentials = new BasicAWSCredentials(accessKey,
secretKey)ipFinder.setAwsCredentials(awsCredentials)ipFinder.setBucketName(configHelper.envOrElseConfig("ignite.aws.s3.bucket_name"))discoverySPI.setIpFinder(ipFinder)
But that fails too, it throws the same exception with the "1%1",When I check
the bucket I can see a file called "0:0:0:0:0:0:0:1:1%1" and I think that is
the IPv6 address of the loopback interface and in some way spark or ignite
is failing to handle that address.In the attempt log I can see this:
>>> +----------------------------------------------------------------------+>>>
>>> Ignite ver.
>>> 1.6.0#20160518-sha1:0b22c45bb9b97692208fd0705ddf8045ff34a031>>>
>>> +----------------------------------------------------------------------+>>>
>>> OS name: Linux 4.1.17-22.30.amzn1.x86_64 amd64>>> CPU(s): 4>>> Heap:
>>> 11.0GB>>> VM name: 6268@ip-172-31-41-59>>> Local node
>>> [ID=B284DCEF-BB88-4AF1-BAA2-C7F337A8E579, order=2, clientMode=true]>>>
>>> Local node addresses: [ip-172-31-41-59.ec2.internal/*0:0:0:0:0:0:0:1%1*,
>>> /127.0.0.1, /172.31.41.59]>>> Local ports: TCP:11211 TCP:47100 TCP:48100
Notice the "*0:0:0:0:0:0:0:1%1*"I also tried to do a:
cfg.setDiscoverySpi(discoverySPI)cfg.setLocalHost("127.0.0.1")cfg.setClientMode(true)cfg.setPeerClassLoadingEnabled(false)
But in that case it hangs in:
16/10/20 22:49:27 INFO IgniteKernal: Security status [authentication=off,
tls/ssl=off]16/10/20 22:49:27 INFO GridTcpRestProtocol: Command protocol
successfully started [name=TCP binary, host=0.0.0.0/0.0.0.0,
port=11211]16/10/20 22:49:28 WARN CustomTcpDiscoveryS3IpFinder: Amazon
client configuration is not set (will use default).16/10/20 22:49:28 WARN
TcpDiscoverySpi: Failed to connect to any address from IP finder (will retry
to join topology every 2 secs): [/127.0.0.1:47500]16/10/20 22:51:36 INFO
ContextCleaner: Cleaned accumulator 1
Any help is really appreciated.Thanks,P.S Sorry if the post appears twice
but I sent the post to the mailing list some days ago and it doesn't show in
here and I really need your help.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-Spark-Streaming-on-Amazon-EMR-Exception-tp8410.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite + Spark Streaming on Amazon EMR Exception
Posted by vkulichenko <va...@gmail.com>.
Well, the exception is actually coming from Spark, not Ignite, so the issue
is most likely there. System provided I suggested should fix the issue in
any case though. Not sure how this should be done in EMR.
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-Spark-Streaming-on-Amazon-EMR-Exception-tp8410p8569.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite + Spark Streaming on Amazon EMR Exception
Posted by Geektimus <al...@gmail.com>.
Hi Val,Thanks for your suggestion, the thing at this point is that this is
running inside EMR (Elastic Map Reduce in Amazon) in embedded mode and I
don't have control over the system properties. I tried to use them inside a
bootstrap action for the cluster and I found that it was deprecated for the
kind of instance that I'm using.Also, I tried to use the new json conf API
for EMR with
{ "Classification": "spark-env", "Configurations": [
{ "Classification": "export", "Properties": {
"JAVA_HOME": "/usr/lib/jvm/java-1.8.0", "JAVA_OPTS":
"-Djava.net.preferIPv4Stack=true",
"java.net.preferIPv4Stack": "true" } } ],
"Properties": {} }
*Note:* This is just a fragment.None of them work in the cluster, Im seeing
IPs like "0:0:0:0:0:0:0:1%1" or "0:0:0:0:0:0:0:1%lo". Btw, I found that the
"%1" and "%lo" are the scope IDs for a IPv6.I don't know if the issue is an
incompatibility between Ignite and Spark Streaming or just an error with
Spark handling IPv6'sFYI, In the Spark code I found this marvelous comment
<https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/util/Utils.scala#L948>
in the method that is failing! ¬¬
// This is potentially broken - when dealing with ipv6 addresses for
example, sigh ...// but then hadoop does not support ipv6 right now.// For
now, we assume that if port exists, then it is valid - not check if it is an
int > 0
I'm still trying to find a solution.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-Spark-Streaming-on-Amazon-EMR-Exception-tp8410p8558.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite + Spark Streaming on Amazon EMR Exception
Posted by vkulichenko <va...@gmail.com>.
Hi,
You can try to set -Djava.net.preferIPv4Stack=true system property to bind
to IPv4 addresses only. Probably this will help.
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-Spark-Streaming-on-Amazon-EMR-Exception-tp8410p8423.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.