You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@zeppelin.apache.org by "arvindsherigar@northwesternmutual.com" <ar...@northwesternmutual.com> on 2019/04/29 14:00:00 UTC
Re: [EXTERNAL] Re: Issue working with R/SparkR
DataStax distribution version 6.7
From: Jeff Zhang <zj...@gmail.com>
Reply-To: "users@zeppelin.apache.org" <us...@zeppelin.apache.org>
Date: Sunday, April 28, 2019 at 11:01 PM
To: users <us...@zeppelin.apache.org>
Subject: [EXTERNAL] Re: Issue working with R/SparkR
Which spark version do you use ?
arvindsherigar@northwesternmutual.com<ma...@northwesternmutual.com> <ar...@northwesternmutual.com>> 于2019年4月29日周一 上午10:23写道：
https://zeppelin.apache.org/docs/0.8.1/interpreter/r.html<https://urldefense.proofpoint.com/v2/url?u=https-3A__zeppelin.apache.org_docs_0.8.1_interpreter_r.html&d=DwMFaQ&c=oBIBfyrVf5fVcKOfTTdY_d7WoqYVCfQW6jezwvkj5jg&r=2ozYxxrO5uy6i7CP-nMc0wdBTNfkSGa4age9CqpA0b3LBZZ_7ZITfmsqWijkzea1&m=R9XdjivdKYjYFmH_RUlm1VSWtiBcfNWHMgP6hbxtMJI&s=-p1MV9gJn5XAIepg-nkje92IM9e_kPaXsv8m7AYXSnA&e=>
I am trying out a basic command as shown in the link above and I get the following error. Any help will be appreciated.
%spark.r
2+2
java.lang.ClassCastException: scala.Tuple2 cannot be cast to java.lang.Integer at scala.runtime.BoxesRunTime.unboxToInt(BoxesRunTime.java:101) at org.apache.spark.SparkRBackend$.init(SparkRBackend.scala:42) at org.apache.spark.SparkRBackend.init(SparkRBackend.scala) at org.apache.zeppelin.spark.SparkRInterpreter.open(SparkRInterpreter.java:88) at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69) at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:616) at org.apache.zeppelin.scheduler.Job.run(Job.java:188) at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:140) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
Thanks
Arvind
This e-mail and any attachments may contain confidential information of Northwestern Mutual. If you are not the intended recipient of this message, be aware that any disclosure, copying, distribution or use of this e-mail and any attachments is prohibited. If you have received this e-mail in error, please notify Northwestern Mutual immediately by returning it to the sender and delete all copies from your system. Please be advised that communications with {SECURE MESSAGE} in the subject line have been sent using a secure messaging system. Communications that do not have this tag may not be secure and could be observed by a third party. Thank you for your cooperation.
--
Best Regards
Jeff Zhang
This e-mail and any attachments may contain confidential information of Northwestern Mutual. If you are not the intended recipient of this message, be aware that any disclosure, copying, distribution or use of this e-mail and any attachments is prohibited. If you have received this e-mail in error, please notify Northwestern Mutual immediately by returning it to the sender and delete all copies from your system. Please be advised that communications with {SECURE MESSAGE} in the subject line have been sent using a secure messaging system. Communications that do not have this tag may not be secure and could be observed by a third party. Thank you for your cooperation.
Re: [EXTERNAL] Re: Issue working with R/SparkR
Posted by "arvindsherigar@northwesternmutual.com" <ar...@northwesternmutual.com>.
Had to install certificate for archive.apache.org using the following steps. This is for others who may run into similar issues
https://confluence.atlassian.com/kb/how-to-import-a-public-ssl-certificate-into-a-jvm-867025849.html
From: "arvindsherigar@northwesternmutual.com" <ar...@northwesternmutual.com>
Reply-To: "users@zeppelin.apache.org" <us...@zeppelin.apache.org>
Date: Tuesday, April 30, 2019 at 11:01 AM
To: Jeff Zhang <zj...@gmail.com>
Cc: "users@zeppelin.apache.org" <us...@zeppelin.apache.org>
Subject: Re: [EXTERNAL] Re: Issue working with R/SparkR
Ok, ran into another problem while building the code. Let me know whether this is related to jdk 1.8 I am using instead of jdk 1.7 as per the “building from source page” or something else.
https://zeppelin.apache.org/docs/latest/setup/basics/how_to_build.html<https://urldefense.proofpoint.com/v2/url?u=https-3A__zeppelin.apache.org_docs_latest_setup_basics_how-5Fto-5Fbuild.html&d=DwMGaQ&c=oBIBfyrVf5fVcKOfTTdY_d7WoqYVCfQW6jezwvkj5jg&r=2ozYxxrO5uy6i7CP-nMc0wdBTNfkSGa4age9CqpA0b3LBZZ_7ZITfmsqWijkzea1&m=gLej1ZjJ4W_ha21ri5zuo9WehM-GB9m_NgNW96KoYQM&s=6Mw6a614oitRGJtr7LQAhrrFmNf776RktsXemLdfc60&e=>
[cid:image001.png@01D4FF43.DF190E80]
[cid:image002.png@01D4FF43.DF190E80]
From: Jeff Zhang <zj...@gmail.com>
Date: Monday, April 29, 2019 at 10:50 PM
To: "SHERIGAR, ARVIND" <ar...@northwesternmutual.com>
Cc: "users@zeppelin.apache.org" <us...@zeppelin.apache.org>
Subject: Re: [EXTERNAL] Re: Issue working with R/SparkR
mvn clean package -pl spark/interpreter -am -Pscala-2.11
This e-mail and any attachments may contain confidential information of Northwestern Mutual. If you are not the intended recipient of this message, be aware that any disclosure, copying, distribution or use of this e-mail and any attachments is prohibited. If you have received this e-mail in error, please notify Northwestern Mutual immediately by returning it to the sender and delete all copies from your system. Please be advised that communications with {SECURE MESSAGE} in the subject line have been sent using a secure messaging system. Communications that do not have this tag may not be secure and could be observed by a third party. Thank you for your cooperation.
This e-mail and any attachments may contain confidential information of Northwestern Mutual. If you are not the intended recipient of this message, be aware that any disclosure, copying, distribution or use of this e-mail and any attachments is prohibited. If you have received this e-mail in error, please notify Northwestern Mutual immediately by returning it to the sender and delete all copies from your system. Please be advised that communications with {SECURE MESSAGE} in the subject line have been sent using a secure messaging system. Communications that do not have this tag may not be secure and could be observed by a third party. Thank you for your cooperation.
Re: [EXTERNAL] Re: Issue working with R/SparkR
Posted by "arvindsherigar@northwesternmutual.com" <ar...@northwesternmutual.com>.
Ok, ran into another problem while building the code. Let me know whether this is related to jdk 1.8 I am using instead of jdk 1.7 as per the “building from source page” or something else.
https://zeppelin.apache.org/docs/latest/setup/basics/how_to_build.html
[cid:image001.png@01D4FF43.DF190E80]
[cid:image002.png@01D4FF43.DF190E80]
From: Jeff Zhang <zj...@gmail.com>
Date: Monday, April 29, 2019 at 10:50 PM
To: "SHERIGAR, ARVIND" <ar...@northwesternmutual.com>
Cc: "users@zeppelin.apache.org" <us...@zeppelin.apache.org>
Subject: Re: [EXTERNAL] Re: Issue working with R/SparkR
mvn clean package -pl spark/interpreter -am -Pscala-2.11
This e-mail and any attachments may contain confidential information of Northwestern Mutual. If you are not the intended recipient of this message, be aware that any disclosure, copying, distribution or use of this e-mail and any attachments is prohibited. If you have received this e-mail in error, please notify Northwestern Mutual immediately by returning it to the sender and delete all copies from your system. Please be advised that communications with {SECURE MESSAGE} in the subject line have been sent using a secure messaging system. Communications that do not have this tag may not be secure and could be observed by a third party. Thank you for your cooperation.
Re: [EXTERNAL] Re: Issue working with R/SparkR
Posted by Jeff Zhang <zj...@gmail.com>.
The problem is that zeppelin's beam interpreter doesn't support scala 2.11,
we need to upgrade it.
For a workaround, you can just build spark interpreter via the following
command:
mvn clean package -pl spark/interpreter -am -Pscala-2.11
arvindsherigar@northwesternmutual.com <ar...@northwesternmutual.com>
于2019年4月30日周二 上午11:20写道：
> I am doing a local fix for this problem, but I am unable to compile the
> code. I checked maven central and this jar does not exist. Am I missing
> something here? I could change the pom in the project, but wanted to make
> sure I don’t break something else.
>
>
>
> [ERROR] Failed to execute goal on project zeppelin-beam: Could not resolve
> dependencies for project
> org.apache.zeppelin:zeppelin-beam:jar:0.9.0-SNAPSHOT: *Failure to find
> org.apache.beam:beam-runners-flink_2.11:jar:2.0.0* in
> https://repo.maven.apache.org/maven2 was cached in the local repository,
> resolution will not be reattempted until the update interval of central has
> elapsed or updates are forced
>
>
>
> *https://mvnrepository.com/artifact/org.apache.beam/beam-runners-flink
> <https://mvnrepository.com/artifact/org.apache.beam/beam-runners-flink>*
>
>
>
> Thanks
>
> Arvind
>
>
>
>
>
> *From: *Jeff Zhang <zj...@gmail.com>
> *Reply-To: *"users@zeppelin.apache.org" <us...@zeppelin.apache.org>
> *Date: *Monday, April 29, 2019 at 9:54 AM
> *To: *users <us...@zeppelin.apache.org>
> *Subject: *Re: [EXTERNAL] Re: Issue working with R/SparkR
>
>
>
>
>
> That's right
>
>
>
> arvindsherigar@northwesternmutual.com <
> arvindsherigar@northwesternmutual.com> 于2019年4月29日周一 下午10:33写道：
>
> Ok, I talked to the DSE architect and he told me that they had to do a
> security patch (spark version example -> 2.2.2.5), which broke this piece
> of zeppelin code - *bolded*.
>
>
> https://github.com/apache/zeppelin/blob/735064fdc57ae958fabae85b399bb5af3cb79144/spark/interpreter/src/main/scala/org/apache/spark/SparkRBackend.scala#L34-L44
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_zeppelin_blob_735064fdc57ae958fabae85b399bb5af3cb79144_spark_interpreter_src_main_scala_org_apache_spark_SparkRBackend.scala-23L34-2DL44&d=DwMFaQ&c=oBIBfyrVf5fVcKOfTTdY_d7WoqYVCfQW6jezwvkj5jg&r=2ozYxxrO5uy6i7CP-nMc0wdBTNfkSGa4age9CqpA0b3LBZZ_7ZITfmsqWijkzea1&m=p8jIjG5K5TauQWxRvMhReOCieAiUTuZCr22KAO-YRW0&s=B1shgXQUvveI8LYid8Sji-gKZDnypoDJ_AM9B-L7pow&e=>
>
>
>
> def init(version: SparkVersion) : Unit = {
>
> val rBackendClass = classOf[RBackend]
>
> *if** (version.isSecretSocketSupported)* {
>
> val result = rBackendClass.getMethod("init").invoke(backend).
> asInstanceOf[Tuple2[Int, Object]]
>
> portNumber = result._1
>
> val rAuthHelper = result._2
>
> secret = rAuthHelper.getClass.getMethod("secret"
> ).invoke(rAuthHelper).asInstanceOf[String]
>
> } else {
>
> * portNumber **=** rBackendClass.getMethod(**"init"*
> *).invoke(backend).**asInstanceOf**[**Int**]*
>
> }
>
> }
>
>
>
>
>
> *From: *"arvindsherigar@northwesternmutual.com" <
> arvindsherigar@northwesternmutual.com>
> *Reply-To: *"users@zeppelin.apache.org" <us...@zeppelin.apache.org>
> *Date: *Monday, April 29, 2019 at 9:01 AM
> *To: *"users@zeppelin.apache.org" <us...@zeppelin.apache.org>
> *Subject: *Re: [EXTERNAL] Re: Issue working with R/SparkR
>
>
>
> DataStax distribution version 6.7
>
>
>
> *From: *Jeff Zhang <zj...@gmail.com>
> *Reply-To: *"users@zeppelin.apache.org" <us...@zeppelin.apache.org>
> *Date: *Sunday, April 28, 2019 at 11:01 PM
> *To: *users <us...@zeppelin.apache.org>
> *Subject: *[EXTERNAL] Re: Issue working with R/SparkR
>
>
>
> Which spark version do you use ?
>
>
>
> arvindsherigar@northwesternmutual.com <
> arvindsherigar@northwesternmutual.com> 于2019年4月29日周一 上午10:23写道：
>
> https://zeppelin.apache.org/docs/0.8.1/interpreter/r.html
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__zeppelin.apache.org_docs_0.8.1_interpreter_r.html&d=DwMFaQ&c=oBIBfyrVf5fVcKOfTTdY_d7WoqYVCfQW6jezwvkj5jg&r=2ozYxxrO5uy6i7CP-nMc0wdBTNfkSGa4age9CqpA0b3LBZZ_7ZITfmsqWijkzea1&m=R9XdjivdKYjYFmH_RUlm1VSWtiBcfNWHMgP6hbxtMJI&s=-p1MV9gJn5XAIepg-nkje92IM9e_kPaXsv8m7AYXSnA&e=>
>
>
>
> I am trying out a basic command as shown in the link above and I get the
> following error. Any help will be appreciated.
>
>
>
> *%spark.r *
>
> *2+2*
>
>
>
> *java.lang.ClassCastException: scala.Tuple2 cannot be cast to
> java.lang.Integer* at
> scala.runtime.BoxesRunTime.unboxToInt(BoxesRunTime.java:101) at
> org.apache.spark.SparkRBackend$.init(SparkRBackend.scala:42) at
> org.apache.spark.SparkRBackend.init(SparkRBackend.scala) at
> org.apache.zeppelin.spark.SparkRInterpreter.open(SparkRInterpreter.java:88)
> at
> org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:616)
> at org.apache.zeppelin.scheduler.Job.run(Job.java:188) at
> org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:140)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>
>
>
> Thanks
>
> Arvind
>
>
>
> This e-mail and any attachments may contain confidential information of
> Northwestern Mutual. If you are not the intended recipient of this message,
> be aware that any disclosure, copying, distribution or use of this e-mail
> and any attachments is prohibited. If you have received this e-mail in
> error, please notify Northwestern Mutual immediately by returning it to the
> sender and delete all copies from your system. Please be advised that
> communications with {SECURE MESSAGE} in the subject line have been sent
> using a secure messaging system. Communications that do not have this tag
> may not be secure and could be observed by a third party. Thank you for
> your cooperation.
>
>
>
>
> --
>
> Best Regards
>
> Jeff Zhang
>
>
>
> This e-mail and any attachments may contain confidential information of
> Northwestern Mutual. If you are not the intended recipient of this message,
> be aware that any disclosure, copying, distribution or use of this e-mail
> and any attachments is prohibited. If you have received this e-mail in
> error, please notify Northwestern Mutual immediately by returning it to the
> sender and delete all copies from your system. Please be advised that
> communications with {SECURE MESSAGE} in the subject line have been sent
> using a secure messaging system. Communications that do not have this tag
> may not be secure and could be observed by a third party. Thank you for
> your cooperation.
>
>
>
> This e-mail and any attachments may contain confidential information of
> Northwestern Mutual. If you are not the intended recipient of this message,
> be aware that any disclosure, copying, distribution or use of this e-mail
> and any attachments is prohibited. If you have received this e-mail in
> error, please notify Northwestern Mutual immediately by returning it to the
> sender and delete all copies from your system. Please be advised that
> communications with {SECURE MESSAGE} in the subject line have been sent
> using a secure messaging system. Communications that do not have this tag
> may not be secure and could be observed by a third party. Thank you for
> your cooperation.
>
>
>
>
> --
>
> Best Regards
>
> Jeff Zhang
>
>
> This e-mail and any attachments may contain confidential information of
> Northwestern Mutual. If you are not the intended recipient of this message,
> be aware that any disclosure, copying, distribution or use of this e-mail
> and any attachments is prohibited. If you have received this e-mail in
> error, please notify Northwestern Mutual immediately by returning it to the
> sender and delete all copies from your system. Please be advised that
> communications with {SECURE MESSAGE} in the subject line have been sent
> using a secure messaging system. Communications that do not have this tag
> may not be secure and could be observed by a third party. Thank you for
> your cooperation.
>
>
>
--
Best Regards
Jeff Zhang
Re: [EXTERNAL] Re: Issue working with R/SparkR
Posted by "arvindsherigar@northwesternmutual.com" <ar...@northwesternmutual.com>.
I am doing a local fix for this problem, but I am unable to compile the code. I checked maven central and this jar does not exist. Am I missing something here? I could change the pom in the project, but wanted to make sure I don’t break something else.
[ERROR] Failed to execute goal on project zeppelin-beam: Could not resolve dependencies for project org.apache.zeppelin:zeppelin-beam:jar:0.9.0-SNAPSHOT: Failure to find org.apache.beam:beam-runners-flink_2.11:jar:2.0.0 in https://repo.maven.apache.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced
https://mvnrepository.com/artifact/org.apache.beam/beam-runners-flink
Thanks
Arvind
From: Jeff Zhang <zj...@gmail.com>
Reply-To: "users@zeppelin.apache.org" <us...@zeppelin.apache.org>
Date: Monday, April 29, 2019 at 9:54 AM
To: users <us...@zeppelin.apache.org>
Subject: Re: [EXTERNAL] Re: Issue working with R/SparkR
That's right
arvindsherigar@northwesternmutual.com<ma...@northwesternmutual.com> <ar...@northwesternmutual.com>> 于2019年4月29日周一 下午10:33写道：
Ok, I talked to the DSE architect and he told me that they had to do a security patch (spark version example -> 2.2.2.5), which broke this piece of zeppelin code - bolded.
https://github.com/apache/zeppelin/blob/735064fdc57ae958fabae85b399bb5af3cb79144/spark/interpreter/src/main/scala/org/apache/spark/SparkRBackend.scala#L34-L44<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_zeppelin_blob_735064fdc57ae958fabae85b399bb5af3cb79144_spark_interpreter_src_main_scala_org_apache_spark_SparkRBackend.scala-23L34-2DL44&d=DwMFaQ&c=oBIBfyrVf5fVcKOfTTdY_d7WoqYVCfQW6jezwvkj5jg&r=2ozYxxrO5uy6i7CP-nMc0wdBTNfkSGa4age9CqpA0b3LBZZ_7ZITfmsqWijkzea1&m=p8jIjG5K5TauQWxRvMhReOCieAiUTuZCr22KAO-YRW0&s=B1shgXQUvveI8LYid8Sji-gKZDnypoDJ_AM9B-L7pow&e=>
def init(version: SparkVersion) : Unit = {
val rBackendClass = classOf[RBackend]
if (version.isSecretSocketSupported) {
val result = rBackendClass.getMethod("init").invoke(backend).asInstanceOf[Tuple2[Int, Object]]
portNumber = result._1
val rAuthHelper = result._2
secret = rAuthHelper.getClass.getMethod("secret").invoke(rAuthHelper).asInstanceOf[String]
} else {
portNumber = rBackendClass.getMethod("init").invoke(backend).asInstanceOf[Int]
}
}
From: "arvindsherigar@northwesternmutual.com<ma...@northwesternmutual.com>" <ar...@northwesternmutual.com>>
Reply-To: "users@zeppelin.apache.org<ma...@zeppelin.apache.org>" <us...@zeppelin.apache.org>>
Date: Monday, April 29, 2019 at 9:01 AM
To: "users@zeppelin.apache.org<ma...@zeppelin.apache.org>" <us...@zeppelin.apache.org>>
Subject: Re: [EXTERNAL] Re: Issue working with R/SparkR
DataStax distribution version 6.7
From: Jeff Zhang <zj...@gmail.com>>
Reply-To: "users@zeppelin.apache.org<ma...@zeppelin.apache.org>" <us...@zeppelin.apache.org>>
Date: Sunday, April 28, 2019 at 11:01 PM
To: users <us...@zeppelin.apache.org>>
Subject: [EXTERNAL] Re: Issue working with R/SparkR
Which spark version do you use ?
arvindsherigar@northwesternmutual.com<ma...@northwesternmutual.com> <ar...@northwesternmutual.com>> 于2019年4月29日周一 上午10:23写道：
https://zeppelin.apache.org/docs/0.8.1/interpreter/r.html<https://urldefense.proofpoint.com/v2/url?u=https-3A__zeppelin.apache.org_docs_0.8.1_interpreter_r.html&d=DwMFaQ&c=oBIBfyrVf5fVcKOfTTdY_d7WoqYVCfQW6jezwvkj5jg&r=2ozYxxrO5uy6i7CP-nMc0wdBTNfkSGa4age9CqpA0b3LBZZ_7ZITfmsqWijkzea1&m=R9XdjivdKYjYFmH_RUlm1VSWtiBcfNWHMgP6hbxtMJI&s=-p1MV9gJn5XAIepg-nkje92IM9e_kPaXsv8m7AYXSnA&e=>
I am trying out a basic command as shown in the link above and I get the following error. Any help will be appreciated.
%spark.r
2+2
java.lang.ClassCastException: scala.Tuple2 cannot be cast to java.lang.Integer at scala.runtime.BoxesRunTime.unboxToInt(BoxesRunTime.java:101) at org.apache.spark.SparkRBackend$.init(SparkRBackend.scala:42) at org.apache.spark.SparkRBackend.init(SparkRBackend.scala) at org.apache.zeppelin.spark.SparkRInterpreter.open(SparkRInterpreter.java:88) at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69) at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:616) at org.apache.zeppelin.scheduler.Job.run(Job.java:188) at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:140) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
Thanks
Arvind
This e-mail and any attachments may contain confidential information of Northwestern Mutual. If you are not the intended recipient of this message, be aware that any disclosure, copying, distribution or use of this e-mail and any attachments is prohibited. If you have received this e-mail in error, please notify Northwestern Mutual immediately by returning it to the sender and delete all copies from your system. Please be advised that communications with {SECURE MESSAGE} in the subject line have been sent using a secure messaging system. Communications that do not have this tag may not be secure and could be observed by a third party. Thank you for your cooperation.
--
Best Regards
Jeff Zhang
This e-mail and any attachments may contain confidential information of Northwestern Mutual. If you are not the intended recipient of this message, be aware that any disclosure, copying, distribution or use of this e-mail and any attachments is prohibited. If you have received this e-mail in error, please notify Northwestern Mutual immediately by returning it to the sender and delete all copies from your system. Please be advised that communications with {SECURE MESSAGE} in the subject line have been sent using a secure messaging system. Communications that do not have this tag may not be secure and could be observed by a third party. Thank you for your cooperation.
This e-mail and any attachments may contain confidential information of Northwestern Mutual. If you are not the intended recipient of this message, be aware that any disclosure, copying, distribution or use of this e-mail and any attachments is prohibited. If you have received this e-mail in error, please notify Northwestern Mutual immediately by returning it to the sender and delete all copies from your system. Please be advised that communications with {SECURE MESSAGE} in the subject line have been sent using a secure messaging system. Communications that do not have this tag may not be secure and could be observed by a third party. Thank you for your cooperation.
--
Best Regards
Jeff Zhang
This e-mail and any attachments may contain confidential information of Northwestern Mutual. If you are not the intended recipient of this message, be aware that any disclosure, copying, distribution or use of this e-mail and any attachments is prohibited. If you have received this e-mail in error, please notify Northwestern Mutual immediately by returning it to the sender and delete all copies from your system. Please be advised that communications with {SECURE MESSAGE} in the subject line have been sent using a secure messaging system. Communications that do not have this tag may not be secure and could be observed by a third party. Thank you for your cooperation.
Re: [EXTERNAL] Re: Issue working with R/SparkR
Posted by Jeff Zhang <zj...@gmail.com>.
That's right
arvindsherigar@northwesternmutual.com <ar...@northwesternmutual.com>
于2019年4月29日周一 下午10:33写道：
> Ok, I talked to the DSE architect and he told me that they had to do a
> security patch (spark version example -> 2.2.2.5), which broke this piece
> of zeppelin code - *bolded*.
>
>
> https://github.com/apache/zeppelin/blob/735064fdc57ae958fabae85b399bb5af3cb79144/spark/interpreter/src/main/scala/org/apache/spark/SparkRBackend.scala#L34-L44
>
>
>
> def init(version: SparkVersion) : Unit = {
>
> val rBackendClass = classOf[RBackend]
>
> *if** (version.isSecretSocketSupported)* {
>
> val result = rBackendClass.getMethod("init").invoke(backend).
> asInstanceOf[Tuple2[Int, Object]]
>
> portNumber = result._1
>
> val rAuthHelper = result._2
>
> secret = rAuthHelper.getClass.getMethod("secret"
> ).invoke(rAuthHelper).asInstanceOf[String]
>
> } else {
>
> * portNumber **=** rBackendClass.getMethod(**"init"*
> *).invoke(backend).**asInstanceOf**[**Int**]*
>
> }
>
> }
>
>
>
>
>
> *From: *"arvindsherigar@northwesternmutual.com" <
> arvindsherigar@northwesternmutual.com>
> *Reply-To: *"users@zeppelin.apache.org" <us...@zeppelin.apache.org>
> *Date: *Monday, April 29, 2019 at 9:01 AM
> *To: *"users@zeppelin.apache.org" <us...@zeppelin.apache.org>
> *Subject: *Re: [EXTERNAL] Re: Issue working with R/SparkR
>
>
>
> DataStax distribution version 6.7
>
>
>
> *From: *Jeff Zhang <zj...@gmail.com>
> *Reply-To: *"users@zeppelin.apache.org" <us...@zeppelin.apache.org>
> *Date: *Sunday, April 28, 2019 at 11:01 PM
> *To: *users <us...@zeppelin.apache.org>
> *Subject: *[EXTERNAL] Re: Issue working with R/SparkR
>
>
>
> Which spark version do you use ?
>
>
>
> arvindsherigar@northwesternmutual.com <
> arvindsherigar@northwesternmutual.com> 于2019年4月29日周一 上午10:23写道：
>
> https://zeppelin.apache.org/docs/0.8.1/interpreter/r.html
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__zeppelin.apache.org_docs_0.8.1_interpreter_r.html&d=DwMFaQ&c=oBIBfyrVf5fVcKOfTTdY_d7WoqYVCfQW6jezwvkj5jg&r=2ozYxxrO5uy6i7CP-nMc0wdBTNfkSGa4age9CqpA0b3LBZZ_7ZITfmsqWijkzea1&m=R9XdjivdKYjYFmH_RUlm1VSWtiBcfNWHMgP6hbxtMJI&s=-p1MV9gJn5XAIepg-nkje92IM9e_kPaXsv8m7AYXSnA&e=>
>
>
>
> I am trying out a basic command as shown in the link above and I get the
> following error. Any help will be appreciated.
>
>
>
> *%spark.r *
>
> *2+2*
>
>
>
> *java.lang.ClassCastException: scala.Tuple2 cannot be cast to
> java.lang.Integer* at
> scala.runtime.BoxesRunTime.unboxToInt(BoxesRunTime.java:101) at
> org.apache.spark.SparkRBackend$.init(SparkRBackend.scala:42) at
> org.apache.spark.SparkRBackend.init(SparkRBackend.scala) at
> org.apache.zeppelin.spark.SparkRInterpreter.open(SparkRInterpreter.java:88)
> at
> org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:616)
> at org.apache.zeppelin.scheduler.Job.run(Job.java:188) at
> org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:140)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>
>
>
> Thanks
>
> Arvind
>
>
>
> This e-mail and any attachments may contain confidential information of
> Northwestern Mutual. If you are not the intended recipient of this message,
> be aware that any disclosure, copying, distribution or use of this e-mail
> and any attachments is prohibited. If you have received this e-mail in
> error, please notify Northwestern Mutual immediately by returning it to the
> sender and delete all copies from your system. Please be advised that
> communications with {SECURE MESSAGE} in the subject line have been sent
> using a secure messaging system. Communications that do not have this tag
> may not be secure and could be observed by a third party. Thank you for
> your cooperation.
>
>
>
>
>
> --
>
> Best Regards
>
> Jeff Zhang
>
>
>
> This e-mail and any attachments may contain confidential information of
> Northwestern Mutual. If you are not the intended recipient of this message,
> be aware that any disclosure, copying, distribution or use of this e-mail
> and any attachments is prohibited. If you have received this e-mail in
> error, please notify Northwestern Mutual immediately by returning it to the
> sender and delete all copies from your system. Please be advised that
> communications with {SECURE MESSAGE} in the subject line have been sent
> using a secure messaging system. Communications that do not have this tag
> may not be secure and could be observed by a third party. Thank you for
> your cooperation.
>
>
>
> This e-mail and any attachments may contain confidential information of
> Northwestern Mutual. If you are not the intended recipient of this message,
> be aware that any disclosure, copying, distribution or use of this e-mail
> and any attachments is prohibited. If you have received this e-mail in
> error, please notify Northwestern Mutual immediately by returning it to the
> sender and delete all copies from your system. Please be advised that
> communications with {SECURE MESSAGE} in the subject line have been sent
> using a secure messaging system. Communications that do not have this tag
> may not be secure and could be observed by a third party. Thank you for
> your cooperation.
>
>
>
--
Best Regards
Jeff Zhang
Re: [EXTERNAL] Re: Issue working with R/SparkR
Posted by "arvindsherigar@northwesternmutual.com" <ar...@northwesternmutual.com>.
Ok, I talked to the DSE architect and he told me that they had to do a security patch (spark version example -> 2.2.2.5), which broke this piece of zeppelin code - bolded.
https://github.com/apache/zeppelin/blob/735064fdc57ae958fabae85b399bb5af3cb79144/spark/interpreter/src/main/scala/org/apache/spark/SparkRBackend.scala#L34-L44
def init(version: SparkVersion) : Unit = {
val rBackendClass = classOf[RBackend]
if (version.isSecretSocketSupported) {
val result = rBackendClass.getMethod("init").invoke(backend).asInstanceOf[Tuple2[Int, Object]]
portNumber = result._1
val rAuthHelper = result._2
secret = rAuthHelper.getClass.getMethod("secret").invoke(rAuthHelper).asInstanceOf[String]
} else {
portNumber = rBackendClass.getMethod("init").invoke(backend).asInstanceOf[Int]
}
}
From: "arvindsherigar@northwesternmutual.com" <ar...@northwesternmutual.com>
Reply-To: "users@zeppelin.apache.org" <us...@zeppelin.apache.org>
Date: Monday, April 29, 2019 at 9:01 AM
To: "users@zeppelin.apache.org" <us...@zeppelin.apache.org>
Subject: Re: [EXTERNAL] Re: Issue working with R/SparkR
DataStax distribution version 6.7
From: Jeff Zhang <zj...@gmail.com>
Reply-To: "users@zeppelin.apache.org" <us...@zeppelin.apache.org>
Date: Sunday, April 28, 2019 at 11:01 PM
To: users <us...@zeppelin.apache.org>
Subject: [EXTERNAL] Re: Issue working with R/SparkR
Which spark version do you use ?
arvindsherigar@northwesternmutual.com<ma...@northwesternmutual.com> <ar...@northwesternmutual.com>> 于2019年4月29日周一 上午10:23写道：
https://zeppelin.apache.org/docs/0.8.1/interpreter/r.html<https://urldefense.proofpoint.com/v2/url?u=https-3A__zeppelin.apache.org_docs_0.8.1_interpreter_r.html&d=DwMFaQ&c=oBIBfyrVf5fVcKOfTTdY_d7WoqYVCfQW6jezwvkj5jg&r=2ozYxxrO5uy6i7CP-nMc0wdBTNfkSGa4age9CqpA0b3LBZZ_7ZITfmsqWijkzea1&m=R9XdjivdKYjYFmH_RUlm1VSWtiBcfNWHMgP6hbxtMJI&s=-p1MV9gJn5XAIepg-nkje92IM9e_kPaXsv8m7AYXSnA&e=>
I am trying out a basic command as shown in the link above and I get the following error. Any help will be appreciated.
%spark.r
2+2
java.lang.ClassCastException: scala.Tuple2 cannot be cast to java.lang.Integer at scala.runtime.BoxesRunTime.unboxToInt(BoxesRunTime.java:101) at org.apache.spark.SparkRBackend$.init(SparkRBackend.scala:42) at org.apache.spark.SparkRBackend.init(SparkRBackend.scala) at org.apache.zeppelin.spark.SparkRInterpreter.open(SparkRInterpreter.java:88) at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69) at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:616) at org.apache.zeppelin.scheduler.Job.run(Job.java:188) at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:140) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
Thanks
Arvind
This e-mail and any attachments may contain confidential information of Northwestern Mutual. If you are not the intended recipient of this message, be aware that any disclosure, copying, distribution or use of this e-mail and any attachments is prohibited. If you have received this e-mail in error, please notify Northwestern Mutual immediately by returning it to the sender and delete all copies from your system. Please be advised that communications with {SECURE MESSAGE} in the subject line have been sent using a secure messaging system. Communications that do not have this tag may not be secure and could be observed by a third party. Thank you for your cooperation.
--
Best Regards
Jeff Zhang
This e-mail and any attachments may contain confidential information of Northwestern Mutual. If you are not the intended recipient of this message, be aware that any disclosure, copying, distribution or use of this e-mail and any attachments is prohibited. If you have received this e-mail in error, please notify Northwestern Mutual immediately by returning it to the sender and delete all copies from your system. Please be advised that communications with {SECURE MESSAGE} in the subject line have been sent using a secure messaging system. Communications that do not have this tag may not be secure and could be observed by a third party. Thank you for your cooperation.
This e-mail and any attachments may contain confidential information of Northwestern Mutual. If you are not the intended recipient of this message, be aware that any disclosure, copying, distribution or use of this e-mail and any attachments is prohibited. If you have received this e-mail in error, please notify Northwestern Mutual immediately by returning it to the sender and delete all copies from your system. Please be advised that communications with {SECURE MESSAGE} in the subject line have been sent using a secure messaging system. Communications that do not have this tag may not be secure and could be observed by a third party. Thank you for your cooperation.