You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@livy.apache.org by Junaid Nasir <jn...@an10.io> on 2017/10/09 12:11:27 UTC
Spark cassandra connector with livy
Hi,
How can I include Cassandra-connect-package to Livy spark session?
I use the following spark-submit command/spark-submit --master
spark://10.128.1.1:7077 --packages
datastax:spark-cassandra-connector:2.0.1-s_2.11 --conf
spark.cassandra.connection.host="10.128.1.1,10.128.1.2,10.128.1.3"
how can i translate it to livy's session request or add this to livy conf file?
Also how to provide list of string and Map of key=val as mentioned in livy's
documentation
Regards,Junaid Nasir
Re: Spark cassandra connector with livy
Posted by Saisai Shao <sa...@gmail.com>.
Please set "spark.jars.packages" to the package you wanted in batch POST
protocol "conf" field.
Thanks
Jerry
On Tue, Oct 10, 2017 at 3:05 AM, Junaid Nasir <jn...@an10.io> wrote:
> more info regarding the problem.
> when i pass {"kind": "pyspark","jars":["datastax:
> spark-cassandra-connector:2.0.1-s_2.11"]} via post to /sessions
> session status changes to dead. livy logs show this
>
> 17/10/09 18:45:15 WARN ContextLauncher: Child process exited with code 1.
> 17/10/09 18:45:15 ERROR RSCClient: Failed to connect to context.
> java.io.IOException: Child process exited with code 1.
> at com.cloudera.livy.rsc.ContextLauncher$ChildProcess$1.run(ContextLauncher.java:383)
> at com.cloudera.livy.rsc.ContextLauncher$ChildProcess$2.run(ContextLauncher.java:432)
> at java.lang.Thread.run(Thread.java:748)
> 17/10/09 18:45:15 INFO RSCClient: Failing pending job 115078db-7ec3-4a5c-b74e-e24d6d811413 due to shutdown.
> 17/10/09 18:45:15 WARN InteractiveSession: (Fail to get rsc uri,java.util.concurrent.ExecutionException: java.io.IOException: Child process exited with code 1.)
> 17/10/09 18:45:15 INFO InteractiveSession: Stopping InteractiveSession 1...
> 17/10/09 18:45:15 INFO InteractiveSession: Stopped InteractiveSession 1.
>
> [image: Mixmax]
> <https://mixmax.com/r/592fe3b429b79365389d2354> Not using Mixmax yet?
> <https://mixmax.com/r/592fe3b429b79365389d2354>
>
>
>
> On Mon, Oct 9, 2017 5:11 PM, Junaid Nasir jnasir@an10.io wrote:
>
>> Hi,
>>
>> How can I include Cassandra-connect-package
>> <https://github.com/datastax/spark-cassandra-connector/> to Livy spark
>> session?
>>
>> I use the following spark-submit command
>>
>> /spark-submit --master spark://10.128.1.1:7077 --packages
>> datastax:spark-cassandra-connector:2.0.1-s_2.11 --conf
>> spark.cassandra.connection.host="10.128.1.1,10.128.1.2,10.128.1.3"
>>
>>
>> how can i translate it to livy's session request or add this to livy conf
>> file?
>>
>> Also how to provide list of string and Map of key=val as mentioned in livy's
>> documentation
>> <https://livy.incubator.apache.org/docs/latest/rest-api.html#session>
>>
>> Regards,
>> Junaid Nasir
>>
>
Re: Spark cassandra connector with livy
Posted by Junaid Nasir <jn...@an10.io>.
more info regarding the problem.when i pass {"kind":
"pyspark","jars":["datastax:spark-cassandra-connector:2.0.1-s_2.11"]} via post
to /sessionssession status changes to dead. livy logs show this
17/10/09 18:45:15 WARN ContextLauncher: Child process exited with code 1.17/10/09 18:45:15 ERROR RSCClient: Failed to connect to context.java.io.IOException: Child process exited with code 1. at com.cloudera.livy.rsc.ContextLauncher$ChildProcess$1.run(ContextLauncher.java:383) at com.cloudera.livy.rsc.ContextLauncher$ChildProcess$2.run(ContextLauncher.java:432) at java.lang.Thread.run(Thread.java:748)17/10/09 18:45:15 INFO RSCClient: Failing pending job 115078db-7ec3-4a5c-b74e-e24d6d811413 due to shutdown.17/10/09 18:45:15 WARN InteractiveSession: (Fail to get rsc uri,java.util.concurrent.ExecutionException: java.io.IOException: Child process exited with code 1.)17/10/09 18:45:15 INFO InteractiveSession: Stopping InteractiveSession 1...17/10/09 18:45:15 INFO InteractiveSession: Stopped InteractiveSession 1.
Not using Mixmax yet?
On Mon, Oct 9, 2017 5:11 PM, Junaid Nasir jnasir@an10.io wrote:
Hi,
How can I include Cassandra-connect-package to Livy spark session?
I use the following spark-submit command/spark-submit --master
spark://10.128.1.1:7077 --packages
datastax:spark-cassandra-connector:2.0.1-s_2.11 --conf
spark.cassandra.connection.host="10.128.1.1,10.128.1.2,10.128.1.3"
how can i translate it to livy's session request or add this to livy conf file?
Also how to provide list of string and Map of key=val as mentioned in livy's
documentation
Regards,Junaid Nasir