JAr files into python3

[Joaquin Alzola <Jo...@lebara.com>]

HI List,


I have the following script which will be used in Spark.



#!/usr/bin/env python3

from pyspark_cassandra import CassandraSparkContext, Row

from pyspark import SparkContext, SparkConf

from pyspark.sql import SQLContext

import os



os.environ['CLASSPATH']="/mnt/spark/lib"



conf = SparkConf().setAppName("test").setMaster("spark://192.168.23.31:7077").set("spark.cassandra.connection.host", "192.168.23.31")

sc = CassandraSparkContext(conf=conf) sqlContext = SQLContext(sc)

df = sqlContext.read.format("org.apache.spark.sql.cassandra").options(keyspace="lebara_diameter_codes", table="nl_lebara_diameter_codes").load()

list = df.select("errorcode2001").where("errorcode2001 > 1200").collect()

list2 = df.select("date").collect()

print([i for i in list[0]])

print(type(list[0]))



The error that it throws is the following one (which is logical because I do not load the jar files):

py4j.protocol.Py4JJavaError: An error occurred while calling o29.load.

: java.lang.ClassNotFoundException: Failed to find data source: org.apache.spark.sql.cassandra. Please find packages at http://spark-packages.org



Is there a way to load those jar files into python or the classpath when calling sqlContext.read.format("org.apache.spark.sql.cassandra")?



Or on the other hand I have to create python scripts with #!/usr/bin/env pyspark?



BR



Joaqun



This email is confidential and may be subject to privilege. If you are not the intended recipient, please do not copy or disclose its content but contact the sender immediately upon receipt.