You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Mina Aslani <as...@gmail.com> on 2018/05/11 02:14:13 UTC
java.lang.NullPointerException
Hi,
I get
java.lang.NullPointerException at
org.apache.spark.sql.SparkSession.sessionState$lzycompute(SparkSession.scala:128)
When I try to createDataFrame using the sparkSession, see below:
SparkConf conf = new SparkConf().setMaster(....).setAppName("test");
conf.set("spark.driver.allowMultipleContexts", "true");
//Read messages in batch of duration seconds
JavaStreamingContext jssc = new JavaStreamingContext(conf,
Durations.seconds(1));
SparkSession spark = SparkSession.builder().config(conf).getOrCreate();
List<Row> data = Arrays.asList(
RowFactory.create(val1, val2, val3)
);
StructType schema = new StructType(new StructField[]{
createStructField("field1", StringType, false),
createStructField("field2", StringType, false),
createStructField("field3", StringType, false)
});
Dataset<Row> df = spark.createDataFrame(data, schema);
Any idea? I am using JAVA therefore,I cannot convert my RDD to DF(). T
Therefore, I try to get specific field/values and createDataFrame manually.
Regards,
Mina