Spark setup

[George Davies <ge...@gmail.com>]

I have already have a standalone ignite cluster running on k8s and can run
SQL statements against it fine.

Part of a requirements on the system i am building is to perform v-pivots
on the query result set.

I've seen spark come up as a good solution to v-pivots and so I'm trying to
set up a simple master + executor cluster.

I have added all the ignite libs to the classpath per the docs but when i
attempt to launch the master i get the error:

Error SparkUncaughtExceptionHandler:91 - Uncaught Exception in thread
Thread[main,5,main]
java.io.IOException: failure to login
Caused by: javax.security.auth.login.LoginException:
java.lang.NullPointerException: invalid null input: name

Any pointers on what I am doing incorrectly? I dont have a separate HDFS
cluster to log in to, I just want to use spark over the ignite caches.

Re: Spark setup

[Andrei Aleksandrov <ae...@gmail.com>]

Hi,

Ignite 2.7.5 requires the spark of 2.3.X version.

You should start the separate Spark and Ignite cluster:

https://apacheignite.readme.io/docs/getting-started
https://spark.apache.org/docs/2.3.0/spark-standalone.html

After that you should provide all required Ignite libraries to you 
driver and executor classpath. You can just copy all Ignite jars to 
every Spark node and add them to Spark classpath or try to use next 
script to submit your spark job:

LIBS_DIR=$1
EXAMPLE_CLASS=$2
PATH_TO_JAR=$3
JARS=find $LIBS_DIR -name '*.jar'
EXECUTOR_PATH=""
  for eachjarinlib in $JARS ; do
if [ "$eachjarinlib" != "ABCDEFGHIJKLMNOPQRSTUVWXYZ.JAR" ]; then
     EXECUTOR_PATH=file:$eachjarinlib:$EXECUTOR_PATH
fi
done
spark-submit --deploy-mode client --master 
spark://andrei-ThinkPad-P51s:7077 --conf 
"spark.driver.extraClassPath=$EXECUTOR_PATH" --conf 
"spark.executor.extraClassPath=$EXECUTOR_PATH" --class $EXAMPLE_CLASS 
$PATH_TO_JAR $4 $5 $6 $7

Libs can be collected in one place in case of you used maven project as 
next:

<plugin><groupId>org.apache.maven.plugins</groupId><artifactId>maven-dependency-plugin</artifactId><version>3.1.1</version><executions><execution><id>copy-sources</id><phase>package</phase><goals><goal>copy-dependencies</goal></goals><configuration><outputDirectory>target/libs</outputDirectory><overWriteReleases>false</overWriteReleases><overWriteSnapshots>false</overWriteSnapshots><overWriteIfNewer>true</overWriteIfNewer></configuration></execution></executions></plugin>

After that your start command should be like next:

bash run_example.sh ./target/libs/* com.some.your.ClassName 
./target/your.jar client.xml

Some Spark job example you can see here (the code from this link can be 
used with Ignite as well):

https://docs.gridgain.com/docs/cross-database-queries

BR,
Andrei

9/22/2019 8:36 PM, George Davies пишет:
> I have already have a standalone ignite cluster running on k8s and can 
> run SQL statements against it fine.
>
> Part of a requirements on the system i am building is to perform 
> v-pivots on the query result set.
>
> I've seen spark come up as a good solution to v-pivots and so I'm 
> trying to set up a simple master + executor cluster.
>
> I have added all the ignite libs to the classpath per the docs but 
> when i attempt to launch the master i get the error:
>
> Error SparkUncaughtExceptionHandler:91 - Uncaught Exception in thread 
> Thread[main,5,main]
> java.io.IOException: failure to login
> Caused by: javax.security.auth.login.LoginException: 
> java.lang.NullPointerException: invalid null input: name
>
> Any pointers on what I am doing incorrectly? I dont have a separate 
> HDFS cluster to log in to, I just want to use spark over the ignite 
> caches.
>
>
>