Spark-Ignite integration: unable to find CONFIG

[rsmanda <rs...@yahoo.com>]

Hi,

When I try to execute the sample given here
"https://github.com/apache/ignite/blob/master/examples/src/main/spark/org/apache/ignite/examples/spark/JavaIgniteDataFrameWriteExample.java",
it throws an error "java.io.FileNotFoundException: File
C:/Apache%20Ignite/example-ignite.xml does not exist".

The file exists and is readable (tried this with a separate sample program).

The config file works fine while starting the Ignite cluster:

String CONFIG = "C:\\Apache Ignite\\example-ignite.xml":
ignite = Ignition.start(CONFIG);

but gives an error when I try to save a DataSet to Ignite:

equiJoinDS.write()
                .format(IgniteDataFrameSettings.FORMAT_IGNITE())
                .mode(SaveMode.Append)
                .option(IgniteDataFrameSettings.OPTION_CONFIG_FILE(),
CONFIG)
                .option(IgniteDataFrameSettings.OPTION_TABLE(),
"raw_message")
               
.option(IgniteDataFrameSettings.OPTION_CREATE_TABLE_PRIMARY_KEY_FIELDS(),
"id")
               
.option(IgniteDataFrameSettings.OPTION_CREATE_TABLE_PARAMETERS(),
"template=replicated")
                .save();


Complete error stack below:

Exception in thread "main" class org.apache.ignite.IgniteCheckedException:
Failed to instantiate Spring XML application context
[springUrl=file:/C:/Apache%20Ignite/example-ignite.xml, err=IOException
parsing XML document from URL [file:/C:/Apache%20Ignite/example-ignite.xml];
nested exception is java.io.FileNotFoundException: File
C:/Apache%20Ignite/example-ignite.xml does not exist]
	at
org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.applicationContext(IgniteSpringHelperImpl.java:392)
	at
org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.loadConfigurations(IgniteSpringHelperImpl.java:104)
	at
org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.loadConfigurations(IgniteSpringHelperImpl.java:98)
	at
org.apache.ignite.internal.IgnitionEx.loadConfigurations(IgnitionEx.java:744)
	at
org.apache.ignite.internal.IgnitionEx.loadConfigurations(IgnitionEx.java:783)
	at
org.apache.ignite.internal.IgnitionEx.loadConfiguration(IgnitionEx.java:823)
	at
org.apache.ignite.spark.impl.IgniteRelationProvider$$anonfun$configProvider$1$1.apply(IgniteRelationProvider.scala:216)
	at
org.apache.ignite.spark.impl.IgniteRelationProvider$$anonfun$configProvider$1$1.apply(IgniteRelationProvider.scala:213)
	at org.apache.ignite.spark.Once.apply(IgniteContext.scala:222)
	at org.apache.ignite.spark.IgniteContext.ignite(IgniteContext.scala:144)
	at org.apache.ignite.spark.IgniteContext.<init>(IgniteContext.scala:63)
	at org.apache.ignite.spark.IgniteContext$.apply(IgniteContext.scala:192)
	at
org.apache.ignite.spark.impl.IgniteRelationProvider.igniteContext(IgniteRelationProvider.scala:236)
	at
org.apache.ignite.spark.impl.IgniteRelationProvider.createRelation(IgniteRelationProvider.scala:100)
	at
org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:46)
	at
org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
	at
org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
	at
org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:86)
	at
org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131)
	at
org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:127)
	at
org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155)
	at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
	at
org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)
	at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127)
	at
org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:80)
	at
org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:80)
	at
org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:654)
	at
org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:654)
	at
org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:77)
	at
org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:654)
	at
org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:273)
	at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:267)
	at SparkTReX.findMatch(SparkTReX.java:127)
	at SparkTReX.runRecon(SparkTReX.java:96)
	at SparkTReX.main(SparkTReX.java:82)
Caused by: org.springframework.beans.factory.BeanDefinitionStoreException:
IOException parsing XML document from URL
[file:/C:/Apache%20Ignite/example-ignite.xml]; nested exception is
java.io.FileNotFoundException: File C:/Apache%20Ignite/example-ignite.xml
does not exist
	at
org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:344)
	at
org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:304)
	at
org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.applicationContext(IgniteSpringHelperImpl.java:379)
	... 34 more
Caused by: java.io.FileNotFoundException: File
C:/Apache%20Ignite/example-ignite.xml does not exist
	at
org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:539)
	at
org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:752)
	at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:529)
	at
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:409)
	at
org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:142)
	at
org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:346)
	at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:766)
	at org.apache.hadoop.fs.FsUrlConnection.connect(FsUrlConnection.java:50)
	at
org.apache.hadoop.fs.FsUrlConnection.getInputStream(FsUrlConnection.java:59)
	at
org.springframework.core.io.UrlResource.getInputStream(UrlResource.java:169)
	at
org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:330)
	... 36 more






--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Spark-Ignite integration: unable to find CONFIG

[aealexsandrov <ae...@gmail.com>]

Hi,

The most obvious reason that you have some mistake in the file name.

I checked the path with white space and ignite was successfully started. In
case if you are sure that path to file is ok then could you please attach
your full example and I will try to start it with spark local installation.

BR,
Andrei



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/