You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@ignite.apache.org by vkulichenko <va...@gmail.com> on 2017/08/09 21:54:53 UTC

Re: Running Spark SQL on Spark Thrift Server with Ignite

Hi Ravi,

I don't think it currently will, because this will require integration with
data frames. We have it plans, but it is not implemented yet. I think you
should use IgniteRDD or Ignite APIs directly.

Can you describe business use case you're trying to implement?

-Val



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Running-Spark-SQL-on-Spark-Thrift-Server-with-Ignite-tp16087p16091.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Running Spark SQL on Spark Thrift Server with Ignite

Posted by vkulichenko <va...@gmail.com>.

Ravi,

You can take a look at these pages:

https://apacheignite-fs.readme.io/docs/ignitecontext-igniterdd
https://apacheignite-fs.readme.io/docs/testing-integration-with-spark-shell

-Val



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Running-Spark-SQL-on-Spark-Thrift-Server-with-Ignite-tp16087p16184.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Running Spark SQL on Spark Thrift Server with Ignite

Posted by ravi <me...@gmail.com>.

Hi Val,
  Thanks for the reply. Can you share some example or sample code snippet
for the steps you have explained?. The link you have shared doesn't explain
mapping Hive/Spar RDD to Ignite RDD context?

Regards
Ravi.P



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Running-Spark-SQL-on-Spark-Thrift-Server-with-Ignite-tp16087p16153.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Running Spark SQL on Spark Thrift Server with Ignite

Posted by vkulichenko <va...@gmail.com>.

Ravi,

If you need to speed up SQL, you should make sure Ignite uses indexes to
execute queries. I think you can do the following:
- Create Hive RDD and map it to RDD of key value pairs.
- Create new IgniteRDD on top of a cache and use IgniteRDD#savePairs method
to load data from Hive to Ignite.
- IgniteRDD#sql method to execute queries.

Note that SQL needs to be configured in Ignite (i.e. you need to specify
queryable fields, indexes, etc.). More information here:
https://apacheignite.readme.io/docs/sql-queries

-Val



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Running-Spark-SQL-on-Spark-Thrift-Server-with-Ignite-tp16087p16115.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.