Re: Ignite RDD Map Reduce function

[vkulichenko <va...@gmail.com>]

Hi,

Please properly subscribe to the mailing list so that the community can
receive email notifications. Here is the instruction:
http://apache-ignite-users.70518.x6.nabble.com/mailing_list/MailingListOptions.jtp?forum=1


pmazumdar wrote
> Can someone please provide an example as to how to use map and reduce
> functions of ignite RDD (used with Spark)? I want to calculate weighted
> average on the data that resides on ignite cache using ignite RDD. i.e.
> 
> IgniteRDD<Double, Double> igniteRDD =
> ic.fromCache(CacheConfig.partitionedCache());
> 
> I now want to use Map and Reduce functions on igniteRDD to calculate
> weighted average. Could someone please assist?

IgniteRDD extends standard Spark RDD, so all APIs are available. You can use
standard map() and reduce() functions, as well as all others.

In some cases you can speed up the process using SQL queries that you can
run directly against data in Ignite using IgniteRDD.sql() method. See
details here:
https://apacheignite-fs.readme.io/docs/ignitecontext-igniterdd#section-running-sql-queries-against-ignite-cache

-Val



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-RDD-Map-Reduce-function-tp6880p6888.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Ignite RDD Map Reduce function

[vkulichenko <va...@gmail.com>]

pmazumdar wrote
> IgniteRDD's map function expects a function and a ClassTag. What is this
> ClassTag here? I am new to Ignite and Spark. If you could please explain
> with an example, it would be very helpful for me. Thanks.

map() method accepts only a function. ClassTag is not a parameter, it's just
a special handling of generics used in Scala. Note that all this stuff is
Spark API not Ignite API, so I recommend to refer to Spark docs and
examples.

However, if you never used Spark, I don't see any reason why you use
IgniteRDD. Why not use Ignite APIs directly? For map-reduce, for example,
you can use compute tasks [1].

[1] https://apacheignite.readme.io/docs/compute-tasks#computetask

-Val



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-RDD-Map-Reduce-function-tp6880p6934.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.