You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Mohit Anchlia <mo...@gmail.com> on 2015/03/10 23:04:16 UTC

Hadoop Map vs Spark stream Map

Hi,

I am trying to understand Hadoop Map method compared to spark Map and I
noticed that spark Map only receives 3 arguments 1) input value 2) output
key 3) output value, however in hadoop map it has 4 values 1) input key 2)
input value 3) output key 4) output value. Is there any reason it was
designed this way? Just trying to undersand:

Hadoop:

public void map(K key, V val,
                       OutputCollector<K, V> output, Reporter reporter)

--------


		// Count each word in each batch

		JavaPairDStream<String, Integer> *pairs* = words.mapToPair(

		  *new* *PairFunction<String, String, Integer>()* {

		    @Override *public* Tuple2<String, Integer> call(String s)
*throws* Exception {

		      *return* *new* Tuple2<String, Integer>(s, 1);

		    }

		  });