You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Eduardo Costa Alfaia <e....@unibs.it> on 2014/04/04 17:03:43 UTC
reduceByKeyAndWindow Java
Hi guys,
I would like knowing if the part of code is right to use in Window.
JavaPairDStream<String, Integer> wordCounts = words.map(
103 new PairFunction<String, String, Integer>() {
104 @Override
105 public Tuple2<String, Integer> call(String s) {
106 return new Tuple2<String, Integer>(s, 1);
107 }
108 }).reduceByKeyAndWindow(
109 new Function2<Integer, Integer, Integer>() {
110 public Integer call(Integer i1, Integer i2) { return i1 +
i2; }
111 },
112 new Function2<Integer, Integer, Integer>() {
113 public Integer call(Integer i1, Integer i2) { return i1 -
i2; }
114 },
115 new Duration(60 * 5 * 1000),
116 new Duration(1 * 1000)
117 );
Thanks
--
Informativa sulla Privacy: http://www.unibs.it/node/8155
Re: reduceByKeyAndWindow Java
Posted by Eduardo Costa Alfaia <e....@unibs.it>.
Hi TD
Could you explain me this code part?
.reduceByKeyAndWindow(
109 new Function2<Integer, Integer, Integer>() {
110 public Integer call(Integer i1, Integer i2) { return i1 +
i2; }
111 },
112 new Function2<Integer, Integer, Integer>() {
113 public Integer call(Integer i1, Integer i2) { return i1 -
i2; }
114 },
115 new Duration(60 * 5 * 1000),
116 new Duration(1 * 1000)
117 );
Thanks
Em 4/4/14, 22:56, Tathagata Das escreveu:
> I havent really compiled the code, but it looks good to me. Why? Is
> there any problem you are facing?
>
> TD
>
>
> On Fri, Apr 4, 2014 at 8:03 AM, Eduardo Costa Alfaia
> <e.costaalfaia@unibs.it <ma...@unibs.it>> wrote:
>
>
> Hi guys,
>
> I would like knowing if the part of code is right to use in Window.
>
> JavaPairDStream<String, Integer> wordCounts = words.map(
> 103 new PairFunction<String, String, Integer>() {
> 104 @Override
> 105 public Tuple2<String, Integer> call(String s) {
> 106 return new Tuple2<String, Integer>(s, 1);
> 107 }
> 108 }).reduceByKeyAndWindow(
> 109 new Function2<Integer, Integer, Integer>() {
> 110 public Integer call(Integer i1, Integer i2) { return
> i1 + i2; }
> 111 },
> 112 new Function2<Integer, Integer, Integer>() {
> 113 public Integer call(Integer i1, Integer i2) { return
> i1 - i2; }
> 114 },
> 115 new Duration(60 * 5 * 1000),
> 116 new Duration(1 * 1000)
> 117 );
>
>
>
> Thanks
>
> --
> Informativa sulla Privacy: http://www.unibs.it/node/8155
>
>
--
Informativa sulla Privacy: http://www.unibs.it/node/8155
Re: reduceByKeyAndWindow Java
Posted by Eduardo Costa Alfaia <e....@unibs.it>.
Hi Tathagata,
You are right, this code compile, but I am some problems with high
memory consummation, I sent today some email about this, but no response
until now.
Thanks
Em 4/4/14, 22:56, Tathagata Das escreveu:
> I havent really compiled the code, but it looks good to me. Why? Is
> there any problem you are facing?
>
> TD
>
>
> On Fri, Apr 4, 2014 at 8:03 AM, Eduardo Costa Alfaia
> <e.costaalfaia@unibs.it <ma...@unibs.it>> wrote:
>
>
> Hi guys,
>
> I would like knowing if the part of code is right to use in Window.
>
> JavaPairDStream<String, Integer> wordCounts = words.map(
> 103 new PairFunction<String, String, Integer>() {
> 104 @Override
> 105 public Tuple2<String, Integer> call(String s) {
> 106 return new Tuple2<String, Integer>(s, 1);
> 107 }
> 108 }).reduceByKeyAndWindow(
> 109 new Function2<Integer, Integer, Integer>() {
> 110 public Integer call(Integer i1, Integer i2) { return
> i1 + i2; }
> 111 },
> 112 new Function2<Integer, Integer, Integer>() {
> 113 public Integer call(Integer i1, Integer i2) { return
> i1 - i2; }
> 114 },
> 115 new Duration(60 * 5 * 1000),
> 116 new Duration(1 * 1000)
> 117 );
>
>
>
> Thanks
>
> --
> Informativa sulla Privacy: http://www.unibs.it/node/8155
>
>
--
Informativa sulla Privacy: http://www.unibs.it/node/8155
Re: reduceByKeyAndWindow Java
Posted by Tathagata Das <ta...@gmail.com>.
I havent really compiled the code, but it looks good to me. Why? Is there
any problem you are facing?
TD
On Fri, Apr 4, 2014 at 8:03 AM, Eduardo Costa Alfaia <e.costaalfaia@unibs.it
> wrote:
>
> Hi guys,
>
> I would like knowing if the part of code is right to use in Window.
>
> JavaPairDStream<String, Integer> wordCounts = words.map(
> 103 new PairFunction<String, String, Integer>() {
> 104 @Override
> 105 public Tuple2<String, Integer> call(String s) {
> 106 return new Tuple2<String, Integer>(s, 1);
> 107 }
> 108 }).reduceByKeyAndWindow(
> 109 new Function2<Integer, Integer, Integer>() {
> 110 public Integer call(Integer i1, Integer i2) { return i1 +
> i2; }
> 111 },
> 112 new Function2<Integer, Integer, Integer>() {
> 113 public Integer call(Integer i1, Integer i2) { return i1 -
> i2; }
> 114 },
> 115 new Duration(60 * 5 * 1000),
> 116 new Duration(1 * 1000)
> 117 );
>
>
>
> Thanks
>
> --
> Informativa sulla Privacy: http://www.unibs.it/node/8155
>