You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Luke Xiong <le...@gmail.com> on 2023/04/29 05:56:05 UTC

Fwd: A WordCount job using DataStream API but behave like the batch WordCount example

Dear experts,

Is it possible to write a WordCount job that uses the DataStream API, but
make it behave like the batch version WordCount example?

More specifically, I hope the job can get a DataStream of the final (word,
count) records when fed a text file.

For example, given a text file:
```input.txt
hello hello world hello world
hello world world world hello world
```

In the flink WordCount examples, the batch version outputs:
```batch.version.output
hello 5
world 6
```

while the stream version outputs:
```stream.version.output
(hello,1)
(hello,2)
(world,1)
(hello,3)
(world,2)
(hello,4)
(world,3)
(world,4)
(world,5)
(hello,5)
(world,6)
```
Is it possible to have a DataStream that only has two elements: (hello, 5)
and (world, 6)?

Regards,
Luke

Re: A WordCount job using DataStream API but behave like the batch WordCount example

Posted by Marco Villalobos <mv...@kineteque.com>.
Hi Luke,

A batch has a beginning and an end. Although a stream has a beginning, it has no end.

Thus, you'll have three choices:

1. You can collect your aggregated sum at a period interval
2. You can aggregate the sum in a time window. If your window is large enough, you'll get the same result.
3. You can also run the stream in batch mode. 

Remember, a stream does not end (unless it is run in batch mode).

> On Apr 29, 2023, at 9:11 AM, Marco Villalobos <mv...@kineteque.com> wrote:
> 
> Hi Luke,
> 
> A batch has a beginning and an end. Although a stream has a beginning, it has no end.
> 
> Thus, you'll have three choices:
> 
> 1. You can collect your aggregated sum at a period interval
> 2. You can aggregate the sum in a time window. If your window is large enough, you'll get the same result.
> 3. You can also run the stream in batch mode. 
> 
> Remember, a stream does not end (unless it is run in batch mode).
> 
> 
> 
>> On Apr 28, 2023, at 10:56 PM, Luke Xiong <le...@gmail.com> wrote:
>> 
>> Dear experts,
>> 
>> Is it possible to write a WordCount job that uses the DataStream API, but make it behave like the batch version WordCount example?
>> 
>> More specifically, I hope the job can get a DataStream of the final (word, count) records when fed a text file.
>> 
>> For example, given a text file:
>> ```input.txt
>> hello hello world hello world
>> hello world world world hello world
>> ```
>> 
>> In the flink WordCount examples, the batch version outputs:
>> ```batch.version.output
>> hello 5
>> world 6
>> ```
>> 
>> while the stream version outputs:
>> ```stream.version.output
>> (hello,1)
>> (hello,2)
>> (world,1)
>> (hello,3)
>> (world,2)
>> (hello,4)
>> (world,3)
>> (world,4)
>> (world,5)
>> (hello,5)
>> (world,6)
>> ```
>> Is it possible to have a DataStream that only has two elements: (hello, 5) and (world, 6)?
>> 
>> Regards,
>> Luke
> 


Re: A WordCount job using DataStream API but behave like the batch WordCount example

Posted by Luke Xiong <le...@gmail.com>.
Hi Ken,

Never mind. I did a map of  Tuple2.of(C, 1) -> Tuple2.of(F, 1)  before
keyBy, then replaced the map after sum with  Tuple2.of(F, <count>) -> d . I
got the expected result. So it's very likely something is wrong with C in
determining equalness. Thank you all the same.

- Luke

On Sun, Apr 30, 2023 at 12:56 PM Ken Krugler <kk...@transpac.com>
wrote:

> Hi Luke,
>
> Without more details, I don’t have a good explanation for why you’re
> seeing duplicate results.
>
> — Ken
>
> On Apr 29, 2023, at 7:38 PM, Luke Xiong <le...@gmail.com> wrote:
>
> Hi Ken,
>
> My workflow looks like this:
>
> dataStream
>     .map(A -> B)
>     .flatMap(B -> some Tuple2.of(C, 1))
>     .keyBy(t, t.f0) // a.k.a. C
>     .sum(1)
>     .map(Tuple2.of(C, <count>) -> d)
>     ;
>
> So just illustrative, and I am not writing a WordCount job either.
>
> - Luke
>
>
> On Sat, Apr 29, 2023 at 10:31 PM Ken Krugler <kk...@transpac.com>
> wrote:
>
>> Hi Luke,
>>
>> What’s your workflow look like?
>>
>> The one I included in my email generates what I’d expect.
>>
>> — Ken
>>
>> On Apr 29, 2023, at 7:22 PM, Luke Xiong <le...@gmail.com> wrote:
>>
>> Hi Marco and Ken,
>>
>> Thanks for the tips. I tried setting runtime mode to BATCH and it seems
>> to work. However, I notice there are duplicate keys in the resulting
>> stream. For example, it has
>> (hello, 2)
>> (hello, 2)
>> instead of
>> (hello, 4)
>>
>> I suspect there might be a bug in the equals method of the keyed object,
>> but it doesn't seem likely, because I can get expected result with
>>
>> .collect(Collectors.groupingByConcurrent(Function.identity(), Collectors.counting()))
>>
>> with the same stream using the Java Stream API.
>>
>> Is there any other reason that causes it, and what should I do to get a stream with only one element per key?
>>
>> -lxiong
>>
>>
>> On Sat, Apr 29, 2023 at 5:24 PM Ken Krugler <kk...@transpac.com>
>> wrote:
>>
>>> Hi Luke,
>>>
>>> If you’re reading from a bounded source (like a file, and you’re not
>>> doing any monitoring of the directory or the file to load more data) then
>>> yes, you can get a final count using a stream.
>>>
>>> The key point is that the Flink file source will send out
>>> a Watermark.MAX_WATERMARK value when it is done, and this will trigger any
>>> pending aggregations.
>>>
>>> Though you do need to make sure your workflow mode is either explicitly
>>> BOUNDED, or AUTOMATIC.
>>>
>>> Something like…
>>>
>>>         *final* StreamExecutionEnvironment env = ...;
>>>         env.setRuntimeMode(RuntimeExecutionMode.*AUTOMATIC*);
>>>
>>>         DataStream<String> lines = ...;
>>>
>>>         lines
>>>             .flatMap(*new* SplitLineFunction())
>>>             .keyBy(t -> t.f0)
>>>             .sum(1)
>>>             .print();
>>>
>>>         env.execute();
>>>
>>> …with a typical line splitter, cheesy version like...
>>>
>>>     *private* *static* *class* SplitLineFunction *implements*
>>> FlatMapFunction<String, Tuple2<String, Integer>> {
>>>
>>>         @Override
>>>         *public* *void* flatMap(String in, Collector<Tuple2<String,
>>> Integer>> out) *throws* Exception {
>>>             *for* (String word : in.split(" ")) {
>>>                 out.collect(Tuple2.*of*(word, 1));
>>>             }
>>>         }
>>>     }
>>>
>>> — Ken
>>>
>>> On Apr 28, 2023, at 10:56 PM, Luke Xiong <le...@gmail.com> wrote:
>>>
>>> Dear experts,
>>>
>>> Is it possible to write a WordCount job that uses the DataStream API,
>>> but make it behave like the batch version WordCount example?
>>>
>>> More specifically, I hope the job can get a DataStream of the final
>>> (word, count) records when fed a text file.
>>>
>>> For example, given a text file:
>>> ```input.txt
>>> hello hello world hello world
>>> hello world world world hello world
>>> ```
>>>
>>> In the flink WordCount examples, the batch version outputs:
>>> ```batch.version.output
>>> hello 5
>>> world 6
>>> ```
>>>
>>> while the stream version outputs:
>>> ```stream.version.output
>>> (hello,1)
>>> (hello,2)
>>> (world,1)
>>> (hello,3)
>>> (world,2)
>>> (hello,4)
>>> (world,3)
>>> (world,4)
>>> (world,5)
>>> (hello,5)
>>> (world,6)
>>> ```
>>> Is it possible to have a DataStream that only has two elements: (hello,
>>> 5) and (world, 6)?
>>>
>>> Regards,
>>> Luke
>>>
>>>
>>> --------------------------
>>> Ken Krugler
>>> http://www.scaleunlimited.com
>>> Custom big data solutions
>>> Flink, Pinot, Solr, Elasticsearch
>>>
>>>
>>>
>>>
>> --------------------------
>> Ken Krugler
>> http://www.scaleunlimited.com
>> Custom big data solutions
>> Flink, Pinot, Solr, Elasticsearch
>>
>>
>>
>>
> --------------------------
> Ken Krugler
> http://www.scaleunlimited.com
> Custom big data solutions
> Flink, Pinot, Solr, Elasticsearch
>
>
>
>

Re: A WordCount job using DataStream API but behave like the batch WordCount example

Posted by Ken Krugler <kk...@transpac.com>.
Hi Luke,

Without more details, I don’t have a good explanation for why you’re seeing duplicate results.

— Ken

> On Apr 29, 2023, at 7:38 PM, Luke Xiong <le...@gmail.com> wrote:
> 
> Hi Ken,
> 
> My workflow looks like this:
> 
> dataStream
>     .map(A -> B)
>     .flatMap(B -> some Tuple2.of(C, 1))
>     .keyBy(t, t.f0) // a.k.a. C
>     .sum(1)
>     .map(Tuple2.of(C, <count>) -> d)
>     ;
> 
> So just illustrative, and I am not writing a WordCount job either.
> 
> - Luke
> 
> 
> On Sat, Apr 29, 2023 at 10:31 PM Ken Krugler <kkrugler_lists@transpac.com <ma...@transpac.com>> wrote:
> Hi Luke,
> 
> What’s your workflow look like?
> 
> The one I included in my email generates what I’d expect.
> 
> — Ken
> 
>> On Apr 29, 2023, at 7:22 PM, Luke Xiong <leixure@gmail.com <ma...@gmail.com>> wrote:
>> 
>> Hi Marco and Ken,
>> 
>> Thanks for the tips. I tried setting runtime mode to BATCH and it seems to work. However, I notice there are duplicate keys in the resulting stream. For example, it has
>> (hello, 2)
>> (hello, 2)
>> instead of
>> (hello, 4)
>> 
>> I suspect there might be a bug in the equals method of the keyed object, but it doesn't seem likely, because I can get expected result with
>> .collect(Collectors.groupingByConcurrent(Function.identity(), Collectors.counting()))
>> with the same stream using the Java Stream API.
>> Is there any other reason that causes it, and what should I do to get a stream with only one element per key?
>> -lxiong
>> 
>> On Sat, Apr 29, 2023 at 5:24 PM Ken Krugler <kkrugler_lists@transpac.com <ma...@transpac.com>> wrote:
>> Hi Luke,
>> 
>> If you’re reading from a bounded source (like a file, and you’re not doing any monitoring of the directory or the file to load more data) then yes, you can get a final count using a stream.
>> 
>> The key point is that the Flink file source will send out a Watermark.MAX_WATERMARK value when it is done, and this will trigger any pending aggregations.
>> 
>> Though you do need to make sure your workflow mode is either explicitly BOUNDED, or AUTOMATIC.
>> 
>> Something like…
>> 
>>         final StreamExecutionEnvironment env = ...;
>>         env.setRuntimeMode(RuntimeExecutionMode.AUTOMATIC);
>>         
>>         DataStream<String> lines = ...;
>>         
>>         lines
>>             .flatMap(new SplitLineFunction())
>>             .keyBy(t -> t.f0)
>>             .sum(1)
>>             .print();
>>         
>>         env.execute();
>> 
>> …with a typical line splitter, cheesy version like...
>> 
>>     private static class SplitLineFunction implements FlatMapFunction<String, Tuple2<String, Integer>> {
>> 
>>         @Override
>>         public void flatMap(String in, Collector<Tuple2<String, Integer>> out) throws Exception {
>>             for (String word : in.split(" ")) {
>>                 out.collect(Tuple2.of(word, 1));
>>             }
>>         }
>>     }
>> 
>> — Ken
>> 
>>> On Apr 28, 2023, at 10:56 PM, Luke Xiong <leixure@gmail.com <ma...@gmail.com>> wrote:
>>> 
>>> Dear experts,
>>> 
>>> Is it possible to write a WordCount job that uses the DataStream API, but make it behave like the batch version WordCount example?
>>> 
>>> More specifically, I hope the job can get a DataStream of the final (word, count) records when fed a text file.
>>> 
>>> For example, given a text file:
>>> ```input.txt
>>> hello hello world hello world
>>> hello world world world hello world
>>> ```
>>> 
>>> In the flink WordCount examples, the batch version outputs:
>>> ```batch.version.output
>>> hello 5
>>> world 6
>>> ```
>>> 
>>> while the stream version outputs:
>>> ```stream.version.output
>>> (hello,1)
>>> (hello,2)
>>> (world,1)
>>> (hello,3)
>>> (world,2)
>>> (hello,4)
>>> (world,3)
>>> (world,4)
>>> (world,5)
>>> (hello,5)
>>> (world,6)
>>> ```
>>> Is it possible to have a DataStream that only has two elements: (hello, 5) and (world, 6)?
>>> 
>>> Regards,
>>> Luke
>> 
>> --------------------------
>> Ken Krugler
>> http://www.scaleunlimited.com <http://www.scaleunlimited.com/>
>> Custom big data solutions
>> Flink, Pinot, Solr, Elasticsearch
>> 
>> 
>> 
> 
> --------------------------
> Ken Krugler
> http://www.scaleunlimited.com <http://www.scaleunlimited.com/>
> Custom big data solutions
> Flink, Pinot, Solr, Elasticsearch
> 
> 
> 

--------------------------
Ken Krugler
http://www.scaleunlimited.com
Custom big data solutions
Flink, Pinot, Solr, Elasticsearch




Re: A WordCount job using DataStream API but behave like the batch WordCount example

Posted by Ken Krugler <kk...@transpac.com>.
Hi Luke,

What’s your workflow look like?

The one I included in my email generates what I’d expect.

— Ken

> On Apr 29, 2023, at 7:22 PM, Luke Xiong <le...@gmail.com> wrote:
> 
> Hi Marco and Ken,
> 
> Thanks for the tips. I tried setting runtime mode to BATCH and it seems to work. However, I notice there are duplicate keys in the resulting stream. For example, it has
> (hello, 2)
> (hello, 2)
> instead of
> (hello, 4)
> 
> I suspect there might be a bug in the equals method of the keyed object, but it doesn't seem likely, because I can get expected result with
> .collect(Collectors.groupingByConcurrent(Function.identity(), Collectors.counting()))
> with the same stream using the Java Stream API.
> Is there any other reason that causes it, and what should I do to get a stream with only one element per key?
> -lxiong
> 
> On Sat, Apr 29, 2023 at 5:24 PM Ken Krugler <kkrugler_lists@transpac.com <ma...@transpac.com>> wrote:
> Hi Luke,
> 
> If you’re reading from a bounded source (like a file, and you’re not doing any monitoring of the directory or the file to load more data) then yes, you can get a final count using a stream.
> 
> The key point is that the Flink file source will send out a Watermark.MAX_WATERMARK value when it is done, and this will trigger any pending aggregations.
> 
> Though you do need to make sure your workflow mode is either explicitly BOUNDED, or AUTOMATIC.
> 
> Something like…
> 
>         final StreamExecutionEnvironment env = ...;
>         env.setRuntimeMode(RuntimeExecutionMode.AUTOMATIC);
>         
>         DataStream<String> lines = ...;
>         
>         lines
>             .flatMap(new SplitLineFunction())
>             .keyBy(t -> t.f0)
>             .sum(1)
>             .print();
>         
>         env.execute();
> 
> …with a typical line splitter, cheesy version like...
> 
>     private static class SplitLineFunction implements FlatMapFunction<String, Tuple2<String, Integer>> {
> 
>         @Override
>         public void flatMap(String in, Collector<Tuple2<String, Integer>> out) throws Exception {
>             for (String word : in.split(" ")) {
>                 out.collect(Tuple2.of(word, 1));
>             }
>         }
>     }
> 
> — Ken
> 
>> On Apr 28, 2023, at 10:56 PM, Luke Xiong <leixure@gmail.com <ma...@gmail.com>> wrote:
>> 
>> Dear experts,
>> 
>> Is it possible to write a WordCount job that uses the DataStream API, but make it behave like the batch version WordCount example?
>> 
>> More specifically, I hope the job can get a DataStream of the final (word, count) records when fed a text file.
>> 
>> For example, given a text file:
>> ```input.txt
>> hello hello world hello world
>> hello world world world hello world
>> ```
>> 
>> In the flink WordCount examples, the batch version outputs:
>> ```batch.version.output
>> hello 5
>> world 6
>> ```
>> 
>> while the stream version outputs:
>> ```stream.version.output
>> (hello,1)
>> (hello,2)
>> (world,1)
>> (hello,3)
>> (world,2)
>> (hello,4)
>> (world,3)
>> (world,4)
>> (world,5)
>> (hello,5)
>> (world,6)
>> ```
>> Is it possible to have a DataStream that only has two elements: (hello, 5) and (world, 6)?
>> 
>> Regards,
>> Luke
> 
> --------------------------
> Ken Krugler
> http://www.scaleunlimited.com <http://www.scaleunlimited.com/>
> Custom big data solutions
> Flink, Pinot, Solr, Elasticsearch
> 
> 
> 

--------------------------
Ken Krugler
http://www.scaleunlimited.com
Custom big data solutions
Flink, Pinot, Solr, Elasticsearch