How to iterate over DataSet elements without converting it to List

[subash basnet <ya...@gmail.com>]

Hello there,

I have been stuck on how to iterate over the DataSet, perform operations
and return a new modified DataSet similar to that of list operation as
shown below.
Eg:
for (Centroid centroid : centroids.collect()) {
    for (Tuple2<Integer, Point> element : clusteredPoints.collect()) {
       //perform necessary operations
     }
//add elements
}
//return elements list

I need to iterate over dataSet without converting it to List.


Best Regards,
Subash Basnet

Re: How to iterate over DataSet elements without converting it to List

[Judit Fehér <fe...@gmail.com>]

Hi,

if you want to iterate through a DataSet you can simply use the map
function on the DataSets instead of for loops.
In your example you have nested loops, instead of this you can join the two
datasets
and then perform the map function.
It looks like you may want to implement a k-means algorithm. If this is the
case then you might want to check out the Flink k-means example in the
machine learning library, you will also see here how to iterate through a
DataSet with the map function:
https://github.com/apache/flink/blob/master/flink-examples/flink-examples-batch/src/main/java/org/apache/flink/examples/java/clustering/KMeans.java#L100-L107

Best Regards,
Judit

2016-02-18 15:29 GMT+01:00 subash basnet <ya...@gmail.com>:

> Hello there,
>
> I have been stuck on how to iterate over the DataSet, perform operations
> and return a new modified DataSet similar to that of list operation as
> shown below.
> Eg:
> for (Centroid centroid : centroids.collect()) {
>     for (Tuple2<Integer, Point> element : clusteredPoints.collect()) {
>        //perform necessary operations
>      }
> //add elements
> }
> //return elements list
>
> I need to iterate over dataSet without converting it to List.
>
>
> Best Regards,
> Subash Basnet
>