You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by tanguy racinet <ta...@gmail.com> on 2015/02/05 20:37:54 UTC
flink loop
Hi,
We are trying to develop the Apriori algorith with the Flink for our Data
minning project.
In our understanding, Flink could handle loop within the workflow.
However, our knowledge is limited and we cannot find a nice way to do it.
Here is the flow of my algorithm :
GenerateCandidates ----> CalculateFrequentItemSet
mapper ----> reducer
We would like to use the reducer result as the mapper's entry for a
predefined number of times (loop x times).
Is there any smart way to that with Flink. Or should we just copy paste the
loop x times ?
Thank you,
<http://eitictlabs-rennes.fr/>
*Racinet Tanguy*
*EIT ICT Labs Master School Student*
*Distributed Systems and Services*
Tel : +33 6 63 20 89 16 / +49 176 3749 8854
Mail : tanracinet@gmail.com
ᐧ
Re: flink loop
Posted by Till Rohrmann <ti...@gmail.com>.
You can instantiate a RichMapper object in which you have access to the
RuntimeContext with getRuntimeContext(). If this happens within an
iteration, then you can cast the RuntimeContext to a
IterationRuntimeContext which has the method getSuperstepNumber(). This
method gives you the current iteration number.
The RichMapper can be used instead of the original map function. You can
find a small example here [1].
Cheers,
Till
[1] http://flink.apache.org/docs/0.8/programming_guide.html#rich-functions
On Mon, Feb 9, 2015 at 4:46 PM, tanguy racinet <ta...@gmail.com> wrote:
> Hello,
>
> Thank you for your help, our algorithm is working nicely now.
>
> We are now trying to add some more value to it. In order to do so, we
> would like to have the last iteration being slightly different than the
> others.
> We would like to avoid code repetition if possible.
>
> Is there any way to know if we are on the last iteration inside functions
> such as mappers ? Or do we have to duplicate our code and just change a few
> things for the last iteration ?
>
> Regards,
> ᐧ
>
> <http://eitictlabs-rennes.fr/>
>
>
> *Racinet Tanguy*
>
> *EIT ICT Labs Master School Student*
> *Distributed Systems and Services*
>
> Tel : +33 6 63 20 89 16 / +49 176 3749 8854
> Mail : tanracinet@gmail.com
>
> On Sun, Feb 8, 2015 at 1:32 PM, Till Rohrmann <tr...@apache.org>
> wrote:
>
>> It will definitely cost you something but checking whether a tuple of
>> strings is in ascending order should still be linear in the length of the
>> concatenated string. The far more serious problem is actually the cross
>> operation which is really costly. But if you need the cartesian product,
>> then you have to do it.
>>
>> On Sun, Feb 8, 2015 at 1:24 PM, tanguy racinet <ta...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> Thank you, that is what we were thinking about but we are working with
>>> Strings and not int inside permutations.
>>> So we are wondering if the ordering wouldn't be too expensive
>>> ᐧ
>>>
>>> <http://eitictlabs-rennes.fr/>
>>>
>>>
>>> *Racinet Tanguy*
>>>
>>> *EIT ICT Labs Master School Student*
>>> *Distributed Systems and Services*
>>>
>>> Tel : +33 6 63 20 89 16 / +49 176 3749 8854
>>> Mail : tanracinet@gmail.com
>>>
>>> On Sun, Feb 8, 2015 at 1:07 PM, Till Rohrmann <tr...@apache.org>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> you could apply a filter operation after the cross operation which
>>>> filters all combinations out which are not in ascending order.
>>>>
>>>> Cheers,
>>>>
>>>> Till
>>>>
>>>> On Sun, Feb 8, 2015 at 12:38 PM, tanguy racinet <ta...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Thank you for you reply. It helped us solve the looping problems in a
>>>>> nicer way.
>>>>>
>>>>> We are struggling with some aspects of the cross function.
>>>>> Still trying to implement the Apriori algorithm, we need to create
>>>>> combinations of frequent itemSets.
>>>>> Our problem is that the crossing gives us duplicates, for instance
>>>>> :(1, 2, 3, 4) and (2, 1, 4, 3) are equivalent for us so we are trying to
>>>>> find a way to remove that kind of duplicate in our DataSet.
>>>>>
>>>>> We already removed duplicates inside our combinations (1, 1, 2) => (1,
>>>>> 2).
>>>>>
>>>>> We were thinking about using HashSet but they are not serializable and
>>>>> we cannot use them inside the workflow, but only inside functions.
>>>>>
>>>>> Can you think of any way to remove those duplicates ?
>>>>>
>>>>> Thank you,
>>>>> ᐧ
>>>>>
>>>>> <http://eitictlabs-rennes.fr/>
>>>>>
>>>>>
>>>>> *Racinet Tanguy*
>>>>>
>>>>> *EIT ICT Labs Master School Student*
>>>>> *Distributed Systems and Services*
>>>>>
>>>>> Tel : +33 6 63 20 89 16 / +49 176 3749 8854
>>>>> Mail : tanracinet@gmail.com
>>>>>
>>>>> On Thu, Feb 5, 2015 at 8:51 PM, Vasiliki Kalavri <
>>>>> vasilikikalavri@gmail.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I'm not familiar with the particular algorithm, but you can most
>>>>>> probably use one of the two iterate operators in Flink.
>>>>>>
>>>>>> You can read a description and see some examples in the
>>>>>> documentation:
>>>>>>
>>>>>> http://flink.apache.org/docs/0.8/programming_guide.html#iteration-operators
>>>>>>
>>>>>> Let us know if you have any questions!
>>>>>>
>>>>>> Cheers,
>>>>>> V.
>>>>>>
>>>>>> On 5 February 2015 at 20:37, tanguy racinet <ta...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> We are trying to develop the Apriori algorith with the Flink for our
>>>>>>> Data minning project.
>>>>>>> In our understanding, Flink could handle loop within the workflow.
>>>>>>> However, our knowledge is limited and we cannot find a nice way to
>>>>>>> do it.
>>>>>>>
>>>>>>> Here is the flow of my algorithm :
>>>>>>> GenerateCandidates ----> CalculateFrequentItemSet
>>>>>>> mapper ----> reducer
>>>>>>>
>>>>>>> We would like to use the reducer result as the mapper's entry for a
>>>>>>> predefined number of times (loop x times).
>>>>>>>
>>>>>>> Is there any smart way to that with Flink. Or should we just copy
>>>>>>> paste the loop x times ?
>>>>>>>
>>>>>>> Thank you,
>>>>>>> <http://eitictlabs-rennes.fr/>
>>>>>>>
>>>>>>>
>>>>>>> *Racinet Tanguy*
>>>>>>>
>>>>>>> *EIT ICT Labs Master School Student*
>>>>>>> *Distributed Systems and Services*
>>>>>>>
>>>>>>> Tel : +33 6 63 20 89 16 / +49 176 3749 8854
>>>>>>> Mail : tanracinet@gmail.com
>>>>>>>
>>>>>>> ᐧ
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: flink loop
Posted by tanguy racinet <ta...@gmail.com>.
Hello,
Thank you for your help, our algorithm is working nicely now.
We are now trying to add some more value to it. In order to do so, we would
like to have the last iteration being slightly different than the others.
We would like to avoid code repetition if possible.
Is there any way to know if we are on the last iteration inside functions
such as mappers ? Or do we have to duplicate our code and just change a few
things for the last iteration ?
Regards,
ᐧ
<http://eitictlabs-rennes.fr/>
*Racinet Tanguy*
*EIT ICT Labs Master School Student*
*Distributed Systems and Services*
Tel : +33 6 63 20 89 16 / +49 176 3749 8854
Mail : tanracinet@gmail.com
On Sun, Feb 8, 2015 at 1:32 PM, Till Rohrmann <tr...@apache.org> wrote:
> It will definitely cost you something but checking whether a tuple of
> strings is in ascending order should still be linear in the length of the
> concatenated string. The far more serious problem is actually the cross
> operation which is really costly. But if you need the cartesian product,
> then you have to do it.
>
> On Sun, Feb 8, 2015 at 1:24 PM, tanguy racinet <ta...@gmail.com>
> wrote:
>
>> Hi,
>>
>> Thank you, that is what we were thinking about but we are working with
>> Strings and not int inside permutations.
>> So we are wondering if the ordering wouldn't be too expensive
>> ᐧ
>>
>> <http://eitictlabs-rennes.fr/>
>>
>>
>> *Racinet Tanguy*
>>
>> *EIT ICT Labs Master School Student*
>> *Distributed Systems and Services*
>>
>> Tel : +33 6 63 20 89 16 / +49 176 3749 8854
>> Mail : tanracinet@gmail.com
>>
>> On Sun, Feb 8, 2015 at 1:07 PM, Till Rohrmann <tr...@apache.org>
>> wrote:
>>
>>> Hi,
>>>
>>> you could apply a filter operation after the cross operation which
>>> filters all combinations out which are not in ascending order.
>>>
>>> Cheers,
>>>
>>> Till
>>>
>>> On Sun, Feb 8, 2015 at 12:38 PM, tanguy racinet <ta...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> Thank you for you reply. It helped us solve the looping problems in a
>>>> nicer way.
>>>>
>>>> We are struggling with some aspects of the cross function.
>>>> Still trying to implement the Apriori algorithm, we need to create
>>>> combinations of frequent itemSets.
>>>> Our problem is that the crossing gives us duplicates, for instance :(1,
>>>> 2, 3, 4) and (2, 1, 4, 3) are equivalent for us so we are trying to find a
>>>> way to remove that kind of duplicate in our DataSet.
>>>>
>>>> We already removed duplicates inside our combinations (1, 1, 2) => (1,
>>>> 2).
>>>>
>>>> We were thinking about using HashSet but they are not serializable and
>>>> we cannot use them inside the workflow, but only inside functions.
>>>>
>>>> Can you think of any way to remove those duplicates ?
>>>>
>>>> Thank you,
>>>> ᐧ
>>>>
>>>> <http://eitictlabs-rennes.fr/>
>>>>
>>>>
>>>> *Racinet Tanguy*
>>>>
>>>> *EIT ICT Labs Master School Student*
>>>> *Distributed Systems and Services*
>>>>
>>>> Tel : +33 6 63 20 89 16 / +49 176 3749 8854
>>>> Mail : tanracinet@gmail.com
>>>>
>>>> On Thu, Feb 5, 2015 at 8:51 PM, Vasiliki Kalavri <
>>>> vasilikikalavri@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I'm not familiar with the particular algorithm, but you can most
>>>>> probably use one of the two iterate operators in Flink.
>>>>>
>>>>> You can read a description and see some examples in the documentation:
>>>>>
>>>>> http://flink.apache.org/docs/0.8/programming_guide.html#iteration-operators
>>>>>
>>>>> Let us know if you have any questions!
>>>>>
>>>>> Cheers,
>>>>> V.
>>>>>
>>>>> On 5 February 2015 at 20:37, tanguy racinet <ta...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> We are trying to develop the Apriori algorith with the Flink for our
>>>>>> Data minning project.
>>>>>> In our understanding, Flink could handle loop within the workflow.
>>>>>> However, our knowledge is limited and we cannot find a nice way to do
>>>>>> it.
>>>>>>
>>>>>> Here is the flow of my algorithm :
>>>>>> GenerateCandidates ----> CalculateFrequentItemSet
>>>>>> mapper ----> reducer
>>>>>>
>>>>>> We would like to use the reducer result as the mapper's entry for a
>>>>>> predefined number of times (loop x times).
>>>>>>
>>>>>> Is there any smart way to that with Flink. Or should we just copy
>>>>>> paste the loop x times ?
>>>>>>
>>>>>> Thank you,
>>>>>> <http://eitictlabs-rennes.fr/>
>>>>>>
>>>>>>
>>>>>> *Racinet Tanguy*
>>>>>>
>>>>>> *EIT ICT Labs Master School Student*
>>>>>> *Distributed Systems and Services*
>>>>>>
>>>>>> Tel : +33 6 63 20 89 16 / +49 176 3749 8854
>>>>>> Mail : tanracinet@gmail.com
>>>>>>
>>>>>> ᐧ
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: flink loop
Posted by Till Rohrmann <tr...@apache.org>.
It will definitely cost you something but checking whether a tuple of
strings is in ascending order should still be linear in the length of the
concatenated string. The far more serious problem is actually the cross
operation which is really costly. But if you need the cartesian product,
then you have to do it.
On Sun, Feb 8, 2015 at 1:24 PM, tanguy racinet <ta...@gmail.com> wrote:
> Hi,
>
> Thank you, that is what we were thinking about but we are working with
> Strings and not int inside permutations.
> So we are wondering if the ordering wouldn't be too expensive
> ᐧ
>
> <http://eitictlabs-rennes.fr/>
>
>
> *Racinet Tanguy*
>
> *EIT ICT Labs Master School Student*
> *Distributed Systems and Services*
>
> Tel : +33 6 63 20 89 16 / +49 176 3749 8854
> Mail : tanracinet@gmail.com
>
> On Sun, Feb 8, 2015 at 1:07 PM, Till Rohrmann <tr...@apache.org>
> wrote:
>
>> Hi,
>>
>> you could apply a filter operation after the cross operation which
>> filters all combinations out which are not in ascending order.
>>
>> Cheers,
>>
>> Till
>>
>> On Sun, Feb 8, 2015 at 12:38 PM, tanguy racinet <ta...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> Thank you for you reply. It helped us solve the looping problems in a
>>> nicer way.
>>>
>>> We are struggling with some aspects of the cross function.
>>> Still trying to implement the Apriori algorithm, we need to create
>>> combinations of frequent itemSets.
>>> Our problem is that the crossing gives us duplicates, for instance :(1,
>>> 2, 3, 4) and (2, 1, 4, 3) are equivalent for us so we are trying to find a
>>> way to remove that kind of duplicate in our DataSet.
>>>
>>> We already removed duplicates inside our combinations (1, 1, 2) => (1,
>>> 2).
>>>
>>> We were thinking about using HashSet but they are not serializable and
>>> we cannot use them inside the workflow, but only inside functions.
>>>
>>> Can you think of any way to remove those duplicates ?
>>>
>>> Thank you,
>>> ᐧ
>>>
>>> <http://eitictlabs-rennes.fr/>
>>>
>>>
>>> *Racinet Tanguy*
>>>
>>> *EIT ICT Labs Master School Student*
>>> *Distributed Systems and Services*
>>>
>>> Tel : +33 6 63 20 89 16 / +49 176 3749 8854
>>> Mail : tanracinet@gmail.com
>>>
>>> On Thu, Feb 5, 2015 at 8:51 PM, Vasiliki Kalavri <
>>> vasilikikalavri@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> I'm not familiar with the particular algorithm, but you can most
>>>> probably use one of the two iterate operators in Flink.
>>>>
>>>> You can read a description and see some examples in the documentation:
>>>>
>>>> http://flink.apache.org/docs/0.8/programming_guide.html#iteration-operators
>>>>
>>>> Let us know if you have any questions!
>>>>
>>>> Cheers,
>>>> V.
>>>>
>>>> On 5 February 2015 at 20:37, tanguy racinet <ta...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> We are trying to develop the Apriori algorith with the Flink for our
>>>>> Data minning project.
>>>>> In our understanding, Flink could handle loop within the workflow.
>>>>> However, our knowledge is limited and we cannot find a nice way to do
>>>>> it.
>>>>>
>>>>> Here is the flow of my algorithm :
>>>>> GenerateCandidates ----> CalculateFrequentItemSet
>>>>> mapper ----> reducer
>>>>>
>>>>> We would like to use the reducer result as the mapper's entry for a
>>>>> predefined number of times (loop x times).
>>>>>
>>>>> Is there any smart way to that with Flink. Or should we just copy
>>>>> paste the loop x times ?
>>>>>
>>>>> Thank you,
>>>>> <http://eitictlabs-rennes.fr/>
>>>>>
>>>>>
>>>>> *Racinet Tanguy*
>>>>>
>>>>> *EIT ICT Labs Master School Student*
>>>>> *Distributed Systems and Services*
>>>>>
>>>>> Tel : +33 6 63 20 89 16 / +49 176 3749 8854
>>>>> Mail : tanracinet@gmail.com
>>>>>
>>>>> ᐧ
>>>>>
>>>>
>>>>
>>>
>>
>
Re: flink loop
Posted by tanguy racinet <ta...@gmail.com>.
Hi,
Thank you, that is what we were thinking about but we are working with
Strings and not int inside permutations.
So we are wondering if the ordering wouldn't be too expensive
ᐧ
<http://eitictlabs-rennes.fr/>
*Racinet Tanguy*
*EIT ICT Labs Master School Student*
*Distributed Systems and Services*
Tel : +33 6 63 20 89 16 / +49 176 3749 8854
Mail : tanracinet@gmail.com
On Sun, Feb 8, 2015 at 1:07 PM, Till Rohrmann <tr...@apache.org> wrote:
> Hi,
>
> you could apply a filter operation after the cross operation which filters
> all combinations out which are not in ascending order.
>
> Cheers,
>
> Till
>
> On Sun, Feb 8, 2015 at 12:38 PM, tanguy racinet <ta...@gmail.com>
> wrote:
>
>> Hi,
>>
>> Thank you for you reply. It helped us solve the looping problems in a
>> nicer way.
>>
>> We are struggling with some aspects of the cross function.
>> Still trying to implement the Apriori algorithm, we need to create
>> combinations of frequent itemSets.
>> Our problem is that the crossing gives us duplicates, for instance :(1,
>> 2, 3, 4) and (2, 1, 4, 3) are equivalent for us so we are trying to find a
>> way to remove that kind of duplicate in our DataSet.
>>
>> We already removed duplicates inside our combinations (1, 1, 2) => (1, 2).
>>
>> We were thinking about using HashSet but they are not serializable and we
>> cannot use them inside the workflow, but only inside functions.
>>
>> Can you think of any way to remove those duplicates ?
>>
>> Thank you,
>> ᐧ
>>
>> <http://eitictlabs-rennes.fr/>
>>
>>
>> *Racinet Tanguy*
>>
>> *EIT ICT Labs Master School Student*
>> *Distributed Systems and Services*
>>
>> Tel : +33 6 63 20 89 16 / +49 176 3749 8854
>> Mail : tanracinet@gmail.com
>>
>> On Thu, Feb 5, 2015 at 8:51 PM, Vasiliki Kalavri <
>> vasilikikalavri@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I'm not familiar with the particular algorithm, but you can most
>>> probably use one of the two iterate operators in Flink.
>>>
>>> You can read a description and see some examples in the documentation:
>>>
>>> http://flink.apache.org/docs/0.8/programming_guide.html#iteration-operators
>>>
>>> Let us know if you have any questions!
>>>
>>> Cheers,
>>> V.
>>>
>>> On 5 February 2015 at 20:37, tanguy racinet <ta...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> We are trying to develop the Apriori algorith with the Flink for our
>>>> Data minning project.
>>>> In our understanding, Flink could handle loop within the workflow.
>>>> However, our knowledge is limited and we cannot find a nice way to do
>>>> it.
>>>>
>>>> Here is the flow of my algorithm :
>>>> GenerateCandidates ----> CalculateFrequentItemSet
>>>> mapper ----> reducer
>>>>
>>>> We would like to use the reducer result as the mapper's entry for a
>>>> predefined number of times (loop x times).
>>>>
>>>> Is there any smart way to that with Flink. Or should we just copy paste
>>>> the loop x times ?
>>>>
>>>> Thank you,
>>>> <http://eitictlabs-rennes.fr/>
>>>>
>>>>
>>>> *Racinet Tanguy*
>>>>
>>>> *EIT ICT Labs Master School Student*
>>>> *Distributed Systems and Services*
>>>>
>>>> Tel : +33 6 63 20 89 16 / +49 176 3749 8854
>>>> Mail : tanracinet@gmail.com
>>>>
>>>> ᐧ
>>>>
>>>
>>>
>>
>
Re: flink loop
Posted by Till Rohrmann <tr...@apache.org>.
Hi,
you could apply a filter operation after the cross operation which filters
all combinations out which are not in ascending order.
Cheers,
Till
On Sun, Feb 8, 2015 at 12:38 PM, tanguy racinet <ta...@gmail.com>
wrote:
> Hi,
>
> Thank you for you reply. It helped us solve the looping problems in a
> nicer way.
>
> We are struggling with some aspects of the cross function.
> Still trying to implement the Apriori algorithm, we need to create
> combinations of frequent itemSets.
> Our problem is that the crossing gives us duplicates, for instance :(1, 2,
> 3, 4) and (2, 1, 4, 3) are equivalent for us so we are trying to find a way
> to remove that kind of duplicate in our DataSet.
>
> We already removed duplicates inside our combinations (1, 1, 2) => (1, 2).
>
> We were thinking about using HashSet but they are not serializable and we
> cannot use them inside the workflow, but only inside functions.
>
> Can you think of any way to remove those duplicates ?
>
> Thank you,
> ᐧ
>
> <http://eitictlabs-rennes.fr/>
>
>
> *Racinet Tanguy*
>
> *EIT ICT Labs Master School Student*
> *Distributed Systems and Services*
>
> Tel : +33 6 63 20 89 16 / +49 176 3749 8854
> Mail : tanracinet@gmail.com
>
> On Thu, Feb 5, 2015 at 8:51 PM, Vasiliki Kalavri <
> vasilikikalavri@gmail.com> wrote:
>
>> Hi,
>>
>> I'm not familiar with the particular algorithm, but you can most probably
>> use one of the two iterate operators in Flink.
>>
>> You can read a description and see some examples in the documentation:
>>
>> http://flink.apache.org/docs/0.8/programming_guide.html#iteration-operators
>>
>> Let us know if you have any questions!
>>
>> Cheers,
>> V.
>>
>> On 5 February 2015 at 20:37, tanguy racinet <ta...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> We are trying to develop the Apriori algorith with the Flink for our
>>> Data minning project.
>>> In our understanding, Flink could handle loop within the workflow.
>>> However, our knowledge is limited and we cannot find a nice way to do it.
>>>
>>> Here is the flow of my algorithm :
>>> GenerateCandidates ----> CalculateFrequentItemSet
>>> mapper ----> reducer
>>>
>>> We would like to use the reducer result as the mapper's entry for a
>>> predefined number of times (loop x times).
>>>
>>> Is there any smart way to that with Flink. Or should we just copy paste
>>> the loop x times ?
>>>
>>> Thank you,
>>> <http://eitictlabs-rennes.fr/>
>>>
>>>
>>> *Racinet Tanguy*
>>>
>>> *EIT ICT Labs Master School Student*
>>> *Distributed Systems and Services*
>>>
>>> Tel : +33 6 63 20 89 16 / +49 176 3749 8854
>>> Mail : tanracinet@gmail.com
>>>
>>> ᐧ
>>>
>>
>>
>
Re: flink loop
Posted by tanguy racinet <ta...@gmail.com>.
Hi,
Thank you for you reply. It helped us solve the looping problems in a nicer
way.
We are struggling with some aspects of the cross function.
Still trying to implement the Apriori algorithm, we need to create
combinations of frequent itemSets.
Our problem is that the crossing gives us duplicates, for instance :(1, 2,
3, 4) and (2, 1, 4, 3) are equivalent for us so we are trying to find a way
to remove that kind of duplicate in our DataSet.
We already removed duplicates inside our combinations (1, 1, 2) => (1, 2).
We were thinking about using HashSet but they are not serializable and we
cannot use them inside the workflow, but only inside functions.
Can you think of any way to remove those duplicates ?
Thank you,
ᐧ
<http://eitictlabs-rennes.fr/>
*Racinet Tanguy*
*EIT ICT Labs Master School Student*
*Distributed Systems and Services*
Tel : +33 6 63 20 89 16 / +49 176 3749 8854
Mail : tanracinet@gmail.com
On Thu, Feb 5, 2015 at 8:51 PM, Vasiliki Kalavri <va...@gmail.com>
wrote:
> Hi,
>
> I'm not familiar with the particular algorithm, but you can most probably
> use one of the two iterate operators in Flink.
>
> You can read a description and see some examples in the documentation:
> http://flink.apache.org/docs/0.8/programming_guide.html#iteration-operators
>
> Let us know if you have any questions!
>
> Cheers,
> V.
>
> On 5 February 2015 at 20:37, tanguy racinet <ta...@gmail.com> wrote:
>
>> Hi,
>>
>> We are trying to develop the Apriori algorith with the Flink for our Data
>> minning project.
>> In our understanding, Flink could handle loop within the workflow.
>> However, our knowledge is limited and we cannot find a nice way to do it.
>>
>> Here is the flow of my algorithm :
>> GenerateCandidates ----> CalculateFrequentItemSet
>> mapper ----> reducer
>>
>> We would like to use the reducer result as the mapper's entry for a
>> predefined number of times (loop x times).
>>
>> Is there any smart way to that with Flink. Or should we just copy paste
>> the loop x times ?
>>
>> Thank you,
>> <http://eitictlabs-rennes.fr/>
>>
>>
>> *Racinet Tanguy*
>>
>> *EIT ICT Labs Master School Student*
>> *Distributed Systems and Services*
>>
>> Tel : +33 6 63 20 89 16 / +49 176 3749 8854
>> Mail : tanracinet@gmail.com
>>
>> ᐧ
>>
>
>
Re: flink loop
Posted by Vasiliki Kalavri <va...@gmail.com>.
Hi,
I'm not familiar with the particular algorithm, but you can most probably
use one of the two iterate operators in Flink.
You can read a description and see some examples in the documentation:
http://flink.apache.org/docs/0.8/programming_guide.html#iteration-operators
Let us know if you have any questions!
Cheers,
V.
On 5 February 2015 at 20:37, tanguy racinet <ta...@gmail.com> wrote:
> Hi,
>
> We are trying to develop the Apriori algorith with the Flink for our Data
> minning project.
> In our understanding, Flink could handle loop within the workflow.
> However, our knowledge is limited and we cannot find a nice way to do it.
>
> Here is the flow of my algorithm :
> GenerateCandidates ----> CalculateFrequentItemSet
> mapper ----> reducer
>
> We would like to use the reducer result as the mapper's entry for a
> predefined number of times (loop x times).
>
> Is there any smart way to that with Flink. Or should we just copy paste
> the loop x times ?
>
> Thank you,
> <http://eitictlabs-rennes.fr/>
>
>
> *Racinet Tanguy*
>
> *EIT ICT Labs Master School Student*
> *Distributed Systems and Services*
>
> Tel : +33 6 63 20 89 16 / +49 176 3749 8854
> Mail : tanracinet@gmail.com
>
> ᐧ
>