You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Andra Lungu <lu...@gmail.com> on 2015/03/04 16:31:12 UTC
[MultipleProgramsTestBase][Cluster vs. Collection mode] Inconsistent Behavior
Hello,
I have implemented a Bulk Synchronous Version of Triangle Count. The code
can be found here:
https://github.com/andralungu/gelly-partitioning/tree/triangles
In this algorithm, the messages sent differ as the superstep differs. In
order to distinguish between superstep numbers, I used the
getSuperstepNumber() function.
In order to test the overall implementation, I have extended
MultipleProgramsTestBase... nothing unusual until here. The problem is that
in CLUSTER mode, the test passes and the result is the one expected because
the superstep number changes, as can be seen below:
[Messenger]Step 1
[Messenger]Step 1
[Messenger]Step 1
[Messenger]Step 1
[Messenger]Step 1
[Messenger]Step 1
[Messenger]Step 1
[Update]Step 1
[Update]Step 1
[Update]Step 1
[Update]Step 1
[Update]Step 1
[Update]Step 1
[Messenger]Step 2
[Messenger]Step 2
[Messenger]Step 2
[Messenger]Step 2
[Messenger]Step 2
[Messenger]Step 2
[Update]Step 2
[Update]Step 2
[Update]Step 2
[Update]Step 2
[Update]Step 2
[Messenger]Step 3
[Messenger]Step 3
[Messenger]Step 3
[Messenger]Step 3
[Messenger]Step 3
[Update]Step 3
[Update]Step 3
[Update]Step 3
For COLLECTION, the superstep number remains 1, and the result is obviously
not the one I expected.
[Messenger]Step 1
[Messenger]Step 1
[Messenger]Step 1
[Messenger]Step 1
[Messenger]Step 1
[Messenger]Step 1
[Messenger]Step 1
[Update]Step 1
[Update]Step 1
[Update]Step 1
[Update]Step 1
[Update]Step 1
[Update]Step 1
[Update]Step 1
[Messenger]Step 1
[Messenger]Step 1
[Messenger]Step 1
[Messenger]Step 1
[Messenger]Step 1
[Messenger]Step 1
[Messenger]Step 1
[Update]Step 1
[Update]Step 1
[Update]Step 1
[Update]Step 1
[Update]Step 1
[Update]Step 1
[Update]Step 1
[Messenger]Step 1
[Messenger]Step 1
[Messenger]Step 1
[Messenger]Step 1
[Messenger]Step 1
[Messenger]Step 1
[Messenger]Step 1
[Update]Step 1
[Update]Step 1
[Update]Step 1
[Update]Step 1
[Update]Step 1
[Update]Step 1
[Update]Step 1
Does anyone have an idea what could have triggered this behaviour?
Thanks in advance!
Andra
Re: [MultipleProgramsTestBase][Cluster vs. Collection mode]
Inconsistent Behavior
Posted by Vasiliki Kalavri <va...@gmail.com>.
Thanks!
I created an issue [1] and will try to look into this during the weekend
(unless someone gets to it faster :))
Cheers,
V.
[1]: http://issues.apache.org/jira/browse/FLINK-1652
On 4 March 2015 at 20:29, Stephan Ewen <se...@apache.org> wrote:
> My gut feeling would be that the IterationRuntimeContext should be the same
> across all iterations.
>
> It only needs to support returning a different superstep number in each
> superstep.
>
> On Wed, Mar 4, 2015 at 7:13 PM, Vasiliki Kalavri <
> vasilikikalavri@gmail.com>
> wrote:
>
> > Hi,
> >
> > I have found the issue, but will need some help to resolve it :)
> >
> > In CollectionExecutor, if the superstep number is > 0 (I guess this means
> > iteration?)
> > a new IterationRuntimeUDFContext is created for every unary and binary
> > operator, being assigned this superstep number in its constructor.
> >
> > However, in VertexCentricIteration, MessagingFunction and
> > VertexUpdateFunction are assigned a unique IterationRuntimeContext in the
> > open method of their wrappers, like this:
> >
> > public void open(Configuration parameters) throws Exception {
> > if (getIterationRuntimeContext().getSuperstepNumber() == 1) {
> > this.vertexUpdateFunction.init(getIterationRuntimeContext());
> > }
> > }
> >
> > There is no comment, but I suppose the above was written having something
> > in mind that I'm not aware of...
> > Is the IterationRuntimeContext supposed to be unique and the
> > CollectionExecutor has to make sure to update the superstep number
> > accordingly (like it does with the aggregators) or shall we assign the
> new
> > context in every superstep?
> >
> > Thanks!
> >
> > -Vasia.
> >
> > On 4 March 2015 at 17:46, Vasiliki Kalavri <va...@gmail.com>
> > wrote:
> >
> > > Hi Andra,
> > >
> > > judging from the output, it seems that all 3 supersteps are executed in
> > > the second case as well,
> > > but getSuperstepNumber() is returning the wrong superstep number.
> > > I confirmed that this is also the case
> > > in VertexCentricConnectedComponentsITCase
> > > and SpargelConnectedComponentsITCase, i.e. the superstep number is
> wrong,
> > > but the results produced are correct.
> > > I'll try to find out what's wrong.
> > >
> > > -V.
> > >
> > > On 4 March 2015 at 16:31, Andra Lungu <lu...@gmail.com> wrote:
> > >
> > >> Hello,
> > >>
> > >> I have implemented a Bulk Synchronous Version of Triangle Count. The
> > code
> > >> can be found here:
> > >> https://github.com/andralungu/gelly-partitioning/tree/triangles
> > >>
> > >> In this algorithm, the messages sent differ as the superstep differs.
> In
> > >> order to distinguish between superstep numbers, I used the
> > >> getSuperstepNumber() function.
> > >>
> > >> In order to test the overall implementation, I have extended
> > >> MultipleProgramsTestBase... nothing unusual until here. The problem is
> > >> that
> > >> in CLUSTER mode, the test passes and the result is the one expected
> > >> because
> > >> the superstep number changes, as can be seen below:
> > >> [Messenger]Step 1
> > >> [Messenger]Step 1
> > >> [Messenger]Step 1
> > >> [Messenger]Step 1
> > >> [Messenger]Step 1
> > >> [Messenger]Step 1
> > >> [Messenger]Step 1
> > >> [Update]Step 1
> > >> [Update]Step 1
> > >> [Update]Step 1
> > >> [Update]Step 1
> > >> [Update]Step 1
> > >> [Update]Step 1
> > >> [Messenger]Step 2
> > >> [Messenger]Step 2
> > >> [Messenger]Step 2
> > >> [Messenger]Step 2
> > >> [Messenger]Step 2
> > >> [Messenger]Step 2
> > >> [Update]Step 2
> > >> [Update]Step 2
> > >> [Update]Step 2
> > >> [Update]Step 2
> > >> [Update]Step 2
> > >> [Messenger]Step 3
> > >> [Messenger]Step 3
> > >> [Messenger]Step 3
> > >> [Messenger]Step 3
> > >> [Messenger]Step 3
> > >> [Update]Step 3
> > >> [Update]Step 3
> > >> [Update]Step 3
> > >>
> > >> For COLLECTION, the superstep number remains 1, and the result is
> > >> obviously
> > >> not the one I expected.
> > >> [Messenger]Step 1
> > >> [Messenger]Step 1
> > >> [Messenger]Step 1
> > >> [Messenger]Step 1
> > >> [Messenger]Step 1
> > >> [Messenger]Step 1
> > >> [Messenger]Step 1
> > >> [Update]Step 1
> > >> [Update]Step 1
> > >> [Update]Step 1
> > >> [Update]Step 1
> > >> [Update]Step 1
> > >> [Update]Step 1
> > >> [Update]Step 1
> > >> [Messenger]Step 1
> > >> [Messenger]Step 1
> > >> [Messenger]Step 1
> > >> [Messenger]Step 1
> > >> [Messenger]Step 1
> > >> [Messenger]Step 1
> > >> [Messenger]Step 1
> > >> [Update]Step 1
> > >> [Update]Step 1
> > >> [Update]Step 1
> > >> [Update]Step 1
> > >> [Update]Step 1
> > >> [Update]Step 1
> > >> [Update]Step 1
> > >> [Messenger]Step 1
> > >> [Messenger]Step 1
> > >> [Messenger]Step 1
> > >> [Messenger]Step 1
> > >> [Messenger]Step 1
> > >> [Messenger]Step 1
> > >> [Messenger]Step 1
> > >> [Update]Step 1
> > >> [Update]Step 1
> > >> [Update]Step 1
> > >> [Update]Step 1
> > >> [Update]Step 1
> > >> [Update]Step 1
> > >> [Update]Step 1
> > >>
> > >> Does anyone have an idea what could have triggered this behaviour?
> > >>
> > >> Thanks in advance!
> > >> Andra
> > >>
> > >
> > >
> >
>
Re: [MultipleProgramsTestBase][Cluster vs. Collection mode]
Inconsistent Behavior
Posted by Stephan Ewen <se...@apache.org>.
My gut feeling would be that the IterationRuntimeContext should be the same
across all iterations.
It only needs to support returning a different superstep number in each
superstep.
On Wed, Mar 4, 2015 at 7:13 PM, Vasiliki Kalavri <va...@gmail.com>
wrote:
> Hi,
>
> I have found the issue, but will need some help to resolve it :)
>
> In CollectionExecutor, if the superstep number is > 0 (I guess this means
> iteration?)
> a new IterationRuntimeUDFContext is created for every unary and binary
> operator, being assigned this superstep number in its constructor.
>
> However, in VertexCentricIteration, MessagingFunction and
> VertexUpdateFunction are assigned a unique IterationRuntimeContext in the
> open method of their wrappers, like this:
>
> public void open(Configuration parameters) throws Exception {
> if (getIterationRuntimeContext().getSuperstepNumber() == 1) {
> this.vertexUpdateFunction.init(getIterationRuntimeContext());
> }
> }
>
> There is no comment, but I suppose the above was written having something
> in mind that I'm not aware of...
> Is the IterationRuntimeContext supposed to be unique and the
> CollectionExecutor has to make sure to update the superstep number
> accordingly (like it does with the aggregators) or shall we assign the new
> context in every superstep?
>
> Thanks!
>
> -Vasia.
>
> On 4 March 2015 at 17:46, Vasiliki Kalavri <va...@gmail.com>
> wrote:
>
> > Hi Andra,
> >
> > judging from the output, it seems that all 3 supersteps are executed in
> > the second case as well,
> > but getSuperstepNumber() is returning the wrong superstep number.
> > I confirmed that this is also the case
> > in VertexCentricConnectedComponentsITCase
> > and SpargelConnectedComponentsITCase, i.e. the superstep number is wrong,
> > but the results produced are correct.
> > I'll try to find out what's wrong.
> >
> > -V.
> >
> > On 4 March 2015 at 16:31, Andra Lungu <lu...@gmail.com> wrote:
> >
> >> Hello,
> >>
> >> I have implemented a Bulk Synchronous Version of Triangle Count. The
> code
> >> can be found here:
> >> https://github.com/andralungu/gelly-partitioning/tree/triangles
> >>
> >> In this algorithm, the messages sent differ as the superstep differs. In
> >> order to distinguish between superstep numbers, I used the
> >> getSuperstepNumber() function.
> >>
> >> In order to test the overall implementation, I have extended
> >> MultipleProgramsTestBase... nothing unusual until here. The problem is
> >> that
> >> in CLUSTER mode, the test passes and the result is the one expected
> >> because
> >> the superstep number changes, as can be seen below:
> >> [Messenger]Step 1
> >> [Messenger]Step 1
> >> [Messenger]Step 1
> >> [Messenger]Step 1
> >> [Messenger]Step 1
> >> [Messenger]Step 1
> >> [Messenger]Step 1
> >> [Update]Step 1
> >> [Update]Step 1
> >> [Update]Step 1
> >> [Update]Step 1
> >> [Update]Step 1
> >> [Update]Step 1
> >> [Messenger]Step 2
> >> [Messenger]Step 2
> >> [Messenger]Step 2
> >> [Messenger]Step 2
> >> [Messenger]Step 2
> >> [Messenger]Step 2
> >> [Update]Step 2
> >> [Update]Step 2
> >> [Update]Step 2
> >> [Update]Step 2
> >> [Update]Step 2
> >> [Messenger]Step 3
> >> [Messenger]Step 3
> >> [Messenger]Step 3
> >> [Messenger]Step 3
> >> [Messenger]Step 3
> >> [Update]Step 3
> >> [Update]Step 3
> >> [Update]Step 3
> >>
> >> For COLLECTION, the superstep number remains 1, and the result is
> >> obviously
> >> not the one I expected.
> >> [Messenger]Step 1
> >> [Messenger]Step 1
> >> [Messenger]Step 1
> >> [Messenger]Step 1
> >> [Messenger]Step 1
> >> [Messenger]Step 1
> >> [Messenger]Step 1
> >> [Update]Step 1
> >> [Update]Step 1
> >> [Update]Step 1
> >> [Update]Step 1
> >> [Update]Step 1
> >> [Update]Step 1
> >> [Update]Step 1
> >> [Messenger]Step 1
> >> [Messenger]Step 1
> >> [Messenger]Step 1
> >> [Messenger]Step 1
> >> [Messenger]Step 1
> >> [Messenger]Step 1
> >> [Messenger]Step 1
> >> [Update]Step 1
> >> [Update]Step 1
> >> [Update]Step 1
> >> [Update]Step 1
> >> [Update]Step 1
> >> [Update]Step 1
> >> [Update]Step 1
> >> [Messenger]Step 1
> >> [Messenger]Step 1
> >> [Messenger]Step 1
> >> [Messenger]Step 1
> >> [Messenger]Step 1
> >> [Messenger]Step 1
> >> [Messenger]Step 1
> >> [Update]Step 1
> >> [Update]Step 1
> >> [Update]Step 1
> >> [Update]Step 1
> >> [Update]Step 1
> >> [Update]Step 1
> >> [Update]Step 1
> >>
> >> Does anyone have an idea what could have triggered this behaviour?
> >>
> >> Thanks in advance!
> >> Andra
> >>
> >
> >
>
Re: [MultipleProgramsTestBase][Cluster vs. Collection mode]
Inconsistent Behavior
Posted by Vasiliki Kalavri <va...@gmail.com>.
Hi,
I have found the issue, but will need some help to resolve it :)
In CollectionExecutor, if the superstep number is > 0 (I guess this means
iteration?)
a new IterationRuntimeUDFContext is created for every unary and binary
operator, being assigned this superstep number in its constructor.
However, in VertexCentricIteration, MessagingFunction and
VertexUpdateFunction are assigned a unique IterationRuntimeContext in the
open method of their wrappers, like this:
public void open(Configuration parameters) throws Exception {
if (getIterationRuntimeContext().getSuperstepNumber() == 1) {
this.vertexUpdateFunction.init(getIterationRuntimeContext());
}
}
There is no comment, but I suppose the above was written having something
in mind that I'm not aware of...
Is the IterationRuntimeContext supposed to be unique and the
CollectionExecutor has to make sure to update the superstep number
accordingly (like it does with the aggregators) or shall we assign the new
context in every superstep?
Thanks!
-Vasia.
On 4 March 2015 at 17:46, Vasiliki Kalavri <va...@gmail.com>
wrote:
> Hi Andra,
>
> judging from the output, it seems that all 3 supersteps are executed in
> the second case as well,
> but getSuperstepNumber() is returning the wrong superstep number.
> I confirmed that this is also the case
> in VertexCentricConnectedComponentsITCase
> and SpargelConnectedComponentsITCase, i.e. the superstep number is wrong,
> but the results produced are correct.
> I'll try to find out what's wrong.
>
> -V.
>
> On 4 March 2015 at 16:31, Andra Lungu <lu...@gmail.com> wrote:
>
>> Hello,
>>
>> I have implemented a Bulk Synchronous Version of Triangle Count. The code
>> can be found here:
>> https://github.com/andralungu/gelly-partitioning/tree/triangles
>>
>> In this algorithm, the messages sent differ as the superstep differs. In
>> order to distinguish between superstep numbers, I used the
>> getSuperstepNumber() function.
>>
>> In order to test the overall implementation, I have extended
>> MultipleProgramsTestBase... nothing unusual until here. The problem is
>> that
>> in CLUSTER mode, the test passes and the result is the one expected
>> because
>> the superstep number changes, as can be seen below:
>> [Messenger]Step 1
>> [Messenger]Step 1
>> [Messenger]Step 1
>> [Messenger]Step 1
>> [Messenger]Step 1
>> [Messenger]Step 1
>> [Messenger]Step 1
>> [Update]Step 1
>> [Update]Step 1
>> [Update]Step 1
>> [Update]Step 1
>> [Update]Step 1
>> [Update]Step 1
>> [Messenger]Step 2
>> [Messenger]Step 2
>> [Messenger]Step 2
>> [Messenger]Step 2
>> [Messenger]Step 2
>> [Messenger]Step 2
>> [Update]Step 2
>> [Update]Step 2
>> [Update]Step 2
>> [Update]Step 2
>> [Update]Step 2
>> [Messenger]Step 3
>> [Messenger]Step 3
>> [Messenger]Step 3
>> [Messenger]Step 3
>> [Messenger]Step 3
>> [Update]Step 3
>> [Update]Step 3
>> [Update]Step 3
>>
>> For COLLECTION, the superstep number remains 1, and the result is
>> obviously
>> not the one I expected.
>> [Messenger]Step 1
>> [Messenger]Step 1
>> [Messenger]Step 1
>> [Messenger]Step 1
>> [Messenger]Step 1
>> [Messenger]Step 1
>> [Messenger]Step 1
>> [Update]Step 1
>> [Update]Step 1
>> [Update]Step 1
>> [Update]Step 1
>> [Update]Step 1
>> [Update]Step 1
>> [Update]Step 1
>> [Messenger]Step 1
>> [Messenger]Step 1
>> [Messenger]Step 1
>> [Messenger]Step 1
>> [Messenger]Step 1
>> [Messenger]Step 1
>> [Messenger]Step 1
>> [Update]Step 1
>> [Update]Step 1
>> [Update]Step 1
>> [Update]Step 1
>> [Update]Step 1
>> [Update]Step 1
>> [Update]Step 1
>> [Messenger]Step 1
>> [Messenger]Step 1
>> [Messenger]Step 1
>> [Messenger]Step 1
>> [Messenger]Step 1
>> [Messenger]Step 1
>> [Messenger]Step 1
>> [Update]Step 1
>> [Update]Step 1
>> [Update]Step 1
>> [Update]Step 1
>> [Update]Step 1
>> [Update]Step 1
>> [Update]Step 1
>>
>> Does anyone have an idea what could have triggered this behaviour?
>>
>> Thanks in advance!
>> Andra
>>
>
>
Re: [MultipleProgramsTestBase][Cluster vs. Collection mode]
Inconsistent Behavior
Posted by Vasiliki Kalavri <va...@gmail.com>.
Hi Andra,
judging from the output, it seems that all 3 supersteps are executed in the
second case as well,
but getSuperstepNumber() is returning the wrong superstep number.
I confirmed that this is also the case
in VertexCentricConnectedComponentsITCase
and SpargelConnectedComponentsITCase, i.e. the superstep number is wrong,
but the results produced are correct.
I'll try to find out what's wrong.
-V.
On 4 March 2015 at 16:31, Andra Lungu <lu...@gmail.com> wrote:
> Hello,
>
> I have implemented a Bulk Synchronous Version of Triangle Count. The code
> can be found here:
> https://github.com/andralungu/gelly-partitioning/tree/triangles
>
> In this algorithm, the messages sent differ as the superstep differs. In
> order to distinguish between superstep numbers, I used the
> getSuperstepNumber() function.
>
> In order to test the overall implementation, I have extended
> MultipleProgramsTestBase... nothing unusual until here. The problem is that
> in CLUSTER mode, the test passes and the result is the one expected because
> the superstep number changes, as can be seen below:
> [Messenger]Step 1
> [Messenger]Step 1
> [Messenger]Step 1
> [Messenger]Step 1
> [Messenger]Step 1
> [Messenger]Step 1
> [Messenger]Step 1
> [Update]Step 1
> [Update]Step 1
> [Update]Step 1
> [Update]Step 1
> [Update]Step 1
> [Update]Step 1
> [Messenger]Step 2
> [Messenger]Step 2
> [Messenger]Step 2
> [Messenger]Step 2
> [Messenger]Step 2
> [Messenger]Step 2
> [Update]Step 2
> [Update]Step 2
> [Update]Step 2
> [Update]Step 2
> [Update]Step 2
> [Messenger]Step 3
> [Messenger]Step 3
> [Messenger]Step 3
> [Messenger]Step 3
> [Messenger]Step 3
> [Update]Step 3
> [Update]Step 3
> [Update]Step 3
>
> For COLLECTION, the superstep number remains 1, and the result is obviously
> not the one I expected.
> [Messenger]Step 1
> [Messenger]Step 1
> [Messenger]Step 1
> [Messenger]Step 1
> [Messenger]Step 1
> [Messenger]Step 1
> [Messenger]Step 1
> [Update]Step 1
> [Update]Step 1
> [Update]Step 1
> [Update]Step 1
> [Update]Step 1
> [Update]Step 1
> [Update]Step 1
> [Messenger]Step 1
> [Messenger]Step 1
> [Messenger]Step 1
> [Messenger]Step 1
> [Messenger]Step 1
> [Messenger]Step 1
> [Messenger]Step 1
> [Update]Step 1
> [Update]Step 1
> [Update]Step 1
> [Update]Step 1
> [Update]Step 1
> [Update]Step 1
> [Update]Step 1
> [Messenger]Step 1
> [Messenger]Step 1
> [Messenger]Step 1
> [Messenger]Step 1
> [Messenger]Step 1
> [Messenger]Step 1
> [Messenger]Step 1
> [Update]Step 1
> [Update]Step 1
> [Update]Step 1
> [Update]Step 1
> [Update]Step 1
> [Update]Step 1
> [Update]Step 1
>
> Does anyone have an idea what could have triggered this behaviour?
>
> Thanks in advance!
> Andra
>