You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Andra Lungu <lu...@gmail.com> on 2015/07/24 18:56:43 UTC
A soft reminder
Hi guys,
I opened these JIRAs a while ago and, unfortunately, no one had time to
look at them :|.
[1] https://issues.apache.org/jira/browse/FLINK-2361
[2] https://issues.apache.org/jira/browse/FLINK-2360
[1] could probably be hacked, but [2] is a huge problem that was
encountered by other people outside of Gelly.
Thanks for the help!
Andra
Re: A soft reminder
Posted by Stephan Ewen <se...@apache.org>.
I think I have found two problems in the hash table. One may be the loss of
records, the other one the EOF.
Give me a bit to confirm, test, and prepare a patch.
On Thu, Jul 30, 2015 at 7:46 PM, Gábor Gévay <gg...@gmail.com> wrote:
> It is working with setSolutionSetUnmanagedMemory(true), thanks!
>
> Gabor
>
>
>
> 2015-07-30 19:23 GMT+02:00 Andra Lungu <lu...@gmail.com>:
> > Could you try adding the following lines to your code?
> >
> > VertexCentricConfiguration parameters = new VertexCentricConfiguration();
> > parameters.setSolutionSetUnmanagedMemory(true);
> >
> > return graphWithScoredVertices.runVertexCentricIteration(new
> > VertexLabelUpdater(delta),
> > new LabelMessenger(), maxIterations, parameters);
> >
> > Change the functions appropriately in the runVertexCentricIteration call.
> > The error seems to be caused by this CompactingHashTable and this
> parameter
> > deactivates its usage. I am very curious what happens in your case. If
> you
> > could tell us the outcome, I'd greately appreciate it.
> >
> >
> > On Thu, Jul 30, 2015 at 7:17 PM, Gábor Gévay <gg...@gmail.com> wrote:
> >
> >> Yes, in a VertexCentricIteration with a few million nodes, running
> >> locally on my laptop with about 10 GB of memory given to java.
> >>
> >> Best,
> >> Gabor
> >>
> >>
> >>
> >> 2015-07-30 18:32 GMT+02:00 Andra Lungu <lu...@gmail.com>:
> >> > Hi Gabor,
> >> >
> >> > Within a delta iteration right?
> >> >
> >> > On Thu, Jul 30, 2015 at 6:31 PM, Gábor Gévay <gg...@gmail.com>
> wrote:
> >> >
> >> >> Hi,
> >> >>
> >> >> I have also run into this problem just now. It only happens with much
> >> data.
> >> >>
> >> >> Best regards,
> >> >> Gabor
> >> >>
> >> >>
> >> >>
> >> >> 2015-07-27 11:35 GMT+02:00 Felix Neutatz <ne...@googlemail.com>:
> >> >> > Hi,
> >> >> >
> >> >> > I also encountered the EOF exception for a delta iteration with
> "more
> >> >> > data". With less data it works ...
> >> >> >
> >> >> > Best regards,
> >> >> > Felix
> >> >> > Am 27.07.2015 10:25 vorm. schrieb "Andra Lungu" <
> >> lungu.andra@gmail.com>:
> >> >> >
> >> >> >> Hi Stephan,
> >> >> >>
> >> >> >> I tried to debug a bit around the EOF Exception. It seems that I
> am
> >> >> pretty
> >> >> >> useless on my own. I have no idea how to fix it. And it occurs in
> >> quite
> >> >> a
> >> >> >> lot of experiments... I believe it's something behind a delta
> >> iteration
> >> >> >> [this is the common point between my experiments and the others'
> >> code].
> >> >> >>
> >> >> >> Some pointers would be greatly appreciated :)
> >> >> >> Thanks!
> >> >> >> Andra
> >> >> >>
> >> >> >> On Mon, Jul 27, 2015 at 9:32 AM, Stephan Ewen <se...@apache.org>
> >> wrote:
> >> >> >>
> >> >> >> > Hey!
> >> >> >> >
> >> >> >> > It is pretty much as you said: No one had the time to fix this
> so
> >> far.
> >> >> >> They
> >> >> >> > are on the list, though,
> >> >> >> >
> >> >> >> > Do you want to pick up one of them?
> >> >> >> >
> >> >> >> > Greetings,
> >> >> >> > Stephan
> >> >> >> >
> >> >> >> >
> >> >> >> > On Fri, Jul 24, 2015 at 6:56 PM, Andra Lungu <
> >> lungu.andra@gmail.com>
> >> >> >> > wrote:
> >> >> >> >
> >> >> >> > > Hi guys,
> >> >> >> > >
> >> >> >> > > I opened these JIRAs a while ago and, unfortunately, no one
> had
> >> >> time to
> >> >> >> > > look at them :|.
> >> >> >> > >
> >> >> >> > > [1] https://issues.apache.org/jira/browse/FLINK-2361
> >> >> >> > > [2] https://issues.apache.org/jira/browse/FLINK-2360
> >> >> >> > >
> >> >> >> > > [1] could probably be hacked, but [2] is a huge problem that
> was
> >> >> >> > > encountered by other people outside of Gelly.
> >> >> >> > >
> >> >> >> > > Thanks for the help!
> >> >> >> > > Andra
> >> >> >> > >
> >> >> >> >
> >> >> >>
> >> >>
> >>
>
Re: A soft reminder
Posted by Gábor Gévay <gg...@gmail.com>.
It is working with setSolutionSetUnmanagedMemory(true), thanks!
Gabor
2015-07-30 19:23 GMT+02:00 Andra Lungu <lu...@gmail.com>:
> Could you try adding the following lines to your code?
>
> VertexCentricConfiguration parameters = new VertexCentricConfiguration();
> parameters.setSolutionSetUnmanagedMemory(true);
>
> return graphWithScoredVertices.runVertexCentricIteration(new
> VertexLabelUpdater(delta),
> new LabelMessenger(), maxIterations, parameters);
>
> Change the functions appropriately in the runVertexCentricIteration call.
> The error seems to be caused by this CompactingHashTable and this parameter
> deactivates its usage. I am very curious what happens in your case. If you
> could tell us the outcome, I'd greately appreciate it.
>
>
> On Thu, Jul 30, 2015 at 7:17 PM, Gábor Gévay <gg...@gmail.com> wrote:
>
>> Yes, in a VertexCentricIteration with a few million nodes, running
>> locally on my laptop with about 10 GB of memory given to java.
>>
>> Best,
>> Gabor
>>
>>
>>
>> 2015-07-30 18:32 GMT+02:00 Andra Lungu <lu...@gmail.com>:
>> > Hi Gabor,
>> >
>> > Within a delta iteration right?
>> >
>> > On Thu, Jul 30, 2015 at 6:31 PM, Gábor Gévay <gg...@gmail.com> wrote:
>> >
>> >> Hi,
>> >>
>> >> I have also run into this problem just now. It only happens with much
>> data.
>> >>
>> >> Best regards,
>> >> Gabor
>> >>
>> >>
>> >>
>> >> 2015-07-27 11:35 GMT+02:00 Felix Neutatz <ne...@googlemail.com>:
>> >> > Hi,
>> >> >
>> >> > I also encountered the EOF exception for a delta iteration with "more
>> >> > data". With less data it works ...
>> >> >
>> >> > Best regards,
>> >> > Felix
>> >> > Am 27.07.2015 10:25 vorm. schrieb "Andra Lungu" <
>> lungu.andra@gmail.com>:
>> >> >
>> >> >> Hi Stephan,
>> >> >>
>> >> >> I tried to debug a bit around the EOF Exception. It seems that I am
>> >> pretty
>> >> >> useless on my own. I have no idea how to fix it. And it occurs in
>> quite
>> >> a
>> >> >> lot of experiments... I believe it's something behind a delta
>> iteration
>> >> >> [this is the common point between my experiments and the others'
>> code].
>> >> >>
>> >> >> Some pointers would be greatly appreciated :)
>> >> >> Thanks!
>> >> >> Andra
>> >> >>
>> >> >> On Mon, Jul 27, 2015 at 9:32 AM, Stephan Ewen <se...@apache.org>
>> wrote:
>> >> >>
>> >> >> > Hey!
>> >> >> >
>> >> >> > It is pretty much as you said: No one had the time to fix this so
>> far.
>> >> >> They
>> >> >> > are on the list, though,
>> >> >> >
>> >> >> > Do you want to pick up one of them?
>> >> >> >
>> >> >> > Greetings,
>> >> >> > Stephan
>> >> >> >
>> >> >> >
>> >> >> > On Fri, Jul 24, 2015 at 6:56 PM, Andra Lungu <
>> lungu.andra@gmail.com>
>> >> >> > wrote:
>> >> >> >
>> >> >> > > Hi guys,
>> >> >> > >
>> >> >> > > I opened these JIRAs a while ago and, unfortunately, no one had
>> >> time to
>> >> >> > > look at them :|.
>> >> >> > >
>> >> >> > > [1] https://issues.apache.org/jira/browse/FLINK-2361
>> >> >> > > [2] https://issues.apache.org/jira/browse/FLINK-2360
>> >> >> > >
>> >> >> > > [1] could probably be hacked, but [2] is a huge problem that was
>> >> >> > > encountered by other people outside of Gelly.
>> >> >> > >
>> >> >> > > Thanks for the help!
>> >> >> > > Andra
>> >> >> > >
>> >> >> >
>> >> >>
>> >>
>>
Re: A soft reminder
Posted by Andra Lungu <lu...@gmail.com>.
Could you try adding the following lines to your code?
VertexCentricConfiguration parameters = new VertexCentricConfiguration();
parameters.setSolutionSetUnmanagedMemory(true);
return graphWithScoredVertices.runVertexCentricIteration(new
VertexLabelUpdater(delta),
new LabelMessenger(), maxIterations, parameters);
Change the functions appropriately in the runVertexCentricIteration call.
The error seems to be caused by this CompactingHashTable and this parameter
deactivates its usage. I am very curious what happens in your case. If you
could tell us the outcome, I'd greately appreciate it.
On Thu, Jul 30, 2015 at 7:17 PM, Gábor Gévay <gg...@gmail.com> wrote:
> Yes, in a VertexCentricIteration with a few million nodes, running
> locally on my laptop with about 10 GB of memory given to java.
>
> Best,
> Gabor
>
>
>
> 2015-07-30 18:32 GMT+02:00 Andra Lungu <lu...@gmail.com>:
> > Hi Gabor,
> >
> > Within a delta iteration right?
> >
> > On Thu, Jul 30, 2015 at 6:31 PM, Gábor Gévay <gg...@gmail.com> wrote:
> >
> >> Hi,
> >>
> >> I have also run into this problem just now. It only happens with much
> data.
> >>
> >> Best regards,
> >> Gabor
> >>
> >>
> >>
> >> 2015-07-27 11:35 GMT+02:00 Felix Neutatz <ne...@googlemail.com>:
> >> > Hi,
> >> >
> >> > I also encountered the EOF exception for a delta iteration with "more
> >> > data". With less data it works ...
> >> >
> >> > Best regards,
> >> > Felix
> >> > Am 27.07.2015 10:25 vorm. schrieb "Andra Lungu" <
> lungu.andra@gmail.com>:
> >> >
> >> >> Hi Stephan,
> >> >>
> >> >> I tried to debug a bit around the EOF Exception. It seems that I am
> >> pretty
> >> >> useless on my own. I have no idea how to fix it. And it occurs in
> quite
> >> a
> >> >> lot of experiments... I believe it's something behind a delta
> iteration
> >> >> [this is the common point between my experiments and the others'
> code].
> >> >>
> >> >> Some pointers would be greatly appreciated :)
> >> >> Thanks!
> >> >> Andra
> >> >>
> >> >> On Mon, Jul 27, 2015 at 9:32 AM, Stephan Ewen <se...@apache.org>
> wrote:
> >> >>
> >> >> > Hey!
> >> >> >
> >> >> > It is pretty much as you said: No one had the time to fix this so
> far.
> >> >> They
> >> >> > are on the list, though,
> >> >> >
> >> >> > Do you want to pick up one of them?
> >> >> >
> >> >> > Greetings,
> >> >> > Stephan
> >> >> >
> >> >> >
> >> >> > On Fri, Jul 24, 2015 at 6:56 PM, Andra Lungu <
> lungu.andra@gmail.com>
> >> >> > wrote:
> >> >> >
> >> >> > > Hi guys,
> >> >> > >
> >> >> > > I opened these JIRAs a while ago and, unfortunately, no one had
> >> time to
> >> >> > > look at them :|.
> >> >> > >
> >> >> > > [1] https://issues.apache.org/jira/browse/FLINK-2361
> >> >> > > [2] https://issues.apache.org/jira/browse/FLINK-2360
> >> >> > >
> >> >> > > [1] could probably be hacked, but [2] is a huge problem that was
> >> >> > > encountered by other people outside of Gelly.
> >> >> > >
> >> >> > > Thanks for the help!
> >> >> > > Andra
> >> >> > >
> >> >> >
> >> >>
> >>
>
Re: A soft reminder
Posted by Gábor Gévay <gg...@gmail.com>.
Yes, in a VertexCentricIteration with a few million nodes, running
locally on my laptop with about 10 GB of memory given to java.
Best,
Gabor
2015-07-30 18:32 GMT+02:00 Andra Lungu <lu...@gmail.com>:
> Hi Gabor,
>
> Within a delta iteration right?
>
> On Thu, Jul 30, 2015 at 6:31 PM, Gábor Gévay <gg...@gmail.com> wrote:
>
>> Hi,
>>
>> I have also run into this problem just now. It only happens with much data.
>>
>> Best regards,
>> Gabor
>>
>>
>>
>> 2015-07-27 11:35 GMT+02:00 Felix Neutatz <ne...@googlemail.com>:
>> > Hi,
>> >
>> > I also encountered the EOF exception for a delta iteration with "more
>> > data". With less data it works ...
>> >
>> > Best regards,
>> > Felix
>> > Am 27.07.2015 10:25 vorm. schrieb "Andra Lungu" <lu...@gmail.com>:
>> >
>> >> Hi Stephan,
>> >>
>> >> I tried to debug a bit around the EOF Exception. It seems that I am
>> pretty
>> >> useless on my own. I have no idea how to fix it. And it occurs in quite
>> a
>> >> lot of experiments... I believe it's something behind a delta iteration
>> >> [this is the common point between my experiments and the others' code].
>> >>
>> >> Some pointers would be greatly appreciated :)
>> >> Thanks!
>> >> Andra
>> >>
>> >> On Mon, Jul 27, 2015 at 9:32 AM, Stephan Ewen <se...@apache.org> wrote:
>> >>
>> >> > Hey!
>> >> >
>> >> > It is pretty much as you said: No one had the time to fix this so far.
>> >> They
>> >> > are on the list, though,
>> >> >
>> >> > Do you want to pick up one of them?
>> >> >
>> >> > Greetings,
>> >> > Stephan
>> >> >
>> >> >
>> >> > On Fri, Jul 24, 2015 at 6:56 PM, Andra Lungu <lu...@gmail.com>
>> >> > wrote:
>> >> >
>> >> > > Hi guys,
>> >> > >
>> >> > > I opened these JIRAs a while ago and, unfortunately, no one had
>> time to
>> >> > > look at them :|.
>> >> > >
>> >> > > [1] https://issues.apache.org/jira/browse/FLINK-2361
>> >> > > [2] https://issues.apache.org/jira/browse/FLINK-2360
>> >> > >
>> >> > > [1] could probably be hacked, but [2] is a huge problem that was
>> >> > > encountered by other people outside of Gelly.
>> >> > >
>> >> > > Thanks for the help!
>> >> > > Andra
>> >> > >
>> >> >
>> >>
>>
Re: A soft reminder
Posted by Andra Lungu <lu...@gmail.com>.
Hi Gabor,
Within a delta iteration right?
On Thu, Jul 30, 2015 at 6:31 PM, Gábor Gévay <gg...@gmail.com> wrote:
> Hi,
>
> I have also run into this problem just now. It only happens with much data.
>
> Best regards,
> Gabor
>
>
>
> 2015-07-27 11:35 GMT+02:00 Felix Neutatz <ne...@googlemail.com>:
> > Hi,
> >
> > I also encountered the EOF exception for a delta iteration with "more
> > data". With less data it works ...
> >
> > Best regards,
> > Felix
> > Am 27.07.2015 10:25 vorm. schrieb "Andra Lungu" <lu...@gmail.com>:
> >
> >> Hi Stephan,
> >>
> >> I tried to debug a bit around the EOF Exception. It seems that I am
> pretty
> >> useless on my own. I have no idea how to fix it. And it occurs in quite
> a
> >> lot of experiments... I believe it's something behind a delta iteration
> >> [this is the common point between my experiments and the others' code].
> >>
> >> Some pointers would be greatly appreciated :)
> >> Thanks!
> >> Andra
> >>
> >> On Mon, Jul 27, 2015 at 9:32 AM, Stephan Ewen <se...@apache.org> wrote:
> >>
> >> > Hey!
> >> >
> >> > It is pretty much as you said: No one had the time to fix this so far.
> >> They
> >> > are on the list, though,
> >> >
> >> > Do you want to pick up one of them?
> >> >
> >> > Greetings,
> >> > Stephan
> >> >
> >> >
> >> > On Fri, Jul 24, 2015 at 6:56 PM, Andra Lungu <lu...@gmail.com>
> >> > wrote:
> >> >
> >> > > Hi guys,
> >> > >
> >> > > I opened these JIRAs a while ago and, unfortunately, no one had
> time to
> >> > > look at them :|.
> >> > >
> >> > > [1] https://issues.apache.org/jira/browse/FLINK-2361
> >> > > [2] https://issues.apache.org/jira/browse/FLINK-2360
> >> > >
> >> > > [1] could probably be hacked, but [2] is a huge problem that was
> >> > > encountered by other people outside of Gelly.
> >> > >
> >> > > Thanks for the help!
> >> > > Andra
> >> > >
> >> >
> >>
>
Re: A soft reminder
Posted by Gábor Gévay <gg...@gmail.com>.
Hi,
I have also run into this problem just now. It only happens with much data.
Best regards,
Gabor
2015-07-27 11:35 GMT+02:00 Felix Neutatz <ne...@googlemail.com>:
> Hi,
>
> I also encountered the EOF exception for a delta iteration with "more
> data". With less data it works ...
>
> Best regards,
> Felix
> Am 27.07.2015 10:25 vorm. schrieb "Andra Lungu" <lu...@gmail.com>:
>
>> Hi Stephan,
>>
>> I tried to debug a bit around the EOF Exception. It seems that I am pretty
>> useless on my own. I have no idea how to fix it. And it occurs in quite a
>> lot of experiments... I believe it's something behind a delta iteration
>> [this is the common point between my experiments and the others' code].
>>
>> Some pointers would be greatly appreciated :)
>> Thanks!
>> Andra
>>
>> On Mon, Jul 27, 2015 at 9:32 AM, Stephan Ewen <se...@apache.org> wrote:
>>
>> > Hey!
>> >
>> > It is pretty much as you said: No one had the time to fix this so far.
>> They
>> > are on the list, though,
>> >
>> > Do you want to pick up one of them?
>> >
>> > Greetings,
>> > Stephan
>> >
>> >
>> > On Fri, Jul 24, 2015 at 6:56 PM, Andra Lungu <lu...@gmail.com>
>> > wrote:
>> >
>> > > Hi guys,
>> > >
>> > > I opened these JIRAs a while ago and, unfortunately, no one had time to
>> > > look at them :|.
>> > >
>> > > [1] https://issues.apache.org/jira/browse/FLINK-2361
>> > > [2] https://issues.apache.org/jira/browse/FLINK-2360
>> > >
>> > > [1] could probably be hacked, but [2] is a huge problem that was
>> > > encountered by other people outside of Gelly.
>> > >
>> > > Thanks for the help!
>> > > Andra
>> > >
>> >
>>
Re: A soft reminder
Posted by Felix Neutatz <ne...@googlemail.com>.
Hi,
I also encountered the EOF exception for a delta iteration with "more
data". With less data it works ...
Best regards,
Felix
Am 27.07.2015 10:25 vorm. schrieb "Andra Lungu" <lu...@gmail.com>:
> Hi Stephan,
>
> I tried to debug a bit around the EOF Exception. It seems that I am pretty
> useless on my own. I have no idea how to fix it. And it occurs in quite a
> lot of experiments... I believe it's something behind a delta iteration
> [this is the common point between my experiments and the others' code].
>
> Some pointers would be greatly appreciated :)
> Thanks!
> Andra
>
> On Mon, Jul 27, 2015 at 9:32 AM, Stephan Ewen <se...@apache.org> wrote:
>
> > Hey!
> >
> > It is pretty much as you said: No one had the time to fix this so far.
> They
> > are on the list, though,
> >
> > Do you want to pick up one of them?
> >
> > Greetings,
> > Stephan
> >
> >
> > On Fri, Jul 24, 2015 at 6:56 PM, Andra Lungu <lu...@gmail.com>
> > wrote:
> >
> > > Hi guys,
> > >
> > > I opened these JIRAs a while ago and, unfortunately, no one had time to
> > > look at them :|.
> > >
> > > [1] https://issues.apache.org/jira/browse/FLINK-2361
> > > [2] https://issues.apache.org/jira/browse/FLINK-2360
> > >
> > > [1] could probably be hacked, but [2] is a huge problem that was
> > > encountered by other people outside of Gelly.
> > >
> > > Thanks for the help!
> > > Andra
> > >
> >
>
Re: A soft reminder
Posted by Andra Lungu <lu...@gmail.com>.
Hi Stephan,
I tried to debug a bit around the EOF Exception. It seems that I am pretty
useless on my own. I have no idea how to fix it. And it occurs in quite a
lot of experiments... I believe it's something behind a delta iteration
[this is the common point between my experiments and the others' code].
Some pointers would be greatly appreciated :)
Thanks!
Andra
On Mon, Jul 27, 2015 at 9:32 AM, Stephan Ewen <se...@apache.org> wrote:
> Hey!
>
> It is pretty much as you said: No one had the time to fix this so far. They
> are on the list, though,
>
> Do you want to pick up one of them?
>
> Greetings,
> Stephan
>
>
> On Fri, Jul 24, 2015 at 6:56 PM, Andra Lungu <lu...@gmail.com>
> wrote:
>
> > Hi guys,
> >
> > I opened these JIRAs a while ago and, unfortunately, no one had time to
> > look at them :|.
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-2361
> > [2] https://issues.apache.org/jira/browse/FLINK-2360
> >
> > [1] could probably be hacked, but [2] is a huge problem that was
> > encountered by other people outside of Gelly.
> >
> > Thanks for the help!
> > Andra
> >
>
Re: A soft reminder
Posted by Stephan Ewen <se...@apache.org>.
Hey!
It is pretty much as you said: No one had the time to fix this so far. They
are on the list, though,
Do you want to pick up one of them?
Greetings,
Stephan
On Fri, Jul 24, 2015 at 6:56 PM, Andra Lungu <lu...@gmail.com> wrote:
> Hi guys,
>
> I opened these JIRAs a while ago and, unfortunately, no one had time to
> look at them :|.
>
> [1] https://issues.apache.org/jira/browse/FLINK-2361
> [2] https://issues.apache.org/jira/browse/FLINK-2360
>
> [1] could probably be hacked, but [2] is a huge problem that was
> encountered by other people outside of Gelly.
>
> Thanks for the help!
> Andra
>