You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Arvid Heise <ar...@gmail.com> on 2015/02/02 12:48:25 UTC
Re: ReduceGroup fails on server
OK, patch indeed worked for my workflow. Thank you very much.
Any idea when this patch will be in a non-snapshot?
Best,
Arvid
On Fri, Jan 30, 2015 at 6:04 PM, Arvid Heise <ar...@gmail.com> wrote:
> Sorry building the 0.9 branch took longer than expected. I will follow
> that up on Monday.
>
> Alternatively I would be grateful for a 0.8.0 patch ;)
>
> Best,
>
> Arvid
>
> On Fri, Jan 30, 2015 at 5:09 PM, Aljoscha Krettek <al...@apache.org>
> wrote:
>
>> Hi Arvid,
>> I have a fix that I hope fixes your problem:
>> https://github.com/aljoscha/flink/tree/serializer-factories-fix
>>
>> Could you try building it and running your example?
>>
>> Cheers,
>> Aljoscha
>>
>> On Fri, Jan 30, 2015 at 3:30 PM, Aljoscha Krettek <al...@apache.org>
>> wrote:
>> > We have a bit of a divide in how we handle TypeSerializer and
>> > TypeComparator: TypeSerializer does not handle duplication but relies
>> > on outside code (RuntimeStatefulSerializerFactory) to perform the
>> > duplication. TypeComparator does duplication itself. There is also the
>> > RuntimeComparatorFactory. This, however, does not perform duplication
>> > but simply hands out the same TypeComparator instance multiple times.
>> > With comparators, the user of the comparator is responsible for
>> > calling duplicate() on the comparator.
>> >
>> > I started work on a branch where I add a duplicate() method to
>> > TypeSerializer that simply return itself for stateless serializers and
>> > does a deep copy if it is stateful. The TypeSerializers no longer have
>> > method isStateful() and I replaced the two serializer factories by one
>> > factory that always returns a "duplicate" of the serializer it holds.
>> >
>> > I think we should consolidate the two approaches. So either let the
>> > factories handle all the duplication or let the user of the
>> > comparator/serializer always handle duplication. I think the former is
>> > better, since the latter makes it very easy to forget to duplicate.
>> >
>> > On Fri, Jan 30, 2015 at 11:34 AM, Aljoscha Krettek <al...@apache.org>
>> wrote:
>> >> Maybe we can get rid of the serializer factories altogether. We could
>> >> enhance the Serializers with a method duplicate() that does nothing
>> >> for stateless serializers and does a deep copy for stateful
>> >> serializers. This would also consolidate all the knowledge about
>> >> stateful/stateless in one place.
>> >>
>> >> On Fri, Jan 30, 2015 at 11:25 AM, Aljoscha Krettek <
>> aljoscha@apache.org> wrote:
>> >>> Of course it doesn't work. The ClassLoader is declared transient in
>> >>> the serializer factory, so it is Null once the factory has been
>> >>> serialized/deserialized once. I open a Jira issue:
>> >>> https://issues.apache.org/jira/browse/FLINK-1463
>> >>>
>> >>> On Thu, Jan 29, 2015 at 9:43 PM, Stephan Ewen <se...@apache.org>
>> wrote:
>> >>>> Thanks for reporting this, Arvid. I remember that we saw something
>> similar
>> >>>> a while back. I will take a look at this later today (I am in PST
>> zone
>> >>>> now), can hopefully fix this.
>> >>>>
>> >>>> Greetings,
>> >>>> Stephan
>>
>
>
Re: ReduceGroup fails on server
Posted by Aljoscha Krettek <al...@apache.org>.
Hi,
we have some bug fixes queued up. So a 0.8.1 bug fix release should be
expected in the upcoming weeks.
Cheers,
Aljoscha
On Mon, Feb 2, 2015 at 12:48 PM, Arvid Heise <ar...@gmail.com> wrote:
> OK, patch indeed worked for my workflow. Thank you very much.
>
> Any idea when this patch will be in a non-snapshot?
>
> Best,
>
> Arvid
>
> On Fri, Jan 30, 2015 at 6:04 PM, Arvid Heise <ar...@gmail.com> wrote:
>
>> Sorry building the 0.9 branch took longer than expected. I will follow
>> that up on Monday.
>>
>> Alternatively I would be grateful for a 0.8.0 patch ;)
>>
>> Best,
>>
>> Arvid
>>
>> On Fri, Jan 30, 2015 at 5:09 PM, Aljoscha Krettek <al...@apache.org>
>> wrote:
>>
>>> Hi Arvid,
>>> I have a fix that I hope fixes your problem:
>>> https://github.com/aljoscha/flink/tree/serializer-factories-fix
>>>
>>> Could you try building it and running your example?
>>>
>>> Cheers,
>>> Aljoscha
>>>
>>> On Fri, Jan 30, 2015 at 3:30 PM, Aljoscha Krettek <al...@apache.org>
>>> wrote:
>>> > We have a bit of a divide in how we handle TypeSerializer and
>>> > TypeComparator: TypeSerializer does not handle duplication but relies
>>> > on outside code (RuntimeStatefulSerializerFactory) to perform the
>>> > duplication. TypeComparator does duplication itself. There is also the
>>> > RuntimeComparatorFactory. This, however, does not perform duplication
>>> > but simply hands out the same TypeComparator instance multiple times.
>>> > With comparators, the user of the comparator is responsible for
>>> > calling duplicate() on the comparator.
>>> >
>>> > I started work on a branch where I add a duplicate() method to
>>> > TypeSerializer that simply return itself for stateless serializers and
>>> > does a deep copy if it is stateful. The TypeSerializers no longer have
>>> > method isStateful() and I replaced the two serializer factories by one
>>> > factory that always returns a "duplicate" of the serializer it holds.
>>> >
>>> > I think we should consolidate the two approaches. So either let the
>>> > factories handle all the duplication or let the user of the
>>> > comparator/serializer always handle duplication. I think the former is
>>> > better, since the latter makes it very easy to forget to duplicate.
>>> >
>>> > On Fri, Jan 30, 2015 at 11:34 AM, Aljoscha Krettek <al...@apache.org>
>>> wrote:
>>> >> Maybe we can get rid of the serializer factories altogether. We could
>>> >> enhance the Serializers with a method duplicate() that does nothing
>>> >> for stateless serializers and does a deep copy for stateful
>>> >> serializers. This would also consolidate all the knowledge about
>>> >> stateful/stateless in one place.
>>> >>
>>> >> On Fri, Jan 30, 2015 at 11:25 AM, Aljoscha Krettek <
>>> aljoscha@apache.org> wrote:
>>> >>> Of course it doesn't work. The ClassLoader is declared transient in
>>> >>> the serializer factory, so it is Null once the factory has been
>>> >>> serialized/deserialized once. I open a Jira issue:
>>> >>> https://issues.apache.org/jira/browse/FLINK-1463
>>> >>>
>>> >>> On Thu, Jan 29, 2015 at 9:43 PM, Stephan Ewen <se...@apache.org>
>>> wrote:
>>> >>>> Thanks for reporting this, Arvid. I remember that we saw something
>>> similar
>>> >>>> a while back. I will take a look at this later today (I am in PST
>>> zone
>>> >>>> now), can hopefully fix this.
>>> >>>>
>>> >>>> Greetings,
>>> >>>> Stephan
>>>
>>
>>