You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Udi Meiri <eh...@google.com> on 2019/05/14 01:14:33 UTC
pickling typing types in Python 3.5+
It seems like pickling of typing types is broken in 3.5 and 3.6, fixed in
3.7:
https://github.com/python/typing/issues/511
Here are my attempts:
https://gist.github.com/udim/ec213305ca865390c391001e8778e91d
My ideas:
1. I know that we override type object handling in pickler.py
(_nested_type_wrapper), and perhaps this mechanism can be used to pickle
typing classes correctly. The question is how.
2. Exclude/stub out these classes when pickling a pipeline - they are only
used for verification during pipeline construction anyway. This could be a
temporary solution for versions 3.5 and 3.6.
Any ideas / opinions?
Re: pickling typing types in Python 3.5+
Posted by Robert Bradshaw <ro...@google.com>.
I looked into CloudPickle a while back, and would be supportive of the change.
On Mon, Jul 1, 2019 at 11:06 PM Valentyn Tymofieiev <va...@google.com> wrote:
>
> I have checked that cloudpickle (an alternative to dill) is able to pickle and unpickle typing types on Python 3.5, 3.6, which seems to be a recent change, see: https://github.com/cloudpipe/cloudpickle/issues/63#issuecomment-501624383.
>
> I am evaluating cloudpickle as a potential avenue to address several other issues we found in Beam while working on Python 3 support, such as:
>
> https://issues.apache.org/jira/browse/BEAM-6522
> https://issues.apache.org/jira/browse/BEAM-7284
> https://issues.apache.org/jira/browse/BEAM-5878?focusedCommentId=16834554
> https://github.com/uqfoundation/dill/issues/300
> https://issues.apache.org/jira/browse/BEAM-7540
>
> Once I have more information on cloudpickle vs dill in Beam, I'll bring it to the mailing list.
>
> On Wed, May 15, 2019 at 5:25 AM Robert Bradshaw <ro...@google.com> wrote:
>>
>> (2) seems reasonable.
>>
>> On Tue, May 14, 2019 at 3:15 AM Udi Meiri <eh...@google.com> wrote:
>> >
>> > It seems like pickling of typing types is broken in 3.5 and 3.6, fixed in 3.7:
>> > https://github.com/python/typing/issues/511
>> >
>> > Here are my attempts:
>> > https://gist.github.com/udim/ec213305ca865390c391001e8778e91d
>> >
>> >
>> > My ideas:
>> > 1. I know that we override type object handling in pickler.py (_nested_type_wrapper), and perhaps this mechanism can be used to pickle typing classes correctly. The question is how.
>> >
>> > 2. Exclude/stub out these classes when pickling a pipeline - they are only used for verification during pipeline construction anyway. This could be a temporary solution for versions 3.5 and 3.6.
>> >
>> > Any ideas / opinions?
Re: pickling typing types in Python 3.5+
Posted by Valentyn Tymofieiev <va...@google.com>.
I have checked that cloudpickle (an alternative to dill) is able to pickle
and unpickle typing types on Python 3.5, 3.6, which seems to be a recent
change, see:
https://github.com/cloudpipe/cloudpickle/issues/63#issuecomment-501624383.
I am evaluating cloudpickle as a potential avenue to address several other
issues we found in Beam while working on Python 3 support, such as:
https://issues.apache.org/jira/browse/BEAM-6522
https://issues.apache.org/jira/browse/BEAM-7284
https://issues.apache.org/jira/browse/BEAM-5878?focusedCommentId=16834554
https://github.com/uqfoundation/dill/issues/300
https://issues.apache.org/jira/browse/BEAM-7540
Once I have more information on cloudpickle vs dill in Beam, I'll bring it
to the mailing list.
On Wed, May 15, 2019 at 5:25 AM Robert Bradshaw <ro...@google.com> wrote:
> (2) seems reasonable.
>
> On Tue, May 14, 2019 at 3:15 AM Udi Meiri <eh...@google.com> wrote:
> >
> > It seems like pickling of typing types is broken in 3.5 and 3.6, fixed
> in 3.7:
> > https://github.com/python/typing/issues/511
> >
> > Here are my attempts:
> > https://gist.github.com/udim/ec213305ca865390c391001e8778e91d
> >
> >
> > My ideas:
> > 1. I know that we override type object handling in pickler.py
> (_nested_type_wrapper), and perhaps this mechanism can be used to pickle
> typing classes correctly. The question is how.
> >
> > 2. Exclude/stub out these classes when pickling a pipeline - they are
> only used for verification during pipeline construction anyway. This could
> be a temporary solution for versions 3.5 and 3.6.
> >
> > Any ideas / opinions?
>
Re: pickling typing types in Python 3.5+
Posted by Robert Bradshaw <ro...@google.com>.
(2) seems reasonable.
On Tue, May 14, 2019 at 3:15 AM Udi Meiri <eh...@google.com> wrote:
>
> It seems like pickling of typing types is broken in 3.5 and 3.6, fixed in 3.7:
> https://github.com/python/typing/issues/511
>
> Here are my attempts:
> https://gist.github.com/udim/ec213305ca865390c391001e8778e91d
>
>
> My ideas:
> 1. I know that we override type object handling in pickler.py (_nested_type_wrapper), and perhaps this mechanism can be used to pickle typing classes correctly. The question is how.
>
> 2. Exclude/stub out these classes when pickling a pipeline - they are only used for verification during pipeline construction anyway. This could be a temporary solution for versions 3.5 and 3.6.
>
> Any ideas / opinions?