You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Wei Zhong <we...@gmail.com> on 2019/10/09 03:23:50 UTC
Re: [DISCUSS] Flink Python UDF Environment and Dependency Management
Hi Jincheng, Dian and Jeff,
Thank you for your replies and comments in google doc! I think we have come to an agreement on the desgin doc with only minor changes as follow:
- Using the API "set_python_executable" instead of "set_environment_variable" to set the python executable file path.
- Making the argument "requirements_cached_dir" of API "set_python_requirements" optional to support only upload a requirement.txt file.
I'm also glad to hear any other opinions!
Thanks,
Wei
> 在 2019年9月26日，15:23，Dian Fu <di...@gmail.com> 写道：
>
> Hi Wei,
>
> Thanks a lot for bringing up this discussion. Python dependency management is very important for Python users. I have left a few comments on the design doc.
>
> Thanks,
> Dian
>
>> 在 2019年9月26日，下午12:23，jincheng sun <su...@gmail.com> 写道：
>>
>> Thanks for bring up the discussion, Wei.
>> Overall the design doc looks good. I have left a few comments.
>>
>> BTW: Dependency Management is very important for Python UDFs, welcome
>> anyone left your suggestions!
>>
>> Best,
>> Jincheng
>>
>> Wei Zhong <we...@gmail.com> 于2019年9月26日周四 上午11:59写道：
>>
>>> Hi everyone,
>>>
>>> In FLIP-58 [1] we have a plan to support Python UDF. As a critical part of
>>> python UDF, the environment and dependency management of users' python code
>>> has not been fully discussed.
>>>
>>> I'd like to start a discussion on "Flink Python UDF Environment and
>>> Dependency Management". Here is the design doc I drafted:
>>>
>>>
>>> https://docs.google.com/document/d/1vq5J3TSyhscQXbpRhz-Yd3KCX62PBJeC_a_h3amUvJ4/edit?usp=sharing
>>>
>>> Please take a look, and feedbacks are welcome.
>>>
>>> Thanks,
>>> Wei
>>>
>>> [1]:
>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table
>>> <https://cwiki.apache.org/confluence/display/FLINK/FLIP-58:+Flink+Python+User-Defined+Stateless+Function+for+Table>
>>>
>>>
>
Re: [DISCUSS] Flink Python UDF Environment and Dependency Management
Posted by Wei Zhong <we...@gmail.com>.
Thank you Jincheng. I have got the permission.
> 在 2019年10月12日，18:09，jincheng sun <su...@gmail.com> 写道：
>
> Hi Wei Zhong,
>
> I have given you edit permission, could you please re-login and check it. :)
>
> Best,
> Jincheng
>
>
> Wei Zhong <weizhong0618@gmail.com <ma...@gmail.com>> 于2019年10月12日周六 下午6:06写道：
> Hi Jincheng,
>
> As communicated in this email thread, I’m proposing to convert the design doc to a FLIP and bring up the VOTE. It would be great if you can grant me the write access to Confluence :). My Confluence ID is zhongwei.
>
> Thanks,
> Wei Zhong
>
>
>> 在 2019年10月12日，17:41，jincheng sun <sunjincheng121@gmail.com <ma...@gmail.com>> 写道：
>>
>> Hi,
>>
>> + 1 to bring up the VOTE and create the FLIP.
>>
>> Best,
>> Jincheng
>>
>> Dian Fu <dian0511.fu@gmail.com <ma...@gmail.com>> 于2019年10月12日周六 上午10:12写道：
>> Hi Wei,
>>
>> Thanks for the great work! It seems that it has reached an agreement on the design. Should we start VOTE on this design? I'm also wondering if a FLIP is deserved as it introduces user facing API. If so, we should create a FLIP before VOTE.
>>
>> Thanks,
>> Dian
>>
>> > 在 2019年10月9日，上午11:23，Wei Zhong <weizhong0618@gmail.com <ma...@gmail.com>> 写道：
>> >
>> > Hi Jincheng, Dian and Jeff,
>> >
>> > Thank you for your replies and comments in google doc! I think we have come to an agreement on the desgin doc with only minor changes as follow:
>> > - Using the API "set_python_executable" instead of "set_environment_variable" to set the python executable file path.
>> > - Making the argument "requirements_cached_dir" of API "set_python_requirements" optional to support only upload a requirement.txt file.
>> >
>> > I'm also glad to hear any other opinions!
>> >
>> > Thanks,
>> > Wei
>> >
>> >
>> >> 在 2019年9月26日，15:23，Dian Fu <dian0511.fu@gmail.com <ma...@gmail.com>> 写道：
>> >>
>> >> Hi Wei,
>> >>
>> >> Thanks a lot for bringing up this discussion. Python dependency management is very important for Python users. I have left a few comments on the design doc.
>> >>
>> >> Thanks,
>> >> Dian
>> >>
>> >>> 在 2019年9月26日，下午12:23，jincheng sun <sunjincheng121@gmail.com <ma...@gmail.com>> 写道：
>> >>>
>> >>> Thanks for bring up the discussion, Wei.
>> >>> Overall the design doc looks good. I have left a few comments.
>> >>>
>> >>> BTW: Dependency Management is very important for Python UDFs, welcome
>> >>> anyone left your suggestions!
>> >>>
>> >>> Best,
>> >>> Jincheng
>> >>>
>> >>> Wei Zhong <weizhong0618@gmail.com <ma...@gmail.com>> 于2019年9月26日周四 上午11:59写道：
>> >>>
>> >>>> Hi everyone,
>> >>>>
>> >>>> In FLIP-58 [1] we have a plan to support Python UDF. As a critical part of
>> >>>> python UDF, the environment and dependency management of users' python code
>> >>>> has not been fully discussed.
>> >>>>
>> >>>> I'd like to start a discussion on "Flink Python UDF Environment and
>> >>>> Dependency Management". Here is the design doc I drafted:
>> >>>>
>> >>>>
>> >>>> https://docs.google.com/document/d/1vq5J3TSyhscQXbpRhz-Yd3KCX62PBJeC_a_h3amUvJ4/edit?usp=sharing <https://docs.google.com/document/d/1vq5J3TSyhscQXbpRhz-Yd3KCX62PBJeC_a_h3amUvJ4/edit?usp=sharing>
>> >>>>
>> >>>> Please take a look, and feedbacks are welcome.
>> >>>>
>> >>>> Thanks,
>> >>>> Wei
>> >>>>
>> >>>> [1]:
>> >>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table <https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table>
>> >>>> <https://cwiki.apache.org/confluence/display/FLINK/FLIP-58:+Flink+Python+User-Defined+Stateless+Function+for+Table <https://cwiki.apache.org/confluence/display/FLINK/FLIP-58:+Flink+Python+User-Defined+Stateless+Function+for+Table>>
>> >>>>
>> >>>>
>> >>
>> >
>>
>
Re: [DISCUSS] Flink Python UDF Environment and Dependency Management
Posted by jincheng sun <su...@gmail.com>.
Hi Wei Zhong,
I have given you edit permission, could you please re-login and check it. :)
Best,
Jincheng
Wei Zhong <we...@gmail.com> 于2019年10月12日周六 下午6:06写道：
> Hi Jincheng,
>
> As communicated in this email thread, I’m proposing to convert the design
> doc to a FLIP and bring up the VOTE. It would be great if you can grant me
> the write access to Confluence :). My Confluence ID is zhongwei.
>
> Thanks,
> Wei Zhong
>
>
> 在 2019年10月12日，17:41，jincheng sun <su...@gmail.com> 写道：
>
> Hi,
>
> + 1 to bring up the VOTE and create the FLIP.
>
> Best,
> Jincheng
>
> Dian Fu <di...@gmail.com> 于2019年10月12日周六 上午10:12写道：
>
>> Hi Wei,
>>
>> Thanks for the great work! It seems that it has reached an agreement on
>> the design. Should we start VOTE on this design? I'm also wondering if a
>> FLIP is deserved as it introduces user facing API. If so, we should create
>> a FLIP before VOTE.
>>
>> Thanks,
>> Dian
>>
>> > 在 2019年10月9日，上午11:23，Wei Zhong <we...@gmail.com> 写道：
>> >
>> > Hi Jincheng, Dian and Jeff,
>> >
>> > Thank you for your replies and comments in google doc! I think we have
>> come to an agreement on the desgin doc with only minor changes as follow:
>> > - Using the API "set_python_executable" instead of
>> "set_environment_variable" to set the python executable file path.
>> > - Making the argument "requirements_cached_dir" of API
>> "set_python_requirements" optional to support only upload a requirement.txt
>> file.
>> >
>> > I'm also glad to hear any other opinions!
>> >
>> > Thanks,
>> > Wei
>> >
>> >
>> >> 在 2019年9月26日，15:23，Dian Fu <di...@gmail.com> 写道：
>> >>
>> >> Hi Wei,
>> >>
>> >> Thanks a lot for bringing up this discussion. Python dependency
>> management is very important for Python users. I have left a few comments
>> on the design doc.
>> >>
>> >> Thanks,
>> >> Dian
>> >>
>> >>> 在 2019年9月26日，下午12:23，jincheng sun <su...@gmail.com> 写道：
>> >>>
>> >>> Thanks for bring up the discussion, Wei.
>> >>> Overall the design doc looks good. I have left a few comments.
>> >>>
>> >>> BTW: Dependency Management is very important for Python UDFs, welcome
>> >>> anyone left your suggestions!
>> >>>
>> >>> Best,
>> >>> Jincheng
>> >>>
>> >>> Wei Zhong <we...@gmail.com> 于2019年9月26日周四 上午11:59写道：
>> >>>
>> >>>> Hi everyone,
>> >>>>
>> >>>> In FLIP-58 [1] we have a plan to support Python UDF. As a critical
>> part of
>> >>>> python UDF, the environment and dependency management of users'
>> python code
>> >>>> has not been fully discussed.
>> >>>>
>> >>>> I'd like to start a discussion on "Flink Python UDF Environment and
>> >>>> Dependency Management". Here is the design doc I drafted:
>> >>>>
>> >>>>
>> >>>>
>> https://docs.google.com/document/d/1vq5J3TSyhscQXbpRhz-Yd3KCX62PBJeC_a_h3amUvJ4/edit?usp=sharing
>> >>>>
>> >>>> Please take a look, and feedbacks are welcome.
>> >>>>
>> >>>> Thanks,
>> >>>> Wei
>> >>>>
>> >>>> [1]:
>> >>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table
>> >>>> <
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-58:+Flink+Python+User-Defined+Stateless+Function+for+Table
>> >
>> >>>>
>> >>>>
>> >>
>> >
>>
>>
>
Re: [DISCUSS] Flink Python UDF Environment and Dependency Management
Posted by Wei Zhong <we...@gmail.com>.
Hi Jincheng,
As communicated in this email thread, I’m proposing to convert the design doc to a FLIP and bring up the VOTE. It would be great if you can grant me the write access to Confluence :). My Confluence ID is zhongwei.
Thanks,
Wei Zhong
> 在 2019年10月12日，17:41，jincheng sun <su...@gmail.com> 写道：
>
> Hi,
>
> + 1 to bring up the VOTE and create the FLIP.
>
> Best,
> Jincheng
>
> Dian Fu <dian0511.fu@gmail.com <ma...@gmail.com>> 于2019年10月12日周六 上午10:12写道：
> Hi Wei,
>
> Thanks for the great work! It seems that it has reached an agreement on the design. Should we start VOTE on this design? I'm also wondering if a FLIP is deserved as it introduces user facing API. If so, we should create a FLIP before VOTE.
>
> Thanks,
> Dian
>
> > 在 2019年10月9日，上午11:23，Wei Zhong <weizhong0618@gmail.com <ma...@gmail.com>> 写道：
> >
> > Hi Jincheng, Dian and Jeff,
> >
> > Thank you for your replies and comments in google doc! I think we have come to an agreement on the desgin doc with only minor changes as follow:
> > - Using the API "set_python_executable" instead of "set_environment_variable" to set the python executable file path.
> > - Making the argument "requirements_cached_dir" of API "set_python_requirements" optional to support only upload a requirement.txt file.
> >
> > I'm also glad to hear any other opinions!
> >
> > Thanks,
> > Wei
> >
> >
> >> 在 2019年9月26日，15:23，Dian Fu <dian0511.fu@gmail.com <ma...@gmail.com>> 写道：
> >>
> >> Hi Wei,
> >>
> >> Thanks a lot for bringing up this discussion. Python dependency management is very important for Python users. I have left a few comments on the design doc.
> >>
> >> Thanks,
> >> Dian
> >>
> >>> 在 2019年9月26日，下午12:23，jincheng sun <sunjincheng121@gmail.com <ma...@gmail.com>> 写道：
> >>>
> >>> Thanks for bring up the discussion, Wei.
> >>> Overall the design doc looks good. I have left a few comments.
> >>>
> >>> BTW: Dependency Management is very important for Python UDFs, welcome
> >>> anyone left your suggestions!
> >>>
> >>> Best,
> >>> Jincheng
> >>>
> >>> Wei Zhong <weizhong0618@gmail.com <ma...@gmail.com>> 于2019年9月26日周四 上午11:59写道：
> >>>
> >>>> Hi everyone,
> >>>>
> >>>> In FLIP-58 [1] we have a plan to support Python UDF. As a critical part of
> >>>> python UDF, the environment and dependency management of users' python code
> >>>> has not been fully discussed.
> >>>>
> >>>> I'd like to start a discussion on "Flink Python UDF Environment and
> >>>> Dependency Management". Here is the design doc I drafted:
> >>>>
> >>>>
> >>>> https://docs.google.com/document/d/1vq5J3TSyhscQXbpRhz-Yd3KCX62PBJeC_a_h3amUvJ4/edit?usp=sharing <https://docs.google.com/document/d/1vq5J3TSyhscQXbpRhz-Yd3KCX62PBJeC_a_h3amUvJ4/edit?usp=sharing>
> >>>>
> >>>> Please take a look, and feedbacks are welcome.
> >>>>
> >>>> Thanks,
> >>>> Wei
> >>>>
> >>>> [1]:
> >>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table <https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table>
> >>>> <https://cwiki.apache.org/confluence/display/FLINK/FLIP-58:+Flink+Python+User-Defined+Stateless+Function+for+Table <https://cwiki.apache.org/confluence/display/FLINK/FLIP-58:+Flink+Python+User-Defined+Stateless+Function+for+Table>>
> >>>>
> >>>>
> >>
> >
>
Re: [DISCUSS] Flink Python UDF Environment and Dependency Management
Posted by jincheng sun <su...@gmail.com>.
Hi,
+ 1 to bring up the VOTE and create the FLIP.
Best,
Jincheng
Dian Fu <di...@gmail.com> 于2019年10月12日周六 上午10:12写道：
> Hi Wei,
>
> Thanks for the great work! It seems that it has reached an agreement on
> the design. Should we start VOTE on this design? I'm also wondering if a
> FLIP is deserved as it introduces user facing API. If so, we should create
> a FLIP before VOTE.
>
> Thanks,
> Dian
>
> > 在 2019年10月9日，上午11:23，Wei Zhong <we...@gmail.com> 写道：
> >
> > Hi Jincheng, Dian and Jeff,
> >
> > Thank you for your replies and comments in google doc! I think we have
> come to an agreement on the desgin doc with only minor changes as follow:
> > - Using the API "set_python_executable" instead of
> "set_environment_variable" to set the python executable file path.
> > - Making the argument "requirements_cached_dir" of API
> "set_python_requirements" optional to support only upload a requirement.txt
> file.
> >
> > I'm also glad to hear any other opinions!
> >
> > Thanks,
> > Wei
> >
> >
> >> 在 2019年9月26日，15:23，Dian Fu <di...@gmail.com> 写道：
> >>
> >> Hi Wei,
> >>
> >> Thanks a lot for bringing up this discussion. Python dependency
> management is very important for Python users. I have left a few comments
> on the design doc.
> >>
> >> Thanks,
> >> Dian
> >>
> >>> 在 2019年9月26日，下午12:23，jincheng sun <su...@gmail.com> 写道：
> >>>
> >>> Thanks for bring up the discussion, Wei.
> >>> Overall the design doc looks good. I have left a few comments.
> >>>
> >>> BTW: Dependency Management is very important for Python UDFs, welcome
> >>> anyone left your suggestions!
> >>>
> >>> Best,
> >>> Jincheng
> >>>
> >>> Wei Zhong <we...@gmail.com> 于2019年9月26日周四 上午11:59写道：
> >>>
> >>>> Hi everyone,
> >>>>
> >>>> In FLIP-58 [1] we have a plan to support Python UDF. As a critical
> part of
> >>>> python UDF, the environment and dependency management of users'
> python code
> >>>> has not been fully discussed.
> >>>>
> >>>> I'd like to start a discussion on "Flink Python UDF Environment and
> >>>> Dependency Management". Here is the design doc I drafted:
> >>>>
> >>>>
> >>>>
> https://docs.google.com/document/d/1vq5J3TSyhscQXbpRhz-Yd3KCX62PBJeC_a_h3amUvJ4/edit?usp=sharing
> >>>>
> >>>> Please take a look, and feedbacks are welcome.
> >>>>
> >>>> Thanks,
> >>>> Wei
> >>>>
> >>>> [1]:
> >>>>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table
> >>>> <
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-58:+Flink+Python+User-Defined+Stateless+Function+for+Table
> >
> >>>>
> >>>>
> >>
> >
>
>
Re: [DISCUSS] Flink Python UDF Environment and Dependency Management
Posted by Dian Fu <di...@gmail.com>.
Hi Wei,
Thanks for the great work! It seems that it has reached an agreement on the design. Should we start VOTE on this design? I'm also wondering if a FLIP is deserved as it introduces user facing API. If so, we should create a FLIP before VOTE.
Thanks,
Dian
> 在 2019年10月9日，上午11:23，Wei Zhong <we...@gmail.com> 写道：
>
> Hi Jincheng, Dian and Jeff,
>
> Thank you for your replies and comments in google doc! I think we have come to an agreement on the desgin doc with only minor changes as follow:
> - Using the API "set_python_executable" instead of "set_environment_variable" to set the python executable file path.
> - Making the argument "requirements_cached_dir" of API "set_python_requirements" optional to support only upload a requirement.txt file.
>
> I'm also glad to hear any other opinions!
>
> Thanks,
> Wei
>
>
>> 在 2019年9月26日，15:23，Dian Fu <di...@gmail.com> 写道：
>>
>> Hi Wei,
>>
>> Thanks a lot for bringing up this discussion. Python dependency management is very important for Python users. I have left a few comments on the design doc.
>>
>> Thanks,
>> Dian
>>
>>> 在 2019年9月26日，下午12:23，jincheng sun <su...@gmail.com> 写道：
>>>
>>> Thanks for bring up the discussion, Wei.
>>> Overall the design doc looks good. I have left a few comments.
>>>
>>> BTW: Dependency Management is very important for Python UDFs, welcome
>>> anyone left your suggestions!
>>>
>>> Best,
>>> Jincheng
>>>
>>> Wei Zhong <we...@gmail.com> 于2019年9月26日周四 上午11:59写道：
>>>
>>>> Hi everyone,
>>>>
>>>> In FLIP-58 [1] we have a plan to support Python UDF. As a critical part of
>>>> python UDF, the environment and dependency management of users' python code
>>>> has not been fully discussed.
>>>>
>>>> I'd like to start a discussion on "Flink Python UDF Environment and
>>>> Dependency Management". Here is the design doc I drafted:
>>>>
>>>>
>>>> https://docs.google.com/document/d/1vq5J3TSyhscQXbpRhz-Yd3KCX62PBJeC_a_h3amUvJ4/edit?usp=sharing
>>>>
>>>> Please take a look, and feedbacks are welcome.
>>>>
>>>> Thanks,
>>>> Wei
>>>>
>>>> [1]:
>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table
>>>> <https://cwiki.apache.org/confluence/display/FLINK/FLIP-58:+Flink+Python+User-Defined+Stateless+Function+for+Table>
>>>>
>>>>
>>
>