You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by John Tipper <jo...@hotmail.com> on 2022/06/16 09:03:57 UTC
The configured managed memory fraction for Python worker process must be within (0, 1], was: %s
Hi all,
I'm trying to run a PyFlink unit test to test some PyFlink SQL and where my code uses a Python UDF. I can't share my code but the test case is similar to the code here: https://github.com/apache/flink/blob/f8172cdbbc27344896d961be4b0b9cdbf000b5cd/flink-python/pyflink/testing/test_case_utils.py When I have some simple SQL everything is fine. When I add a more complex query I get an error, which looks like it's memory related.
java.lang.IllegalArgumentException: The configured managed memory fraction
for Python worker process must be within (0, 1], was: %s. It may be because
the consumer type "Python" was missing or set to 0 for the config option
"taskmanager.memory.managed.consumer-weights".0.0
In my test case setUp(), I try to set that value like this, but it seems to have no effect:
self.t_env.get_config().get_configuration().set_string("taskmanager.memory.managed.consumer-weights", "PYTHON:30")
Am I not setting it correctly, or is there something else I need to do to fix this error?
Many thanks,
John
Re: The configured managed memory fraction for Python worker process must be within (0, 1], was: %s
Posted by Dian Fu <di...@gmail.com>.
Hi John,
A simple workaround is by setting configuration option
`python.fn-execution.memory.managed` as `false`.
Regards,
Dian
On Fri, Jun 17, 2022 at 6:50 PM John Tipper <jo...@hotmail.com> wrote:
> Thanks Dian - I’m restricted to 1.13 for the moment because this is
> running inside AWS Kinesis. Is there a way to manually bypass that issue?
>
> J
>
> Sent from my iPhone
>
> On 17 Jun 2022, at 04:59, Dian Fu <di...@gmail.com> wrote:
>
>
> >> This error generally occurs in jobs where there are transfers between
> Table and datastream.
> AFAIK, this issue should have already been fixed, see
> https://issues.apache.org/jira/browse/FLINK-26920 and
> https://issues.apache.org/jira/browse/FLINK-23133 for more details.
>
> Regards,
> Dian
>
> On Fri, Jun 17, 2022 at 10:17 AM Xingbo Huang <hx...@gmail.com> wrote:
>
>> Hi John,
>>
>> Because I can't see your code, I can only provide some possible reasons
>> for this error:
>> 1. This error generally occurs in jobs where there are transfers between
>> Table and datastream. But given that you said you just used the sql +
>> python udf, this shouldn't be the case.
>> 2. The default value of `taskmanager.memory.managed.consumer-weights` is
>> `OPERATOR:70,STATE_BACKEND:70,PYTHON:30`, so in your case, there is
>> actually no need to set it to `PYTHON:30`
>> 3. In fact, for pure sql+python udf jobs, if you don't set error value
>> `PYTHON:0` in `taskmanager.memory.managed.consumer-weights`, I really can't
>> think of any situation where this problem will occur.
>>
>> Best,
>> Xingbo
>>
>> John Tipper <jo...@hotmail.com> 于2022年6月16日周四 19:41写道：
>>
>>> Hi Xingbo,
>>>
>>> Yes, there are a number of temporary views being created, where each is
>>> being created using SQL (CREATE TEMPORARY VIEW ...) rather than explicit
>>> calls to the Table and DataStream APIs.
>>>
>>> Is this a good pattern or are there caveats I should be aware of please?
>>>
>>> Many thanks,
>>>
>>> John
>>>
>>>
>>> ------------------------------
>>> *From:* Xingbo Huang <hx...@gmail.com>
>>> *Sent:* 16 June 2022 12:34
>>> *To:* John Tipper <jo...@hotmail.com>
>>> *Cc:* user@flink.apache.org <us...@flink.apache.org>
>>> *Subject:* Re: The configured managed memory fraction for Python worker
>>> process must be within (0, 1], was: %s
>>>
>>> Hi John,
>>>
>>> Does your job logic include conversion between Table and DataStream? For
>>> example, methods such as `create_temporary_view(path: str, data_stream:
>>> DataStream): -> Table` are used.
>>>
>>> Best,
>>> Xingbo
>>>
>>> John Tipper <jo...@hotmail.com> 于2022年6月16日周四 18:31写道：
>>>
>>> Hi Xingbo,
>>>
>>> I’m afraid I can’t share my code but Flink is 1.13. The main Flink code
>>> is running inside Kinesis on AWS so I cannot change the version.
>>>
>>> Many thanks,
>>>
>>> John
>>>
>>> Sent from my iPhone
>>>
>>> On 16 Jun 2022, at 10:37, Xingbo Huang <hx...@gmail.com> wrote:
>>>
>>>
>>> Hi John,
>>>
>>> Could you provide the code snippet and the version of pyflink you used?
>>>
>>> Best,
>>> Xingbo
>>>
>>>
>>> John Tipper <jo...@hotmail.com> 于2022年6月16日周四 17:05写道：
>>>
>>> Hi all,
>>>
>>> I'm trying to run a PyFlink unit test to test some PyFlink SQL and where
>>> my code uses a Python UDF. I can't share my code but the test case is
>>> similar to the code here:
>>> https://github.com/apache/flink/blob/f8172cdbbc27344896d961be4b0b9cdbf000b5cd/flink-python/pyflink/testing/test_case_utils.py When
>>> I have some simple SQL everything is fine. When I add a more complex query
>>> I get an error, which looks like it's memory related.
>>>
>>> java.lang.IllegalArgumentException: The configured managed memory fraction
>>> for Python worker process must be within (0, 1], was: %s. It may be because
>>> the consumer type "Python" was missing or set to 0 for the config option
>>> "taskmanager.memory.managed.consumer-weights".0.0
>>>
>>>
>>>
>>> In my test case setUp(), I try to set that value like this, but it seems
>>> to have no effect:
>>>
>>> self.t_env.get_config().get_configuration().set_string("taskmanager.memory.managed.consumer-weights",
>>> "PYTHON:30")
>>>
>>>
>>> Am I not setting it correctly, or is there something else I need to do
>>> to fix this error?
>>>
>>> Many thanks,
>>>
>>> John
>>>
>>>
Re: The configured managed memory fraction for Python worker process must be within (0, 1], was: %s
Posted by John Tipper <jo...@hotmail.com>.
Thanks Dian - I’m restricted to 1.13 for the moment because this is running inside AWS Kinesis. Is there a way to manually bypass that issue?
J
Sent from my iPhone
On 17 Jun 2022, at 04:59, Dian Fu <di...@gmail.com> wrote:
>> This error generally occurs in jobs where there are transfers between Table and datastream.
AFAIK, this issue should have already been fixed, see https://issues.apache.org/jira/browse/FLINK-26920 and https://issues.apache.org/jira/browse/FLINK-23133 for more details.
Regards,
Dian
On Fri, Jun 17, 2022 at 10:17 AM Xingbo Huang <hx...@gmail.com>> wrote:
Hi John,
Because I can't see your code, I can only provide some possible reasons for this error:
1. This error generally occurs in jobs where there are transfers between Table and datastream. But given that you said you just used the sql + python udf, this shouldn't be the case.
2. The default value of `taskmanager.memory.managed.consumer-weights` is `OPERATOR:70,STATE_BACKEND:70,PYTHON:30`, so in your case, there is actually no need to set it to `PYTHON:30`
3. In fact, for pure sql+python udf jobs, if you don't set error value `PYTHON:0` in `taskmanager.memory.managed.consumer-weights`, I really can't think of any situation where this problem will occur.
Best,
Xingbo
John Tipper <jo...@hotmail.com>> 于2022年6月16日周四 19:41写道：
Hi Xingbo,
Yes, there are a number of temporary views being created, where each is being created using SQL (CREATE TEMPORARY VIEW ...) rather than explicit calls to the Table and DataStream APIs.
Is this a good pattern or are there caveats I should be aware of please?
Many thanks,
John
________________________________
From: Xingbo Huang <hx...@gmail.com>>
Sent: 16 June 2022 12:34
To: John Tipper <jo...@hotmail.com>>
Cc: user@flink.apache.org<ma...@flink.apache.org> <us...@flink.apache.org>>
Subject: Re: The configured managed memory fraction for Python worker process must be within (0, 1], was: %s
Hi John,
Does your job logic include conversion between Table and DataStream? For example, methods such as `create_temporary_view(path: str, data_stream: DataStream): -> Table` are used.
Best,
Xingbo
John Tipper <jo...@hotmail.com>> 于2022年6月16日周四 18:31写道：
Hi Xingbo,
I’m afraid I can’t share my code but Flink is 1.13. The main Flink code is running inside Kinesis on AWS so I cannot change the version.
Many thanks,
John
Sent from my iPhone
On 16 Jun 2022, at 10:37, Xingbo Huang <hx...@gmail.com>> wrote:
Hi John,
Could you provide the code snippet and the version of pyflink you used?
Best,
Xingbo
John Tipper <jo...@hotmail.com>> 于2022年6月16日周四 17:05写道：
Hi all,
I'm trying to run a PyFlink unit test to test some PyFlink SQL and where my code uses a Python UDF. I can't share my code but the test case is similar to the code here: https://github.com/apache/flink/blob/f8172cdbbc27344896d961be4b0b9cdbf000b5cd/flink-python/pyflink/testing/test_case_utils.py When I have some simple SQL everything is fine. When I add a more complex query I get an error, which looks like it's memory related.
java.lang.IllegalArgumentException: The configured managed memory fraction
for Python worker process must be within (0, 1], was: %s. It may be because
the consumer type "Python" was missing or set to 0 for the config option
"taskmanager.memory.managed.consumer-weights".0.0
In my test case setUp(), I try to set that value like this, but it seems to have no effect:
self.t_env.get_config().get_configuration().set_string("taskmanager.memory.managed.consumer-weights", "PYTHON:30")
Am I not setting it correctly, or is there something else I need to do to fix this error?
Many thanks,
John
Re: The configured managed memory fraction for Python worker process must be within (0, 1], was: %s
Posted by Dian Fu <di...@gmail.com>.
>> This error generally occurs in jobs where there are transfers between
Table and datastream.
AFAIK, this issue should have already been fixed, see
https://issues.apache.org/jira/browse/FLINK-26920 and
https://issues.apache.org/jira/browse/FLINK-23133 for more details.
Regards,
Dian
On Fri, Jun 17, 2022 at 10:17 AM Xingbo Huang <hx...@gmail.com> wrote:
> Hi John,
>
> Because I can't see your code, I can only provide some possible reasons
> for this error:
> 1. This error generally occurs in jobs where there are transfers between
> Table and datastream. But given that you said you just used the sql +
> python udf, this shouldn't be the case.
> 2. The default value of `taskmanager.memory.managed.consumer-weights` is
> `OPERATOR:70,STATE_BACKEND:70,PYTHON:30`, so in your case, there is
> actually no need to set it to `PYTHON:30`
> 3. In fact, for pure sql+python udf jobs, if you don't set error value
> `PYTHON:0` in `taskmanager.memory.managed.consumer-weights`, I really can't
> think of any situation where this problem will occur.
>
> Best,
> Xingbo
>
> John Tipper <jo...@hotmail.com> 于2022年6月16日周四 19:41写道：
>
>> Hi Xingbo,
>>
>> Yes, there are a number of temporary views being created, where each is
>> being created using SQL (CREATE TEMPORARY VIEW ...) rather than explicit
>> calls to the Table and DataStream APIs.
>>
>> Is this a good pattern or are there caveats I should be aware of please?
>>
>> Many thanks,
>>
>> John
>>
>>
>> ------------------------------
>> *From:* Xingbo Huang <hx...@gmail.com>
>> *Sent:* 16 June 2022 12:34
>> *To:* John Tipper <jo...@hotmail.com>
>> *Cc:* user@flink.apache.org <us...@flink.apache.org>
>> *Subject:* Re: The configured managed memory fraction for Python worker
>> process must be within (0, 1], was: %s
>>
>> Hi John,
>>
>> Does your job logic include conversion between Table and DataStream? For
>> example, methods such as `create_temporary_view(path: str, data_stream:
>> DataStream): -> Table` are used.
>>
>> Best,
>> Xingbo
>>
>> John Tipper <jo...@hotmail.com> 于2022年6月16日周四 18:31写道：
>>
>> Hi Xingbo,
>>
>> I’m afraid I can’t share my code but Flink is 1.13. The main Flink code
>> is running inside Kinesis on AWS so I cannot change the version.
>>
>> Many thanks,
>>
>> John
>>
>> Sent from my iPhone
>>
>> On 16 Jun 2022, at 10:37, Xingbo Huang <hx...@gmail.com> wrote:
>>
>>
>> Hi John,
>>
>> Could you provide the code snippet and the version of pyflink you used?
>>
>> Best,
>> Xingbo
>>
>>
>> John Tipper <jo...@hotmail.com> 于2022年6月16日周四 17:05写道：
>>
>> Hi all,
>>
>> I'm trying to run a PyFlink unit test to test some PyFlink SQL and where
>> my code uses a Python UDF. I can't share my code but the test case is
>> similar to the code here:
>> https://github.com/apache/flink/blob/f8172cdbbc27344896d961be4b0b9cdbf000b5cd/flink-python/pyflink/testing/test_case_utils.py When
>> I have some simple SQL everything is fine. When I add a more complex query
>> I get an error, which looks like it's memory related.
>>
>> java.lang.IllegalArgumentException: The configured managed memory fraction
>> for Python worker process must be within (0, 1], was: %s. It may be because
>> the consumer type "Python" was missing or set to 0 for the config option
>> "taskmanager.memory.managed.consumer-weights".0.0
>>
>>
>>
>> In my test case setUp(), I try to set that value like this, but it seems
>> to have no effect:
>>
>> self.t_env.get_config().get_configuration().set_string("taskmanager.memory.managed.consumer-weights",
>> "PYTHON:30")
>>
>>
>> Am I not setting it correctly, or is there something else I need to do to
>> fix this error?
>>
>> Many thanks,
>>
>> John
>>
>>
Re: The configured managed memory fraction for Python worker process must be within (0, 1], was: %s
Posted by Xingbo Huang <hx...@gmail.com>.
Hi John,
Because I can't see your code, I can only provide some possible reasons for
this error:
1. This error generally occurs in jobs where there are transfers between
Table and datastream. But given that you said you just used the sql +
python udf, this shouldn't be the case.
2. The default value of `taskmanager.memory.managed.consumer-weights` is
`OPERATOR:70,STATE_BACKEND:70,PYTHON:30`, so in your case, there is
actually no need to set it to `PYTHON:30`
3. In fact, for pure sql+python udf jobs, if you don't set error value
`PYTHON:0` in `taskmanager.memory.managed.consumer-weights`, I really can't
think of any situation where this problem will occur.
Best,
Xingbo
John Tipper <jo...@hotmail.com> 于2022年6月16日周四 19:41写道：
> Hi Xingbo,
>
> Yes, there are a number of temporary views being created, where each is
> being created using SQL (CREATE TEMPORARY VIEW ...) rather than explicit
> calls to the Table and DataStream APIs.
>
> Is this a good pattern or are there caveats I should be aware of please?
>
> Many thanks,
>
> John
>
>
> ------------------------------
> *From:* Xingbo Huang <hx...@gmail.com>
> *Sent:* 16 June 2022 12:34
> *To:* John Tipper <jo...@hotmail.com>
> *Cc:* user@flink.apache.org <us...@flink.apache.org>
> *Subject:* Re: The configured managed memory fraction for Python worker
> process must be within (0, 1], was: %s
>
> Hi John,
>
> Does your job logic include conversion between Table and DataStream? For
> example, methods such as `create_temporary_view(path: str, data_stream:
> DataStream): -> Table` are used.
>
> Best,
> Xingbo
>
> John Tipper <jo...@hotmail.com> 于2022年6月16日周四 18:31写道：
>
> Hi Xingbo,
>
> I’m afraid I can’t share my code but Flink is 1.13. The main Flink code is
> running inside Kinesis on AWS so I cannot change the version.
>
> Many thanks,
>
> John
>
> Sent from my iPhone
>
> On 16 Jun 2022, at 10:37, Xingbo Huang <hx...@gmail.com> wrote:
>
>
> Hi John,
>
> Could you provide the code snippet and the version of pyflink you used?
>
> Best,
> Xingbo
>
>
> John Tipper <jo...@hotmail.com> 于2022年6月16日周四 17:05写道：
>
> Hi all,
>
> I'm trying to run a PyFlink unit test to test some PyFlink SQL and where
> my code uses a Python UDF. I can't share my code but the test case is
> similar to the code here:
> https://github.com/apache/flink/blob/f8172cdbbc27344896d961be4b0b9cdbf000b5cd/flink-python/pyflink/testing/test_case_utils.py When
> I have some simple SQL everything is fine. When I add a more complex query
> I get an error, which looks like it's memory related.
>
> java.lang.IllegalArgumentException: The configured managed memory fraction
> for Python worker process must be within (0, 1], was: %s. It may be because
> the consumer type "Python" was missing or set to 0 for the config option
> "taskmanager.memory.managed.consumer-weights".0.0
>
>
>
> In my test case setUp(), I try to set that value like this, but it seems
> to have no effect:
>
> self.t_env.get_config().get_configuration().set_string("taskmanager.memory.managed.consumer-weights",
> "PYTHON:30")
>
>
> Am I not setting it correctly, or is there something else I need to do to
> fix this error?
>
> Many thanks,
>
> John
>
>
Re: The configured managed memory fraction for Python worker process must be within (0, 1], was: %s
Posted by John Tipper <jo...@hotmail.com>.
Hi Xingbo,
Yes, there are a number of temporary views being created, where each is being created using SQL (CREATE TEMPORARY VIEW ...) rather than explicit calls to the Table and DataStream APIs.
Is this a good pattern or are there caveats I should be aware of please?
Many thanks,
John
________________________________
From: Xingbo Huang <hx...@gmail.com>
Sent: 16 June 2022 12:34
To: John Tipper <jo...@hotmail.com>
Cc: user@flink.apache.org <us...@flink.apache.org>
Subject: Re: The configured managed memory fraction for Python worker process must be within (0, 1], was: %s
Hi John,
Does your job logic include conversion between Table and DataStream? For example, methods such as `create_temporary_view(path: str, data_stream: DataStream): -> Table` are used.
Best,
Xingbo
John Tipper <jo...@hotmail.com>> 于2022年6月16日周四 18:31写道：
Hi Xingbo,
I’m afraid I can’t share my code but Flink is 1.13. The main Flink code is running inside Kinesis on AWS so I cannot change the version.
Many thanks,
John
Sent from my iPhone
On 16 Jun 2022, at 10:37, Xingbo Huang <hx...@gmail.com>> wrote:
Hi John,
Could you provide the code snippet and the version of pyflink you used?
Best,
Xingbo
John Tipper <jo...@hotmail.com>> 于2022年6月16日周四 17:05写道：
Hi all,
I'm trying to run a PyFlink unit test to test some PyFlink SQL and where my code uses a Python UDF. I can't share my code but the test case is similar to the code here: https://github.com/apache/flink/blob/f8172cdbbc27344896d961be4b0b9cdbf000b5cd/flink-python/pyflink/testing/test_case_utils.py When I have some simple SQL everything is fine. When I add a more complex query I get an error, which looks like it's memory related.
java.lang.IllegalArgumentException: The configured managed memory fraction
for Python worker process must be within (0, 1], was: %s. It may be because
the consumer type "Python" was missing or set to 0 for the config option
"taskmanager.memory.managed.consumer-weights".0.0
In my test case setUp(), I try to set that value like this, but it seems to have no effect:
self.t_env.get_config().get_configuration().set_string("taskmanager.memory.managed.consumer-weights", "PYTHON:30")
Am I not setting it correctly, or is there something else I need to do to fix this error?
Many thanks,
John
Re: The configured managed memory fraction for Python worker process must be within (0, 1], was: %s
Posted by Xingbo Huang <hx...@gmail.com>.
Hi John,
Does your job logic include conversion between Table and DataStream? For
example, methods such as `create_temporary_view(path: str, data_stream:
DataStream): -> Table` are used.
Best,
Xingbo
John Tipper <jo...@hotmail.com> 于2022年6月16日周四 18:31写道：
> Hi Xingbo,
>
> I’m afraid I can’t share my code but Flink is 1.13. The main Flink code is
> running inside Kinesis on AWS so I cannot change the version.
>
> Many thanks,
>
> John
>
> Sent from my iPhone
>
> On 16 Jun 2022, at 10:37, Xingbo Huang <hx...@gmail.com> wrote:
>
>
> Hi John,
>
> Could you provide the code snippet and the version of pyflink you used?
>
> Best,
> Xingbo
>
>
> John Tipper <jo...@hotmail.com> 于2022年6月16日周四 17:05写道：
>
>> Hi all,
>>
>> I'm trying to run a PyFlink unit test to test some PyFlink SQL and where
>> my code uses a Python UDF. I can't share my code but the test case is
>> similar to the code here:
>> https://github.com/apache/flink/blob/f8172cdbbc27344896d961be4b0b9cdbf000b5cd/flink-python/pyflink/testing/test_case_utils.py When
>> I have some simple SQL everything is fine. When I add a more complex query
>> I get an error, which looks like it's memory related.
>>
>> java.lang.IllegalArgumentException: The configured managed memory fraction
>> for Python worker process must be within (0, 1], was: %s. It may be because
>> the consumer type "Python" was missing or set to 0 for the config option
>> "taskmanager.memory.managed.consumer-weights".0.0
>>
>>
>>
>> In my test case setUp(), I try to set that value like this, but it seems
>> to have no effect:
>>
>> self.t_env.get_config().get_configuration().set_string("taskmanager.memory.managed.consumer-weights",
>> "PYTHON:30")
>>
>>
>> Am I not setting it correctly, or is there something else I need to do to
>> fix this error?
>>
>> Many thanks,
>>
>> John
>>
>>
Re: The configured managed memory fraction for Python worker process must be within (0, 1], was: %s
Posted by John Tipper <jo...@hotmail.com>.
Hi Xingbo,
I’m afraid I can’t share my code but Flink is 1.13. The main Flink code is running inside Kinesis on AWS so I cannot change the version.
Many thanks,
John
Sent from my iPhone
On 16 Jun 2022, at 10:37, Xingbo Huang <hx...@gmail.com> wrote:
Hi John,
Could you provide the code snippet and the version of pyflink you used?
Best,
Xingbo
John Tipper <jo...@hotmail.com>> 于2022年6月16日周四 17:05写道：
Hi all,
I'm trying to run a PyFlink unit test to test some PyFlink SQL and where my code uses a Python UDF. I can't share my code but the test case is similar to the code here: https://github.com/apache/flink/blob/f8172cdbbc27344896d961be4b0b9cdbf000b5cd/flink-python/pyflink/testing/test_case_utils.py When I have some simple SQL everything is fine. When I add a more complex query I get an error, which looks like it's memory related.
java.lang.IllegalArgumentException: The configured managed memory fraction
for Python worker process must be within (0, 1], was: %s. It may be because
the consumer type "Python" was missing or set to 0 for the config option
"taskmanager.memory.managed.consumer-weights".0.0
In my test case setUp(), I try to set that value like this, but it seems to have no effect:
self.t_env.get_config().get_configuration().set_string("taskmanager.memory.managed.consumer-weights", "PYTHON:30")
Am I not setting it correctly, or is there something else I need to do to fix this error?
Many thanks,
John
Re: The configured managed memory fraction for Python worker process must be within (0, 1], was: %s
Posted by Xingbo Huang <hx...@gmail.com>.
Hi John,
Could you provide the code snippet and the version of pyflink you used?
Best,
Xingbo
John Tipper <jo...@hotmail.com> 于2022年6月16日周四 17:05写道：
> Hi all,
>
> I'm trying to run a PyFlink unit test to test some PyFlink SQL and where
> my code uses a Python UDF. I can't share my code but the test case is
> similar to the code here:
> https://github.com/apache/flink/blob/f8172cdbbc27344896d961be4b0b9cdbf000b5cd/flink-python/pyflink/testing/test_case_utils.py When
> I have some simple SQL everything is fine. When I add a more complex query
> I get an error, which looks like it's memory related.
>
> java.lang.IllegalArgumentException: The configured managed memory fraction
> for Python worker process must be within (0, 1], was: %s. It may be because
> the consumer type "Python" was missing or set to 0 for the config option
> "taskmanager.memory.managed.consumer-weights".0.0
>
>
>
> In my test case setUp(), I try to set that value like this, but it seems
> to have no effect:
>
> self.t_env.get_config().get_configuration().set_string("taskmanager.memory.managed.consumer-weights",
> "PYTHON:30")
>
>
> Am I not setting it correctly, or is there something else I need to do to
> fix this error?
>
> Many thanks,
>
> John
>
>