You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Lydian Lee <ti...@gmail.com> on 2025/01/26 23:53:41 UTC
Unable to run in python-worker-harness after bumpping from 2.41.0 to 2.60.0
Hi,
We are trying to bump an old pipeline using flink runner and beam python
SDK.
The version changes are:
- Flink: 1.15.4 -> 1.18.0
- Beam: 2.41.0 -> 2.60.0
- python: 3.9. (no change)
However, after bumping the version, we noticed that the
python-worker-harness is unable to process function properly. Here's the
error log we kept received:
```
Starting worker with command ['/opt/apache/beam/boot', '--id=6-1',
'--logging_endpoint=localhost:46759',
'--artifact_endpoint=localhost:44123',
'--provision_endpoint=localhost:42447', '--control_endpoint=localho
2025/01/26 23:41:26 Provision info:
pipeline_options:{fields:{key:"beam:option:allow_non_deterministic_key_coders:v1"
value:{bool_value:false}}
fields:{key:"beam:option:allow_non_restored_state:v1"
value:{bool_value:false}} fields:{key:"beam:o
2025/01/26 23:41:44 boot.go: error logging message over FnAPI. endpoint
localhost:46759 error: EOF message follows
2025/01/26 23:41:44 DEBUG Received signal: terminated 2025/01/26 23:41:44
boot.go: error logging message over FnAPI. endpoint localhost:46759 error:
EOF message follows
2025/01/26 23:41:44 WARN Python (worker 6-1) exited 1 times: signal:
terminated restarting SDK process
```
It probably unrelated to the job itself as the job was working fine in the
earlier version:
[image: Untitled.jpeg]
Wondering if anyone have any idea how I can fix this issue? Thanks
Re: Unable to run in python-worker-harness after bumpping from 2.41.0 to 2.60.0
Posted by Jaehyeon Kim <do...@gmail.com>.
I also had the same log message but it was okay for me. For s3 access, it
should be fine as long as the app can recognise aws credentials.
Checked my earlier failure log and it was because the python harness
process failed to get started. If it happens, it gets discarded and a new
process is started with a new id (1-1, 2-1, ...).
2024-05-23 22:31:52,733 WARN org.apache.flink.metrics.MetricGroup
[] - The operator name Source:
ReadWordsFromKafka/ReadFromKafka/KafkaIO.Read/
KafkaIO.Read.ReadFromKafkaViaUnbounded/Read(KafkaUnboundedSource) exceeded
the 80 characters length limit and was truncated.
2024-05-23 22:31:53,911 INFO
org.apache.beam.runners.fnexecution.logging.GrpcLoggingService [] - Beam Fn
Logging client connected.
2024-05-23 22:31:53,934 INFO
org.apache.beam.runners.fnexecution.logging.GrpcLoggingService [] - Beam Fn
Logging client connected.
2024-05-23 22:31:58,571 INFO
org.apache.beam.runners.fnexecution.environment.ProcessEnvironmentFactory
[] - Still waiting for startup of environment '/opt/apache/beam/boot' for
worker id 1-1
2024-05-23 22:32:00,844 INFO
org.apache.beam.runners.fnexecution.logging.GrpcLoggingService [] - Beam Fn
Logging client connected.
...
2024-05-23 22:32:00,872 INFO /usr/local/lib/python3.10
/site-packages/apache_beam/runners/worker/sdk_worker.py:247 [] -
Initializing SDKHarness with unbounded number of workers.
2024-05-23 22:32:00,872 INFO /usr/local/lib/python3.10
/site-packages/apache_beam/runners/worker/sdk_worker_main.py:211 [] -
Python sdk harness starting.
...
2024-05-23 22:32:03,572 INFO
org.apache.beam.runners.fnexecution.environment.ProcessEnvironmentFactory
[] - Still waiting for startup of environment '/opt/apache/beam/boot' for
worker id 1-1
2024-05-23 22:32:05,959 WARN
org.apache.beam.runners.fnexecution.logging.GrpcLoggingService [] - Logging
client failed unexpectedly.
org.apache.beam.vendor.grpc.v1p60p1.io.grpc.StatusRuntimeException:
CANCELLED: client cancelled
...
2024-05-23 22:32:08,572 INFO
org.apache.beam.runners.fnexecution.environment.ProcessEnvironmentFactory
[] - Still waiting for startup of environment '/opt/apache/beam/boot' for
worker id 1-1
2024-05-23 22:32:08,581 WARN org.apache.flink.runtime.taskmanager.Task
[] -
CalculateAvgWordLen/GetAvgWordLength/CombinePerKey/GroupByKey ->
[4]{CalculateAvgWordLen,
WriteWordLenToKafka} -> [2]WriteWordLenToKafka/WriteToKafka/{Kafka
ProducerRecord, KafkaIO.WriteRecords} (1/1)#0
(6c7c9be7467596ad77dc91f9aeb1683e_fa7f845a998dd8979a79c90087d7ce62_0_0)
switched from INITIALIZING to FAILED with failure cause:
org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.util.concurrent.UncheckedExecutionException:
java.lang.IllegalStateException: Process died with exit code 1
...
Caused by: java.lang.IllegalStateException: Process died with exit code 1
at org.apache.beam.runners.fnexecution.environment.ProcessManager$
RunningProcess.isAliveOrThrow(ProcessManager.java:75)
at
org.apache.beam.runners.fnexecution.environment.ProcessEnvironmentFactory.createEnvironment
(ProcessEnvironmentFactory.java:110)
at org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory$1.
load(DefaultJobBundleFactory.java:259)
at org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory$1.
load(DefaultJobBundleFactory.java:232)
at
org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.cache.LocalCache$
LoadingValueReference.loadFuture(LocalCache.java:3571)
at
org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.cache.LocalCache$
Segment.loadSync(LocalCache.java:2313)
at
org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.cache.LocalCache$
Segment.lockedGetOrLoad(LocalCache.java:2190)
at
org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.cache.LocalCache$
Segment.get(LocalCache.java:2080)
... 20 more
On Thu, 30 Jan 2025 at 20:10, Lydian Lee <ti...@gmail.com> wrote:
> I actually found an interesting log line from task manager when I tried to
> run with PROCESS:
>
> ```
> python3.9/site-packages/apache_beam/runners/worker/sdk_worker_main.py:134
> [] - Pipeline_options: {'streaming': True, 'job_name':
> 'ml-ofs.debit-amplify-transaction-v2-2-test', 'gcp_oauth_scopes': ['
> https://www.googleapis.com/auth/bigquery', '
> https://www.googleapis.com/auth/cloud-platform', '
> https://www.googleapis.com/auth/devstorage.full_control', '
> https://www.googleapis.com/auth/userinfo.email', '
> https://www.googleapis.com/auth/datastore', '
> https://www.googleapis.com/auth/spanner.admin', '
> https://www.googleapis.com/auth/spanner.data', '
> https://www.googleapis.com/auth/bigquery', '
> https://www.googleapis.com/auth/cloud-platform', '
> https://www.googleapis.com/auth/devstorage.full_control', '
> https://www.googleapis.com/auth/userinfo.email', '
> https://www.googleapis.com/auth/datastore', '
> https://www.googleapis.com/auth/spanner.admin', '
> https://www.googleapis.com/auth/spanner.data'], 'experiments':
> ['disable_logging_submission_environment', 'beam_fn_api'], 'sdk_location':
> 'container', 'job_endpoint': 'localhost:37379', 'environment_type':
> 'PROCESS', 'environment_config': '{"command":"/opt/apache/beam/boot"}',
> 'sdk_worker_parallelism': '1', 'environment_cache_millis': '0'}
> ```
>
> However, my job is configured to run with S3, and it seems like by
> default it keeps trying to use gcp. Is there any way I can avoid that?
> (Not sure if that is the main reason which causes the issue)
>
>
>
> On Thu, Jan 30, 2025 at 12:28 AM Lydian Lee <ti...@gmail.com>
> wrote:
>
>> I did see something like this in the flink-main-container, but I think it
>> is because the harness is terminated, and then causes the error in the
>> flink-main-container.
>> Or do you think it is actually the other way?
>>
>> [image: Untitled 2.png]
>>
>> On Wed, Jan 29, 2025 at 1:15 PM Jaehyeon Kim <do...@gmail.com> wrote:
>>
>>> Do you see something like this in task manager logs? I encountered this
>>> error when I deployed my pipeline via the application deployment mode with
>>> Beam 2.56.0 and Flink 1.16.
>>>
>>> [image: image.png]
>>>
>>> On Thu, 30 Jan 2025 at 04:52, Lydian Lee <ti...@gmail.com>
>>> wrote:
>>>
>>>> Hi Jaehyeon,
>>>>
>>>> Do you have any suggestions on how I can show more information for
>>>> debugging this python harness process error? Thanks for your help.
>>>>
>>>>
>>>> On Mon, Jan 27, 2025 at 5:44 PM Jaehyeon Kim <do...@gmail.com> wrote:
>>>>
>>>>> Your original log messages include error messages in this file -
>>>>> https://github.com/apache/beam/blob/master/sdks/python/container/boot.go#L253
>>>>> .
>>>>>
>>>>> It may be due to failure of the Python harness process.
>>>>>
>>>>> On Tue, 28 Jan 2025 at 06:22, Lydian Lee <ti...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> It seems like there's also similar issue from others:
>>>>>> https://github.com/apache/beam/issues/29683
>>>>>>
>>>>>> For me, here's how I launch the job with the following argos:
>>>>>> --runner=portableRunner
>>>>>> --streaming
>>>>>> --environment_type=EXTERNAL
>>>>>> --environment_config=localhost:50000
>>>>>> --experiments=disable_logging_submission_environment # without this,
>>>>>> it is complaining on rpc error: code = Unknown desc = ; failed to retrieve
>>>>>> chunk for /tmp/staged/submission_environment_dependencies.txt
>>>>>>
>>>>>> For launching the harness:
>>>>>> /opt/apache/beam/boot --worker_pool
>>>>>>
>>>>>>
>>>>>> I've tested the different pipeline which all works in 2.41.0, but now
>>>>>> they all failed with 2.60.0. I've also tested with 2.57.0, and that one
>>>>>> seems to already have the same issue.
>>>>>>
>>>>>> Something worth mentioning is I also need to install
>>>>>> `google-cloud-storage` when bumping the version, otherwise it will complain
>>>>>> about missing the google cloud library, although I am not using gcs at all.
>>>>>> ```
>>>>>> Starting worker with command ['/opt/apache/beam/boot', '--id=31-1',
>>>>>> '--logging_endpoint=localhost:40649',
>>>>>> '--artifact_endpoint=localhost:36751',
>>>>>> '--provision_endpoint=localhost:45679', '--control_endpoint=localh 2
>>>>>> 025/01/23 23:46:14 Provision info:
>>>>>> pipeline_options:{fields:{key:"beam:option:allow_non_deterministic_key_coders:v1"
>>>>>> value:{bool_value:false}}
>>>>>> fields:{key:"beam:option:allow_non_restored_state:v1"
>>>>>> value:{bool_value:false}} fields:{key:"beam:o 2
>>>>>> 025/01/23 23:46:28 boot.go: error logging message over FnAPI.
>>>>>> endpoint localhost:40649 error: EOF message follows
>>>>>> 2025/01/23 23:46:28 DEBUG Received signal: terminated
>>>>>> 2025/01/23 23:46:28 boot.go: error logging message over FnAPI.
>>>>>> endpoint localhost:40649 error: EOF message follows
>>>>>> 2025/01/23 23:46:28 ERROR Failed to import GCSFileSystem; loading of
>>>>>> this filesystem will be skipped. Error details: cannot import name
>>>>>> 'storage' from 'google.cloud' (unknown location)
>>>>>> ```
>>>>>>
>>>>>> Thanks for your help.
>>>>>>
>>>>>
Re: Unable to run in python-worker-harness after bumpping from 2.41.0 to 2.60.0
Posted by Lydian Lee <ti...@gmail.com>.
I actually found an interesting log line from task manager when I tried to
run with PROCESS:
```
python3.9/site-packages/apache_beam/runners/worker/sdk_worker_main.py:134
[] - Pipeline_options: {'streaming': True, 'job_name':
'ml-ofs.debit-amplify-transaction-v2-2-test', 'gcp_oauth_scopes': ['
https://www.googleapis.com/auth/bigquery', '
https://www.googleapis.com/auth/cloud-platform', '
https://www.googleapis.com/auth/devstorage.full_control', '
https://www.googleapis.com/auth/userinfo.email', '
https://www.googleapis.com/auth/datastore', '
https://www.googleapis.com/auth/spanner.admin', '
https://www.googleapis.com/auth/spanner.data', '
https://www.googleapis.com/auth/bigquery', '
https://www.googleapis.com/auth/cloud-platform', '
https://www.googleapis.com/auth/devstorage.full_control', '
https://www.googleapis.com/auth/userinfo.email', '
https://www.googleapis.com/auth/datastore', '
https://www.googleapis.com/auth/spanner.admin', '
https://www.googleapis.com/auth/spanner.data'], 'experiments':
['disable_logging_submission_environment', 'beam_fn_api'], 'sdk_location':
'container', 'job_endpoint': 'localhost:37379', 'environment_type':
'PROCESS', 'environment_config': '{"command":"/opt/apache/beam/boot"}',
'sdk_worker_parallelism': '1', 'environment_cache_millis': '0'}
```
However, my job is configured to run with S3, and it seems like by default
it keeps trying to use gcp. Is there any way I can avoid that? (Not sure
if that is the main reason which causes the issue)
On Thu, Jan 30, 2025 at 12:28 AM Lydian Lee <ti...@gmail.com> wrote:
> I did see something like this in the flink-main-container, but I think it
> is because the harness is terminated, and then causes the error in the
> flink-main-container.
> Or do you think it is actually the other way?
>
> [image: Untitled 2.png]
>
> On Wed, Jan 29, 2025 at 1:15 PM Jaehyeon Kim <do...@gmail.com> wrote:
>
>> Do you see something like this in task manager logs? I encountered this
>> error when I deployed my pipeline via the application deployment mode with
>> Beam 2.56.0 and Flink 1.16.
>>
>> [image: image.png]
>>
>> On Thu, 30 Jan 2025 at 04:52, Lydian Lee <ti...@gmail.com> wrote:
>>
>>> Hi Jaehyeon,
>>>
>>> Do you have any suggestions on how I can show more information for
>>> debugging this python harness process error? Thanks for your help.
>>>
>>>
>>> On Mon, Jan 27, 2025 at 5:44 PM Jaehyeon Kim <do...@gmail.com> wrote:
>>>
>>>> Your original log messages include error messages in this file -
>>>> https://github.com/apache/beam/blob/master/sdks/python/container/boot.go#L253
>>>> .
>>>>
>>>> It may be due to failure of the Python harness process.
>>>>
>>>> On Tue, 28 Jan 2025 at 06:22, Lydian Lee <ti...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> It seems like there's also similar issue from others:
>>>>> https://github.com/apache/beam/issues/29683
>>>>>
>>>>> For me, here's how I launch the job with the following argos:
>>>>> --runner=portableRunner
>>>>> --streaming
>>>>> --environment_type=EXTERNAL
>>>>> --environment_config=localhost:50000
>>>>> --experiments=disable_logging_submission_environment # without this,
>>>>> it is complaining on rpc error: code = Unknown desc = ; failed to retrieve
>>>>> chunk for /tmp/staged/submission_environment_dependencies.txt
>>>>>
>>>>> For launching the harness:
>>>>> /opt/apache/beam/boot --worker_pool
>>>>>
>>>>>
>>>>> I've tested the different pipeline which all works in 2.41.0, but now
>>>>> they all failed with 2.60.0. I've also tested with 2.57.0, and that one
>>>>> seems to already have the same issue.
>>>>>
>>>>> Something worth mentioning is I also need to install
>>>>> `google-cloud-storage` when bumping the version, otherwise it will complain
>>>>> about missing the google cloud library, although I am not using gcs at all.
>>>>> ```
>>>>> Starting worker with command ['/opt/apache/beam/boot', '--id=31-1',
>>>>> '--logging_endpoint=localhost:40649',
>>>>> '--artifact_endpoint=localhost:36751',
>>>>> '--provision_endpoint=localhost:45679', '--control_endpoint=localh 2
>>>>> 025/01/23 23:46:14 Provision info:
>>>>> pipeline_options:{fields:{key:"beam:option:allow_non_deterministic_key_coders:v1"
>>>>> value:{bool_value:false}}
>>>>> fields:{key:"beam:option:allow_non_restored_state:v1"
>>>>> value:{bool_value:false}} fields:{key:"beam:o 2
>>>>> 025/01/23 23:46:28 boot.go: error logging message over FnAPI. endpoint
>>>>> localhost:40649 error: EOF message follows
>>>>> 2025/01/23 23:46:28 DEBUG Received signal: terminated
>>>>> 2025/01/23 23:46:28 boot.go: error logging message over FnAPI.
>>>>> endpoint localhost:40649 error: EOF message follows
>>>>> 2025/01/23 23:46:28 ERROR Failed to import GCSFileSystem; loading of
>>>>> this filesystem will be skipped. Error details: cannot import name
>>>>> 'storage' from 'google.cloud' (unknown location)
>>>>> ```
>>>>>
>>>>> Thanks for your help.
>>>>>
>>>>
Re: Unable to run in python-worker-harness after bumpping from 2.41.0 to 2.60.0
Posted by Lydian Lee <ti...@gmail.com>.
I did see something like this in the flink-main-container, but I think it
is because the harness is terminated, and then causes the error in the
flink-main-container.
Or do you think it is actually the other way?
[image: Untitled 2.png]
On Wed, Jan 29, 2025 at 1:15 PM Jaehyeon Kim <do...@gmail.com> wrote:
> Do you see something like this in task manager logs? I encountered this
> error when I deployed my pipeline via the application deployment mode with
> Beam 2.56.0 and Flink 1.16.
>
> [image: image.png]
>
> On Thu, 30 Jan 2025 at 04:52, Lydian Lee <ti...@gmail.com> wrote:
>
>> Hi Jaehyeon,
>>
>> Do you have any suggestions on how I can show more information for
>> debugging this python harness process error? Thanks for your help.
>>
>>
>> On Mon, Jan 27, 2025 at 5:44 PM Jaehyeon Kim <do...@gmail.com> wrote:
>>
>>> Your original log messages include error messages in this file -
>>> https://github.com/apache/beam/blob/master/sdks/python/container/boot.go#L253
>>> .
>>>
>>> It may be due to failure of the Python harness process.
>>>
>>> On Tue, 28 Jan 2025 at 06:22, Lydian Lee <ti...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> It seems like there's also similar issue from others:
>>>> https://github.com/apache/beam/issues/29683
>>>>
>>>> For me, here's how I launch the job with the following argos:
>>>> --runner=portableRunner
>>>> --streaming
>>>> --environment_type=EXTERNAL
>>>> --environment_config=localhost:50000
>>>> --experiments=disable_logging_submission_environment # without this,
>>>> it is complaining on rpc error: code = Unknown desc = ; failed to retrieve
>>>> chunk for /tmp/staged/submission_environment_dependencies.txt
>>>>
>>>> For launching the harness:
>>>> /opt/apache/beam/boot --worker_pool
>>>>
>>>>
>>>> I've tested the different pipeline which all works in 2.41.0, but now
>>>> they all failed with 2.60.0. I've also tested with 2.57.0, and that one
>>>> seems to already have the same issue.
>>>>
>>>> Something worth mentioning is I also need to install
>>>> `google-cloud-storage` when bumping the version, otherwise it will complain
>>>> about missing the google cloud library, although I am not using gcs at all.
>>>> ```
>>>> Starting worker with command ['/opt/apache/beam/boot', '--id=31-1',
>>>> '--logging_endpoint=localhost:40649',
>>>> '--artifact_endpoint=localhost:36751',
>>>> '--provision_endpoint=localhost:45679', '--control_endpoint=localh 2
>>>> 025/01/23 23:46:14 Provision info:
>>>> pipeline_options:{fields:{key:"beam:option:allow_non_deterministic_key_coders:v1"
>>>> value:{bool_value:false}}
>>>> fields:{key:"beam:option:allow_non_restored_state:v1"
>>>> value:{bool_value:false}} fields:{key:"beam:o 2
>>>> 025/01/23 23:46:28 boot.go: error logging message over FnAPI. endpoint
>>>> localhost:40649 error: EOF message follows
>>>> 2025/01/23 23:46:28 DEBUG Received signal: terminated
>>>> 2025/01/23 23:46:28 boot.go: error logging message over FnAPI. endpoint
>>>> localhost:40649 error: EOF message follows
>>>> 2025/01/23 23:46:28 ERROR Failed to import GCSFileSystem; loading of
>>>> this filesystem will be skipped. Error details: cannot import name
>>>> 'storage' from 'google.cloud' (unknown location)
>>>> ```
>>>>
>>>> Thanks for your help.
>>>>
>>>
Re: Unable to run in python-worker-harness after bumpping from 2.41.0 to 2.60.0
Posted by Jaehyeon Kim <do...@gmail.com>.
Do you see something like this in task manager logs? I encountered this
error when I deployed my pipeline via the application deployment mode with
Beam 2.56.0 and Flink 1.16.
[image: image.png]
On Thu, 30 Jan 2025 at 04:52, Lydian Lee <ti...@gmail.com> wrote:
> Hi Jaehyeon,
>
> Do you have any suggestions on how I can show more information for
> debugging this python harness process error? Thanks for your help.
>
>
> On Mon, Jan 27, 2025 at 5:44 PM Jaehyeon Kim <do...@gmail.com> wrote:
>
>> Your original log messages include error messages in this file -
>> https://github.com/apache/beam/blob/master/sdks/python/container/boot.go#L253
>> .
>>
>> It may be due to failure of the Python harness process.
>>
>> On Tue, 28 Jan 2025 at 06:22, Lydian Lee <ti...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> It seems like there's also similar issue from others:
>>> https://github.com/apache/beam/issues/29683
>>>
>>> For me, here's how I launch the job with the following argos:
>>> --runner=portableRunner
>>> --streaming
>>> --environment_type=EXTERNAL
>>> --environment_config=localhost:50000
>>> --experiments=disable_logging_submission_environment # without this, it
>>> is complaining on rpc error: code = Unknown desc = ; failed to retrieve
>>> chunk for /tmp/staged/submission_environment_dependencies.txt
>>>
>>> For launching the harness:
>>> /opt/apache/beam/boot --worker_pool
>>>
>>>
>>> I've tested the different pipeline which all works in 2.41.0, but now
>>> they all failed with 2.60.0. I've also tested with 2.57.0, and that one
>>> seems to already have the same issue.
>>>
>>> Something worth mentioning is I also need to install
>>> `google-cloud-storage` when bumping the version, otherwise it will complain
>>> about missing the google cloud library, although I am not using gcs at all.
>>> ```
>>> Starting worker with command ['/opt/apache/beam/boot', '--id=31-1',
>>> '--logging_endpoint=localhost:40649',
>>> '--artifact_endpoint=localhost:36751',
>>> '--provision_endpoint=localhost:45679', '--control_endpoint=localh 2
>>> 025/01/23 23:46:14 Provision info:
>>> pipeline_options:{fields:{key:"beam:option:allow_non_deterministic_key_coders:v1"
>>> value:{bool_value:false}}
>>> fields:{key:"beam:option:allow_non_restored_state:v1"
>>> value:{bool_value:false}} fields:{key:"beam:o 2
>>> 025/01/23 23:46:28 boot.go: error logging message over FnAPI. endpoint
>>> localhost:40649 error: EOF message follows
>>> 2025/01/23 23:46:28 DEBUG Received signal: terminated
>>> 2025/01/23 23:46:28 boot.go: error logging message over FnAPI. endpoint
>>> localhost:40649 error: EOF message follows
>>> 2025/01/23 23:46:28 ERROR Failed to import GCSFileSystem; loading of
>>> this filesystem will be skipped. Error details: cannot import name
>>> 'storage' from 'google.cloud' (unknown location)
>>> ```
>>>
>>> Thanks for your help.
>>>
>>
Re: Unable to run in python-worker-harness after bumpping from 2.41.0 to 2.60.0
Posted by Lydian Lee <ti...@gmail.com>.
Hi Jaehyeon,
Do you have any suggestions on how I can show more information for
debugging this python harness process error? Thanks for your help.
On Mon, Jan 27, 2025 at 5:44 PM Jaehyeon Kim <do...@gmail.com> wrote:
> Your original log messages include error messages in this file -
> https://github.com/apache/beam/blob/master/sdks/python/container/boot.go#L253
> .
>
> It may be due to failure of the Python harness process.
>
> On Tue, 28 Jan 2025 at 06:22, Lydian Lee <ti...@gmail.com> wrote:
>
>> Hi,
>>
>> It seems like there's also similar issue from others:
>> https://github.com/apache/beam/issues/29683
>>
>> For me, here's how I launch the job with the following argos:
>> --runner=portableRunner
>> --streaming
>> --environment_type=EXTERNAL
>> --environment_config=localhost:50000
>> --experiments=disable_logging_submission_environment # without this, it
>> is complaining on rpc error: code = Unknown desc = ; failed to retrieve
>> chunk for /tmp/staged/submission_environment_dependencies.txt
>>
>> For launching the harness:
>> /opt/apache/beam/boot --worker_pool
>>
>>
>> I've tested the different pipeline which all works in 2.41.0, but now
>> they all failed with 2.60.0. I've also tested with 2.57.0, and that one
>> seems to already have the same issue.
>>
>> Something worth mentioning is I also need to install
>> `google-cloud-storage` when bumping the version, otherwise it will complain
>> about missing the google cloud library, although I am not using gcs at all.
>> ```
>> Starting worker with command ['/opt/apache/beam/boot', '--id=31-1',
>> '--logging_endpoint=localhost:40649',
>> '--artifact_endpoint=localhost:36751',
>> '--provision_endpoint=localhost:45679', '--control_endpoint=localh 2
>> 025/01/23 23:46:14 Provision info:
>> pipeline_options:{fields:{key:"beam:option:allow_non_deterministic_key_coders:v1"
>> value:{bool_value:false}}
>> fields:{key:"beam:option:allow_non_restored_state:v1"
>> value:{bool_value:false}} fields:{key:"beam:o 2
>> 025/01/23 23:46:28 boot.go: error logging message over FnAPI. endpoint
>> localhost:40649 error: EOF message follows
>> 2025/01/23 23:46:28 DEBUG Received signal: terminated
>> 2025/01/23 23:46:28 boot.go: error logging message over FnAPI. endpoint
>> localhost:40649 error: EOF message follows
>> 2025/01/23 23:46:28 ERROR Failed to import GCSFileSystem; loading of this
>> filesystem will be skipped. Error details: cannot import name 'storage'
>> from 'google.cloud' (unknown location)
>> ```
>>
>> Thanks for your help.
>>
>
Re: Unable to run in python-worker-harness after bumpping from 2.41.0 to 2.60.0
Posted by Jaehyeon Kim <do...@gmail.com>.
Your original log messages include error messages in this file -
https://github.com/apache/beam/blob/master/sdks/python/container/boot.go#L253
.
It may be due to failure of the Python harness process.
On Tue, 28 Jan 2025 at 06:22, Lydian Lee <ti...@gmail.com> wrote:
> Hi,
>
> It seems like there's also similar issue from others:
> https://github.com/apache/beam/issues/29683
>
> For me, here's how I launch the job with the following argos:
> --runner=portableRunner
> --streaming
> --environment_type=EXTERNAL
> --environment_config=localhost:50000
> --experiments=disable_logging_submission_environment # without this, it
> is complaining on rpc error: code = Unknown desc = ; failed to retrieve
> chunk for /tmp/staged/submission_environment_dependencies.txt
>
> For launching the harness:
> /opt/apache/beam/boot --worker_pool
>
>
> I've tested the different pipeline which all works in 2.41.0, but now they
> all failed with 2.60.0. I've also tested with 2.57.0, and that one seems to
> already have the same issue.
>
> Something worth mentioning is I also need to install
> `google-cloud-storage` when bumping the version, otherwise it will complain
> about missing the google cloud library, although I am not using gcs at all.
> ```
> Starting worker with command ['/opt/apache/beam/boot', '--id=31-1',
> '--logging_endpoint=localhost:40649',
> '--artifact_endpoint=localhost:36751',
> '--provision_endpoint=localhost:45679', '--control_endpoint=localh 2
> 025/01/23 23:46:14 Provision info:
> pipeline_options:{fields:{key:"beam:option:allow_non_deterministic_key_coders:v1"
> value:{bool_value:false}}
> fields:{key:"beam:option:allow_non_restored_state:v1"
> value:{bool_value:false}} fields:{key:"beam:o 2
> 025/01/23 23:46:28 boot.go: error logging message over FnAPI. endpoint
> localhost:40649 error: EOF message follows
> 2025/01/23 23:46:28 DEBUG Received signal: terminated
> 2025/01/23 23:46:28 boot.go: error logging message over FnAPI. endpoint
> localhost:40649 error: EOF message follows
> 2025/01/23 23:46:28 ERROR Failed to import GCSFileSystem; loading of this
> filesystem will be skipped. Error details: cannot import name 'storage'
> from 'google.cloud' (unknown location)
> ```
>
> Thanks for your help.
>
Re: Unable to run in python-worker-harness after bumpping from 2.41.0 to 2.60.0
Posted by Lydian Lee <ti...@gmail.com>.
Hi,
It seems like there's also similar issue from others:
https://github.com/apache/beam/issues/29683
For me, here's how I launch the job with the following argos:
--runner=portableRunner
--streaming
--environment_type=EXTERNAL
--environment_config=localhost:50000
--experiments=disable_logging_submission_environment # without this, it is
complaining on rpc error: code = Unknown desc = ; failed to retrieve chunk
for /tmp/staged/submission_environment_dependencies.txt
For launching the harness:
/opt/apache/beam/boot --worker_pool
I've tested the different pipeline which all works in 2.41.0, but now they
all failed with 2.60.0. I've also tested with 2.57.0, and that one seems to
already have the same issue.
Something worth mentioning is I also need to install `google-cloud-storage`
when bumping the version, otherwise it will complain about missing the
google cloud library, although I am not using gcs at all.
```
Starting worker with command ['/opt/apache/beam/boot', '--id=31-1',
'--logging_endpoint=localhost:40649',
'--artifact_endpoint=localhost:36751',
'--provision_endpoint=localhost:45679', '--control_endpoint=localh 2
025/01/23 23:46:14 Provision info:
pipeline_options:{fields:{key:"beam:option:allow_non_deterministic_key_coders:v1"
value:{bool_value:false}}
fields:{key:"beam:option:allow_non_restored_state:v1"
value:{bool_value:false}} fields:{key:"beam:o 2
025/01/23 23:46:28 boot.go: error logging message over FnAPI. endpoint
localhost:40649 error: EOF message follows
2025/01/23 23:46:28 DEBUG Received signal: terminated
2025/01/23 23:46:28 boot.go: error logging message over FnAPI. endpoint
localhost:40649 error: EOF message follows
2025/01/23 23:46:28 ERROR Failed to import GCSFileSystem; loading of this
filesystem will be skipped. Error details: cannot import name 'storage'
from 'google.cloud' (unknown location)
```
Thanks for your help.
Re: Unable to run in python-worker-harness after bumpping from 2.41.0 to 2.60.0
Posted by XQ Hu via user <us...@beam.apache.org>.
Hi Lydian,
I do not know how you setup your FlinkRunner. Recently, I tested several
ways (you can check https://github.com/liferoad/beam-ml-flink) to run a
simple Beam ML job on FlinkRunner and they all seem to be working fine.
XQ
On Mon, Jan 27, 2025 at 12:28 PM Ahmet Altay <al...@google.com> wrote:
> I do not know the answer. Adding @Kenneth Knowles <kl...@google.com> and @XQ
> Hu <xq...@google.com> in case they might be able to answer it.
>
> On Sun, Jan 26, 2025 at 3:54 PM Lydian Lee <ti...@gmail.com>
> wrote:
>
>> Hi,
>>
>> We are trying to bump an old pipeline using flink runner and beam python
>> SDK.
>> The version changes are:
>> - Flink: 1.15.4 -> 1.18.0
>> - Beam: 2.41.0 -> 2.60.0
>> - python: 3.9. (no change)
>>
>> However, after bumping the version, we noticed that the
>> python-worker-harness is unable to process function properly. Here's the
>> error log we kept received:
>>
>> ```
>> Starting worker with command ['/opt/apache/beam/boot', '--id=6-1',
>> '--logging_endpoint=localhost:46759',
>> '--artifact_endpoint=localhost:44123',
>> '--provision_endpoint=localhost:42447', '--control_endpoint=localho
>> 2025/01/26 23:41:26 Provision info:
>> pipeline_options:{fields:{key:"beam:option:allow_non_deterministic_key_coders:v1"
>> value:{bool_value:false}}
>> fields:{key:"beam:option:allow_non_restored_state:v1"
>> value:{bool_value:false}} fields:{key:"beam:o
>> 2025/01/26 23:41:44 boot.go: error logging message over FnAPI. endpoint
>> localhost:46759 error: EOF message follows
>> 2025/01/26 23:41:44 DEBUG Received signal: terminated 2025/01/26 23:41:44
>> boot.go: error logging message over FnAPI. endpoint localhost:46759 error:
>> EOF message follows
>> 2025/01/26 23:41:44 WARN Python (worker 6-1) exited 1 times: signal:
>> terminated restarting SDK process
>> ```
>> It probably unrelated to the job itself as the job was working fine in
>> the earlier version:
>> [image: Untitled.jpeg]
>> Wondering if anyone have any idea how I can fix this issue? Thanks
>>
>
Re: Unable to run in python-worker-harness after bumpping from 2.41.0 to 2.60.0
Posted by Ahmet Altay via user <us...@beam.apache.org>.
I do not know the answer. Adding @Kenneth Knowles <kl...@google.com> and @XQ
Hu <xq...@google.com> in case they might be able to answer it.
On Sun, Jan 26, 2025 at 3:54 PM Lydian Lee <ti...@gmail.com> wrote:
> Hi,
>
> We are trying to bump an old pipeline using flink runner and beam python
> SDK.
> The version changes are:
> - Flink: 1.15.4 -> 1.18.0
> - Beam: 2.41.0 -> 2.60.0
> - python: 3.9. (no change)
>
> However, after bumping the version, we noticed that the
> python-worker-harness is unable to process function properly. Here's the
> error log we kept received:
>
> ```
> Starting worker with command ['/opt/apache/beam/boot', '--id=6-1',
> '--logging_endpoint=localhost:46759',
> '--artifact_endpoint=localhost:44123',
> '--provision_endpoint=localhost:42447', '--control_endpoint=localho
> 2025/01/26 23:41:26 Provision info:
> pipeline_options:{fields:{key:"beam:option:allow_non_deterministic_key_coders:v1"
> value:{bool_value:false}}
> fields:{key:"beam:option:allow_non_restored_state:v1"
> value:{bool_value:false}} fields:{key:"beam:o
> 2025/01/26 23:41:44 boot.go: error logging message over FnAPI. endpoint
> localhost:46759 error: EOF message follows
> 2025/01/26 23:41:44 DEBUG Received signal: terminated 2025/01/26 23:41:44
> boot.go: error logging message over FnAPI. endpoint localhost:46759 error:
> EOF message follows
> 2025/01/26 23:41:44 WARN Python (worker 6-1) exited 1 times: signal:
> terminated restarting SDK process
> ```
> It probably unrelated to the job itself as the job was working fine in the
> earlier version:
> [image: Untitled.jpeg]
> Wondering if anyone have any idea how I can fix this issue? Thanks
>