[jira] [Comment Edited] (BEAM-6860) WriteToText crash with "GlobalWindow -> ._IntervalWindowBase"

["Paul (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-6860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16860164#comment-16860164 ] 

Paul edited comment on BEAM-6860 at 6/10/19 5:08 PM:
-----------------------------------------------------

FWIW I'm having this same issue on apache_beam 2.10, 2.11, 2.12, 2.13 python sdk; I confirmed that this works on 2.9.0 as well, which I saw in a stack overflow thread here ([https://stackoverflow.com/questions/55109403/apache-beam-python-sdk-upgrade-issue).]

It also fails not only on WriteToText, but also WriteToBigQuery. Specifically, this does not work when implementing windowing on a bounded data set and adding element timestamps. While doing this, I was working off of this example: [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/complete/game/hourly_team_score.py]

However, this example actually works in later versions of the beam python SDK, so I'm wondering if this is a user-issue with a very poor error message or if we're explicitly/implicitly using a feature of beam that the example does not.

 

Environment was:

centos7 linux, kernel 3.10

python 2.7.16

apache beam versions: 2.9.0-2.13.0


was (Author: pshahid):
FWIW I'm having this same issue on apache_beam 2.10, 2.11, 2.12, 2.13 python sdk; I confirmed that this works on 2.9.0 as well, which I saw in a stack overflow thread here ([https://stackoverflow.com/questions/55109403/apache-beam-python-sdk-upgrade-issue).]

It also fails not only on WriteToText, but also WriteToBigQuery. Specifically, this does not work when implementing windowing on a bounded data set and adding element timestamps. While doing this, I was working off of this example: [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/complete/game/hourly_team_score.py]

However, this example actually works in later versions of the beam python SDK, so I'm wondering if this is a user-issue with a very poor error message or if we're explicitly/implicitly using a feature of beam that the example does not.

> WriteToText crash with "GlobalWindow -> ._IntervalWindowBase"
> -------------------------------------------------------------
>
>                 Key: BEAM-6860
>                 URL: https://issues.apache.org/jira/browse/BEAM-6860
>             Project: Beam
>          Issue Type: Bug
>          Components: beam-model
>    Affects Versions: 2.11.0
>         Environment: macOS, DirectRunner, python 2.7.15 via pyenv/pyenv-virtualenv
>            Reporter: Henrik
>            Priority: Major
>              Labels: newbie
>
> Main error:
> > Cannot convert GlobalWindow to apache_beam.utils.windowed_value._IntervalWindowBase
> This is very hard for me to debug. Doing a DoPar call before, printing the input, gives me just what I want; so the lines of data to serialise are "alright"; just JSON strings, in fact.
> Stacktrace:
> {code:java}
> Traceback (most recent call last):
>   File "./okr_end_ride.py", line 254, in <module>
>     run()
>   File "./okr_end_ride.py", line 250, in run
>     run_pipeline(pipeline_options, known_args)
>   File "./okr_end_ride.py", line 198, in run_pipeline
>     | 'write_all'             >> WriteToText(known_args.output, file_name_suffix=".txt")
>   File "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/pipeline.py", line 426, in __exit__
>     self.run().wait_until_finish()
>   File "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/pipeline.py", line 406, in run
>     self._options).run(False)
>   File "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/pipeline.py", line 419, in run
>     return self.runner.run_pipeline(self, self._options)
>   File "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/runners/direct/direct_runner.py", line 132, in run_pipeline
>     return runner.run_pipeline(pipeline, options)
>   File "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py", line 275, in run_pipeline
>     default_environment=self._default_environment))
>   File "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py", line 278, in run_via_runner_api
>     return self.run_stages(*self.create_stages(pipeline_proto))
>   File "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py", line 354, in run_stages
>     stage_context.safe_coders)
>   File "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py", line 509, in run_stage
>     data_input, data_output)
>   File "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py", line 1206, in process_bundle
>     result_future = self._controller.control_handler.push(process_bundle)
>   File "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py", line 821, in push
>     response = self.worker.do_instruction(request)
>   File "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", line 265, in do_instruction
>     request.instruction_id)
>   File "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", line 281, in process_bundle
>     delayed_applications = bundle_processor.process_bundle(instruction_id)
>   File "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py", line 552, in process_bundle
>     op.finish()
>   File "apache_beam/runners/worker/operations.py", line 549, in apache_beam.runners.worker.operations.DoOperation.finish
>   File "apache_beam/runners/worker/operations.py", line 550, in apache_beam.runners.worker.operations.DoOperation.finish
>   File "apache_beam/runners/worker/operations.py", line 551, in apache_beam.runners.worker.operations.DoOperation.finish
>   File "apache_beam/runners/common.py", line 758, in apache_beam.runners.common.DoFnRunner.finish
>   File "apache_beam/runners/common.py", line 752, in apache_beam.runners.common.DoFnRunner._invoke_bundle_method
>   File "apache_beam/runners/common.py", line 777, in apache_beam.runners.common.DoFnRunner._reraise_augmented
>   File "apache_beam/runners/common.py", line 750, in apache_beam.runners.common.DoFnRunner._invoke_bundle_method
>   File "apache_beam/runners/common.py", line 361, in apache_beam.runners.common.DoFnInvoker.invoke_finish_bundle
>   File "apache_beam/runners/common.py", line 364, in apache_beam.runners.common.DoFnInvoker.invoke_finish_bundle
>   File "apache_beam/runners/common.py", line 887, in apache_beam.runners.common._OutputProcessor.finish_bundle_outputs
>   File "apache_beam/runners/worker/operations.py", line 130, in apache_beam.runners.worker.operations.SingletonConsumerSet.receive
>   File "apache_beam/runners/worker/operations.py", line 110, in apache_beam.runners.worker.operations.ConsumerSet.update_counters_start
>   File "apache_beam/runners/worker/opcounters.py", line 195, in apache_beam.runners.worker.opcounters.OperationCounters.update_from
>   File "apache_beam/runners/worker/opcounters.py", line 213, in apache_beam.runners.worker.opcounters.OperationCounters.do_sample
>   File "apache_beam/coders/coder_impl.py", line 991, in apache_beam.coders.coder_impl.WindowedValueCoderImpl.get_estimated_size_and_observables
>   File "apache_beam/coders/coder_impl.py", line 1007, in apache_beam.coders.coder_impl.WindowedValueCoderImpl.get_estimated_size_and_observables
>   File "apache_beam/coders/coder_impl.py", line 791, in apache_beam.coders.coder_impl.SequenceCoderImpl.estimate_size
>   File "apache_beam/coders/coder_impl.py", line 805, in apache_beam.coders.coder_impl.SequenceCoderImpl.get_estimated_size_and_observables
>   File "apache_beam/coders/coder_impl.py", line 145, in apache_beam.coders.coder_impl.CoderImpl.get_estimated_size_and_observables
>   File "apache_beam/coders/coder_impl.py", line 487, in apache_beam.coders.coder_impl.IntervalWindowCoderImpl.estimate_size
> TypeError: Cannot convert GlobalWindow to apache_beam.utils.windowed_value._IntervalWindowBase [while running 'write_all/Write/WriteImpl/WriteBundles']
> {code}
> Slack thread: [https://the-asf.slack.com/archives/C9H0YNP3P/p1552928806101200] including code snippet
> Your issue tracker seems to be missing components for both the python on go SDKs.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)