You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "shunping (via GitHub)" <gi...@apache.org> on 2025/12/02 20:02:24 UTC

[I] [Bug]: Prism panic when TestStream is followed by a stateful dofn [beam]

shunping opened a new issue, #36965:
URL: https://github.com/apache/beam/issues/36965

   ### What happened?
   
   This is observed in some of our test workflow. Even though it is not causing the test to fail, I believe it is only because of luck, aka our tests do not cover this case. 
   
   
   Thanks @aIbrahiim for bringing this up in one of the fix PR #36927.
   
   To reproduce, we can use the following code:
   
   ```python
   import logging
   import apache_beam as beam
   from apache_beam.coders import VarIntCoder
   from apache_beam.transforms.userstate import ReadModifyWriteStateSpec
   from apache_beam.utils.timestamp import Timestamp
   from apache_beam.options.pipeline_options import PipelineOptions
   from apache_beam.testing.test_stream import TestStream
   
   logging.basicConfig(level=logging.INFO)
   #logging.basicConfig(level=logging.WARNING)
   
   class MyDoFn(beam.DoFn):
     COUNT = ReadModifyWriteStateSpec('count', VarIntCoder())
   
     def __init__(self):
       pass
   
     def process(
         self,
         element,
         count_state=beam.DoFn.StateParam(COUNT),
     ):
       print(element)
       count = count_state.read()
       if not count:
         count = 1
       else:
         count += 1
   
       count_state.write(count)
       yield count
   
   
   
   
   options = PipelineOptions([
       "--streaming",
       "--environment_type=LOOPBACK",
       "--runner=PrismRunner",
       "--prism_log_kind=dev",
       #"--prism_beam_version_override=v2.66.0"
       # "--runner=PortableRunner",
       # "--job_endpoint=localhost:8073",
   ])
   
   with beam.Pipeline(options=options) as p:
     now = Timestamp.now()
     _ = (
         p | TestStream().add_elements([(1, 2), (3, 4)])
         | 'MyDoFn' >> beam.ParDo(MyDoFn())
         | beam.LogElements(
             prefix="result=",
             level=logging.WARNING,
             with_timestamp=True,
             with_window=True,
             use_epoch_time=True))
   ```
   
   ### Issue Priority
   
   Priority: 2 (default / most bugs should be filed as P2)
   
   ### Issue Components
   
   - [ ] Component: Python SDK
   - [ ] Component: Java SDK
   - [ ] Component: Go SDK
   - [ ] Component: Typescript SDK
   - [ ] Component: IO connector
   - [ ] Component: Beam YAML
   - [ ] Component: Beam examples
   - [ ] Component: Beam playground
   - [ ] Component: Beam katas
   - [ ] Component: Website
   - [ ] Component: Infrastructure
   - [ ] Component: Spark Runner
   - [ ] Component: Flink Runner
   - [ ] Component: Samza Runner
   - [ ] Component: Twister2 Runner
   - [ ] Component: Hazelcast Jet Runner
   - [ ] Component: Google Cloud Dataflow Runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug]: Prism panic when TestStream is followed by a stateful dofn [beam]

Posted by "shunping (via GitHub)" <gi...@apache.org>.
shunping commented on issue #36965:
URL: https://github.com/apache/beam/issues/36965#issuecomment-3993027970

   Not yet! Feel free to give it a try.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug]: Prism panic when TestStream is followed by a stateful dofn [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #36965:
URL: https://github.com/apache/beam/issues/36965#issuecomment-5205002597

   This issue has been marked as stale due to 150 days of inactivity. It will be closed in 30 days if no further activity occurs. If you think that’s incorrect or this issue still needs to be addressed, please simply write any comment. If closed, you can reopen the issue at any time. Thank you for your contributions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug]: Prism panic when TestStream is followed by a stateful dofn [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #36965: [Bug]: Prism panic when TestStream is followed by a stateful dofn
URL: https://github.com/apache/beam/issues/36965


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug]: Prism panic when TestStream is followed by a stateful dofn [beam]

Posted by "boy397 (via GitHub)" <gi...@apache.org>.
boy397 commented on issue #36965:
URL: https://github.com/apache/beam/issues/36965#issuecomment-4019667067

   Hi @shunping  do you have any suggestions??


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug]: Prism panic when TestStream is followed by a stateful dofn [beam]

Posted by "boy397 (via GitHub)" <gi...@apache.org>.
boy397 commented on issue #36965:
URL: https://github.com/apache/beam/issues/36965#issuecomment-3993135455

   you have worked on it, so is there any suggestion you want to provide me before i start??
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug]: Prism panic when TestStream is followed by a stateful dofn [beam]

Posted by "boy397 (via GitHub)" <gi...@apache.org>.
boy397 commented on issue #36965:
URL: https://github.com/apache/beam/issues/36965#issuecomment-3993019445

   @shunping Is this issue solved yet? If not can i contribute to it??
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug]: Prism panic when TestStream is followed by a stateful dofn [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #36965:
URL: https://github.com/apache/beam/issues/36965#issuecomment-5559276152

   This issue has been closed due to lack of activity. If you think that is incorrect, you can reopen the issue at any time.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org