You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/06/04 18:21:53 UTC

[GitHub] [beam] damccorm opened a new issue, #20633: Python DirectRunner InputStream Issues (RuntimeError: VarLong too long)

damccorm opened a new issue, #20633:
URL: https://github.com/apache/beam/issues/20633

   I have a Beam workflow that runs fine on Dataflow. However, when I run it on my local direct runner (Windows 10, Python SDK), I run into some strange behaviour if I have a negative number in my PCollection (see below for an example).
   
   Minimal code example to reproduce the exception:
   ```
   
   from apache_beam.options.pipeline_options import PipelineOptions
   from apache_beam.pipeline import
   Pipeline
   import apache_beam
   
   if __name__ == '__main__':
       pipeline_options = PipelineOptions()
   
   
      with Pipeline(options=pipeline_options) as p:
           lines = (
                   p
              
       | apache_beam.Create([
                       {"Test": -1}
                   ]))
   
           lines |
   apache_beam.io.WriteToText('./lines')
   
   ```
   
   Exception:
   ```
   
     File "apache_beam\coders\coder_impl.py", line 222, in apache_beam.coders.coder_impl.StreamCoderImpl.decode
   
    File "apache_beam\coders\coder_impl.py", line 224, in apache_beam.coders.coder_impl.StreamCoderImpl.decode
   
    File "apache_beam\coders\coder_impl.py", line 470, in apache_beam.coders.coder_impl.FastPrimitivesCoderImpl.decode_from_stream
   
    File "apache_beam\coders\coder_impl.py", line 450, in apache_beam.coders.coder_impl.FastPrimitivesCoderImpl.decode_from_stream
   
    File "apache_beam\coders\stream.pyx", line 193, in apache_beam.coders.stream.InputStream.read_var_int64
   RuntimeError:
   VarLong too long. [while running 'Create/Map(decode)']
   
   ```
   
    Any help is highly appreciated.
   
    
   
   Imported from Jira [BEAM-11037](https://issues.apache.org/jira/browse/BEAM-11037). Original Jira may contain additional context.
   Reported by: Stauffer.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] Abacn commented on issue #20633: Python DirectRunner InputStream Issues (RuntimeError: VarLong too long)

Posted by GitBox <gi...@apache.org>.
Abacn commented on issue #20633:
URL: https://github.com/apache/beam/issues/20633#issuecomment-1170179145

   .take-issue


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] TheNeuralBit closed issue #20633: Python DirectRunner InputStream Issues (RuntimeError: VarLong too long)

Posted by GitBox <gi...@apache.org>.
TheNeuralBit closed issue #20633: Python DirectRunner InputStream Issues (RuntimeError: VarLong too long)
URL: https://github.com/apache/beam/issues/20633


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] Abacn commented on issue #20633: Python DirectRunner InputStream Issues (RuntimeError: VarLong too long)

Posted by GitBox <gi...@apache.org>.
Abacn commented on issue #20633:
URL: https://github.com/apache/beam/issues/20633#issuecomment-1170075888

   Likely a cython compatibility issue with windows.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] Abacn commented on issue #20633: Python DirectRunner InputStream Issues (RuntimeError: VarLong too long)

Posted by GitBox <gi...@apache.org>.
Abacn commented on issue #20633:
URL: https://github.com/apache/beam/issues/20633#issuecomment-1170174934

   Apparently the current implementation has problem for platforms that has sizeof(long) returns 4 (windows does)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org