[jira] [Created] (BEAM-14176) Beam dataflow hangs with requirements.txt

["Ryan Thompson (Jira)" <ji...@apache.org>]

Ryan Thompson created BEAM-14176:
------------------------------------

             Summary: Beam dataflow hangs with requirements.txt
                 Key: BEAM-14176
                 URL: https://issues.apache.org/jira/browse/BEAM-14176
             Project: Beam
          Issue Type: Bug
          Components: sdk-py-core
            Reporter: Ryan Thompson


Similar to this question:

https://stackoverflow.com/questions/62032382/dataflow-fails-when-i-add-requirements-txt-python

Note: I could resolve this also by using setup.py.  However, it would be nice to have a better error message instead of hanging.

 

When trying to use a requirements.txt file and deploy to dataflow, beam is hanging.

Here was the following last message.

INFO:apache_beam.runners.portability.stager:Executing command: 



['/Users/ryanthompson/.virtualenvs/hackathon/bin/python', '-m', 'pip', 'download', '--dest', '/var/folders/6j/0z_b3j512gd6_mszhyy5p5qc0037d6/T/dataflow-requirements-cache', '-r', '/var/folders/6j/0z_b3j512gd6_mszhyy5p5qc0037d6/T/tmp68jk51_9/tmp_requirements.txt', '--exists-action', 'i', '--no-binary', ':all:']

Here is a program that replicates:

import logging

import argparse
import apache_beam as beam
from apache_beam import Create
from apache_beam.options.pipeline_options import PipelineOptions, SetupOptions
import apache_beam.io.gcp.gcsfilesystem as gcsfs

import py_midicsv as pm


def midi_to_csv(file_name) -> str:
fs = gcsfs.GCSFileSystem(PipelineOptions())
file = fs.open(file_name, 'rb')
return pm.midi_to_csv(file)

def run(argv=None):
parser = argparse.ArgumentParser()
known_args, pipeline_args = parser.parse_known_args(argv)
# For gs testing.
input_filenames = ['gs://clouddfe-ryanthompson/hackathon/classical/bach/bach_846.mid']
output_name = 'gs://clouddfe-ryanthompson/hackathon/output/midi_out'

options = PipelineOptions(pipeline_args)
options.view_as(SetupOptions).save_main_session = True
options.view_as(SetupOptions).requirements_file = 'pipelines/requirements.txt'
with beam.Pipeline(options=options) as p:
(p | Create(input_filenames)
mapped = input_pcol | 'Read File from GCS' >> beam.Map(midi_to_csv)
written = mapped | 'Write to output files' >> beam.Map(logging.info))


if __name__ == '__main__':
logging.getLogger().setLevel(logging.INFO)
run()

 

Here is my requirements.txt file:

py-midicsv

 

Other possibly relevant information. 

I tested with python 3.6, on macbook, with pycharm console



--
This message was sent by Atlassian Jira
(v8.20.1#820001)