[GitHub] [beam] ConverJens edited a comment on pull request #13180: BEAM-9094 Configure S3 client for IO to s3 compatible object stores

[GitBox <gi...@apache.org>]

ConverJens edited a comment on pull request #13180:
URL: https://github.com/apache/beam/pull/13180#issuecomment-762308020


   @dandy10 @pabloem 
   Great work with this PR!
   I'm trying to get s3 (Minio) to work for TFX, and I get it to work for all but the beam components where I get this strange error:
   
   ```
   Traceback (most recent call last):
     File "apache_beam/runners/common.py", line 1213, in apache_beam.runners.common.DoFnRunner.process
     File "apache_beam/runners/common.py", line 742, in apache_beam.runners.common.PerWindowInvoker.invoke_process
     File "apache_beam/runners/common.py", line 867, in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
     File "/usr/local/lib/python3.7/dist-packages/apache_beam/io/iobase.py", line 1129, in process
       self.writer = self.sink.open_writer(init_result, str(uuid.uuid4()))
     File "/usr/local/lib/python3.7/dist-packages/apache_beam/options/value_provider.py", line 135, in _f
       return fnc(self, *args, **kwargs)
     File "/usr/local/lib/python3.7/dist-packages/apache_beam/io/filebasedsink.py", line 196, in open_writer
       return FileBasedSinkWriter(self, writer_path)
     File "/usr/local/lib/python3.7/dist-packages/apache_beam/io/filebasedsink.py", line 417, in __init__
       self.temp_handle = self.sink.open(temp_shard_path)
     File "/usr/local/lib/python3.7/dist-packages/apache_beam/options/value_provider.py", line 135, in _f
       return fnc(self, *args, **kwargs)
     File "/usr/local/lib/python3.7/dist-packages/apache_beam/io/filebasedsink.py", line 138, in open
       return FileSystems.create(temp_path, self.mime_type, self.compression_type)
     File "/usr/local/lib/python3.7/dist-packages/apache_beam/io/filesystems.py", line 229, in create
       return filesystem.create(path, mime_type, compression_type)
     File "/usr/local/lib/python3.7/dist-packages/apache_beam/io/aws/s3filesystem.py", line 171, in create
       return self._path_open(path, 'wb', mime_type, compression_type)
     File "/usr/local/lib/python3.7/dist-packages/apache_beam/io/aws/s3filesystem.py", line 151, in _path_open
       raw_file = s3io.S3IO(options=self._options).open(
     File "/usr/local/lib/python3.7/dist-packages/apache_beam/io/aws/s3io.py", line 63, in __init__
       raise ValueError('Must provide one of client or options')
   ValueError: Must provide one of client or options
   ```
   
   Do you have any idea what I'm doing wrong? 
   
   These are the beam pipeline args that I'm supplying and I know for sure that at least the multi process and nr_of_workers arguments are applied:
   ```
   '--direct_running_mode=multi_processing',
   f'--direct_num_workers={NR_OF_CPUS}',
   '--s3_endpoint_url=minio-service.kubeflow:9000',
   f'--s3_access_key={ACCESS_KEY}',
   f'--s3_secret_access_key={SECRET_ACCESS_KEY},
   '--s3_verify=False'
   ```
   
   Help would be greatly appreciated!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org