You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hop.apache.org by "UniverseColumbus (via GitHub)" <gi...@apache.org> on 2023/02/01 21:18:07 UTC
[GitHub] [hop] UniverseColumbus opened a new issue, #2218: [Bug]: VFS connection to S3 bucket using the "Text File Input" action
UniverseColumbus opened a new issue, #2218:
URL: https://github.com/apache/hop/issues/2218
### Apache Hop version?
2.3.0
### Java version?
11.0.16
### Operating system
macOS
### What happened?
I have encountered a problem when trying to download & upload files from an S3 bucket using a "Text File Input" action in Apache hop GUI (version 2.3.0) running on the local macOS environment using the following VFS url (s3://sdi-snowflake-bucket/snowflake-data/fake-data.csv). AWS credentials (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY) are being set as [system environment variables](https://hop.apache.org/manual/latest/vfs/aws-s3-vfs.html#_variables) which I can verify exist using the “Write to Log” action in hop. When trying to download the S3 files there is an error in hop that says:
ERROR: org.apache.commons.vfs2.FileSystemException: Could not determine the type of file "s3:///sdi-snowflake-bucket/snowflake-data/fake-data.csv".
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403
The AWS credentials set as system environment variables work when using the [“Shell” action](https://hop.apache.org/manual/latest/workflow/actions/shell.html) with AWS cli and the following command: aws s3 ls s3://sdi-snowflake-bucket/snowflake-data/fake-data.csv
Also, downloading a file from S3 using aws cli and the following command works: aws s3 cp s3://sdi-snowflake-bucket/snowflake-data/fake-data.csv .
Additionally, the AWS credentials set as system environment variables work fine with variable substitution when targeting a S3 bucket with the “COPY INTO” sql statement from Snowflake.
Lastly, I would like to mention that it is possible to upload and download files from our S3 bucket when everything is set to a public access level in AWS using the “Text File Input/Output” action. However, the credentials don't work otherwise with the "Text File Input" action through the VFS connection to AWS, even though they work fine using AWS cli and the “Execute SQL Script” action in hop.
### Issue Priority
Priority: 2
### Issue Component
Component: VFS
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@hop.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hop] bamaer commented on issue #2218: [Bug]: VFS connection to S3 bucket using the "Text File Input" action
Posted by "bamaer (via GitHub)" <gi...@apache.org>.
bamaer commented on issue #2218:
URL: https://github.com/apache/hop/issues/2218#issuecomment-1416127299
can you post the output of `aws s3 ls s3://sdi-snowflake-bucket/snowflake-data/test-data.csv`
this error seems to thrown when Hop tries to [get the file listing](https://github.com/apache/hop/blob/91fdf18ecb3667ef328072fc7c82d548ed80e2d8/plugins/tech/aws/src/main/java/org/apache/hop/vfs/s3/s3/vfs/S3FileObject.java#L149) for your S3 object or its parent (virtual) folder.
also: check if there are no conflicting settings on your system as mentioned in the [hop chat](
https://chat.project-hop.org/hop/pl/oyjk8sx83bbgueobshff5gwmka), related to a similar discussion that turned out to be a configuration problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hop] UniverseColumbus closed issue #2218: [Bug]: VFS connection to S3 bucket using the "Text File Input" action
Posted by "UniverseColumbus (via GitHub)" <gi...@apache.org>.
UniverseColumbus closed issue #2218: [Bug]: VFS connection to S3 bucket using the "Text File Input" action
URL: https://github.com/apache/hop/issues/2218
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hop] UniverseColumbus commented on issue #2218: [Bug]: VFS connection to S3 bucket using the "Text File Input" action
Posted by "UniverseColumbus (via GitHub)" <gi...@apache.org>.
UniverseColumbus commented on issue #2218:
URL: https://github.com/apache/hop/issues/2218#issuecomment-1412786784
Here is the stack trace:
2023/02/01 14:53:01 - Hop - Pipeline opened.
2023/02/01 14:53:01 - Hop - Launching pipeline [testing]...
2023/02/01 14:53:01 - Hop - Started the pipeline execution.
2023/02/01 14:53:01 - testing - Executing this pipeline using the Local Pipeline Engine with run configuration 'local'
2023/02/01 14:53:01 - testing - Execution started for pipeline [testing]
2023/02/01 14:53:02 - FileInputList - ERROR: org.apache.commons.vfs2.FileSystemException: Could not determine the type of file "s3:///sdi-snowflake-bucket/snowflake-data/test-data.csv".
2023/02/01 14:53:02 - FileInputList - at org.apache.commons.vfs2.provider.AbstractFileObject.attach(AbstractFileObject.java:171)
2023/02/01 14:53:02 - FileInputList - at org.apache.commons.vfs2.provider.AbstractFileObject.getType(AbstractFileObject.java:1370)
2023/02/01 14:53:02 - FileInputList - at org.apache.hop.core.fileinput.FileInputList.createFileList(FileInputList.java:226)
2023/02/01 14:53:02 - FileInputList - at org.apache.hop.core.fileinput.FileInputList.createFileList(FileInputList.java:172)
2023/02/01 14:53:02 - FileInputList - at org.apache.hop.pipeline.transforms.file.BaseFileInputMeta.getFileInputList(BaseFileInputMeta.java:123)
2023/02/01 14:53:02 - FileInputList - at org.apache.hop.pipeline.transforms.file.BaseFileInputTransform.init(BaseFileInputTransform.java:76)
2023/02/01 14:53:02 - FileInputList - at org.apache.hop.pipeline.transforms.fileinput.text.TextFileInput.init(TextFileInput.java:55)
2023/02/01 14:53:02 - FileInputList - at org.apache.hop.pipeline.transform.TransformInitThread.run(TransformInitThread.java:66)
2023/02/01 14:53:02 - FileInputList - at java.base/java.lang.Thread.run(Thread.java:834)
2023/02/01 14:53:02 - FileInputList - Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: Q1ZV7Y7Q3JD1CGCJ; S3 Extended Request ID: rbynPyMJzsWQhqbgiTj8yb0u1eUcGnRnd3xSGCHyXJr8MF/PxnjGkOKuilKop7rUJ/2Z5FszC3w=; Proxy: null), S3 Extended Request ID: rbynPyMJzsWQhqbgiTj8yb0u1eUcGnRnd3xSGCHyXJr8MF/PxnjGkOKuilKop7rUJ/2Z5FszC3w=
2023/02/01 14:53:02 - FileInputList - at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1879)
2023/02/01 14:53:02 - FileInputList - at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1418)
2023/02/01 14:53:02 - FileInputList - at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1387)
2023/02/01 14:53:02 - FileInputList - at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157)
2023/02/01 14:53:02 - FileInputList - at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814)
2023/02/01 14:53:02 - FileInputList - at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781)
2023/02/01 14:53:02 - FileInputList - at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)
2023/02/01 14:53:02 - FileInputList - at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)
2023/02/01 14:53:02 - FileInputList - at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)
2023/02/01 14:53:02 - FileInputList - at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561)
2023/02/01 14:53:02 - FileInputList - at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541)
2023/02/01 14:53:02 - FileInputList - at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5456)
2023/02/01 14:53:02 - FileInputList - at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5403)
2023/02/01 14:53:02 - FileInputList - at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5397)
2023/02/01 14:53:02 - FileInputList - at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:928)
2023/02/01 14:53:02 - FileInputList - at org.apache.hop.vfs.s3.s3.vfs.S3FileObject.handleAttachException(S3FileObject.java:149)
2023/02/01 14:53:02 - FileInputList - at org.apache.hop.vfs.s3.s3common.S3CommonFileObject.doAttach(S3CommonFileObject.java:207)
2023/02/01 14:53:02 - FileInputList - at org.apache.commons.vfs2.provider.AbstractFileObject.attach(AbstractFileObject.java:162)
2023/02/01 14:53:02 - FileInputList - ... 8 more
2023/02/01 14:53:02 - testing - Pipeline duration : 0.833 seconds [ 0.833" ]
2023/02/01 14:53:02 - testing - Execution finished on a local pipeline engine with run configuration 'local'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org