You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hop.apache.org by "rbrenchley (via GitHub)" <gi...@apache.org> on 2025/07/28 20:16:12 UTC
[I] [Feature Request]: AWS S3 bucket connector transform for a pipeline (hop)
rbrenchley opened a new issue, #5588:
URL: https://github.com/apache/hop/issues/5588
### What would you like to happen?
Please create a connector to AWS for pulling data from an S3 bucket.
The following parameters should be passed into the transform:
- Request type: ex GET/PUT
- AWS Service URL
- AccessKey
- SecretKey
### Issue Priority
Priority: 2
### Issue Component
Component: Hop Gui, Component: Actions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: AWS S3 bucket connector transform for a pipeline (hop)
Posted by "bamaer (via GitHub)" <gi...@apache.org>.
bamaer commented on issue #5588:
URL: https://github.com/apache/hop/issues/5588#issuecomment-3151218174
if one of the developers needs it, wants to work on it or if someone or some company sponsors the development effort
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: AWS S3 bucket connector transform for a pipeline (hop)
Posted by "hansva (via GitHub)" <gi...@apache.org>.
hansva commented on issue #5588:
URL: https://github.com/apache/hop/issues/5588#issuecomment-3129812828
You can just use [s3://](https://hop.apache.org//manual/latest/vfs/aws-s3-vfs.html) in any of our file transforms, I'm not sure what this request is covering?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: AWS S3 bucket connector transform for a pipeline (hop)
Posted by "rbrenchley (via GitHub)" <gi...@apache.org>.
rbrenchley commented on issue #5588:
URL: https://github.com/apache/hop/issues/5588#issuecomment-3151182314
Will you be adding this feature in the near future?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: AWS S3 bucket connector transform for a pipeline (hop)
Posted by "hansva (via GitHub)" <gi...@apache.org>.
hansva commented on issue #5588:
URL: https://github.com/apache/hop/issues/5588#issuecomment-3133354497
This probably means your default aws configuration on your system does not have a default bucket, append the bucket name after the s3://<bucketname>/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: AWS S3 bucket connector transform for a pipeline (hop)
Posted by "hansva (via GitHub)" <gi...@apache.org>.
hansva commented on issue #5588:
URL: https://github.com/apache/hop/issues/5588#issuecomment-3151114345
For Azure and for Google we have created a way to connect to multiple accounts at the same time, this has not yet been done for AWS
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: AWS S3 bucket connector transform for a pipeline (hop)
Posted by "rbrenchley (via GitHub)" <gi...@apache.org>.
rbrenchley commented on issue #5588:
URL: https://github.com/apache/hop/issues/5588#issuecomment-3152321304
AWS has as request signing interceptor found at: https://github.com/awsdocs/aws-request-signing-apache-interceptor
How would one go about incorporating this into Hop?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: AWS S3 bucket connector transform for a pipeline (hop)
Posted by "rbrenchley (via GitHub)" <gi...@apache.org>.
rbrenchley closed issue #5588: [Feature Request]: AWS S3 bucket connector transform for a pipeline
URL: https://github.com/apache/hop/issues/5588
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: AWS S3 bucket connector transform for a pipeline (hop)
Posted by "hansva (via GitHub)" <gi...@apache.org>.
hansva commented on issue #5588:
URL: https://github.com/apache/hop/issues/5588#issuecomment-5268064721
This has been resolved with our s3 connection metadata type
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: AWS S3 bucket connector transform for a pipeline (hop)
Posted by "rbrenchley (via GitHub)" <gi...@apache.org>.
rbrenchley commented on issue #5588:
URL: https://github.com/apache/hop/issues/5588#issuecomment-3150817402
I need to be able to connect to multiple separate buckets including a Minio bucket with different access id's and keys. How would you recommend doing this?
This is where a component to connect to s3 would work well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: AWS S3 bucket connector transform for a pipeline (hop)
Posted by "hansva (via GitHub)" <gi...@apache.org>.
hansva closed issue #5588: [Feature Request]: AWS S3 bucket connector transform for a pipeline
URL: https://github.com/apache/hop/issues/5588
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: AWS S3 bucket connector transform for a pipeline (hop)
Posted by "hansva (via GitHub)" <gi...@apache.org>.
hansva commented on issue #5588:
URL: https://github.com/apache/hop/issues/5588#issuecomment-3151259971
I can't speak for all contributors as I do not know their needs/priorities. Most work for what their company, clients need or what they need themselves. If certain features need fast tracking there are parties providing [commercial support or development](https://hop.apache.org/community/commercial/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: AWS S3 bucket connector transform for a pipeline (hop)
Posted by "rbrenchley (via GitHub)" <gi...@apache.org>.
rbrenchley commented on issue #5588:
URL: https://github.com/apache/hop/issues/5588#issuecomment-3165146085
Something simple that you could do in a future release;, include the boto3 library with the Jython jar. While it would be nice to have a component specific for AWS, you wouldn't really need to do that if you had boto3.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: AWS S3 bucket connector transform for a pipeline (hop)
Posted by "rbrenchley (via GitHub)" <gi...@apache.org>.
rbrenchley commented on issue #5588:
URL: https://github.com/apache/hop/issues/5588#issuecomment-3134085290
Working.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: AWS S3 bucket connector transform for a pipeline (hop)
Posted by "hansva (via GitHub)" <gi...@apache.org>.
hansva commented on issue #5588:
URL: https://github.com/apache/hop/issues/5588#issuecomment-3152394675
The simplest work around would be to use shell commands to do what you want
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: AWS S3 bucket connector transform for a pipeline (hop)
Posted by "hansva (via GitHub)" <gi...@apache.org>.
hansva commented on issue #5588:
URL: https://github.com/apache/hop/issues/5588#issuecomment-3133362652
But normally it should be able to list buckets does the aws cli work on your system?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: AWS S3 bucket connector transform for a pipeline (hop)
Posted by "rbrenchley (via GitHub)" <gi...@apache.org>.
rbrenchley commented on issue #5588:
URL: https://github.com/apache/hop/issues/5588#issuecomment-3133279720
I've tried that, and I get the following error:
Error browsing to location: 's3://'
FileSystemException: Could not determine the type of file "s3:///".
Root cause: IllegalArgumentException: bucketName cannot be empty
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: AWS S3 bucket connector transform for a pipeline (hop)
Posted by "rbrenchley (via GitHub)" <gi...@apache.org>.
rbrenchley commented on issue #5588:
URL: https://github.com/apache/hop/issues/5588#issuecomment-3133383293
Error browsing to location: 's3:///myhoptestbucket'
FileSystemException: Could not list the contents of folder "s3:///myhoptestbucket".
I don't have the client but postman works just fine using the web service.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: AWS S3 bucket connector transform for a pipeline (hop)
Posted by "hansva (via GitHub)" <gi...@apache.org>.
hansva commented on issue #5588:
URL: https://github.com/apache/hop/issues/5588#issuecomment-3133497749
We use the system settings did you setup your environment as described in the [aws](https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html) docs? The reason we don't want to create specific transforms is because once that is set up everything will work, you can read directly from s3 or use operations like copy/move/delete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: AWS S3 bucket connector transform for a pipeline (hop)
Posted by "hansva (via GitHub)" <gi...@apache.org>.
hansva commented on issue #5588:
URL: https://github.com/apache/hop/issues/5588#issuecomment-3160875576
The script transforms can do imports if the library is in the class path
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: AWS S3 bucket connector transform for a pipeline (hop)
Posted by "rbrenchley (via GitHub)" <gi...@apache.org>.
rbrenchley commented on issue #5588:
URL: https://github.com/apache/hop/issues/5588#issuecomment-3160864089
Does hop have a HMAC-SHA256 function? I know you can do this with Java or Java script but you have to do a require or import statement, which I don't think works with any of the script components.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org