You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@hop.apache.org by Rob Burgess <ba...@hotmail.com> on 2024/07/22 20:28:08 UTC

Uploading files to S3 bucket

Hi

What is the best Hop action to use to upload files to a S3 bucket?

Thanks

Rob

Re: Uploading files to S3 bucket

Posted by Hans Van Akelyen <ha...@gmail.com>.
Hi Rob,

All our transforms (except for CSV input) can read an write directly to S3.
It will use the default configured AWS account (there is no support for profiles).
Multi-account support is on the roadmap.

More information can be found under the Virtual File System section in the documentation [1].

1 consideration that Diego has mentioned is that listing of bucket content when a bucket contains 10K+ files can be slow, first using the CLI to list the content saving that to a txt and then using that as an input can be faster.

Cheers,
Hans

[1] https://hop.apache.org//manual/latest/vfs/aws-s3-vfs.html
On 23 Jul 2024 at 02:45 +0200, Diego Mainou <di...@bizcubed.com.au>, wrote:
> Hi Rob,
>
> 2 ways.
>
> 1. Mount the S3 as a VFS and simply use the file copy/move step
> 2. Use the bash step and execute commands through the cli
>
> the top one is easy albeit sometimes may be a bit slower than the S3 native cli.
>
>
> Diego Mainou
> Business Development Manager
> M. +61 415 152 091
> E. diego.mainou@bizcubed.com.au
> www.bizcubed.com.au
> https://www.linkedin.com/in/diegomainou/
>
>
> Want to hear more about what BizCubed is doing?
>
> • Follow us on LinkedIn
> • Sign up for our newsletter
>
>
> From: "Rob Burgess" <ba...@hotmail.com>
> To: "users" <us...@hop.apache.org>
> Sent: Tuesday, 23 July, 2024 6:28:08 AM
> Subject: Uploading files to S3 bucket
>
> Hi
>
> What is the best Hop action to use to upload files to a S3 bucket?
>
> Thanks
>
> Rob

Re: Uploading files to S3 bucket

Posted by Diego Mainou <di...@bizcubed.com.au>.
Hi Rob, 

2 ways. 

1. Mount the S3 as a VFS and simply use the file copy/move step 
2. Use the bash step and execute commands through the cli 

the top one is easy albeit sometimes may be a bit slower than the S3 native cli. 


[ https://www.bizcubed.com.au/ | 
						                
										             ] 	Diego Mainou 
Business Development Manager 
M. +61 415 152 091 
E. [ mailto:diego.mainou@bizcubed.com.au | diego.mainou@bizcubed.com.au ] 
[ https://www.bizcubed.com.au/ | www.bizcubed.com.au ] 
[ https://www.linkedin.com/in/diegomainou/ | https://www.linkedin.com/in/diegomainou/ ] 
		[ https://www.bizcubed.com.au/ | 
																																																
																																																 	 ] 

Want to hear more about what BizCubed is doing? 


    * [ https://www.linkedin.com/company/bizcubed | Follow us on LinkedIn ] 
    * [ https://go.bizcubed.com.au/newsletter-registration | Sign up for our newsletter ] 


From: "Rob Burgess" <ba...@hotmail.com> 
To: "users" <us...@hop.apache.org> 
Sent: Tuesday, 23 July, 2024 6:28:08 AM 
Subject: Uploading files to S3 bucket 

Hi 

What is the best Hop action to use to upload files to a S3 bucket? 

Thanks 

Rob