Is there a way to configure migrating data from postgresql to aws s3 as a stream ( currently appears to be attempting to pull all of the data from the table into memory and then send to aws? )

[Reid Thompson <Re...@omnicell.com>]

Looking for some guidance.


For small tables, the format below will work fine.
However, I am attempting 

create table s3.datafolder.schema1_table1 as select * from postgresql.schema1.table1;

s3.datafolder.schema1_table1 is in AWS.
postgresql.schema1.table1 is a couple of hundred GB.
The request is failing with out of memory errors.
Is there a way to instruct drill to stream the data to s3 as it's read?
I presume it's attempting to pull all of the data from the table into
memory, then write it?  (which obviously isn't going to work on a host
with 16GB of RAM).

thanks,
reid