You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@drill.apache.org by Joshua Schlesser <jo...@cursivelabs.com> on 2015/12/11 23:02:56 UTC
s3a storage not writable
I am trying to create new tables on a storage configured to use s3a storage.
I copied followed the s3 instructions in the docs and copied dfs storage and replaced the connection with the s3a uri and set the root workspace to writable. tmp was already writable.
I execute the use storage; sql command to change my default schema.
i then execute create table tablename as select * from schema.tablename
the select when run outside of the context of a create table runs fine.
I receive the following error when trying to create a table:
Error: PARSE ERROR: Unable to create or drop tables/views. Schema [xxxxxx] is immutable
I have confirmed that I can perform select queries just fine from the s3 bucket and the permissions associated with the keys are full access.
Im running 1.3.0.
Is there anything i need to do special to enable s3a storages to be writable?
Thanks in advance for any insights.
Josh Schlesser
Re: s3a storage not writable
Posted by Tomer Shiran <ts...@dremio.com>.
Thanks for sharing your findings!
On Sat, Dec 12, 2015 at 10:55 AM, Joshua Schlesser <jo...@cursivelabs.com>
wrote:
> Answer to my own question.
> You can label the root workspace as writable but that is ignored.
> Also it seems you cant assume that top level ‘folders’ in your s3 buckets
> will magically get created in a create table statement even though s3 is an
> object store and folders are not actually real things
>
> http://docs.aws.amazon.com/AmazonS3/latest/UG/FolderOperations.html <
> http://docs.aws.amazon.com/AmazonS3/latest/UG/FolderOperations.html>
>
> so for instance i have a tmp workspace
> I had to create an empty folder/prefix called tmp in the bucket and create
> table statements started to work
> after creating the initial folder its possible to put slashes in table
> names
>
> use fh;
>
> the following doesnt work
> create table `/abcde/fgh` as …
> Error: PARSE ERROR: Unable to create or drop tables/views. Schema [fh] is
> immutable
> create table `abcde/fgh`
> Error: PARSE ERROR: Unable to create or drop tables/views. Schema [fh] is
> immutable
> create table root.`abcde/fgh`
> Error: PARSE ERROR: Unable to create or drop tables/views. Schema [fh] is
> immutable
>
>
> does work
> create table tmp.`abcde`
> create table tmp.`abcde/fgh`
>
> the nesting works as you would expect when querying parquet files
> select count(*) from tmp.`abcde` … counts all data in the abcde dir and
> all sub folders
> select count(*) from tmp.`abcde/fgh` counts just the data in the fgh
> directory
>
> Cheers
>
> > On Dec 11, 2015, at 2:02 PM, Joshua Schlesser <jo...@cursivelabs.com>
> wrote:
> >
> > I am trying to create new tables on a storage configured to use s3a
> storage.
> > I copied followed the s3 instructions in the docs and copied dfs storage
> and replaced the connection with the s3a uri and set the root workspace to
> writable. tmp was already writable.
> >
> > I execute the use storage; sql command to change my default schema.
> > i then execute create table tablename as select * from schema.tablename
> >
> > the select when run outside of the context of a create table runs fine.
> >
> > I receive the following error when trying to create a table:
> > Error: PARSE ERROR: Unable to create or drop tables/views. Schema
> [xxxxxx] is immutable
> >
> > I have confirmed that I can perform select queries just fine from the s3
> bucket and the permissions associated with the keys are full access.
> >
> > Im running 1.3.0.
> >
> > Is there anything i need to do special to enable s3a storages to be
> writable?
> >
> > Thanks in advance for any insights.
> >
> > Josh Schlesser
> >
> >
> >
>
Re: s3a storage not writable
Posted by Joshua Schlesser <jo...@cursivelabs.com>.
Answer to my own question.
You can label the root workspace as writable but that is ignored.
Also it seems you cant assume that top level ‘folders’ in your s3 buckets will magically get created in a create table statement even though s3 is an object store and folders are not actually real things
http://docs.aws.amazon.com/AmazonS3/latest/UG/FolderOperations.html <http://docs.aws.amazon.com/AmazonS3/latest/UG/FolderOperations.html>
so for instance i have a tmp workspace
I had to create an empty folder/prefix called tmp in the bucket and create table statements started to work
after creating the initial folder its possible to put slashes in table names
use fh;
the following doesnt work
create table `/abcde/fgh` as …
Error: PARSE ERROR: Unable to create or drop tables/views. Schema [fh] is immutable
create table `abcde/fgh`
Error: PARSE ERROR: Unable to create or drop tables/views. Schema [fh] is immutable
create table root.`abcde/fgh`
Error: PARSE ERROR: Unable to create or drop tables/views. Schema [fh] is immutable
does work
create table tmp.`abcde`
create table tmp.`abcde/fgh`
the nesting works as you would expect when querying parquet files
select count(*) from tmp.`abcde` … counts all data in the abcde dir and all sub folders
select count(*) from tmp.`abcde/fgh` counts just the data in the fgh directory
Cheers
> On Dec 11, 2015, at 2:02 PM, Joshua Schlesser <jo...@cursivelabs.com> wrote:
>
> I am trying to create new tables on a storage configured to use s3a storage.
> I copied followed the s3 instructions in the docs and copied dfs storage and replaced the connection with the s3a uri and set the root workspace to writable. tmp was already writable.
>
> I execute the use storage; sql command to change my default schema.
> i then execute create table tablename as select * from schema.tablename
>
> the select when run outside of the context of a create table runs fine.
>
> I receive the following error when trying to create a table:
> Error: PARSE ERROR: Unable to create or drop tables/views. Schema [xxxxxx] is immutable
>
> I have confirmed that I can perform select queries just fine from the s3 bucket and the permissions associated with the keys are full access.
>
> Im running 1.3.0.
>
> Is there anything i need to do special to enable s3a storages to be writable?
>
> Thanks in advance for any insights.
>
> Josh Schlesser
>
>
>