hive:4.0.1 docker with s3a managed location

[chris snow <ch...@gmail.com>]

I'm using the hive docker-compose quickstart with 4.0.1

I've created a schema:

CREATE SCHEMA mydb MANAGEDLOCATION 's3a://datastore/mydb';

But when I DESCRIBE it, I see the following:

+----------+----------+-----------------------------------------+-----------------------------------------+-------------+-------------+-----------------+----------------+
| db_name  | comment  |                location                 |
       managedlocation             | owner_name  | owner_type  |
connector_name  | remote_dbname  |
+----------+----------+-----------------------------------------+-----------------------------------------+-------------+-------------+-----------------+----------------+
| mydb    |          | file:/opt/hive/data/warehouse/mydb.db  |
file:/opt/hive/data/warehouse/mydb.db  | hive        | USER        |
              |                |
+----------+----------+-----------------------------------------+-----------------------------------------+-------------+-------------+-----------------+----------------+

0: jdbc:hive2://localhost:10000/> use mydb;
0: jdbc:hive2://localhost:10000/> create table test (col1 int);
org.apache.hadoop.hive.ql.metadata.HiveException:
MetaException(message:file:/opt/hive/data/warehouse/mydb.db/test is
not a directory or unable to create one)

Any ideas?  Should the managedlocation in DESCRIBE show s3a:// ?

Re: hive:4.0.1 docker with s3a managed location

[Denys Kuzmenko <dk...@apache.org>]

Hi Chris,

I wanted to try the same with the official docker image, however, looks like /opt/hadoop/share/hadoop/tools/lib/hadoop-aws-3.3.6.jar is not even added to classpath: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found

I'll create a ticket for that.

But to answer your original question: "Should the managedlocation in DESCRIBE show s3a:// ?"
- Yes, it should.

Here is are the same steps done on my internal test env. Note, it's slightly different from upstream:

CREATE SCHEMA mydb MANAGEDLOCATION 's3a://dw-bucket/data/warehouse/tablespace/managed/mydb'

describe schema mydb;
+----------+----------+----------------------------------------------------+----------------------------------------------------+-----------------+-------------+-----------------+----------------+
| db_name  | comment  |                      location                      |                  managedlocation                   |   owner_name    | owner_type  | connector_name  | remote_dbname  |
+----------+----------+----------------------------------------------------+----------------------------------------------------+-----------------+-------------+-----------------+----------------+
| mydb     |          | s3a://dw-bucket/data/warehouse/tablespace/external/hive/mydb.db | s3a://dw-bucket/data/warehouse/tablespace/managed/mydb | hive  | USER        |                 |                |
+----------+----------+----------------------------------------------------+----------------------------------------------------+-----------------+-------------+-----------------+----------------+

use mydb;
create table test (col1 int);

describe formatted test;

Location:                     | s3a://dw-bucket/data/warehouse/tablespace/managed/mydb/test | NULL                                               |
| Table Type:                   | MANAGED_TABLE                                      | NULL

Regards,
Denys