You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iceberg.apache.org by Ivan Yang <iv...@gmail.com> on 2021/07/12 23:31:26 UTC

Trino Iceberg

Dear Iceberg community,

I recently try to set up Presto/Trino to connect to the Iceberg data store. I generated Iceberg db/table/data using a Flink job via Glue Catalog. So after Flink job run. My Iceberg meta and data files are in AWS s3. I can use flink sql-client to query the iceberg table. Now I try to set up Trino catalog to query the same icerberg table. I don’t know how to configure trino/etc/catalog/iceberg.properties file. Here is the CATALOG I run in the flink sql-client.
	
CREATE CATALOG my_ice WITH (
  'type'='iceberg',
  'warehouse'='s3://raw-corpus-test/my-iceberg',
  'catalog-impl'='org.apache.iceberg.aws.glue.GlueCatalog',
  'io-impl'='org.apache.iceberg.aws.s3.S3FileIO',
  'lock-impl'='org.apache.iceberg.aws.glue.DynamoLockManager',
  'lock.table'='myGlueLockTable'
);

I saw documentation of

connector.name=iceberg
hive.metastore.uri=thrift://localhost:9083

My question is what value to put for hive.metastore.uri in my case where my meta and data files are in s3?

Thanks,
Ivan

Re: Trino Iceberg

Posted by Ivan Yang <iv...@gmail.com>.
Hi Yan,

Ah, thanks for pointing out the PR, that will likely fit in my approach well. Do you have any idea when this will be merged into Trino Iceberg connector? In the meantime, If s3 is my iceberg data storage, what available ingestion and query methods will you suggest?

Thanks,
Ivan

> On Jul 12, 2021, at 11:26 PM, Yan Yan <yy...@gmail.com> wrote:
> 
> Hi Ivan,
> 
> Today Trino Iceberg connector does not support Glue catalog. However Jack has authored a PR for this support in Trino community that you may be interested in: https://github.com/trinodb/trino/pull/6977 <https://github.com/trinodb/trino/pull/6977> 
> 
> Yan
> 
> On Mon, Jul 12, 2021 at 4:31 PM Ivan Yang <ivanygyang@gmail.com <ma...@gmail.com>> wrote:
> Dear Iceberg community,
> 
> I recently try to set up Presto/Trino to connect to the Iceberg data store. I generated Iceberg db/table/data using a Flink job via Glue Catalog. So after Flink job run. My Iceberg meta and data files are in AWS s3. I can use flink sql-client to query the iceberg table. Now I try to set up Trino catalog to query the same icerberg table. I don’t know how to configure trino/etc/catalog/iceberg.properties file. Here is the CATALOG I run in the flink sql-client.
> 	
> CREATE CATALOG my_ice WITH (
>   'type'='iceberg',
>   'warehouse'='s3://raw-corpus-test/my-iceberg <>',
>   'catalog-impl'='org.apache.iceberg.aws.glue.GlueCatalog',
>   'io-impl'='org.apache.iceberg.aws.s3.S3FileIO',
>   'lock-impl'='org.apache.iceberg.aws.glue.DynamoLockManager',
>   'lock.table'='myGlueLockTable'
> );
> 
> I saw documentation of
> 
> connector.name <http://connector.name/>=iceberg
> hive.metastore.uri=thrift://localhost:9083 <>
> 
> My question is what value to put for hive.metastore.uri in my case where my meta and data files are in s3?
> 
> Thanks,
> Ivan
> 
> 
>

Re: Trino Iceberg

Posted by Yan Yan <yy...@gmail.com>.
Hi Ivan,

Today Trino Iceberg connector does not support Glue catalog. However Jack
has authored a PR for this support in Trino community that you may be
interested in: https://github.com/trinodb/trino/pull/6977

Yan

On Mon, Jul 12, 2021 at 4:31 PM Ivan Yang <iv...@gmail.com> wrote:

> Dear Iceberg community,
>
> I recently try to set up Presto/Trino to connect to the Iceberg data
> store. I generated Iceberg db/table/data using a Flink job via Glue
> Catalog. So after Flink job run. My Iceberg meta and data files are in AWS
> s3. I can use flink sql-client to query the iceberg table. Now I try to set
> up Trino catalog to query the same icerberg table. I don’t know how to
> configure trino/etc/catalog/iceberg.properties file. Here is the CATALOG I
> run in the flink sql-client.
> CREATE CATALOG my_ice WITH (
>   'type'='iceberg',
>   'warehouse'='s3://raw-corpus-test/my-iceberg',
>   'catalog-impl'='org.apache.iceberg.aws.glue.GlueCatalog',
>   'io-impl'='org.apache.iceberg.aws.s3.S3FileIO',
>   'lock-impl'='org.apache.iceberg.aws.glue.DynamoLockManager',
>   'lock.table'='myGlueLockTable'
> );
>
> I saw documentation of
>
> connector.name=iceberg
> hive.metastore.uri=thrift://localhost:9083
>
> My question is what value to put for hive.metastore.uri in my case where
> my meta and data files are in s3?
>
> Thanks,
> Ivan
>
>
>
>