You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/07/15 22:55:23 UTC
[GitHub] [iceberg] jackye1995 opened a new pull request #2831: Doc: add documentation for JDBC and DynamoDB catalogs
jackye1995 opened a new pull request #2831:
URL: https://github.com/apache/iceberg/pull/2831
@yyanyy @cwsteinbach
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] yyanyy commented on a change in pull request #2831: Doc: add documentation for JDBC and DynamoDB catalogs
Posted by GitBox <gi...@apache.org>.
yyanyy commented on a change in pull request #2831:
URL: https://github.com/apache/iceberg/pull/2831#discussion_r673585442
##########
File path: site/docs/aws.md
##########
@@ -238,6 +242,59 @@ LOCATION 's3://my-special-table-bucket'
PARTITIONED BY (category);
```
+### DynamoDB Catalog
+
+Iceberg supports using a [DynamoDB](https://aws.amazon.com/dynamodb) table to record and manage database and table information.
+
+#### Configurations
+
+The DynamoDB catalog supports the following configurations:
+
+| Property | Default | Description |
+| --------------------------------- | -------------------------------------------------- | ------------------------------------------------------ |
+| dynamodb.table-name | iceberg | name of the DynamoDB table used by DynamoDbCatalog |
+
+
+#### Internal Table Design
+
+The DynamoDB table is designed with the following columns:
+
+| Column | Key | Type | Description |
+| ----------------- | --------------- | ----------- |--------------------------------------------------------------------- |
+| identifier | partition key | string | table identifier such as `db1.table1`, or `NAMESPACE` for namespaces |
+| namespace | sort key | string | namespace name. A [global secondary index (GSI)](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/GSI.html) is created with namespace as partition key, identifier as sort key, no other projected columns |
+| v | | string | row version, used for optimistic locking |
+| updated_at | | number | timestamp (millis) of the last update |
+| created_at | | number | timestamp (millis) of the table creation |
+| p.<property_key\> | | string | Iceberg-defined table properties including `table_type`, `metadata_location` and `previous_metadata_location` or namespace properties
+
+This design has the following benefits:
+
+1. table name is used directly as partition key to avoid any potential [hot partition issue](https://aws.amazon.com/premiumsupport/knowledge-center/dynamodb-table-throttled/), comparing to use namespace as partition key and table name as sort key
Review comment:
if my understanding from the comment above was correct, I think "comparing to use namespace as partition key and table name as sort key" here may imply that identifier will be always including table name; what about something like "avoid potential hot partition issue if there are heavy write traffic to tables within the same namespace, since users can configure the key to be on table level"
##########
File path: site/docs/aws.md
##########
@@ -238,6 +242,59 @@ LOCATION 's3://my-special-table-bucket'
PARTITIONED BY (category);
```
+### DynamoDB Catalog
+
+Iceberg supports using a [DynamoDB](https://aws.amazon.com/dynamodb) table to record and manage database and table information.
+
+#### Configurations
+
+The DynamoDB catalog supports the following configurations:
+
+| Property | Default | Description |
+| --------------------------------- | -------------------------------------------------- | ------------------------------------------------------ |
+| dynamodb.table-name | iceberg | name of the DynamoDB table used by DynamoDbCatalog |
+
+
+#### Internal Table Design
+
+The DynamoDB table is designed with the following columns:
+
+| Column | Key | Type | Description |
+| ----------------- | --------------- | ----------- |--------------------------------------------------------------------- |
+| identifier | partition key | string | table identifier such as `db1.table1`, or `NAMESPACE` for namespaces |
Review comment:
sorry what do you mean by "or `NAMESPACE` for namespaces"? this `identifier` can be just the namespaces for sharing a single table with all tables within namespaces? And for the namespace case, is that always be this exact "NAMESPACE" string, or this is just an example?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jackye1995 commented on a change in pull request #2831: Doc: add documentation for JDBC and DynamoDB catalogs
Posted by GitBox <gi...@apache.org>.
jackye1995 commented on a change in pull request #2831:
URL: https://github.com/apache/iceberg/pull/2831#discussion_r673591122
##########
File path: site/docs/aws.md
##########
@@ -238,6 +242,59 @@ LOCATION 's3://my-special-table-bucket'
PARTITIONED BY (category);
```
+### DynamoDB Catalog
+
+Iceberg supports using a [DynamoDB](https://aws.amazon.com/dynamodb) table to record and manage database and table information.
+
+#### Configurations
+
+The DynamoDB catalog supports the following configurations:
+
+| Property | Default | Description |
+| --------------------------------- | -------------------------------------------------- | ------------------------------------------------------ |
+| dynamodb.table-name | iceberg | name of the DynamoDB table used by DynamoDbCatalog |
+
+
+#### Internal Table Design
+
+The DynamoDB table is designed with the following columns:
+
+| Column | Key | Type | Description |
+| ----------------- | --------------- | ----------- |--------------------------------------------------------------------- |
+| identifier | partition key | string | table identifier such as `db1.table1`, or `NAMESPACE` for namespaces |
Review comment:
yes, that exact string
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] yyanyy commented on a change in pull request #2831: Doc: add documentation for JDBC and DynamoDB catalogs
Posted by GitBox <gi...@apache.org>.
yyanyy commented on a change in pull request #2831:
URL: https://github.com/apache/iceberg/pull/2831#discussion_r673584376
##########
File path: site/docs/aws.md
##########
@@ -238,6 +242,59 @@ LOCATION 's3://my-special-table-bucket'
PARTITIONED BY (category);
```
+### DynamoDB Catalog
+
+Iceberg supports using a [DynamoDB](https://aws.amazon.com/dynamodb) table to record and manage database and table information.
+
+#### Configurations
+
+The DynamoDB catalog supports the following configurations:
+
+| Property | Default | Description |
+| --------------------------------- | -------------------------------------------------- | ------------------------------------------------------ |
+| dynamodb.table-name | iceberg | name of the DynamoDB table used by DynamoDbCatalog |
+
+
+#### Internal Table Design
+
+The DynamoDB table is designed with the following columns:
+
+| Column | Key | Type | Description |
+| ----------------- | --------------- | ----------- |--------------------------------------------------------------------- |
+| identifier | partition key | string | table identifier such as `db1.table1`, or `NAMESPACE` for namespaces |
Review comment:
sorry what do you mean by "or `NAMESPACE` for namespaces"? this `identifier` can be just the namespaces for sharing a single table with all tables within namespaces? And for the namespace case, is that always be this exact "NAMESPACE" string, or this is just an example (i.e. we repeat the same value in `namespace` column here)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] XuQianJin-Stars commented on pull request #2831: Doc: add documentation for JDBC and DynamoDB catalogs
Posted by GitBox <gi...@apache.org>.
XuQianJin-Stars commented on pull request #2831:
URL: https://github.com/apache/iceberg/pull/2831#issuecomment-881835549
+1 LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] XuQianJin-Stars commented on a change in pull request #2831: Doc: add documentation for JDBC and DynamoDB catalogs
Posted by GitBox <gi...@apache.org>.
XuQianJin-Stars commented on a change in pull request #2831:
URL: https://github.com/apache/iceberg/pull/2831#discussion_r671086236
##########
File path: site/docs/jdbc.md
##########
@@ -0,0 +1,48 @@
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+# Iceberg JDBC Integration
+
+## JDBC Catalog
+
+Iceberg supports using a table in a relational database to manage Iceberg tables through JDBC.
+The database that JDBC connects to must support atomic transaction to allow the JDBC catalog implementation to
+properly support atomic Iceberg table commits and read serializable isolation.
+
+### Configurations
+
+Because each database and database service provider might require different configurations,
+the JDBC catalog allows arbitrary configurations through:
+
+| Property | Default | Description |
+| -------------------- | --------------------------------- | ------------------------------------------------------ |
+| uri | | the JDBC connection string |
+| jdbc.<property_key\> | | any key value pairs to configure the JDBC connection |
+
+For example, you can start a Spark session with a MySQL JDBC connection using the following configurations:
Review comment:
Is it possible to add here to operate from the jdbc api level?
```
static final String JDBC_DRIVER = "com.mysql.cj.jdbc.Driver";
static final String DB_URL = "jdbc:mysql://localhost:3306/test";
Class.forName(JDBC_DRIVER);
Map<String, String> properties = new HashMap<>();
properties.put(CatalogProperties.URI, DB_URL);
properties.put(JdbcCatalog.PROPERTY_PREFIX + "user", USER);
properties.put(JdbcCatalog.PROPERTY_PREFIX + "password", PASSWORD);
properties.put(CatalogProperties.WAREHOUSE_LOCATION, WAREHOUSE_LOCATION);
JdbcCatalog catalog = new JdbcCatalog();
Configuration conf = new Configuration();
catalog.setConf(conf);
catalog.initialize("test_jdbc_catalog", properties);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] yyanyy merged pull request #2831: Doc: add documentation for JDBC and DynamoDB catalogs
Posted by GitBox <gi...@apache.org>.
yyanyy merged pull request #2831:
URL: https://github.com/apache/iceberg/pull/2831
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jackye1995 commented on a change in pull request #2831: Doc: add documentation for JDBC and DynamoDB catalogs
Posted by GitBox <gi...@apache.org>.
jackye1995 commented on a change in pull request #2831:
URL: https://github.com/apache/iceberg/pull/2831#discussion_r671444819
##########
File path: site/docs/jdbc.md
##########
@@ -0,0 +1,48 @@
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+# Iceberg JDBC Integration
+
+## JDBC Catalog
+
+Iceberg supports using a table in a relational database to manage Iceberg tables through JDBC.
+The database that JDBC connects to must support atomic transaction to allow the JDBC catalog implementation to
+properly support atomic Iceberg table commits and read serializable isolation.
+
+### Configurations
+
+Because each database and database service provider might require different configurations,
+the JDBC catalog allows arbitrary configurations through:
+
+| Property | Default | Description |
+| -------------------- | --------------------------------- | ------------------------------------------------------ |
+| uri | | the JDBC connection string |
+| jdbc.<property_key\> | | any key value pairs to configure the JDBC connection |
+
+For example, you can start a Spark session with a MySQL JDBC connection using the following configurations:
Review comment:
thanks, added one block for Java API.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org