You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/09/07 12:04:38 UTC
[GitHub] [iceberg] openinx opened a new pull request #3085: Docs: Add flink iceberg connector
openinx opened a new pull request #3085:
URL: https://github.com/apache/iceberg/pull/3085
Add document for the PR #2666 . I decided to prepare a document for this because I've seen people were asking how to use the flink `'connector'='iceberg'` to create iceberg table in https://github.com/apache/iceberg/issues/3079
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jackye1995 commented on a change in pull request #3085: Docs: Add flink iceberg connector
Posted by GitBox <gi...@apache.org>.
jackye1995 commented on a change in pull request #3085:
URL: https://github.com/apache/iceberg/pull/3085#discussion_r708884766
##########
File path: site/docs/flink-connector.md
##########
@@ -0,0 +1,138 @@
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+Apache Iceberg supports creating flink table directly without creating the explicit flink catalog in flink SQL in [#2666](https://github.com/apache/iceberg/pull/2666). That means we can just create an iceberg table by specifying `'connector'='iceberg'` table option in flink SQL which is similar to usage in the flink official [document](https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/connectors/table/overview/).
+
+In flink, the SQL `CREATE TABLE test (..) WITH ('connector'='iceberg', ...)` will create an flink table in current flink catalog (use [GenericInMemoryCatalog](https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/table/catalogs/#genericinmemorycatalog) by default),
+which is just mapping to the underlying iceberg table instead of maintaining iceberg table directly in current flink catalog.
+
+To create the table in flink SQL by using SQL syntax `CREATE TABLE test (..) WITH ('connector'='iceberg', ...)`, flink iceberg connector provides the following table properties:
+
+* `connector`: Use the constant `iceberg`.
+* `catalog-name`: User-specified catalog name. It's required because the connector don't have any default value.
+* `catalog-type`: Default to use `hive` if don't specify any value. The optional values are:
+ * `hive`: The hive metastore catalog.
+ * `hadoop`: The hadoop catalog.
+ * `custom`: The customized catalog, see [custom catalog](./custom-catalog.md) for more details.
+* `catalog-database`: The iceberg database name in the backend catalog, use the current flink database name by default.
+* `catalog-table`: The iceberg table name in the backend catalog. Default to use the table name in the flink `CREATE TABLE` sentence.
+
+## Table managed in hive catalog.
+
+Before executing the following SQL, please make sure you've configured the flink SQL client correctly according to the quick start [document](./flink.md).
+
+The following SQL will create an flink table in the current flink catalog, which maps to the iceberg table `default_database.iceberg_table` managed in iceberg catalog.
+
+```sql
+CREATE TABLE flink_table (
+ id BIGINT,
+ data STRING
+) WITH (
+ 'connector'='iceberg',
+ 'catalog-name'='hive_prod',
+ 'uri'='thrift://localhost:9083',
+ 'warehouse'='hdfs://nn:8020/path/to/warehouse'
+);
+```
+
+If you want to create a flink table mapping to a different iceberg table managed in hive catalog (such as `hive_db.hive_iceberg_table` in hive), then you can create flink table as following:
+
+```sql
+CREATE TABLE flink_table (
+ id BIGINT,
+ data STRING
+) WITH (
+ 'connector'='iceberg',
+ 'catalog-name'='hive_prod',
+ 'catalog-database'='hive_db',
+ 'catalog-table'='hive_iceberg_table',
+ 'uri'='thrift://localhost:9083',
+ 'warehouse'='hdfs://nn:8020/path/to/warehouse'
+);
+```
+
+!!! Note
+ The underlying catalog database (`hive_db` in the above example) will be created automatically if it does not exist when writing records into the flink table.
+
+## Table managed in hadoop catalog
+
+The following SQL will create an flink table in current flink catalog, which maps to the iceberg table `default_database.flink_table` managed in hadoop catalog.
+
+```sql
+CREATE TABLE flink_table (
+ id BIGINT,
+ data STRING
+) WITH (
+ 'connector'='iceberg',
+ 'catalog-name'='hadoop_prod',
+ 'catalog-type'='hadoop',
+ 'warehouse'='hdfs://nn:8020/path/to/warehouse'
+);
+```
+
+## Table managed in custom catalog
+
+The following SQL will create an flink table in current flink catalog, which maps to the iceberg table `default_database.flink_table` managed in custom catalog.
+
+```sql
+CREATE TABLE flink_table (
+ id BIGINT,
+ data STRING
+) WITH (
+ 'connector'='iceberg',
+ 'catalog-name'='custom_prod',
+ 'catalog-type'='custom',
+ 'catalog-impl'='com.my.custom.CatalogImpl',
+ -- More table properties for the customized catalog
+ 'my-additional-catalog-config'='my-value',
+ ...
+);
+```
+
+Please refer to [AWS](./aws.md#catalogs), [Nessie](./nessie.md), [JDBC](./jdbc.md) catalog for more details.
+
+## A complete example.
+
+Take the hive catalog as an example:
+
+```sql
+CREATE TABLE flink_table (
+ id BIGINT,
+ data STRING
+) WITH (
+ 'connector'='iceberg',
+ 'catalog-name'='hive_prod',
+ 'uri'='thrift://localhost:9083',
+ 'warehouse'='file:///path/to/warehouse'
+);
+
+INSERT INTO flink_table VALUES (1, 'AAA'), (2, 'BBB'), (3, 'CCC');
+
+SET execution.result-mode=tableau;
+SELECT * FROM flink_table;
+
++----+------+
+| id | data |
++----+------+
+| 1 | AAA |
+| 2 | BBB |
+| 3 | CCC |
++----+------+
+3 rows in set
+```
+
+Please refer to [document](./flink.md) for queries and writes.
Review comment:
nit: For more details, please refer to the Iceberg Flink document.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] openinx commented on a change in pull request #3085: Docs: Add flink iceberg connector
Posted by GitBox <gi...@apache.org>.
openinx commented on a change in pull request #3085:
URL: https://github.com/apache/iceberg/pull/3085#discussion_r711543543
##########
File path: site/docs/flink-connector.md
##########
@@ -0,0 +1,138 @@
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+Apache Iceberg supports creating flink table directly without creating the explicit flink catalog in flink SQL in [#2666](https://github.com/apache/iceberg/pull/2666). That means we can just create an iceberg table by specifying `'connector'='iceberg'` table option in flink SQL which is similar to usage in the flink official [document](https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/connectors/table/overview/).
+
+In flink, the SQL `CREATE TABLE test (..) WITH ('connector'='iceberg', ...)` will create an flink table in current flink catalog (use [GenericInMemoryCatalog](https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/table/catalogs/#genericinmemorycatalog) by default),
+which is just mapping to the underlying iceberg table instead of maintaining iceberg table directly in current flink catalog.
+
+To create the table in flink SQL by using SQL syntax `CREATE TABLE test (..) WITH ('connector'='iceberg', ...)`, flink iceberg connector provides the following table properties:
+
+* `connector`: Use the constant `iceberg`.
+* `catalog-name`: User-specified catalog name. It's required because the connector don't have any default value.
+* `catalog-type`: Default to use `hive` if don't specify any value. The optional values are:
+ * `hive`: The hive metastore catalog.
+ * `hadoop`: The hadoop catalog.
+ * `custom`: The customized catalog, see [custom catalog](./custom-catalog.md) for more details.
+* `catalog-database`: The iceberg database name in the backend catalog, use the current flink database name by default.
+* `catalog-table`: The iceberg table name in the backend catalog. Default to use the table name in the flink `CREATE TABLE` sentence.
+
+## Table managed in hive catalog.
+
+Before executing the following SQL, please make sure you've configured the flink SQL client correctly according to the quick start [document](./flink.md).
+
+The following SQL will create an flink table in the current flink catalog, which maps to the iceberg table `default_database.iceberg_table` managed in iceberg catalog.
+
+```sql
+CREATE TABLE flink_table (
+ id BIGINT,
+ data STRING
+) WITH (
+ 'connector'='iceberg',
+ 'catalog-name'='hive_prod',
+ 'uri'='thrift://localhost:9083',
+ 'warehouse'='hdfs://nn:8020/path/to/warehouse'
+);
+```
+
+If you want to create a flink table mapping to a different iceberg table managed in hive catalog (such as `hive_db.hive_iceberg_table` in hive), then you can create flink table as following:
+
+```sql
+CREATE TABLE flink_table (
+ id BIGINT,
+ data STRING
+) WITH (
+ 'connector'='iceberg',
+ 'catalog-name'='hive_prod',
+ 'catalog-database'='hive_db',
+ 'catalog-table'='hive_iceberg_table',
+ 'uri'='thrift://localhost:9083',
+ 'warehouse'='hdfs://nn:8020/path/to/warehouse'
+);
+```
+
+!!! Note
+ The underlying catalog database (`hive_db` in the above example) will be created automatically if it does not exist when writing records into the flink table.
Review comment:
Yes, that's the correct behavior. Because the `CREATE TABLE test (..) WITH ('connector'='iceberg', ...)` is actually creating a table in in-memory catalog by default and it won't do any real thing in the underlying fs. The database & table will be created only when the records are planing to write to table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jackye1995 commented on a change in pull request #3085: Docs: Add flink iceberg connector
Posted by GitBox <gi...@apache.org>.
jackye1995 commented on a change in pull request #3085:
URL: https://github.com/apache/iceberg/pull/3085#discussion_r708881627
##########
File path: site/docs/flink-connector.md
##########
@@ -0,0 +1,138 @@
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+Apache Iceberg supports creating flink table directly without creating the explicit flink catalog in flink SQL in [#2666](https://github.com/apache/iceberg/pull/2666). That means we can just create an iceberg table by specifying `'connector'='iceberg'` table option in flink SQL which is similar to usage in the flink official [document](https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/connectors/table/overview/).
Review comment:
nit: make sure all "Flink" are capitalized.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jackye1995 commented on a change in pull request #3085: Docs: Add flink iceberg connector
Posted by GitBox <gi...@apache.org>.
jackye1995 commented on a change in pull request #3085:
URL: https://github.com/apache/iceberg/pull/3085#discussion_r708882358
##########
File path: site/docs/flink-connector.md
##########
@@ -0,0 +1,138 @@
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+Apache Iceberg supports creating flink table directly without creating the explicit flink catalog in flink SQL in [#2666](https://github.com/apache/iceberg/pull/2666). That means we can just create an iceberg table by specifying `'connector'='iceberg'` table option in flink SQL which is similar to usage in the flink official [document](https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/connectors/table/overview/).
+
+In flink, the SQL `CREATE TABLE test (..) WITH ('connector'='iceberg', ...)` will create an flink table in current flink catalog (use [GenericInMemoryCatalog](https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/table/catalogs/#genericinmemorycatalog) by default),
+which is just mapping to the underlying iceberg table instead of maintaining iceberg table directly in current flink catalog.
+
+To create the table in flink SQL by using SQL syntax `CREATE TABLE test (..) WITH ('connector'='iceberg', ...)`, flink iceberg connector provides the following table properties:
+
+* `connector`: Use the constant `iceberg`.
+* `catalog-name`: User-specified catalog name. It's required because the connector don't have any default value.
+* `catalog-type`: Default to use `hive` if don't specify any value. The optional values are:
+ * `hive`: The hive metastore catalog.
+ * `hadoop`: The hadoop catalog.
+ * `custom`: The customized catalog, see [custom catalog](./custom-catalog.md) for more details.
+* `catalog-database`: The iceberg database name in the backend catalog, use the current flink database name by default.
+* `catalog-table`: The iceberg table name in the backend catalog. Default to use the table name in the flink `CREATE TABLE` sentence.
+
+## Table managed in hive catalog.
+
+Before executing the following SQL, please make sure you've configured the flink SQL client correctly according to the quick start [document](./flink.md).
+
+The following SQL will create an flink table in the current flink catalog, which maps to the iceberg table `default_database.iceberg_table` managed in iceberg catalog.
+
+```sql
+CREATE TABLE flink_table (
+ id BIGINT,
+ data STRING
+) WITH (
+ 'connector'='iceberg',
+ 'catalog-name'='hive_prod',
+ 'uri'='thrift://localhost:9083',
+ 'warehouse'='hdfs://nn:8020/path/to/warehouse'
+);
+```
+
+If you want to create a flink table mapping to a different iceberg table managed in hive catalog (such as `hive_db.hive_iceberg_table` in hive), then you can create flink table as following:
Review comment:
nit: capitalize "Hive"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] openinx merged pull request #3085: Docs: Add flink iceberg connector
Posted by GitBox <gi...@apache.org>.
openinx merged pull request #3085:
URL: https://github.com/apache/iceberg/pull/3085
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jackye1995 commented on a change in pull request #3085: Docs: Add flink iceberg connector
Posted by GitBox <gi...@apache.org>.
jackye1995 commented on a change in pull request #3085:
URL: https://github.com/apache/iceberg/pull/3085#discussion_r708880797
##########
File path: site/docs/flink-connector.md
##########
@@ -0,0 +1,138 @@
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+Apache Iceberg supports creating flink table directly without creating the explicit flink catalog in flink SQL in [#2666](https://github.com/apache/iceberg/pull/2666). That means we can just create an iceberg table by specifying `'connector'='iceberg'` table option in flink SQL which is similar to usage in the flink official [document](https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/connectors/table/overview/).
Review comment:
should it be the other way? `Flink supports creating Iceberg tables directly ...`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] openinx commented on a change in pull request #3085: Docs: Add flink iceberg connector
Posted by GitBox <gi...@apache.org>.
openinx commented on a change in pull request #3085:
URL: https://github.com/apache/iceberg/pull/3085#discussion_r711551137
##########
File path: site/mkdocs.yml
##########
@@ -75,7 +75,9 @@ nav:
- Maintenance Procedures: spark-procedures.md
- Structured Streaming: spark-structured-streaming.md
- Time Travel: spark-queries/#time-travel
- - Flink: flink.md
+ - Flink:
+ - Getting Started: flink.md
Review comment:
Yes, I think it's necessary to break the flink into more pages so that people could get the correct point in the separate page. We will also introduce more feature for iceberg flink integration work, such as flink table maintaince API , flip-27 unified source/sink etc.
I will prefer to publish a separate PR to address the break-down thing, let this one focus on adding the iceberg flink connector.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jackye1995 commented on a change in pull request #3085: Docs: Add flink iceberg connector
Posted by GitBox <gi...@apache.org>.
jackye1995 commented on a change in pull request #3085:
URL: https://github.com/apache/iceberg/pull/3085#discussion_r708885434
##########
File path: site/mkdocs.yml
##########
@@ -75,7 +75,9 @@ nav:
- Maintenance Procedures: spark-procedures.md
- Structured Streaming: spark-structured-streaming.md
- Time Travel: spark-queries/#time-travel
- - Flink: flink.md
+ - Flink:
+ - Getting Started: flink.md
Review comment:
I am thinking we might want to break down the flink documentation like Spark (no need to do in this PR), just to make it more consistent between the 2 and easier to navigate. What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jackye1995 commented on a change in pull request #3085: Docs: Add flink iceberg connector
Posted by GitBox <gi...@apache.org>.
jackye1995 commented on a change in pull request #3085:
URL: https://github.com/apache/iceberg/pull/3085#discussion_r712380171
##########
File path: site/docs/flink-connector.md
##########
@@ -0,0 +1,138 @@
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+Apache Flink supports creating Iceberg table directly without creating the explicit Flink catalog in Flink SQL. That means we can just create an iceberg table by specifying `'connector'='iceberg'` table option in Flink SQL which is similar to usage in the Flink official [document](https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/connectors/table/overview/).
+
+In Flink, the SQL `CREATE TABLE test (..) WITH ('connector'='iceberg', ...)` will create an Flink table in current Flink catalog (use [GenericInMemoryCatalog](https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/table/catalogs/#genericinmemorycatalog) by default),
Review comment:
nit: a Flink table
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jackye1995 commented on a change in pull request #3085: Docs: Add flink iceberg connector
Posted by GitBox <gi...@apache.org>.
jackye1995 commented on a change in pull request #3085:
URL: https://github.com/apache/iceberg/pull/3085#discussion_r708881237
##########
File path: site/docs/flink-connector.md
##########
@@ -0,0 +1,138 @@
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+Apache Iceberg supports creating flink table directly without creating the explicit flink catalog in flink SQL in [#2666](https://github.com/apache/iceberg/pull/2666). That means we can just create an iceberg table by specifying `'connector'='iceberg'` table option in flink SQL which is similar to usage in the flink official [document](https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/connectors/table/overview/).
Review comment:
Also I think we don't need to mention PR links in the documentation (at least I did not see anything referenced in other places)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on a change in pull request #3085: Docs: Add flink iceberg connector
Posted by GitBox <gi...@apache.org>.
stevenzwu commented on a change in pull request #3085:
URL: https://github.com/apache/iceberg/pull/3085#discussion_r703605163
##########
File path: site/docs/flink-connector.md
##########
@@ -0,0 +1,138 @@
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+Apache Iceberg support creating flink table directly without creating explicit flink catalog in flink SQL in [#2666](https://github.com/apache/iceberg/pull/2666). That means we can just create an iceberg table with specifying 'connector'='iceberg' table option in flink SQL which is similar to the flink official [document](https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/connectors/table/overview/).
+
+In flink, the SQL `CREATE TABLE test (..) WITH ('connector'='iceberg', ...)` will create an flink table in current flink catalog (use [GenericInMemoryCatalog](https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/table/catalogs/#genericinmemorycatalog) by default),
+which is just map to the underlying iceberg table instead of maintaining iceberg table.
+
+To create flink table backend iceberg table in flink SQL by using syntax `CREATE TABLE test (..) WITH ('connector'='iceberg', ...)`, flink iceberg connector provides the following table properties:
+
+* `connector`: Use the constant `iceberg`.
+* `catalog-name`: User-specified catalog name.
+* `catalog-type`: The optional values are:
+ * `hive`: The hive metastore catalog. Use `hive` by default if we don't specify any value for `catalog-type`.
+ * `hadoop`: The hadoop catalog.
+ * `custom`: All database and tables are maintained in the customized catalog, see [custom catalog](https://iceberg.apache.org/custom-catalog/) for more details.
+* `catalog-database`: The iceberg database name in the backend catalog, use the current flink database name by default.
+* `catalog-table`: The iceberg table name in the backend catalog.
+
+## Table managed in hive catalog.
+
+Before executing the following SQL, please make sure you've configured the flink SQL client correctly according to the quick start [document](./flink.md).
+
+The following SQL will create an flink table in the current flink catalog, which maps to the iceberg table `default_database.iceberg_table` managed in iceberg catalog.
+
+```sql
+CREATE TABLE flink_table (
+ id BIGINT,
+ data STRING
+) WITH (
+ 'connector'='iceberg',
+ 'catalog-name'='hive_prod',
+ 'uri'='thrift://localhost:9083',
+ 'warehouse'='file:///path/to/warehouse'
+);
+```
+
+If you want to create a flink table mapping to a different iceberg table managed in hive catalog (such as `hive_db.hive_iceberg_table` in hive), then you can create flink table as following:
+
+```sql
+CREATE TABLE flink_table (
+ id BIGINT,
+ data STRING
+) WITH (
+ 'connector'='iceberg',
+ 'catalog-name'='hive_prod',
+ 'catalog-database'='hive_db',
+ 'catalog-table'='hive_iceberg_table',
+ 'uri'='thrift://localhost:9083',
+ 'warehouse'='file:///path/to/warehouse'
+);
+```
+
+!!! Note
+ The underlying catalog database (`hive_db` in the above example) will be created automatically if it does not exist when writing records into the flink table, same thing with the underlying catalog table (`hive_iceberg_table` in the above example).
Review comment:
> same thing with the underlying catalog table (`hive_iceberg_table` in the above example
is this part redundant? the SQL is to create a new table. this is the intention and probably doesn't need to be called out as a note
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jackye1995 commented on a change in pull request #3085: Docs: Add flink iceberg connector
Posted by GitBox <gi...@apache.org>.
jackye1995 commented on a change in pull request #3085:
URL: https://github.com/apache/iceberg/pull/3085#discussion_r708882179
##########
File path: site/docs/flink-connector.md
##########
@@ -0,0 +1,138 @@
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+Apache Iceberg supports creating flink table directly without creating the explicit flink catalog in flink SQL in [#2666](https://github.com/apache/iceberg/pull/2666). That means we can just create an iceberg table by specifying `'connector'='iceberg'` table option in flink SQL which is similar to usage in the flink official [document](https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/connectors/table/overview/).
+
+In flink, the SQL `CREATE TABLE test (..) WITH ('connector'='iceberg', ...)` will create an flink table in current flink catalog (use [GenericInMemoryCatalog](https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/table/catalogs/#genericinmemorycatalog) by default),
+which is just mapping to the underlying iceberg table instead of maintaining iceberg table directly in current flink catalog.
+
+To create the table in flink SQL by using SQL syntax `CREATE TABLE test (..) WITH ('connector'='iceberg', ...)`, flink iceberg connector provides the following table properties:
+
+* `connector`: Use the constant `iceberg`.
+* `catalog-name`: User-specified catalog name. It's required because the connector don't have any default value.
+* `catalog-type`: Default to use `hive` if don't specify any value. The optional values are:
+ * `hive`: The hive metastore catalog.
+ * `hadoop`: The hadoop catalog.
+ * `custom`: The customized catalog, see [custom catalog](./custom-catalog.md) for more details.
+* `catalog-database`: The iceberg database name in the backend catalog, use the current flink database name by default.
+* `catalog-table`: The iceberg table name in the backend catalog. Default to use the table name in the flink `CREATE TABLE` sentence.
+
+## Table managed in hive catalog.
+
+Before executing the following SQL, please make sure you've configured the flink SQL client correctly according to the quick start [document](./flink.md).
+
+The following SQL will create an flink table in the current flink catalog, which maps to the iceberg table `default_database.iceberg_table` managed in iceberg catalog.
Review comment:
nit: a Flink table
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] openinx commented on a change in pull request #3085: Docs: Add flink iceberg connector
Posted by GitBox <gi...@apache.org>.
openinx commented on a change in pull request #3085:
URL: https://github.com/apache/iceberg/pull/3085#discussion_r703994325
##########
File path: site/docs/flink-connector.md
##########
@@ -0,0 +1,138 @@
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+Apache Iceberg support creating flink table directly without creating explicit flink catalog in flink SQL in [#2666](https://github.com/apache/iceberg/pull/2666). That means we can just create an iceberg table with specifying 'connector'='iceberg' table option in flink SQL which is similar to the flink official [document](https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/connectors/table/overview/).
+
+In flink, the SQL `CREATE TABLE test (..) WITH ('connector'='iceberg', ...)` will create an flink table in current flink catalog (use [GenericInMemoryCatalog](https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/table/catalogs/#genericinmemorycatalog) by default),
+which is just map to the underlying iceberg table instead of maintaining iceberg table.
+
+To create flink table backend iceberg table in flink SQL by using syntax `CREATE TABLE test (..) WITH ('connector'='iceberg', ...)`, flink iceberg connector provides the following table properties:
+
+* `connector`: Use the constant `iceberg`.
+* `catalog-name`: User-specified catalog name.
+* `catalog-type`: The optional values are:
+ * `hive`: The hive metastore catalog. Use `hive` by default if we don't specify any value for `catalog-type`.
+ * `hadoop`: The hadoop catalog.
+ * `custom`: All database and tables are maintained in the customized catalog, see [custom catalog](https://iceberg.apache.org/custom-catalog/) for more details.
+* `catalog-database`: The iceberg database name in the backend catalog, use the current flink database name by default.
+* `catalog-table`: The iceberg table name in the backend catalog.
+
+## Table managed in hive catalog.
+
+Before executing the following SQL, please make sure you've configured the flink SQL client correctly according to the quick start [document](./flink.md).
+
+The following SQL will create an flink table in the current flink catalog, which maps to the iceberg table `default_database.iceberg_table` managed in iceberg catalog.
+
+```sql
+CREATE TABLE flink_table (
+ id BIGINT,
+ data STRING
+) WITH (
+ 'connector'='iceberg',
+ 'catalog-name'='hive_prod',
+ 'uri'='thrift://localhost:9083',
+ 'warehouse'='file:///path/to/warehouse'
+);
+```
+
+If you want to create a flink table mapping to a different iceberg table managed in hive catalog (such as `hive_db.hive_iceberg_table` in hive), then you can create flink table as following:
+
+```sql
+CREATE TABLE flink_table (
+ id BIGINT,
+ data STRING
+) WITH (
+ 'connector'='iceberg',
+ 'catalog-name'='hive_prod',
+ 'catalog-database'='hive_db',
+ 'catalog-table'='hive_iceberg_table',
+ 'uri'='thrift://localhost:9083',
+ 'warehouse'='file:///path/to/warehouse'
+);
+```
+
+!!! Note
+ The underlying catalog database (`hive_db` in the above example) will be created automatically if it does not exist when writing records into the flink table, same thing with the underlying catalog table (`hive_iceberg_table` in the above example).
Review comment:
Okay, let's remove this line.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on a change in pull request #3085: Docs: Add flink iceberg connector
Posted by GitBox <gi...@apache.org>.
stevenzwu commented on a change in pull request #3085:
URL: https://github.com/apache/iceberg/pull/3085#discussion_r703606434
##########
File path: site/docs/flink-connector.md
##########
@@ -0,0 +1,138 @@
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+Apache Iceberg support creating flink table directly without creating explicit flink catalog in flink SQL in [#2666](https://github.com/apache/iceberg/pull/2666). That means we can just create an iceberg table with specifying 'connector'='iceberg' table option in flink SQL which is similar to the flink official [document](https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/connectors/table/overview/).
+
+In flink, the SQL `CREATE TABLE test (..) WITH ('connector'='iceberg', ...)` will create an flink table in current flink catalog (use [GenericInMemoryCatalog](https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/table/catalogs/#genericinmemorycatalog) by default),
+which is just map to the underlying iceberg table instead of maintaining iceberg table.
+
+To create flink table backend iceberg table in flink SQL by using syntax `CREATE TABLE test (..) WITH ('connector'='iceberg', ...)`, flink iceberg connector provides the following table properties:
Review comment:
> To create flink table backend iceberg table
This part reads weird
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] openinx commented on a change in pull request #3085: Docs: Add flink iceberg connector
Posted by GitBox <gi...@apache.org>.
openinx commented on a change in pull request #3085:
URL: https://github.com/apache/iceberg/pull/3085#discussion_r703990637
##########
File path: site/docs/flink-connector.md
##########
@@ -0,0 +1,138 @@
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+Apache Iceberg support creating flink table directly without creating explicit flink catalog in flink SQL in [#2666](https://github.com/apache/iceberg/pull/2666). That means we can just create an iceberg table with specifying 'connector'='iceberg' table option in flink SQL which is similar to the flink official [document](https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/connectors/table/overview/).
+
+In flink, the SQL `CREATE TABLE test (..) WITH ('connector'='iceberg', ...)` will create an flink table in current flink catalog (use [GenericInMemoryCatalog](https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/table/catalogs/#genericinmemorycatalog) by default),
+which is just map to the underlying iceberg table instead of maintaining iceberg table.
+
+To create flink table backend iceberg table in flink SQL by using syntax `CREATE TABLE test (..) WITH ('connector'='iceberg', ...)`, flink iceberg connector provides the following table properties:
Review comment:
OK, let's make this more clear. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] openinx commented on a change in pull request #3085: Docs: Add flink iceberg connector
Posted by GitBox <gi...@apache.org>.
openinx commented on a change in pull request #3085:
URL: https://github.com/apache/iceberg/pull/3085#discussion_r711539940
##########
File path: site/docs/flink-connector.md
##########
@@ -0,0 +1,138 @@
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+Apache Iceberg supports creating flink table directly without creating the explicit flink catalog in flink SQL in [#2666](https://github.com/apache/iceberg/pull/2666). That means we can just create an iceberg table by specifying `'connector'='iceberg'` table option in flink SQL which is similar to usage in the flink official [document](https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/connectors/table/overview/).
Review comment:
Okay, all sounds reasonable to me !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jackye1995 commented on a change in pull request #3085: Docs: Add flink iceberg connector
Posted by GitBox <gi...@apache.org>.
jackye1995 commented on a change in pull request #3085:
URL: https://github.com/apache/iceberg/pull/3085#discussion_r708883313
##########
File path: site/docs/flink-connector.md
##########
@@ -0,0 +1,138 @@
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+Apache Iceberg supports creating flink table directly without creating the explicit flink catalog in flink SQL in [#2666](https://github.com/apache/iceberg/pull/2666). That means we can just create an iceberg table by specifying `'connector'='iceberg'` table option in flink SQL which is similar to usage in the flink official [document](https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/connectors/table/overview/).
+
+In flink, the SQL `CREATE TABLE test (..) WITH ('connector'='iceberg', ...)` will create an flink table in current flink catalog (use [GenericInMemoryCatalog](https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/table/catalogs/#genericinmemorycatalog) by default),
+which is just mapping to the underlying iceberg table instead of maintaining iceberg table directly in current flink catalog.
+
+To create the table in flink SQL by using SQL syntax `CREATE TABLE test (..) WITH ('connector'='iceberg', ...)`, flink iceberg connector provides the following table properties:
+
+* `connector`: Use the constant `iceberg`.
+* `catalog-name`: User-specified catalog name. It's required because the connector don't have any default value.
+* `catalog-type`: Default to use `hive` if don't specify any value. The optional values are:
+ * `hive`: The hive metastore catalog.
+ * `hadoop`: The hadoop catalog.
+ * `custom`: The customized catalog, see [custom catalog](./custom-catalog.md) for more details.
+* `catalog-database`: The iceberg database name in the backend catalog, use the current flink database name by default.
+* `catalog-table`: The iceberg table name in the backend catalog. Default to use the table name in the flink `CREATE TABLE` sentence.
+
+## Table managed in hive catalog.
+
+Before executing the following SQL, please make sure you've configured the flink SQL client correctly according to the quick start [document](./flink.md).
+
+The following SQL will create an flink table in the current flink catalog, which maps to the iceberg table `default_database.iceberg_table` managed in iceberg catalog.
+
+```sql
+CREATE TABLE flink_table (
+ id BIGINT,
+ data STRING
+) WITH (
+ 'connector'='iceberg',
+ 'catalog-name'='hive_prod',
+ 'uri'='thrift://localhost:9083',
+ 'warehouse'='hdfs://nn:8020/path/to/warehouse'
+);
+```
+
+If you want to create a flink table mapping to a different iceberg table managed in hive catalog (such as `hive_db.hive_iceberg_table` in hive), then you can create flink table as following:
+
+```sql
+CREATE TABLE flink_table (
+ id BIGINT,
+ data STRING
+) WITH (
+ 'connector'='iceberg',
+ 'catalog-name'='hive_prod',
+ 'catalog-database'='hive_db',
+ 'catalog-table'='hive_iceberg_table',
+ 'uri'='thrift://localhost:9083',
+ 'warehouse'='hdfs://nn:8020/path/to/warehouse'
+);
+```
+
+!!! Note
+ The underlying catalog database (`hive_db` in the above example) will be created automatically if it does not exist when writing records into the flink table.
Review comment:
> when writing records into the flink table.
so it won't create the database when creating the table, but only after the first INSERT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] openinx commented on pull request #3085: Docs: Add flink iceberg connector
Posted by GitBox <gi...@apache.org>.
openinx commented on pull request #3085:
URL: https://github.com/apache/iceberg/pull/3085#issuecomment-924529203
Get this merged, thanks @stevenzwu & @jackye1995 for reviewing !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on a change in pull request #3085: Docs: Add flink iceberg connector
Posted by GitBox <gi...@apache.org>.
stevenzwu commented on a change in pull request #3085:
URL: https://github.com/apache/iceberg/pull/3085#discussion_r704002960
##########
File path: site/docs/flink-connector.md
##########
@@ -15,21 +15,21 @@
- limitations under the License.
-->
-Apache Iceberg support creating flink table directly without creating explicit flink catalog in flink SQL in [#2666](https://github.com/apache/iceberg/pull/2666). That means we can just create an iceberg table with specifying 'connector'='iceberg' table option in flink SQL which is similar to the flink official [document](https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/connectors/table/overview/).
+Apache Iceberg supports creating flink table directly without creating the explicit flink catalog in flink SQL in [#2666](https://github.com/apache/iceberg/pull/2666). That means we can just create an iceberg table by specifying `'connector'='iceberg'` table option in flink SQL which is similar to usage in the flink official [document](https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/connectors/table/overview/).
In flink, the SQL `CREATE TABLE test (..) WITH ('connector'='iceberg', ...)` will create an flink table in current flink catalog (use [GenericInMemoryCatalog](https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/table/catalogs/#genericinmemorycatalog) by default),
-which is just map to the underlying iceberg table instead of maintaining iceberg table.
+which is just mapping to the underlying iceberg table instead of maintaining iceberg table directly in current flink catalog.
-To create flink table backend iceberg table in flink SQL by using syntax `CREATE TABLE test (..) WITH ('connector'='iceberg', ...)`, flink iceberg connector provides the following table properties:
+To create the table in flink SQL by using SQL syntax `CREATE TABLE test (..) WITH ('connector'='iceberg', ...)`, flink iceberg connector provides the following table properties:
* `connector`: Use the constant `iceberg`.
-* `catalog-name`: User-specified catalog name.
-* `catalog-type`: The optional values are:
- * `hive`: The hive metastore catalog. Use `hive` by default if we don't specify any value for `catalog-type`.
+* `catalog-name`: User-specified catalog name. It's required because the connector don't have any default value.
+* `catalog-type`: Default to use `hive` if don't specify any value. The optional values are:
+ * `hive`: The hive metastore catalog.
* `hadoop`: The hadoop catalog.
- * `custom`: All database and tables are maintained in the customized catalog, see [custom catalog](https://iceberg.apache.org/custom-catalog/) for more details.
+ * `custom`: The customized catalog, see [custom catalog](./custom-catalog.md) for more details.
* `catalog-database`: The iceberg database name in the backend catalog, use the current flink database name by default.
-* `catalog-table`: The iceberg table name in the backend catalog.
+* `catalog-table`: The iceberg table name in the backend catalog. Default to use the `<table-name>` in the flink DDL `CREATE TABLE <table-name> (..) WITH ('connector'='iceberg', ...)`.
Review comment:
Is this property needed at all with `CREATE TABLE <table-name>`? I assume `<table-name>` is required in the SQL syntax.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] openinx commented on a change in pull request #3085: Docs: Add flink iceberg connector
Posted by GitBox <gi...@apache.org>.
openinx commented on a change in pull request #3085:
URL: https://github.com/apache/iceberg/pull/3085#discussion_r703991546
##########
File path: site/docs/flink-connector.md
##########
@@ -0,0 +1,138 @@
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+Apache Iceberg support creating flink table directly without creating explicit flink catalog in flink SQL in [#2666](https://github.com/apache/iceberg/pull/2666). That means we can just create an iceberg table with specifying 'connector'='iceberg' table option in flink SQL which is similar to the flink official [document](https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/connectors/table/overview/).
+
+In flink, the SQL `CREATE TABLE test (..) WITH ('connector'='iceberg', ...)` will create an flink table in current flink catalog (use [GenericInMemoryCatalog](https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/table/catalogs/#genericinmemorycatalog) by default),
+which is just map to the underlying iceberg table instead of maintaining iceberg table.
+
+To create flink table backend iceberg table in flink SQL by using syntax `CREATE TABLE test (..) WITH ('connector'='iceberg', ...)`, flink iceberg connector provides the following table properties:
+
+* `connector`: Use the constant `iceberg`.
+* `catalog-name`: User-specified catalog name.
+* `catalog-type`: The optional values are:
Review comment:
In the following `hive` part, we describe the default behavior:
> `hive`: The hive metastore catalog. Use `hive` by default if we don't specify any value for `catalog-type`.
We can move it ad the head of this part if you think it's necessary to !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on a change in pull request #3085: Docs: Add flink iceberg connector
Posted by GitBox <gi...@apache.org>.
stevenzwu commented on a change in pull request #3085:
URL: https://github.com/apache/iceberg/pull/3085#discussion_r703609260
##########
File path: site/docs/flink-connector.md
##########
@@ -0,0 +1,138 @@
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+Apache Iceberg support creating flink table directly without creating explicit flink catalog in flink SQL in [#2666](https://github.com/apache/iceberg/pull/2666). That means we can just create an iceberg table with specifying 'connector'='iceberg' table option in flink SQL which is similar to the flink official [document](https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/connectors/table/overview/).
+
+In flink, the SQL `CREATE TABLE test (..) WITH ('connector'='iceberg', ...)` will create an flink table in current flink catalog (use [GenericInMemoryCatalog](https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/table/catalogs/#genericinmemorycatalog) by default),
+which is just map to the underlying iceberg table instead of maintaining iceberg table.
+
+To create flink table backend iceberg table in flink SQL by using syntax `CREATE TABLE test (..) WITH ('connector'='iceberg', ...)`, flink iceberg connector provides the following table properties:
+
+* `connector`: Use the constant `iceberg`.
+* `catalog-name`: User-specified catalog name.
+* `catalog-type`: The optional values are:
Review comment:
can we call out the default value?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on a change in pull request #3085: Docs: Add flink iceberg connector
Posted by GitBox <gi...@apache.org>.
stevenzwu commented on a change in pull request #3085:
URL: https://github.com/apache/iceberg/pull/3085#discussion_r703610791
##########
File path: site/mkdocs.yml
##########
@@ -75,7 +75,9 @@ nav:
- Maintenance Procedures: spark-procedures.md
- Structured Streaming: spark-structured-streaming.md
- Time Travel: spark-queries/#time-travel
- - Flink: flink.md
+ - Flink:
+ - Getting Started: flink.md
+ - Iceberg Connector: flink-connector.md
Review comment:
Iceberg Connector -> Flink Connector?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] openinx commented on a change in pull request #3085: Docs: Add flink iceberg connector
Posted by GitBox <gi...@apache.org>.
openinx commented on a change in pull request #3085:
URL: https://github.com/apache/iceberg/pull/3085#discussion_r708194612
##########
File path: site/docs/flink-connector.md
##########
@@ -15,21 +15,21 @@
- limitations under the License.
-->
-Apache Iceberg support creating flink table directly without creating explicit flink catalog in flink SQL in [#2666](https://github.com/apache/iceberg/pull/2666). That means we can just create an iceberg table with specifying 'connector'='iceberg' table option in flink SQL which is similar to the flink official [document](https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/connectors/table/overview/).
+Apache Iceberg supports creating flink table directly without creating the explicit flink catalog in flink SQL in [#2666](https://github.com/apache/iceberg/pull/2666). That means we can just create an iceberg table by specifying `'connector'='iceberg'` table option in flink SQL which is similar to usage in the flink official [document](https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/connectors/table/overview/).
In flink, the SQL `CREATE TABLE test (..) WITH ('connector'='iceberg', ...)` will create an flink table in current flink catalog (use [GenericInMemoryCatalog](https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/table/catalogs/#genericinmemorycatalog) by default),
-which is just map to the underlying iceberg table instead of maintaining iceberg table.
+which is just mapping to the underlying iceberg table instead of maintaining iceberg table directly in current flink catalog.
-To create flink table backend iceberg table in flink SQL by using syntax `CREATE TABLE test (..) WITH ('connector'='iceberg', ...)`, flink iceberg connector provides the following table properties:
+To create the table in flink SQL by using SQL syntax `CREATE TABLE test (..) WITH ('connector'='iceberg', ...)`, flink iceberg connector provides the following table properties:
* `connector`: Use the constant `iceberg`.
-* `catalog-name`: User-specified catalog name.
-* `catalog-type`: The optional values are:
- * `hive`: The hive metastore catalog. Use `hive` by default if we don't specify any value for `catalog-type`.
+* `catalog-name`: User-specified catalog name. It's required because the connector don't have any default value.
+* `catalog-type`: Default to use `hive` if don't specify any value. The optional values are:
+ * `hive`: The hive metastore catalog.
* `hadoop`: The hadoop catalog.
- * `custom`: All database and tables are maintained in the customized catalog, see [custom catalog](https://iceberg.apache.org/custom-catalog/) for more details.
+ * `custom`: The customized catalog, see [custom catalog](./custom-catalog.md) for more details.
* `catalog-database`: The iceberg database name in the backend catalog, use the current flink database name by default.
-* `catalog-table`: The iceberg table name in the backend catalog.
+* `catalog-table`: The iceberg table name in the backend catalog. Default to use the `<table-name>` in the flink DDL `CREATE TABLE <table-name> (..) WITH ('connector'='iceberg', ...)`.
Review comment:
Okay, I've made this more clear in the updated PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jackye1995 commented on a change in pull request #3085: Docs: Add flink iceberg connector
Posted by GitBox <gi...@apache.org>.
jackye1995 commented on a change in pull request #3085:
URL: https://github.com/apache/iceberg/pull/3085#discussion_r708884016
##########
File path: site/docs/flink-connector.md
##########
@@ -0,0 +1,138 @@
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+Apache Iceberg supports creating flink table directly without creating the explicit flink catalog in flink SQL in [#2666](https://github.com/apache/iceberg/pull/2666). That means we can just create an iceberg table by specifying `'connector'='iceberg'` table option in flink SQL which is similar to usage in the flink official [document](https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/connectors/table/overview/).
+
+In flink, the SQL `CREATE TABLE test (..) WITH ('connector'='iceberg', ...)` will create an flink table in current flink catalog (use [GenericInMemoryCatalog](https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/table/catalogs/#genericinmemorycatalog) by default),
+which is just mapping to the underlying iceberg table instead of maintaining iceberg table directly in current flink catalog.
+
+To create the table in flink SQL by using SQL syntax `CREATE TABLE test (..) WITH ('connector'='iceberg', ...)`, flink iceberg connector provides the following table properties:
+
+* `connector`: Use the constant `iceberg`.
+* `catalog-name`: User-specified catalog name. It's required because the connector don't have any default value.
+* `catalog-type`: Default to use `hive` if don't specify any value. The optional values are:
+ * `hive`: The hive metastore catalog.
+ * `hadoop`: The hadoop catalog.
+ * `custom`: The customized catalog, see [custom catalog](./custom-catalog.md) for more details.
+* `catalog-database`: The iceberg database name in the backend catalog, use the current flink database name by default.
+* `catalog-table`: The iceberg table name in the backend catalog. Default to use the table name in the flink `CREATE TABLE` sentence.
+
+## Table managed in hive catalog.
+
+Before executing the following SQL, please make sure you've configured the flink SQL client correctly according to the quick start [document](./flink.md).
+
+The following SQL will create an flink table in the current flink catalog, which maps to the iceberg table `default_database.iceberg_table` managed in iceberg catalog.
+
+```sql
+CREATE TABLE flink_table (
+ id BIGINT,
+ data STRING
+) WITH (
+ 'connector'='iceberg',
+ 'catalog-name'='hive_prod',
+ 'uri'='thrift://localhost:9083',
+ 'warehouse'='hdfs://nn:8020/path/to/warehouse'
+);
+```
+
+If you want to create a flink table mapping to a different iceberg table managed in hive catalog (such as `hive_db.hive_iceberg_table` in hive), then you can create flink table as following:
+
+```sql
+CREATE TABLE flink_table (
+ id BIGINT,
+ data STRING
+) WITH (
+ 'connector'='iceberg',
+ 'catalog-name'='hive_prod',
+ 'catalog-database'='hive_db',
+ 'catalog-table'='hive_iceberg_table',
+ 'uri'='thrift://localhost:9083',
+ 'warehouse'='hdfs://nn:8020/path/to/warehouse'
+);
+```
+
+!!! Note
+ The underlying catalog database (`hive_db` in the above example) will be created automatically if it does not exist when writing records into the flink table.
+
+## Table managed in hadoop catalog
+
+The following SQL will create an flink table in current flink catalog, which maps to the iceberg table `default_database.flink_table` managed in hadoop catalog.
+
+```sql
+CREATE TABLE flink_table (
+ id BIGINT,
+ data STRING
+) WITH (
+ 'connector'='iceberg',
+ 'catalog-name'='hadoop_prod',
+ 'catalog-type'='hadoop',
+ 'warehouse'='hdfs://nn:8020/path/to/warehouse'
+);
+```
+
+## Table managed in custom catalog
+
+The following SQL will create an flink table in current flink catalog, which maps to the iceberg table `default_database.flink_table` managed in custom catalog.
+
+```sql
+CREATE TABLE flink_table (
+ id BIGINT,
+ data STRING
+) WITH (
+ 'connector'='iceberg',
+ 'catalog-name'='custom_prod',
+ 'catalog-type'='custom',
+ 'catalog-impl'='com.my.custom.CatalogImpl',
+ -- More table properties for the customized catalog
+ 'my-additional-catalog-config'='my-value',
+ ...
+);
+```
+
+Please refer to [AWS](./aws.md#catalogs), [Nessie](./nessie.md), [JDBC](./jdbc.md) catalog for more details.
Review comment:
I think we can simply say "check sections under the Integrations tab for all custom catalogs" to avoid listing all the pages.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] openinx commented on pull request #3085: Docs: Add flink iceberg connector
Posted by GitBox <gi...@apache.org>.
openinx commented on pull request #3085:
URL: https://github.com/apache/iceberg/pull/3085#issuecomment-919082630
@rdblue , @jackye1995 , would you like to have a double check ? I think I need at least one apache iceberg committer to approve this PR before merging this, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org