You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2022/11/08 03:06:26 UTC
[GitHub] [flink-table-store] wxplovecc opened a new pull request, #357: [FLINK-29922] Support create external table for hive catalog
wxplovecc opened a new pull request, #357:
URL: https://github.com/apache/flink-table-store/pull/357
support hive catalog to create external table
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-table-store] JingsongLi commented on a diff in pull request #357: [FLINK-29922] Support create external table for hive catalog
Posted by GitBox <gi...@apache.org>.
JingsongLi commented on code in PR #357:
URL: https://github.com/apache/flink-table-store/pull/357#discussion_r1020246545
##########
flink-table-store-core/src/main/java/org/apache/flink/table/store/table/TableType.java:
##########
@@ -0,0 +1,25 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.store.table;
+
+/** Enum of catalog table type. */
+public enum TableType {
+ MANAGED_TABLE,
Review Comment:
`MANAGED` and `EXTERNAL` is OK.
Can you let it implements `DescribedEnum` just like `ChangelogProducer`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-table-store] SteNicholas commented on a diff in pull request #357: [FLINK-29922] Support create external table for hive catalog
Posted by GitBox <gi...@apache.org>.
SteNicholas commented on code in PR #357:
URL: https://github.com/apache/flink-table-store/pull/357#discussion_r1020706975
##########
flink-table-store-hive/flink-table-store-hive-catalog/src/main/java/org/apache/flink/table/store/hive/HiveCatalog.java:
##########
@@ -226,6 +227,13 @@ public void createTable(ObjectPath tablePath, UpdateSchema updateSchema, boolean
e);
}
Table table = newHmsTable(tablePath);
+
+ if (hiveConf.getEnum(TABLE_TYPE.key(), TableType.MANAGED_TABLE)
Review Comment:
It's better to move this logic to `newHmsTable` which initialzes the `Table` and sets the parameters of the `Table`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-table-store] JingsongLi merged pull request #357: [FLINK-29922] Support create external table for hive catalog
Posted by GitBox <gi...@apache.org>.
JingsongLi merged PR #357:
URL: https://github.com/apache/flink-table-store/pull/357
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-table-store] SteNicholas commented on a diff in pull request #357: [FLINK-29922] Support create external table for hive catalog
Posted by GitBox <gi...@apache.org>.
SteNicholas commented on code in PR #357:
URL: https://github.com/apache/flink-table-store/pull/357#discussion_r1019779213
##########
flink-table-store-core/src/main/java/org/apache/flink/table/store/CatalogOptions.java:
##########
@@ -49,6 +50,12 @@ private CatalogOptions() {}
.noDefaultValue()
.withDescription("Uri of metastore server.");
+ public static final ConfigOption<TableType> TABLE_TYPE =
+ ConfigOptions.key("table.type")
+ .enumType(TableType.class)
+ .defaultValue(TableType.MANAGED_TABLE)
+ .withDescription("Table type of hms table, MANAGED_TABLE/EXTERNAL_TABLE.");
Review Comment:
```suggestion
.withDescription("Type of table.");
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-table-store] SteNicholas commented on a diff in pull request #357: [FLINK-29922] Support create external table for hive catalog
Posted by GitBox <gi...@apache.org>.
SteNicholas commented on code in PR #357:
URL: https://github.com/apache/flink-table-store/pull/357#discussion_r1016658666
##########
flink-table-store-core/src/main/java/org/apache/flink/table/store/CatalogOptions.java:
##########
@@ -49,6 +49,12 @@ private CatalogOptions() {}
.noDefaultValue()
.withDescription("Uri of metastore server.");
+ public static final ConfigOption<Boolean> HMS_EXTERNAL =
+ ConfigOptions.key("hms.external")
Review Comment:
@wxplovecc，could this introduce the `TableType` enum and `table.type` config option which default value is `MANAGED_TABLE`? In the `CatalogOptions`, the `hms.external` is specific for core module.
cc @JingsongLi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-table-store] wxplovecc commented on a diff in pull request #357: [FLINK-29922] Support create external table for hive catalog
Posted by GitBox <gi...@apache.org>.
wxplovecc commented on code in PR #357:
URL: https://github.com/apache/flink-table-store/pull/357#discussion_r1021091587
##########
flink-table-store-hive/flink-table-store-hive-catalog/src/main/java/org/apache/flink/table/store/hive/HiveCatalog.java:
##########
@@ -226,6 +227,13 @@ public void createTable(ObjectPath tablePath, UpdateSchema updateSchema, boolean
e);
}
Table table = newHmsTable(tablePath);
+
+ if (hiveConf.getEnum(TABLE_TYPE.key(), TableType.MANAGED_TABLE)
Review Comment:
done
##########
flink-table-store-core/src/main/java/org/apache/flink/table/store/table/TableType.java:
##########
@@ -0,0 +1,25 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.store.table;
+
+/** Enum of catalog table type. */
+public enum TableType {
+ MANAGED_TABLE,
Review Comment:
updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-table-store] JingsongLi commented on a diff in pull request #357: [FLINK-29922] Support create external table for hive catalog
Posted by GitBox <gi...@apache.org>.
JingsongLi commented on code in PR #357:
URL: https://github.com/apache/flink-table-store/pull/357#discussion_r1020246545
##########
flink-table-store-core/src/main/java/org/apache/flink/table/store/table/TableType.java:
##########
@@ -0,0 +1,25 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.store.table;
+
+/** Enum of catalog table type. */
+public enum TableType {
+ MANAGED_TABLE,
Review Comment:
`MANAGED` is OK.
Can you let it implements `DescribedEnum` just like `ChangelogProducer`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-table-store] SteNicholas commented on a diff in pull request #357: [FLINK-29922] Support create external table for hive catalog
Posted by GitBox <gi...@apache.org>.
SteNicholas commented on code in PR #357:
URL: https://github.com/apache/flink-table-store/pull/357#discussion_r1016658666
##########
flink-table-store-core/src/main/java/org/apache/flink/table/store/CatalogOptions.java:
##########
@@ -49,6 +49,12 @@ private CatalogOptions() {}
.noDefaultValue()
.withDescription("Uri of metastore server.");
+ public static final ConfigOption<Boolean> HMS_EXTERNAL =
+ ConfigOptions.key("hms.external")
Review Comment:
Could this introduce the `TableType` enum and `table.type` config option which default value is `MANAGED_TABLE`? In the `CatalogOptions`, the `hms.external` is specific for core module.
cc @JingsongLi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-table-store] wxplovecc commented on a diff in pull request #357: [FLINK-29922] Support create external table for hive catalog
Posted by GitBox <gi...@apache.org>.
wxplovecc commented on code in PR #357:
URL: https://github.com/apache/flink-table-store/pull/357#discussion_r1018875597
##########
flink-table-store-core/src/main/java/org/apache/flink/table/store/CatalogOptions.java:
##########
@@ -49,6 +49,12 @@ private CatalogOptions() {}
.noDefaultValue()
.withDescription("Uri of metastore server.");
+ public static final ConfigOption<Boolean> HMS_EXTERNAL =
+ ConfigOptions.key("hms.external")
Review Comment:
@SteNicholas Thanks for review, updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-table-store] SteNicholas commented on a diff in pull request #357: [FLINK-29922] Support create external table for hive catalog
Posted by GitBox <gi...@apache.org>.
SteNicholas commented on code in PR #357:
URL: https://github.com/apache/flink-table-store/pull/357#discussion_r1020706445
##########
flink-table-store-hive/flink-table-store-hive-catalog/src/main/java/org/apache/flink/table/store/hive/HiveCatalog.java:
##########
@@ -226,6 +227,13 @@ public void createTable(ObjectPath tablePath, UpdateSchema updateSchema, boolean
e);
}
Table table = newHmsTable(tablePath);
+
+ if (hiveConf.getEnum(TABLE_TYPE.key(), TableType.MANAGED_TABLE)
+ .equals(TableType.EXTERNAL_TABLE)) {
+ table.setTableType(TableType.EXTERNAL_TABLE.toString());
Review Comment:
```suggestion
table.setTableType(TableType.EXTERNAL_TABLE.name());
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-table-store] SteNicholas commented on a diff in pull request #357: [FLINK-29922] Support create external table for hive catalog
Posted by GitBox <gi...@apache.org>.
SteNicholas commented on code in PR #357:
URL: https://github.com/apache/flink-table-store/pull/357#discussion_r1016658666
##########
flink-table-store-core/src/main/java/org/apache/flink/table/store/CatalogOptions.java:
##########
@@ -49,6 +49,12 @@ private CatalogOptions() {}
.noDefaultValue()
.withDescription("Uri of metastore server.");
+ public static final ConfigOption<Boolean> HMS_EXTERNAL =
+ ConfigOptions.key("hms.external")
Review Comment:
Could this introduce the `TableType` enum and `table.type` config option which default value is `MANAGED_TABLE`? cc @JingsongLi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-table-store] SteNicholas commented on a diff in pull request #357: [FLINK-29922] Support create external table for hive catalog
Posted by GitBox <gi...@apache.org>.
SteNicholas commented on code in PR #357:
URL: https://github.com/apache/flink-table-store/pull/357#discussion_r1021115008
##########
flink-table-store-core/src/main/java/org/apache/flink/table/store/table/TableType.java:
##########
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.store.table;
+
+import org.apache.flink.configuration.DescribedEnum;
+import org.apache.flink.configuration.description.InlineElement;
+
+import static org.apache.flink.configuration.description.TextElement.text;
+
+/** Enum of catalog table type. */
+public enum TableType implements DescribedEnum {
+ MANAGED("MANAGED_TABLE", "Hive manage the lifecycle of the table."),
Review Comment:
```suggestion
MANAGED("MANAGED_TABLE", "Table Store owns the table where the entire lifecycle of the table data is managed."),
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-table-store] SteNicholas commented on a diff in pull request #357: [FLINK-29922] Support create external table for hive catalog
Posted by GitBox <gi...@apache.org>.
SteNicholas commented on code in PR #357:
URL: https://github.com/apache/flink-table-store/pull/357#discussion_r1021115008
##########
flink-table-store-core/src/main/java/org/apache/flink/table/store/table/TableType.java:
##########
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.store.table;
+
+import org.apache.flink.configuration.DescribedEnum;
+import org.apache.flink.configuration.description.InlineElement;
+
+import static org.apache.flink.configuration.description.TextElement.text;
+
+/** Enum of catalog table type. */
+public enum TableType implements DescribedEnum {
+ MANAGED("MANAGED_TABLE", "Hive manage the lifecycle of the table."),
Review Comment:
```suggestion
MANAGED("MANAGED_TABLE", "Table Store owned table where the entire lifecycle of the table data is managed."),
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-table-store] SteNicholas commented on a diff in pull request #357: [FLINK-29922] Support create external table for hive catalog
Posted by GitBox <gi...@apache.org>.
SteNicholas commented on code in PR #357:
URL: https://github.com/apache/flink-table-store/pull/357#discussion_r1021116470
##########
flink-table-store-core/src/main/java/org/apache/flink/table/store/table/TableType.java:
##########
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.store.table;
+
+import org.apache.flink.configuration.DescribedEnum;
+import org.apache.flink.configuration.description.InlineElement;
+
+import static org.apache.flink.configuration.description.TextElement.text;
+
+/** Enum of catalog table type. */
+public enum TableType implements DescribedEnum {
+ MANAGED("MANAGED_TABLE", "Hive manage the lifecycle of the table."),
+ EXTERNAL("EXTERNAL_TABLE", "Files are already present or in remote locations.");
Review Comment:
```suggestion
EXTERNAL("EXTERNAL_TABLE", "The table where Table Store has loose coupling with the data stored in external locations.");
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-table-store] wxplovecc commented on a diff in pull request #357: [FLINK-29922] Support create external table for hive catalog
Posted by GitBox <gi...@apache.org>.
wxplovecc commented on code in PR #357:
URL: https://github.com/apache/flink-table-store/pull/357#discussion_r1021144454
##########
flink-table-store-core/src/main/java/org/apache/flink/table/store/table/TableType.java:
##########
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.store.table;
+
+import org.apache.flink.configuration.DescribedEnum;
+import org.apache.flink.configuration.description.InlineElement;
+
+import static org.apache.flink.configuration.description.TextElement.text;
+
+/** Enum of catalog table type. */
+public enum TableType implements DescribedEnum {
+ MANAGED("MANAGED_TABLE", "Hive manage the lifecycle of the table."),
+ EXTERNAL("EXTERNAL_TABLE", "Files are already present or in remote locations.");
Review Comment:
updated @SteNicholas
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-table-store] SteNicholas commented on a diff in pull request #357: [FLINK-29922] Support create external table for hive catalog
Posted by GitBox <gi...@apache.org>.
SteNicholas commented on code in PR #357:
URL: https://github.com/apache/flink-table-store/pull/357#discussion_r1019779486
##########
flink-table-store-core/src/main/java/org/apache/flink/table/store/table/TableType.java:
##########
@@ -0,0 +1,25 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.store.table;
+
+/** TableType enum of catalog table. */
Review Comment:
```suggestion
/** Enum of catalog table type. */
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org