You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "zhuangchong (via GitHub)" <gi...@apache.org> on 2024/05/21 04:58:32 UTC
[PR] [PIP-22] Introduce repair metastore procedure [paimon]
zhuangchong opened a new pull request, #3355:
URL: https://github.com/apache/paimon/pull/3355
<!-- Please specify the module before the PR name: [core] ... or [flink] ... -->
### Purpose
<!-- Linking this pull request to the issue -->
Introduce repair metastore procedure
<!-- What is the purpose of the change -->
### Tests
<!-- List UT and IT cases to verify this change -->
HiveCatalogITCaseBase#testRepairTable
### API and Format
<!-- Does this change affect API or storage format -->
### Documentation
<!-- Does this change introduce a new feature -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [PIP-22] Introduce repair metastore procedure [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #3355:
URL: https://github.com/apache/paimon/pull/3355#issuecomment-2129025811
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [PIP-22] Introduce repair metastore procedure [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi closed pull request #3355: [PIP-22] Introduce repair metastore procedure
URL: https://github.com/apache/paimon/pull/3355
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [PIP-22] Introduce repair metastore procedure [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #3355:
URL: https://github.com/apache/paimon/pull/3355#discussion_r1609644441
##########
paimon-hive/paimon-hive-catalog/src/main/java/org/apache/paimon/hive/HiveCatalog.java:
##########
@@ -471,6 +468,131 @@ public boolean caseSensitive() {
return false;
}
+ @Override
+ public void repairCatalog() {
+ List<String> databases = listDatabases(new Path(warehouse));
+ for (String database : databases) {
+ repairDatabase(database);
+ }
+ }
+
+ @Override
+ public void repairDatabase(String databaseName) {
+ repairHmsDatabase(databaseName);
+
+ List<String> tables = listTables(databaseName, true);
+
+ CompletableFuture<Void> allOf =
+ CompletableFuture.allOf(
+ tables.stream()
+ .map(table -> Identifier.create(databaseName, table))
+ .map(
+ identifier ->
+ CompletableFuture.runAsync(
+ () -> {
+ try {
+ repairTable(identifier);
+ } catch (TableNotExistException e) {
+ LOG.error(
+ "Table {} does not exist in the paimon.",
+ identifier.getFullName());
+ // ignore
+ }
+ },
+ COMMON_IO_FORK_JOIN_POOL))
+ .toArray(CompletableFuture[]::new));
+ allOf.join();
+ }
+
+ private void repairHmsDatabase(String databaseName) {
+ checkNotSystemDatabase(databaseName);
+ try {
+ Database database = client.getDatabase(databaseName);
+ Path newPath = newDatabasePath(databaseName);
+ if (!locationHelper.getDatabaseLocation(database).equals(newPath.toUri().toString())) {
Review Comment:
If the database already exists, we will use the location set in it. We cannot change the user's location.
Please add tests to cover this.
##########
paimon-hive/paimon-hive-catalog/src/main/java/org/apache/paimon/hive/HiveCatalog.java:
##########
@@ -471,6 +468,131 @@ public boolean caseSensitive() {
return false;
}
+ @Override
+ public void repairCatalog() {
+ List<String> databases = listDatabases(new Path(warehouse));
+ for (String database : databases) {
+ repairDatabase(database);
+ }
+ }
+
+ @Override
+ public void repairDatabase(String databaseName) {
+ repairHmsDatabase(databaseName);
+
+ List<String> tables = listTables(databaseName, true);
+
+ CompletableFuture<Void> allOf =
+ CompletableFuture.allOf(
+ tables.stream()
+ .map(table -> Identifier.create(databaseName, table))
+ .map(
+ identifier ->
+ CompletableFuture.runAsync(
+ () -> {
+ try {
+ repairTable(identifier);
+ } catch (TableNotExistException e) {
+ LOG.error(
+ "Table {} does not exist in the paimon.",
+ identifier.getFullName());
+ // ignore
+ }
+ },
+ COMMON_IO_FORK_JOIN_POOL))
+ .toArray(CompletableFuture[]::new));
+ allOf.join();
+ }
+
+ private void repairHmsDatabase(String databaseName) {
+ checkNotSystemDatabase(databaseName);
+ try {
+ Database database = client.getDatabase(databaseName);
+ Path newPath = newDatabasePath(databaseName);
+ if (!locationHelper.getDatabaseLocation(database).equals(newPath.toUri().toString())) {
+ locationHelper.specifyDatabaseLocation(newPath, database);
+ client.alterDatabase(databaseName, database);
+ }
+ } catch (NoSuchObjectException e) {
+ try {
+ createDatabase(databaseName, true);
+ } catch (DatabaseAlreadyExistException e1) {
+ // ignore
+ }
+ } catch (TException e) {
+ throw new RuntimeException(
+ "Failed to determine if database "
+ + databaseName
+ + " exists in hive metastore.",
+ e);
+ }
+ }
+
+ @Override
+ public void repairTable(Identifier identifier) throws TableNotExistException {
+ checkNotSystemTable(identifier, "repairTable");
+ validateIdentifierNameCaseInsensitive(identifier);
+
+ FileStoreTable paimonTable = getDataTable(identifier, true);
+ validateFieldNameCaseInsensitive(paimonTable.rowType().getFieldNames());
+
+ TableSchema tableSchema = paimonTable.schema();
+ try {
+ try {
+ Table table =
+ client.getTable(identifier.getDatabaseName(), identifier.getObjectName());
Review Comment:
If the table already exists, we will use the location set in it instead of table in the path.
Please add tests to cover this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [PIP-22] Introduce repair metastore procedure [paimon]
Posted by "zhuangchong (via GitHub)" <gi...@apache.org>.
zhuangchong commented on code in PR #3355:
URL: https://github.com/apache/paimon/pull/3355#discussion_r1611119913
##########
paimon-hive/paimon-hive-catalog/src/main/java/org/apache/paimon/hive/HiveCatalog.java:
##########
@@ -471,6 +468,131 @@ public boolean caseSensitive() {
return false;
}
+ @Override
+ public void repairCatalog() {
+ List<String> databases = listDatabases(new Path(warehouse));
+ for (String database : databases) {
+ repairDatabase(database);
+ }
+ }
+
+ @Override
+ public void repairDatabase(String databaseName) {
+ repairHmsDatabase(databaseName);
+
+ List<String> tables = listTables(databaseName, true);
+
+ CompletableFuture<Void> allOf =
+ CompletableFuture.allOf(
+ tables.stream()
+ .map(table -> Identifier.create(databaseName, table))
+ .map(
+ identifier ->
+ CompletableFuture.runAsync(
+ () -> {
+ try {
+ repairTable(identifier);
+ } catch (TableNotExistException e) {
+ LOG.error(
+ "Table {} does not exist in the paimon.",
+ identifier.getFullName());
+ // ignore
+ }
+ },
+ COMMON_IO_FORK_JOIN_POOL))
+ .toArray(CompletableFuture[]::new));
+ allOf.join();
+ }
+
+ private void repairHmsDatabase(String databaseName) {
+ checkNotSystemDatabase(databaseName);
+ try {
+ Database database = client.getDatabase(databaseName);
+ Path newPath = newDatabasePath(databaseName);
+ if (!locationHelper.getDatabaseLocation(database).equals(newPath.toUri().toString())) {
Review Comment:
done.
##########
paimon-hive/paimon-hive-catalog/src/main/java/org/apache/paimon/hive/HiveCatalog.java:
##########
@@ -471,6 +468,131 @@ public boolean caseSensitive() {
return false;
}
+ @Override
+ public void repairCatalog() {
+ List<String> databases = listDatabases(new Path(warehouse));
+ for (String database : databases) {
+ repairDatabase(database);
+ }
+ }
+
+ @Override
+ public void repairDatabase(String databaseName) {
+ repairHmsDatabase(databaseName);
+
+ List<String> tables = listTables(databaseName, true);
+
+ CompletableFuture<Void> allOf =
+ CompletableFuture.allOf(
+ tables.stream()
+ .map(table -> Identifier.create(databaseName, table))
+ .map(
+ identifier ->
+ CompletableFuture.runAsync(
+ () -> {
+ try {
+ repairTable(identifier);
+ } catch (TableNotExistException e) {
+ LOG.error(
+ "Table {} does not exist in the paimon.",
+ identifier.getFullName());
+ // ignore
+ }
+ },
+ COMMON_IO_FORK_JOIN_POOL))
+ .toArray(CompletableFuture[]::new));
+ allOf.join();
+ }
+
+ private void repairHmsDatabase(String databaseName) {
+ checkNotSystemDatabase(databaseName);
+ try {
+ Database database = client.getDatabase(databaseName);
+ Path newPath = newDatabasePath(databaseName);
+ if (!locationHelper.getDatabaseLocation(database).equals(newPath.toUri().toString())) {
+ locationHelper.specifyDatabaseLocation(newPath, database);
+ client.alterDatabase(databaseName, database);
+ }
+ } catch (NoSuchObjectException e) {
+ try {
+ createDatabase(databaseName, true);
+ } catch (DatabaseAlreadyExistException e1) {
+ // ignore
+ }
+ } catch (TException e) {
+ throw new RuntimeException(
+ "Failed to determine if database "
+ + databaseName
+ + " exists in hive metastore.",
+ e);
+ }
+ }
+
+ @Override
+ public void repairTable(Identifier identifier) throws TableNotExistException {
+ checkNotSystemTable(identifier, "repairTable");
+ validateIdentifierNameCaseInsensitive(identifier);
+
+ FileStoreTable paimonTable = getDataTable(identifier, true);
+ validateFieldNameCaseInsensitive(paimonTable.rowType().getFieldNames());
+
+ TableSchema tableSchema = paimonTable.schema();
+ try {
+ try {
+ Table table =
+ client.getTable(identifier.getDatabaseName(), identifier.getObjectName());
Review Comment:
done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [PIP-22] Introduce repair metastore procedure [paimon]
Posted by "zhuangchong (via GitHub)" <gi...@apache.org>.
zhuangchong commented on PR #3355:
URL: https://github.com/apache/paimon/pull/3355#issuecomment-2122492182
Thanks. @JingsongLi
I have modified the comment, please help with code review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [PIP-22] Introduce repair metastore procedure [paimon]
Posted by "zhuangchong (via GitHub)" <gi...@apache.org>.
zhuangchong commented on code in PR #3355:
URL: https://github.com/apache/paimon/pull/3355#discussion_r1608212406
##########
paimon-core/src/main/java/org/apache/paimon/catalog/Catalog.java:
##########
@@ -258,6 +258,18 @@ default boolean caseSensitive() {
return true;
}
+ default void repairCatalog() throws DatabaseNotExistException {
+ throw new UnsupportedOperationException();
+ }
+
+ default void repairDatabase(String databaseName) throws DatabaseNotExistException {
+ throw new UnsupportedOperationException();
+ }
+
+ default void repairTable(Identifier identifier) throws TableNotExistException {
Review Comment:
Querying the paimon table by identifier in the file system may not exist. `TableNotExistException` is the exception thrown on the getDataTable method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [PIP-22] Introduce repair metastore procedure [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #3355:
URL: https://github.com/apache/paimon/pull/3355#issuecomment-2121756670
Thanks @zhuangchong , left some comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [PIP-22] Introduce repair metastore procedure [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #3355:
URL: https://github.com/apache/paimon/pull/3355#discussion_r1607634437
##########
paimon-core/src/main/java/org/apache/paimon/catalog/Catalog.java:
##########
@@ -258,6 +258,18 @@ default boolean caseSensitive() {
return true;
}
+ default void repairCatalog() throws DatabaseNotExistException {
Review Comment:
Add some comments.
##########
paimon-core/src/main/java/org/apache/paimon/catalog/Catalog.java:
##########
@@ -258,6 +258,18 @@ default boolean caseSensitive() {
return true;
}
+ default void repairCatalog() throws DatabaseNotExistException {
+ throw new UnsupportedOperationException();
+ }
+
+ default void repairDatabase(String databaseName) throws DatabaseNotExistException {
Review Comment:
Add some comments.
##########
paimon-core/src/main/java/org/apache/paimon/catalog/Catalog.java:
##########
@@ -258,6 +258,18 @@ default boolean caseSensitive() {
return true;
}
+ default void repairCatalog() throws DatabaseNotExistException {
Review Comment:
Why throw `DatabaseNotExistException`?
##########
paimon-core/src/main/java/org/apache/paimon/catalog/Catalog.java:
##########
@@ -258,6 +258,18 @@ default boolean caseSensitive() {
return true;
}
+ default void repairCatalog() throws DatabaseNotExistException {
+ throw new UnsupportedOperationException();
+ }
+
+ default void repairDatabase(String databaseName) throws DatabaseNotExistException {
+ throw new UnsupportedOperationException();
+ }
+
+ default void repairTable(Identifier identifier) throws TableNotExistException {
Review Comment:
Add some comments.
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/procedure/RepairProcedure.java:
##########
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.flink.procedure;
+
+import org.apache.paimon.catalog.Catalog;
+import org.apache.paimon.catalog.Identifier;
+import org.apache.paimon.utils.StringUtils;
+
+import org.apache.flink.table.procedure.ProcedureContext;
+
+/**
+ * Repair procedure. Usage:
+ *
+ * <pre><code>
+ * -- repair all databases and tables in catalog
+ * CALL sys.repair()
+ *
+ * -- repair all tables in a specific database
+ * CALL sys.repair('databaseName')
+ *
+ * -- repair a table
+ * CALL sys.repair('databaseName.tableName')
+ * </code></pre>
+ */
+public class RepairProcedure extends ProcedureBase {
+
+ public static final String IDENTIFIER = "repair";
+
+ @Override
+ public String identifier() {
+ return IDENTIFIER;
+ }
+
+ public String[] call(ProcedureContext procedureContext)
+ throws Catalog.TableNotExistException, Catalog.DatabaseNotExistException {
+ return call(procedureContext, "");
Review Comment:
use null instead of "".
##########
paimon-flink/paimon-flink-cdc/src/main/java/org/apache/paimon/flink/sink/cdc/CdcRecordUtils.java:
##########
@@ -90,7 +90,7 @@ public static Optional<GenericRow> toGenericRow(CdcRecord record, List<DataField
int idx = fieldNames.indexOf(key);
if (idx < 0) {
- LOG.info("Field " + key + " not found. Waiting for schema update.");
+ LOG.info("Field {} not found. Waiting for schema update.", key);
Review Comment:
revert here? performance path should just use string concat.
##########
paimon-flink/paimon-flink-cdc/src/main/java/org/apache/paimon/flink/sink/cdc/CdcRecordUtils.java:
##########
@@ -105,11 +105,9 @@ public static Optional<GenericRow> toGenericRow(CdcRecord record, List<DataField
genericRow.setField(idx, TypeUtils.castFromCdcValueString(value, type));
} catch (Exception e) {
LOG.info(
- "Failed to convert value "
- + value
- + " to type "
- + type
- + ". Waiting for schema update.",
+ "Failed to convert value {} to type {}. Waiting for schema update.",
Review Comment:
revert, ditto
##########
paimon-hive/paimon-hive-catalog/src/main/java/org/apache/paimon/hive/HiveCatalog.java:
##########
@@ -471,6 +482,129 @@ public boolean caseSensitive() {
return false;
}
+ @Override
+ public void repairCatalog() throws DatabaseNotExistException {
+ FileSystemCatalog fileSystemCatalog = new FileSystemCatalog(fileIO, new Path(warehouse));
+ List<String> databases = fileSystemCatalog.listDatabases();
+ for (String database : databases) {
+ repairDatabase(database, fileSystemCatalog);
+ }
+ }
+
+ @Override
+ public void repairDatabase(String databaseName) throws DatabaseNotExistException {
+ repairDatabase(databaseName, new FileSystemCatalog(fileIO, new Path(warehouse)));
+ }
+
+ private void repairDatabase(String databaseName, FileSystemCatalog fileSystemCatalog)
+ throws DatabaseNotExistException {
+ if (!databaseExists(databaseName)) {
+ try {
+ createDatabase(databaseName, true);
+ } catch (DatabaseAlreadyExistException e) {
+ // ignore
+ }
+ }
+
+ List<String> tables = fileSystemCatalog.listTables(databaseName);
Review Comment:
We should check location here, if the database already exists, and use custom location. We should just repair the custom location.
Maybe you can just extract some methods from `FileSystemCatalog` to `AbstractCatalog` instead of using `FileSystemCatalog` directly.
##########
paimon-hive/paimon-hive-catalog/src/main/java/org/apache/paimon/hive/HiveCatalog.java:
##########
@@ -471,6 +482,129 @@ public boolean caseSensitive() {
return false;
}
+ @Override
+ public void repairCatalog() throws DatabaseNotExistException {
+ FileSystemCatalog fileSystemCatalog = new FileSystemCatalog(fileIO, new Path(warehouse));
+ List<String> databases = fileSystemCatalog.listDatabases();
+ for (String database : databases) {
+ repairDatabase(database, fileSystemCatalog);
+ }
+ }
+
+ @Override
+ public void repairDatabase(String databaseName) throws DatabaseNotExistException {
+ repairDatabase(databaseName, new FileSystemCatalog(fileIO, new Path(warehouse)));
+ }
+
+ private void repairDatabase(String databaseName, FileSystemCatalog fileSystemCatalog)
+ throws DatabaseNotExistException {
+ if (!databaseExists(databaseName)) {
+ try {
+ createDatabase(databaseName, true);
+ } catch (DatabaseAlreadyExistException e) {
+ // ignore
+ }
+ }
+
+ List<String> tables = fileSystemCatalog.listTables(databaseName);
+
+ CompletableFuture<Void> allOf =
+ CompletableFuture.allOf(
+ tables.stream()
+ .map(table -> Identifier.create(databaseName, table))
+ .map(
+ identifier ->
+ CompletableFuture.runAsync(
+ () -> {
+ try {
+ repairTable(
+ identifier,
+ fileSystemCatalog);
+ } catch (TableNotExistException e) {
+ throw new RuntimeException(e);
+ }
+ },
+ COMMON_IO_FORK_JOIN_POOL))
+ .toArray(CompletableFuture[]::new));
+ allOf.join();
+ }
+
+ @Override
+ public void repairTable(Identifier identifier) throws TableNotExistException {
+ repairTable(identifier, new FileSystemCatalog(fileIO, new Path(warehouse)));
+ }
+
+ private void repairTable(Identifier identifier, FileSystemCatalog fileSystemCatalog)
+ throws TableNotExistException {
+ org.apache.paimon.table.Table paimonTable = fileSystemCatalog.getTable(identifier);
+
+ checkNotSystemTable(identifier, "repairTable");
+ validateIdentifierNameCaseInsensitive(identifier);
+ validateFieldNameCaseInsensitive(paimonTable.rowType().getFieldNames());
+
+ HashMap<String, String> newOptions = new HashMap<>(paimonTable.options());
+ if (!paimonTable.partitionKeys().isEmpty()) {
+ // set metastore.partitioned-table = true
+ newOptions.put(METASTORE_PARTITIONED_TABLE.key(), "true");
Review Comment:
Why? We can just respect the `METASTORE_PARTITIONED_TABLE` option in the table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [PIP-22] Introduce repair metastore procedure [paimon]
Posted by "zhuangchong (via GitHub)" <gi...@apache.org>.
zhuangchong commented on PR #3355:
URL: https://github.com/apache/paimon/pull/3355#issuecomment-2126384301
@JingsongLi
I added a new test `HiveCatalogITCaseBase#testRepairTableWithCustomLocation`,
1. If the database in hive already exists and a custom location is used, use the custom location lookup table in the file system;
2. If the table already exists and a custom location is used, updating the Hive table will not change the location;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org