You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "zhangjun0x01 (via GitHub)" <gi...@apache.org> on 2023/10/23 09:58:16 UTC
[PR] [flink] add `drop partition` for flink sql [incubator-paimon]
zhangjun0x01 opened a new pull request, #2167:
URL: https://github.com/apache/incubator-paimon/pull/2167
<!-- Please specify the module before the PR name: [core] ... or [flink] ... -->
### Purpose
<!-- Linking this pull request to the issue -->
add `drop partition` for flink sql
<!-- What is the purpose of the change -->
### Tests
<!-- List UT and IT cases to verify this change -->
org.apache.paimon.flink.CatalogTableITCase#testDropPartition
### API and Format
<!-- Does this change affect API or storage format -->
### Documentation
<!-- Does this change introduce a new feature -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] add `drop partition` for flink sql [incubator-paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #2167:
URL: https://github.com/apache/incubator-paimon/pull/2167
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] add `drop partition` for flink sql [incubator-paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #2167:
URL: https://github.com/apache/incubator-paimon/pull/2167#discussion_r1371248448
##########
paimon-core/src/main/java/org/apache/paimon/catalog/AbstractCatalog.java:
##########
@@ -113,6 +118,16 @@ public void createDatabase(String name, boolean ignoreIfExists)
createDatabaseImpl(name);
}
+ @Override
+ public void dropPartition(Identifier identifier, Map<String, String> partitionSpec)
+ throws TableNotExistException, PartitionNotExistException {
+ Table table = getTable(identifier);
+ AbstractFileStoreTable fileStoreTable = (AbstractFileStoreTable) table;
+ FileStoreCommit commit = fileStoreTable.store().newCommit(UUID.randomUUID().toString());
+ commit.dropPartitions(
Review Comment:
It looks that we should delete partitions of metastore in `FileStoreCommit`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] add `drop partition` for flink sql [incubator-paimon]
Posted by "yuzelin (via GitHub)" <gi...@apache.org>.
yuzelin commented on PR #2167:
URL: https://github.com/apache/incubator-paimon/pull/2167#issuecomment-1807544252
Resolve conflict.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] add `drop partition` for flink sql [incubator-paimon]
Posted by "zhangjun0x01 (via GitHub)" <gi...@apache.org>.
zhangjun0x01 commented on PR #2167:
URL: https://github.com/apache/incubator-paimon/pull/2167#issuecomment-1779563517
> I think maybe we should not delete partition for metastore, this can be finished by users. Because we have time travel, the partitions are still visible.
I removed the code follow your suggestion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] add `drop partition` for flink sql [incubator-paimon]
Posted by "zhuangchong (via GitHub)" <gi...@apache.org>.
zhuangchong commented on code in PR #2167:
URL: https://github.com/apache/incubator-paimon/pull/2167#discussion_r1369435379
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/FlinkCatalog.java:
##########
@@ -842,8 +851,23 @@ public final void createPartition(
@Override
public final void dropPartition(
ObjectPath tablePath, CatalogPartitionSpec partitionSpec, boolean ignoreIfNotExists)
- throws CatalogException {
- throw new UnsupportedOperationException();
+ throws PartitionNotExistException, CatalogException {
Review Comment:
I understand that paimon catalog should be used to drop partition. For example, when using paimon hive catalog, hive metastore needs to be updated at the same time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] add `drop partition` for flink sql [incubator-paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #2167:
URL: https://github.com/apache/incubator-paimon/pull/2167#issuecomment-1778668883
I think maybe we should not delete partition for metastore, this can be finished by users.
Because we have time travel, the partitions are still visible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] add `drop partition` for flink sql [incubator-paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #2167:
URL: https://github.com/apache/incubator-paimon/pull/2167#discussion_r1373993084
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/FlinkCatalog.java:
##########
@@ -826,7 +826,12 @@ public final CatalogPartition getPartition(
@Override
public final boolean partitionExists(ObjectPath tablePath, CatalogPartitionSpec partitionSpec)
throws CatalogException {
- return false;
+ try {
+ List<CatalogPartitionSpec> partitionSpecs = getPartitionSpecs(tablePath, partitionSpec);
Review Comment:
Here we need an effective method to check partition. For example, we can provide a `SnapshotReaderImpl.withPartitionFilter` to optimize scanning.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] add `drop partition` for flink sql [incubator-paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #2167:
URL: https://github.com/apache/incubator-paimon/pull/2167#discussion_r1371248448
##########
paimon-core/src/main/java/org/apache/paimon/catalog/AbstractCatalog.java:
##########
@@ -113,6 +118,16 @@ public void createDatabase(String name, boolean ignoreIfExists)
createDatabaseImpl(name);
}
+ @Override
+ public void dropPartition(Identifier identifier, Map<String, String> partitionSpec)
+ throws TableNotExistException, PartitionNotExistException {
+ Table table = getTable(identifier);
+ AbstractFileStoreTable fileStoreTable = (AbstractFileStoreTable) table;
+ FileStoreCommit commit = fileStoreTable.store().newCommit(UUID.randomUUID().toString());
+ commit.dropPartitions(
Review Comment:
It looks that we should delete partitions of metastore in `FileStoreCommit`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] add `drop partition` for flink sql [incubator-paimon]
Posted by "zhangjun0x01 (via GitHub)" <gi...@apache.org>.
zhangjun0x01 commented on code in PR #2167:
URL: https://github.com/apache/incubator-paimon/pull/2167#discussion_r1370439871
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/FlinkCatalog.java:
##########
@@ -842,8 +851,23 @@ public final void createPartition(
@Override
public final void dropPartition(
ObjectPath tablePath, CatalogPartitionSpec partitionSpec, boolean ignoreIfNotExists)
- throws CatalogException {
- throw new UnsupportedOperationException();
+ throws PartitionNotExistException, CatalogException {
+
+ if (!partitionExists(tablePath, partitionSpec)) {
+ throw new PartitionNotExistException(getName(), tablePath, partitionSpec);
+ }
+
+ try {
+ Identifier identifier = toIdentifier(tablePath);
+ Table table = catalog.getTable(identifier);
+ AbstractFileStoreTable fileStoreTable = (AbstractFileStoreTable) table;
+ FileStoreCommit commit = fileStoreTable.store().newCommit(UUID.randomUUID().toString());
+ List<Map<String, String>> partitions =
+ Collections.singletonList(partitionSpec.getPartitionSpec());
+ commit.dropPartitions(partitions, BatchWriteBuilder.COMMIT_IDENTIFIER);
+ } catch (Catalog.TableNotExistException e) {
+ throw new CatalogException(e);
+ }
Review Comment:
yes， you are right, I ignored this parameter `METASTORE_PARTITIONED_TABLE`, I update it
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/FlinkCatalog.java:
##########
@@ -842,8 +851,23 @@ public final void createPartition(
@Override
public final void dropPartition(
ObjectPath tablePath, CatalogPartitionSpec partitionSpec, boolean ignoreIfNotExists)
- throws CatalogException {
- throw new UnsupportedOperationException();
+ throws PartitionNotExistException, CatalogException {
+
+ if (!partitionExists(tablePath, partitionSpec)) {
+ throw new PartitionNotExistException(getName(), tablePath, partitionSpec);
+ }
+
+ try {
+ Identifier identifier = toIdentifier(tablePath);
+ Table table = catalog.getTable(identifier);
+ AbstractFileStoreTable fileStoreTable = (AbstractFileStoreTable) table;
+ FileStoreCommit commit = fileStoreTable.store().newCommit(UUID.randomUUID().toString());
+ List<Map<String, String>> partitions =
+ Collections.singletonList(partitionSpec.getPartitionSpec());
+ commit.dropPartitions(partitions, BatchWriteBuilder.COMMIT_IDENTIFIER);
+ } catch (Catalog.TableNotExistException e) {
+ throw new CatalogException(e);
+ }
Review Comment:
yes， you are right, I ignored this parameter `METASTORE_PARTITIONED_TABLE`, I update it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] add `drop partition` for flink sql [incubator-paimon]
Posted by "zhuangchong (via GitHub)" <gi...@apache.org>.
zhuangchong commented on code in PR #2167:
URL: https://github.com/apache/incubator-paimon/pull/2167#discussion_r1369435610
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/FlinkCatalog.java:
##########
@@ -842,8 +851,23 @@ public final void createPartition(
@Override
public final void dropPartition(
ObjectPath tablePath, CatalogPartitionSpec partitionSpec, boolean ignoreIfNotExists)
- throws CatalogException {
- throw new UnsupportedOperationException();
+ throws PartitionNotExistException, CatalogException {
+
+ if (!partitionExists(tablePath, partitionSpec)) {
+ throw new PartitionNotExistException(getName(), tablePath, partitionSpec);
+ }
+
+ try {
+ Identifier identifier = toIdentifier(tablePath);
+ Table table = catalog.getTable(identifier);
+ AbstractFileStoreTable fileStoreTable = (AbstractFileStoreTable) table;
+ FileStoreCommit commit = fileStoreTable.store().newCommit(UUID.randomUUID().toString());
+ List<Map<String, String>> partitions =
+ Collections.singletonList(partitionSpec.getPartitionSpec());
+ commit.dropPartitions(partitions, BatchWriteBuilder.COMMIT_IDENTIFIER);
+ } catch (Catalog.TableNotExistException e) {
+ throw new CatalogException(e);
+ }
Review Comment:
I understand that paimon catalog should be used to drop partition. For example, when using paimon hive catalog, hive metastore needs to be updated at the same time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] add `drop partition` for flink sql [incubator-paimon]
Posted by "zhuangchong (via GitHub)" <gi...@apache.org>.
zhuangchong commented on code in PR #2167:
URL: https://github.com/apache/incubator-paimon/pull/2167#discussion_r1369435379
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/FlinkCatalog.java:
##########
@@ -842,8 +851,23 @@ public final void createPartition(
@Override
public final void dropPartition(
ObjectPath tablePath, CatalogPartitionSpec partitionSpec, boolean ignoreIfNotExists)
- throws CatalogException {
- throw new UnsupportedOperationException();
+ throws PartitionNotExistException, CatalogException {
Review Comment:
I understand that paimon catalog should be used to drop partition. For example, when using paimon hive catalog, hive metastore needs to be updated at the same time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] add `drop partition` for flink sql [incubator-paimon]
Posted by "zhangjun0x01 (via GitHub)" <gi...@apache.org>.
zhangjun0x01 commented on code in PR #2167:
URL: https://github.com/apache/incubator-paimon/pull/2167#discussion_r1369495294
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/FlinkCatalog.java:
##########
@@ -842,8 +851,23 @@ public final void createPartition(
@Override
public final void dropPartition(
ObjectPath tablePath, CatalogPartitionSpec partitionSpec, boolean ignoreIfNotExists)
- throws CatalogException {
- throw new UnsupportedOperationException();
+ throws PartitionNotExistException, CatalogException {
+
+ if (!partitionExists(tablePath, partitionSpec)) {
+ throw new PartitionNotExistException(getName(), tablePath, partitionSpec);
+ }
+
+ try {
+ Identifier identifier = toIdentifier(tablePath);
+ Table table = catalog.getTable(identifier);
+ AbstractFileStoreTable fileStoreTable = (AbstractFileStoreTable) table;
+ FileStoreCommit commit = fileStoreTable.store().newCommit(UUID.randomUUID().toString());
+ List<Map<String, String>> partitions =
+ Collections.singletonList(partitionSpec.getPartitionSpec());
+ commit.dropPartitions(partitions, BatchWriteBuilder.COMMIT_IDENTIFIER);
+ } catch (Catalog.TableNotExistException e) {
+ throw new CatalogException(e);
+ }
Review Comment:
when we create a paimon partitioned table with hive catalog , [we do not save the partitions info in the hive catalog](https://github.com/apache/incubator-paimon/blob/fa68a1e91ee77aeef0c0b99160cc093e56d2405a/paimon-hive/paimon-hive-catalog/src/main/java/org/apache/paimon/hive/HiveCatalog.java#L455) , so when we drop table partition, we only need to modify the paimon metadata.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] add `drop partition` for flink sql [incubator-paimon]
Posted by "zhuangchong (via GitHub)" <gi...@apache.org>.
zhuangchong commented on code in PR #2167:
URL: https://github.com/apache/incubator-paimon/pull/2167#discussion_r1369631946
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/FlinkCatalog.java:
##########
@@ -842,8 +851,23 @@ public final void createPartition(
@Override
public final void dropPartition(
ObjectPath tablePath, CatalogPartitionSpec partitionSpec, boolean ignoreIfNotExists)
- throws CatalogException {
- throw new UnsupportedOperationException();
+ throws PartitionNotExistException, CatalogException {
+
+ if (!partitionExists(tablePath, partitionSpec)) {
+ throw new PartitionNotExistException(getName(), tablePath, partitionSpec);
+ }
+
+ try {
+ Identifier identifier = toIdentifier(tablePath);
+ Table table = catalog.getTable(identifier);
+ AbstractFileStoreTable fileStoreTable = (AbstractFileStoreTable) table;
+ FileStoreCommit commit = fileStoreTable.store().newCommit(UUID.randomUUID().toString());
+ List<Map<String, String>> partitions =
+ Collections.singletonList(partitionSpec.getPartitionSpec());
+ commit.dropPartitions(partitions, BatchWriteBuilder.COMMIT_IDENTIFIER);
+ } catch (Catalog.TableNotExistException e) {
+ throw new CatalogException(e);
+ }
Review Comment:
It's handled here.
https://github.com/apache/incubator-paimon/blob/fa68a1e91ee77aeef0c0b99160cc093e56d2405a/paimon-hive/paimon-hive-catalog/src/main/java/org/apache/paimon/hive/HiveCatalog.java#L479-L496
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] add `drop partition` for flink sql [incubator-paimon]
Posted by "zhangjun0x01 (via GitHub)" <gi...@apache.org>.
zhangjun0x01 commented on code in PR #2167:
URL: https://github.com/apache/incubator-paimon/pull/2167#discussion_r1375261387
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/FlinkCatalog.java:
##########
@@ -826,7 +826,12 @@ public final CatalogPartition getPartition(
@Override
public final boolean partitionExists(ObjectPath tablePath, CatalogPartitionSpec partitionSpec)
throws CatalogException {
- return false;
+ try {
+ List<CatalogPartitionSpec> partitionSpecs = getPartitionSpecs(tablePath, partitionSpec);
Review Comment:
I updated it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org