You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "zhuangchong (via GitHub)" <gi...@apache.org> on 2023/07/27 12:07:20 UTC
[GitHub] [incubator-paimon] zhuangchong opened a new pull request, #1676: [spark] Support spark to modify the default value of the column.
zhuangchong opened a new pull request, #1676:
URL: https://github.com/apache/incubator-paimon/pull/1676
<!-- Please specify the module before the PR name: [core] ... or [flink] ... -->
### Purpose
<!-- Linking this pull request to the issue -->
Linked issue: close #1675
Support spark to modify the default value of the column.
```sql
ALTER TABLE testAddColumn ADD COLUMN d STRING DEFAULT 0 COMMENT 'd field';
ALTER TABLE testAlterColumnDefaultValue ALTER COLUMN c SET DEFAULT 100;
```
<!-- What is the purpose of the change -->
### Tests
<!-- List UT and IT cases to verify this change -->
### API and Format
<!-- Does this change affect API or storage format -->
### Documentation
<!-- Does this change introduce a new feature -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] zhuangchong commented on a diff in pull request #1676: [spark] Support spark to modify the default value of the column.
Posted by "zhuangchong (via GitHub)" <gi...@apache.org>.
zhuangchong commented on code in PR #1676:
URL: https://github.com/apache/incubator-paimon/pull/1676#discussion_r1277251322
##########
docs/content/how-to/altering-tables.md:
##########
@@ -413,6 +413,48 @@ Supported Type Conversions.
{{< generated/column_type_conversion >}}
+## Changing Column Default Value
+
+The following SQL changes default value of column `col_a` to `0`.
+
+{{< tabs "change-column-default-value" >}}
+
+{{< tab "Flink" >}}
+
+```sql
+ALTER TABLE my_table SET (
+ 'fields.col_a.default-value' = '0'
+);
+```
+
+{{< /tab >}}
+
+{{< tab "Spark3" >}}
+
+```sql
+-- alter column
+ALTER TABLE my_table ALTER COLUMN col_a SET DEFAULT 0;
+
+-- or set properties
+ALTER TABLE my_table SET TBLPROPERTIES (
+ 'fields.col_a.default-value' = '0'
+);
+```
+
+{{< /tab >}}
+
+{{< tab "Trino" >}}
+
+```sql
+ALTER TABLE my_table SET PROPERTIES fields.col_a.default-value = '0';
Review Comment:
I found that what I wrote was wrong, I will delete this part.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] JingsongLi commented on a diff in pull request #1676: [spark] Support spark to modify the default value of the column.
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #1676:
URL: https://github.com/apache/incubator-paimon/pull/1676#discussion_r1277220948
##########
paimon-spark/paimon-spark-common/src/main/java/org/apache/paimon/spark/SparkCatalog.java:
##########
@@ -315,48 +320,77 @@ public boolean dropTable(Identifier ident) {
}
}
- private SchemaChange toSchemaChange(TableChange change) {
+ private List<SchemaChange> toSchemaChange(TableChange change) {
+ final List<SchemaChange> schemaChanges = new ArrayList<>();
if (change instanceof TableChange.SetProperty) {
TableChange.SetProperty set = (TableChange.SetProperty) change;
validateAlterProperty(set.property());
- return SchemaChange.setOption(set.property(), set.value());
+ schemaChanges.add(SchemaChange.setOption(set.property(), set.value()));
+ return schemaChanges;
} else if (change instanceof TableChange.RemoveProperty) {
TableChange.RemoveProperty remove = (TableChange.RemoveProperty) change;
validateAlterProperty(remove.property());
- return SchemaChange.removeOption(remove.property());
+ schemaChanges.add(SchemaChange.removeOption(remove.property()));
+ return schemaChanges;
} else if (change instanceof TableChange.AddColumn) {
TableChange.AddColumn add = (TableChange.AddColumn) change;
validateAlterNestedField(add.fieldNames());
SchemaChange.Move move = getMove(add.position(), add.fieldNames());
- return SchemaChange.addColumn(
- add.fieldNames()[0],
- toPaimonType(add.dataType()).copy(add.isNullable()),
- add.comment(),
- move);
+ schemaChanges.add(
+ SchemaChange.addColumn(
+ add.fieldNames()[0],
+ toPaimonType(add.dataType()).copy(add.isNullable()),
+ add.comment(),
+ move));
+ if (Objects.nonNull(add.defaultValue())) {
Review Comment:
`add.defaultValue()` is only working for Spark 3.4.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] zhuangchong closed pull request #1676: [spark] Support spark to modify the default value of the column.
Posted by "zhuangchong (via GitHub)" <gi...@apache.org>.
zhuangchong closed pull request #1676: [spark] Support spark to modify the default value of the column.
URL: https://github.com/apache/incubator-paimon/pull/1676
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] zhuangchong commented on a diff in pull request #1676: [spark] Support spark to modify the default value of the column.
Posted by "zhuangchong (via GitHub)" <gi...@apache.org>.
zhuangchong commented on code in PR #1676:
URL: https://github.com/apache/incubator-paimon/pull/1676#discussion_r1277342875
##########
paimon-spark/paimon-spark-common/src/main/java/org/apache/paimon/spark/SparkCatalog.java:
##########
@@ -315,48 +320,77 @@ public boolean dropTable(Identifier ident) {
}
}
- private SchemaChange toSchemaChange(TableChange change) {
+ private List<SchemaChange> toSchemaChange(TableChange change) {
+ final List<SchemaChange> schemaChanges = new ArrayList<>();
if (change instanceof TableChange.SetProperty) {
TableChange.SetProperty set = (TableChange.SetProperty) change;
validateAlterProperty(set.property());
- return SchemaChange.setOption(set.property(), set.value());
+ schemaChanges.add(SchemaChange.setOption(set.property(), set.value()));
+ return schemaChanges;
} else if (change instanceof TableChange.RemoveProperty) {
TableChange.RemoveProperty remove = (TableChange.RemoveProperty) change;
validateAlterProperty(remove.property());
- return SchemaChange.removeOption(remove.property());
+ schemaChanges.add(SchemaChange.removeOption(remove.property()));
+ return schemaChanges;
} else if (change instanceof TableChange.AddColumn) {
TableChange.AddColumn add = (TableChange.AddColumn) change;
validateAlterNestedField(add.fieldNames());
SchemaChange.Move move = getMove(add.position(), add.fieldNames());
- return SchemaChange.addColumn(
- add.fieldNames()[0],
- toPaimonType(add.dataType()).copy(add.isNullable()),
- add.comment(),
- move);
+ schemaChanges.add(
+ SchemaChange.addColumn(
+ add.fieldNames()[0],
+ toPaimonType(add.dataType()).copy(add.isNullable()),
+ add.comment(),
+ move));
+ if (Objects.nonNull(add.defaultValue())) {
Review Comment:
Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] zhuangchong closed pull request #1676: [spark] Support spark to modify the default value of the column.
Posted by "zhuangchong (via GitHub)" <gi...@apache.org>.
zhuangchong closed pull request #1676: [spark] Support spark to modify the default value of the column.
URL: https://github.com/apache/incubator-paimon/pull/1676
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] zhuangchong commented on a diff in pull request #1676: [spark] Support spark to modify the default value of the column.
Posted by "zhuangchong (via GitHub)" <gi...@apache.org>.
zhuangchong commented on code in PR #1676:
URL: https://github.com/apache/incubator-paimon/pull/1676#discussion_r1277247038
##########
docs/content/how-to/altering-tables.md:
##########
@@ -413,6 +413,48 @@ Supported Type Conversions.
{{< generated/column_type_conversion >}}
+## Changing Column Default Value
+
+The following SQL changes default value of column `col_a` to `0`.
+
+{{< tabs "change-column-default-value" >}}
+
+{{< tab "Flink" >}}
+
+```sql
+ALTER TABLE my_table SET (
+ 'fields.col_a.default-value' = '0'
+);
+```
+
+{{< /tab >}}
+
+{{< tab "Spark3" >}}
+
+```sql
+-- alter column
+ALTER TABLE my_table ALTER COLUMN col_a SET DEFAULT 0;
+
+-- or set properties
+ALTER TABLE my_table SET TBLPROPERTIES (
+ 'fields.col_a.default-value' = '0'
+);
+```
+
+{{< /tab >}}
+
+{{< tab "Trino" >}}
+
+```sql
+ALTER TABLE my_table SET PROPERTIES fields.col_a.default-value = '0';
Review Comment:
I copied this part.
https://github.com/apache/incubator-paimon/blob/cfddb74dd34fc51ff8a76b46bf54a3ed8e597d5d/docs/content/how-to/altering-tables.md?plain=1#L55-L61
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] JingsongLi commented on a diff in pull request #1676: [spark] Support spark to modify the default value of the column.
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #1676:
URL: https://github.com/apache/incubator-paimon/pull/1676#discussion_r1277341680
##########
paimon-spark/paimon-spark-common/src/main/java/org/apache/paimon/spark/SparkCatalog.java:
##########
@@ -315,48 +320,77 @@ public boolean dropTable(Identifier ident) {
}
}
- private SchemaChange toSchemaChange(TableChange change) {
+ private List<SchemaChange> toSchemaChange(TableChange change) {
+ final List<SchemaChange> schemaChanges = new ArrayList<>();
if (change instanceof TableChange.SetProperty) {
TableChange.SetProperty set = (TableChange.SetProperty) change;
validateAlterProperty(set.property());
- return SchemaChange.setOption(set.property(), set.value());
+ schemaChanges.add(SchemaChange.setOption(set.property(), set.value()));
+ return schemaChanges;
} else if (change instanceof TableChange.RemoveProperty) {
TableChange.RemoveProperty remove = (TableChange.RemoveProperty) change;
validateAlterProperty(remove.property());
- return SchemaChange.removeOption(remove.property());
+ schemaChanges.add(SchemaChange.removeOption(remove.property()));
+ return schemaChanges;
} else if (change instanceof TableChange.AddColumn) {
TableChange.AddColumn add = (TableChange.AddColumn) change;
validateAlterNestedField(add.fieldNames());
SchemaChange.Move move = getMove(add.position(), add.fieldNames());
- return SchemaChange.addColumn(
- add.fieldNames()[0],
- toPaimonType(add.dataType()).copy(add.isNullable()),
- add.comment(),
- move);
+ schemaChanges.add(
+ SchemaChange.addColumn(
+ add.fieldNames()[0],
+ toPaimonType(add.dataType()).copy(add.isNullable()),
+ add.comment(),
+ move));
+ if (Objects.nonNull(add.defaultValue())) {
Review Comment:
It is difficult to support both 3.3 and 3.4.
I think we can just close this PR now. Until we really need this feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] zhuangchong closed pull request #1676: [spark] Support spark to modify the default value of the column.
Posted by "zhuangchong (via GitHub)" <gi...@apache.org>.
zhuangchong closed pull request #1676: [spark] Support spark to modify the default value of the column.
URL: https://github.com/apache/incubator-paimon/pull/1676
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] JingsongLi commented on a diff in pull request #1676: [spark] Support spark to modify the default value of the column.
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #1676:
URL: https://github.com/apache/incubator-paimon/pull/1676#discussion_r1277214191
##########
docs/content/how-to/altering-tables.md:
##########
@@ -413,6 +413,48 @@ Supported Type Conversions.
{{< generated/column_type_conversion >}}
+## Changing Column Default Value
+
+The following SQL changes default value of column `col_a` to `0`.
+
+{{< tabs "change-column-default-value" >}}
+
+{{< tab "Flink" >}}
+
+```sql
+ALTER TABLE my_table SET (
+ 'fields.col_a.default-value' = '0'
+);
+```
+
+{{< /tab >}}
+
+{{< tab "Spark3" >}}
+
+```sql
+-- alter column
+ALTER TABLE my_table ALTER COLUMN col_a SET DEFAULT 0;
+
+-- or set properties
+ALTER TABLE my_table SET TBLPROPERTIES (
+ 'fields.col_a.default-value' = '0'
+);
+```
+
+{{< /tab >}}
+
+{{< tab "Trino" >}}
+
+```sql
+ALTER TABLE my_table SET PROPERTIES fields.col_a.default-value = '0';
Review Comment:
Have you tested this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] zhuangchong commented on a diff in pull request #1676: [spark] Support spark to modify the default value of the column.
Posted by "zhuangchong (via GitHub)" <gi...@apache.org>.
zhuangchong commented on code in PR #1676:
URL: https://github.com/apache/incubator-paimon/pull/1676#discussion_r1277260110
##########
paimon-spark/paimon-spark-common/src/main/java/org/apache/paimon/spark/SparkCatalog.java:
##########
@@ -315,48 +320,77 @@ public boolean dropTable(Identifier ident) {
}
}
- private SchemaChange toSchemaChange(TableChange change) {
+ private List<SchemaChange> toSchemaChange(TableChange change) {
+ final List<SchemaChange> schemaChanges = new ArrayList<>();
if (change instanceof TableChange.SetProperty) {
TableChange.SetProperty set = (TableChange.SetProperty) change;
validateAlterProperty(set.property());
- return SchemaChange.setOption(set.property(), set.value());
+ schemaChanges.add(SchemaChange.setOption(set.property(), set.value()));
+ return schemaChanges;
} else if (change instanceof TableChange.RemoveProperty) {
TableChange.RemoveProperty remove = (TableChange.RemoveProperty) change;
validateAlterProperty(remove.property());
- return SchemaChange.removeOption(remove.property());
+ schemaChanges.add(SchemaChange.removeOption(remove.property()));
+ return schemaChanges;
} else if (change instanceof TableChange.AddColumn) {
TableChange.AddColumn add = (TableChange.AddColumn) change;
validateAlterNestedField(add.fieldNames());
SchemaChange.Move move = getMove(add.position(), add.fieldNames());
- return SchemaChange.addColumn(
- add.fieldNames()[0],
- toPaimonType(add.dataType()).copy(add.isNullable()),
- add.comment(),
- move);
+ schemaChanges.add(
+ SchemaChange.addColumn(
+ add.fieldNames()[0],
+ toPaimonType(add.dataType()).copy(add.isNullable()),
+ add.comment(),
+ move));
+ if (Objects.nonNull(add.defaultValue())) {
Review Comment:
`UpdateColumnDefaultValue` also exists in spark 3.4, how should I deal with this situation？
is to close this PR Or is it written under the spark3.4 module?
##########
paimon-spark/paimon-spark-common/src/main/java/org/apache/paimon/spark/SparkCatalog.java:
##########
@@ -315,48 +320,77 @@ public boolean dropTable(Identifier ident) {
}
}
- private SchemaChange toSchemaChange(TableChange change) {
+ private List<SchemaChange> toSchemaChange(TableChange change) {
+ final List<SchemaChange> schemaChanges = new ArrayList<>();
if (change instanceof TableChange.SetProperty) {
TableChange.SetProperty set = (TableChange.SetProperty) change;
validateAlterProperty(set.property());
- return SchemaChange.setOption(set.property(), set.value());
+ schemaChanges.add(SchemaChange.setOption(set.property(), set.value()));
+ return schemaChanges;
} else if (change instanceof TableChange.RemoveProperty) {
TableChange.RemoveProperty remove = (TableChange.RemoveProperty) change;
validateAlterProperty(remove.property());
- return SchemaChange.removeOption(remove.property());
+ schemaChanges.add(SchemaChange.removeOption(remove.property()));
+ return schemaChanges;
} else if (change instanceof TableChange.AddColumn) {
TableChange.AddColumn add = (TableChange.AddColumn) change;
validateAlterNestedField(add.fieldNames());
SchemaChange.Move move = getMove(add.position(), add.fieldNames());
- return SchemaChange.addColumn(
- add.fieldNames()[0],
- toPaimonType(add.dataType()).copy(add.isNullable()),
- add.comment(),
- move);
+ schemaChanges.add(
+ SchemaChange.addColumn(
+ add.fieldNames()[0],
+ toPaimonType(add.dataType()).copy(add.isNullable()),
+ add.comment(),
+ move));
+ if (Objects.nonNull(add.defaultValue())) {
Review Comment:
`UpdateColumnDefaultValue` also exists in spark 3.4, how should I deal with this situation？
is to close this PR Or is it written under the spark3.4 module?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org