You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@amoro.apache.org by "Aireed (via GitHub)" <gi...@apache.org> on 2025/01/02 12:01:03 UTC
[PR] [AMORO-x]fix: create table like using mixed_hive [amoro]
Aireed opened a new pull request, #3387:
URL: https://github.com/apache/amoro/pull/3387
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://amoro.apache.org/how-to-contribute/
2. If the PR is related to an issue in https://github.com/apache/amoro/issues, add '[AMORO-XXXX]' in your PR title, e.g., '[AMORO-XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][AMORO-XXXX] Your PR title ...'.
-->
## Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about its use case.
2. If you fix a bug, you can clarify why it is a bug.
3. Use Fix/Resolve/Close #{ISSUE_NUMBER} to link this PR to its related issue
-->
Enviroment:
format: mixed_hive
engine: spark3.3
catalog: `sparkUnifiedSessionCatalog`
bug:
table created by using `create table like using mixed_hive` is not a mixed_hive table.
```
spark-sql> create table ctl_demo(id int ,name string, dt string) using mixed_hive partitioned by (dt);
Time taken: 1.137 seconds
spark-sql> show create table ctl_demo;
CREATE TABLE spark_catalog.amoro_v7.ctl_demo (
id INT,
name STRING,
dt STRING)
USING mixed_hive
PARTITIONED BY (dt)
TBLPROPERTIES (
'base.hive.location-root' = 'hdfs://easyops-cluster/user/warehouse/amoro_v7.db/ctl_demo/hive',
'base.write.format' = 'parquet',
'delta.write.format' = 'parquet',
'flink.max-continuous-empty-commits' = '2147483647',
'schema.name-mapping.default' = '[ {
"field-id" : 1,
"names" : [ "id" ]
}, {
"field-id" : 2,
"names" : [ "name" ]
}, {
"field-id" : 3,
"names" : [ "dt" ]
} ]',
'self-optimizing.group' = 'amovo_v7',
'table.create-timestamp' = '1735817876108',
'write.metadata.delete-after-commit.enabled' = 'true',
'write.parquet.compression-codec' = 'zstd')
Time taken: 0.893 seconds, Fetched 1 row(s)
spark-sql> create table ctl_demo_like like ctl_demo using mixed_hive;
Time taken: 0.501 seconds
spark-sql> show create table ctl_demo_like;
CREATE TABLE amoro_v7.ctl_demo_like (
id INT,
name STRING,
dt STRING)
USING mixed_hive
PARTITIONED BY (dt)
Time taken: 0.103 seconds, Fetched 1 row(s)
spark-sql>
```
## Brief change log
<!--
Clearly describe the changes made in modules, classes, methods, etc.
-->
- modify provider to `mixed_hive`
- support like sourceTable without databases
## How was this patch tested?
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] Run test locally before making a pull request
## Documentation
- Does this pull request introduce a new feature? (yes / no)
- If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-3387]fix: create table like using mixed_hive [amoro]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3387:
URL: https://github.com/apache/amoro/pull/3387#issuecomment-2767683125
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@amoro.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-3387]fix: create table like using mixed_hive [amoro]
Posted by "baiyangtx (via GitHub)" <gi...@apache.org>.
baiyangtx commented on code in PR #3387:
URL: https://github.com/apache/amoro/pull/3387#discussion_r1923177607
##########
amoro-format-mixed/amoro-mixed-spark/v3.3/amoro-mixed-spark-3.3/src/main/scala/org/apache/amoro/spark/sql/catalyst/analysis/RewriteMixedFormatCommand.scala:
##########
@@ -82,35 +86,48 @@ case class RewriteMixedFormatCommand(sparkSession: SparkSession) extends Rule[Lo
optionsMap += (WriteMode.WRITE_MODE_KEY -> WriteMode.OVERWRITE_DYNAMIC.mode)
val newTableSpec = tableSpec.copy(properties = propertiesMap)
c.copy(tableSpec = newTableSpec, writeOptions = optionsMap)
- case CreateTableLikeCommand(targetTable, sourceTable, _, provider, properties, ifNotExists)
- if isCreateMixedFormatTableLikeCommand(targetTable, provider) =>
- val (sourceCatalog, sourceIdentifier) = buildCatalogAndIdentifier(sparkSession, sourceTable)
- val (targetCatalog, targetIdentifier) = buildCatalogAndIdentifier(sparkSession, targetTable)
- val table = sourceCatalog.loadTable(sourceIdentifier)
- var targetProperties = properties
- targetProperties += ("provider" -> "arctic")
- table match {
- case keyedTable: MixedSparkTable =>
- keyedTable.table() match {
- case table: KeyedTable =>
- targetProperties += ("primary.keys" -> String.join(
- ",",
- table.primaryKeySpec().fieldNames()))
+ case CreateTableLikeCommand(targetTable, sourceTable, _, provider, properties, ifNotExists) =>
+ isCreateMixedFormatTableLikeCommand(targetTable, provider) match {
+ case true =>
+ val (sourceCatalog, sourceIdentifier) =
+ buildCatalogAndIdentifier(sparkSession, sourceTable)
+ val (targetCatalog, targetIdentifier) =
+ buildCatalogAndIdentifier(sparkSession, targetTable)
+ val table = sourceCatalog.loadTable(sourceIdentifier)
+ var targetProperties = properties
+ targetProperties += ("provider" -> "mixed_hive")
+ table match {
+ case keyedTable: MixedSparkTable =>
+ keyedTable.table() match {
+ case table: KeyedTable =>
+ targetProperties += ("primary.keys" -> String.join(
+ ",",
+ table.primaryKeySpec().fieldNames()))
+ case _ =>
+ }
case _ =>
}
+ val tableSpec = TableSpec(
+ properties = targetProperties.toMap,
+ provider = provider,
+ options = Map.empty,
+ location = None,
+ comment = None,
+ serde = None,
+ external = false)
+ val seq: Seq[String] = Seq(
+ targetTable.database.getOrElse(sparkSession.catalog.currentDatabase),
+ targetTable.identifier)
+ val name = ResolvedDBObjectName(targetCatalog, seq)
+ CreateTable(name, table.schema(), table.partitioning(), tableSpec, ifNotExists)
case _ =>
+ provider.isDefined match {
+ case true =>
+ throw new UnsupportedOperationException(
+ s"format ${provider.get} does not support create table like command!!!")
+ case false => plan
Review Comment:
This implementation may impact the logic of other extensions when multiple extensions are used together.
It is recommended to handle only the scenarios you are concerned with in the `case conditions =>`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-3387]fix: create table like using mixed_hive [amoro]
Posted by "Aireed (via GitHub)" <gi...@apache.org>.
Aireed commented on code in PR #3387:
URL: https://github.com/apache/amoro/pull/3387#discussion_r1924738646
##########
amoro-format-mixed/amoro-mixed-spark/v3.3/amoro-mixed-spark-3.3/src/main/scala/org/apache/amoro/spark/sql/catalyst/analysis/RewriteMixedFormatCommand.scala:
##########
@@ -82,35 +86,48 @@ case class RewriteMixedFormatCommand(sparkSession: SparkSession) extends Rule[Lo
optionsMap += (WriteMode.WRITE_MODE_KEY -> WriteMode.OVERWRITE_DYNAMIC.mode)
val newTableSpec = tableSpec.copy(properties = propertiesMap)
c.copy(tableSpec = newTableSpec, writeOptions = optionsMap)
- case CreateTableLikeCommand(targetTable, sourceTable, _, provider, properties, ifNotExists)
- if isCreateMixedFormatTableLikeCommand(targetTable, provider) =>
- val (sourceCatalog, sourceIdentifier) = buildCatalogAndIdentifier(sparkSession, sourceTable)
- val (targetCatalog, targetIdentifier) = buildCatalogAndIdentifier(sparkSession, targetTable)
- val table = sourceCatalog.loadTable(sourceIdentifier)
- var targetProperties = properties
- targetProperties += ("provider" -> "arctic")
- table match {
- case keyedTable: MixedSparkTable =>
- keyedTable.table() match {
- case table: KeyedTable =>
- targetProperties += ("primary.keys" -> String.join(
- ",",
- table.primaryKeySpec().fieldNames()))
+ case CreateTableLikeCommand(targetTable, sourceTable, _, provider, properties, ifNotExists) =>
+ isCreateMixedFormatTableLikeCommand(targetTable, provider) match {
+ case true =>
+ val (sourceCatalog, sourceIdentifier) =
+ buildCatalogAndIdentifier(sparkSession, sourceTable)
+ val (targetCatalog, targetIdentifier) =
+ buildCatalogAndIdentifier(sparkSession, targetTable)
+ val table = sourceCatalog.loadTable(sourceIdentifier)
+ var targetProperties = properties
+ targetProperties += ("provider" -> "mixed_hive")
+ table match {
+ case keyedTable: MixedSparkTable =>
+ keyedTable.table() match {
+ case table: KeyedTable =>
+ targetProperties += ("primary.keys" -> String.join(
+ ",",
+ table.primaryKeySpec().fieldNames()))
+ case _ =>
+ }
case _ =>
}
+ val tableSpec = TableSpec(
+ properties = targetProperties.toMap,
+ provider = provider,
+ options = Map.empty,
+ location = None,
+ comment = None,
+ serde = None,
+ external = false)
+ val seq: Seq[String] = Seq(
+ targetTable.database.getOrElse(sparkSession.catalog.currentDatabase),
+ targetTable.identifier)
+ val name = ResolvedDBObjectName(targetCatalog, seq)
+ CreateTable(name, table.schema(), table.partitioning(), tableSpec, ifNotExists)
case _ =>
+ provider.isDefined match {
+ case true =>
+ throw new UnsupportedOperationException(
+ s"format ${provider.get} does not support create table like command!!!")
+ case false => plan
Review Comment:
thanks for your reviews. I remove the `provider.isDefined` check and return plan directly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-3387]fix: create table like using mixed_hive [amoro]
Posted by "zhoujinsong (via GitHub)" <gi...@apache.org>.
zhoujinsong merged PR #3387:
URL: https://github.com/apache/amoro/pull/3387
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-3387]fix: create table like using mixed_hive [amoro]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #3387:
URL: https://github.com/apache/amoro/pull/3387#issuecomment-2603826179
## [Codecov](https://app.codecov.io/gh/apache/amoro/pull/3387?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 27.80%. Comparing base [(`decd55d`)](https://app.codecov.io/gh/apache/amoro/commit/decd55dec29e253d81e29ee960244d2999996432?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`12b2d86`)](https://app.codecov.io/gh/apache/amoro/commit/12b2d8630196d36a6c768beb19c4ea0cef14f12d?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
> Report is 3 commits behind head on master.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #3387 +/- ##
============================================
+ Coverage 21.64% 27.80% +6.15%
- Complexity 2316 3589 +1273
============================================
Files 427 599 +172
Lines 39751 48597 +8846
Branches 5630 6266 +636
============================================
+ Hits 8605 13510 +4905
- Misses 30419 34149 +3730
- Partials 727 938 +211
```
| [Flag](https://app.codecov.io/gh/apache/amoro/pull/3387/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [core](https://app.codecov.io/gh/apache/amoro/pull/3387/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `27.80% <ø> (?)` | |
| [trino](https://app.codecov.io/gh/apache/amoro/pull/3387/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/amoro/pull/3387?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-3387]fix: create table like using mixed_hive [amoro]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3387:
URL: https://github.com/apache/amoro/pull/3387#issuecomment-2675850885
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@amoro.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-3387]fix: create table like using mixed_hive [amoro]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3387:
URL: https://github.com/apache/amoro/pull/3387#issuecomment-2787905204
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-3387]fix: create table like using mixed_hive [amoro]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #3387: [AMORO-3387]fix: create table like using mixed_hive
URL: https://github.com/apache/amoro/pull/3387
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org