You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "KnightChess (via GitHub)" <gi...@apache.org> on 2024/07/04 11:58:16 UTC
[PR] [HUDI-7949] insert into hudi table with columns specified [hudi]
KnightChess opened a new pull request, #11568:
URL: https://github.com/apache/hudi/pull/11568
as #11552 describe, hoodie insert not support column specified, support this case:
- user can specified column order when insert into/overwrite
### Change Logs
only support spark version >= 3.2
- if user specified column list, reorder plan with `Project`
### Impact
none
### Risk level (write none, low medium or high below)
none
### Documentation Update
none
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7949] insert into hudi table with columns specified [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 merged PR #11568:
URL: https://github.com/apache/hudi/pull/11568
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7949] insert into hudi table with columns specified [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #11568:
URL: https://github.com/apache/hudi/pull/11568#discussion_r1667210894
##########
hudi-spark-datasource/hudi-spark3.1.x/src/main/scala/org/apache/spark/sql/HoodieSpark31CatalystPlanUtils.scala:
##########
@@ -83,4 +82,13 @@ object HoodieSpark31CatalystPlanUtils extends HoodieSpark3CatalystPlanUtils {
override def unapplyShowIndexes(plan: LogicalPlan): Option[(LogicalPlan, Seq[Attribute])] = None
override def unapplyRefreshIndex(plan: LogicalPlan): Option[(LogicalPlan, String)] = None
+
+ override def unapplyInsertIntoStatement(plan: LogicalPlan): Option[(LogicalPlan, Seq[String], Map[String, Option[String]], LogicalPlan, Boolean, Boolean)] = {
+ plan match {
+ case insert: InsertIntoStatement =>
+ Some((insert.table, Seq.empty, insert.partitionSpec, insert.query, insert.overwrite, insert.ifPartitionNotExists))
Review Comment:
Should we log some msg here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7949] insert into hudi table with columns specified [hudi]
Posted by "KnightChess (via GitHub)" <gi...@apache.org>.
KnightChess commented on PR #11568:
URL: https://github.com/apache/hudi/pull/11568#issuecomment-2223023363
@codope got it o( ̄▽ ̄)o
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7949] insert into hudi table with columns specified [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11568:
URL: https://github.com/apache/hudi/pull/11568#issuecomment-2211756816
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "19a29026785304260e8c534e9e1edc80aa0c1e9b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24714",
"triggerID" : "19a29026785304260e8c534e9e1edc80aa0c1e9b",
"triggerType" : "PUSH"
}, {
"hash" : "a0ee4874c79268370a765067614fbfa8797d0f0b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24746",
"triggerID" : "a0ee4874c79268370a765067614fbfa8797d0f0b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 19a29026785304260e8c534e9e1edc80aa0c1e9b Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24714)
* a0ee4874c79268370a765067614fbfa8797d0f0b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24746)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7949] insert into hudi table with columns specified [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #11568:
URL: https://github.com/apache/hudi/pull/11568#discussion_r1667210673
##########
hudi-spark-datasource/hudi-spark2/src/main/scala/org/apache/spark/sql/HoodieSpark2CatalystPlanUtils.scala:
##########
@@ -61,10 +61,10 @@ object HoodieSpark2CatalystPlanUtils extends HoodieCatalystPlansUtils {
Join(left, right, joinType, None)
}
- override def unapplyInsertIntoStatement(plan: LogicalPlan): Option[(LogicalPlan, Map[String, Option[String]], LogicalPlan, Boolean, Boolean)] = {
+ override def unapplyInsertIntoStatement(plan: LogicalPlan): Option[(LogicalPlan, Seq[String], Map[String, Option[String]], LogicalPlan, Boolean, Boolean)] = {
plan match {
case InsertIntoTable(table, partition, query, overwrite, ifPartitionNotExists) =>
- Some((table, partition, query, overwrite, ifPartitionNotExists))
+ Some((table, Seq.empty, partition, query, overwrite, ifPartitionNotExists))
Review Comment:
Do you think we should log some msg here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7949] insert into hudi table with columns specified [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #11568:
URL: https://github.com/apache/hudi/pull/11568#discussion_r1667210845
##########
hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/HoodieSpark3CatalystPlanUtils.scala:
##########
@@ -56,15 +57,6 @@ trait HoodieSpark3CatalystPlanUtils extends HoodieCatalystPlansUtils {
Join(left, right, joinType, None, JoinHint.NONE)
}
- override def unapplyInsertIntoStatement(plan: LogicalPlan): Option[(LogicalPlan, Map[String, Option[String]], LogicalPlan, Boolean, Boolean)] = {
- plan match {
- case insert: InsertIntoStatement =>
Review Comment:
Why remove this impl?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7949] insert into hudi table with columns specified [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #11568:
URL: https://github.com/apache/hudi/pull/11568#discussion_r1667210457
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/HoodieCatalystPlansUtils.scala:
##########
@@ -112,7 +113,7 @@ trait HoodieCatalystPlansUtils {
* Decomposes [[InsertIntoStatement]] into its arguments allowing to accommodate for API
* changes in Spark 3.3
*/
- def unapplyInsertIntoStatement(plan: LogicalPlan): Option[(LogicalPlan, Map[String, Option[String]], LogicalPlan, Boolean, Boolean)]
+ def unapplyInsertIntoStatement(plan: LogicalPlan): Option[(LogicalPlan, Seq[String], Map[String, Option[String]], LogicalPlan, Boolean, Boolean)]
Review Comment:
Can we add a doc for the new param?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7949] insert into hudi table with columns specified [hudi]
Posted by "KnightChess (via GitHub)" <gi...@apache.org>.
KnightChess commented on PR #11568:
URL: https://github.com/apache/hudi/pull/11568#issuecomment-2222856656
@codope hi, need specified like `insert into aaa (id, day, price, name, hour) values (2, '01', 12.2, 'bbb', '02')` will trigger this logical, but you case not specified columns, and I test rollback this pr also has problem you point, so it's not caused by this submission
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7949] insert into hudi table with columns specified [hudi]
Posted by "leesf (via GitHub)" <gi...@apache.org>.
leesf commented on code in PR #11568:
URL: https://github.com/apache/hudi/pull/11568#discussion_r1667793800
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/TestInsertTable.scala:
##########
@@ -2826,4 +2826,163 @@ class TestInsertTable extends HoodieSparkSqlTestBase {
spark.sessionState.conf.unsetConf("hoodie.datasource.insert.dup.policy")
spark.sessionState.conf.unsetConf("hoodie.datasource.write.operation")
}
+
+ test("Test insert into with special cols") {
+ withTempDir { tmp =>
+ if (HoodieSparkUtils.gteqSpark3_2) {
+ val targetTableA = generateTableName
+ val tablePathA = s"${tmp.getCanonicalPath}/$targetTableA"
+ if (HoodieSparkUtils.isSpark3_4) {
+ spark.sql("set spark.sql.defaultColumn.enabled = false")
+ }
+
+ spark.sql(
+ s"""
+ |create table if not exists $targetTableA (
+ | id bigint,
+ | name string,
+ | price double
+ |) using hudi
+ |tblproperties (
+ | primaryKey = 'id',
+ | type = 'mor',
+ | preCombineField = 'name'
+ |) location '$tablePathA'
+ |""".stripMargin)
+
+ spark.sql(s"insert into $targetTableA (id, price, name) values (1, 12.1, 'aaa')")
+
+ checkAnswer(s"select id, price, name from $targetTableA")(
+ Seq(1, 12.1, "aaa")
+ )
+
+ val targetTableB = generateTableName
+ val tablePathB = s"${tmp.getCanonicalPath}/$targetTableB"
+
+ spark.sql(
+ s"""
+ |create table if not exists $targetTableB (
+ | id bigint,
+ | name string,
+ | price double,
+ | day string,
+ | hour string
+ |) using hudi
+ |tblproperties (
+ | primaryKey = 'id',
+ | type = 'mor',
+ | preCombineField = 'name'
+ |) partitioned by (day, hour)
+ |location '$tablePathB'
+ |""".stripMargin)
+
+ spark.sql(s"insert into $targetTableB (id, day, price, name, hour) " +
+ s"values (2, '01', 12.2, 'bbb', '02')")
+
+ spark.sql(s"insert into $targetTableB (id, day, price, name, hour) " +
+ s"select id, '01' as dt, price, name, '03' as hour from $targetTableA")
+
+ spark.sql(s"insert into $targetTableB partition(day='02', hour) (id, hour, price, name) " +
+ s"values (3, '01', 12.3, 'ccc')")
+
+ spark.sql(s"insert into $targetTableB partition(day='02', hour='02') (id, price, name) " +
+ s"values (4, 12.4, 'ddd')")
+
+ checkAnswer(s"select id, price, name, day, hour from $targetTableB")(
+ Seq(2, 12.2, "bbb", "01", "02"),
+ Seq(1, 12.1, "aaa", "01", "03"),
+ Seq(3, 12.3, "ccc", "02", "01"),
+ Seq(4, 12.4, "ddd", "02", "02")
+ )
+
+ if (HoodieSparkUtils.isSpark3_4) {
+ spark.sql("set spark.sql.defaultColumn.enabled = true")
+ checkExceptionContain(s"insert into $targetTableB (id, day, price, name, hour) " +
+ s"select id, '01' as dt, price, name, '03' as hour from $targetTableA")(
+ "hudi not support specified cols when enable default columns")
+ }
+ }
+ }
+ }
+
+ test("Test insert overwrite with special cols") {
Review Comment:
ditto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7949] insert into hudi table with columns specified [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11568:
URL: https://github.com/apache/hudi/pull/11568#issuecomment-2209094499
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "19a29026785304260e8c534e9e1edc80aa0c1e9b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24714",
"triggerID" : "19a29026785304260e8c534e9e1edc80aa0c1e9b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 19a29026785304260e8c534e9e1edc80aa0c1e9b Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24714)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7949] insert into hudi table with columns specified [hudi]
Posted by "leesf (via GitHub)" <gi...@apache.org>.
leesf commented on PR #11568:
URL: https://github.com/apache/hudi/pull/11568#issuecomment-2212620852
LGTM, with minor comments
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7949] insert into hudi table with columns specified [hudi]
Posted by "KnightChess (via GitHub)" <gi...@apache.org>.
KnightChess commented on PR #11568:
URL: https://github.com/apache/hudi/pull/11568#issuecomment-2223061487
@codope if specified, look like work well.
<img width="1159" alt="image" src="https://github.com/apache/hudi/assets/20125927/9ab0f7b8-76d1-4281-b693-bc1de1e412cc">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7949] insert into hudi table with columns specified [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11568:
URL: https://github.com/apache/hudi/pull/11568#issuecomment-2208847684
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "19a29026785304260e8c534e9e1edc80aa0c1e9b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24714",
"triggerID" : "19a29026785304260e8c534e9e1edc80aa0c1e9b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 19a29026785304260e8c534e9e1edc80aa0c1e9b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24714)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7949] insert into hudi table with columns specified [hudi]
Posted by "leesf (via GitHub)" <gi...@apache.org>.
leesf commented on code in PR #11568:
URL: https://github.com/apache/hudi/pull/11568#discussion_r1667793749
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/TestInsertTable.scala:
##########
@@ -2826,4 +2826,163 @@ class TestInsertTable extends HoodieSparkSqlTestBase {
spark.sessionState.conf.unsetConf("hoodie.datasource.insert.dup.policy")
spark.sessionState.conf.unsetConf("hoodie.datasource.write.operation")
}
+
+ test("Test insert into with special cols") {
Review Comment:
special -> specified
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7949] insert into hudi table with columns specified [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #11568:
URL: https://github.com/apache/hudi/pull/11568#issuecomment-2212278490
@leesf Do you have intreast to reivew this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7949] insert into hudi table with columns specified [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11568:
URL: https://github.com/apache/hudi/pull/11568#issuecomment-2211780369
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "19a29026785304260e8c534e9e1edc80aa0c1e9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24714",
"triggerID" : "19a29026785304260e8c534e9e1edc80aa0c1e9b",
"triggerType" : "PUSH"
}, {
"hash" : "a0ee4874c79268370a765067614fbfa8797d0f0b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24746",
"triggerID" : "a0ee4874c79268370a765067614fbfa8797d0f0b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a0ee4874c79268370a765067614fbfa8797d0f0b Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24746)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7949] insert into hudi table with columns specified [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on PR #11568:
URL: https://github.com/apache/hudi/pull/11568#issuecomment-2222779094
@KnightChess Looks like this PR does not handle partition spec? Can you please check HUDI-7964?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7949] insert into hudi table with columns specified [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on PR #11568:
URL: https://github.com/apache/hudi/pull/11568#issuecomment-2222998105
@KnightChess Yes it is not due to this PR. I just tested by creating a parquet table and it's still the same behavior. So, issue is something unrelated to Hudi. You could try as well:
```
spark-sql> drop table if exists test_table;
Time taken: 0.459 seconds
spark-sql>
> create table test_table (
> ts BIGINT,
> id STRING,
> rider STRING,
> driver STRING,
> fare DOUBLE,
> city STRING,
> state STRING
> )
> USING parquet
> PARTITIONED BY (state, city)
> location 'file:///tmp/hudi_test_table';
spark-sql> INSERT INTO test_table VALUES (1695159649,'trip1','rider-A','driver-K',19.10,'san_francisco','california');
spark-sql> INSERT INTO test_table VALUES (1695091554,'trip2','rider-C','driver-M',27.70,'austin','texas');
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7949] insert into hudi table with columns specified [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #11568:
URL: https://github.com/apache/hudi/pull/11568#discussion_r1667210583
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/analysis/HoodieAnalysis.scala:
##########
@@ -408,12 +408,20 @@ case class ResolveImplementationsEarly() extends Rule[LogicalPlan] {
override def apply(plan: LogicalPlan): LogicalPlan = {
plan match {
// Convert to InsertIntoHoodieTableCommand
- case iis @ MatchInsertIntoStatement(relation @ ResolvesToHudiTable(_), partition, query, overwrite, _) if query.resolved =>
+ case iis @ MatchInsertIntoStatement(relation @ ResolvesToHudiTable(_), userSpecifiedCols, partition, query, overwrite, _) if query.resolved =>
relation match {
// NOTE: In Spark >= 3.2, Hudi relations will be resolved as [[DataSourceV2Relation]]s by default;
// However, currently, fallback will be applied downgrading them to V1 relations, hence
// we need to check whether we could proceed here, or has to wait until fallback rule kicks in
- case lr: LogicalRelation => new InsertIntoHoodieTableCommand(lr, query, partition, overwrite)
+ case lr: LogicalRelation =>
+ // Create a project if this is an INSERT INTO query with specified cols.
+ val projectByUserSpecified = if (userSpecifiedCols.nonEmpty) {
+ assert(lr.catalogTable.isDefined, "Missing catalog table")
Review Comment:
Use `ValidationUtils.checkState` instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7949] insert into hudi table with columns specified [hudi]
Posted by "KnightChess (via GitHub)" <gi...@apache.org>.
KnightChess commented on code in PR #11568:
URL: https://github.com/apache/hudi/pull/11568#discussion_r1667357115
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/analysis/HoodieAnalysis.scala:
##########
@@ -408,12 +408,20 @@ case class ResolveImplementationsEarly() extends Rule[LogicalPlan] {
override def apply(plan: LogicalPlan): LogicalPlan = {
plan match {
// Convert to InsertIntoHoodieTableCommand
- case iis @ MatchInsertIntoStatement(relation @ ResolvesToHudiTable(_), partition, query, overwrite, _) if query.resolved =>
+ case iis @ MatchInsertIntoStatement(relation @ ResolvesToHudiTable(_), userSpecifiedCols, partition, query, overwrite, _) if query.resolved =>
relation match {
// NOTE: In Spark >= 3.2, Hudi relations will be resolved as [[DataSourceV2Relation]]s by default;
// However, currently, fallback will be applied downgrading them to V1 relations, hence
// we need to check whether we could proceed here, or has to wait until fallback rule kicks in
- case lr: LogicalRelation => new InsertIntoHoodieTableCommand(lr, query, partition, overwrite)
+ case lr: LogicalRelation =>
+ // Create a project if this is an INSERT INTO query with specified cols.
+ val projectByUserSpecified = if (userSpecifiedCols.nonEmpty) {
+ assert(lr.catalogTable.isDefined, "Missing catalog table")
Review Comment:
done
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/HoodieCatalystPlansUtils.scala:
##########
@@ -112,7 +113,7 @@ trait HoodieCatalystPlansUtils {
* Decomposes [[InsertIntoStatement]] into its arguments allowing to accommodate for API
* changes in Spark 3.3
*/
- def unapplyInsertIntoStatement(plan: LogicalPlan): Option[(LogicalPlan, Map[String, Option[String]], LogicalPlan, Boolean, Boolean)]
+ def unapplyInsertIntoStatement(plan: LogicalPlan): Option[(LogicalPlan, Seq[String], Map[String, Option[String]], LogicalPlan, Boolean, Boolean)]
Review Comment:
done
##########
hudi-spark-datasource/hudi-spark2/src/main/scala/org/apache/spark/sql/HoodieSpark2CatalystPlanUtils.scala:
##########
@@ -61,10 +61,10 @@ object HoodieSpark2CatalystPlanUtils extends HoodieCatalystPlansUtils {
Join(left, right, joinType, None)
}
- override def unapplyInsertIntoStatement(plan: LogicalPlan): Option[(LogicalPlan, Map[String, Option[String]], LogicalPlan, Boolean, Boolean)] = {
+ override def unapplyInsertIntoStatement(plan: LogicalPlan): Option[(LogicalPlan, Seq[String], Map[String, Option[String]], LogicalPlan, Boolean, Boolean)] = {
plan match {
case InsertIntoTable(table, partition, query, overwrite, ifPartitionNotExists) =>
- Some((table, partition, query, overwrite, ifPartitionNotExists))
+ Some((table, Seq.empty, partition, query, overwrite, ifPartitionNotExists))
Review Comment:
I think it unnecessary, because it is not supported at the sql grammar level
##########
hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/HoodieSpark3CatalystPlanUtils.scala:
##########
@@ -56,15 +57,6 @@ trait HoodieSpark3CatalystPlanUtils extends HoodieCatalystPlansUtils {
Join(left, right, joinType, None, JoinHint.NONE)
}
- override def unapplyInsertIntoStatement(plan: LogicalPlan): Option[(LogicalPlan, Map[String, Option[String]], LogicalPlan, Boolean, Boolean)] = {
- plan match {
- case insert: InsertIntoStatement =>
Review Comment:
every subclass has it own impl, so I remove it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7949] insert into hudi table with columns specified [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11568:
URL: https://github.com/apache/hudi/pull/11568#issuecomment-2211755305
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "19a29026785304260e8c534e9e1edc80aa0c1e9b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24714",
"triggerID" : "19a29026785304260e8c534e9e1edc80aa0c1e9b",
"triggerType" : "PUSH"
}, {
"hash" : "a0ee4874c79268370a765067614fbfa8797d0f0b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a0ee4874c79268370a765067614fbfa8797d0f0b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 19a29026785304260e8c534e9e1edc80aa0c1e9b Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24714)
* a0ee4874c79268370a765067614fbfa8797d0f0b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7949] insert into hudi table with columns specified [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11568:
URL: https://github.com/apache/hudi/pull/11568#issuecomment-2208830193
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "19a29026785304260e8c534e9e1edc80aa0c1e9b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "19a29026785304260e8c534e9e1edc80aa0c1e9b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 19a29026785304260e8c534e9e1edc80aa0c1e9b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org