You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/10/28 03:26:23 UTC
[GitHub] [hudi] dongkelun opened a new pull request #3882: [HUDI-2642] Add support ignoring case in update sql operation
dongkelun opened a new pull request #3882:
URL: https://github.com/apache/hudi/pull/3882
## *Tips*
- *Thank you very much for contributing to Apache Hudi.*
- *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.*
## What is the purpose of the pull request
*Add support ignoring case in update sql operation*
## Brief change log
*(for example:)*
- *Add support ignoring case in update sql operation*
## Verify this pull request
- *Added unit test in TestUpdateTable.*
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] dongkelun commented on pull request #3882: [HUDI-2642] Add support ignoring case in update sql operation
Posted by GitBox <gi...@apache.org>.
dongkelun commented on pull request #3882:
URL: https://github.com/apache/hudi/pull/3882#issuecomment-954367643
@xushiyan @YannByron Can you also help review this PR associated with [Add support ignoring case in merge into](https://github.com/apache/hudi/pull/3700)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3882: [HUDI-2642] Add support ignoring case in update sql operation
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3882:
URL: https://github.com/apache/hudi/pull/3882#issuecomment-953479741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "df86af3f27c7044cc0ea2a9a624ad1294dd3a054",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2921",
"triggerID" : "df86af3f27c7044cc0ea2a9a624ad1294dd3a054",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* df86af3f27c7044cc0ea2a9a624ad1294dd3a054 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2921)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3882: [HUDI-2642] Add support ignoring case in update sql operation
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3882:
URL: https://github.com/apache/hudi/pull/3882#issuecomment-953479741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "df86af3f27c7044cc0ea2a9a624ad1294dd3a054",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2921",
"triggerID" : "df86af3f27c7044cc0ea2a9a624ad1294dd3a054",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* df86af3f27c7044cc0ea2a9a624ad1294dd3a054 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2921)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] YannByron commented on a change in pull request #3882: [HUDI-2642] Add support ignoring case in update sql operation
Posted by GitBox <gi...@apache.org>.
YannByron commented on a change in pull request #3882:
URL: https://github.com/apache/hudi/pull/3882#discussion_r758093580
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/UpdateHoodieTableCommand.scala
##########
@@ -51,7 +51,10 @@ case class UpdateHoodieTableCommand(updateTable: UpdateTable) extends RunnableCo
}.toMap
val updateExpressions = table.output
- .map(attr => name2UpdateValue.getOrElse(attr.name, attr))
+ .map(attr => {
+ val UpdateValueOption = name2UpdateValue.find(f => sparkSession.sessionState.conf.resolver(f._1, attr.name))
Review comment:
Got it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3882: [HUDI-2642] Add support ignoring case in update sql operation
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3882:
URL: https://github.com/apache/hudi/pull/3882#issuecomment-961588723
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "df86af3f27c7044cc0ea2a9a624ad1294dd3a054",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2921",
"triggerID" : "df86af3f27c7044cc0ea2a9a624ad1294dd3a054",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* df86af3f27c7044cc0ea2a9a624ad1294dd3a054 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2921)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] YannByron commented on pull request #3882: [HUDI-2642] Add support ignoring case in update sql operation
Posted by GitBox <gi...@apache.org>.
YannByron commented on pull request #3882:
URL: https://github.com/apache/hudi/pull/3882#issuecomment-980825721
@dongkelun LGTM, just left one minor comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3882: [HUDI-2642] Add support ignoring case in update sql operation
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3882:
URL: https://github.com/apache/hudi/pull/3882#issuecomment-953479741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "df86af3f27c7044cc0ea2a9a624ad1294dd3a054",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2921",
"triggerID" : "df86af3f27c7044cc0ea2a9a624ad1294dd3a054",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* df86af3f27c7044cc0ea2a9a624ad1294dd3a054 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2921)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3882: [HUDI-2642] Add support ignoring case in update sql operation
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3882:
URL: https://github.com/apache/hudi/pull/3882#issuecomment-953479741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "df86af3f27c7044cc0ea2a9a624ad1294dd3a054",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "df86af3f27c7044cc0ea2a9a624ad1294dd3a054",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* df86af3f27c7044cc0ea2a9a624ad1294dd3a054 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3882: [HUDI-2642] Add support ignoring case in update sql operation
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3882:
URL: https://github.com/apache/hudi/pull/3882#issuecomment-953479741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "df86af3f27c7044cc0ea2a9a624ad1294dd3a054",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2921",
"triggerID" : "df86af3f27c7044cc0ea2a9a624ad1294dd3a054",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* df86af3f27c7044cc0ea2a9a624ad1294dd3a054 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2921)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3882: [HUDI-2642] Add support ignoring case in update sql operation
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3882:
URL: https://github.com/apache/hudi/pull/3882#issuecomment-961588723
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "df86af3f27c7044cc0ea2a9a624ad1294dd3a054",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2921",
"triggerID" : "df86af3f27c7044cc0ea2a9a624ad1294dd3a054",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* df86af3f27c7044cc0ea2a9a624ad1294dd3a054 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2921)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan merged pull request #3882: [HUDI-2642] Add support ignoring case in update sql operation
Posted by GitBox <gi...@apache.org>.
xushiyan merged pull request #3882:
URL: https://github.com/apache/hudi/pull/3882
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3882: [HUDI-2642] Add support ignoring case in update sql operation
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3882:
URL: https://github.com/apache/hudi/pull/3882#issuecomment-961588723
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "df86af3f27c7044cc0ea2a9a624ad1294dd3a054",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2921",
"triggerID" : "df86af3f27c7044cc0ea2a9a624ad1294dd3a054",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* df86af3f27c7044cc0ea2a9a624ad1294dd3a054 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2921)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] dongkelun commented on pull request #3882: [HUDI-2642] Add support ignoring case in update sql operation
Posted by GitBox <gi...@apache.org>.
dongkelun commented on pull request #3882:
URL: https://github.com/apache/hudi/pull/3882#issuecomment-976112535
@xushiyan Hi, can you help review this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] dongkelun commented on a change in pull request #3882: [HUDI-2642] Add support ignoring case in update sql operation
Posted by GitBox <gi...@apache.org>.
dongkelun commented on a change in pull request #3882:
URL: https://github.com/apache/hudi/pull/3882#discussion_r757842535
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/UpdateHoodieTableCommand.scala
##########
@@ -51,7 +51,10 @@ case class UpdateHoodieTableCommand(updateTable: UpdateTable) extends RunnableCo
}.toMap
val updateExpressions = table.output
- .map(attr => name2UpdateValue.getOrElse(attr.name, attr))
+ .map(attr => {
+ val UpdateValueOption = name2UpdateValue.find(f => sparkSession.sessionState.conf.resolver(f._1, attr.name))
Review comment:
Because its value will be used later. If use `name2UpdateValue.exists`, it will be obtained again later, resulting in duplicate searches
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3882: [HUDI-2642] Add support ignoring case in update sql operation
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3882:
URL: https://github.com/apache/hudi/pull/3882#issuecomment-953479741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "df86af3f27c7044cc0ea2a9a624ad1294dd3a054",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2921",
"triggerID" : "df86af3f27c7044cc0ea2a9a624ad1294dd3a054",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* df86af3f27c7044cc0ea2a9a624ad1294dd3a054 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2921)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] YannByron commented on a change in pull request #3882: [HUDI-2642] Add support ignoring case in update sql operation
Posted by GitBox <gi...@apache.org>.
YannByron commented on a change in pull request #3882:
URL: https://github.com/apache/hudi/pull/3882#discussion_r757834193
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/UpdateHoodieTableCommand.scala
##########
@@ -51,7 +51,10 @@ case class UpdateHoodieTableCommand(updateTable: UpdateTable) extends RunnableCo
}.toMap
val updateExpressions = table.output
- .map(attr => name2UpdateValue.getOrElse(attr.name, attr))
+ .map(attr => {
+ val UpdateValueOption = name2UpdateValue.find(f => sparkSession.sessionState.conf.resolver(f._1, attr.name))
Review comment:
is `name2UpdateValue.exists` better ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org