You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/05/27 03:27:01 UTC
[GitHub] [spark] aniket486 opened a new pull request #28649: [SPARK-31829] Check for partition existence before computation
aniket486 opened a new pull request #28649:
URL: https://github.com/apache/spark/pull/28649
for insert overwrite if not exists queries on Hive Serde Table.
Example query:
INSERT OVERWRITE table T partition(p='existing') IF NOT EXISTS select ... ;
where T is a Hive Serde table.
For Datasource table, it does avoid the computation and exits early (due to work done in SPARK-20831).
InsertSuite already covers the test scenario.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28649: [SPARK-31829] Check for hive table partition existence before computation for insert overwrite if not exists query
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28649:
URL: https://github.com/apache/spark/pull/28649#issuecomment-634397248
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] aniket486 commented on pull request #28649: [SPARK-31829] Check for partition existence before computation
Posted by GitBox <gi...@apache.org>.
aniket486 commented on pull request #28649:
URL: https://github.com/apache/spark/pull/28649#issuecomment-665350022
Thanks for the review, @cloud-fan. PTAL.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] c21 commented on a change in pull request #28649: [SPARK-31829] Check for partition existence before computation
Posted by GitBox <gi...@apache.org>.
c21 commented on a change in pull request #28649:
URL: https://github.com/apache/spark/pull/28649#discussion_r462057050
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala
##########
@@ -166,6 +175,17 @@ case class InsertIntoHiveTable(
if (isDynamic.init.zip(isDynamic.tail).contains((true, false))) {
throw new AnalysisException(ErrorMsg.PARTITION_DYN_STA_ORDER.getMsg)
}
+ } else if (numStaticPartitions > 0) {
+ // scalastyle:off
+ // ifNotExists is only valid with static partition, refer to
+ // https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML#LanguageManualDML-InsertingdataintoHiveTablesfromqueries
+ // scalastyle:on
+ if (!oldPart.isEmpty && ifPartitionNotExists) {
+ val partitionStr = partitionSpec.map(_.productIterator.mkString("=")).mkString("/")
+ logWarning(s"Partition '$partitionStr' already exist, skip running job to overwrite " +
Review comment:
+1 for adding a warning message here, I feel previous behavior with silently exiting query might be quite confusing for users and developers.
nit: as we could have multiple `INSERT OVERWRITE` in one query, do we also want to include table name (`table.identifier.table`) in warning message?
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala
##########
@@ -148,6 +148,15 @@ case class InsertIntoHiveTable(
|Table partitions: ${table.partitionColumnNames.mkString(",")}""".stripMargin)
}
+ lazy val oldPart = if ( numDynamicPartitions == 0) {
Review comment:
+1
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala
##########
@@ -166,6 +175,17 @@ case class InsertIntoHiveTable(
if (isDynamic.init.zip(isDynamic.tail).contains((true, false))) {
throw new AnalysisException(ErrorMsg.PARTITION_DYN_STA_ORDER.getMsg)
}
+ } else if (numStaticPartitions > 0) {
+ // scalastyle:off
+ // ifNotExists is only valid with static partition, refer to
+ // https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML#LanguageManualDML-InsertingdataintoHiveTablesfromqueries
+ // scalastyle:on
+ if (!oldPart.isEmpty && ifPartitionNotExists) {
Review comment:
nit: `if (oldPart.isDefined && ifPartitionNotExists)`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] aniket486 commented on pull request #28649: [SPARK-31829] Check for partition existence before computation
Posted by GitBox <gi...@apache.org>.
aniket486 commented on pull request #28649:
URL: https://github.com/apache/spark/pull/28649#issuecomment-659632448
@gatorsmile @viirya @cloud-fan - can you please take a look?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #28649: [SPARK-31829] Check for partition existence before computation
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #28649:
URL: https://github.com/apache/spark/pull/28649#discussion_r459867268
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala
##########
@@ -148,6 +148,15 @@ case class InsertIntoHiveTable(
|Table partitions: ${table.partitionColumnNames.mkString(",")}""".stripMargin)
}
+ val oldPart = if ( numDynamicPartitions == 0) {
Review comment:
or just make it a lazy val, so that we only compute it when needed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #28649: [SPARK-31829] Check for partition existence before computation
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #28649:
URL: https://github.com/apache/spark/pull/28649#discussion_r459866396
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala
##########
@@ -148,6 +148,15 @@ case class InsertIntoHiveTable(
|Table partitions: ${table.partitionColumnNames.mkString(",")}""".stripMargin)
}
+ val oldPart = if ( numDynamicPartitions == 0) {
Review comment:
`numDynamicPartitions == 0 && numStaticPartitions > 0`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #28649: [SPARK-31829] Check for partition existence before computation
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #28649:
URL: https://github.com/apache/spark/pull/28649#discussion_r459867319
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala
##########
@@ -166,6 +175,17 @@ case class InsertIntoHiveTable(
if (isDynamic.init.zip(isDynamic.tail).contains((true, false))) {
throw new AnalysisException(ErrorMsg.PARTITION_DYN_STA_ORDER.getMsg)
}
+ } else if (numStaticPartitions > 0) {
+ // scalastyle:off
+ // ifNotExists is only valid with static partition, refer to
+ // https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML#LanguageManualDML-InsertingdataintoHiveTablesfromqueries
+ // scalastyle:on
+ if (!oldPart.isEmpty && ifPartitionNotExists) {
+ val partitionStr = partitionSpec.map(_.productIterator.mkString("=")).mkString("/")
+ logWarning(s"Partition '$partitionStr' already exist, skip running job to overwrite " +
Review comment:
do we have this warning message before?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #28649: [SPARK-31829] Check for partition existence before computation
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #28649:
URL: https://github.com/apache/spark/pull/28649#discussion_r462036036
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala
##########
@@ -148,6 +148,15 @@ case class InsertIntoHiveTable(
|Table partitions: ${table.partitionColumnNames.mkString(",")}""".stripMargin)
}
+ lazy val oldPart = if ( numDynamicPartitions == 0) {
Review comment:
if it's a lazy val, do we still need the `if`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] aniket486 commented on pull request #28649: [SPARK-31829] Check for partition existence before computation
Posted by GitBox <gi...@apache.org>.
aniket486 commented on pull request #28649:
URL: https://github.com/apache/spark/pull/28649#issuecomment-663347876
Current implementation calls "[saveAsHiveFile](https://github.com/apache/spark/blob/master/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala#L205)" which triggers job to run the "select ... " query only to discard the output [later](https://github.com/apache/spark/blob/master/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala#L274) (by not calling externalCatalog.loadPartition)".
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] aniket486 commented on a change in pull request #28649: [SPARK-31829] Check for partition existence before computation
Posted by GitBox <gi...@apache.org>.
aniket486 commented on a change in pull request #28649:
URL: https://github.com/apache/spark/pull/28649#discussion_r461716620
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala
##########
@@ -166,6 +175,17 @@ case class InsertIntoHiveTable(
if (isDynamic.init.zip(isDynamic.tail).contains((true, false))) {
throw new AnalysisException(ErrorMsg.PARTITION_DYN_STA_ORDER.getMsg)
}
+ } else if (numStaticPartitions > 0) {
+ // scalastyle:off
+ // ifNotExists is only valid with static partition, refer to
+ // https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML#LanguageManualDML-InsertingdataintoHiveTablesfromqueries
+ // scalastyle:on
+ if (!oldPart.isEmpty && ifPartitionNotExists) {
+ val partitionStr = partitionSpec.map(_.productIterator.mkString("=")).mkString("/")
+ logWarning(s"Partition '$partitionStr' already exist, skip running job to overwrite " +
Review comment:
No, we didn't have this warning before. But, it's good to see this explicitly in the log to understand why no job was run.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] commented on pull request #28649: [SPARK-31829] Check for hive table partition existence before computation for insert overwrite if not exists query
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #28649:
URL: https://github.com/apache/spark/pull/28649#issuecomment-774567548
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28649: [SPARK-31829] Check for partition existence before computation
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28649:
URL: https://github.com/apache/spark/pull/28649#issuecomment-634396991
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28649: [SPARK-31829] Check for partition existence before computation
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28649:
URL: https://github.com/apache/spark/pull/28649#issuecomment-634396991
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #28649: [SPARK-31829] Check for partition existence before computation
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #28649:
URL: https://github.com/apache/spark/pull/28649#issuecomment-661821006
can you be more specific about "avoid the computation"?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] closed pull request #28649: [SPARK-31829] Check for hive table partition existence before computation for insert overwrite if not exists query
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #28649:
URL: https://github.com/apache/spark/pull/28649
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org