You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/03/14 07:41:37 UTC
[GitHub] [spark] LantaoJin opened a new pull request #27911:
[SPARK-31154][SQL] Expose basic write metrics for InsertIntoDataSourceCommand
LantaoJin opened a new pull request #27911: [SPARK-31154][SQL] Expose basic write metrics for InsertIntoDataSourceCommand
URL: https://github.com/apache/spark/pull/27911
### What changes were proposed in this pull request?
Spark provides interface `InsertableRelation` and the `InsertIntoDataSourceCommand` to delegate the inserting processing to a data source. Unlike `DataWritingCommand`, the metrics in InsertIntoDataSourceCommand is empty and has no chance to update. So we cannot get "number of written files" or "number of output rows" from its metrics.
For example, if a table is a Spark parquet table. We can get the writing metrics by:
```scala
val df = sql("INSERT INTO TABLE test_table SELECT 1, 'a'")
val numFiles = df.queryExecution.sparkPlan.metrics("numFiles").value
```
But if it is a Delta table, we cannot.
### Does this PR introduce any user-facing change?
Add a method in `InsertableRelation`. But still keep the old one.
### How was this patch tested?
Add a unit suite.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27911:
[SPARK-31154][SQL] Expose basic write metrics for InsertIntoDataSourceCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27911: [SPARK-31154][SQL] Expose basic write metrics for InsertIntoDataSourceCommand
URL: https://github.com/apache/spark/pull/27911#issuecomment-599023715
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LantaoJin commented on issue #27911: [SPARK-31154][SQL]
Expose basic write metrics for InsertIntoDataSourceCommand
Posted by GitBox <gi...@apache.org>.
LantaoJin commented on issue #27911: [SPARK-31154][SQL] Expose basic write metrics for InsertIntoDataSourceCommand
URL: https://github.com/apache/spark/pull/27911#issuecomment-599296615
ping @cloud-fan @Dooyoung-Hwang
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27911:
[SPARK-31154][SQL] Expose basic write metrics for InsertIntoDataSourceCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27911: [SPARK-31154][SQL] Expose basic write metrics for InsertIntoDataSourceCommand
URL: https://github.com/apache/spark/pull/27911#issuecomment-599077813
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27911: [SPARK-31154][SQL]
Expose basic write metrics for InsertIntoDataSourceCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27911: [SPARK-31154][SQL] Expose basic write metrics for InsertIntoDataSourceCommand
URL: https://github.com/apache/spark/pull/27911#issuecomment-599023716
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24521/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27911: [SPARK-31154][SQL]
Expose basic write metrics for InsertIntoDataSourceCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27911: [SPARK-31154][SQL] Expose basic write metrics for InsertIntoDataSourceCommand
URL: https://github.com/apache/spark/pull/27911#issuecomment-599042630
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27911:
[SPARK-31154][SQL] Expose basic write metrics for InsertIntoDataSourceCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27911: [SPARK-31154][SQL] Expose basic write metrics for InsertIntoDataSourceCommand
URL: https://github.com/apache/spark/pull/27911#issuecomment-599122491
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27911:
[SPARK-31154][SQL] Expose basic write metrics for InsertIntoDataSourceCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27911: [SPARK-31154][SQL] Expose basic write metrics for InsertIntoDataSourceCommand
URL: https://github.com/apache/spark/pull/27911#issuecomment-599122493
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119799/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27911: [SPARK-31154][SQL]
Expose basic write metrics for InsertIntoDataSourceCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27911: [SPARK-31154][SQL] Expose basic write metrics for InsertIntoDataSourceCommand
URL: https://github.com/apache/spark/pull/27911#issuecomment-599077587
**[Test build #119799 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119799/testReport)** for PR 27911 at commit [`bb6738f`](https://github.com/apache/spark/commit/bb6738fe9fc5f80836e64d6c44cd1508882ab20a).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27911:
[SPARK-31154][SQL] Expose basic write metrics for InsertIntoDataSourceCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27911: [SPARK-31154][SQL] Expose basic write metrics for InsertIntoDataSourceCommand
URL: https://github.com/apache/spark/pull/27911#issuecomment-599077819
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24529/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27911:
[SPARK-31154][SQL] Expose basic write metrics for InsertIntoDataSourceCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27911: [SPARK-31154][SQL] Expose basic write metrics for InsertIntoDataSourceCommand
URL: https://github.com/apache/spark/pull/27911#issuecomment-599042630
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27911: [SPARK-31154][SQL]
Expose basic write metrics for InsertIntoDataSourceCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27911: [SPARK-31154][SQL] Expose basic write metrics for InsertIntoDataSourceCommand
URL: https://github.com/apache/spark/pull/27911#issuecomment-599077819
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24529/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27911: [SPARK-31154][SQL]
Expose basic write metrics for InsertIntoDataSourceCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27911: [SPARK-31154][SQL] Expose basic write metrics for InsertIntoDataSourceCommand
URL: https://github.com/apache/spark/pull/27911#issuecomment-599122493
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119799/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LantaoJin removed a comment on issue #27911:
[SPARK-31154][SQL] Expose basic write metrics for InsertIntoDataSourceCommand
Posted by GitBox <gi...@apache.org>.
LantaoJin removed a comment on issue #27911: [SPARK-31154][SQL] Expose basic write metrics for InsertIntoDataSourceCommand
URL: https://github.com/apache/spark/pull/27911#issuecomment-599025859
How to rerun the checks?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27911:
[SPARK-31154][SQL] Expose basic write metrics for InsertIntoDataSourceCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27911: [SPARK-31154][SQL] Expose basic write metrics for InsertIntoDataSourceCommand
URL: https://github.com/apache/spark/pull/27911#issuecomment-599023716
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24521/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27911: [SPARK-31154][SQL]
Expose basic write metrics for InsertIntoDataSourceCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27911: [SPARK-31154][SQL] Expose basic write metrics for InsertIntoDataSourceCommand
URL: https://github.com/apache/spark/pull/27911#issuecomment-599042632
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119791/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27911:
[SPARK-31154][SQL] Expose basic write metrics for InsertIntoDataSourceCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27911: [SPARK-31154][SQL] Expose basic write metrics for InsertIntoDataSourceCommand
URL: https://github.com/apache/spark/pull/27911#issuecomment-599042632
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119791/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27911: [SPARK-31154][SQL]
Expose basic write metrics for InsertIntoDataSourceCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27911: [SPARK-31154][SQL] Expose basic write metrics for InsertIntoDataSourceCommand
URL: https://github.com/apache/spark/pull/27911#issuecomment-599023658
**[Test build #119791 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119791/testReport)** for PR 27911 at commit [`4a1c41e`](https://github.com/apache/spark/commit/4a1c41e157ef008015e3b679d90af7300734ad7c).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27911: [SPARK-31154][SQL]
Expose basic write metrics for InsertIntoDataSourceCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27911: [SPARK-31154][SQL] Expose basic write metrics for InsertIntoDataSourceCommand
URL: https://github.com/apache/spark/pull/27911#issuecomment-599077813
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27911:
[SPARK-31154][SQL] Expose basic write metrics for InsertIntoDataSourceCommand
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27911: [SPARK-31154][SQL] Expose basic write metrics for InsertIntoDataSourceCommand
URL: https://github.com/apache/spark/pull/27911#issuecomment-599077587
**[Test build #119799 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119799/testReport)** for PR 27911 at commit [`bb6738f`](https://github.com/apache/spark/commit/bb6738fe9fc5f80836e64d6c44cd1508882ab20a).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27911: [SPARK-31154][SQL]
Expose basic write metrics for InsertIntoDataSourceCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27911: [SPARK-31154][SQL] Expose basic write metrics for InsertIntoDataSourceCommand
URL: https://github.com/apache/spark/pull/27911#issuecomment-599122491
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LantaoJin commented on issue #27911: [SPARK-31154][SQL]
Expose basic write metrics for InsertIntoDataSourceCommand
Posted by GitBox <gi...@apache.org>.
LantaoJin commented on issue #27911: [SPARK-31154][SQL] Expose basic write metrics for InsertIntoDataSourceCommand
URL: https://github.com/apache/spark/pull/27911#issuecomment-599025859
How to rerun the checks?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27911:
[SPARK-31154][SQL] Expose basic write metrics for InsertIntoDataSourceCommand
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27911: [SPARK-31154][SQL] Expose basic write metrics for InsertIntoDataSourceCommand
URL: https://github.com/apache/spark/pull/27911#issuecomment-599023658
**[Test build #119791 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119791/testReport)** for PR 27911 at commit [`4a1c41e`](https://github.com/apache/spark/commit/4a1c41e157ef008015e3b679d90af7300734ad7c).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27911: [SPARK-31154][SQL]
Expose basic write metrics for InsertIntoDataSourceCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27911: [SPARK-31154][SQL] Expose basic write metrics for InsertIntoDataSourceCommand
URL: https://github.com/apache/spark/pull/27911#issuecomment-599122256
**[Test build #119799 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119799/testReport)** for PR 27911 at commit [`bb6738f`](https://github.com/apache/spark/commit/bb6738fe9fc5f80836e64d6c44cd1508882ab20a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LantaoJin commented on a change in pull request #27911:
[SPARK-31154][SQL] Expose basic write metrics for InsertIntoDataSourceCommand
Posted by GitBox <gi...@apache.org>.
LantaoJin commented on a change in pull request #27911: [SPARK-31154][SQL] Expose basic write metrics for InsertIntoDataSourceCommand
URL: https://github.com/apache/spark/pull/27911#discussion_r395009741
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala
##########
@@ -292,7 +293,11 @@ trait PrunedFilteredScan {
*/
@Stable
trait InsertableRelation {
- def insert(data: DataFrame, overwrite: Boolean): Unit
+ def insert(data: DataFrame, overwrite: Boolean): Unit = {}
+
+ def insert(data: DataFrame, overwrite: Boolean, metrics: Map[String, SQLMetric]): Unit = {
Review comment:
Does DSv2 have the similar interface? Current Delta Lake is using this command and besides this metrics, we have implemented ensure distribution for InsertIntoDataSource to support bucketing delta table. Does DSv1 accepte no more improvements? It has to move to DSv2?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27911: [SPARK-31154][SQL]
Expose basic write metrics for InsertIntoDataSourceCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27911: [SPARK-31154][SQL] Expose basic write metrics for InsertIntoDataSourceCommand
URL: https://github.com/apache/spark/pull/27911#issuecomment-599042555
**[Test build #119791 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119791/testReport)** for PR 27911 at commit [`4a1c41e`](https://github.com/apache/spark/commit/4a1c41e157ef008015e3b679d90af7300734ad7c).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27911: [SPARK-31154][SQL]
Expose basic write metrics for InsertIntoDataSourceCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27911: [SPARK-31154][SQL] Expose basic write metrics for InsertIntoDataSourceCommand
URL: https://github.com/apache/spark/pull/27911#issuecomment-599023715
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27911:
[SPARK-31154][SQL] Expose basic write metrics for InsertIntoDataSourceCommand
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27911: [SPARK-31154][SQL] Expose basic write metrics for InsertIntoDataSourceCommand
URL: https://github.com/apache/spark/pull/27911#discussion_r394825460
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala
##########
@@ -292,7 +293,11 @@ trait PrunedFilteredScan {
*/
@Stable
trait InsertableRelation {
- def insert(data: DataFrame, overwrite: Boolean): Unit
+ def insert(data: DataFrame, overwrite: Boolean): Unit = {}
+
+ def insert(data: DataFrame, overwrite: Boolean, metrics: Map[String, SQLMetric]): Unit = {
Review comment:
The source isn't necessarily related to files. It's a bit odd to assume the number of files here. Also, seems like this is what DSv2 should cover?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org