You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/08/26 15:54:59 UTC
[GitHub] [spark] liangz1 opened a new pull request #33853: [WIP] Add df.withMetadata and make withColumn public
liangz1 opened a new pull request #33853:
URL: https://github.com/apache/spark/pull/33853
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error type or message, please read the guideline first in
'core/src/main/resources/error/README.md'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on pull request #33853: [WIP] Add df.withMetadata and make withColumn public
Posted by GitBox <gi...@apache.org>.
HyukjinKwon edited a comment on pull request #33853:
URL: https://github.com/apache/spark/pull/33853#issuecomment-909763962
We should add a test BTW - please describe "How was this patch tested?".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33853: [SPARK-36642] Add df.withMetadata: a syntax suger to update the metadata of a dataframe
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33853:
URL: https://github.com/apache/spark/pull/33853#issuecomment-910434812
**[Test build #142927 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142927/testReport)** for PR 33853 at commit [`3331254`](https://github.com/apache/spark/commit/3331254ba3e695571faf47d2a3afb740e0d3506c).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33853: [WIP] Add df.withMetadata and make withColumn public
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33853:
URL: https://github.com/apache/spark/pull/33853#issuecomment-909167036
**[Test build #142876 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142876/testReport)** for PR 33853 at commit [`8577fc9`](https://github.com/apache/spark/commit/8577fc9003e6e5070abbc5220eb13572d914a48a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33853: [SPARK-36642] Add df.withMetadata: a syntax suger to update the metadata of a dataframe
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33853:
URL: https://github.com/apache/spark/pull/33853#issuecomment-910767053
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142928/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33853: [WIP] Add df.withMetadata and make withColumn public
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33853:
URL: https://github.com/apache/spark/pull/33853#issuecomment-908997742
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47379/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33853: [SPARK-36642] Add df.withMetadata: a syntax suger to update the metadata of a dataframe
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33853:
URL: https://github.com/apache/spark/pull/33853#issuecomment-910757431
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142927/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33853: [SPARK-36642] Add df.withMetadata: a syntax suger to update the metadata of a dataframe
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33853:
URL: https://github.com/apache/spark/pull/33853#issuecomment-910455037
**[Test build #142928 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142928/testReport)** for PR 33853 at commit [`8b5bd77`](https://github.com/apache/spark/commit/8b5bd77739a571beec4d278d03834b18b6873040).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33853: [WIP] Add df.withMetadata and make withColumn public
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33853:
URL: https://github.com/apache/spark/pull/33853#issuecomment-906577371
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47325/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33853: Add df.withMetadata and make withColumn public
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33853:
URL: https://github.com/apache/spark/pull/33853#issuecomment-910112398
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47414/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33853: [SPARK-36642] Add df.withMetadata: a syntax suger to update the metadata of a dataframe
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33853:
URL: https://github.com/apache/spark/pull/33853#issuecomment-910530200
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47429/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on pull request #33853: [WIP] Add df.withMetadata and make withColumn public
Posted by GitBox <gi...@apache.org>.
HyukjinKwon edited a comment on pull request #33853:
URL: https://github.com/apache/spark/pull/33853#issuecomment-909763962
We should add a test BTW - please describe "How was this patch tested?".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33853: [WIP] Add df.withMetadata and make withColumn public
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33853:
URL: https://github.com/apache/spark/pull/33853#issuecomment-908962034
**[Test build #142876 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142876/testReport)** for PR 33853 at commit [`8577fc9`](https://github.com/apache/spark/commit/8577fc9003e6e5070abbc5220eb13572d914a48a).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33853: Add df.withMetadata and make withColumn public
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33853:
URL: https://github.com/apache/spark/pull/33853#issuecomment-910046622
**[Test build #142911 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142911/testReport)** for PR 33853 at commit [`603166d`](https://github.com/apache/spark/commit/603166d46ec2d504f01001c1c03e4cf645617b49).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33853: [WIP] Add df.withMetadata and make withColumn public
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33853:
URL: https://github.com/apache/spark/pull/33853#issuecomment-909167036
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on pull request #33853: Add df.withMetadata and make withColumn public
Posted by GitBox <gi...@apache.org>.
zero323 commented on pull request #33853:
URL: https://github.com/apache/spark/pull/33853#issuecomment-910309216
> I'm neutral on this change.
Same here. The current API is verbose, but metadata always felt more like an internal API (it's not properly documented for starters) and this type of operation can be easily added within the project (for example, with type classes).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] WeichenXu123 closed pull request #33853: [SPARK-36642][SQL] Add df.withMetadata: a syntax suger to update the metadata of a dataframe
Posted by GitBox <gi...@apache.org>.
WeichenXu123 closed pull request #33853:
URL: https://github.com/apache/spark/pull/33853
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33853: Add df.withMetadata and make withColumn public
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33853:
URL: https://github.com/apache/spark/pull/33853#discussion_r700065420
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
##########
@@ -702,6 +702,18 @@ class DataFrameSuite extends QueryTest
"The size of column names: 2 isn't equal to the size of metadata elements: 1"))
}
+ test("withMetadata: replace metadata of a column") {
Review comment:
Lets also add a jira prefix in the test title: "SPARK-XXXXX: withMetadata ... "
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33853: [SPARK-36642] Add df.withMetadata: a syntax suger to update the metadata of a dataframe
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33853:
URL: https://github.com/apache/spark/pull/33853#issuecomment-910486187
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47430/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] WeichenXu123 commented on a change in pull request #33853: [SPARK-36642][SQL] Add df.withMetadata: a syntax suger to update the metadata of a dataframe
Posted by GitBox <gi...@apache.org>.
WeichenXu123 commented on a change in pull request #33853:
URL: https://github.com/apache/spark/pull/33853#discussion_r700704639
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
##########
@@ -702,6 +702,18 @@ class DataFrameSuite extends QueryTest
"The size of column names: 2 isn't equal to the size of metadata elements: 1"))
}
+ test("SPARK-36642: withMetadata: replace metadata of a column") {
+ val metadata = new MetadataBuilder().putLong("key", 1L).build()
+ val df1 = sparkContext.parallelize(Array(1, 2, 3)).toDF("x")
+ val df2 = df1.withMetadata("x", metadata)
+ assert(df2.schema(0).metadata === metadata)
+
Review comment:
nit, add a test which column existing metadata, test replace it with a new metadata
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33853: [WIP] Add df.withMetadata and make withColumn public
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33853:
URL: https://github.com/apache/spark/pull/33853#issuecomment-908986350
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47379/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33853: [WIP] Add df.withMetadata and make withColumn public
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33853:
URL: https://github.com/apache/spark/pull/33853#issuecomment-909173740
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142876/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #33853: [WIP] Add df.withMetadata and make withColumn public
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #33853:
URL: https://github.com/apache/spark/pull/33853#discussion_r699274062
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
##########
@@ -2491,6 +2502,16 @@ class Dataset[T] private[sql](
}
}
+ /**
+ * Returns a new Dataset by updating an exisitng column with metadata.
+ *
+ * @group untypedrel
+ * @since 3.3.0
+ */
+ def withMetadata(existingColumnName: String, metadata: Metadata): DataFrame = {
+ withColumn(existingColumnName, col(existingColumnName), metadata)
Review comment:
It would be nice to know the motivation here (maybe we can start with a JIRA ticket?). Adding metadata with `as` (`alias` in Python) is a bit verbose / repetitive but it is not that common operation anyway.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] WeichenXu123 commented on pull request #33853: [SPARK-36642][SQL] Add df.withMetadata: a syntax suger to update the metadata of a dataframe
Posted by GitBox <gi...@apache.org>.
WeichenXu123 commented on pull request #33853:
URL: https://github.com/apache/spark/pull/33853#issuecomment-914792425
merged to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33853: [SPARK-36642] Add df.withMetadata: a syntax suger to update the metadata of a dataframe
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33853:
URL: https://github.com/apache/spark/pull/33853#issuecomment-910735290
**[Test build #142927 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142927/testReport)** for PR 33853 at commit [`3331254`](https://github.com/apache/spark/commit/3331254ba3e695571faf47d2a3afb740e0d3506c).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33853: [SPARK-36642] Add df.withMetadata: a syntax suger to update the metadata of a dataframe
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33853:
URL: https://github.com/apache/spark/pull/33853#issuecomment-910434812
**[Test build #142927 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142927/testReport)** for PR 33853 at commit [`3331254`](https://github.com/apache/spark/commit/3331254ba3e695571faf47d2a3afb740e0d3506c).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] WeichenXu123 commented on a change in pull request #33853: [SPARK-36642][SQL] Add df.withMetadata: a syntax suger to update the metadata of a dataframe
Posted by GitBox <gi...@apache.org>.
WeichenXu123 commented on a change in pull request #33853:
URL: https://github.com/apache/spark/pull/33853#discussion_r700704417
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
##########
@@ -2491,6 +2491,16 @@ class Dataset[T] private[sql](
}
}
+ /**
+ * Returns a new Dataset by updating an existing column with metadata.
+ *
+ * @group untypedrel
+ * @since 3.3.0
+ */
+ def withMetadata(columnName: String, metadata: Metadata): DataFrame = {
Review comment:
shall we use function name `withColumnMetadata` to make it more clear ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33853: Add df.withMetadata and make withColumn public
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33853:
URL: https://github.com/apache/spark/pull/33853#issuecomment-910112368
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47414/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33853: [SPARK-36642] Add df.withMetadata: a syntax suger to update the metadata of a dataframe
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33853:
URL: https://github.com/apache/spark/pull/33853#issuecomment-910757431
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142927/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33853: [SPARK-36642] Add df.withMetadata: a syntax suger to update the metadata of a dataframe
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33853:
URL: https://github.com/apache/spark/pull/33853#issuecomment-910493231
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47430/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33853: [SPARK-36642] Add df.withMetadata: a syntax suger to update the metadata of a dataframe
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33853:
URL: https://github.com/apache/spark/pull/33853#issuecomment-910517820
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47429/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] liangz1 commented on a change in pull request #33853: [SPARK-36642][SQL] Add df.withMetadata: a syntax suger to update the metadata of a dataframe
Posted by GitBox <gi...@apache.org>.
liangz1 commented on a change in pull request #33853:
URL: https://github.com/apache/spark/pull/33853#discussion_r700708369
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
##########
@@ -702,6 +702,18 @@ class DataFrameSuite extends QueryTest
"The size of column names: 2 isn't equal to the size of metadata elements: 1"))
}
+ test("SPARK-36642: withMetadata: replace metadata of a column") {
+ val metadata = new MetadataBuilder().putLong("key", 1L).build()
+ val df1 = sparkContext.parallelize(Array(1, 2, 3)).toDF("x")
+ val df2 = df1.withMetadata("x", metadata)
+ assert(df2.schema(0).metadata === metadata)
+
Review comment:
I think that might not be necessary because by default (`df1`), column `"x"` has a metadata `Metadata.empty`, so the new metadata is already "replacing" the existing default metadata.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #33853: [WIP] Add df.withMetadata and make withColumn public
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #33853:
URL: https://github.com/apache/spark/pull/33853#discussion_r699274062
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
##########
@@ -2491,6 +2502,16 @@ class Dataset[T] private[sql](
}
}
+ /**
+ * Returns a new Dataset by updating an exisitng column with metadata.
+ *
+ * @group untypedrel
+ * @since 3.3.0
+ */
+ def withMetadata(existingColumnName: String, metadata: Metadata): DataFrame = {
+ withColumn(existingColumnName, col(existingColumnName), metadata)
Review comment:
It would be nice to know the motivation here (maybe we can start with a JIRA ticket?). Adding metadata with `as` (`alias` in Python) is a bit verbose / repetitive but it is not that common operation anyway.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33853: [WIP] Add df.withMetadata and make withColumn public
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33853:
URL: https://github.com/apache/spark/pull/33853#issuecomment-908962034
**[Test build #142876 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142876/testReport)** for PR 33853 at commit [`8577fc9`](https://github.com/apache/spark/commit/8577fc9003e6e5070abbc5220eb13572d914a48a).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33853: Add df.withMetadata and make withColumn public
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33853:
URL: https://github.com/apache/spark/pull/33853#issuecomment-910249913
**[Test build #142911 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142911/testReport)** for PR 33853 at commit [`603166d`](https://github.com/apache/spark/commit/603166d46ec2d504f01001c1c03e4cf645617b49).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33853: Add df.withMetadata and make withColumn public
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33853:
URL: https://github.com/apache/spark/pull/33853#issuecomment-910268295
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142911/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33853: [SPARK-36642] Add df.withMetadata: a syntax suger to update the metadata of a dataframe
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33853:
URL: https://github.com/apache/spark/pull/33853#issuecomment-910493271
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47430/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33853: Add df.withMetadata and make withColumn public
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33853:
URL: https://github.com/apache/spark/pull/33853#issuecomment-910268295
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142911/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33853: [WIP] Add df.withMetadata and make withColumn public
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33853:
URL: https://github.com/apache/spark/pull/33853#issuecomment-908962034
**[Test build #142876 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142876/testReport)** for PR 33853 at commit [`8577fc9`](https://github.com/apache/spark/commit/8577fc9003e6e5070abbc5220eb13572d914a48a).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33853: [SPARK-36642] Add df.withMetadata: a syntax suger to update the metadata of a dataframe
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33853:
URL: https://github.com/apache/spark/pull/33853#issuecomment-910493271
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47430/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33853: [WIP] Add df.withMetadata and make withColumn public
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33853:
URL: https://github.com/apache/spark/pull/33853#issuecomment-906577371
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47325/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33853: [WIP] Add df.withMetadata and make withColumn public
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33853:
URL: https://github.com/apache/spark/pull/33853#issuecomment-909173740
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142876/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33853: [WIP] Add df.withMetadata and make withColumn public
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33853:
URL: https://github.com/apache/spark/pull/33853#issuecomment-906727106
**[Test build #142821 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142821/testReport)** for PR 33853 at commit [`de6af8b`](https://github.com/apache/spark/commit/de6af8b78ff78b47fcd2e0f85ca8d395c3454dec).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33853: [SPARK-36642] Add df.withMetadata: a syntax suger to update the metadata of a dataframe
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33853:
URL: https://github.com/apache/spark/pull/33853#issuecomment-910471912
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47429/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33853: [WIP] Add df.withMetadata and make withColumn public
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33853:
URL: https://github.com/apache/spark/pull/33853#issuecomment-909173740
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142876/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #33853: [WIP] Add df.withMetadata and make withColumn public
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #33853:
URL: https://github.com/apache/spark/pull/33853#discussion_r699274062
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
##########
@@ -2491,6 +2502,16 @@ class Dataset[T] private[sql](
}
}
+ /**
+ * Returns a new Dataset by updating an exisitng column with metadata.
+ *
+ * @group untypedrel
+ * @since 3.3.0
+ */
+ def withMetadata(existingColumnName: String, metadata: Metadata): DataFrame = {
+ withColumn(existingColumnName, col(existingColumnName), metadata)
Review comment:
It would be nice to know the motivation here (maybe we can start with a JIRA ticket?). Adding metadata with `as` (`alias` in Python) is a bit verbose / repetitive but it is not that common operation anyway.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33853: Add df.withMetadata and make withColumn public
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33853:
URL: https://github.com/apache/spark/pull/33853#issuecomment-910112398
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47414/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #33853: [WIP] Add df.withMetadata and make withColumn public
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #33853:
URL: https://github.com/apache/spark/pull/33853#issuecomment-909763864
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] liangz1 commented on pull request #33853: [SPARK-36642] Add df.withMetadata: a syntax suger to update the metadata of a dataframe
Posted by GitBox <gi...@apache.org>.
liangz1 commented on pull request #33853:
URL: https://github.com/apache/spark/pull/33853#issuecomment-910448508
@zero323 Thanks for your comments. I agree that the metadata is not well known to most starters and we may want to improve the documentation for it in a follow-up ticket.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33853: [WIP] Add df.withMetadata and make withColumn public
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33853:
URL: https://github.com/apache/spark/pull/33853#issuecomment-906734268
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142821/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33853: [WIP] Add df.withMetadata and make withColumn public
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33853:
URL: https://github.com/apache/spark/pull/33853#issuecomment-906537222
**[Test build #142821 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142821/testReport)** for PR 33853 at commit [`de6af8b`](https://github.com/apache/spark/commit/de6af8b78ff78b47fcd2e0f85ca8d395c3454dec).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33853: Add df.withMetadata and make withColumn public
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33853:
URL: https://github.com/apache/spark/pull/33853#issuecomment-910046622
**[Test build #142911 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142911/testReport)** for PR 33853 at commit [`603166d`](https://github.com/apache/spark/commit/603166d46ec2d504f01001c1c03e4cf645617b49).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33853: [SPARK-36642] Add df.withMetadata: a syntax suger to update the metadata of a dataframe
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33853:
URL: https://github.com/apache/spark/pull/33853#issuecomment-910767053
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142928/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33853: [WIP] Add df.withMetadata and make withColumn public
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33853:
URL: https://github.com/apache/spark/pull/33853#issuecomment-908997742
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47379/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33853: [WIP] Add df.withMetadata and make withColumn public
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33853:
URL: https://github.com/apache/spark/pull/33853#issuecomment-908997742
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33853: [WIP] Add df.withMetadata and make withColumn public
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33853:
URL: https://github.com/apache/spark/pull/33853#issuecomment-908987636
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47379/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33853: [WIP] Add df.withMetadata and make withColumn public
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33853:
URL: https://github.com/apache/spark/pull/33853#issuecomment-908962034
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #33853: [SPARK-36642] Add df.withMetadata: a syntax suger to update the metadata of a dataframe
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #33853:
URL: https://github.com/apache/spark/pull/33853#discussion_r700363016
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
##########
@@ -2491,6 +2491,16 @@ class Dataset[T] private[sql](
}
}
+ /**
+ * Returns a new Dataset by updating an exisitng column with metadata.
+ *
+ * @group untypedrel
+ * @since 3.3.0
+ */
+ def withMetadata(existingColumnName: String, metadata: Metadata): DataFrame = {
Review comment:
nit: existingColumnName is too verbose - how about `columnName`?
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
##########
@@ -2491,6 +2491,16 @@ class Dataset[T] private[sql](
}
}
+ /**
+ * Returns a new Dataset by updating an exisitng column with metadata.
Review comment:
typo: exisitng -> existing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33853: [SPARK-36642] Add df.withMetadata: a syntax suger to update the metadata of a dataframe
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33853:
URL: https://github.com/apache/spark/pull/33853#issuecomment-910764998
**[Test build #142928 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142928/testReport)** for PR 33853 at commit [`8b5bd77`](https://github.com/apache/spark/commit/8b5bd77739a571beec4d278d03834b18b6873040).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33853: [SPARK-36642] Add df.withMetadata: a syntax suger to update the metadata of a dataframe
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33853:
URL: https://github.com/apache/spark/pull/33853#issuecomment-910455037
**[Test build #142928 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142928/testReport)** for PR 33853 at commit [`8b5bd77`](https://github.com/apache/spark/commit/8b5bd77739a571beec4d278d03834b18b6873040).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #33853: [WIP] Add df.withMetadata and make withColumn public
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #33853:
URL: https://github.com/apache/spark/pull/33853#issuecomment-909763962
We should add a test BTW.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33853: [WIP] Add df.withMetadata and make withColumn public
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33853:
URL: https://github.com/apache/spark/pull/33853#issuecomment-909173740
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142876/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33853: Add df.withMetadata and make withColumn public
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33853:
URL: https://github.com/apache/spark/pull/33853#issuecomment-910079457
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47414/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] liangz1 commented on a change in pull request #33853: [SPARK-36642][SQL] Add df.withMetadata: a syntax suger to update the metadata of a dataframe
Posted by GitBox <gi...@apache.org>.
liangz1 commented on a change in pull request #33853:
URL: https://github.com/apache/spark/pull/33853#discussion_r700706967
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
##########
@@ -2491,6 +2491,16 @@ class Dataset[T] private[sql](
}
}
+ /**
+ * Returns a new Dataset by updating an existing column with metadata.
+ *
+ * @group untypedrel
+ * @since 3.3.0
+ */
+ def withMetadata(columnName: String, metadata: Metadata): DataFrame = {
Review comment:
That would also work, but I prefer the current shorter version, since the first param requires a column name, it's not likely for the users to be confused about whether the metadata is for a column or something else. Also, since this is syntax sugar, a shorter name would better justify the new API.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33853: [WIP] Add df.withMetadata and make withColumn public
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33853:
URL: https://github.com/apache/spark/pull/33853#issuecomment-908962034
**[Test build #142876 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142876/testReport)** for PR 33853 at commit [`8577fc9`](https://github.com/apache/spark/commit/8577fc9003e6e5070abbc5220eb13572d914a48a).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33853: [WIP] Add df.withMetadata and make withColumn public
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33853:
URL: https://github.com/apache/spark/pull/33853#issuecomment-906734268
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142821/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33853: [SPARK-36642] Add df.withMetadata: a syntax suger to update the metadata of a dataframe
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33853:
URL: https://github.com/apache/spark/pull/33853#issuecomment-910530200
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47429/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #33853: Add df.withMetadata and make withColumn public
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #33853:
URL: https://github.com/apache/spark/pull/33853#issuecomment-910122046
Can we file a JIRA and link it in the PR title? I'm neutral on this change. Would need to wait for other committers to review and approve.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #33853: [WIP] Add df.withMetadata and make withColumn public
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #33853:
URL: https://github.com/apache/spark/pull/33853#issuecomment-909763864
Okay, had an offline discussion. I am fine with the justification shown in the PR description given that it's going to be used frequently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33853: [WIP] Add df.withMetadata and make withColumn public
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33853:
URL: https://github.com/apache/spark/pull/33853#issuecomment-908997742
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33853: [WIP] Add df.withMetadata and make withColumn public
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33853:
URL: https://github.com/apache/spark/pull/33853#discussion_r697084452
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
##########
@@ -2462,8 +2462,19 @@ class Dataset[T] private[sql](
/**
* Returns a new Dataset by adding a column with metadata.
+ *
+ * `column`'s expression must only refer to attributes supplied by this Dataset. It is an
+ * error to add a column that refers to some other Dataset.
+ *
+ * @note this method introduces a projection internally. Therefore, calling it multiple times,
+ * for instance, via loops in order to add multiple columns can generate big plans which
+ * can cause performance issues and even `StackOverflowException`. To avoid this,
+ * use `select` with the multiple columns at once.
+ *
+ * @group untypedrel
+ * @since 3.3.0
*/
- private[spark] def withColumn(colName: String, col: Column, metadata: Metadata): DataFrame =
+ def withColumn(colName: String, col: Column, metadata: Metadata): DataFrame =
Review comment:
Why can't we simply use `Column.as` with metadata?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33853: [WIP] Add df.withMetadata and make withColumn public
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33853:
URL: https://github.com/apache/spark/pull/33853#issuecomment-906537222
**[Test build #142821 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142821/testReport)** for PR 33853 at commit [`de6af8b`](https://github.com/apache/spark/commit/de6af8b78ff78b47fcd2e0f85ca8d395c3454dec).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33853: [WIP] Add df.withMetadata and make withColumn public
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33853:
URL: https://github.com/apache/spark/pull/33853#discussion_r697084219
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
##########
@@ -2491,6 +2502,16 @@ class Dataset[T] private[sql](
}
}
+ /**
+ * Returns a new Dataset by updating an exisitng column with metadata.
+ *
+ * @group untypedrel
+ * @since 3.3.0
+ */
+ def withMetadata(existingColumnName: String, metadata: Metadata): DataFrame = {
+ withColumn(existingColumnName, col(existingColumnName), metadata)
Review comment:
I wouldn't add this one. This can be easily done in app side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33853: [WIP] Add df.withMetadata and make withColumn public
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33853:
URL: https://github.com/apache/spark/pull/33853#issuecomment-906569063
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47325/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org