You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/03/28 00:00:11 UTC
[GitHub] [spark] huaxingao opened a new pull request #28058:
[SPARK-31290][SQL][R] Add back the deprecated R APIs
huaxingao opened a new pull request #28058: [SPARK-31290][SQL][R] Add back the deprecated R APIs
URL: https://github.com/apache/spark/pull/28058
### What changes were proposed in this pull request?
Add back the deprecated R APIs removed by
https://github.com/apache/spark/pull/22843/
https://github.com/apache/spark/pull/22815
These APIs are
```
sparkR.init
jsonFile
parquetFile
registerTempTable
saveAsParquetFile
unionAll
createExternalTable
dropTempTable
```
### Why are the changes needed?
Amend Spark's Semantic Versioning Policy
### Does this PR introduce any user-facing change?
Yes
The removed R APIs are put back.
### How was this patch tested?
Add back the removed tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #28058:
[SPARK-31290][R] Add back the deprecated R APIs
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #28058: [SPARK-31290][R] Add back the deprecated R APIs
URL: https://github.com/apache/spark/pull/28058#discussion_r399892241
##########
File path: R/pkg/R/catalog.R
##########
@@ -130,6 +159,29 @@ clearCache <- function() {
invisible(callJMethod(catalog, "clearCache"))
}
+#' Drops the temporary table with the given table name in the catalog.
Review comment:
`#' (Deprecated) Drop Temporary Table` seems missed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28058:
[SPARK-31290][SQL][R] Add back the deprecated R APIs
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28058: [SPARK-31290][SQL][R] Add back the deprecated R APIs
URL: https://github.com/apache/spark/pull/28058#issuecomment-605362768
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on issue #28058:
[SPARK-31290][R] Add back the deprecated R APIs
Posted by GitBox <gi...@apache.org>.
HyukjinKwon edited a comment on issue #28058: [SPARK-31290][R] Add back the deprecated R APIs
URL: https://github.com/apache/spark/pull/28058#issuecomment-605758357
It seems more complicated than I thought. Let me try to clarify the current status here:
#### `jsonFile` and `parquetFile`
- Scala side, `jsonFile`, `parquetFile` were removed at https://github.com/apache/spark/pull/10559 and recovered at https://github.com/apache/spark/pull/13637 in the same Spark 2.0.
- PySpark side, they were removed at https://github.com/apache/spark/pull/10559 and were not recovered afterwards.
- SparkR side, they were never removed
#### `jsonRDD`
- SparkR made RDD API as private APIs as of SPARK-7230; RDD related APIs are supposed to be removed as they are uesless.
#### `saveAsParquetFile`
- Scala side `saveAsParquetFile` was removed at https://github.com/apache/spark/pull/10559
- PySpark side, they were removed at https://github.com/apache/spark/pull/10559.
- SparkR side, they were never removed.
I skimmed user-mailing list with keyword search, and I found no complaints. This implies, no complaints found about:
- `jsonFile`, `parquetFile` and `jsonRDD` missing in PySpark from Spark 2.0
- `saveAsParquetFile` in Scala and PySpark from Spark 2.0
So, I guess it's fine to don't add `jsonFile`, `parquetFile`, `jsonRDD` and `saveAsParquetFile` back.
<br/>
**TL;DR**: from my rough investigation, I think it's fine to don't add `jsonFile`, `parquetFile`, `jsonRDD` and `saveAsParquetFile` because:
- people don't quite care about `jsonFile`, `parquetFile`, `jsonRDD` and `saveAsParquetFile`.
- `jsonFile` and `parquetFile` don't exist in PySpark.
- `jsonRDD` is useless as RDD APIs became private in SparkR.
- `saveAsParquetFile` does not exist in Scala and PySpark sides.
WDYT @falaki and @mengxr?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28058: [SPARK-31290][R]
Add back the deprecated R APIs
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28058: [SPARK-31290][R] Add back the deprecated R APIs
URL: https://github.com/apache/spark/pull/28058#issuecomment-605772865
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #28058: [SPARK-31290][R] Add
back the deprecated R APIs
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on issue #28058: [SPARK-31290][R] Add back the deprecated R APIs
URL: https://github.com/apache/spark/pull/28058#issuecomment-606974763
I discussed offline with @marmbrus. Looks we're fine to exclude these four above.
Merged to master and branch-3.0.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] rxin commented on issue #28058: [SPARK-31290][R] Add back
the deprecated R APIs
Posted by GitBox <gi...@apache.org>.
rxin commented on issue #28058: [SPARK-31290][R] Add back the deprecated R APIs
URL: https://github.com/apache/spark/pull/28058#issuecomment-605721398
If they have already been removed prior to 3.0 and nobody has said anything, I don't think we should add those back in.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on issue #28058:
[SPARK-31290][R] Add back the deprecated R APIs
Posted by GitBox <gi...@apache.org>.
HyukjinKwon edited a comment on issue #28058: [SPARK-31290][R] Add back the deprecated R APIs
URL: https://github.com/apache/spark/pull/28058#issuecomment-605726816
@huaxingao, seems like we roughly agreed upon excluding `jsonFile`, `parquetFile` and `saveAsParquetFile` at SPARK-12600. `jsonRDD` seems not making sense to add back since SparkR dropped RDD support officially. Can we remove these four in this PR?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #28058:
[SPARK-31290][R] Add back the deprecated R APIs
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #28058: [SPARK-31290][R] Add back the deprecated R APIs
URL: https://github.com/apache/spark/pull/28058#discussion_r399744816
##########
File path: R/pkg/R/DataFrame.R
##########
@@ -946,6 +973,17 @@ setMethod("write.parquet",
invisible(handledCallJMethod(write, "parquet", path))
})
+#' @rdname write.parquet
+#' @name saveAsParquetFile
+#' @aliases saveAsParquetFile,SparkDataFrame,character-method
+#' @note saveAsParquetFile since 1.4.0
+setMethod("saveAsParquetFile",
Review comment:
Okay, using the rubric we added, in R or PySpark, it's very easy to keep the compatibility - if-else based on version and redirect to the alternative. There seems direct alternative too.
Seems like we haven't seen a lot of complaints when we remove `saveAsParquetFile` at SPARK-12600. I skimmed user-mailing list with keyword search, and I found 0 complaints.
Having write API in `DataFrame` seems inconsistent with Scala and PySpark sides, and also namespace used in R side (e.g., `write.json`) - it causes a cognitive cost.
Shall we remove these one out?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on issue #28058:
[SPARK-31290][R] Add back the deprecated R APIs
Posted by GitBox <gi...@apache.org>.
HyukjinKwon edited a comment on issue #28058: [SPARK-31290][R] Add back the deprecated R APIs
URL: https://github.com/apache/spark/pull/28058#issuecomment-605726258
Just to clarify things, what the community agreed on is the rubric added at https://spark.apache.org/versioning-policy.html.
Some APIs listed here, `jsonFile`, `parquetFile` and `saveAsParquetFile` were removed in Scala and PySpark before Spark 3.0. I skimmed user mailing list by keyword search and nobody complained.
One last thing is, SparkR dropped RDD support. So I don't think it also makes sense to add `jsonRDD` back.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28058: [SPARK-31290][R]
Add back the deprecated R APIs
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28058: [SPARK-31290][R] Add back the deprecated R APIs
URL: https://github.com/apache/spark/pull/28058#issuecomment-605740098
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on issue #28058:
[SPARK-31290][R] Add back the deprecated R APIs
Posted by GitBox <gi...@apache.org>.
HyukjinKwon edited a comment on issue #28058: [SPARK-31290][R] Add back the deprecated R APIs
URL: https://github.com/apache/spark/pull/28058#issuecomment-605758357
It seems more complicated than I though. Let me try to clarify the current status here:
#### `jsonFile` and `parquetFile`
- Scala side, `jsonFile`, `parquetFile` were removed at https://github.com/apache/spark/pull/10559 and recovered at https://github.com/apache/spark/pull/13637 in the same Spark 2.0.
- PySpark side, they were removed at https://github.com/apache/spark/pull/10559 and were not recovered afterwards.
- SparkR side, they were never removed
#### `jsonRDD`
- SparkR made RDD API as private APIs as of SPARK-7230; RDD related APIs are supposed to be removed as they are uesless.
#### `saveAsParquetFile`
- Scala side `saveAsParquetFile` was removed at https://github.com/apache/spark/pull/10559
- PySpark side, they were removed at https://github.com/apache/spark/pull/10559.
- SparkR side, they were never removed.
I skimmed user-mailing list with keyword search, and I found no complaints. This implies, no complaints found about:
- `jsonFile`, `parquetFile` and `jsonRDD` missing in PySpark from Spark 2.0
- `saveAsParquetFile` in Scala and PySpark from Spark 2.0
So, I guess it's fine to don't add `jsonFile`, `parquetFile`, `jsonRDD` and `saveAsParquetFile` back.
<br/>
**TR;DL**: from my rough investigation, I think it's fine to don't add `jsonFile`, `parquetFile`, `jsonRDD` and `saveAsParquetFile` because:
- people don't quite care about `jsonFile`, `parquetFile`, `jsonRDD` and `saveAsParquetFile`.
- `jsonFile` and `parquetFile` don't exist in PySpark.
- `jsonRDD` is useless as RDD APIs became private in SparkR.
- `saveAsParquetFile` does not exist in Scala and PySpark sides.
WDYT @falaki and @mengxr?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #28058: [SPARK-31290][R]
Add back the deprecated R APIs
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #28058: [SPARK-31290][R] Add back the deprecated R APIs
URL: https://github.com/apache/spark/pull/28058#issuecomment-605732614
**[Test build #120563 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120563/testReport)** for PR 28058 at commit [`754f938`](https://github.com/apache/spark/commit/754f93863bfec47d6c53a796ad70bdfb3a187441).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #28058:
[SPARK-31290][SQL][R] Add back the deprecated R APIs
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #28058: [SPARK-31290][SQL][R] Add back the deprecated R APIs
URL: https://github.com/apache/spark/pull/28058#issuecomment-605362512
**[Test build #120517 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120517/testReport)** for PR 28058 at commit [`69e2953`](https://github.com/apache/spark/commit/69e295363ac623456695a553325b0e4e7a48f806).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on issue #28058: [SPARK-31290][R] Add
back the deprecated R APIs
Posted by GitBox <gi...@apache.org>.
huaxingao commented on issue #28058: [SPARK-31290][R] Add back the deprecated R APIs
URL: https://github.com/apache/spark/pull/28058#issuecomment-606976158
Thank you all for the help!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28058:
[SPARK-31290][R] Add back the deprecated R APIs
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28058: [SPARK-31290][R] Add back the deprecated R APIs
URL: https://github.com/apache/spark/pull/28058#issuecomment-605764080
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25271/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28058: [SPARK-31290][SQL][R]
Add back the deprecated R APIs
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28058: [SPARK-31290][SQL][R] Add back the deprecated R APIs
URL: https://github.com/apache/spark/pull/28058#issuecomment-605368897
**[Test build #120517 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120517/testReport)** for PR 28058 at commit [`69e2953`](https://github.com/apache/spark/commit/69e295363ac623456695a553325b0e4e7a48f806).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28058:
[SPARK-31290][R] Add back the deprecated R APIs
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28058: [SPARK-31290][R] Add back the deprecated R APIs
URL: https://github.com/apache/spark/pull/28058#issuecomment-605732881
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25269/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28058: [SPARK-31290][R] Add
back the deprecated R APIs
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28058: [SPARK-31290][R] Add back the deprecated R APIs
URL: https://github.com/apache/spark/pull/28058#issuecomment-605763711
**[Test build #120567 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120567/testReport)** for PR 28058 at commit [`ce5969b`](https://github.com/apache/spark/commit/ce5969b95aa7de4a09fb39ca58dc8f2a7dc2f0f0).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28058: [SPARK-31290][R] Add
back the deprecated R APIs
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28058: [SPARK-31290][R] Add back the deprecated R APIs
URL: https://github.com/apache/spark/pull/28058#issuecomment-605772758
**[Test build #120567 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120567/testReport)** for PR 28058 at commit [`ce5969b`](https://github.com/apache/spark/commit/ce5969b95aa7de4a09fb39ca58dc8f2a7dc2f0f0).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on issue #28058: [SPARK-31290][SQL][R]
Add back the deprecated R APIs
Posted by GitBox <gi...@apache.org>.
huaxingao commented on issue #28058: [SPARK-31290][SQL][R] Add back the deprecated R APIs
URL: https://github.com/apache/spark/pull/28058#issuecomment-605362027
cc @HyukjinKwon @felixcheung
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on issue #28058:
[SPARK-31290][R] Add back the deprecated R APIs
Posted by GitBox <gi...@apache.org>.
HyukjinKwon edited a comment on issue #28058: [SPARK-31290][R] Add back the deprecated R APIs
URL: https://github.com/apache/spark/pull/28058#issuecomment-605726258
Just to clarify things, what the community agreed on is the rubric added at https://spark.apache.org/versioning-policy.html.
Several APIs listed here, `jsonFile`, `parquetFile` and `saveAsParquetFile` were removed in Scala and PySpark before Spark 3.0 at SPARK-12600. I skimmed user mailing list by keyword search and nobody complained.
One last thing is, SparkR dropped RDD support. So I don't think it also makes sense to add `jsonRDD` back.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] felixcheung commented on issue #28058: [SPARK-31290][R] Add
back the deprecated R APIs
Posted by GitBox <gi...@apache.org>.
felixcheung commented on issue #28058: [SPARK-31290][R] Add back the deprecated R APIs
URL: https://github.com/apache/spark/pull/28058#issuecomment-605721121
They don’t have 1:1 to Scala. Some R APIs are older and long since removed even in Scala before 3.0.
I’m fine with this change if this is what the community has agreed on.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on issue #28058:
[SPARK-31290][R] Add back the deprecated R APIs
Posted by GitBox <gi...@apache.org>.
HyukjinKwon edited a comment on issue #28058: [SPARK-31290][R] Add back the deprecated R APIs
URL: https://github.com/apache/spark/pull/28058#issuecomment-605758357
It seems more complicated than I thought. Let me try to clarify the current status here:
#### `jsonFile` and `parquetFile`
- Scala side, `jsonFile`, `parquetFile` were removed at https://github.com/apache/spark/pull/10559 and recovered at https://github.com/apache/spark/pull/13637 in the same Spark 2.0.
- PySpark side, they were removed at https://github.com/apache/spark/pull/10559 and were not recovered afterwards.
- SparkR side, they were never removed
#### `jsonRDD`
- SparkR made RDD API as private APIs as of SPARK-7230; RDD related APIs are supposed to be removed as they are uesless.
#### `saveAsParquetFile`
- Scala side `saveAsParquetFile` was removed at https://github.com/apache/spark/pull/10559
- PySpark side, it was also removed at https://github.com/apache/spark/pull/10559.
- SparkR side, it was never removed.
I skimmed user-mailing list with keyword search, and I found no complaints. This implies, no complaints found about:
- `jsonFile`, `parquetFile` and `jsonRDD` missing in PySpark from Spark 2.0
- `saveAsParquetFile` in Scala and PySpark from Spark 2.0
So, I guess it's fine to don't add `jsonFile`, `parquetFile`, `jsonRDD` and `saveAsParquetFile` back.
<br/>
**TL;DR**: from my rough investigation, I think it's fine to don't add `jsonFile`, `parquetFile`, `jsonRDD` and `saveAsParquetFile` because:
- people don't quite care about `jsonFile`, `parquetFile`, `jsonRDD` and `saveAsParquetFile`.
- `jsonFile` and `parquetFile` don't exist in PySpark.
- `jsonRDD` is useless as RDD APIs became private in SparkR.
- `saveAsParquetFile` does not exist in Scala and PySpark sides.
WDYT @falaki and @mengxr?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28058:
[SPARK-31290][SQL][R] Add back the deprecated R APIs
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28058: [SPARK-31290][SQL][R] Add back the deprecated R APIs
URL: https://github.com/apache/spark/pull/28058#issuecomment-605368966
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120517/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28058:
[SPARK-31290][SQL][R] Add back the deprecated R APIs
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28058: [SPARK-31290][SQL][R] Add back the deprecated R APIs
URL: https://github.com/apache/spark/pull/28058#issuecomment-605362768
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28058: [SPARK-31290][R]
Add back the deprecated R APIs
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28058: [SPARK-31290][R] Add back the deprecated R APIs
URL: https://github.com/apache/spark/pull/28058#issuecomment-605732877
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28058: [SPARK-31290][SQL][R]
Add back the deprecated R APIs
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28058: [SPARK-31290][SQL][R] Add back the deprecated R APIs
URL: https://github.com/apache/spark/pull/28058#issuecomment-605362512
**[Test build #120517 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120517/testReport)** for PR 28058 at commit [`69e2953`](https://github.com/apache/spark/commit/69e295363ac623456695a553325b0e4e7a48f806).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on issue #28058:
[SPARK-31290][R] Add back the deprecated R APIs
Posted by GitBox <gi...@apache.org>.
HyukjinKwon edited a comment on issue #28058: [SPARK-31290][R] Add back the deprecated R APIs
URL: https://github.com/apache/spark/pull/28058#issuecomment-605758357
It seems more complicated than I though. Let me try to clarify the current status here:
#### `jsonFile` and `parquetFile`
- Scala side, `jsonFile`, `parquetFile` were removed at https://github.com/apache/spark/pull/10559 and recovered at https://github.com/apache/spark/pull/13637 in the same Spark 2.0.
- PySpark side, they were removed at https://github.com/apache/spark/pull/10559 and were not recovered afterwards.
- SparkR side, they were never removed
#### `jsonRDD`
- SparkR made RDD API as private APIs as of SPARK-7230; RDD related APIs are supposed to be removed as they are uesless.
#### `saveAsParquetFile`
- Scala side `saveAsParquetFile` was removed at https://github.com/apache/spark/pull/10559
- PySpark side, they were removed at https://github.com/apache/spark/pull/10559.
- SparkR side, they were never removed.
I skimmed user-mailing list with keyword search, and I found no complaints. This implies, no complaints found about:
- `jsonFile`, `parquetFile` and `jsonRDD` missing in PySpark from Spark 2.0
- `saveAsParquetFile` in Scala and PySpark from Spark 2.0
So, I guess it's fine to don't add `jsonFile`, `parquetFile`, `jsonRDD` and `saveAsParquetFile` back.
<br/>
**TL;DR**: from my rough investigation, I think it's fine to don't add `jsonFile`, `parquetFile`, `jsonRDD` and `saveAsParquetFile` because:
- people don't quite care about `jsonFile`, `parquetFile`, `jsonRDD` and `saveAsParquetFile`.
- `jsonFile` and `parquetFile` don't exist in PySpark.
- `jsonRDD` is useless as RDD APIs became private in SparkR.
- `saveAsParquetFile` does not exist in Scala and PySpark sides.
WDYT @falaki and @mengxr?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #28058: [SPARK-31290][R] Add
back the deprecated R APIs
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on issue #28058: [SPARK-31290][R] Add back the deprecated R APIs
URL: https://github.com/apache/spark/pull/28058#issuecomment-605726258
Just to clarify things, what the community agreed on is the rubric added at https://spark.apache.org/versioning-policy.html.
Some APIs listed here, `jsonFile`, `parquetFile` and `saveAsParquetFile` were removed in Scala and PySpark before Spark 3.0. I skimmed nobody complained in user mailing list by keyword search.
One last thing is, SparkR dropped RDD support. So I don't think it also makes sense to add `jsonRDD`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #28058: [SPARK-31290][R] Add
back the deprecated R APIs
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on issue #28058: [SPARK-31290][R] Add back the deprecated R APIs
URL: https://github.com/apache/spark/pull/28058#issuecomment-605740449
@huaxingao, don't forget to update PR description :-)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #28058: [SPARK-31290][R] Add
back the deprecated R APIs
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on issue #28058: [SPARK-31290][R] Add back the deprecated R APIs
URL: https://github.com/apache/spark/pull/28058#issuecomment-605726816
@huaxingao, seems like we roughly agreed upon excluding `jsonFile`, `parquetFile` and `saveAsParquetFile`. `jsonRDD` seems not making sense to add back since SparkR dropped RDD support officially. Can we remove these four in this PR?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28058:
[SPARK-31290][SQL][R] Add back the deprecated R APIs
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28058: [SPARK-31290][SQL][R] Add back the deprecated R APIs
URL: https://github.com/apache/spark/pull/28058#issuecomment-605362772
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25223/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28058: [SPARK-31290][R] Add
back the deprecated R APIs
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28058: [SPARK-31290][R] Add back the deprecated R APIs
URL: https://github.com/apache/spark/pull/28058#issuecomment-605732614
**[Test build #120563 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120563/testReport)** for PR 28058 at commit [`754f938`](https://github.com/apache/spark/commit/754f93863bfec47d6c53a796ad70bdfb3a187441).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28058:
[SPARK-31290][R] Add back the deprecated R APIs
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28058: [SPARK-31290][R] Add back the deprecated R APIs
URL: https://github.com/apache/spark/pull/28058#issuecomment-605732877
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28058: [SPARK-31290][R] Add
back the deprecated R APIs
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28058: [SPARK-31290][R] Add back the deprecated R APIs
URL: https://github.com/apache/spark/pull/28058#issuecomment-605740023
**[Test build #120563 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120563/testReport)** for PR 28058 at commit [`754f938`](https://github.com/apache/spark/commit/754f93863bfec47d6c53a796ad70bdfb3a187441).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28058:
[SPARK-31290][SQL][R] Add back the deprecated R APIs
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28058: [SPARK-31290][SQL][R] Add back the deprecated R APIs
URL: https://github.com/apache/spark/pull/28058#issuecomment-605368963
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28058: [SPARK-31290][R]
Add back the deprecated R APIs
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28058: [SPARK-31290][R] Add back the deprecated R APIs
URL: https://github.com/apache/spark/pull/28058#issuecomment-605732881
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25269/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on issue #28058:
[SPARK-31290][R] Add back the deprecated R APIs
Posted by GitBox <gi...@apache.org>.
HyukjinKwon edited a comment on issue #28058: [SPARK-31290][R] Add back the deprecated R APIs
URL: https://github.com/apache/spark/pull/28058#issuecomment-605758357
It seems more complicated than I thought. Let me try to clarify the current status here:
#### `jsonFile` and `parquetFile`
- Scala side, `jsonFile`, `parquetFile` were removed at https://github.com/apache/spark/pull/10559 and recovered at https://github.com/apache/spark/pull/13637 in the same Spark 2.0.
- PySpark side, they were removed at https://github.com/apache/spark/pull/10559 and were not recovered afterwards.
- SparkR side, they were never removed
#### `jsonRDD`
- SparkR made RDD API as private APIs as of SPARK-7230; RDD related APIs are supposed to be removed as they are uesless.
#### `saveAsParquetFile`
- Scala side `saveAsParquetFile` was removed at https://github.com/apache/spark/pull/10559
- PySpark side, it was also removed at https://github.com/apache/spark/pull/10559.
- SparkR side, it was never removed.
I skimmed user-mailing list with keyword search, and I found no complaints. This implies, no complaints found about:
- `jsonFile`, `parquetFile` and `jsonRDD` missing in PySpark from Spark 2.0
- `saveAsParquetFile` in Scala and PySpark from Spark 2.0
So, I guess it's fine to don't add `jsonFile`, `parquetFile`, `jsonRDD` and `saveAsParquetFile` back.
<br/>
**TL;DR**: from my rough investigation, I think it's fine to don't add `jsonFile`, `parquetFile`, `jsonRDD` and `saveAsParquetFile` because:
- people don't quite care about `jsonFile`, `parquetFile`, `jsonRDD` and `saveAsParquetFile`.
- `jsonFile` and `parquetFile` don't exist in PySpark.
- `jsonRDD` is useless as RDD APIs became private in SparkR.
- `saveAsParquetFile` does not exist in Scala and PySpark sides.
WDYT @falaki, @mengxr, @felixcheung?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #28058: [SPARK-31290][R] Add
back the deprecated R APIs
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on issue #28058: [SPARK-31290][R] Add back the deprecated R APIs
URL: https://github.com/apache/spark/pull/28058#issuecomment-605758357
It seems more complicated than I though. Let me try to clarify the current status here:
#### `jsonFile` and `parquetFile`
- Scala side, `jsonFile`, `parquetFile` were removed at https://github.com/apache/spark/pull/10559 and recovered at https://github.com/apache/spark/pull/13637 in the same Spark 2.0.
- PySpark side, they were removed at https://github.com/apache/spark/pull/10559 and were not recovered afterwards.
- SparkR side, they were never removed
#### `jsonRDD`
- SparkR made RDD API as private APIs as of SPARK-7230; RDD related APIs are supposed to be removed as they are uesless.
#### `saveAsParquetFile`
- Scala side `saveAsParquetFile` was removed at https://github.com/apache/spark/pull/10559
- PySpark side, they were removed at https://github.com/apache/spark/pull/10559.
- SparkR side, they were never removed.
I skimmed user-mailing list with keyword search, and I found no complaints. This implies, no complaints found about:
- `jsonFile`, `parquetFile` and `jsonRDD` missing in PySpark from Spark 2.0
- `saveAsParquetFile` in Scala and PySpark from Spark 2.0
So, I guess it's fine to don't add `jsonFile`, `parquetFile`, `jsonRDD` and `saveAsParquetFile` back.
<br/>
TR;DL: from my rough investigation, I think it's fine to don't add `jsonFile`, `parquetFile`, `jsonRDD` and `saveAsParquetFile` because:
- people don't quite care about `jsonFile`, `parquetFile`, `jsonRDD` and `saveAsParquetFile`.
- `jsonFile` and `parquetFile` don't exist in PySpark.
- `jsonRDD` is useless as RDD APIs became private in SparkR.
- `saveAsParquetFile` does not exist in Scala and PySpark sides.
WDYT @falaki and @mengxr?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] falaki commented on issue #28058: [SPARK-31290][SQL][R] Add
back the deprecated R APIs
Posted by GitBox <gi...@apache.org>.
falaki commented on issue #28058: [SPARK-31290][SQL][R] Add back the deprecated R APIs
URL: https://github.com/apache/spark/pull/28058#issuecomment-605392260
If we have added these back to Scala, adding them to R seems good.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #28058: [SPARK-31290][R]
Add back the deprecated R APIs
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #28058: [SPARK-31290][R] Add back the deprecated R APIs
URL: https://github.com/apache/spark/pull/28058
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28058:
[SPARK-31290][R] Add back the deprecated R APIs
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28058: [SPARK-31290][R] Add back the deprecated R APIs
URL: https://github.com/apache/spark/pull/28058#issuecomment-605740101
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120563/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao edited a comment on issue #28058:
[SPARK-31290][SQL][R] Add back the deprecated R APIs
Posted by GitBox <gi...@apache.org>.
huaxingao edited a comment on issue #28058: [SPARK-31290][SQL][R] Add back the deprecated R APIs
URL: https://github.com/apache/spark/pull/28058#issuecomment-605371895
Do we need to add back ```saveAsParquetFile```? Seems it is not added back in scala? Also, do we need to add back ```sparkR.init```, ```sparkRSQL.init```, ```sparkRHive.init```?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28058:
[SPARK-31290][R] Add back the deprecated R APIs
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28058: [SPARK-31290][R] Add back the deprecated R APIs
URL: https://github.com/apache/spark/pull/28058#issuecomment-605740098
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on issue #28058:
[SPARK-31290][R] Add back the deprecated R APIs
Posted by GitBox <gi...@apache.org>.
HyukjinKwon edited a comment on issue #28058: [SPARK-31290][R] Add back the deprecated R APIs
URL: https://github.com/apache/spark/pull/28058#issuecomment-605726258
Just to clarify things, what the community agreed on is the rubric added at https://spark.apache.org/versioning-policy.html.
Several APIs listed here, `jsonFile`, `parquetFile` and `saveAsParquetFile` were removed in Scala and PySpark before Spark 3.0. I skimmed user mailing list by keyword search and nobody complained.
One last thing is, SparkR dropped RDD support. So I don't think it also makes sense to add `jsonRDD` back.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28058:
[SPARK-31290][SQL][R] Add back the deprecated R APIs
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28058: [SPARK-31290][SQL][R] Add back the deprecated R APIs
URL: https://github.com/apache/spark/pull/28058#issuecomment-605368963
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28058:
[SPARK-31290][R] Add back the deprecated R APIs
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28058: [SPARK-31290][R] Add back the deprecated R APIs
URL: https://github.com/apache/spark/pull/28058#issuecomment-605772865
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on issue #28058: [SPARK-31290][SQL][R]
Add back the deprecated R APIs
Posted by GitBox <gi...@apache.org>.
huaxingao commented on issue #28058: [SPARK-31290][SQL][R] Add back the deprecated R APIs
URL: https://github.com/apache/spark/pull/28058#issuecomment-605371895
Do we need to add back ```saveAsParquetFile```? Seems it is not added back in scala? Also, do we need to add back this ```sparkR.init```, ```sparkRSQL.init```, ```sparkRHive.init```?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gatorsmile commented on issue #28058: [SPARK-31290][SQL][R]
Add back the deprecated R APIs
Posted by GitBox <gi...@apache.org>.
gatorsmile commented on issue #28058: [SPARK-31290][SQL][R] Add back the deprecated R APIs
URL: https://github.com/apache/spark/pull/28058#issuecomment-605391349
Also cc @mengxr @falaki Could you take a look?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28058: [SPARK-31290][R]
Add back the deprecated R APIs
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28058: [SPARK-31290][R] Add back the deprecated R APIs
URL: https://github.com/apache/spark/pull/28058#issuecomment-605740101
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120563/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gatorsmile commented on a change in pull request #28058:
[SPARK-31290][R] Add back the deprecated R APIs
Posted by GitBox <gi...@apache.org>.
gatorsmile commented on a change in pull request #28058: [SPARK-31290][R] Add back the deprecated R APIs
URL: https://github.com/apache/spark/pull/28058#discussion_r399876242
##########
File path: R/pkg/R/DataFrame.R
##########
@@ -946,6 +973,17 @@ setMethod("write.parquet",
invisible(handledCallJMethod(write, "parquet", path))
})
+#' @rdname write.parquet
+#' @name saveAsParquetFile
+#' @aliases saveAsParquetFile,SparkDataFrame,character-method
+#' @note saveAsParquetFile since 1.4.0
+setMethod("saveAsParquetFile",
Review comment:
OK. If pySpark already removed it in 2.0, I think we can get rid of saveAsParquetFile in sparkR in this release too.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #28058: [SPARK-31290][R]
Add back the deprecated R APIs
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #28058: [SPARK-31290][R] Add back the deprecated R APIs
URL: https://github.com/apache/spark/pull/28058#issuecomment-605546112
cc @marmbrus
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28058: [SPARK-31290][R]
Add back the deprecated R APIs
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28058: [SPARK-31290][R] Add back the deprecated R APIs
URL: https://github.com/apache/spark/pull/28058#issuecomment-605772873
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120567/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28058:
[SPARK-31290][SQL][R] Add back the deprecated R APIs
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28058: [SPARK-31290][SQL][R] Add back the deprecated R APIs
URL: https://github.com/apache/spark/pull/28058#issuecomment-605368966
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120517/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28058: [SPARK-31290][R]
Add back the deprecated R APIs
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28058: [SPARK-31290][R] Add back the deprecated R APIs
URL: https://github.com/apache/spark/pull/28058#issuecomment-605764074
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28058:
[SPARK-31290][R] Add back the deprecated R APIs
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28058: [SPARK-31290][R] Add back the deprecated R APIs
URL: https://github.com/apache/spark/pull/28058#issuecomment-605764074
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28058: [SPARK-31290][R]
Add back the deprecated R APIs
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28058: [SPARK-31290][R] Add back the deprecated R APIs
URL: https://github.com/apache/spark/pull/28058#issuecomment-605764080
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25271/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #28058:
[SPARK-31290][R] Add back the deprecated R APIs
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #28058: [SPARK-31290][R] Add back the deprecated R APIs
URL: https://github.com/apache/spark/pull/28058#discussion_r399743569
##########
File path: R/pkg/R/DataFrame.R
##########
@@ -946,6 +973,17 @@ setMethod("write.parquet",
invisible(handledCallJMethod(write, "parquet", path))
})
+#' @rdname write.parquet
+#' @name saveAsParquetFile
+#' @aliases saveAsParquetFile,SparkDataFrame,character-method
+#' @note saveAsParquetFile since 1.4.0
+setMethod("saveAsParquetFile",
Review comment:
`jsonFile` and `parquetFile` too.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #28058: [SPARK-31290][R]
Add back the deprecated R APIs
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #28058: [SPARK-31290][R] Add back the deprecated R APIs
URL: https://github.com/apache/spark/pull/28058#issuecomment-605763711
**[Test build #120567 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120567/testReport)** for PR 28058 at commit [`ce5969b`](https://github.com/apache/spark/commit/ce5969b95aa7de4a09fb39ca58dc8f2a7dc2f0f0).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28058:
[SPARK-31290][SQL][R] Add back the deprecated R APIs
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28058: [SPARK-31290][SQL][R] Add back the deprecated R APIs
URL: https://github.com/apache/spark/pull/28058#issuecomment-605362772
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25223/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on issue #28058:
[SPARK-31290][R] Add back the deprecated R APIs
Posted by GitBox <gi...@apache.org>.
HyukjinKwon edited a comment on issue #28058: [SPARK-31290][R] Add back the deprecated R APIs
URL: https://github.com/apache/spark/pull/28058#issuecomment-605726258
Just to clarify things, what the community agreed on is the rubric added at https://spark.apache.org/versioning-policy.html.
Some APIs listed here, `jsonFile`, `parquetFile` and `saveAsParquetFile` were removed in Scala and PySpark before Spark 3.0. I skimmed user mailing list by keyword search and nobody complained.
One last thing is, SparkR dropped RDD support. So I don't think it also makes sense to add `jsonRDD`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #28058:
[SPARK-31290][R] Add back the deprecated R APIs
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #28058: [SPARK-31290][R] Add back the deprecated R APIs
URL: https://github.com/apache/spark/pull/28058#discussion_r399744816
##########
File path: R/pkg/R/DataFrame.R
##########
@@ -946,6 +973,17 @@ setMethod("write.parquet",
invisible(handledCallJMethod(write, "parquet", path))
})
+#' @rdname write.parquet
+#' @name saveAsParquetFile
+#' @aliases saveAsParquetFile,SparkDataFrame,character-method
+#' @note saveAsParquetFile since 1.4.0
+setMethod("saveAsParquetFile",
Review comment:
Okay, using the rubric we added, in R or PySpark, it's very easy to keep the compatibility in user app - if-else based on version and redirect to the alternative. There seems direct alternative too.
Seems like we haven't seen a lot of complaints when we remove `saveAsParquetFile` at SPARK-12600. I skimmed user-mailing list with keyword search, and I found 0 complaints.
Having write API in `DataFrame` seems inconsistent with Scala and PySpark sides, and also namespace used in R side (e.g., `write.json`) - it causes a cognitive cost.
Shall we remove these one out?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #28058:
[SPARK-31290][R] Add back the deprecated R APIs
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #28058: [SPARK-31290][R] Add back the deprecated R APIs
URL: https://github.com/apache/spark/pull/28058#discussion_r399743329
##########
File path: R/pkg/R/DataFrame.R
##########
@@ -946,6 +973,17 @@ setMethod("write.parquet",
invisible(handledCallJMethod(write, "parquet", path))
})
+#' @rdname write.parquet
+#' @name saveAsParquetFile
+#' @aliases saveAsParquetFile,SparkDataFrame,character-method
+#' @note saveAsParquetFile since 1.4.0
+setMethod("saveAsParquetFile",
Review comment:
Are we sure we want to keep this too? Scala and PySpark sides don't have it either as of SPARK-12600 cc @rxin and @marmbrus
https://github.com/apache/spark/commit/77ab49b8575d2ebd678065fa70b0343d532ab9c2#diff-c3d0394b2fc08fb2842ff0362a5ac6c9
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28058:
[SPARK-31290][R] Add back the deprecated R APIs
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28058: [SPARK-31290][R] Add back the deprecated R APIs
URL: https://github.com/apache/spark/pull/28058#issuecomment-605772873
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120567/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org