You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Reynold Xin (JIRA)" <ji...@apache.org> on 2016/01/04 21:08:39 UTC
[jira] [Created] (SPARK-12625) SparkR is using deprecated APIs
Reynold Xin created SPARK-12625:
-----------------------------------
Summary: SparkR is using deprecated APIs
Key: SPARK-12625
URL: https://issues.apache.org/jira/browse/SPARK-12625
Project: Spark
Issue Type: Bug
Components: SparkR, SQL
Reporter: Reynold Xin
Priority: Blocker
See the test failure for the following pull request:
https://github.com/apache/spark/pull/10559
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48642/consoleFull
SparkR is using a lot of deprecated APIs.
{code}
========================================================================
Running SparkR tests
========================================================================
Loading required package: methods
Attaching package: 'SparkR'
The following object is masked from 'package:testthat':
describe
The following objects are masked from 'package:stats':
cov, filter, lag, na.omit, predict, sd, var
The following objects are masked from 'package:base':
colnames, colnames<-, intersect, rank, rbind, sample, subset,
summary, table, transform
SerDe functionality : ...................
functions on binary files : Re-using existing Spark Context. Please stop SparkR with sparkR.stop() or restart R to create a new Spark Context
....
binary functions : Re-using existing Spark Context. Please stop SparkR with sparkR.stop() or restart R to create a new Spark Context
...........
broadcast variables : Re-using existing Spark Context. Please stop SparkR with sparkR.stop() or restart R to create a new Spark Context
..
functions in client.R : .....
test functions in sparkR.R : Re-using existing Spark Context. Please stop SparkR with sparkR.stop() or restart R to create a new Spark Context
........Re-using existing Spark Context. Please stop SparkR with sparkR.stop() or restart R to create a new Spark Context
..........
include an external JAR in SparkContext : ..
include R packages : Re-using existing Spark Context. Please stop SparkR with sparkR.stop() or restart R to create a new Spark Context
MLlib functions : Re-using existing Spark Context. Please stop SparkR with sparkR.stop() or restart R to create a new Spark Context
............
parallelize() and collect() : Re-using existing Spark Context. Please stop SparkR with sparkR.stop() or restart R to create a new Spark Context
.............................
basic RDD functions : Re-using existing Spark Context. Please stop SparkR with sparkR.stop() or restart R to create a new Spark Context
...........................................................................................................................................................................................................................................................................................................................................................................................................................................
partitionBy, groupByKey, reduceByKey etc. : Re-using existing Spark Context. Please stop SparkR with sparkR.stop() or restart R to create a new Spark Context
....................
SparkSQL functions : Re-using existing Spark Context. Please stop SparkR with sparkR.stop() or restart R to create a new Spark Context
.......................................................1...................................................................................2.3....4......................................................................................................................5............6.....................................................................................................................................7............................................................................8........................................................................
tests RDD function take() : Re-using existing Spark Context. Please stop SparkR with sparkR.stop() or restart R to create a new Spark Context
................
the textFile() function : Re-using existing Spark Context. Please stop SparkR with sparkR.stop() or restart R to create a new Spark Context
.............
functions in utils.R : Re-using existing Spark Context. Please stop SparkR with sparkR.stop() or restart R to create a new Spark Context
.......................
1. Error: create DataFrame from RDD --------------------------------------------
1: withCallingHandlers(eval(code, new_test_environment), error = capture_calls, message = function(c) invokeRestart("muffleMessage"))
2: eval(code, new_test_environment)
3: eval(expr, envir, enclos)
4: insertInto(df, "people") at test_sparkSQL.R:174
5: insertInto(df, "people")
6: .local(x, tableName, ...)
7: callJMethod(x@sdf, "insertInto", tableName, overwrite)
8: invokeJava(isStatic = FALSE, objId$id, methodName, ...)
9: stop(readString(conn))
2. Error: read/write json files ------------------------------------------------
1: withCallingHandlers(eval(code, new_test_environment), error = capture_calls, message = function(c) invokeRestart("muffleMessage"))
2: eval(code, new_test_environment)
3: eval(expr, envir, enclos)
4: write.df(df, jsonPath2, "json", mode = "overwrite") at test_sparkSQL.R:404
5: write.df(df, jsonPath2, "json", mode = "overwrite")
6: .local(df, path, ...)
7: callJMethod(df@sdf, "save", source, jmode, options)
8: invokeJava(isStatic = FALSE, objId$id, methodName, ...)
9: stop(readString(conn))
3. Error: jsonRDD() on a RDD with json string ----------------------------------
1: withCallingHandlers(eval(code, new_test_environment), error = capture_calls, message = function(c) invokeRestart("muffleMessage"))
2: eval(code, new_test_environment)
3: eval(expr, envir, enclos)
4: jsonRDD(sqlContext, rdd) at test_sparkSQL.R:426
5: callJMethod(sqlContext, "jsonRDD", callJMethod(getJRDD(rdd), "rdd"), samplingRatio)
6: invokeJava(isStatic = FALSE, objId$id, methodName, ...)
7: stop(readString(conn))
4. Error: insertInto() on a registered table -----------------------------------
1: withCallingHandlers(eval(code, new_test_environment), error = capture_calls, message = function(c) invokeRestart("muffleMessage"))
2: eval(code, new_test_environment)
3: eval(expr, envir, enclos)
4: write.df(df, parquetPath, "parquet", "overwrite") at test_sparkSQL.R:465
5: write.df(df, parquetPath, "parquet", "overwrite")
6: .local(df, path, ...)
7: callJMethod(df@sdf, "save", source, jmode, options)
8: invokeJava(isStatic = FALSE, objId$id, methodName, ...)
9: stop(readString(conn))
5. Error: subsetting -----------------------------------------------------------
error in evaluating the argument 'i' in selecting a method for function '[':
Calls: %in% -> %in% -> callJMethod -> invokeJava
1: withCallingHandlers(eval(code, new_test_environment), error = capture_calls, message = function(c) invokeRestart("muffleMessage"))
2: eval(code, new_test_environment)
3: eval(expr, envir, enclos)
4: df[df$age %in% c(19, 30), 1:2] at test_sparkSQL.R:844
6. Error: test HiveContext -----------------------------------------------------
1: withCallingHandlers(eval(code, new_test_environment), error = capture_calls, message = function(c) invokeRestart("muffleMessage"))
2: eval(code, new_test_environment)
3: eval(expr, envir, enclos)
4: saveAsTable(df, "json2", "json", "append", path = jsonPath2) at test_sparkSQL.R:905
5: saveAsTable(df, "json2", "json", "append", path = jsonPath2)
6: callJMethod(df@sdf, "saveAsTable", tableName, source, jmode, options)
7: invokeJava(isStatic = FALSE, objId$id, methodName, ...)
8: stop(readString(conn))
7. Error: filter() on a DataFrame ----------------------------------------------
error in evaluating the argument 'condition' in selecting a method for function 'where':
Calls: %in% -> %in% -> callJMethod -> invokeJava
1: withCallingHandlers(eval(code, new_test_environment), error = capture_calls, message = function(c) invokeRestart("muffleMessage"))
2: eval(code, new_test_environment)
3: eval(expr, envir, enclos)
4: where(df, df$age %in% c(19)) at test_sparkSQL.R:1260
8. Error: read/write Parquet files ---------------------------------------------
1: withCallingHandlers(eval(code, new_test_environment), error = capture_calls, message = function(c) invokeRestart("muffleMessage"))
2: eval(code, new_test_environment)
3: eval(expr, envir, enclos)
4: write.df(df, parquetPath, "parquet", mode = "overwrite") at test_sparkSQL.R:1472
5: write.df(df, parquetPath, "parquet", mode = "overwrite")
6: .local(df, path, ...)
7: callJMethod(df@sdf, "save", source, jmode, options)
8: invokeJava(isStatic = FALSE, objId$id, methodName, ...)
9: stop(readString(conn))
Error: Test failures
Execution halted
Loading required package: methods
Attaching package: 'SparkR'
The following object is masked from 'package:testthat':
describe
The following objects are masked from 'package:stats':
cov, filter, lag, na.omit, predict, sd, var
The following objects are masked from 'package:base':
colnames, colnames<-, intersect, rank, rbind, sample, subset,
summary, table, transform
SerDe functionality : ...................
functions on binary files : Re-using existing Spark Context. Please stop SparkR with sparkR.stop() or restart R to create a new Spark Context
....
binary functions : Re-using existing Spark Context. Please stop SparkR with sparkR.stop() or restart R to create a new Spark Context
...........
broadcast variables : Re-using existing Spark Context. Please stop SparkR with sparkR.stop() or restart R to create a new Spark Context
..
functions in client.R : .....
test functions in sparkR.R : Re-using existing Spark Context. Please stop SparkR with sparkR.stop() or restart R to create a new Spark Context
........Re-using existing Spark Context. Please stop SparkR with sparkR.stop() or restart R to create a new Spark Context
..........
include an external JAR in SparkContext : ..
include R packages : Re-using existing Spark Context. Please stop SparkR with sparkR.stop() or restart R to create a new Spark Context
MLlib functions : Re-using existing Spark Context. Please stop SparkR with sparkR.stop() or restart R to create a new Spark Context
............
parallelize() and collect() : Re-using existing Spark Context. Please stop SparkR with sparkR.stop() or restart R to create a new Spark Context
.............................
basic RDD functions : Re-using existing Spark Context. Please stop SparkR with sparkR.stop() or restart R to create a new Spark Context
...........................................................................................................................................................................................................................................................................................................................................................................................................................................
partitionBy, groupByKey, reduceByKey etc. : Re-using existing Spark Context. Please stop SparkR with sparkR.stop() or restart R to create a new Spark Context
....................
SparkSQL functions : Re-using existing Spark Context. Please stop SparkR with sparkR.stop() or restart R to create a new Spark Context
.......................................................1...................................................................................2.3....4......................................................................................................................5............6.....................................................................................................................................7............................................................................8........................................................................
tests RDD function take() : Re-using existing Spark Context. Please stop SparkR with sparkR.stop() or restart R to create a new Spark Context
................
the textFile() function : Re-using existing Spark Context. Please stop SparkR with sparkR.stop() or restart R to create a new Spark Context
.............
functions in utils.R : Re-using existing Spark Context. Please stop SparkR with sparkR.stop() or restart R to create a new Spark Context
.......................
1. Error: create DataFrame from RDD --------------------------------------------
1: withCallingHandlers(eval(code, new_test_environment), error = capture_calls, message = function(c) invokeRestart("muffleMessage"))
2: eval(code, new_test_environment)
3: eval(expr, envir, enclos)
4: insertInto(df, "people") at test_sparkSQL.R:174
5: insertInto(df, "people")
6: .local(x, tableName, ...)
7: callJMethod(x@sdf, "insertInto", tableName, overwrite)
8: invokeJava(isStatic = FALSE, objId$id, methodName, ...)
9: stop(readString(conn))
2. Error: read/write json files ------------------------------------------------
1: withCallingHandlers(eval(code, new_test_environment), error = capture_calls, message = function(c) invokeRestart("muffleMessage"))
2: eval(code, new_test_environment)
3: eval(expr, envir, enclos)
4: write.df(df, jsonPath2, "json", mode = "overwrite") at test_sparkSQL.R:404
5: write.df(df, jsonPath2, "json", mode = "overwrite")
6: .local(df, path, ...)
7: callJMethod(df@sdf, "save", source, jmode, options)
8: invokeJava(isStatic = FALSE, objId$id, methodName, ...)
9: stop(readString(conn))
3. Error: jsonRDD() on a RDD with json string ----------------------------------
1: withCallingHandlers(eval(code, new_test_environment), error = capture_calls, message = function(c) invokeRestart("muffleMessage"))
2: eval(code, new_test_environment)
3: eval(expr, envir, enclos)
4: jsonRDD(sqlContext, rdd) at test_sparkSQL.R:426
5: callJMethod(sqlContext, "jsonRDD", callJMethod(getJRDD(rdd), "rdd"), samplingRatio)
6: invokeJava(isStatic = FALSE, objId$id, methodName, ...)
7: stop(readString(conn))
4. Error: insertInto() on a registered table -----------------------------------
1: withCallingHandlers(eval(code, new_test_environment), error = capture_calls, message = function(c) invokeRestart("muffleMessage"))
2: eval(code, new_test_environment)
3: eval(expr, envir, enclos)
4: write.df(df, parquetPath, "parquet", "overwrite") at test_sparkSQL.R:465
5: write.df(df, parquetPath, "parquet", "overwrite")
6: .local(df, path, ...)
7: callJMethod(df@sdf, "save", source, jmode, options)
8: invokeJava(isStatic = FALSE, objId$id, methodName, ...)
9: stop(readString(conn))
5. Error: subsetting -----------------------------------------------------------
error in evaluating the argument 'i' in selecting a method for function '[':
Calls: %in% -> %in% -> callJMethod -> invokeJava
1: withCallingHandlers(eval(code, new_test_environment), error = capture_calls, message = function(c) invokeRestart("muffleMessage"))
2: eval(code, new_test_environment)
3: eval(expr, envir, enclos)
4: df[df$age %in% c(19, 30), 1:2] at test_sparkSQL.R:844
6. Error: test HiveContext -----------------------------------------------------
1: withCallingHandlers(eval(code, new_test_environment), error = capture_calls, message = function(c) invokeRestart("muffleMessage"))
2: eval(code, new_test_environment)
3: eval(expr, envir, enclos)
4: saveAsTable(df, "json2", "json", "append", path = jsonPath2) at test_sparkSQL.R:905
5: saveAsTable(df, "json2", "json", "append", path = jsonPath2)
6: callJMethod(df@sdf, "saveAsTable", tableName, source, jmode, options)
7: invokeJava(isStatic = FALSE, objId$id, methodName, ...)
8: stop(readString(conn))
7. Error: filter() on a DataFrame ----------------------------------------------
error in evaluating the argument 'condition' in selecting a method for function 'where':
Calls: %in% -> %in% -> callJMethod -> invokeJava
1: withCallingHandlers(eval(code, new_test_environment), error = capture_calls, message = function(c) invokeRestart("muffleMessage"))
2: eval(code, new_test_environment)
3: eval(expr, envir, enclos)
4: where(df, df$age %in% c(19)) at test_sparkSQL.R:1260
8. Error: read/write Parquet files ---------------------------------------------
1: withCallingHandlers(eval(code, new_test_environment), error = capture_calls, message = function(c) invokeRestart("muffleMessage"))
2: eval(code, new_test_environment)
3: eval(expr, envir, enclos)
4: write.df(df, parquetPath, "parquet", mode = "overwrite") at test_sparkSQL.R:1472
5: write.df(df, parquetPath, "parquet", mode = "overwrite")
6: .local(df, path, ...)
7: callJMethod(df@sdf, "save", source, jmode, options)
8: invokeJava(isStatic = FALSE, objId$id, methodName, ...)
9: stop(readString(conn))
Error: Test failures
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org