You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "obfuscated_dvlper (Jira)" <ji...@apache.org> on 2021/05/03 18:40:00 UTC
[jira] [Commented] (SPARK-34791) SparkR throws node stack overflow
[ https://issues.apache.org/jira/browse/SPARK-34791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17338549#comment-17338549 ]
obfuscated_dvlper commented on SPARK-34791:
-------------------------------------------
sorry about reporting this late. I tried with Apache Spark 3.1.1 but still same issue. I didn't update R yet, still using R-4.0.2 I will attempt to try with R-4.0.3 and share my findings soon.
{code:java}
> source('sample.R')
> myclsr = myclosure_func()
> myclsr$get_some_date('2021-01-01')
[1] "2020-12-31"
>
> ## spark.lapply throws node stack overflow
> result = spark.lapply(c('2021-01-01', '2021-01-02'), function (rdate) {
+ source('sample.R')
+ another_closure = myclosure_func()
+ return(another_closure$get_some_date(rdate))
+ })
Error: node stack overflow
>
> sessionInfo()
R version 4.0.2 (2020-06-22)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: CentOS Linux 7 (Core)
Matrix products: default
BLAS: /../R/R-4.0.2/lib/libRblas.so
LAPACK: /../R/R-4.0.2/lib/libRlapack.so
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] SparkR_3.1.1
loaded via a namespace (and not attached):
[1] compiler_4.0.2
>
{code}
> SparkR throws node stack overflow
> ---------------------------------
>
> Key: SPARK-34791
> URL: https://issues.apache.org/jira/browse/SPARK-34791
> Project: Spark
> Issue Type: Question
> Components: SparkR
> Affects Versions: 3.0.1
> Reporter: obfuscated_dvlper
> Priority: Major
>
> SparkR throws "node stack overflow" error upon running code (sample below) on R-4.0.2 with Spark 3.0.1.
> Same piece of code works on R-3.3.3 with Spark 2.2.1 (& SparkR 2.4.5)
> {code:java}
> source('sample.R')
> myclsr = myclosure_func()
> myclsr$get_some_date('2021-01-01')
> ## spark.lapply throws node stack overflow
> result = spark.lapply(c('2021-01-01', '2021-01-02'), function (rdate) {
> source('sample.R')
> another_closure = myclosure_func()
> return(another_closure$get_some_date(rdate))
> })
> {code}
> Sample.R
> {code:java}
> ## util function, which calls itself
> getPreviousBusinessDate <- function(asofdate) {
> asdt <- asofdate;
> asdt <- as.Date(asofdate)-1;
> wd <- format(as.Date(asdt),"%A")
> if(wd == "Saturday" | wd == "Sunday") {
> return (getPreviousBusinessDate(asdt));
> }
> return (asdt);
> }
> ## closure which calls util function
> myclosure_func = function() {
> myclosure = list()
> get_some_date = function (random_date) {
> return (getPreviousBusinessDate(random_date))
> }
> myclosure$get_some_date = get_some_date
> return(myclosure)
> }
> {code}
> This seems to have caused by sourcing sample.R twice. Once before invoking Spark session and another within Spark session.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org