You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Hyukjin Kwon <gu...@gmail.com> on 2018/11/02 02:34:39 UTC
[discuss] SparkR CRAN feasibility check server problem
Hi all,
I want to raise the CRAN failure issue because it started to block Spark
PRs time to time. Since the number
of PRs grows hugely in Spark community, this is critical to not block other
PRs.
There has been a problem at CRAN (See
https://github.com/apache/spark/pull/20005 for analysis).
To cut it short, the root cause is malformed package info from
https://cran.r-project.org/src/contrib/PACKAGES
from server side, and this had to be fixed by requesting it to CRAN
sysaadmin's help.
https://issues.apache.org/jira/browse/SPARK-24152 <- newly open. I am
pretty sure it's the same issue
https://issues.apache.org/jira/browse/SPARK-25923 <- reopen/resolved 2 times
https://issues.apache.org/jira/browse/SPARK-22812
This happened 5 times for roughly about 10 months, causing blocking almost
all PRs in Apache Spark.
Historically, it blocked whole PRs for few days once, and whole Spark
community had to stop working.
I assume this has been not a super big big issue so far for other projects
or other people because apparently
higher version of R has some logics to handle this malformed documents (at
least I verified R 3.4.0 works fine).
For our side, Jenkins has low R version (R 3.1.1 if that's not updated from
what I have seen before),
which is unable to parse the malformed server's response.
So, I want to talk about how we are going to handle this. Possible
solutions are:
1. We should start a talk with CRAN sysadmin to permanently prevent this
issue
2. We upgrade R to 3.4.0 in Jenkins (however we will not be able to test
low R versions)
3. ...
If if we fine, I would like to suggest to forward this email to CRAN
sysadmin to discuss further about this.
Adding Liang-Chi Felix and Shivaram who I already talked about this few
times before.
Thanks all.
Re: [discuss] SparkR CRAN feasibility check server problem
Posted by Liang-Chi Hsieh <vi...@gmail.com>.
Just got reply from CRAN admin. It should be fixed now.
Hyukjin Kwon wrote
> Thanks, Liang-chi.
>
> On Thu, 13 Dec 2018, 8:29 am Liang-Chi Hsieh <
> viirya@
> wrote:
>
> >
>> Sorry for late. There is a malformed record at CRAN package page again.
>> I've
>> already asked CRAN admin for help. It should be fixed soon according to
>> past
>> experience.
>>
>> Related discussion will be in
>> https://issues.apache.org/jira/browse/SPARK-24152. I will post here if I
>> get
>> reply from CRAN admin.
>>
>> Thanks.
>>
>>
>> Liang-Chi Hsieh wrote
>> > Thanks for letting me know! I will look into it and ask CRAN admin for
>> > help.
>> >
>> >
>> > Hyukjin Kwon wrote
>> >> Looks it's happening again. Liang-Chi, do you mind if I ask it again?
>> >>
>> >> FYI, R 3.4 is officially deprecated as of
>> >> https://github.com/apache/spark/pull/23012
>> >> We could upgrade R version to 3.4.x in Jenkins, which deals with the
>> >> malformed(?) responses after 3.0 release.
>> >> Then, we could get rid of this problem..!
>> >>
>> >> 2018년 11월 12일 (월) 오후 1:47, Hyukjin Kwon <
>> >
>> >> gurwls223@
>> >
>> >> >님이 작성:
>> >>
>> >>> I made a PR to officially drop R prior to version 3.4 (
>> >>> https://github.com/apache/spark/pull/23012).
>> >>> The tests will probably fail for now since it produces warnings for
>> >>> using
>> >>> R 3.1.x.
>> >>>
>> >>> 2018년 11월 11일 (일) 오전 3:00, Felix Cheung <
>> >
>> >> felixcheung_m@
>> >
>> >> >님이 작성:
>> >>>
>> >>>> It’s a great point about min R version. From what I see, mostly
>> because
>> >>>> of fixes and packages support, most users of R are fairly up to
>> date?
>> >>>> So
>> >>>> perhaps 3.4 as min version is reasonable esp. for Spark 3.
>> >>>>
>> >>>> Are we getting traction with CRAN sysadmin? It seems like this has
>> been
>> >>>> broken a few times.
>> >>>>
>> >>>>
>> >>>> ------------------------------
>> >>>> *From:* Liang-Chi Hsieh <
>> >
>> >> viirya@
>> >
>> >> >
>> >>>> *Sent:* Saturday, November 10, 2018 2:32 AM
>> >>>> *To:*
>> >
>> >> dev@.apache
>> >
>> >>>> *Subject:* Re: [discuss] SparkR CRAN feasibility check server
>> problem
>> >>>>
>> >>>>
>> >>>> Yeah, thanks Hyukjin Kwon for bringing this up for discussion.
>> >>>>
>> >>>> I don't know how higher versions of R are widely used across R
>> >>>> community.
>> >>>> If
>> >>>> R version 3.1.x was not very commonly used, I think we can discuss
>> to
>> >>>> upgrade minimum R version in next Spark version.
>> >>>>
>> >>>> If we ended up with not upgrading, we can discuss with CRAN sysadmin
>> to
>> >>>> fix
>> >>>> it by the service side automatically that prevents malformed R
>> packages
>> >>>> info. So we don't need to fix it manually every time.
>> >>>>
>> >>>>
>> >>>>
>> >>>> Hyukjin Kwon wrote
>> >>>> >> Can upgrading R able to fix the issue. Is this perhaps not
>> >>>> necessarily
>> >>>> > malform but some new format for new versions perhaps?
>> >>>> > That's my guess. I am not totally sure about it tho.
>> >>>> >
>> >>>> >> Anyway we should consider upgrading R version if that fixes the
>> >>>> problem.
>> >>>> > Yea, we should. If we should, it should be more them R 3.4. Maybe
>> >>>> it's
>> >>>> > good
>> >>>> > time to start to talk about minimum R version. 3.1.x is too old.
>> It's
>> >>>> > released 4.5 years ago.
>> >>>> > R 3.4.0 is released 1.5 years ago. Considering the timing for
>> Spark
>> >>>> 3.0,
>> >>>> > deprecating lower versions, bumping up R to 3.4 might be
>> reasonable
>> >>>> > option.
>> >>>> >
>> >>>> > Adding Shane as well.
>> >>>> >
>> >>>> > If we ended up with not upgrading it, I will forward this email to
>> >>>> CRAN
>> >>>> > sysadmin to discuss further anyway.
>> >>>> >
>> >>>> >
>> >>>> >
>> >>>> > 2018년 11월 2일 (금) 오후 12:51, Felix Cheung <
>> >>>>
>> >>>> > felixcheung@
>> >>>>
>> >>>> > >님이 작성:
>> >>>> >
>> >>>> >> Thanks for being this up and much appreciate with keeping on top
>> of
>> >>>> this
>> >>>> >> at all times.
>> >>>> >>
>> >>>> >> Can upgrading R able to fix the issue. Is this perhaps not
>> >>>> necessarily
>> >>>> >> malform but some new format for new versions perhaps? Anyway we
>> >>>> should
>> >>>> >> consider upgrading R version if that fixes the problem.
>> >>>> >>
>> >>>> >> As an option we could also disable the repo check in Jenkins but
>> I
>> >>>> can
>> >>>> >> see
>> >>>> >> that could also be problematic.
>> >>>> >>
>> >>>> >>
>> >>>> >> On Thu, Nov 1, 2018 at 7:35 PM Hyukjin Kwon <
>> >>>>
>> >>>> > gurwls223@
>> >>>>
>> >>>> > > wrote:
>> >>>> >>
>> >>>> >>> Hi all,
>> >>>> >>>
>> >>>> >>> I want to raise the CRAN failure issue because it started to
>> block
>> >>>> Spark
>> >>>> >>> PRs time to time. Since the number
>> >>>> >>> of PRs grows hugely in Spark community, this is critical to not
>> >>>> block
>> >>>> >>> other PRs.
>> >>>> >>>
>> >>>> >>> There has been a problem at CRAN (See
>> >>>> >>> https://github.com/apache/spark/pull/20005 for analysis).
>> >>>> >>> To cut it short, the root cause is malformed package info from
>> >>>> >>> https://cran.r-project.org/src/contrib/PACKAGES
>> >>>> >>> from server side, and this had to be fixed by requesting it to
>> CRAN
>> >>>> >>> sysaadmin's help.
>> >>>> >>>
>> >>>> >>> https://issues.apache.org/jira/browse/SPARK-24152 <- newly open.
>> I
>> >>>> am
>> >>>> >>> pretty sure it's the same issue
>> >>>> >>> https://issues.apache.org/jira/browse/SPARK-25923 <-
>> >>>> reopen/resolved
>> >>>> 2
>> >>>> >>> times
>> >>>> >>> https://issues.apache.org/jira/browse/SPARK-22812
>> >>>> >>>
>> >>>> >>> This happened 5 times for roughly about 10 months, causing
>> blocking
>> >>>> >>> almost all PRs in Apache Spark.
>> >>>> >>> Historically, it blocked whole PRs for few days once, and whole
>> >>>> Spark
>> >>>> >>> community had to stop working.
>> >>>> >>>
>> >>>> >>> I assume this has been not a super big big issue so far for
>> other
>> >>>> >>> projects or other people because apparently
>> >>>> >>> higher version of R has some logics to handle this malformed
>> >>>> documents
>> >>>> >>> (at least I verified R 3.4.0 works fine).
>> >>>> >>>
>> >>>> >>> For our side, Jenkins has low R version (R 3.1.1 if that's not
>> >>>> updated
>> >>>> >>> from what I have seen before),
>> >>>> >>> which is unable to parse the malformed server's response.
>> >>>> >>>
>> >>>> >>> So, I want to talk about how we are going to handle this.
>> Possible
>> >>>> >>> solutions are:
>> >>>> >>>
>> >>>> >>> 1. We should start a talk with CRAN sysadmin to permanently
>> prevent
>> >>>> this
>> >>>> >>> issue
>> >>>> >>> 2. We upgrade R to 3.4.0 in Jenkins (however we will not be able
>> to
>> >>>> test
>> >>>> >>> low R versions)
>> >>>> >>> 3. ...
>> >>>> >>>
>> >>>> >>> If if we fine, I would like to suggest to forward this email to
>> >>>> CRAN
>> >>>> >>> sysadmin to discuss further about this.
>> >>>> >>>
>> >>>> >>> Adding Liang-Chi Felix and Shivaram who I already talked about
>> this
>> >>>> few
>> >>>> >>> times before.
>> >>>> >>>
>> >>>> >>> Thanks all.
>> >>>> >>>
>> >>>> >>>
>> >>>> >>>
>> >>>> >>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>> --
>> >>>> Sent from:
>> http://apache-spark-developers-list.1001551.n3.nabble.com/
>> >>>>
>> >>>>
>> ---------------------------------------------------------------------
>> >>>> To unsubscribe e-mail:
>> >
>> >> dev-unsubscribe@.apache
>> >
>> >>>>
>> >>>>
>> >
>> >
>> >
>> >
>> >
>> > --
>> > Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe e-mail:
>>
>> > dev-unsubscribe@.apache
>>
>>
>>
>>
>>
>> --
>> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail:
> dev-unsubscribe@.apache
>>
>>
--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: [discuss] SparkR CRAN feasibility check server problem
Posted by Hyukjin Kwon <gu...@gmail.com>.
Thanks, Liang-chi.
On Thu, 13 Dec 2018, 8:29 am Liang-Chi Hsieh <viirya@gmail.com wrote:
>
> Sorry for late. There is a malformed record at CRAN package page again.
> I've
> already asked CRAN admin for help. It should be fixed soon according to
> past
> experience.
>
> Related discussion will be in
> https://issues.apache.org/jira/browse/SPARK-24152. I will post here if I
> get
> reply from CRAN admin.
>
> Thanks.
>
>
> Liang-Chi Hsieh wrote
> > Thanks for letting me know! I will look into it and ask CRAN admin for
> > help.
> >
> >
> > Hyukjin Kwon wrote
> >> Looks it's happening again. Liang-Chi, do you mind if I ask it again?
> >>
> >> FYI, R 3.4 is officially deprecated as of
> >> https://github.com/apache/spark/pull/23012
> >> We could upgrade R version to 3.4.x in Jenkins, which deals with the
> >> malformed(?) responses after 3.0 release.
> >> Then, we could get rid of this problem..!
> >>
> >> 2018년 11월 12일 (월) 오후 1:47, Hyukjin Kwon <
> >
> >> gurwls223@
> >
> >> >님이 작성:
> >>
> >>> I made a PR to officially drop R prior to version 3.4 (
> >>> https://github.com/apache/spark/pull/23012).
> >>> The tests will probably fail for now since it produces warnings for
> >>> using
> >>> R 3.1.x.
> >>>
> >>> 2018년 11월 11일 (일) 오전 3:00, Felix Cheung <
> >
> >> felixcheung_m@
> >
> >> >님이 작성:
> >>>
> >>>> It’s a great point about min R version. From what I see, mostly
> because
> >>>> of fixes and packages support, most users of R are fairly up to date?
> >>>> So
> >>>> perhaps 3.4 as min version is reasonable esp. for Spark 3.
> >>>>
> >>>> Are we getting traction with CRAN sysadmin? It seems like this has
> been
> >>>> broken a few times.
> >>>>
> >>>>
> >>>> ------------------------------
> >>>> *From:* Liang-Chi Hsieh <
> >
> >> viirya@
> >
> >> >
> >>>> *Sent:* Saturday, November 10, 2018 2:32 AM
> >>>> *To:*
> >
> >> dev@.apache
> >
> >>>> *Subject:* Re: [discuss] SparkR CRAN feasibility check server problem
> >>>>
> >>>>
> >>>> Yeah, thanks Hyukjin Kwon for bringing this up for discussion.
> >>>>
> >>>> I don't know how higher versions of R are widely used across R
> >>>> community.
> >>>> If
> >>>> R version 3.1.x was not very commonly used, I think we can discuss to
> >>>> upgrade minimum R version in next Spark version.
> >>>>
> >>>> If we ended up with not upgrading, we can discuss with CRAN sysadmin
> to
> >>>> fix
> >>>> it by the service side automatically that prevents malformed R
> packages
> >>>> info. So we don't need to fix it manually every time.
> >>>>
> >>>>
> >>>>
> >>>> Hyukjin Kwon wrote
> >>>> >> Can upgrading R able to fix the issue. Is this perhaps not
> >>>> necessarily
> >>>> > malform but some new format for new versions perhaps?
> >>>> > That's my guess. I am not totally sure about it tho.
> >>>> >
> >>>> >> Anyway we should consider upgrading R version if that fixes the
> >>>> problem.
> >>>> > Yea, we should. If we should, it should be more them R 3.4. Maybe
> >>>> it's
> >>>> > good
> >>>> > time to start to talk about minimum R version. 3.1.x is too old.
> It's
> >>>> > released 4.5 years ago.
> >>>> > R 3.4.0 is released 1.5 years ago. Considering the timing for Spark
> >>>> 3.0,
> >>>> > deprecating lower versions, bumping up R to 3.4 might be reasonable
> >>>> > option.
> >>>> >
> >>>> > Adding Shane as well.
> >>>> >
> >>>> > If we ended up with not upgrading it, I will forward this email to
> >>>> CRAN
> >>>> > sysadmin to discuss further anyway.
> >>>> >
> >>>> >
> >>>> >
> >>>> > 2018년 11월 2일 (금) 오후 12:51, Felix Cheung <
> >>>>
> >>>> > felixcheung@
> >>>>
> >>>> > >님이 작성:
> >>>> >
> >>>> >> Thanks for being this up and much appreciate with keeping on top of
> >>>> this
> >>>> >> at all times.
> >>>> >>
> >>>> >> Can upgrading R able to fix the issue. Is this perhaps not
> >>>> necessarily
> >>>> >> malform but some new format for new versions perhaps? Anyway we
> >>>> should
> >>>> >> consider upgrading R version if that fixes the problem.
> >>>> >>
> >>>> >> As an option we could also disable the repo check in Jenkins but I
> >>>> can
> >>>> >> see
> >>>> >> that could also be problematic.
> >>>> >>
> >>>> >>
> >>>> >> On Thu, Nov 1, 2018 at 7:35 PM Hyukjin Kwon <
> >>>>
> >>>> > gurwls223@
> >>>>
> >>>> > > wrote:
> >>>> >>
> >>>> >>> Hi all,
> >>>> >>>
> >>>> >>> I want to raise the CRAN failure issue because it started to block
> >>>> Spark
> >>>> >>> PRs time to time. Since the number
> >>>> >>> of PRs grows hugely in Spark community, this is critical to not
> >>>> block
> >>>> >>> other PRs.
> >>>> >>>
> >>>> >>> There has been a problem at CRAN (See
> >>>> >>> https://github.com/apache/spark/pull/20005 for analysis).
> >>>> >>> To cut it short, the root cause is malformed package info from
> >>>> >>> https://cran.r-project.org/src/contrib/PACKAGES
> >>>> >>> from server side, and this had to be fixed by requesting it to
> CRAN
> >>>> >>> sysaadmin's help.
> >>>> >>>
> >>>> >>> https://issues.apache.org/jira/browse/SPARK-24152 <- newly open.
> I
> >>>> am
> >>>> >>> pretty sure it's the same issue
> >>>> >>> https://issues.apache.org/jira/browse/SPARK-25923 <-
> >>>> reopen/resolved
> >>>> 2
> >>>> >>> times
> >>>> >>> https://issues.apache.org/jira/browse/SPARK-22812
> >>>> >>>
> >>>> >>> This happened 5 times for roughly about 10 months, causing
> blocking
> >>>> >>> almost all PRs in Apache Spark.
> >>>> >>> Historically, it blocked whole PRs for few days once, and whole
> >>>> Spark
> >>>> >>> community had to stop working.
> >>>> >>>
> >>>> >>> I assume this has been not a super big big issue so far for other
> >>>> >>> projects or other people because apparently
> >>>> >>> higher version of R has some logics to handle this malformed
> >>>> documents
> >>>> >>> (at least I verified R 3.4.0 works fine).
> >>>> >>>
> >>>> >>> For our side, Jenkins has low R version (R 3.1.1 if that's not
> >>>> updated
> >>>> >>> from what I have seen before),
> >>>> >>> which is unable to parse the malformed server's response.
> >>>> >>>
> >>>> >>> So, I want to talk about how we are going to handle this. Possible
> >>>> >>> solutions are:
> >>>> >>>
> >>>> >>> 1. We should start a talk with CRAN sysadmin to permanently
> prevent
> >>>> this
> >>>> >>> issue
> >>>> >>> 2. We upgrade R to 3.4.0 in Jenkins (however we will not be able
> to
> >>>> test
> >>>> >>> low R versions)
> >>>> >>> 3. ...
> >>>> >>>
> >>>> >>> If if we fine, I would like to suggest to forward this email to
> >>>> CRAN
> >>>> >>> sysadmin to discuss further about this.
> >>>> >>>
> >>>> >>> Adding Liang-Chi Felix and Shivaram who I already talked about
> this
> >>>> few
> >>>> >>> times before.
> >>>> >>>
> >>>> >>> Thanks all.
> >>>> >>>
> >>>> >>>
> >>>> >>>
> >>>> >>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe e-mail:
> >
> >> dev-unsubscribe@.apache
> >
> >>>>
> >>>>
> >
> >
> >
> >
> >
> > --
> > Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
> >
> > ---------------------------------------------------------------------
> > To unsubscribe e-mail:
>
> > dev-unsubscribe@.apache
>
>
>
>
>
> --
> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
Re: [discuss] SparkR CRAN feasibility check server problem
Posted by Liang-Chi Hsieh <vi...@gmail.com>.
Sorry for late. There is a malformed record at CRAN package page again. I've
already asked CRAN admin for help. It should be fixed soon according to past
experience.
Related discussion will be in
https://issues.apache.org/jira/browse/SPARK-24152. I will post here if I get
reply from CRAN admin.
Thanks.
Liang-Chi Hsieh wrote
> Thanks for letting me know! I will look into it and ask CRAN admin for
> help.
>
>
> Hyukjin Kwon wrote
>> Looks it's happening again. Liang-Chi, do you mind if I ask it again?
>>
>> FYI, R 3.4 is officially deprecated as of
>> https://github.com/apache/spark/pull/23012
>> We could upgrade R version to 3.4.x in Jenkins, which deals with the
>> malformed(?) responses after 3.0 release.
>> Then, we could get rid of this problem..!
>>
>> 2018년 11월 12일 (월) 오후 1:47, Hyukjin Kwon <
>
>> gurwls223@
>
>> >님이 작성:
>>
>>> I made a PR to officially drop R prior to version 3.4 (
>>> https://github.com/apache/spark/pull/23012).
>>> The tests will probably fail for now since it produces warnings for
>>> using
>>> R 3.1.x.
>>>
>>> 2018년 11월 11일 (일) 오전 3:00, Felix Cheung <
>
>> felixcheung_m@
>
>> >님이 작성:
>>>
>>>> It’s a great point about min R version. From what I see, mostly because
>>>> of fixes and packages support, most users of R are fairly up to date?
>>>> So
>>>> perhaps 3.4 as min version is reasonable esp. for Spark 3.
>>>>
>>>> Are we getting traction with CRAN sysadmin? It seems like this has been
>>>> broken a few times.
>>>>
>>>>
>>>> ------------------------------
>>>> *From:* Liang-Chi Hsieh <
>
>> viirya@
>
>> >
>>>> *Sent:* Saturday, November 10, 2018 2:32 AM
>>>> *To:*
>
>> dev@.apache
>
>>>> *Subject:* Re: [discuss] SparkR CRAN feasibility check server problem
>>>>
>>>>
>>>> Yeah, thanks Hyukjin Kwon for bringing this up for discussion.
>>>>
>>>> I don't know how higher versions of R are widely used across R
>>>> community.
>>>> If
>>>> R version 3.1.x was not very commonly used, I think we can discuss to
>>>> upgrade minimum R version in next Spark version.
>>>>
>>>> If we ended up with not upgrading, we can discuss with CRAN sysadmin to
>>>> fix
>>>> it by the service side automatically that prevents malformed R packages
>>>> info. So we don't need to fix it manually every time.
>>>>
>>>>
>>>>
>>>> Hyukjin Kwon wrote
>>>> >> Can upgrading R able to fix the issue. Is this perhaps not
>>>> necessarily
>>>> > malform but some new format for new versions perhaps?
>>>> > That's my guess. I am not totally sure about it tho.
>>>> >
>>>> >> Anyway we should consider upgrading R version if that fixes the
>>>> problem.
>>>> > Yea, we should. If we should, it should be more them R 3.4. Maybe
>>>> it's
>>>> > good
>>>> > time to start to talk about minimum R version. 3.1.x is too old. It's
>>>> > released 4.5 years ago.
>>>> > R 3.4.0 is released 1.5 years ago. Considering the timing for Spark
>>>> 3.0,
>>>> > deprecating lower versions, bumping up R to 3.4 might be reasonable
>>>> > option.
>>>> >
>>>> > Adding Shane as well.
>>>> >
>>>> > If we ended up with not upgrading it, I will forward this email to
>>>> CRAN
>>>> > sysadmin to discuss further anyway.
>>>> >
>>>> >
>>>> >
>>>> > 2018년 11월 2일 (금) 오후 12:51, Felix Cheung <
>>>>
>>>> > felixcheung@
>>>>
>>>> > >님이 작성:
>>>> >
>>>> >> Thanks for being this up and much appreciate with keeping on top of
>>>> this
>>>> >> at all times.
>>>> >>
>>>> >> Can upgrading R able to fix the issue. Is this perhaps not
>>>> necessarily
>>>> >> malform but some new format for new versions perhaps? Anyway we
>>>> should
>>>> >> consider upgrading R version if that fixes the problem.
>>>> >>
>>>> >> As an option we could also disable the repo check in Jenkins but I
>>>> can
>>>> >> see
>>>> >> that could also be problematic.
>>>> >>
>>>> >>
>>>> >> On Thu, Nov 1, 2018 at 7:35 PM Hyukjin Kwon <
>>>>
>>>> > gurwls223@
>>>>
>>>> > > wrote:
>>>> >>
>>>> >>> Hi all,
>>>> >>>
>>>> >>> I want to raise the CRAN failure issue because it started to block
>>>> Spark
>>>> >>> PRs time to time. Since the number
>>>> >>> of PRs grows hugely in Spark community, this is critical to not
>>>> block
>>>> >>> other PRs.
>>>> >>>
>>>> >>> There has been a problem at CRAN (See
>>>> >>> https://github.com/apache/spark/pull/20005 for analysis).
>>>> >>> To cut it short, the root cause is malformed package info from
>>>> >>> https://cran.r-project.org/src/contrib/PACKAGES
>>>> >>> from server side, and this had to be fixed by requesting it to CRAN
>>>> >>> sysaadmin's help.
>>>> >>>
>>>> >>> https://issues.apache.org/jira/browse/SPARK-24152 <- newly open. I
>>>> am
>>>> >>> pretty sure it's the same issue
>>>> >>> https://issues.apache.org/jira/browse/SPARK-25923 <-
>>>> reopen/resolved
>>>> 2
>>>> >>> times
>>>> >>> https://issues.apache.org/jira/browse/SPARK-22812
>>>> >>>
>>>> >>> This happened 5 times for roughly about 10 months, causing blocking
>>>> >>> almost all PRs in Apache Spark.
>>>> >>> Historically, it blocked whole PRs for few days once, and whole
>>>> Spark
>>>> >>> community had to stop working.
>>>> >>>
>>>> >>> I assume this has been not a super big big issue so far for other
>>>> >>> projects or other people because apparently
>>>> >>> higher version of R has some logics to handle this malformed
>>>> documents
>>>> >>> (at least I verified R 3.4.0 works fine).
>>>> >>>
>>>> >>> For our side, Jenkins has low R version (R 3.1.1 if that's not
>>>> updated
>>>> >>> from what I have seen before),
>>>> >>> which is unable to parse the malformed server's response.
>>>> >>>
>>>> >>> So, I want to talk about how we are going to handle this. Possible
>>>> >>> solutions are:
>>>> >>>
>>>> >>> 1. We should start a talk with CRAN sysadmin to permanently prevent
>>>> this
>>>> >>> issue
>>>> >>> 2. We upgrade R to 3.4.0 in Jenkins (however we will not be able to
>>>> test
>>>> >>> low R versions)
>>>> >>> 3. ...
>>>> >>>
>>>> >>> If if we fine, I would like to suggest to forward this email to
>>>> CRAN
>>>> >>> sysadmin to discuss further about this.
>>>> >>>
>>>> >>> Adding Liang-Chi Felix and Shivaram who I already talked about this
>>>> few
>>>> >>> times before.
>>>> >>>
>>>> >>> Thanks all.
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe e-mail:
>
>> dev-unsubscribe@.apache
>
>>>>
>>>>
>
>
>
>
>
> --
> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail:
> dev-unsubscribe@.apache
--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: [discuss] SparkR CRAN feasibility check server problem
Posted by Liang-Chi Hsieh <vi...@gmail.com>.
Thanks for letting me know! I will look into it and ask CRAN admin for help.
Hyukjin Kwon wrote
> Looks it's happening again. Liang-Chi, do you mind if I ask it again?
>
> FYI, R 3.4 is officially deprecated as of
> https://github.com/apache/spark/pull/23012
> We could upgrade R version to 3.4.x in Jenkins, which deals with the
> malformed(?) responses after 3.0 release.
> Then, we could get rid of this problem..!
>
> 2018년 11월 12일 (월) 오후 1:47, Hyukjin Kwon <
> gurwls223@
> >님이 작성:
>
>> I made a PR to officially drop R prior to version 3.4 (
>> https://github.com/apache/spark/pull/23012).
>> The tests will probably fail for now since it produces warnings for using
>> R 3.1.x.
>>
>> 2018년 11월 11일 (일) 오전 3:00, Felix Cheung <
> felixcheung_m@
> >님이 작성:
>>
>>> It’s a great point about min R version. From what I see, mostly because
>>> of fixes and packages support, most users of R are fairly up to date? So
>>> perhaps 3.4 as min version is reasonable esp. for Spark 3.
>>>
>>> Are we getting traction with CRAN sysadmin? It seems like this has been
>>> broken a few times.
>>>
>>>
>>> ------------------------------
>>> *From:* Liang-Chi Hsieh <
> viirya@
> >
>>> *Sent:* Saturday, November 10, 2018 2:32 AM
>>> *To:*
> dev@.apache
>>> *Subject:* Re: [discuss] SparkR CRAN feasibility check server problem
>>>
>>>
>>> Yeah, thanks Hyukjin Kwon for bringing this up for discussion.
>>>
>>> I don't know how higher versions of R are widely used across R
>>> community.
>>> If
>>> R version 3.1.x was not very commonly used, I think we can discuss to
>>> upgrade minimum R version in next Spark version.
>>>
>>> If we ended up with not upgrading, we can discuss with CRAN sysadmin to
>>> fix
>>> it by the service side automatically that prevents malformed R packages
>>> info. So we don't need to fix it manually every time.
>>>
>>>
>>>
>>> Hyukjin Kwon wrote
>>> >> Can upgrading R able to fix the issue. Is this perhaps not
>>> necessarily
>>> > malform but some new format for new versions perhaps?
>>> > That's my guess. I am not totally sure about it tho.
>>> >
>>> >> Anyway we should consider upgrading R version if that fixes the
>>> problem.
>>> > Yea, we should. If we should, it should be more them R 3.4. Maybe it's
>>> > good
>>> > time to start to talk about minimum R version. 3.1.x is too old. It's
>>> > released 4.5 years ago.
>>> > R 3.4.0 is released 1.5 years ago. Considering the timing for Spark
>>> 3.0,
>>> > deprecating lower versions, bumping up R to 3.4 might be reasonable
>>> > option.
>>> >
>>> > Adding Shane as well.
>>> >
>>> > If we ended up with not upgrading it, I will forward this email to
>>> CRAN
>>> > sysadmin to discuss further anyway.
>>> >
>>> >
>>> >
>>> > 2018년 11월 2일 (금) 오후 12:51, Felix Cheung <
>>>
>>> > felixcheung@
>>>
>>> > >님이 작성:
>>> >
>>> >> Thanks for being this up and much appreciate with keeping on top of
>>> this
>>> >> at all times.
>>> >>
>>> >> Can upgrading R able to fix the issue. Is this perhaps not
>>> necessarily
>>> >> malform but some new format for new versions perhaps? Anyway we
>>> should
>>> >> consider upgrading R version if that fixes the problem.
>>> >>
>>> >> As an option we could also disable the repo check in Jenkins but I
>>> can
>>> >> see
>>> >> that could also be problematic.
>>> >>
>>> >>
>>> >> On Thu, Nov 1, 2018 at 7:35 PM Hyukjin Kwon <
>>>
>>> > gurwls223@
>>>
>>> > > wrote:
>>> >>
>>> >>> Hi all,
>>> >>>
>>> >>> I want to raise the CRAN failure issue because it started to block
>>> Spark
>>> >>> PRs time to time. Since the number
>>> >>> of PRs grows hugely in Spark community, this is critical to not
>>> block
>>> >>> other PRs.
>>> >>>
>>> >>> There has been a problem at CRAN (See
>>> >>> https://github.com/apache/spark/pull/20005 for analysis).
>>> >>> To cut it short, the root cause is malformed package info from
>>> >>> https://cran.r-project.org/src/contrib/PACKAGES
>>> >>> from server side, and this had to be fixed by requesting it to CRAN
>>> >>> sysaadmin's help.
>>> >>>
>>> >>> https://issues.apache.org/jira/browse/SPARK-24152 <- newly open. I
>>> am
>>> >>> pretty sure it's the same issue
>>> >>> https://issues.apache.org/jira/browse/SPARK-25923 <- reopen/resolved
>>> 2
>>> >>> times
>>> >>> https://issues.apache.org/jira/browse/SPARK-22812
>>> >>>
>>> >>> This happened 5 times for roughly about 10 months, causing blocking
>>> >>> almost all PRs in Apache Spark.
>>> >>> Historically, it blocked whole PRs for few days once, and whole
>>> Spark
>>> >>> community had to stop working.
>>> >>>
>>> >>> I assume this has been not a super big big issue so far for other
>>> >>> projects or other people because apparently
>>> >>> higher version of R has some logics to handle this malformed
>>> documents
>>> >>> (at least I verified R 3.4.0 works fine).
>>> >>>
>>> >>> For our side, Jenkins has low R version (R 3.1.1 if that's not
>>> updated
>>> >>> from what I have seen before),
>>> >>> which is unable to parse the malformed server's response.
>>> >>>
>>> >>> So, I want to talk about how we are going to handle this. Possible
>>> >>> solutions are:
>>> >>>
>>> >>> 1. We should start a talk with CRAN sysadmin to permanently prevent
>>> this
>>> >>> issue
>>> >>> 2. We upgrade R to 3.4.0 in Jenkins (however we will not be able to
>>> test
>>> >>> low R versions)
>>> >>> 3. ...
>>> >>>
>>> >>> If if we fine, I would like to suggest to forward this email to CRAN
>>> >>> sysadmin to discuss further about this.
>>> >>>
>>> >>> Adding Liang-Chi Felix and Shivaram who I already talked about this
>>> few
>>> >>> times before.
>>> >>>
>>> >>> Thanks all.
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail:
> dev-unsubscribe@.apache
>>>
>>>
--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: [discuss] SparkR CRAN feasibility check server problem
Posted by Hyukjin Kwon <gu...@gmail.com>.
Looks it's happening again. Liang-Chi, do you mind if I ask it again?
FYI, R 3.4 is officially deprecated as of
https://github.com/apache/spark/pull/23012
We could upgrade R version to 3.4.x in Jenkins, which deals with the
malformed(?) responses after 3.0 release.
Then, we could get rid of this problem..!
2018년 11월 12일 (월) 오후 1:47, Hyukjin Kwon <gu...@gmail.com>님이 작성:
> I made a PR to officially drop R prior to version 3.4 (
> https://github.com/apache/spark/pull/23012).
> The tests will probably fail for now since it produces warnings for using
> R 3.1.x.
>
> 2018년 11월 11일 (일) 오전 3:00, Felix Cheung <fe...@hotmail.com>님이 작성:
>
>> It’s a great point about min R version. From what I see, mostly because
>> of fixes and packages support, most users of R are fairly up to date? So
>> perhaps 3.4 as min version is reasonable esp. for Spark 3.
>>
>> Are we getting traction with CRAN sysadmin? It seems like this has been
>> broken a few times.
>>
>>
>> ------------------------------
>> *From:* Liang-Chi Hsieh <vi...@gmail.com>
>> *Sent:* Saturday, November 10, 2018 2:32 AM
>> *To:* dev@spark.apache.org
>> *Subject:* Re: [discuss] SparkR CRAN feasibility check server problem
>>
>>
>> Yeah, thanks Hyukjin Kwon for bringing this up for discussion.
>>
>> I don't know how higher versions of R are widely used across R community.
>> If
>> R version 3.1.x was not very commonly used, I think we can discuss to
>> upgrade minimum R version in next Spark version.
>>
>> If we ended up with not upgrading, we can discuss with CRAN sysadmin to
>> fix
>> it by the service side automatically that prevents malformed R packages
>> info. So we don't need to fix it manually every time.
>>
>>
>>
>> Hyukjin Kwon wrote
>> >> Can upgrading R able to fix the issue. Is this perhaps not necessarily
>> > malform but some new format for new versions perhaps?
>> > That's my guess. I am not totally sure about it tho.
>> >
>> >> Anyway we should consider upgrading R version if that fixes the
>> problem.
>> > Yea, we should. If we should, it should be more them R 3.4. Maybe it's
>> > good
>> > time to start to talk about minimum R version. 3.1.x is too old. It's
>> > released 4.5 years ago.
>> > R 3.4.0 is released 1.5 years ago. Considering the timing for Spark 3.0,
>> > deprecating lower versions, bumping up R to 3.4 might be reasonable
>> > option.
>> >
>> > Adding Shane as well.
>> >
>> > If we ended up with not upgrading it, I will forward this email to CRAN
>> > sysadmin to discuss further anyway.
>> >
>> >
>> >
>> > 2018년 11월 2일 (금) 오후 12:51, Felix Cheung <
>>
>> > felixcheung@
>>
>> > >님이 작성:
>> >
>> >> Thanks for being this up and much appreciate with keeping on top of
>> this
>> >> at all times.
>> >>
>> >> Can upgrading R able to fix the issue. Is this perhaps not necessarily
>> >> malform but some new format for new versions perhaps? Anyway we should
>> >> consider upgrading R version if that fixes the problem.
>> >>
>> >> As an option we could also disable the repo check in Jenkins but I can
>> >> see
>> >> that could also be problematic.
>> >>
>> >>
>> >> On Thu, Nov 1, 2018 at 7:35 PM Hyukjin Kwon <
>>
>> > gurwls223@
>>
>> > > wrote:
>> >>
>> >>> Hi all,
>> >>>
>> >>> I want to raise the CRAN failure issue because it started to block
>> Spark
>> >>> PRs time to time. Since the number
>> >>> of PRs grows hugely in Spark community, this is critical to not block
>> >>> other PRs.
>> >>>
>> >>> There has been a problem at CRAN (See
>> >>> https://github.com/apache/spark/pull/20005 for analysis).
>> >>> To cut it short, the root cause is malformed package info from
>> >>> https://cran.r-project.org/src/contrib/PACKAGES
>> >>> from server side, and this had to be fixed by requesting it to CRAN
>> >>> sysaadmin's help.
>> >>>
>> >>> https://issues.apache.org/jira/browse/SPARK-24152 <- newly open. I am
>> >>> pretty sure it's the same issue
>> >>> https://issues.apache.org/jira/browse/SPARK-25923 <- reopen/resolved
>> 2
>> >>> times
>> >>> https://issues.apache.org/jira/browse/SPARK-22812
>> >>>
>> >>> This happened 5 times for roughly about 10 months, causing blocking
>> >>> almost all PRs in Apache Spark.
>> >>> Historically, it blocked whole PRs for few days once, and whole Spark
>> >>> community had to stop working.
>> >>>
>> >>> I assume this has been not a super big big issue so far for other
>> >>> projects or other people because apparently
>> >>> higher version of R has some logics to handle this malformed documents
>> >>> (at least I verified R 3.4.0 works fine).
>> >>>
>> >>> For our side, Jenkins has low R version (R 3.1.1 if that's not updated
>> >>> from what I have seen before),
>> >>> which is unable to parse the malformed server's response.
>> >>>
>> >>> So, I want to talk about how we are going to handle this. Possible
>> >>> solutions are:
>> >>>
>> >>> 1. We should start a talk with CRAN sysadmin to permanently prevent
>> this
>> >>> issue
>> >>> 2. We upgrade R to 3.4.0 in Jenkins (however we will not be able to
>> test
>> >>> low R versions)
>> >>> 3. ...
>> >>>
>> >>> If if we fine, I would like to suggest to forward this email to CRAN
>> >>> sysadmin to discuss further about this.
>> >>>
>> >>> Adding Liang-Chi Felix and Shivaram who I already talked about this
>> few
>> >>> times before.
>> >>>
>> >>> Thanks all.
>> >>>
>> >>>
>> >>>
>> >>>
>>
>>
>>
>>
>>
>> --
>> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
>>
Re: [discuss] SparkR CRAN feasibility check server problem
Posted by Hyukjin Kwon <gu...@gmail.com>.
I made a PR to officially drop R prior to version 3.4 (
https://github.com/apache/spark/pull/23012).
The tests will probably fail for now since it produces warnings for using R
3.1.x.
2018년 11월 11일 (일) 오전 3:00, Felix Cheung <fe...@hotmail.com>님이 작성:
> It’s a great point about min R version. From what I see, mostly because of
> fixes and packages support, most users of R are fairly up to date? So
> perhaps 3.4 as min version is reasonable esp. for Spark 3.
>
> Are we getting traction with CRAN sysadmin? It seems like this has been
> broken a few times.
>
>
> ------------------------------
> *From:* Liang-Chi Hsieh <vi...@gmail.com>
> *Sent:* Saturday, November 10, 2018 2:32 AM
> *To:* dev@spark.apache.org
> *Subject:* Re: [discuss] SparkR CRAN feasibility check server problem
>
>
> Yeah, thanks Hyukjin Kwon for bringing this up for discussion.
>
> I don't know how higher versions of R are widely used across R community.
> If
> R version 3.1.x was not very commonly used, I think we can discuss to
> upgrade minimum R version in next Spark version.
>
> If we ended up with not upgrading, we can discuss with CRAN sysadmin to fix
> it by the service side automatically that prevents malformed R packages
> info. So we don't need to fix it manually every time.
>
>
>
> Hyukjin Kwon wrote
> >> Can upgrading R able to fix the issue. Is this perhaps not necessarily
> > malform but some new format for new versions perhaps?
> > That's my guess. I am not totally sure about it tho.
> >
> >> Anyway we should consider upgrading R version if that fixes the problem.
> > Yea, we should. If we should, it should be more them R 3.4. Maybe it's
> > good
> > time to start to talk about minimum R version. 3.1.x is too old. It's
> > released 4.5 years ago.
> > R 3.4.0 is released 1.5 years ago. Considering the timing for Spark 3.0,
> > deprecating lower versions, bumping up R to 3.4 might be reasonable
> > option.
> >
> > Adding Shane as well.
> >
> > If we ended up with not upgrading it, I will forward this email to CRAN
> > sysadmin to discuss further anyway.
> >
> >
> >
> > 2018년 11월 2일 (금) 오후 12:51, Felix Cheung <
>
> > felixcheung@
>
> > >님이 작성:
> >
> >> Thanks for being this up and much appreciate with keeping on top of this
> >> at all times.
> >>
> >> Can upgrading R able to fix the issue. Is this perhaps not necessarily
> >> malform but some new format for new versions perhaps? Anyway we should
> >> consider upgrading R version if that fixes the problem.
> >>
> >> As an option we could also disable the repo check in Jenkins but I can
> >> see
> >> that could also be problematic.
> >>
> >>
> >> On Thu, Nov 1, 2018 at 7:35 PM Hyukjin Kwon <
>
> > gurwls223@
>
> > > wrote:
> >>
> >>> Hi all,
> >>>
> >>> I want to raise the CRAN failure issue because it started to block
> Spark
> >>> PRs time to time. Since the number
> >>> of PRs grows hugely in Spark community, this is critical to not block
> >>> other PRs.
> >>>
> >>> There has been a problem at CRAN (See
> >>> https://github.com/apache/spark/pull/20005 for analysis).
> >>> To cut it short, the root cause is malformed package info from
> >>> https://cran.r-project.org/src/contrib/PACKAGES
> >>> from server side, and this had to be fixed by requesting it to CRAN
> >>> sysaadmin's help.
> >>>
> >>> https://issues.apache.org/jira/browse/SPARK-24152 <- newly open. I am
> >>> pretty sure it's the same issue
> >>> https://issues.apache.org/jira/browse/SPARK-25923 <- reopen/resolved 2
> >>> times
> >>> https://issues.apache.org/jira/browse/SPARK-22812
> >>>
> >>> This happened 5 times for roughly about 10 months, causing blocking
> >>> almost all PRs in Apache Spark.
> >>> Historically, it blocked whole PRs for few days once, and whole Spark
> >>> community had to stop working.
> >>>
> >>> I assume this has been not a super big big issue so far for other
> >>> projects or other people because apparently
> >>> higher version of R has some logics to handle this malformed documents
> >>> (at least I verified R 3.4.0 works fine).
> >>>
> >>> For our side, Jenkins has low R version (R 3.1.1 if that's not updated
> >>> from what I have seen before),
> >>> which is unable to parse the malformed server's response.
> >>>
> >>> So, I want to talk about how we are going to handle this. Possible
> >>> solutions are:
> >>>
> >>> 1. We should start a talk with CRAN sysadmin to permanently prevent
> this
> >>> issue
> >>> 2. We upgrade R to 3.4.0 in Jenkins (however we will not be able to
> test
> >>> low R versions)
> >>> 3. ...
> >>>
> >>> If if we fine, I would like to suggest to forward this email to CRAN
> >>> sysadmin to discuss further about this.
> >>>
> >>> Adding Liang-Chi Felix and Shivaram who I already talked about this few
> >>> times before.
> >>>
> >>> Thanks all.
> >>>
> >>>
> >>>
> >>>
>
>
>
>
>
> --
> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
Re: [discuss] SparkR CRAN feasibility check server problem
Posted by Felix Cheung <fe...@hotmail.com>.
It’s a great point about min R version. From what I see, mostly because of fixes and packages support, most users of R are fairly up to date? So perhaps 3.4 as min version is reasonable esp. for Spark 3.
Are we getting traction with CRAN sysadmin? It seems like this has been broken a few times.
________________________________
From: Liang-Chi Hsieh <vi...@gmail.com>
Sent: Saturday, November 10, 2018 2:32 AM
To: dev@spark.apache.org
Subject: Re: [discuss] SparkR CRAN feasibility check server problem
Yeah, thanks Hyukjin Kwon for bringing this up for discussion.
I don't know how higher versions of R are widely used across R community. If
R version 3.1.x was not very commonly used, I think we can discuss to
upgrade minimum R version in next Spark version.
If we ended up with not upgrading, we can discuss with CRAN sysadmin to fix
it by the service side automatically that prevents malformed R packages
info. So we don't need to fix it manually every time.
Hyukjin Kwon wrote
>> Can upgrading R able to fix the issue. Is this perhaps not necessarily
> malform but some new format for new versions perhaps?
> That's my guess. I am not totally sure about it tho.
>
>> Anyway we should consider upgrading R version if that fixes the problem.
> Yea, we should. If we should, it should be more them R 3.4. Maybe it's
> good
> time to start to talk about minimum R version. 3.1.x is too old. It's
> released 4.5 years ago.
> R 3.4.0 is released 1.5 years ago. Considering the timing for Spark 3.0,
> deprecating lower versions, bumping up R to 3.4 might be reasonable
> option.
>
> Adding Shane as well.
>
> If we ended up with not upgrading it, I will forward this email to CRAN
> sysadmin to discuss further anyway.
>
>
>
> 2018년 11월 2일 (금) 오후 12:51, Felix Cheung <
> felixcheung@
> >님이 작성:
>
>> Thanks for being this up and much appreciate with keeping on top of this
>> at all times.
>>
>> Can upgrading R able to fix the issue. Is this perhaps not necessarily
>> malform but some new format for new versions perhaps? Anyway we should
>> consider upgrading R version if that fixes the problem.
>>
>> As an option we could also disable the repo check in Jenkins but I can
>> see
>> that could also be problematic.
>>
>>
>> On Thu, Nov 1, 2018 at 7:35 PM Hyukjin Kwon <
> gurwls223@
> > wrote:
>>
>>> Hi all,
>>>
>>> I want to raise the CRAN failure issue because it started to block Spark
>>> PRs time to time. Since the number
>>> of PRs grows hugely in Spark community, this is critical to not block
>>> other PRs.
>>>
>>> There has been a problem at CRAN (See
>>> https://github.com/apache/spark/pull/20005 for analysis).
>>> To cut it short, the root cause is malformed package info from
>>> https://cran.r-project.org/src/contrib/PACKAGES
>>> from server side, and this had to be fixed by requesting it to CRAN
>>> sysaadmin's help.
>>>
>>> https://issues.apache.org/jira/browse/SPARK-24152 <- newly open. I am
>>> pretty sure it's the same issue
>>> https://issues.apache.org/jira/browse/SPARK-25923 <- reopen/resolved 2
>>> times
>>> https://issues.apache.org/jira/browse/SPARK-22812
>>>
>>> This happened 5 times for roughly about 10 months, causing blocking
>>> almost all PRs in Apache Spark.
>>> Historically, it blocked whole PRs for few days once, and whole Spark
>>> community had to stop working.
>>>
>>> I assume this has been not a super big big issue so far for other
>>> projects or other people because apparently
>>> higher version of R has some logics to handle this malformed documents
>>> (at least I verified R 3.4.0 works fine).
>>>
>>> For our side, Jenkins has low R version (R 3.1.1 if that's not updated
>>> from what I have seen before),
>>> which is unable to parse the malformed server's response.
>>>
>>> So, I want to talk about how we are going to handle this. Possible
>>> solutions are:
>>>
>>> 1. We should start a talk with CRAN sysadmin to permanently prevent this
>>> issue
>>> 2. We upgrade R to 3.4.0 in Jenkins (however we will not be able to test
>>> low R versions)
>>> 3. ...
>>>
>>> If if we fine, I would like to suggest to forward this email to CRAN
>>> sysadmin to discuss further about this.
>>>
>>> Adding Liang-Chi Felix and Shivaram who I already talked about this few
>>> times before.
>>>
>>> Thanks all.
>>>
>>>
>>>
>>>
--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: [discuss] SparkR CRAN feasibility check server problem
Posted by Liang-Chi Hsieh <vi...@gmail.com>.
Yeah, thanks Hyukjin Kwon for bringing this up for discussion.
I don't know how higher versions of R are widely used across R community. If
R version 3.1.x was not very commonly used, I think we can discuss to
upgrade minimum R version in next Spark version.
If we ended up with not upgrading, we can discuss with CRAN sysadmin to fix
it by the service side automatically that prevents malformed R packages
info. So we don't need to fix it manually every time.
Hyukjin Kwon wrote
>> Can upgrading R able to fix the issue. Is this perhaps not necessarily
> malform but some new format for new versions perhaps?
> That's my guess. I am not totally sure about it tho.
>
>> Anyway we should consider upgrading R version if that fixes the problem.
> Yea, we should. If we should, it should be more them R 3.4. Maybe it's
> good
> time to start to talk about minimum R version. 3.1.x is too old. It's
> released 4.5 years ago.
> R 3.4.0 is released 1.5 years ago. Considering the timing for Spark 3.0,
> deprecating lower versions, bumping up R to 3.4 might be reasonable
> option.
>
> Adding Shane as well.
>
> If we ended up with not upgrading it, I will forward this email to CRAN
> sysadmin to discuss further anyway.
>
>
>
> 2018년 11월 2일 (금) 오후 12:51, Felix Cheung <
> felixcheung@
> >님이 작성:
>
>> Thanks for being this up and much appreciate with keeping on top of this
>> at all times.
>>
>> Can upgrading R able to fix the issue. Is this perhaps not necessarily
>> malform but some new format for new versions perhaps? Anyway we should
>> consider upgrading R version if that fixes the problem.
>>
>> As an option we could also disable the repo check in Jenkins but I can
>> see
>> that could also be problematic.
>>
>>
>> On Thu, Nov 1, 2018 at 7:35 PM Hyukjin Kwon <
> gurwls223@
> > wrote:
>>
>>> Hi all,
>>>
>>> I want to raise the CRAN failure issue because it started to block Spark
>>> PRs time to time. Since the number
>>> of PRs grows hugely in Spark community, this is critical to not block
>>> other PRs.
>>>
>>> There has been a problem at CRAN (See
>>> https://github.com/apache/spark/pull/20005 for analysis).
>>> To cut it short, the root cause is malformed package info from
>>> https://cran.r-project.org/src/contrib/PACKAGES
>>> from server side, and this had to be fixed by requesting it to CRAN
>>> sysaadmin's help.
>>>
>>> https://issues.apache.org/jira/browse/SPARK-24152 <- newly open. I am
>>> pretty sure it's the same issue
>>> https://issues.apache.org/jira/browse/SPARK-25923 <- reopen/resolved 2
>>> times
>>> https://issues.apache.org/jira/browse/SPARK-22812
>>>
>>> This happened 5 times for roughly about 10 months, causing blocking
>>> almost all PRs in Apache Spark.
>>> Historically, it blocked whole PRs for few days once, and whole Spark
>>> community had to stop working.
>>>
>>> I assume this has been not a super big big issue so far for other
>>> projects or other people because apparently
>>> higher version of R has some logics to handle this malformed documents
>>> (at least I verified R 3.4.0 works fine).
>>>
>>> For our side, Jenkins has low R version (R 3.1.1 if that's not updated
>>> from what I have seen before),
>>> which is unable to parse the malformed server's response.
>>>
>>> So, I want to talk about how we are going to handle this. Possible
>>> solutions are:
>>>
>>> 1. We should start a talk with CRAN sysadmin to permanently prevent this
>>> issue
>>> 2. We upgrade R to 3.4.0 in Jenkins (however we will not be able to test
>>> low R versions)
>>> 3. ...
>>>
>>> If if we fine, I would like to suggest to forward this email to CRAN
>>> sysadmin to discuss further about this.
>>>
>>> Adding Liang-Chi Felix and Shivaram who I already talked about this few
>>> times before.
>>>
>>> Thanks all.
>>>
>>>
>>>
>>>
--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: [discuss] SparkR CRAN feasibility check server problem
Posted by Hyukjin Kwon <gu...@gmail.com>.
> Can upgrading R able to fix the issue. Is this perhaps not necessarily
malform but some new format for new versions perhaps?
That's my guess. I am not totally sure about it tho.
> Anyway we should consider upgrading R version if that fixes the problem.
Yea, we should. If we should, it should be more them R 3.4. Maybe it's good
time to start to talk about minimum R version. 3.1.x is too old. It's
released 4.5 years ago.
R 3.4.0 is released 1.5 years ago. Considering the timing for Spark 3.0,
deprecating lower versions, bumping up R to 3.4 might be reasonable option.
Adding Shane as well.
If we ended up with not upgrading it, I will forward this email to CRAN
sysadmin to discuss further anyway.
2018년 11월 2일 (금) 오후 12:51, Felix Cheung <fe...@apache.org>님이 작성:
> Thanks for being this up and much appreciate with keeping on top of this
> at all times.
>
> Can upgrading R able to fix the issue. Is this perhaps not necessarily
> malform but some new format for new versions perhaps? Anyway we should
> consider upgrading R version if that fixes the problem.
>
> As an option we could also disable the repo check in Jenkins but I can see
> that could also be problematic.
>
>
> On Thu, Nov 1, 2018 at 7:35 PM Hyukjin Kwon <gu...@gmail.com> wrote:
>
>> Hi all,
>>
>> I want to raise the CRAN failure issue because it started to block Spark
>> PRs time to time. Since the number
>> of PRs grows hugely in Spark community, this is critical to not block
>> other PRs.
>>
>> There has been a problem at CRAN (See
>> https://github.com/apache/spark/pull/20005 for analysis).
>> To cut it short, the root cause is malformed package info from
>> https://cran.r-project.org/src/contrib/PACKAGES
>> from server side, and this had to be fixed by requesting it to CRAN
>> sysaadmin's help.
>>
>> https://issues.apache.org/jira/browse/SPARK-24152 <- newly open. I am
>> pretty sure it's the same issue
>> https://issues.apache.org/jira/browse/SPARK-25923 <- reopen/resolved 2
>> times
>> https://issues.apache.org/jira/browse/SPARK-22812
>>
>> This happened 5 times for roughly about 10 months, causing blocking
>> almost all PRs in Apache Spark.
>> Historically, it blocked whole PRs for few days once, and whole Spark
>> community had to stop working.
>>
>> I assume this has been not a super big big issue so far for other
>> projects or other people because apparently
>> higher version of R has some logics to handle this malformed documents
>> (at least I verified R 3.4.0 works fine).
>>
>> For our side, Jenkins has low R version (R 3.1.1 if that's not updated
>> from what I have seen before),
>> which is unable to parse the malformed server's response.
>>
>> So, I want to talk about how we are going to handle this. Possible
>> solutions are:
>>
>> 1. We should start a talk with CRAN sysadmin to permanently prevent this
>> issue
>> 2. We upgrade R to 3.4.0 in Jenkins (however we will not be able to test
>> low R versions)
>> 3. ...
>>
>> If if we fine, I would like to suggest to forward this email to CRAN
>> sysadmin to discuss further about this.
>>
>> Adding Liang-Chi Felix and Shivaram who I already talked about this few
>> times before.
>>
>> Thanks all.
>>
>>
>>
>>
Re: [discuss] SparkR CRAN feasibility check server problem
Posted by Felix Cheung <fe...@apache.org>.
Thanks for being this up and much appreciate with keeping on top of this at
all times.
Can upgrading R able to fix the issue. Is this perhaps not necessarily
malform but some new format for new versions perhaps? Anyway we should
consider upgrading R version if that fixes the problem.
As an option we could also disable the repo check in Jenkins but I can see
that could also be problematic.
On Thu, Nov 1, 2018 at 7:35 PM Hyukjin Kwon <gu...@gmail.com> wrote:
> Hi all,
>
> I want to raise the CRAN failure issue because it started to block Spark
> PRs time to time. Since the number
> of PRs grows hugely in Spark community, this is critical to not block
> other PRs.
>
> There has been a problem at CRAN (See
> https://github.com/apache/spark/pull/20005 for analysis).
> To cut it short, the root cause is malformed package info from
> https://cran.r-project.org/src/contrib/PACKAGES
> from server side, and this had to be fixed by requesting it to CRAN
> sysaadmin's help.
>
> https://issues.apache.org/jira/browse/SPARK-24152 <- newly open. I am
> pretty sure it's the same issue
> https://issues.apache.org/jira/browse/SPARK-25923 <- reopen/resolved 2
> times
> https://issues.apache.org/jira/browse/SPARK-22812
>
> This happened 5 times for roughly about 10 months, causing blocking almost
> all PRs in Apache Spark.
> Historically, it blocked whole PRs for few days once, and whole Spark
> community had to stop working.
>
> I assume this has been not a super big big issue so far for other projects
> or other people because apparently
> higher version of R has some logics to handle this malformed documents (at
> least I verified R 3.4.0 works fine).
>
> For our side, Jenkins has low R version (R 3.1.1 if that's not updated
> from what I have seen before),
> which is unable to parse the malformed server's response.
>
> So, I want to talk about how we are going to handle this. Possible
> solutions are:
>
> 1. We should start a talk with CRAN sysadmin to permanently prevent this
> issue
> 2. We upgrade R to 3.4.0 in Jenkins (however we will not be able to test
> low R versions)
> 3. ...
>
> If if we fine, I would like to suggest to forward this email to CRAN
> sysadmin to discuss further about this.
>
> Adding Liang-Chi Felix and Shivaram who I already talked about this few
> times before.
>
> Thanks all.
>
>
>
>