You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by roni <ro...@gmail.com> on 2016/03/15 06:44:28 UTC
sparkR issues ?
Hi,
I am working with bioinformatics and trying to convert some scripts to
sparkR to fit into other spark jobs.
I tries a simple example from a bioinf lib and as soon as I start sparkR
environment it does not work.
code as follows -
countData <- matrix(1:100,ncol=4)
condition <- factor(c("A","A","B","B"))
dds <- DESeqDataSetFromMatrix(countData, DataFrame(condition), ~ condition)
Works if i dont initialize the sparkR environment.
if I do library(SparkR) and sqlContext <- sparkRSQL.init(sc) it gives
following error
> dds <- DESeqDataSetFromMatrix(countData, as.data.frame(condition), ~
condition)
Error in DataFrame(colData, row.names = rownames(colData)) :
cannot coerce class "data.frame" to a DataFrame
I am really stumped. I am not using any spark function , so i would expect
it to work as a simple R code.
why it does not work?
Appreciate the help
-R
RE: sparkR issues ?
Posted by "Sun, Rui" <ru...@intel.com>.
Sorry. I am wrong. The issue is not related to as.data.frame(). It seems to be related to DataFrame naming conflict between s4vectors and SparkR.
Refer to https://issues.apache.org/jira/browse/SPARK-12148
From: Sun, Rui [mailto:rui.sun@intel.com]
Sent: Wednesday, March 16, 2016 9:33 AM
To: Alex Kozlov <al...@gmail.com>; roni <ro...@gmail.com>
Cc: user@spark.apache.org
Subject: RE: sparkR issues ?
I have submitted https://issues.apache.org/jira/browse/SPARK-13905 and a PR for it.
From: Alex Kozlov [mailto:alexvk@gmail.com]
Sent: Wednesday, March 16, 2016 12:52 AM
To: roni <ro...@gmail.com>>
Cc: Sun, Rui <ru...@intel.com>>; user@spark.apache.org<ma...@spark.apache.org>
Subject: Re: sparkR issues ?
Hi Roni, you can probably rename the as.data.frame in $SPARK_HOME/R/pkg/R/DataFrame.R and re-install SparkR by running install-dev.sh
On Tue, Mar 15, 2016 at 8:46 AM, roni <ro...@gmail.com>> wrote:
Hi ,
Is there a work around for this?
Do i need to file a bug for this?
Thanks
-R
On Tue, Mar 15, 2016 at 12:28 AM, Sun, Rui <ru...@intel.com>> wrote:
It seems as.data.frame() defined in SparkR convers the versions in R base package.
We can try to see if we can change the implementation of as.data.frame() in SparkR to avoid such covering.
From: Alex Kozlov [mailto:alexvk@gmail.com<ma...@gmail.com>]
Sent: Tuesday, March 15, 2016 2:59 PM
To: roni <ro...@gmail.com>>
Cc: user@spark.apache.org<ma...@spark.apache.org>
Subject: Re: sparkR issues ?
This seems to be a very unfortunate name collision. SparkR defines it's own DataFrame class which shadows what seems to be your own definition.
Is DataFrame something you define? Can you rename it?
On Mon, Mar 14, 2016 at 10:44 PM, roni <ro...@gmail.com>> wrote:
Hi,
I am working with bioinformatics and trying to convert some scripts to sparkR to fit into other spark jobs.
I tries a simple example from a bioinf lib and as soon as I start sparkR environment it does not work.
code as follows -
countData <- matrix(1:100,ncol=4)
condition <- factor(c("A","A","B","B"))
dds <- DESeqDataSetFromMatrix(countData, DataFrame(condition), ~ condition)
Works if i dont initialize the sparkR environment.
if I do library(SparkR) and sqlContext <- sparkRSQL.init(sc) it gives following error
> dds <- DESeqDataSetFromMatrix(countData, as.data.frame(condition), ~ condition)
Error in DataFrame(colData, row.names = rownames(colData)) :
cannot coerce class "data.frame" to a DataFrame
I am really stumped. I am not using any spark function , so i would expect it to work as a simple R code.
why it does not work?
Appreciate the help
-R
--
Alex Kozlov
(408) 507-4987<tel:%28408%29%20507-4987>
(650) 887-2135<tel:%28650%29%20887-2135> efax
alexvk@gmail.com<ma...@gmail.com>
--
Alex Kozlov
(408) 507-4987
(650) 887-2135 efax
alexvk@gmail.com<ma...@gmail.com>
RE: sparkR issues ?
Posted by "Sun, Rui" <ru...@intel.com>.
I have submitted https://issues.apache.org/jira/browse/SPARK-13905 and a PR for it.
From: Alex Kozlov [mailto:alexvk@gmail.com]
Sent: Wednesday, March 16, 2016 12:52 AM
To: roni <ro...@gmail.com>
Cc: Sun, Rui <ru...@intel.com>; user@spark.apache.org
Subject: Re: sparkR issues ?
Hi Roni, you can probably rename the as.data.frame in $SPARK_HOME/R/pkg/R/DataFrame.R and re-install SparkR by running install-dev.sh
On Tue, Mar 15, 2016 at 8:46 AM, roni <ro...@gmail.com>> wrote:
Hi ,
Is there a work around for this?
Do i need to file a bug for this?
Thanks
-R
On Tue, Mar 15, 2016 at 12:28 AM, Sun, Rui <ru...@intel.com>> wrote:
It seems as.data.frame() defined in SparkR convers the versions in R base package.
We can try to see if we can change the implementation of as.data.frame() in SparkR to avoid such covering.
From: Alex Kozlov [mailto:alexvk@gmail.com<ma...@gmail.com>]
Sent: Tuesday, March 15, 2016 2:59 PM
To: roni <ro...@gmail.com>>
Cc: user@spark.apache.org<ma...@spark.apache.org>
Subject: Re: sparkR issues ?
This seems to be a very unfortunate name collision. SparkR defines it's own DataFrame class which shadows what seems to be your own definition.
Is DataFrame something you define? Can you rename it?
On Mon, Mar 14, 2016 at 10:44 PM, roni <ro...@gmail.com>> wrote:
Hi,
I am working with bioinformatics and trying to convert some scripts to sparkR to fit into other spark jobs.
I tries a simple example from a bioinf lib and as soon as I start sparkR environment it does not work.
code as follows -
countData <- matrix(1:100,ncol=4)
condition <- factor(c("A","A","B","B"))
dds <- DESeqDataSetFromMatrix(countData, DataFrame(condition), ~ condition)
Works if i dont initialize the sparkR environment.
if I do library(SparkR) and sqlContext <- sparkRSQL.init(sc) it gives following error
> dds <- DESeqDataSetFromMatrix(countData, as.data.frame(condition), ~ condition)
Error in DataFrame(colData, row.names = rownames(colData)) :
cannot coerce class "data.frame" to a DataFrame
I am really stumped. I am not using any spark function , so i would expect it to work as a simple R code.
why it does not work?
Appreciate the help
-R
--
Alex Kozlov
(408) 507-4987<tel:%28408%29%20507-4987>
(650) 887-2135<tel:%28650%29%20887-2135> efax
alexvk@gmail.com<ma...@gmail.com>
--
Alex Kozlov
(408) 507-4987
(650) 887-2135 efax
alexvk@gmail.com<ma...@gmail.com>
Re: sparkR issues ?
Posted by Alex Kozlov <al...@gmail.com>.
Hi Roni, you can probably rename the as.data.frame in
$SPARK_HOME/R/pkg/R/DataFrame.R and re-install SparkR by running
install-dev.sh
On Tue, Mar 15, 2016 at 8:46 AM, roni <ro...@gmail.com> wrote:
> Hi ,
> Is there a work around for this?
> Do i need to file a bug for this?
> Thanks
> -R
>
> On Tue, Mar 15, 2016 at 12:28 AM, Sun, Rui <ru...@intel.com> wrote:
>
>> It seems as.data.frame() defined in SparkR convers the versions in R base
>> package.
>>
>> We can try to see if we can change the implementation of as.data.frame()
>> in SparkR to avoid such covering.
>>
>>
>>
>> *From:* Alex Kozlov [mailto:alexvk@gmail.com]
>> *Sent:* Tuesday, March 15, 2016 2:59 PM
>> *To:* roni <ro...@gmail.com>
>> *Cc:* user@spark.apache.org
>> *Subject:* Re: sparkR issues ?
>>
>>
>>
>> This seems to be a very unfortunate name collision. SparkR defines it's
>> own DataFrame class which shadows what seems to be your own definition.
>>
>>
>>
>> Is DataFrame something you define? Can you rename it?
>>
>>
>>
>> On Mon, Mar 14, 2016 at 10:44 PM, roni <ro...@gmail.com> wrote:
>>
>> Hi,
>>
>> I am working with bioinformatics and trying to convert some scripts to
>> sparkR to fit into other spark jobs.
>>
>>
>>
>> I tries a simple example from a bioinf lib and as soon as I start sparkR
>> environment it does not work.
>>
>>
>>
>> code as follows -
>>
>> countData <- matrix(1:100,ncol=4)
>>
>> condition <- factor(c("A","A","B","B"))
>>
>> dds <- DESeqDataSetFromMatrix(countData, DataFrame(condition), ~
>> condition)
>>
>>
>>
>> Works if i dont initialize the sparkR environment.
>>
>> if I do library(SparkR) and sqlContext <- sparkRSQL.init(sc) it gives
>> following error
>>
>>
>>
>> > dds <- DESeqDataSetFromMatrix(countData, as.data.frame(condition), ~
>> condition)
>>
>> Error in DataFrame(colData, row.names = rownames(colData)) :
>>
>> cannot coerce class "data.frame" to a DataFrame
>>
>>
>>
>> I am really stumped. I am not using any spark function , so i would
>> expect it to work as a simple R code.
>>
>> why it does not work?
>>
>>
>>
>> Appreciate the help
>>
>> -R
>>
>>
>>
>>
>>
>>
>>
>> --
>>
>> Alex Kozlov
>> (408) 507-4987
>> (650) 887-2135 efax
>> alexvk@gmail.com
>>
>
>
--
Alex Kozlov
(408) 507-4987
(650) 887-2135 efax
alexvk@gmail.com
Re: sparkR issues ?
Posted by roni <ro...@gmail.com>.
Hi ,
Is there a work around for this?
Do i need to file a bug for this?
Thanks
-R
On Tue, Mar 15, 2016 at 12:28 AM, Sun, Rui <ru...@intel.com> wrote:
> It seems as.data.frame() defined in SparkR convers the versions in R base
> package.
>
> We can try to see if we can change the implementation of as.data.frame()
> in SparkR to avoid such covering.
>
>
>
> *From:* Alex Kozlov [mailto:alexvk@gmail.com]
> *Sent:* Tuesday, March 15, 2016 2:59 PM
> *To:* roni <ro...@gmail.com>
> *Cc:* user@spark.apache.org
> *Subject:* Re: sparkR issues ?
>
>
>
> This seems to be a very unfortunate name collision. SparkR defines it's
> own DataFrame class which shadows what seems to be your own definition.
>
>
>
> Is DataFrame something you define? Can you rename it?
>
>
>
> On Mon, Mar 14, 2016 at 10:44 PM, roni <ro...@gmail.com> wrote:
>
> Hi,
>
> I am working with bioinformatics and trying to convert some scripts to
> sparkR to fit into other spark jobs.
>
>
>
> I tries a simple example from a bioinf lib and as soon as I start sparkR
> environment it does not work.
>
>
>
> code as follows -
>
> countData <- matrix(1:100,ncol=4)
>
> condition <- factor(c("A","A","B","B"))
>
> dds <- DESeqDataSetFromMatrix(countData, DataFrame(condition), ~ condition)
>
>
>
> Works if i dont initialize the sparkR environment.
>
> if I do library(SparkR) and sqlContext <- sparkRSQL.init(sc) it gives
> following error
>
>
>
> > dds <- DESeqDataSetFromMatrix(countData, as.data.frame(condition), ~
> condition)
>
> Error in DataFrame(colData, row.names = rownames(colData)) :
>
> cannot coerce class "data.frame" to a DataFrame
>
>
>
> I am really stumped. I am not using any spark function , so i would expect
> it to work as a simple R code.
>
> why it does not work?
>
>
>
> Appreciate the help
>
> -R
>
>
>
>
>
>
>
> --
>
> Alex Kozlov
> (408) 507-4987
> (650) 887-2135 efax
> alexvk@gmail.com
>
RE: sparkR issues ?
Posted by "Sun, Rui" <ru...@intel.com>.
It seems as.data.frame() defined in SparkR convers the versions in R base package.
We can try to see if we can change the implementation of as.data.frame() in SparkR to avoid such covering.
From: Alex Kozlov [mailto:alexvk@gmail.com]
Sent: Tuesday, March 15, 2016 2:59 PM
To: roni <ro...@gmail.com>
Cc: user@spark.apache.org
Subject: Re: sparkR issues ?
This seems to be a very unfortunate name collision. SparkR defines it's own DataFrame class which shadows what seems to be your own definition.
Is DataFrame something you define? Can you rename it?
On Mon, Mar 14, 2016 at 10:44 PM, roni <ro...@gmail.com>> wrote:
Hi,
I am working with bioinformatics and trying to convert some scripts to sparkR to fit into other spark jobs.
I tries a simple example from a bioinf lib and as soon as I start sparkR environment it does not work.
code as follows -
countData <- matrix(1:100,ncol=4)
condition <- factor(c("A","A","B","B"))
dds <- DESeqDataSetFromMatrix(countData, DataFrame(condition), ~ condition)
Works if i dont initialize the sparkR environment.
if I do library(SparkR) and sqlContext <- sparkRSQL.init(sc) it gives following error
> dds <- DESeqDataSetFromMatrix(countData, as.data.frame(condition), ~ condition)
Error in DataFrame(colData, row.names = rownames(colData)) :
cannot coerce class "data.frame" to a DataFrame
I am really stumped. I am not using any spark function , so i would expect it to work as a simple R code.
why it does not work?
Appreciate the help
-R
--
Alex Kozlov
(408) 507-4987
(650) 887-2135 efax
alexvk@gmail.com<ma...@gmail.com>
Re: sparkR issues ?
Posted by roni <ro...@gmail.com>.
Alex,
No I have not defined he "dataframe" its the spark default Dataframe. That
line is just casting Factor as datarame to send to the function.
Thanks
-R
On Mon, Mar 14, 2016 at 11:58 PM, Alex Kozlov <al...@gmail.com> wrote:
> This seems to be a very unfortunate name collision. SparkR defines it's
> own DataFrame class which shadows what seems to be your own definition.
>
> Is DataFrame something you define? Can you rename it?
>
> On Mon, Mar 14, 2016 at 10:44 PM, roni <ro...@gmail.com> wrote:
>
>> Hi,
>> I am working with bioinformatics and trying to convert some scripts to
>> sparkR to fit into other spark jobs.
>>
>> I tries a simple example from a bioinf lib and as soon as I start sparkR
>> environment it does not work.
>>
>> code as follows -
>> countData <- matrix(1:100,ncol=4)
>> condition <- factor(c("A","A","B","B"))
>> dds <- DESeqDataSetFromMatrix(countData, DataFrame(condition), ~
>> condition)
>>
>> Works if i dont initialize the sparkR environment.
>> if I do library(SparkR) and sqlContext <- sparkRSQL.init(sc) it gives
>> following error
>>
>> > dds <- DESeqDataSetFromMatrix(countData, as.data.frame(condition), ~
>> condition)
>> Error in DataFrame(colData, row.names = rownames(colData)) :
>> cannot coerce class "data.frame" to a DataFrame
>>
>> I am really stumped. I am not using any spark function , so i would
>> expect it to work as a simple R code.
>> why it does not work?
>>
>> Appreciate the help
>> -R
>>
>>
>
>
> --
> Alex Kozlov
> (408) 507-4987
> (650) 887-2135 efax
> alexvk@gmail.com
>
Re: sparkR issues ?
Posted by Alex Kozlov <al...@gmail.com>.
This seems to be a very unfortunate name collision. SparkR defines it's
own DataFrame class which shadows what seems to be your own definition.
Is DataFrame something you define? Can you rename it?
On Mon, Mar 14, 2016 at 10:44 PM, roni <ro...@gmail.com> wrote:
> Hi,
> I am working with bioinformatics and trying to convert some scripts to
> sparkR to fit into other spark jobs.
>
> I tries a simple example from a bioinf lib and as soon as I start sparkR
> environment it does not work.
>
> code as follows -
> countData <- matrix(1:100,ncol=4)
> condition <- factor(c("A","A","B","B"))
> dds <- DESeqDataSetFromMatrix(countData, DataFrame(condition), ~ condition)
>
> Works if i dont initialize the sparkR environment.
> if I do library(SparkR) and sqlContext <- sparkRSQL.init(sc) it gives
> following error
>
> > dds <- DESeqDataSetFromMatrix(countData, as.data.frame(condition), ~
> condition)
> Error in DataFrame(colData, row.names = rownames(colData)) :
> cannot coerce class "data.frame" to a DataFrame
>
> I am really stumped. I am not using any spark function , so i would expect
> it to work as a simple R code.
> why it does not work?
>
> Appreciate the help
> -R
>
>
--
Alex Kozlov
(408) 507-4987
(650) 887-2135 efax
alexvk@gmail.com