You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Pradeep Bashyal <pr...@bashyal.com> on 2015/06/29 18:40:33 UTC

[SparkR] Missing Spark APIs in R

Hello,

I noticed that some of the spark-core APIs are not available with version
1.4.0 release of SparkR. For example textFile(), flatMap() etc. The code
seems to be there but is not exported in NAMESPACE. They were all available
as part of the AmpLab Extras previously. I wasn't able to find any
explanations of why they were not included with the release.

Can anyone shed some light on it?

Thanks
Pradeep

Re: [SparkR] Missing Spark APIs in R

Posted by Shivaram Venkataraman <sh...@eecs.berkeley.edu>.

The Spark JIRA and the user, dev mailing lists are the best place to follow
the progress.

Shivaram

On Tue, Jun 30, 2015 at 9:52 AM, Pradeep Bashyal <pr...@bashyal.com>
wrote:

> Thanks Shivaram. I watched your talk and the plan to use ML APIs with R
> flavor looks exciting.
> Is there a different venue where I would be able to follow the SparkR API
> progress?
>
> Thanks
> Pradeep
>
> On Mon, Jun 29, 2015 at 1:12 PM, Shivaram Venkataraman <
> shivaram@eecs.berkeley.edu> wrote:
>
>> The RDD API is pretty complex and we are not yet sure we want to export
>> all those methods in the SparkR API. We are working towards exposing a more
>> limited API in upcoming versions. You can find some more details in the
>> recent Spark Summit talk at
>> https://spark-summit.org/2015/events/sparkr-the-past-the-present-and-the-future/
>>
>> Thanks
>> Shivaram
>>
>> On Mon, Jun 29, 2015 at 9:40 AM, Pradeep Bashyal <pr...@bashyal.com>
>> wrote:
>>
>>> Hello,
>>>
>>> I noticed that some of the spark-core APIs are not available with
>>> version 1.4.0 release of SparkR. For example textFile(), flatMap() etc. The
>>> code seems to be there but is not exported in NAMESPACE. They were all
>>> available as part of the AmpLab Extras previously. I wasn't able to find
>>> any explanations of why they were not included with the release.
>>>
>>> Can anyone shed some light on it?
>>>
>>> Thanks
>>> Pradeep
>>>
>>
>>
>

Re: [SparkR] Missing Spark APIs in R

Posted by Pradeep Bashyal <pr...@bashyal.com>.

Thanks Shivaram. I watched your talk and the plan to use ML APIs with R
flavor looks exciting.
Is there a different venue where I would be able to follow the SparkR API
progress?

Thanks
Pradeep

On Mon, Jun 29, 2015 at 1:12 PM, Shivaram Venkataraman <
shivaram@eecs.berkeley.edu> wrote:

> The RDD API is pretty complex and we are not yet sure we want to export
> all those methods in the SparkR API. We are working towards exposing a more
> limited API in upcoming versions. You can find some more details in the
> recent Spark Summit talk at
> https://spark-summit.org/2015/events/sparkr-the-past-the-present-and-the-future/
>
> Thanks
> Shivaram
>
> On Mon, Jun 29, 2015 at 9:40 AM, Pradeep Bashyal <pr...@bashyal.com>
> wrote:
>
>> Hello,
>>
>> I noticed that some of the spark-core APIs are not available with version
>> 1.4.0 release of SparkR. For example textFile(), flatMap() etc. The code
>> seems to be there but is not exported in NAMESPACE. They were all available
>> as part of the AmpLab Extras previously. I wasn't able to find any
>> explanations of why they were not included with the release.
>>
>> Can anyone shed some light on it?
>>
>> Thanks
>> Pradeep
>>
>
>

Re: [SparkR] Missing Spark APIs in R

Posted by Shivaram Venkataraman <sh...@eecs.berkeley.edu>.

The RDD API is pretty complex and we are not yet sure we want to export all
those methods in the SparkR API. We are working towards exposing a more
limited API in upcoming versions. You can find some more details in the
recent Spark Summit talk at
https://spark-summit.org/2015/events/sparkr-the-past-the-present-and-the-future/

Thanks
Shivaram

On Mon, Jun 29, 2015 at 9:40 AM, Pradeep Bashyal <pr...@bashyal.com>
wrote:

> Hello,
>
> I noticed that some of the spark-core APIs are not available with version
> 1.4.0 release of SparkR. For example textFile(), flatMap() etc. The code
> seems to be there but is not exported in NAMESPACE. They were all available
> as part of the AmpLab Extras previously. I wasn't able to find any
> explanations of why they were not included with the release.
>
> Can anyone shed some light on it?
>
> Thanks
> Pradeep
>