You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by rapelly kartheek <ka...@gmail.com> on 2014/10/12 08:15:08 UTC

How to convert a non-rdd data to rdd.

Hi,

I am trying to write a String that is not an rdd to HDFS. This data is a
variable in Spark Scheduler code. None of the spark File operations are
working because my data is not rdd.

So, I tried using SparkContext.parallelize(data). But it throws error:

[error]
/home/karthik/spark-1.0.0/core/src/main/scala/org/apache/spark/storage/BlockManagerMaster.scala:265:
not found: value SparkContext
[error]      SparkContext.parallelize(result)
[error]      ^
[error] one error found

I realized that this data is part of the Scheduler. So, the Sparkcontext
would not have got created yet.

Any help in "writing scheduler variable data to HDFS" is appreciated!!

-Karthik

Re: How to convert a non-rdd data to rdd.

Posted by "Kartheek.R" <ka...@gmail.com>.
Its a variable in spark-1.0.0/*/storagre/BlockManagerMaster.scala  class.
The return data of AskDriverWithReply() method for the getPeers() method.

Basically, it is a Seq[ArrayBuffer]:

ArraySeq(ArrayBuffer(BlockManagerId(1, s1, 47006, 0), BlockManagerId(0, s1,
34625, 0)), ArrayBuffer(BlockManagerId(1, s1, 47006, 0), BlockManagerId(0,
s2, 34625, 0)), ArrayBuffer(BlockManagerId(1, s1, 47006, 0),
BlockManagerId(0, s2, 34625, 0)), ArrayBuffer(BlockManagerId(1, s1, 47006,
0), BlockManagerId(0, s2, 34625, 0)), ArrayBuffer(BlockManagerId(<driver>,
karthik, 51051, 0), BlockManagerId(1, s1, 47006, 0)))


On Sun, Oct 12, 2014 at 12:59 PM, @Sanjiv Singh [via Apache Spark User
List] <ml...@n3.nabble.com> wrote:

> Hi Karthik,
>
> Can you provide us more detail of dataset "data" that you wanted to
> parallelize with
>
> SparkContext.parallelize(data);
>
>
>
>
> Regards,
> Sanjiv Singh
>
>
> Regards
> Sanjiv Singh
> Mob :  +091 9990-447-339
>
> On Sun, Oct 12, 2014 at 11:45 AM, rapelly kartheek <[hidden email]
> <http://user/SendEmail.jtp?type=node&node=16231&i=0>> wrote:
>
>> Hi,
>>
>> I am trying to write a String that is not an rdd to HDFS. This data is a
>> variable in Spark Scheduler code. None of the spark File operations are
>> working because my data is not rdd.
>>
>> So, I tried using SparkContext.parallelize(data). But it throws error:
>>
>> [error]
>> /home/karthik/spark-1.0.0/core/src/main/scala/org/apache/spark/storage/BlockManagerMaster.scala:265:
>> not found: value SparkContext
>> [error]      SparkContext.parallelize(result)
>> [error]      ^
>> [error] one error found
>>
>> I realized that this data is part of the Scheduler. So, the Sparkcontext
>> would not have got created yet.
>>
>> Any help in "writing scheduler variable data to HDFS" is appreciated!!
>>
>> -Karthik
>>
>
>
>
> ------------------------------
>  If you reply to this email, your message will be added to the discussion
> below:
>
> http://apache-spark-user-list.1001560.n3.nabble.com/How-to-convert-a-non-rdd-data-to-rdd-tp16230p16231.html
>  To unsubscribe from How to convert a non-rdd data to rdd., click here
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=16230&code=a2FydGhlZWsubWJtc0BnbWFpbC5jb218MTYyMzB8LTE1NjA1NDM4NDM=>
> .
> NAML
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>




--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-convert-a-non-rdd-data-to-rdd-tp16230p16232.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: How to convert a non-rdd data to rdd.

Posted by rapelly kartheek <ka...@gmail.com>.
Its a variable in spark-1.0.0/*/storagre/BlockManagerMaster.scala  class.
The return data of AskDriverWithReply() method for the getPeers() method.

Basically, it is a Seq[ArrayBuffer]:

ArraySeq(ArrayBuffer(BlockManagerId(1, s1, 47006, 0), BlockManagerId(0, s1,
34625, 0)), ArrayBuffer(BlockManagerId(1, s1, 47006, 0), BlockManagerId(0,
s2, 34625, 0)), ArrayBuffer(BlockManagerId(1, s1, 47006, 0),
BlockManagerId(0, s2, 34625, 0)), ArrayBuffer(BlockManagerId(1, s1, 47006,
0), BlockManagerId(0, s2, 34625, 0)), ArrayBuffer(BlockManagerId(<driver>,
karthik, 51051, 0), BlockManagerId(1, s1, 47006, 0)))


On Sun, Oct 12, 2014 at 12:59 PM, @Sanjiv Singh [via Apache Spark User
List] <ml...@n3.nabble.com> wrote:

> Hi Karthik,
>
> Can you provide us more detail of dataset "data" that you wanted to
> parallelize with
>
> SparkContext.parallelize(data);
>
>
>
>
> Regards,
> Sanjiv Singh
>
>
> Regards
> Sanjiv Singh
> Mob :  +091 9990-447-339
>
> On Sun, Oct 12, 2014 at 11:45 AM, rapelly kartheek <[hidden email]
> <http://user/SendEmail.jtp?type=node&node=16231&i=0>> wrote:
>
>> Hi,
>>
>> I am trying to write a String that is not an rdd to HDFS. This data is a
>> variable in Spark Scheduler code. None of the spark File operations are
>> working because my data is not rdd.
>>
>> So, I tried using SparkContext.parallelize(data). But it throws error:
>>
>> [error]
>> /home/karthik/spark-1.0.0/core/src/main/scala/org/apache/spark/storage/BlockManagerMaster.scala:265:
>> not found: value SparkContext
>> [error]      SparkContext.parallelize(result)
>> [error]      ^
>> [error] one error found
>>
>> I realized that this data is part of the Scheduler. So, the Sparkcontext
>> would not have got created yet.
>>
>> Any help in "writing scheduler variable data to HDFS" is appreciated!!
>>
>> -Karthik
>>
>
>
>
> ------------------------------
>  If you reply to this email, your message will be added to the discussion
> below:
>
> http://apache-spark-user-list.1001560.n3.nabble.com/How-to-convert-a-non-rdd-data-to-rdd-tp16230p16231.html
>  To unsubscribe from How to convert a non-rdd data to rdd., click here
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=16230&code=a2FydGhlZWsubWJtc0BnbWFpbC5jb218MTYyMzB8LTE1NjA1NDM4NDM=>
> .
> NAML
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>

Re: How to convert a non-rdd data to rdd.

Posted by "@Sanjiv Singh" <sa...@gmail.com>.
Hi Karthik,

Can you provide us more detail of dataset "data" that you wanted to
parallelize with

SparkContext.parallelize(data);




Regards,
Sanjiv Singh


Regards
Sanjiv Singh
Mob :  +091 9990-447-339

On Sun, Oct 12, 2014 at 11:45 AM, rapelly kartheek <ka...@gmail.com>
wrote:

> Hi,
>
> I am trying to write a String that is not an rdd to HDFS. This data is a
> variable in Spark Scheduler code. None of the spark File operations are
> working because my data is not rdd.
>
> So, I tried using SparkContext.parallelize(data). But it throws error:
>
> [error]
> /home/karthik/spark-1.0.0/core/src/main/scala/org/apache/spark/storage/BlockManagerMaster.scala:265:
> not found: value SparkContext
> [error]      SparkContext.parallelize(result)
> [error]      ^
> [error] one error found
>
> I realized that this data is part of the Scheduler. So, the Sparkcontext
> would not have got created yet.
>
> Any help in "writing scheduler variable data to HDFS" is appreciated!!
>
> -Karthik
>

Re: How to convert a non-rdd data to rdd.

Posted by "Kartheek.R" <ka...@gmail.com>.
Does SparkContext exists when this part (AskDriverWithReply()) of the
scheduler code gets executed?

On Sun, Oct 12, 2014 at 1:54 PM, rapelly kartheek <ka...@gmail.com>
wrote:

> Hi Sean,
> I tried even with sc as: sc.parallelize(data). But. I get the error: value
> sc not found.
>
> On Sun, Oct 12, 2014 at 1:47 PM, sowen [via Apache Spark User List] <
> ml-node+s1001560n16233h49@n3.nabble.com> wrote:
>
>> It is a method of the class, not a static method of the object. Since a
>> SparkContext is available as sc in the shell, or you have perhaps created
>> one similarly in your app, write sc.parallelize(...)
>> On Oct 12, 2014 7:15 AM, "rapelly kartheek" <[hidden email]
>> <http://user/SendEmail.jtp?type=node&node=16233&i=0>> wrote:
>>
>>> Hi,
>>>
>>> I am trying to write a String that is not an rdd to HDFS. This data is a
>>> variable in Spark Scheduler code. None of the spark File operations are
>>> working because my data is not rdd.
>>>
>>> So, I tried using SparkContext.parallelize(data). But it throws error:
>>>
>>> [error]
>>> /home/karthik/spark-1.0.0/core/src/main/scala/org/apache/spark/storage/BlockManagerMaster.scala:265:
>>> not found: value SparkContext
>>> [error]      SparkContext.parallelize(result)
>>> [error]      ^
>>> [error] one error found
>>>
>>> I realized that this data is part of the Scheduler. So, the Sparkcontext
>>> would not have got created yet.
>>>
>>> Any help in "writing scheduler variable data to HDFS" is appreciated!!
>>>
>>> -Karthik
>>>
>>
>>
>> ------------------------------
>>  If you reply to this email, your message will be added to the
>> discussion below:
>>
>> http://apache-spark-user-list.1001560.n3.nabble.com/How-to-convert-a-non-rdd-data-to-rdd-tp16230p16233.html
>>  To unsubscribe from How to convert a non-rdd data to rdd., click here
>> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=16230&code=a2FydGhlZWsubWJtc0BnbWFpbC5jb218MTYyMzB8LTE1NjA1NDM4NDM=>
>> .
>> NAML
>> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
>
>




--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-convert-a-non-rdd-data-to-rdd-tp16230p16235.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: How to convert a non-rdd data to rdd.

Posted by "Kartheek.R" <ka...@gmail.com>.
Hi Sean,
I tried even with sc as: sc.parallelize(data). But. I get the error: value
sc not found.

On Sun, Oct 12, 2014 at 1:47 PM, sowen [via Apache Spark User List] <
ml-node+s1001560n16233h49@n3.nabble.com> wrote:

> It is a method of the class, not a static method of the object. Since a
> SparkContext is available as sc in the shell, or you have perhaps created
> one similarly in your app, write sc.parallelize(...)
> On Oct 12, 2014 7:15 AM, "rapelly kartheek" <[hidden email]
> <http://user/SendEmail.jtp?type=node&node=16233&i=0>> wrote:
>
>> Hi,
>>
>> I am trying to write a String that is not an rdd to HDFS. This data is a
>> variable in Spark Scheduler code. None of the spark File operations are
>> working because my data is not rdd.
>>
>> So, I tried using SparkContext.parallelize(data). But it throws error:
>>
>> [error]
>> /home/karthik/spark-1.0.0/core/src/main/scala/org/apache/spark/storage/BlockManagerMaster.scala:265:
>> not found: value SparkContext
>> [error]      SparkContext.parallelize(result)
>> [error]      ^
>> [error] one error found
>>
>> I realized that this data is part of the Scheduler. So, the Sparkcontext
>> would not have got created yet.
>>
>> Any help in "writing scheduler variable data to HDFS" is appreciated!!
>>
>> -Karthik
>>
>
>
> ------------------------------
>  If you reply to this email, your message will be added to the discussion
> below:
>
> http://apache-spark-user-list.1001560.n3.nabble.com/How-to-convert-a-non-rdd-data-to-rdd-tp16230p16233.html
>  To unsubscribe from How to convert a non-rdd data to rdd., click here
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=16230&code=a2FydGhlZWsubWJtc0BnbWFpbC5jb218MTYyMzB8LTE1NjA1NDM4NDM=>
> .
> NAML
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>




--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-convert-a-non-rdd-data-to-rdd-tp16230p16234.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: How to convert a non-rdd data to rdd.

Posted by Sean Owen <so...@cloudera.com>.
It is a method of the class, not a static method of the object. Since a
SparkContext is available as sc in the shell, or you have perhaps created
one similarly in your app, write sc.parallelize(...)
On Oct 12, 2014 7:15 AM, "rapelly kartheek" <ka...@gmail.com> wrote:

> Hi,
>
> I am trying to write a String that is not an rdd to HDFS. This data is a
> variable in Spark Scheduler code. None of the spark File operations are
> working because my data is not rdd.
>
> So, I tried using SparkContext.parallelize(data). But it throws error:
>
> [error]
> /home/karthik/spark-1.0.0/core/src/main/scala/org/apache/spark/storage/BlockManagerMaster.scala:265:
> not found: value SparkContext
> [error]      SparkContext.parallelize(result)
> [error]      ^
> [error] one error found
>
> I realized that this data is part of the Scheduler. So, the Sparkcontext
> would not have got created yet.
>
> Any help in "writing scheduler variable data to HDFS" is appreciated!!
>
> -Karthik
>