You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by rapelly kartheek <ka...@gmail.com> on 2014/10/12 08:15:08 UTC
How to convert a non-rdd data to rdd.
Hi,
I am trying to write a String that is not an rdd to HDFS. This data is a
variable in Spark Scheduler code. None of the spark File operations are
working because my data is not rdd.
So, I tried using SparkContext.parallelize(data). But it throws error:
[error]
/home/karthik/spark-1.0.0/core/src/main/scala/org/apache/spark/storage/BlockManagerMaster.scala:265:
not found: value SparkContext
[error] SparkContext.parallelize(result)
[error] ^
[error] one error found
I realized that this data is part of the Scheduler. So, the Sparkcontext
would not have got created yet.
Any help in "writing scheduler variable data to HDFS" is appreciated!!
-Karthik
Re: How to convert a non-rdd data to rdd.
Posted by "Kartheek.R" <ka...@gmail.com>.
Its a variable in spark-1.0.0/*/storagre/BlockManagerMaster.scala class.
The return data of AskDriverWithReply() method for the getPeers() method.
Basically, it is a Seq[ArrayBuffer]:
ArraySeq(ArrayBuffer(BlockManagerId(1, s1, 47006, 0), BlockManagerId(0, s1,
34625, 0)), ArrayBuffer(BlockManagerId(1, s1, 47006, 0), BlockManagerId(0,
s2, 34625, 0)), ArrayBuffer(BlockManagerId(1, s1, 47006, 0),
BlockManagerId(0, s2, 34625, 0)), ArrayBuffer(BlockManagerId(1, s1, 47006,
0), BlockManagerId(0, s2, 34625, 0)), ArrayBuffer(BlockManagerId(<driver>,
karthik, 51051, 0), BlockManagerId(1, s1, 47006, 0)))
On Sun, Oct 12, 2014 at 12:59 PM, @Sanjiv Singh [via Apache Spark User
List] <ml...@n3.nabble.com> wrote:
> Hi Karthik,
>
> Can you provide us more detail of dataset "data" that you wanted to
> parallelize with
>
> SparkContext.parallelize(data);
>
>
>
>
> Regards,
> Sanjiv Singh
>
>
> Regards
> Sanjiv Singh
> Mob : +091 9990-447-339
>
> On Sun, Oct 12, 2014 at 11:45 AM, rapelly kartheek <[hidden email]
> <http://user/SendEmail.jtp?type=node&node=16231&i=0>> wrote:
>
>> Hi,
>>
>> I am trying to write a String that is not an rdd to HDFS. This data is a
>> variable in Spark Scheduler code. None of the spark File operations are
>> working because my data is not rdd.
>>
>> So, I tried using SparkContext.parallelize(data). But it throws error:
>>
>> [error]
>> /home/karthik/spark-1.0.0/core/src/main/scala/org/apache/spark/storage/BlockManagerMaster.scala:265:
>> not found: value SparkContext
>> [error] SparkContext.parallelize(result)
>> [error] ^
>> [error] one error found
>>
>> I realized that this data is part of the Scheduler. So, the Sparkcontext
>> would not have got created yet.
>>
>> Any help in "writing scheduler variable data to HDFS" is appreciated!!
>>
>> -Karthik
>>
>
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://apache-spark-user-list.1001560.n3.nabble.com/How-to-convert-a-non-rdd-data-to-rdd-tp16230p16231.html
> To unsubscribe from How to convert a non-rdd data to rdd., click here
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=16230&code=a2FydGhlZWsubWJtc0BnbWFpbC5jb218MTYyMzB8LTE1NjA1NDM4NDM=>
> .
> NAML
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-convert-a-non-rdd-data-to-rdd-tp16230p16232.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: How to convert a non-rdd data to rdd.
Posted by rapelly kartheek <ka...@gmail.com>.
Its a variable in spark-1.0.0/*/storagre/BlockManagerMaster.scala class.
The return data of AskDriverWithReply() method for the getPeers() method.
Basically, it is a Seq[ArrayBuffer]:
ArraySeq(ArrayBuffer(BlockManagerId(1, s1, 47006, 0), BlockManagerId(0, s1,
34625, 0)), ArrayBuffer(BlockManagerId(1, s1, 47006, 0), BlockManagerId(0,
s2, 34625, 0)), ArrayBuffer(BlockManagerId(1, s1, 47006, 0),
BlockManagerId(0, s2, 34625, 0)), ArrayBuffer(BlockManagerId(1, s1, 47006,
0), BlockManagerId(0, s2, 34625, 0)), ArrayBuffer(BlockManagerId(<driver>,
karthik, 51051, 0), BlockManagerId(1, s1, 47006, 0)))
On Sun, Oct 12, 2014 at 12:59 PM, @Sanjiv Singh [via Apache Spark User
List] <ml...@n3.nabble.com> wrote:
> Hi Karthik,
>
> Can you provide us more detail of dataset "data" that you wanted to
> parallelize with
>
> SparkContext.parallelize(data);
>
>
>
>
> Regards,
> Sanjiv Singh
>
>
> Regards
> Sanjiv Singh
> Mob : +091 9990-447-339
>
> On Sun, Oct 12, 2014 at 11:45 AM, rapelly kartheek <[hidden email]
> <http://user/SendEmail.jtp?type=node&node=16231&i=0>> wrote:
>
>> Hi,
>>
>> I am trying to write a String that is not an rdd to HDFS. This data is a
>> variable in Spark Scheduler code. None of the spark File operations are
>> working because my data is not rdd.
>>
>> So, I tried using SparkContext.parallelize(data). But it throws error:
>>
>> [error]
>> /home/karthik/spark-1.0.0/core/src/main/scala/org/apache/spark/storage/BlockManagerMaster.scala:265:
>> not found: value SparkContext
>> [error] SparkContext.parallelize(result)
>> [error] ^
>> [error] one error found
>>
>> I realized that this data is part of the Scheduler. So, the Sparkcontext
>> would not have got created yet.
>>
>> Any help in "writing scheduler variable data to HDFS" is appreciated!!
>>
>> -Karthik
>>
>
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://apache-spark-user-list.1001560.n3.nabble.com/How-to-convert-a-non-rdd-data-to-rdd-tp16230p16231.html
> To unsubscribe from How to convert a non-rdd data to rdd., click here
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=16230&code=a2FydGhlZWsubWJtc0BnbWFpbC5jb218MTYyMzB8LTE1NjA1NDM4NDM=>
> .
> NAML
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
Re: How to convert a non-rdd data to rdd.
Posted by "@Sanjiv Singh" <sa...@gmail.com>.
Hi Karthik,
Can you provide us more detail of dataset "data" that you wanted to
parallelize with
SparkContext.parallelize(data);
Regards,
Sanjiv Singh
Regards
Sanjiv Singh
Mob : +091 9990-447-339
On Sun, Oct 12, 2014 at 11:45 AM, rapelly kartheek <ka...@gmail.com>
wrote:
> Hi,
>
> I am trying to write a String that is not an rdd to HDFS. This data is a
> variable in Spark Scheduler code. None of the spark File operations are
> working because my data is not rdd.
>
> So, I tried using SparkContext.parallelize(data). But it throws error:
>
> [error]
> /home/karthik/spark-1.0.0/core/src/main/scala/org/apache/spark/storage/BlockManagerMaster.scala:265:
> not found: value SparkContext
> [error] SparkContext.parallelize(result)
> [error] ^
> [error] one error found
>
> I realized that this data is part of the Scheduler. So, the Sparkcontext
> would not have got created yet.
>
> Any help in "writing scheduler variable data to HDFS" is appreciated!!
>
> -Karthik
>
Re: How to convert a non-rdd data to rdd.
Posted by "Kartheek.R" <ka...@gmail.com>.
Does SparkContext exists when this part (AskDriverWithReply()) of the
scheduler code gets executed?
On Sun, Oct 12, 2014 at 1:54 PM, rapelly kartheek <ka...@gmail.com>
wrote:
> Hi Sean,
> I tried even with sc as: sc.parallelize(data). But. I get the error: value
> sc not found.
>
> On Sun, Oct 12, 2014 at 1:47 PM, sowen [via Apache Spark User List] <
> ml-node+s1001560n16233h49@n3.nabble.com> wrote:
>
>> It is a method of the class, not a static method of the object. Since a
>> SparkContext is available as sc in the shell, or you have perhaps created
>> one similarly in your app, write sc.parallelize(...)
>> On Oct 12, 2014 7:15 AM, "rapelly kartheek" <[hidden email]
>> <http://user/SendEmail.jtp?type=node&node=16233&i=0>> wrote:
>>
>>> Hi,
>>>
>>> I am trying to write a String that is not an rdd to HDFS. This data is a
>>> variable in Spark Scheduler code. None of the spark File operations are
>>> working because my data is not rdd.
>>>
>>> So, I tried using SparkContext.parallelize(data). But it throws error:
>>>
>>> [error]
>>> /home/karthik/spark-1.0.0/core/src/main/scala/org/apache/spark/storage/BlockManagerMaster.scala:265:
>>> not found: value SparkContext
>>> [error] SparkContext.parallelize(result)
>>> [error] ^
>>> [error] one error found
>>>
>>> I realized that this data is part of the Scheduler. So, the Sparkcontext
>>> would not have got created yet.
>>>
>>> Any help in "writing scheduler variable data to HDFS" is appreciated!!
>>>
>>> -Karthik
>>>
>>
>>
>> ------------------------------
>> If you reply to this email, your message will be added to the
>> discussion below:
>>
>> http://apache-spark-user-list.1001560.n3.nabble.com/How-to-convert-a-non-rdd-data-to-rdd-tp16230p16233.html
>> To unsubscribe from How to convert a non-rdd data to rdd., click here
>> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=16230&code=a2FydGhlZWsubWJtc0BnbWFpbC5jb218MTYyMzB8LTE1NjA1NDM4NDM=>
>> .
>> NAML
>> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
>
>
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-convert-a-non-rdd-data-to-rdd-tp16230p16235.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: How to convert a non-rdd data to rdd.
Posted by "Kartheek.R" <ka...@gmail.com>.
Hi Sean,
I tried even with sc as: sc.parallelize(data). But. I get the error: value
sc not found.
On Sun, Oct 12, 2014 at 1:47 PM, sowen [via Apache Spark User List] <
ml-node+s1001560n16233h49@n3.nabble.com> wrote:
> It is a method of the class, not a static method of the object. Since a
> SparkContext is available as sc in the shell, or you have perhaps created
> one similarly in your app, write sc.parallelize(...)
> On Oct 12, 2014 7:15 AM, "rapelly kartheek" <[hidden email]
> <http://user/SendEmail.jtp?type=node&node=16233&i=0>> wrote:
>
>> Hi,
>>
>> I am trying to write a String that is not an rdd to HDFS. This data is a
>> variable in Spark Scheduler code. None of the spark File operations are
>> working because my data is not rdd.
>>
>> So, I tried using SparkContext.parallelize(data). But it throws error:
>>
>> [error]
>> /home/karthik/spark-1.0.0/core/src/main/scala/org/apache/spark/storage/BlockManagerMaster.scala:265:
>> not found: value SparkContext
>> [error] SparkContext.parallelize(result)
>> [error] ^
>> [error] one error found
>>
>> I realized that this data is part of the Scheduler. So, the Sparkcontext
>> would not have got created yet.
>>
>> Any help in "writing scheduler variable data to HDFS" is appreciated!!
>>
>> -Karthik
>>
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://apache-spark-user-list.1001560.n3.nabble.com/How-to-convert-a-non-rdd-data-to-rdd-tp16230p16233.html
> To unsubscribe from How to convert a non-rdd data to rdd., click here
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=16230&code=a2FydGhlZWsubWJtc0BnbWFpbC5jb218MTYyMzB8LTE1NjA1NDM4NDM=>
> .
> NAML
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-convert-a-non-rdd-data-to-rdd-tp16230p16234.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: How to convert a non-rdd data to rdd.
Posted by Sean Owen <so...@cloudera.com>.
It is a method of the class, not a static method of the object. Since a
SparkContext is available as sc in the shell, or you have perhaps created
one similarly in your app, write sc.parallelize(...)
On Oct 12, 2014 7:15 AM, "rapelly kartheek" <ka...@gmail.com> wrote:
> Hi,
>
> I am trying to write a String that is not an rdd to HDFS. This data is a
> variable in Spark Scheduler code. None of the spark File operations are
> working because my data is not rdd.
>
> So, I tried using SparkContext.parallelize(data). But it throws error:
>
> [error]
> /home/karthik/spark-1.0.0/core/src/main/scala/org/apache/spark/storage/BlockManagerMaster.scala:265:
> not found: value SparkContext
> [error] SparkContext.parallelize(result)
> [error] ^
> [error] one error found
>
> I realized that this data is part of the Scheduler. So, the Sparkcontext
> would not have got created yet.
>
> Any help in "writing scheduler variable data to HDFS" is appreciated!!
>
> -Karthik
>