You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by anshu shukla <an...@gmail.com> on 2015/06/22 19:20:29 UTC
Multiple executors writing file using java filewriter
Can not we write some data to a txt file in parallel with multiple
executors running in parallel ??
--
Thanks & Regards,
Anshu Shukla
Re: Multiple executors writing file using java filewriter
Posted by Akhil Das <ak...@sigmoidanalytics.com>.
Then apply a transformation over the dstream to pull those required
information. :)
Thanks
Best Regards
On Tue, Jun 23, 2015 at 3:22 PM, anshu shukla <an...@gmail.com>
wrote:
> Thanks alot ,
>
> Because i just want to log timestamp and unique message id and not full
> RDD .
>
> On Tue, Jun 23, 2015 at 12:41 PM, Akhil Das <ak...@sigmoidanalytics.com>
> wrote:
>
>> Why don't you do a normal .saveAsTextFiles?
>>
>> Thanks
>> Best Regards
>>
>> On Mon, Jun 22, 2015 at 11:55 PM, anshu shukla <an...@gmail.com>
>> wrote:
>>
>>> Thanx for reply !!
>>>
>>> YES , Either it should write on any machine of cluster or Can you
>>> please help me ... that how to do this . Previously i was using writing
>>> using collect () , so some of my tuples are missing while writing.
>>>
>>> //previous logic that was just creating the file on master -
>>>
>>> newinputStream.foreachRDD(new Function2<JavaRDD<String>, Time, Void>()
>>> {
>>>
>>> @Override
>>> public Void call(JavaRDD<String> v1, Time v2) throws Exception {
>>> for(String s:v1.collect()) {
>>> // System.out.println("v1 here is " + v1 + "-------" + s);
>>> spoutlog.batchLogwriter(System.currentTimeMillis(), "spout-MSGID," + msgeditor.getMessageId(s));
>>> // System.out.println(msgeditor.getMessageId(s));
>>> }
>>> return null;
>>> }
>>> });
>>>
>>>
>>>
>>>
>>> On Mon, Jun 22, 2015 at 11:31 PM, Richard Marscher <
>>> rmarscher@localytics.com> wrote:
>>>
>>>> Is spoutLog just a non-spark file writer? If you run that in the map
>>>> call on a cluster its going to be writing in the filesystem of the executor
>>>> its being run on. I'm not sure if that's what you intended.
>>>>
>>>> On Mon, Jun 22, 2015 at 1:35 PM, anshu shukla <an...@gmail.com>
>>>> wrote:
>>>>
>>>>> Running perfectly in local system but not writing to file in cluster mode .ANY suggestions please ..
>>>>>
>>>>>
>>>>> //msgid is long counter
>>>>>
>>>>> JavaDStream<String> newinputStream=inputStream.map(new Function<String, String>() {
>>>>> @Override
>>>>> public String call(String v1) throws Exception {
>>>>> String s1=msgId+"@"+v1;
>>>>> System.out.println(s1);
>>>>> msgId++;
>>>>> try {
>>>>> *//filewriter logic spoutlog.batchLogwriter(System.currentTimeMillis(), "spout-MSGID," + msgeditor.getMessageId(s1));*
>>>>> } catch (Exception e) {
>>>>>
>>>>> System.out.println("exeception is here");
>>>>> e.printStackTrace();
>>>>> throw e;
>>>>> }
>>>>> System.out.println("msgid,"+msgId);
>>>>> return msgeditor.addMessageId(v1,msgId);
>>>>> }
>>>>> });
>>>>>
>>>>>
>>>>> --
>>>>> Thanks & Regards,
>>>>> Anshu Shukla
>>>>>
>>>>> On Mon, Jun 22, 2015 at 10:50 PM, anshu shukla <anshushukla0@gmail.com
>>>>> > wrote:
>>>>>
>>>>>> Can not we write some data to a txt file in parallel with multiple
>>>>>> executors running in parallel ??
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Thanks & Regards,
>>>>>> Anshu Shukla
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Thanks & Regards,
>>>>> Anshu Shukla
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Thanks & Regards,
>>> Anshu Shukla
>>>
>>
>>
>
>
> --
> Thanks & Regards,
> Anshu Shukla
>
Re: Multiple executors writing file using java filewriter
Posted by anshu shukla <an...@gmail.com>.
Thanks alot ,
Because i just want to log timestamp and unique message id and not full
RDD .
On Tue, Jun 23, 2015 at 12:41 PM, Akhil Das <ak...@sigmoidanalytics.com>
wrote:
> Why don't you do a normal .saveAsTextFiles?
>
> Thanks
> Best Regards
>
> On Mon, Jun 22, 2015 at 11:55 PM, anshu shukla <an...@gmail.com>
> wrote:
>
>> Thanx for reply !!
>>
>> YES , Either it should write on any machine of cluster or Can you
>> please help me ... that how to do this . Previously i was using writing
>> using collect () , so some of my tuples are missing while writing.
>>
>> //previous logic that was just creating the file on master -
>>
>> newinputStream.foreachRDD(new Function2<JavaRDD<String>, Time, Void>() {
>>
>> @Override
>> public Void call(JavaRDD<String> v1, Time v2) throws Exception {
>> for(String s:v1.collect()) {
>> // System.out.println("v1 here is " + v1 + "-------" + s);
>> spoutlog.batchLogwriter(System.currentTimeMillis(), "spout-MSGID," + msgeditor.getMessageId(s));
>> // System.out.println(msgeditor.getMessageId(s));
>> }
>> return null;
>> }
>> });
>>
>>
>>
>>
>> On Mon, Jun 22, 2015 at 11:31 PM, Richard Marscher <
>> rmarscher@localytics.com> wrote:
>>
>>> Is spoutLog just a non-spark file writer? If you run that in the map
>>> call on a cluster its going to be writing in the filesystem of the executor
>>> its being run on. I'm not sure if that's what you intended.
>>>
>>> On Mon, Jun 22, 2015 at 1:35 PM, anshu shukla <an...@gmail.com>
>>> wrote:
>>>
>>>> Running perfectly in local system but not writing to file in cluster mode .ANY suggestions please ..
>>>>
>>>>
>>>> //msgid is long counter
>>>>
>>>> JavaDStream<String> newinputStream=inputStream.map(new Function<String, String>() {
>>>> @Override
>>>> public String call(String v1) throws Exception {
>>>> String s1=msgId+"@"+v1;
>>>> System.out.println(s1);
>>>> msgId++;
>>>> try {
>>>> *//filewriter logic spoutlog.batchLogwriter(System.currentTimeMillis(), "spout-MSGID," + msgeditor.getMessageId(s1));*
>>>> } catch (Exception e) {
>>>>
>>>> System.out.println("exeception is here");
>>>> e.printStackTrace();
>>>> throw e;
>>>> }
>>>> System.out.println("msgid,"+msgId);
>>>> return msgeditor.addMessageId(v1,msgId);
>>>> }
>>>> });
>>>>
>>>>
>>>> --
>>>> Thanks & Regards,
>>>> Anshu Shukla
>>>>
>>>> On Mon, Jun 22, 2015 at 10:50 PM, anshu shukla <an...@gmail.com>
>>>> wrote:
>>>>
>>>>> Can not we write some data to a txt file in parallel with multiple
>>>>> executors running in parallel ??
>>>>>
>>>>>
>>>>> --
>>>>> Thanks & Regards,
>>>>> Anshu Shukla
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Thanks & Regards,
>>>> Anshu Shukla
>>>>
>>>
>>>
>>
>>
>> --
>> Thanks & Regards,
>> Anshu Shukla
>>
>
>
--
Thanks & Regards,
Anshu Shukla
Re: Multiple executors writing file using java filewriter
Posted by Akhil Das <ak...@sigmoidanalytics.com>.
Why don't you do a normal .saveAsTextFiles?
Thanks
Best Regards
On Mon, Jun 22, 2015 at 11:55 PM, anshu shukla <an...@gmail.com>
wrote:
> Thanx for reply !!
>
> YES , Either it should write on any machine of cluster or Can you please
> help me ... that how to do this . Previously i was using writing using
> collect () , so some of my tuples are missing while writing.
>
> //previous logic that was just creating the file on master -
>
> newinputStream.foreachRDD(new Function2<JavaRDD<String>, Time, Void>() {
>
> @Override
> public Void call(JavaRDD<String> v1, Time v2) throws Exception {
> for(String s:v1.collect()) {
> // System.out.println("v1 here is " + v1 + "-------" + s);
> spoutlog.batchLogwriter(System.currentTimeMillis(), "spout-MSGID," + msgeditor.getMessageId(s));
> // System.out.println(msgeditor.getMessageId(s));
> }
> return null;
> }
> });
>
>
>
>
> On Mon, Jun 22, 2015 at 11:31 PM, Richard Marscher <
> rmarscher@localytics.com> wrote:
>
>> Is spoutLog just a non-spark file writer? If you run that in the map call
>> on a cluster its going to be writing in the filesystem of the executor its
>> being run on. I'm not sure if that's what you intended.
>>
>> On Mon, Jun 22, 2015 at 1:35 PM, anshu shukla <an...@gmail.com>
>> wrote:
>>
>>> Running perfectly in local system but not writing to file in cluster mode .ANY suggestions please ..
>>>
>>>
>>> //msgid is long counter
>>>
>>> JavaDStream<String> newinputStream=inputStream.map(new Function<String, String>() {
>>> @Override
>>> public String call(String v1) throws Exception {
>>> String s1=msgId+"@"+v1;
>>> System.out.println(s1);
>>> msgId++;
>>> try {
>>> *//filewriter logic spoutlog.batchLogwriter(System.currentTimeMillis(), "spout-MSGID," + msgeditor.getMessageId(s1));*
>>> } catch (Exception e) {
>>>
>>> System.out.println("exeception is here");
>>> e.printStackTrace();
>>> throw e;
>>> }
>>> System.out.println("msgid,"+msgId);
>>> return msgeditor.addMessageId(v1,msgId);
>>> }
>>> });
>>>
>>>
>>> --
>>> Thanks & Regards,
>>> Anshu Shukla
>>>
>>> On Mon, Jun 22, 2015 at 10:50 PM, anshu shukla <an...@gmail.com>
>>> wrote:
>>>
>>>> Can not we write some data to a txt file in parallel with multiple
>>>> executors running in parallel ??
>>>>
>>>>
>>>> --
>>>> Thanks & Regards,
>>>> Anshu Shukla
>>>>
>>>
>>>
>>>
>>> --
>>> Thanks & Regards,
>>> Anshu Shukla
>>>
>>
>>
>
>
> --
> Thanks & Regards,
> Anshu Shukla
>
Re: Multiple executors writing file using java filewriter
Posted by anshu shukla <an...@gmail.com>.
Thanx for reply !!
YES , Either it should write on any machine of cluster or Can you please
help me ... that how to do this . Previously i was using writing using
collect () , so some of my tuples are missing while writing.
//previous logic that was just creating the file on master -
newinputStream.foreachRDD(new Function2<JavaRDD<String>, Time, Void>() {
@Override
public Void call(JavaRDD<String> v1, Time v2) throws Exception {
for(String s:v1.collect()) {
// System.out.println("v1 here is " + v1 + "-------" + s);
spoutlog.batchLogwriter(System.currentTimeMillis(), "spout-MSGID," +
msgeditor.getMessageId(s));
// System.out.println(msgeditor.getMessageId(s));
}
return null;
}
});
On Mon, Jun 22, 2015 at 11:31 PM, Richard Marscher <rmarscher@localytics.com
> wrote:
> Is spoutLog just a non-spark file writer? If you run that in the map call
> on a cluster its going to be writing in the filesystem of the executor its
> being run on. I'm not sure if that's what you intended.
>
> On Mon, Jun 22, 2015 at 1:35 PM, anshu shukla <an...@gmail.com>
> wrote:
>
>> Running perfectly in local system but not writing to file in cluster mode .ANY suggestions please ..
>>
>>
>> //msgid is long counter
>>
>> JavaDStream<String> newinputStream=inputStream.map(new Function<String, String>() {
>> @Override
>> public String call(String v1) throws Exception {
>> String s1=msgId+"@"+v1;
>> System.out.println(s1);
>> msgId++;
>> try {
>> *//filewriter logic spoutlog.batchLogwriter(System.currentTimeMillis(), "spout-MSGID," + msgeditor.getMessageId(s1));*
>> } catch (Exception e) {
>>
>> System.out.println("exeception is here");
>> e.printStackTrace();
>> throw e;
>> }
>> System.out.println("msgid,"+msgId);
>> return msgeditor.addMessageId(v1,msgId);
>> }
>> });
>>
>>
>> --
>> Thanks & Regards,
>> Anshu Shukla
>>
>> On Mon, Jun 22, 2015 at 10:50 PM, anshu shukla <an...@gmail.com>
>> wrote:
>>
>>> Can not we write some data to a txt file in parallel with multiple
>>> executors running in parallel ??
>>>
>>>
>>> --
>>> Thanks & Regards,
>>> Anshu Shukla
>>>
>>
>>
>>
>> --
>> Thanks & Regards,
>> Anshu Shukla
>>
>
>
--
Thanks & Regards,
Anshu Shukla
Re: Multiple executors writing file using java filewriter
Posted by Richard Marscher <rm...@localytics.com>.
Is spoutLog just a non-spark file writer? If you run that in the map call
on a cluster its going to be writing in the filesystem of the executor its
being run on. I'm not sure if that's what you intended.
On Mon, Jun 22, 2015 at 1:35 PM, anshu shukla <an...@gmail.com>
wrote:
> Running perfectly in local system but not writing to file in cluster mode .ANY suggestions please ..
>
>
> //msgid is long counter
>
> JavaDStream<String> newinputStream=inputStream.map(new Function<String, String>() {
> @Override
> public String call(String v1) throws Exception {
> String s1=msgId+"@"+v1;
> System.out.println(s1);
> msgId++;
> try {
> *//filewriter logic spoutlog.batchLogwriter(System.currentTimeMillis(), "spout-MSGID," + msgeditor.getMessageId(s1));*
> } catch (Exception e) {
>
> System.out.println("exeception is here");
> e.printStackTrace();
> throw e;
> }
> System.out.println("msgid,"+msgId);
> return msgeditor.addMessageId(v1,msgId);
> }
> });
>
>
> --
> Thanks & Regards,
> Anshu Shukla
>
> On Mon, Jun 22, 2015 at 10:50 PM, anshu shukla <an...@gmail.com>
> wrote:
>
>> Can not we write some data to a txt file in parallel with multiple
>> executors running in parallel ??
>>
>>
>> --
>> Thanks & Regards,
>> Anshu Shukla
>>
>
>
>
> --
> Thanks & Regards,
> Anshu Shukla
>
Re: Multiple executors writing file using java filewriter
Posted by anshu shukla <an...@gmail.com>.
Running perfectly in local system but not writing to file in cluster
mode .ANY suggestions please ..
//msgid is long counter
JavaDStream<String> newinputStream=inputStream.map(new
Function<String, String>() {
@Override
public String call(String v1) throws Exception {
String s1=msgId+"@"+v1;
System.out.println(s1);
msgId++;
try {
*//filewriter logic
spoutlog.batchLogwriter(System.currentTimeMillis(), "spout-MSGID," +
msgeditor.getMessageId(s1));*
} catch (Exception e) {
System.out.println("exeception is here");
e.printStackTrace();
throw e;
}
System.out.println("msgid,"+msgId);
return msgeditor.addMessageId(v1,msgId);
}
});
--
Thanks & Regards,
Anshu Shukla
On Mon, Jun 22, 2015 at 10:50 PM, anshu shukla <an...@gmail.com>
wrote:
> Can not we write some data to a txt file in parallel with multiple
> executors running in parallel ??
>
>
> --
> Thanks & Regards,
> Anshu Shukla
>
--
Thanks & Regards,
Anshu Shukla