You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by rapelly kartheek <ka...@gmail.com> on 2014/09/19 10:02:46 UTC
rsync problem
Hi,
I'd made some modifications to the spark source code in the master and
reflected them to the slaves using rsync.
I followed this command:
rsync -avL --progress path/to/spark-1.0.0 username@destinationhostname
:path/to/destdirectory.
This worked perfectly. But, I wanted to simultaneously rsync all the
slaves. So, added the other slaves as following:
rsync -avL --progress path/to/spark-1.0.0 username@destinationhostname
:path/to/destdirectory username@slave2:path username@slave3:path and so on.
But this didn't work. Anyway, for now, I did it individually for each node.
Can someone give me the right syntax.
Secondly, after this rsync, I find that my cluster has become tremendously
slow!!!
Sometimes the cluster is just shutting down. Job execution is not happening.
Can someone throw some light on this aspect.
thank you
Karthik
Re: rsync problem
Posted by rapelly kartheek <ka...@gmail.com>.
Hi,
This is the command I am using for submitting my application, SimpleApp:
./bin/spark-submit --class org.apache.spark.examples.SimpleApp
--deploy-mode client --master spark://karthik:7077
$SPARK_HOME/examples/*/scala-*/spark-examples-*.jar /text-data
On Thu, Sep 25, 2014 at 6:52 AM, Tobias Pfeiffer <tg...@preferred.jp> wrote:
> Hi,
>
> I assume you unintentionally did not reply to the list, so I'm adding it
> back to CC.
>
> How do you submit your job to the cluster?
>
> Tobias
>
>
> On Thu, Sep 25, 2014 at 2:21 AM, rapelly kartheek <kartheek.mbms@gmail.com
> > wrote:
>
>> How do I find out whether a node in the cluster is a master or slave??
>> Till now I was thinking that slaves file under the conf folder makes the
>> difference. Also, the MASTER_MASTER_IP in the spark-env.sh file.
>>
>> what else differentiates a slave from the master??
>>
>> On Wed, Sep 24, 2014 at 10:46 PM, rapelly kartheek <
>> kartheek.mbms@gmail.com> wrote:
>>
>>> The job execution is taking place perfectly. Previously, all my print
>>> statements used to be stored in spark/work/*/stdout file. But, now after
>>> doing the rsync, I find that none of the prtint statements are getting
>>> reflected in the stdout file under work folder. But, when I go to the code,
>>> I find the statements in the code. But, they are not reflected into the
>>> stdout file as before.
>>>
>>> Can you please tell me where I went wrong. All I want is to see my
>>> mofication in the code getting relected in output
>>> .
>>>
>>> On Wed, Sep 24, 2014 at 10:22 PM, rapelly kartheek <
>>> kartheek.mbms@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> I have a very important and fundamental doubt: I have rsynced the
>>>> entire spark folder from the master to all slaves in the cluster. When I
>>>> execute a job, its working perfectly. But, when I rsync the entire spark
>>>> folder of the master to all the slaves, is it not that I am sending the
>>>> master configurations to all the slaves and making the slaves behave like
>>>> master??
>>>>
>>>> First of all, is it correct to rsync the entire spark folder??
>>>> But, if I change only one file, then how do I rsync it to all??
>>>>
>>>> On Fri, Sep 19, 2014 at 8:44 PM, rapelly kartheek <
>>>> kartheek.mbms@gmail.com> wrote:
>>>>
>>>>> Thank you Soumya Simantha and Tobias. I've deleted the contents of the
>>>>> work folder in all the nodes.
>>>>> Now its working perfectly as it was before.
>>>>>
>>>>> Thank you
>>>>> Karthik
>>>>>
>>>>> On Fri, Sep 19, 2014 at 4:46 PM, Soumya Simanta <
>>>>> soumya.simanta@gmail.com> wrote:
>>>>>
>>>>>> One possible reason is maybe that the checkpointing directory
>>>>>> $SPARK_HOME/work is rsynced as well.
>>>>>> Try emptying the contents of the work folder on each node and try
>>>>>> again.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, Sep 19, 2014 at 4:53 AM, rapelly kartheek <
>>>>>> kartheek.mbms@gmail.com> wrote:
>>>>>>
>>>>>>> I
>>>>>>> * followed this command:rsync -avL --progress path/to/spark-1.0.0
>>>>>>> username@destinationhostname:*
>>>>>>>
>>>>>>>
>>>>>>> *path/to/destdirectory. Anyway, for now, I did it individually for
>>>>>>> each node.*
>>>>>>>
>>>>>>> I have copied to each node at a time individually using the above
>>>>>>> command. So, I guess the copying may not contain any mixture of files.
>>>>>>> Also, as of now, I am not facing any MethodNotFound exceptions. But, there
>>>>>>> is no job execution taking place.
>>>>>>>
>>>>>>> After sometime, one by one, each goes down and the cluster shuts
>>>>>>> down.
>>>>>>>
>>>>>>> On Fri, Sep 19, 2014 at 2:15 PM, Tobias Pfeiffer <tg...@preferred.jp>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> On Fri, Sep 19, 2014 at 5:17 PM, rapelly kartheek <
>>>>>>>> kartheek.mbms@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> > ,
>>>>>>>>>
>>>>>>>>> * you have copied a lot of files from various hosts to
>>>>>>>>> username@slave3:path*
>>>>>>>>> only from one node to all the other nodes...
>>>>>>>>>
>>>>>>>>
>>>>>>>> I don't think rsync can do that in one command as you described. My
>>>>>>>> guess is that now you have a wild mixture of jar files all across your
>>>>>>>> cluster which will lead to fancy exceptions like MethodNotFound etc.,
>>>>>>>> that's maybe why your cluster is not working correctly.
>>>>>>>>
>>>>>>>> Tobias
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: rsync problem
Posted by Tobias Pfeiffer <tg...@preferred.jp>.
Hi,
I assume you unintentionally did not reply to the list, so I'm adding it
back to CC.
How do you submit your job to the cluster?
Tobias
On Thu, Sep 25, 2014 at 2:21 AM, rapelly kartheek <ka...@gmail.com>
wrote:
> How do I find out whether a node in the cluster is a master or slave??
> Till now I was thinking that slaves file under the conf folder makes the
> difference. Also, the MASTER_MASTER_IP in the spark-env.sh file.
>
> what else differentiates a slave from the master??
>
> On Wed, Sep 24, 2014 at 10:46 PM, rapelly kartheek <
> kartheek.mbms@gmail.com> wrote:
>
>> The job execution is taking place perfectly. Previously, all my print
>> statements used to be stored in spark/work/*/stdout file. But, now after
>> doing the rsync, I find that none of the prtint statements are getting
>> reflected in the stdout file under work folder. But, when I go to the code,
>> I find the statements in the code. But, they are not reflected into the
>> stdout file as before.
>>
>> Can you please tell me where I went wrong. All I want is to see my
>> mofication in the code getting relected in output
>> .
>>
>> On Wed, Sep 24, 2014 at 10:22 PM, rapelly kartheek <
>> kartheek.mbms@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I have a very important and fundamental doubt: I have rsynced the entire
>>> spark folder from the master to all slaves in the cluster. When I execute a
>>> job, its working perfectly. But, when I rsync the entire spark folder of
>>> the master to all the slaves, is it not that I am sending the master
>>> configurations to all the slaves and making the slaves behave like master??
>>>
>>> First of all, is it correct to rsync the entire spark folder??
>>> But, if I change only one file, then how do I rsync it to all??
>>>
>>> On Fri, Sep 19, 2014 at 8:44 PM, rapelly kartheek <
>>> kartheek.mbms@gmail.com> wrote:
>>>
>>>> Thank you Soumya Simantha and Tobias. I've deleted the contents of the
>>>> work folder in all the nodes.
>>>> Now its working perfectly as it was before.
>>>>
>>>> Thank you
>>>> Karthik
>>>>
>>>> On Fri, Sep 19, 2014 at 4:46 PM, Soumya Simanta <
>>>> soumya.simanta@gmail.com> wrote:
>>>>
>>>>> One possible reason is maybe that the checkpointing directory
>>>>> $SPARK_HOME/work is rsynced as well.
>>>>> Try emptying the contents of the work folder on each node and try
>>>>> again.
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Sep 19, 2014 at 4:53 AM, rapelly kartheek <
>>>>> kartheek.mbms@gmail.com> wrote:
>>>>>
>>>>>> I
>>>>>> * followed this command:rsync -avL --progress path/to/spark-1.0.0
>>>>>> username@destinationhostname:*
>>>>>>
>>>>>>
>>>>>> *path/to/destdirectory. Anyway, for now, I did it individually for
>>>>>> each node.*
>>>>>>
>>>>>> I have copied to each node at a time individually using the above
>>>>>> command. So, I guess the copying may not contain any mixture of files.
>>>>>> Also, as of now, I am not facing any MethodNotFound exceptions. But, there
>>>>>> is no job execution taking place.
>>>>>>
>>>>>> After sometime, one by one, each goes down and the cluster shuts
>>>>>> down.
>>>>>>
>>>>>> On Fri, Sep 19, 2014 at 2:15 PM, Tobias Pfeiffer <tg...@preferred.jp>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> On Fri, Sep 19, 2014 at 5:17 PM, rapelly kartheek <
>>>>>>> kartheek.mbms@gmail.com> wrote:
>>>>>>>
>>>>>>>> > ,
>>>>>>>>
>>>>>>>> * you have copied a lot of files from various hosts to
>>>>>>>> username@slave3:path*
>>>>>>>> only from one node to all the other nodes...
>>>>>>>>
>>>>>>>
>>>>>>> I don't think rsync can do that in one command as you described. My
>>>>>>> guess is that now you have a wild mixture of jar files all across your
>>>>>>> cluster which will lead to fancy exceptions like MethodNotFound etc.,
>>>>>>> that's maybe why your cluster is not working correctly.
>>>>>>>
>>>>>>> Tobias
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: rsync problem
Posted by rapelly kartheek <ka...@gmail.com>.
Thank you Soumya Simantha and Tobias. I've deleted the contents of the work
folder in all the nodes.
Now its working perfectly as it was before.
Thank you
Karthik
On Fri, Sep 19, 2014 at 4:46 PM, Soumya Simanta <so...@gmail.com>
wrote:
> One possible reason is maybe that the checkpointing directory
> $SPARK_HOME/work is rsynced as well.
> Try emptying the contents of the work folder on each node and try again.
>
>
>
> On Fri, Sep 19, 2014 at 4:53 AM, rapelly kartheek <kartheek.mbms@gmail.com
> > wrote:
>
>> I
>> * followed this command:rsync -avL --progress path/to/spark-1.0.0
>> username@destinationhostname:*
>>
>>
>> *path/to/destdirectory. Anyway, for now, I did it individually for each
>> node.*
>>
>> I have copied to each node at a time individually using the above
>> command. So, I guess the copying may not contain any mixture of files.
>> Also, as of now, I am not facing any MethodNotFound exceptions. But, there
>> is no job execution taking place.
>>
>> After sometime, one by one, each goes down and the cluster shuts down.
>>
>> On Fri, Sep 19, 2014 at 2:15 PM, Tobias Pfeiffer <tg...@preferred.jp>
>> wrote:
>>
>>> Hi,
>>>
>>> On Fri, Sep 19, 2014 at 5:17 PM, rapelly kartheek <
>>> kartheek.mbms@gmail.com> wrote:
>>>
>>>> > ,
>>>>
>>>> * you have copied a lot of files from various hosts to
>>>> username@slave3:path*
>>>> only from one node to all the other nodes...
>>>>
>>>
>>> I don't think rsync can do that in one command as you described. My
>>> guess is that now you have a wild mixture of jar files all across your
>>> cluster which will lead to fancy exceptions like MethodNotFound etc.,
>>> that's maybe why your cluster is not working correctly.
>>>
>>> Tobias
>>>
>>>
>>>
>>
>
Re: rsync problem
Posted by Soumya Simanta <so...@gmail.com>.
One possible reason is maybe that the checkpointing directory
$SPARK_HOME/work is rsynced as well.
Try emptying the contents of the work folder on each node and try again.
On Fri, Sep 19, 2014 at 4:53 AM, rapelly kartheek <ka...@gmail.com>
wrote:
> I
> * followed this command:rsync -avL --progress path/to/spark-1.0.0
> username@destinationhostname:*
>
>
> *path/to/destdirectory. Anyway, for now, I did it individually for each
> node.*
>
> I have copied to each node at a time individually using the above command.
> So, I guess the copying may not contain any mixture of files. Also, as of
> now, I am not facing any MethodNotFound exceptions. But, there is no job
> execution taking place.
>
> After sometime, one by one, each goes down and the cluster shuts down.
>
> On Fri, Sep 19, 2014 at 2:15 PM, Tobias Pfeiffer <tg...@preferred.jp> wrote:
>
>> Hi,
>>
>> On Fri, Sep 19, 2014 at 5:17 PM, rapelly kartheek <
>> kartheek.mbms@gmail.com> wrote:
>>
>>> > ,
>>>
>>> * you have copied a lot of files from various hosts to
>>> username@slave3:path*
>>> only from one node to all the other nodes...
>>>
>>
>> I don't think rsync can do that in one command as you described. My guess
>> is that now you have a wild mixture of jar files all across your cluster
>> which will lead to fancy exceptions like MethodNotFound etc., that's maybe
>> why your cluster is not working correctly.
>>
>> Tobias
>>
>>
>>
>
Re: rsync problem
Posted by rapelly kartheek <ka...@gmail.com>.
I
* followed this command:rsync -avL --progress path/to/spark-1.0.0
username@destinationhostname:*
*path/to/destdirectory. Anyway, for now, I did it individually for each
node.*
I have copied to each node at a time individually using the above command.
So, I guess the copying may not contain any mixture of files. Also, as of
now, I am not facing any MethodNotFound exceptions. But, there is no job
execution taking place.
After sometime, one by one, each goes down and the cluster shuts down.
On Fri, Sep 19, 2014 at 2:15 PM, Tobias Pfeiffer <tg...@preferred.jp> wrote:
> Hi,
>
> On Fri, Sep 19, 2014 at 5:17 PM, rapelly kartheek <kartheek.mbms@gmail.com
> > wrote:
>
>> > ,
>>
>> * you have copied a lot of files from various hosts to
>> username@slave3:path*
>> only from one node to all the other nodes...
>>
>
> I don't think rsync can do that in one command as you described. My guess
> is that now you have a wild mixture of jar files all across your cluster
> which will lead to fancy exceptions like MethodNotFound etc., that's maybe
> why your cluster is not working correctly.
>
> Tobias
>
>
>
Re: rsync problem
Posted by Tobias Pfeiffer <tg...@preferred.jp>.
Hi,
On Fri, Sep 19, 2014 at 5:17 PM, rapelly kartheek <ka...@gmail.com>
wrote:
> > ,
>
> * you have copied a lot of files from various hosts to
> username@slave3:path*
> only from one node to all the other nodes...
>
I don't think rsync can do that in one command as you described. My guess
is that now you have a wild mixture of jar files all across your cluster
which will lead to fancy exceptions like MethodNotFound etc., that's maybe
why your cluster is not working correctly.
Tobias
Fwd: rsync problem
Posted by rapelly kartheek <ka...@gmail.com>.
---------- Forwarded message ----------
From: rapelly kartheek <ka...@gmail.com>
Date: Fri, Sep 19, 2014 at 1:51 PM
Subject: Re: rsync problem
To: Tobias Pfeiffer <tg...@preferred.jp>
any idea why the cluster is dying down???
On Fri, Sep 19, 2014 at 1:47 PM, rapelly kartheek <ka...@gmail.com>
wrote:
> ,
>
>
> * you have copied a lot of files from various hosts to
> username@slave3:path*
> only from one node to all the other nodes...
>
> On Fri, Sep 19, 2014 at 1:45 PM, rapelly kartheek <kartheek.mbms@gmail.com
> > wrote:
>
>> Hi Tobias,
>>
>> I've copied the files from master to all the slaves.
>>
>> On Fri, Sep 19, 2014 at 1:37 PM, Tobias Pfeiffer <tg...@preferred.jp>
>> wrote:
>>
>>> Hi,
>>>
>>> On Fri, Sep 19, 2014 at 5:02 PM, rapelly kartheek <
>>> kartheek.mbms@gmail.com> wrote:
>>>>
>>>> This worked perfectly. But, I wanted to simultaneously rsync all the
>>>> slaves. So, added the other slaves as following:
>>>>
>>>> rsync -avL --progress path/to/spark-1.0.0 username@destinationhostname
>>>> :path/to/destdirectory username@slave2:path username@slave3:path and
>>>> so on.
>>>>
>>>
>>> The rsync man page says
>>> rsync [OPTION...] SRC... [USER@]HOST:DEST
>>> so as I understand your command, you have copied a lot of files from
>>> various hosts to username@slave3:path. I don't think rsync can copy to
>>> various locations at once.
>>>
>>> Tobias
>>>
>>>
>>
>
Re: rsync problem
Posted by rapelly kartheek <ka...@gmail.com>.
,
* you have copied a lot of files from various hosts to username@slave3:path*
only from one node to all the other nodes...
On Fri, Sep 19, 2014 at 1:45 PM, rapelly kartheek <ka...@gmail.com>
wrote:
> Hi Tobias,
>
> I've copied the files from master to all the slaves.
>
> On Fri, Sep 19, 2014 at 1:37 PM, Tobias Pfeiffer <tg...@preferred.jp> wrote:
>
>> Hi,
>>
>> On Fri, Sep 19, 2014 at 5:02 PM, rapelly kartheek <
>> kartheek.mbms@gmail.com> wrote:
>>>
>>> This worked perfectly. But, I wanted to simultaneously rsync all the
>>> slaves. So, added the other slaves as following:
>>>
>>> rsync -avL --progress path/to/spark-1.0.0 username@destinationhostname
>>> :path/to/destdirectory username@slave2:path username@slave3:path and so
>>> on.
>>>
>>
>> The rsync man page says
>> rsync [OPTION...] SRC... [USER@]HOST:DEST
>> so as I understand your command, you have copied a lot of files from
>> various hosts to username@slave3:path. I don't think rsync can copy to
>> various locations at once.
>>
>> Tobias
>>
>>
>
Re: rsync problem
Posted by rapelly kartheek <ka...@gmail.com>.
Hi Tobias,
I've copied the files from master to all the slaves.
On Fri, Sep 19, 2014 at 1:37 PM, Tobias Pfeiffer <tg...@preferred.jp> wrote:
> Hi,
>
> On Fri, Sep 19, 2014 at 5:02 PM, rapelly kartheek <kartheek.mbms@gmail.com
> > wrote:
>>
>> This worked perfectly. But, I wanted to simultaneously rsync all the
>> slaves. So, added the other slaves as following:
>>
>> rsync -avL --progress path/to/spark-1.0.0 username@destinationhostname
>> :path/to/destdirectory username@slave2:path username@slave3:path and so
>> on.
>>
>
> The rsync man page says
> rsync [OPTION...] SRC... [USER@]HOST:DEST
> so as I understand your command, you have copied a lot of files from
> various hosts to username@slave3:path. I don't think rsync can copy to
> various locations at once.
>
> Tobias
>
>
Re: rsync problem
Posted by Tobias Pfeiffer <tg...@preferred.jp>.
Hi,
On Fri, Sep 19, 2014 at 5:02 PM, rapelly kartheek <ka...@gmail.com>
wrote:
>
> This worked perfectly. But, I wanted to simultaneously rsync all the
> slaves. So, added the other slaves as following:
>
> rsync -avL --progress path/to/spark-1.0.0 username@destinationhostname
> :path/to/destdirectory username@slave2:path username@slave3:path and so
> on.
>
The rsync man page says
rsync [OPTION...] SRC... [USER@]HOST:DEST
so as I understand your command, you have copied a lot of files from
various hosts to username@slave3:path. I don't think rsync can copy to
various locations at once.
Tobias