You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Alejandro Fernandez <af...@hortonworks.com> on 2015/03/10 02:41:42 UTC
Review Request 31878: CopyFromLocal failed to copy Tez tarball to HDFS
failed because multiple processes tried to copy to the same destination
simultaneously
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31878/
-----------------------------------------------------------
Review request for Ambari, Andrew Onischuk, Jonathan Hurley, Nate Cole, and Sid Wagle.
Bugs: AMBARI-9990
https://issues.apache.org/jira/browse/AMBARI-9990
Repository: ambari
Description
-------
Pig Service Check and Hive Server 2 START ran on 2 different machines during the stack installation and failed to copy the tez tarball to HDFS.
I was able to reproduce this locally by calling CopyFromLocal from two clients simultaneously. See the HDFS audit log, datanode logs on c6408 & c6410, and namenode log on c6410.
The copyFromLocal command's behavior is:
* Try to create a temporary file <filename>._COPYING_ and write the real data there
* If hit any exception, delete the file with the name <filename>._COPYING_
Thus we have the following race condition in this test:
Process P1 created file "tez.tar.gz._COPYING_" and wrote data to it
Process P2 fired the same copyFromLocal command and hit exception because it could not get the lease
P2 then deleted the file "tez.tar.gz._COPYING_"
P1 could not close the file "tez.tar.gz._COPYING_" since it had been deleted by P2. The exception would say "could not find lease for file..."
In general we do not have the correct synchronization guarantee for the "copyFromLocal" command.
One solution is for the destination file name to be unique. Because the mv command is synchronized by the namenode, at least one of them will succeed in naming the file.
Diffs
-----
ambari-common/src/main/python/resource_management/libraries/functions/dynamic_variable_interpretation.py 00b8d70
Diff: https://reviews.apache.org/r/31878/diff/
Testing
-------
Unit tests on builds.apache.org passed,
https://builds.apache.org/job/Ambari-trunk-test-patch/1977/
I also deployed a cluster and verified that it was able to copy the tarballs to HDFS when installing YARN, Hive, Pig.
[root@c6408 ~]# su - hdfs -c 'hadoop fs -ls -R /hdp/apps/2.2.2.0-2538/'
dr-xr-xr-x - hdfs hdfs 0 2015-03-10 00:55 /hdp/apps/2.2.2.0-2538/hive
-r--r--r-- 3 hdfs hadoop 82982575 2015-03-10 00:55 /hdp/apps/2.2.2.0-2538/hive/hive.tar.gz
dr-xr-xr-x - hdfs hdfs 0 2015-03-10 00:57 /hdp/apps/2.2.2.0-2538/mapreduce
-r--r--r-- 3 hdfs hadoop 105000 2015-03-10 00:57 /hdp/apps/2.2.2.0-2538/mapreduce/hadoop-streaming.jar
-r--r--r-- 3 hdfs hadoop 192699956 2015-03-09 18:15 /hdp/apps/2.2.2.0-2538/mapreduce/mapreduce.tar.gz
dr-xr-xr-x - hdfs hdfs 0 2015-03-10 00:56 /hdp/apps/2.2.2.0-2538/pig
-r--r--r-- 3 hdfs hadoop 97542246 2015-03-10 00:56 /hdp/apps/2.2.2.0-2538/pig/pig.tar.gz
dr-xr-xr-x - hdfs hdfs 0 2015-03-09 18:15 /hdp/apps/2.2.2.0-2538/tez
-r--r--r-- 3 hdfs hadoop 40656789 2015-03-09 18:15 /hdp/apps/2.2.2.0-2538/tez/tez.tar.gz
Thanks,
Alejandro Fernandez
Re: Review Request 31878: CopyFromLocal failed to copy Tez tarball to
HDFS
failed because multiple processes tried to copy to the same destination
simultaneously
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31878/
-----------------------------------------------------------
(Updated March 11, 2015, 12:36 a.m.)
Review request for Ambari, Andrew Onischuk, Jonathan Hurley, Nate Cole, and Sid Wagle.
Changes
-------
Addressed Jonathan's comment.
Bugs: AMBARI-9990
https://issues.apache.org/jira/browse/AMBARI-9990
Repository: ambari
Description
-------
Pig Service Check and Hive Server 2 START ran on 2 different machines during the stack installation and failed to copy the tez tarball to HDFS.
I was able to reproduce this locally by calling CopyFromLocal from two clients simultaneously. See the HDFS audit log, datanode logs on c6408 & c6410, and namenode log on c6410.
The copyFromLocal command's behavior is:
* Try to create a temporary file <filename>._COPYING_ and write the real data there
* If hit any exception, delete the file with the name <filename>._COPYING_
Thus we have the following race condition in this test:
Process P1 created file "tez.tar.gz._COPYING_" and wrote data to it
Process P2 fired the same copyFromLocal command and hit exception because it could not get the lease
P2 then deleted the file "tez.tar.gz._COPYING_"
P1 could not close the file "tez.tar.gz._COPYING_" since it had been deleted by P2. The exception would say "could not find lease for file..."
In general we do not have the correct synchronization guarantee for the "copyFromLocal" command.
One solution is for the destination file name to be unique. Because the mv command is synchronized by the namenode, at least one of them will succeed in naming the file.
Diffs (updated)
-----
ambari-common/src/main/python/resource_management/libraries/functions/dynamic_variable_interpretation.py 00b8d70
Diff: https://reviews.apache.org/r/31878/diff/
Testing
-------
Unit tests on builds.apache.org passed,
https://builds.apache.org/job/Ambari-trunk-test-patch/1977/
I also deployed a cluster and verified that it was able to copy the tarballs to HDFS when installing YARN, Hive, Pig.
[root@c6408 ~]# su - hdfs -c 'hadoop fs -ls -R /hdp/apps/2.2.2.0-2538/'
dr-xr-xr-x - hdfs hdfs 0 2015-03-10 00:55 /hdp/apps/2.2.2.0-2538/hive
-r--r--r-- 3 hdfs hadoop 82982575 2015-03-10 00:55 /hdp/apps/2.2.2.0-2538/hive/hive.tar.gz
dr-xr-xr-x - hdfs hdfs 0 2015-03-10 00:57 /hdp/apps/2.2.2.0-2538/mapreduce
-r--r--r-- 3 hdfs hadoop 105000 2015-03-10 00:57 /hdp/apps/2.2.2.0-2538/mapreduce/hadoop-streaming.jar
-r--r--r-- 3 hdfs hadoop 192699956 2015-03-09 18:15 /hdp/apps/2.2.2.0-2538/mapreduce/mapreduce.tar.gz
dr-xr-xr-x - hdfs hdfs 0 2015-03-10 00:56 /hdp/apps/2.2.2.0-2538/pig
-r--r--r-- 3 hdfs hadoop 97542246 2015-03-10 00:56 /hdp/apps/2.2.2.0-2538/pig/pig.tar.gz
dr-xr-xr-x - hdfs hdfs 0 2015-03-09 18:15 /hdp/apps/2.2.2.0-2538/tez
-r--r--r-- 3 hdfs hadoop 40656789 2015-03-09 18:15 /hdp/apps/2.2.2.0-2538/tez/tez.tar.gz
Thanks,
Alejandro Fernandez
Re: Review Request 31878: CopyFromLocal failed to copy Tez tarball to
HDFS
failed because multiple processes tried to copy to the same destination
simultaneously
Posted by Nate Cole <nc...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31878/#review75976
-----------------------------------------------------------
Ship it!
ambari-common/src/main/python/resource_management/libraries/functions/dynamic_variable_interpretation.py
<https://reviews.apache.org/r/31878/#comment123333>
I'm assuming this is "atomic enough"
- Nate Cole
On March 9, 2015, 9:41 p.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31878/
> -----------------------------------------------------------
>
> (Updated March 9, 2015, 9:41 p.m.)
>
>
> Review request for Ambari, Andrew Onischuk, Jonathan Hurley, Nate Cole, and Sid Wagle.
>
>
> Bugs: AMBARI-9990
> https://issues.apache.org/jira/browse/AMBARI-9990
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Pig Service Check and Hive Server 2 START ran on 2 different machines during the stack installation and failed to copy the tez tarball to HDFS.
>
> I was able to reproduce this locally by calling CopyFromLocal from two clients simultaneously. See the HDFS audit log, datanode logs on c6408 & c6410, and namenode log on c6410.
>
> The copyFromLocal command's behavior is:
> * Try to create a temporary file <filename>._COPYING_ and write the real data there
> * If hit any exception, delete the file with the name <filename>._COPYING_
>
> Thus we have the following race condition in this test:
> Process P1 created file "tez.tar.gz._COPYING_" and wrote data to it
> Process P2 fired the same copyFromLocal command and hit exception because it could not get the lease
> P2 then deleted the file "tez.tar.gz._COPYING_"
> P1 could not close the file "tez.tar.gz._COPYING_" since it had been deleted by P2. The exception would say "could not find lease for file..."
> In general we do not have the correct synchronization guarantee for the "copyFromLocal" command.
>
> One solution is for the destination file name to be unique. Because the mv command is synchronized by the namenode, at least one of them will succeed in naming the file.
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/dynamic_variable_interpretation.py 00b8d70
>
> Diff: https://reviews.apache.org/r/31878/diff/
>
>
> Testing
> -------
>
> Unit tests on builds.apache.org passed,
> https://builds.apache.org/job/Ambari-trunk-test-patch/1977/
>
> I also deployed a cluster and verified that it was able to copy the tarballs to HDFS when installing YARN, Hive, Pig.
>
> [root@c6408 ~]# su - hdfs -c 'hadoop fs -ls -R /hdp/apps/2.2.2.0-2538/'
> dr-xr-xr-x - hdfs hdfs 0 2015-03-10 00:55 /hdp/apps/2.2.2.0-2538/hive
> -r--r--r-- 3 hdfs hadoop 82982575 2015-03-10 00:55 /hdp/apps/2.2.2.0-2538/hive/hive.tar.gz
> dr-xr-xr-x - hdfs hdfs 0 2015-03-10 00:57 /hdp/apps/2.2.2.0-2538/mapreduce
> -r--r--r-- 3 hdfs hadoop 105000 2015-03-10 00:57 /hdp/apps/2.2.2.0-2538/mapreduce/hadoop-streaming.jar
> -r--r--r-- 3 hdfs hadoop 192699956 2015-03-09 18:15 /hdp/apps/2.2.2.0-2538/mapreduce/mapreduce.tar.gz
> dr-xr-xr-x - hdfs hdfs 0 2015-03-10 00:56 /hdp/apps/2.2.2.0-2538/pig
> -r--r--r-- 3 hdfs hadoop 97542246 2015-03-10 00:56 /hdp/apps/2.2.2.0-2538/pig/pig.tar.gz
> dr-xr-xr-x - hdfs hdfs 0 2015-03-09 18:15 /hdp/apps/2.2.2.0-2538/tez
> -r--r--r-- 3 hdfs hadoop 40656789 2015-03-09 18:15 /hdp/apps/2.2.2.0-2538/tez/tez.tar.gz
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 31878: CopyFromLocal failed to copy Tez tarball to
HDFS
failed because multiple processes tried to copy to the same destination
simultaneously
Posted by Alejandro Fernandez <af...@hortonworks.com>.
> On March 10, 2015, 6:36 p.m., Jonathan Hurley wrote:
> > ambari-common/src/main/python/resource_management/libraries/functions/dynamic_variable_interpretation.py, lines 112-115
> > <https://reviews.apache.org/r/31878/diff/1/?file=889999#file889999line112>
> >
> > That's a lot of code to do something as simple as
> >
> > ```
> > unique_string = str(uuid.uuid4())[:8]
> > ```
> >
> > I know we don't need UUID power here, but it's concise and makes the code cleaner.
Nice, will do.
- Alejandro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31878/#review75924
-----------------------------------------------------------
On March 10, 2015, 1:41 a.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31878/
> -----------------------------------------------------------
>
> (Updated March 10, 2015, 1:41 a.m.)
>
>
> Review request for Ambari, Andrew Onischuk, Jonathan Hurley, Nate Cole, and Sid Wagle.
>
>
> Bugs: AMBARI-9990
> https://issues.apache.org/jira/browse/AMBARI-9990
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Pig Service Check and Hive Server 2 START ran on 2 different machines during the stack installation and failed to copy the tez tarball to HDFS.
>
> I was able to reproduce this locally by calling CopyFromLocal from two clients simultaneously. See the HDFS audit log, datanode logs on c6408 & c6410, and namenode log on c6410.
>
> The copyFromLocal command's behavior is:
> * Try to create a temporary file <filename>._COPYING_ and write the real data there
> * If hit any exception, delete the file with the name <filename>._COPYING_
>
> Thus we have the following race condition in this test:
> Process P1 created file "tez.tar.gz._COPYING_" and wrote data to it
> Process P2 fired the same copyFromLocal command and hit exception because it could not get the lease
> P2 then deleted the file "tez.tar.gz._COPYING_"
> P1 could not close the file "tez.tar.gz._COPYING_" since it had been deleted by P2. The exception would say "could not find lease for file..."
> In general we do not have the correct synchronization guarantee for the "copyFromLocal" command.
>
> One solution is for the destination file name to be unique. Because the mv command is synchronized by the namenode, at least one of them will succeed in naming the file.
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/dynamic_variable_interpretation.py 00b8d70
>
> Diff: https://reviews.apache.org/r/31878/diff/
>
>
> Testing
> -------
>
> Unit tests on builds.apache.org passed,
> https://builds.apache.org/job/Ambari-trunk-test-patch/1977/
>
> I also deployed a cluster and verified that it was able to copy the tarballs to HDFS when installing YARN, Hive, Pig.
>
> [root@c6408 ~]# su - hdfs -c 'hadoop fs -ls -R /hdp/apps/2.2.2.0-2538/'
> dr-xr-xr-x - hdfs hdfs 0 2015-03-10 00:55 /hdp/apps/2.2.2.0-2538/hive
> -r--r--r-- 3 hdfs hadoop 82982575 2015-03-10 00:55 /hdp/apps/2.2.2.0-2538/hive/hive.tar.gz
> dr-xr-xr-x - hdfs hdfs 0 2015-03-10 00:57 /hdp/apps/2.2.2.0-2538/mapreduce
> -r--r--r-- 3 hdfs hadoop 105000 2015-03-10 00:57 /hdp/apps/2.2.2.0-2538/mapreduce/hadoop-streaming.jar
> -r--r--r-- 3 hdfs hadoop 192699956 2015-03-09 18:15 /hdp/apps/2.2.2.0-2538/mapreduce/mapreduce.tar.gz
> dr-xr-xr-x - hdfs hdfs 0 2015-03-10 00:56 /hdp/apps/2.2.2.0-2538/pig
> -r--r--r-- 3 hdfs hadoop 97542246 2015-03-10 00:56 /hdp/apps/2.2.2.0-2538/pig/pig.tar.gz
> dr-xr-xr-x - hdfs hdfs 0 2015-03-09 18:15 /hdp/apps/2.2.2.0-2538/tez
> -r--r--r-- 3 hdfs hadoop 40656789 2015-03-09 18:15 /hdp/apps/2.2.2.0-2538/tez/tez.tar.gz
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 31878: CopyFromLocal failed to copy Tez tarball to
HDFS
failed because multiple processes tried to copy to the same destination
simultaneously
Posted by Jonathan Hurley <jh...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31878/#review75924
-----------------------------------------------------------
Ship it!
I agree that we can let NameNode do the locking here. I don't care if both agents do the same work and the last one in wins.
ambari-common/src/main/python/resource_management/libraries/functions/dynamic_variable_interpretation.py
<https://reviews.apache.org/r/31878/#comment123252>
That's a lot of code to do something as simple as
```
unique_string = str(uuid.uuid4())[:8]
```
I know we don't need UUID power here, but it's concise and makes the code cleaner.
- Jonathan Hurley
On March 9, 2015, 9:41 p.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31878/
> -----------------------------------------------------------
>
> (Updated March 9, 2015, 9:41 p.m.)
>
>
> Review request for Ambari, Andrew Onischuk, Jonathan Hurley, Nate Cole, and Sid Wagle.
>
>
> Bugs: AMBARI-9990
> https://issues.apache.org/jira/browse/AMBARI-9990
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Pig Service Check and Hive Server 2 START ran on 2 different machines during the stack installation and failed to copy the tez tarball to HDFS.
>
> I was able to reproduce this locally by calling CopyFromLocal from two clients simultaneously. See the HDFS audit log, datanode logs on c6408 & c6410, and namenode log on c6410.
>
> The copyFromLocal command's behavior is:
> * Try to create a temporary file <filename>._COPYING_ and write the real data there
> * If hit any exception, delete the file with the name <filename>._COPYING_
>
> Thus we have the following race condition in this test:
> Process P1 created file "tez.tar.gz._COPYING_" and wrote data to it
> Process P2 fired the same copyFromLocal command and hit exception because it could not get the lease
> P2 then deleted the file "tez.tar.gz._COPYING_"
> P1 could not close the file "tez.tar.gz._COPYING_" since it had been deleted by P2. The exception would say "could not find lease for file..."
> In general we do not have the correct synchronization guarantee for the "copyFromLocal" command.
>
> One solution is for the destination file name to be unique. Because the mv command is synchronized by the namenode, at least one of them will succeed in naming the file.
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/dynamic_variable_interpretation.py 00b8d70
>
> Diff: https://reviews.apache.org/r/31878/diff/
>
>
> Testing
> -------
>
> Unit tests on builds.apache.org passed,
> https://builds.apache.org/job/Ambari-trunk-test-patch/1977/
>
> I also deployed a cluster and verified that it was able to copy the tarballs to HDFS when installing YARN, Hive, Pig.
>
> [root@c6408 ~]# su - hdfs -c 'hadoop fs -ls -R /hdp/apps/2.2.2.0-2538/'
> dr-xr-xr-x - hdfs hdfs 0 2015-03-10 00:55 /hdp/apps/2.2.2.0-2538/hive
> -r--r--r-- 3 hdfs hadoop 82982575 2015-03-10 00:55 /hdp/apps/2.2.2.0-2538/hive/hive.tar.gz
> dr-xr-xr-x - hdfs hdfs 0 2015-03-10 00:57 /hdp/apps/2.2.2.0-2538/mapreduce
> -r--r--r-- 3 hdfs hadoop 105000 2015-03-10 00:57 /hdp/apps/2.2.2.0-2538/mapreduce/hadoop-streaming.jar
> -r--r--r-- 3 hdfs hadoop 192699956 2015-03-09 18:15 /hdp/apps/2.2.2.0-2538/mapreduce/mapreduce.tar.gz
> dr-xr-xr-x - hdfs hdfs 0 2015-03-10 00:56 /hdp/apps/2.2.2.0-2538/pig
> -r--r--r-- 3 hdfs hadoop 97542246 2015-03-10 00:56 /hdp/apps/2.2.2.0-2538/pig/pig.tar.gz
> dr-xr-xr-x - hdfs hdfs 0 2015-03-09 18:15 /hdp/apps/2.2.2.0-2538/tez
> -r--r--r-- 3 hdfs hadoop 40656789 2015-03-09 18:15 /hdp/apps/2.2.2.0-2538/tez/tez.tar.gz
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 31878: CopyFromLocal failed to copy Tez tarball to
HDFS
failed because multiple processes tried to copy to the same destination
simultaneously
Posted by Andrew Onischuk <ao...@hortonworks.com>.
> On March 10, 2015, 6:42 p.m., Andrew Onischuk wrote:
> > Alejandro, this will make our deploy longer much, since those fs commands take lot of time on loaded cluster.
> >
> > Can we like set x retries with y delay for CopyFromLocal, in that case we won't increase time of successful deployment.
>
> Andrew Onischuk wrote:
> Meaning this will increase time of every CopyFromLocal x2 or so, and we have a lot of that things called, and they seems to take a lot of time even now.
>
> Alejandro Fernandez wrote:
> What triggers this is YARN History Server START, Hive Sever 2 START, Pig Service Check, and Spark History Server START. This also includes starting after a Rolling Upgrade.
> In all of these cases, we first check if the file is already present in HDFS, so the only performance impact comes from the actual copy the first time.
> I used this approach because it guarantees 100% success. I already wrote a retry-decorator, so I'll see what it takes to make it do a random sleep, but this will require more logic.
What I mean is that any hadoop call takes some time because it initiates jvm, connections, etc. everytime. On loaded machine, 1 hadoop call can somethimes take 15-20sec doesn't matter if it's -copyFromLocal -mv or -ls. We might want to rethink this a little if this is possible. Random sleep is the same thing which takes time. I think we can add to Execute below tries=x, tries_sleep=y, so if it fails due to race condition, it will retry in some time and success?
- Andrew
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31878/#review75927
-----------------------------------------------------------
On March 10, 2015, 1:41 a.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31878/
> -----------------------------------------------------------
>
> (Updated March 10, 2015, 1:41 a.m.)
>
>
> Review request for Ambari, Andrew Onischuk, Jonathan Hurley, Nate Cole, and Sid Wagle.
>
>
> Bugs: AMBARI-9990
> https://issues.apache.org/jira/browse/AMBARI-9990
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Pig Service Check and Hive Server 2 START ran on 2 different machines during the stack installation and failed to copy the tez tarball to HDFS.
>
> I was able to reproduce this locally by calling CopyFromLocal from two clients simultaneously. See the HDFS audit log, datanode logs on c6408 & c6410, and namenode log on c6410.
>
> The copyFromLocal command's behavior is:
> * Try to create a temporary file <filename>._COPYING_ and write the real data there
> * If hit any exception, delete the file with the name <filename>._COPYING_
>
> Thus we have the following race condition in this test:
> Process P1 created file "tez.tar.gz._COPYING_" and wrote data to it
> Process P2 fired the same copyFromLocal command and hit exception because it could not get the lease
> P2 then deleted the file "tez.tar.gz._COPYING_"
> P1 could not close the file "tez.tar.gz._COPYING_" since it had been deleted by P2. The exception would say "could not find lease for file..."
> In general we do not have the correct synchronization guarantee for the "copyFromLocal" command.
>
> One solution is for the destination file name to be unique. Because the mv command is synchronized by the namenode, at least one of them will succeed in naming the file.
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/dynamic_variable_interpretation.py 00b8d70
>
> Diff: https://reviews.apache.org/r/31878/diff/
>
>
> Testing
> -------
>
> Unit tests on builds.apache.org passed,
> https://builds.apache.org/job/Ambari-trunk-test-patch/1977/
>
> I also deployed a cluster and verified that it was able to copy the tarballs to HDFS when installing YARN, Hive, Pig.
>
> [root@c6408 ~]# su - hdfs -c 'hadoop fs -ls -R /hdp/apps/2.2.2.0-2538/'
> dr-xr-xr-x - hdfs hdfs 0 2015-03-10 00:55 /hdp/apps/2.2.2.0-2538/hive
> -r--r--r-- 3 hdfs hadoop 82982575 2015-03-10 00:55 /hdp/apps/2.2.2.0-2538/hive/hive.tar.gz
> dr-xr-xr-x - hdfs hdfs 0 2015-03-10 00:57 /hdp/apps/2.2.2.0-2538/mapreduce
> -r--r--r-- 3 hdfs hadoop 105000 2015-03-10 00:57 /hdp/apps/2.2.2.0-2538/mapreduce/hadoop-streaming.jar
> -r--r--r-- 3 hdfs hadoop 192699956 2015-03-09 18:15 /hdp/apps/2.2.2.0-2538/mapreduce/mapreduce.tar.gz
> dr-xr-xr-x - hdfs hdfs 0 2015-03-10 00:56 /hdp/apps/2.2.2.0-2538/pig
> -r--r--r-- 3 hdfs hadoop 97542246 2015-03-10 00:56 /hdp/apps/2.2.2.0-2538/pig/pig.tar.gz
> dr-xr-xr-x - hdfs hdfs 0 2015-03-09 18:15 /hdp/apps/2.2.2.0-2538/tez
> -r--r--r-- 3 hdfs hadoop 40656789 2015-03-09 18:15 /hdp/apps/2.2.2.0-2538/tez/tez.tar.gz
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 31878: CopyFromLocal failed to copy Tez tarball to
HDFS
failed because multiple processes tried to copy to the same destination
simultaneously
Posted by Andrew Onischuk <ao...@hortonworks.com>.
> On March 10, 2015, 6:42 p.m., Andrew Onischuk wrote:
> > Alejandro, this will make our deploy longer much, since those fs commands take lot of time on loaded cluster.
> >
> > Can we like set x retries with y delay for CopyFromLocal, in that case we won't increase time of successful deployment.
Meaning this will increase time of every CopyFromLocal x2 or so, and we have a lot of that things called, and they seems to take a lot of time even now.
- Andrew
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31878/#review75927
-----------------------------------------------------------
On March 10, 2015, 1:41 a.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31878/
> -----------------------------------------------------------
>
> (Updated March 10, 2015, 1:41 a.m.)
>
>
> Review request for Ambari, Andrew Onischuk, Jonathan Hurley, Nate Cole, and Sid Wagle.
>
>
> Bugs: AMBARI-9990
> https://issues.apache.org/jira/browse/AMBARI-9990
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Pig Service Check and Hive Server 2 START ran on 2 different machines during the stack installation and failed to copy the tez tarball to HDFS.
>
> I was able to reproduce this locally by calling CopyFromLocal from two clients simultaneously. See the HDFS audit log, datanode logs on c6408 & c6410, and namenode log on c6410.
>
> The copyFromLocal command's behavior is:
> * Try to create a temporary file <filename>._COPYING_ and write the real data there
> * If hit any exception, delete the file with the name <filename>._COPYING_
>
> Thus we have the following race condition in this test:
> Process P1 created file "tez.tar.gz._COPYING_" and wrote data to it
> Process P2 fired the same copyFromLocal command and hit exception because it could not get the lease
> P2 then deleted the file "tez.tar.gz._COPYING_"
> P1 could not close the file "tez.tar.gz._COPYING_" since it had been deleted by P2. The exception would say "could not find lease for file..."
> In general we do not have the correct synchronization guarantee for the "copyFromLocal" command.
>
> One solution is for the destination file name to be unique. Because the mv command is synchronized by the namenode, at least one of them will succeed in naming the file.
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/dynamic_variable_interpretation.py 00b8d70
>
> Diff: https://reviews.apache.org/r/31878/diff/
>
>
> Testing
> -------
>
> Unit tests on builds.apache.org passed,
> https://builds.apache.org/job/Ambari-trunk-test-patch/1977/
>
> I also deployed a cluster and verified that it was able to copy the tarballs to HDFS when installing YARN, Hive, Pig.
>
> [root@c6408 ~]# su - hdfs -c 'hadoop fs -ls -R /hdp/apps/2.2.2.0-2538/'
> dr-xr-xr-x - hdfs hdfs 0 2015-03-10 00:55 /hdp/apps/2.2.2.0-2538/hive
> -r--r--r-- 3 hdfs hadoop 82982575 2015-03-10 00:55 /hdp/apps/2.2.2.0-2538/hive/hive.tar.gz
> dr-xr-xr-x - hdfs hdfs 0 2015-03-10 00:57 /hdp/apps/2.2.2.0-2538/mapreduce
> -r--r--r-- 3 hdfs hadoop 105000 2015-03-10 00:57 /hdp/apps/2.2.2.0-2538/mapreduce/hadoop-streaming.jar
> -r--r--r-- 3 hdfs hadoop 192699956 2015-03-09 18:15 /hdp/apps/2.2.2.0-2538/mapreduce/mapreduce.tar.gz
> dr-xr-xr-x - hdfs hdfs 0 2015-03-10 00:56 /hdp/apps/2.2.2.0-2538/pig
> -r--r--r-- 3 hdfs hadoop 97542246 2015-03-10 00:56 /hdp/apps/2.2.2.0-2538/pig/pig.tar.gz
> dr-xr-xr-x - hdfs hdfs 0 2015-03-09 18:15 /hdp/apps/2.2.2.0-2538/tez
> -r--r--r-- 3 hdfs hadoop 40656789 2015-03-09 18:15 /hdp/apps/2.2.2.0-2538/tez/tez.tar.gz
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 31878: CopyFromLocal failed to copy Tez tarball to
HDFS
failed because multiple processes tried to copy to the same destination
simultaneously
Posted by Alejandro Fernandez <af...@hortonworks.com>.
> On March 10, 2015, 6:42 p.m., Andrew Onischuk wrote:
> > Alejandro, this will make our deploy longer much, since those fs commands take lot of time on loaded cluster.
> >
> > Can we like set x retries with y delay for CopyFromLocal, in that case we won't increase time of successful deployment.
>
> Andrew Onischuk wrote:
> Meaning this will increase time of every CopyFromLocal x2 or so, and we have a lot of that things called, and they seems to take a lot of time even now.
>
> Alejandro Fernandez wrote:
> What triggers this is YARN History Server START, Hive Sever 2 START, Pig Service Check, and Spark History Server START. This also includes starting after a Rolling Upgrade.
> In all of these cases, we first check if the file is already present in HDFS, so the only performance impact comes from the actual copy the first time.
> I used this approach because it guarantees 100% success. I already wrote a retry-decorator, so I'll see what it takes to make it do a random sleep, but this will require more logic.
>
> Andrew Onischuk wrote:
> What I mean is that any hadoop call takes some time because it initiates jvm, connections, etc. everytime. On loaded machine, 1 hadoop call can somethimes take 15-20sec doesn't matter if it's -copyFromLocal -mv or -ls. We might want to rethink this a little if this is possible. Random sleep is the same thing which takes time. I think we can add to Execute below tries=x, tries_sleep=y, so if it fails due to race condition, it will retry in some time and success?
These are the possiblities. A = cp and ls (if fails, then retry). B = cp and then mv (guarantees succees)
1. No Race Condition: both strategies will have to execute two hadoop calls.
2. Race Condition.
A. If strategy A fails the first time, it has to keep trying x times, and each time performs it 2 operations, on top of the sleeps.
B. Stategy B only performs 2 operations and the mv is atomic since it is handled by the NameNode
I believe the current implementation is better suited to our needs right now.
- Alejandro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31878/#review75927
-----------------------------------------------------------
On March 10, 2015, 1:41 a.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31878/
> -----------------------------------------------------------
>
> (Updated March 10, 2015, 1:41 a.m.)
>
>
> Review request for Ambari, Andrew Onischuk, Jonathan Hurley, Nate Cole, and Sid Wagle.
>
>
> Bugs: AMBARI-9990
> https://issues.apache.org/jira/browse/AMBARI-9990
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Pig Service Check and Hive Server 2 START ran on 2 different machines during the stack installation and failed to copy the tez tarball to HDFS.
>
> I was able to reproduce this locally by calling CopyFromLocal from two clients simultaneously. See the HDFS audit log, datanode logs on c6408 & c6410, and namenode log on c6410.
>
> The copyFromLocal command's behavior is:
> * Try to create a temporary file <filename>._COPYING_ and write the real data there
> * If hit any exception, delete the file with the name <filename>._COPYING_
>
> Thus we have the following race condition in this test:
> Process P1 created file "tez.tar.gz._COPYING_" and wrote data to it
> Process P2 fired the same copyFromLocal command and hit exception because it could not get the lease
> P2 then deleted the file "tez.tar.gz._COPYING_"
> P1 could not close the file "tez.tar.gz._COPYING_" since it had been deleted by P2. The exception would say "could not find lease for file..."
> In general we do not have the correct synchronization guarantee for the "copyFromLocal" command.
>
> One solution is for the destination file name to be unique. Because the mv command is synchronized by the namenode, at least one of them will succeed in naming the file.
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/dynamic_variable_interpretation.py 00b8d70
>
> Diff: https://reviews.apache.org/r/31878/diff/
>
>
> Testing
> -------
>
> Unit tests on builds.apache.org passed,
> https://builds.apache.org/job/Ambari-trunk-test-patch/1977/
>
> I also deployed a cluster and verified that it was able to copy the tarballs to HDFS when installing YARN, Hive, Pig.
>
> [root@c6408 ~]# su - hdfs -c 'hadoop fs -ls -R /hdp/apps/2.2.2.0-2538/'
> dr-xr-xr-x - hdfs hdfs 0 2015-03-10 00:55 /hdp/apps/2.2.2.0-2538/hive
> -r--r--r-- 3 hdfs hadoop 82982575 2015-03-10 00:55 /hdp/apps/2.2.2.0-2538/hive/hive.tar.gz
> dr-xr-xr-x - hdfs hdfs 0 2015-03-10 00:57 /hdp/apps/2.2.2.0-2538/mapreduce
> -r--r--r-- 3 hdfs hadoop 105000 2015-03-10 00:57 /hdp/apps/2.2.2.0-2538/mapreduce/hadoop-streaming.jar
> -r--r--r-- 3 hdfs hadoop 192699956 2015-03-09 18:15 /hdp/apps/2.2.2.0-2538/mapreduce/mapreduce.tar.gz
> dr-xr-xr-x - hdfs hdfs 0 2015-03-10 00:56 /hdp/apps/2.2.2.0-2538/pig
> -r--r--r-- 3 hdfs hadoop 97542246 2015-03-10 00:56 /hdp/apps/2.2.2.0-2538/pig/pig.tar.gz
> dr-xr-xr-x - hdfs hdfs 0 2015-03-09 18:15 /hdp/apps/2.2.2.0-2538/tez
> -r--r--r-- 3 hdfs hadoop 40656789 2015-03-09 18:15 /hdp/apps/2.2.2.0-2538/tez/tez.tar.gz
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 31878: CopyFromLocal failed to copy Tez tarball to
HDFS
failed because multiple processes tried to copy to the same destination
simultaneously
Posted by Alejandro Fernandez <af...@hortonworks.com>.
> On March 10, 2015, 6:42 p.m., Andrew Onischuk wrote:
> > Alejandro, this will make our deploy longer much, since those fs commands take lot of time on loaded cluster.
> >
> > Can we like set x retries with y delay for CopyFromLocal, in that case we won't increase time of successful deployment.
>
> Andrew Onischuk wrote:
> Meaning this will increase time of every CopyFromLocal x2 or so, and we have a lot of that things called, and they seems to take a lot of time even now.
What triggers this is YARN History Server START, Hive Sever 2 START, Pig Service Check, and Spark History Server START. This also includes starting after a Rolling Upgrade.
In all of these cases, we first check if the file is already present in HDFS, so the only performance impact comes from the actual copy the first time.
I used this approach because it guarantees 100% success. I already wrote a retry-decorator, so I'll see what it takes to make it do a random sleep, but this will require more logic.
- Alejandro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31878/#review75927
-----------------------------------------------------------
On March 10, 2015, 1:41 a.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31878/
> -----------------------------------------------------------
>
> (Updated March 10, 2015, 1:41 a.m.)
>
>
> Review request for Ambari, Andrew Onischuk, Jonathan Hurley, Nate Cole, and Sid Wagle.
>
>
> Bugs: AMBARI-9990
> https://issues.apache.org/jira/browse/AMBARI-9990
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Pig Service Check and Hive Server 2 START ran on 2 different machines during the stack installation and failed to copy the tez tarball to HDFS.
>
> I was able to reproduce this locally by calling CopyFromLocal from two clients simultaneously. See the HDFS audit log, datanode logs on c6408 & c6410, and namenode log on c6410.
>
> The copyFromLocal command's behavior is:
> * Try to create a temporary file <filename>._COPYING_ and write the real data there
> * If hit any exception, delete the file with the name <filename>._COPYING_
>
> Thus we have the following race condition in this test:
> Process P1 created file "tez.tar.gz._COPYING_" and wrote data to it
> Process P2 fired the same copyFromLocal command and hit exception because it could not get the lease
> P2 then deleted the file "tez.tar.gz._COPYING_"
> P1 could not close the file "tez.tar.gz._COPYING_" since it had been deleted by P2. The exception would say "could not find lease for file..."
> In general we do not have the correct synchronization guarantee for the "copyFromLocal" command.
>
> One solution is for the destination file name to be unique. Because the mv command is synchronized by the namenode, at least one of them will succeed in naming the file.
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/dynamic_variable_interpretation.py 00b8d70
>
> Diff: https://reviews.apache.org/r/31878/diff/
>
>
> Testing
> -------
>
> Unit tests on builds.apache.org passed,
> https://builds.apache.org/job/Ambari-trunk-test-patch/1977/
>
> I also deployed a cluster and verified that it was able to copy the tarballs to HDFS when installing YARN, Hive, Pig.
>
> [root@c6408 ~]# su - hdfs -c 'hadoop fs -ls -R /hdp/apps/2.2.2.0-2538/'
> dr-xr-xr-x - hdfs hdfs 0 2015-03-10 00:55 /hdp/apps/2.2.2.0-2538/hive
> -r--r--r-- 3 hdfs hadoop 82982575 2015-03-10 00:55 /hdp/apps/2.2.2.0-2538/hive/hive.tar.gz
> dr-xr-xr-x - hdfs hdfs 0 2015-03-10 00:57 /hdp/apps/2.2.2.0-2538/mapreduce
> -r--r--r-- 3 hdfs hadoop 105000 2015-03-10 00:57 /hdp/apps/2.2.2.0-2538/mapreduce/hadoop-streaming.jar
> -r--r--r-- 3 hdfs hadoop 192699956 2015-03-09 18:15 /hdp/apps/2.2.2.0-2538/mapreduce/mapreduce.tar.gz
> dr-xr-xr-x - hdfs hdfs 0 2015-03-10 00:56 /hdp/apps/2.2.2.0-2538/pig
> -r--r--r-- 3 hdfs hadoop 97542246 2015-03-10 00:56 /hdp/apps/2.2.2.0-2538/pig/pig.tar.gz
> dr-xr-xr-x - hdfs hdfs 0 2015-03-09 18:15 /hdp/apps/2.2.2.0-2538/tez
> -r--r--r-- 3 hdfs hadoop 40656789 2015-03-09 18:15 /hdp/apps/2.2.2.0-2538/tez/tez.tar.gz
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 31878: CopyFromLocal failed to copy Tez tarball to
HDFS
failed because multiple processes tried to copy to the same destination
simultaneously
Posted by Andrew Onischuk <ao...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31878/#review75927
-----------------------------------------------------------
Alejandro, this will make our deploy longer much, since those fs commands take lot of time on loaded cluster.
Can we like set x retries with y delay for CopyFromLocal, in that case we won't increase time of successful deployment.
- Andrew Onischuk
On March 10, 2015, 1:41 a.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31878/
> -----------------------------------------------------------
>
> (Updated March 10, 2015, 1:41 a.m.)
>
>
> Review request for Ambari, Andrew Onischuk, Jonathan Hurley, Nate Cole, and Sid Wagle.
>
>
> Bugs: AMBARI-9990
> https://issues.apache.org/jira/browse/AMBARI-9990
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Pig Service Check and Hive Server 2 START ran on 2 different machines during the stack installation and failed to copy the tez tarball to HDFS.
>
> I was able to reproduce this locally by calling CopyFromLocal from two clients simultaneously. See the HDFS audit log, datanode logs on c6408 & c6410, and namenode log on c6410.
>
> The copyFromLocal command's behavior is:
> * Try to create a temporary file <filename>._COPYING_ and write the real data there
> * If hit any exception, delete the file with the name <filename>._COPYING_
>
> Thus we have the following race condition in this test:
> Process P1 created file "tez.tar.gz._COPYING_" and wrote data to it
> Process P2 fired the same copyFromLocal command and hit exception because it could not get the lease
> P2 then deleted the file "tez.tar.gz._COPYING_"
> P1 could not close the file "tez.tar.gz._COPYING_" since it had been deleted by P2. The exception would say "could not find lease for file..."
> In general we do not have the correct synchronization guarantee for the "copyFromLocal" command.
>
> One solution is for the destination file name to be unique. Because the mv command is synchronized by the namenode, at least one of them will succeed in naming the file.
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/dynamic_variable_interpretation.py 00b8d70
>
> Diff: https://reviews.apache.org/r/31878/diff/
>
>
> Testing
> -------
>
> Unit tests on builds.apache.org passed,
> https://builds.apache.org/job/Ambari-trunk-test-patch/1977/
>
> I also deployed a cluster and verified that it was able to copy the tarballs to HDFS when installing YARN, Hive, Pig.
>
> [root@c6408 ~]# su - hdfs -c 'hadoop fs -ls -R /hdp/apps/2.2.2.0-2538/'
> dr-xr-xr-x - hdfs hdfs 0 2015-03-10 00:55 /hdp/apps/2.2.2.0-2538/hive
> -r--r--r-- 3 hdfs hadoop 82982575 2015-03-10 00:55 /hdp/apps/2.2.2.0-2538/hive/hive.tar.gz
> dr-xr-xr-x - hdfs hdfs 0 2015-03-10 00:57 /hdp/apps/2.2.2.0-2538/mapreduce
> -r--r--r-- 3 hdfs hadoop 105000 2015-03-10 00:57 /hdp/apps/2.2.2.0-2538/mapreduce/hadoop-streaming.jar
> -r--r--r-- 3 hdfs hadoop 192699956 2015-03-09 18:15 /hdp/apps/2.2.2.0-2538/mapreduce/mapreduce.tar.gz
> dr-xr-xr-x - hdfs hdfs 0 2015-03-10 00:56 /hdp/apps/2.2.2.0-2538/pig
> -r--r--r-- 3 hdfs hadoop 97542246 2015-03-10 00:56 /hdp/apps/2.2.2.0-2538/pig/pig.tar.gz
> dr-xr-xr-x - hdfs hdfs 0 2015-03-09 18:15 /hdp/apps/2.2.2.0-2538/tez
> -r--r--r-- 3 hdfs hadoop 40656789 2015-03-09 18:15 /hdp/apps/2.2.2.0-2538/tez/tez.tar.gz
>
>
> Thanks,
>
> Alejandro Fernandez
>
>