You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by ch huang <ju...@gmail.com> on 2013/12/04 02:26:27 UTC
issue about the MR JOB local dir
hi,maillist:
i see three dirs on my local MR job dir ,and i do not know
these dirs usage,anyone knows?
# ls /data/1/mrlocal/yarn/local/
filecache/ nmPrivate/ usercache/
Re: issue about the MR JOB local dir
Posted by ch huang <ju...@gmail.com>.
thank you ,but it seems the doc is littler old , doc says
- *PUBLIC:* <local-dir>/filecache
- *PRIVATE:* <local-dir>/usercache//filecache
- *APPLICATION:* <local-dir>/usercache//appcache/<app-id>/
but here is my nodemanager directory,i guess nmPrivate belongs to private
dir ,and filecache dir is not exist in usercache
# ls /data/mrlocal/1/yarn/local
filecache nmPrivate usercache
[root@CHBM223 conf]# ls /data/mrlocal/1/yarn/local/filecache/
-1058429088916409723 4529188628984375230 -7014620238384418063
1084965746802723478 4537624275313838973 -7168597014714301440
-1624997938511480096 4630901056913375526 7270199361370573766
-1664837184667315424 -4642830643595652223 -7332220817185869511
1725675017861848111 471523682744090087 7332904188082338506
1838346487029342338 4790459366530674957 -7450121760156930096
186504478230003977 4800525395984004560 7478948409771297223
-2348110367263014791 -5080956154405911478 7486468764131639983
-2569725565520513438 524923119076958393 -7755253483162230956
-2590767617048813033 -5270961733852362332 -7859425335924192987
2787947055181616358 -5381775829268220744 7967711417630616031
2816094634154232444 -5845090920164902899 8115657316961272063
286373945366133510 -587409153437667574 -8196745140008584754
2931191327895309259 -5951079387471670627 -8338714062663466224
-304471400571947298 -6076923167039033115 -8473967805299855837
325019546688058584 6080416638029534254 8513492322348652110
-3331048722364108374 6332597539903254606 -8567312237113801580
3360339691049457808 6343222208406792699 8737308241488535006
336835441200377451 6566344665060319340 -8893869581665287805
3628504729266619560 -6639258108397695527 -8895898681278542021
-3801380133229678986 -6653760362065293300 8926294383627727352
3837066533086156807 -6782198269120858036 -8964326004503603190
3929223016635331138 -6814427383139267223 -9049325747073392755
4126862917222506438 -6814979781017122863 -9186700026428986961
[root@CHBM223 conf]# ls /data/mrlocal/1/yarn/local/nmPrivate/
application_1385444985453_0001 application_1385453784842_0010
application_1385522685434_0081
container_1385522685434_0073_01_000014.pid
application_1385445543402_0003 application_1385453784842_0013
container_1385522685434_0073_01_000005.pid
container_1385522685434_0073_01_000017.pid
application_1385445543402_0005 application_1385520079773_0005
container_1385522685434_0073_01_000008.pid
[root@CHBM223 conf]# ls /data/mrlocal/1/yarn/local/usercache/
hdfs helen hive root
On Thu, Dec 5, 2013 at 5:12 AM, Jian He <jh...@hortonworks.com> wrote:
> The following links may help you
> http://hortonworks.com/blog/management-of-application-dependencies-in-yarn/
> http://hortonworks.com/blog/resource-localization-in-yarn-deep-dive/
>
> Thanks,
> Jian
>
>
> On Tue, Dec 3, 2013 at 5:26 PM, ch huang <ju...@gmail.com> wrote:
>
>> hi,maillist:
>> i see three dirs on my local MR job dir ,and i do not know
>> these dirs usage,anyone knows?
>>
>> # ls /data/1/mrlocal/yarn/local/
>> filecache/ nmPrivate/ usercache/
>>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: issue about the MR JOB local dir
Posted by ch huang <ju...@gmail.com>.
thank you ,but it seems the doc is littler old , doc says
- *PUBLIC:* <local-dir>/filecache
- *PRIVATE:* <local-dir>/usercache//filecache
- *APPLICATION:* <local-dir>/usercache//appcache/<app-id>/
but here is my nodemanager directory,i guess nmPrivate belongs to private
dir ,and filecache dir is not exist in usercache
# ls /data/mrlocal/1/yarn/local
filecache nmPrivate usercache
[root@CHBM223 conf]# ls /data/mrlocal/1/yarn/local/filecache/
-1058429088916409723 4529188628984375230 -7014620238384418063
1084965746802723478 4537624275313838973 -7168597014714301440
-1624997938511480096 4630901056913375526 7270199361370573766
-1664837184667315424 -4642830643595652223 -7332220817185869511
1725675017861848111 471523682744090087 7332904188082338506
1838346487029342338 4790459366530674957 -7450121760156930096
186504478230003977 4800525395984004560 7478948409771297223
-2348110367263014791 -5080956154405911478 7486468764131639983
-2569725565520513438 524923119076958393 -7755253483162230956
-2590767617048813033 -5270961733852362332 -7859425335924192987
2787947055181616358 -5381775829268220744 7967711417630616031
2816094634154232444 -5845090920164902899 8115657316961272063
286373945366133510 -587409153437667574 -8196745140008584754
2931191327895309259 -5951079387471670627 -8338714062663466224
-304471400571947298 -6076923167039033115 -8473967805299855837
325019546688058584 6080416638029534254 8513492322348652110
-3331048722364108374 6332597539903254606 -8567312237113801580
3360339691049457808 6343222208406792699 8737308241488535006
336835441200377451 6566344665060319340 -8893869581665287805
3628504729266619560 -6639258108397695527 -8895898681278542021
-3801380133229678986 -6653760362065293300 8926294383627727352
3837066533086156807 -6782198269120858036 -8964326004503603190
3929223016635331138 -6814427383139267223 -9049325747073392755
4126862917222506438 -6814979781017122863 -9186700026428986961
[root@CHBM223 conf]# ls /data/mrlocal/1/yarn/local/nmPrivate/
application_1385444985453_0001 application_1385453784842_0010
application_1385522685434_0081
container_1385522685434_0073_01_000014.pid
application_1385445543402_0003 application_1385453784842_0013
container_1385522685434_0073_01_000005.pid
container_1385522685434_0073_01_000017.pid
application_1385445543402_0005 application_1385520079773_0005
container_1385522685434_0073_01_000008.pid
[root@CHBM223 conf]# ls /data/mrlocal/1/yarn/local/usercache/
hdfs helen hive root
On Thu, Dec 5, 2013 at 5:12 AM, Jian He <jh...@hortonworks.com> wrote:
> The following links may help you
> http://hortonworks.com/blog/management-of-application-dependencies-in-yarn/
> http://hortonworks.com/blog/resource-localization-in-yarn-deep-dive/
>
> Thanks,
> Jian
>
>
> On Tue, Dec 3, 2013 at 5:26 PM, ch huang <ju...@gmail.com> wrote:
>
>> hi,maillist:
>> i see three dirs on my local MR job dir ,and i do not know
>> these dirs usage,anyone knows?
>>
>> # ls /data/1/mrlocal/yarn/local/
>> filecache/ nmPrivate/ usercache/
>>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: issue about the MR JOB local dir
Posted by ch huang <ju...@gmail.com>.
thank you ,but it seems the doc is littler old , doc says
- *PUBLIC:* <local-dir>/filecache
- *PRIVATE:* <local-dir>/usercache//filecache
- *APPLICATION:* <local-dir>/usercache//appcache/<app-id>/
but here is my nodemanager directory,i guess nmPrivate belongs to private
dir ,and filecache dir is not exist in usercache
# ls /data/mrlocal/1/yarn/local
filecache nmPrivate usercache
[root@CHBM223 conf]# ls /data/mrlocal/1/yarn/local/filecache/
-1058429088916409723 4529188628984375230 -7014620238384418063
1084965746802723478 4537624275313838973 -7168597014714301440
-1624997938511480096 4630901056913375526 7270199361370573766
-1664837184667315424 -4642830643595652223 -7332220817185869511
1725675017861848111 471523682744090087 7332904188082338506
1838346487029342338 4790459366530674957 -7450121760156930096
186504478230003977 4800525395984004560 7478948409771297223
-2348110367263014791 -5080956154405911478 7486468764131639983
-2569725565520513438 524923119076958393 -7755253483162230956
-2590767617048813033 -5270961733852362332 -7859425335924192987
2787947055181616358 -5381775829268220744 7967711417630616031
2816094634154232444 -5845090920164902899 8115657316961272063
286373945366133510 -587409153437667574 -8196745140008584754
2931191327895309259 -5951079387471670627 -8338714062663466224
-304471400571947298 -6076923167039033115 -8473967805299855837
325019546688058584 6080416638029534254 8513492322348652110
-3331048722364108374 6332597539903254606 -8567312237113801580
3360339691049457808 6343222208406792699 8737308241488535006
336835441200377451 6566344665060319340 -8893869581665287805
3628504729266619560 -6639258108397695527 -8895898681278542021
-3801380133229678986 -6653760362065293300 8926294383627727352
3837066533086156807 -6782198269120858036 -8964326004503603190
3929223016635331138 -6814427383139267223 -9049325747073392755
4126862917222506438 -6814979781017122863 -9186700026428986961
[root@CHBM223 conf]# ls /data/mrlocal/1/yarn/local/nmPrivate/
application_1385444985453_0001 application_1385453784842_0010
application_1385522685434_0081
container_1385522685434_0073_01_000014.pid
application_1385445543402_0003 application_1385453784842_0013
container_1385522685434_0073_01_000005.pid
container_1385522685434_0073_01_000017.pid
application_1385445543402_0005 application_1385520079773_0005
container_1385522685434_0073_01_000008.pid
[root@CHBM223 conf]# ls /data/mrlocal/1/yarn/local/usercache/
hdfs helen hive root
On Thu, Dec 5, 2013 at 5:12 AM, Jian He <jh...@hortonworks.com> wrote:
> The following links may help you
> http://hortonworks.com/blog/management-of-application-dependencies-in-yarn/
> http://hortonworks.com/blog/resource-localization-in-yarn-deep-dive/
>
> Thanks,
> Jian
>
>
> On Tue, Dec 3, 2013 at 5:26 PM, ch huang <ju...@gmail.com> wrote:
>
>> hi,maillist:
>> i see three dirs on my local MR job dir ,and i do not know
>> these dirs usage,anyone knows?
>>
>> # ls /data/1/mrlocal/yarn/local/
>> filecache/ nmPrivate/ usercache/
>>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: issue about the MR JOB local dir
Posted by ch huang <ju...@gmail.com>.
thank you ,but it seems the doc is littler old , doc says
- *PUBLIC:* <local-dir>/filecache
- *PRIVATE:* <local-dir>/usercache//filecache
- *APPLICATION:* <local-dir>/usercache//appcache/<app-id>/
but here is my nodemanager directory,i guess nmPrivate belongs to private
dir ,and filecache dir is not exist in usercache
# ls /data/mrlocal/1/yarn/local
filecache nmPrivate usercache
[root@CHBM223 conf]# ls /data/mrlocal/1/yarn/local/filecache/
-1058429088916409723 4529188628984375230 -7014620238384418063
1084965746802723478 4537624275313838973 -7168597014714301440
-1624997938511480096 4630901056913375526 7270199361370573766
-1664837184667315424 -4642830643595652223 -7332220817185869511
1725675017861848111 471523682744090087 7332904188082338506
1838346487029342338 4790459366530674957 -7450121760156930096
186504478230003977 4800525395984004560 7478948409771297223
-2348110367263014791 -5080956154405911478 7486468764131639983
-2569725565520513438 524923119076958393 -7755253483162230956
-2590767617048813033 -5270961733852362332 -7859425335924192987
2787947055181616358 -5381775829268220744 7967711417630616031
2816094634154232444 -5845090920164902899 8115657316961272063
286373945366133510 -587409153437667574 -8196745140008584754
2931191327895309259 -5951079387471670627 -8338714062663466224
-304471400571947298 -6076923167039033115 -8473967805299855837
325019546688058584 6080416638029534254 8513492322348652110
-3331048722364108374 6332597539903254606 -8567312237113801580
3360339691049457808 6343222208406792699 8737308241488535006
336835441200377451 6566344665060319340 -8893869581665287805
3628504729266619560 -6639258108397695527 -8895898681278542021
-3801380133229678986 -6653760362065293300 8926294383627727352
3837066533086156807 -6782198269120858036 -8964326004503603190
3929223016635331138 -6814427383139267223 -9049325747073392755
4126862917222506438 -6814979781017122863 -9186700026428986961
[root@CHBM223 conf]# ls /data/mrlocal/1/yarn/local/nmPrivate/
application_1385444985453_0001 application_1385453784842_0010
application_1385522685434_0081
container_1385522685434_0073_01_000014.pid
application_1385445543402_0003 application_1385453784842_0013
container_1385522685434_0073_01_000005.pid
container_1385522685434_0073_01_000017.pid
application_1385445543402_0005 application_1385520079773_0005
container_1385522685434_0073_01_000008.pid
[root@CHBM223 conf]# ls /data/mrlocal/1/yarn/local/usercache/
hdfs helen hive root
On Thu, Dec 5, 2013 at 5:12 AM, Jian He <jh...@hortonworks.com> wrote:
> The following links may help you
> http://hortonworks.com/blog/management-of-application-dependencies-in-yarn/
> http://hortonworks.com/blog/resource-localization-in-yarn-deep-dive/
>
> Thanks,
> Jian
>
>
> On Tue, Dec 3, 2013 at 5:26 PM, ch huang <ju...@gmail.com> wrote:
>
>> hi,maillist:
>> i see three dirs on my local MR job dir ,and i do not know
>> these dirs usage,anyone knows?
>>
>> # ls /data/1/mrlocal/yarn/local/
>> filecache/ nmPrivate/ usercache/
>>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: issue about the MR JOB local dir
Posted by Jian He <jh...@hortonworks.com>.
The following links may help you
http://hortonworks.com/blog/management-of-application-dependencies-in-yarn/
http://hortonworks.com/blog/resource-localization-in-yarn-deep-dive/
Thanks,
Jian
On Tue, Dec 3, 2013 at 5:26 PM, ch huang <ju...@gmail.com> wrote:
> hi,maillist:
> i see three dirs on my local MR job dir ,and i do not know
> these dirs usage,anyone knows?
>
> # ls /data/1/mrlocal/yarn/local/
> filecache/ nmPrivate/ usercache/
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: issue about the MR JOB local dir
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
These are the directories where NodeManager (as configured) will store its local files. Local files includes scripts, jars, libraries - all files sent to nodes via DistributedCache.
Thanks,
+Vinod
On Dec 3, 2013, at 5:26 PM, ch huang wrote:
> hi,maillist:
> i see three dirs on my local MR job dir ,and i do not know these dirs usage,anyone knows?
>
> # ls /data/1/mrlocal/yarn/local/
> filecache/ nmPrivate/ usercache/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: issue about the MR JOB local dir
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
These are the directories where NodeManager (as configured) will store its local files. Local files includes scripts, jars, libraries - all files sent to nodes via DistributedCache.
Thanks,
+Vinod
On Dec 3, 2013, at 5:26 PM, ch huang wrote:
> hi,maillist:
> i see three dirs on my local MR job dir ,and i do not know these dirs usage,anyone knows?
>
> # ls /data/1/mrlocal/yarn/local/
> filecache/ nmPrivate/ usercache/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: issue about the MR JOB local dir
Posted by Jian He <jh...@hortonworks.com>.
The following links may help you
http://hortonworks.com/blog/management-of-application-dependencies-in-yarn/
http://hortonworks.com/blog/resource-localization-in-yarn-deep-dive/
Thanks,
Jian
On Tue, Dec 3, 2013 at 5:26 PM, ch huang <ju...@gmail.com> wrote:
> hi,maillist:
> i see three dirs on my local MR job dir ,and i do not know
> these dirs usage,anyone knows?
>
> # ls /data/1/mrlocal/yarn/local/
> filecache/ nmPrivate/ usercache/
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: issue about the MR JOB local dir
Posted by Jian He <jh...@hortonworks.com>.
The following links may help you
http://hortonworks.com/blog/management-of-application-dependencies-in-yarn/
http://hortonworks.com/blog/resource-localization-in-yarn-deep-dive/
Thanks,
Jian
On Tue, Dec 3, 2013 at 5:26 PM, ch huang <ju...@gmail.com> wrote:
> hi,maillist:
> i see three dirs on my local MR job dir ,and i do not know
> these dirs usage,anyone knows?
>
> # ls /data/1/mrlocal/yarn/local/
> filecache/ nmPrivate/ usercache/
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: issue about the MR JOB local dir
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
These are the directories where NodeManager (as configured) will store its local files. Local files includes scripts, jars, libraries - all files sent to nodes via DistributedCache.
Thanks,
+Vinod
On Dec 3, 2013, at 5:26 PM, ch huang wrote:
> hi,maillist:
> i see three dirs on my local MR job dir ,and i do not know these dirs usage,anyone knows?
>
> # ls /data/1/mrlocal/yarn/local/
> filecache/ nmPrivate/ usercache/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: issue about the MR JOB local dir
Posted by Jian He <jh...@hortonworks.com>.
The following links may help you
http://hortonworks.com/blog/management-of-application-dependencies-in-yarn/
http://hortonworks.com/blog/resource-localization-in-yarn-deep-dive/
Thanks,
Jian
On Tue, Dec 3, 2013 at 5:26 PM, ch huang <ju...@gmail.com> wrote:
> hi,maillist:
> i see three dirs on my local MR job dir ,and i do not know
> these dirs usage,anyone knows?
>
> # ls /data/1/mrlocal/yarn/local/
> filecache/ nmPrivate/ usercache/
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: issue about the MR JOB local dir
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
These are the directories where NodeManager (as configured) will store its local files. Local files includes scripts, jars, libraries - all files sent to nodes via DistributedCache.
Thanks,
+Vinod
On Dec 3, 2013, at 5:26 PM, ch huang wrote:
> hi,maillist:
> i see three dirs on my local MR job dir ,and i do not know these dirs usage,anyone knows?
>
> # ls /data/1/mrlocal/yarn/local/
> filecache/ nmPrivate/ usercache/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.