You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@apex.apache.org by Amol Kekre <am...@datatorrent.com> on 2015/08/25 22:46:17 UTC
Re: HDFS Space Utilization keeps on increasing
Adding dev@apex.incubator.apache.org
Thks,
Amol
On Tue, Aug 25, 2015 at 10:34 AM, David Yan <da...@datatorrent.com> wrote:
> Hi Shashi,
>
> That directory is where Apex stores application information, like
> application jar files, checkpoints, container information, etc.
> Please run this command to see which directory is taking the most space.
>
> $ hdfs dfs -du /user/dtadmin/datatorrent/apps
>
> Then open dtcli and use the get-app-info command look at the information
> of that application. For example:
>
> dt> get-app-info application_1439598948299_0557
>
> The field "state" will tell you whether the application is running or not.
>
> If you don't care about the application, you can safely kill it if it's
> running and delete the HDFS directory by doing hdfs dfs -rm -r
> /user/dtadmin/datatorrent/apps/application_xxx_yyy (replace xxx and yyy
> with appropriate values). Note that doing so will wipe all stored
> information about that application.
>
> David
>
> On Tue, Aug 25, 2015 at 6:32 AM, Shashi Vishwakarma <
> shashi.vish123@gmail.com> wrote:
>
>> Hi,
>>
>> I have DataTorrent 3.x installed on my cluster.Even thought there is no
>> data torrent application is running , still my hdfs space utilization goes
>> on increasing. Below is hdfs path that has occupied most of the space.
>>
>> /user/dtadmin/datatorrent/apps
>>
>> Why this is happening? Am I missing something here?
>>
>> Thanks
>> Shashi
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "apex-dev" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to apex-dev+unsubscribe@googlegroups.com.
>> To post to this group, send email to apex-dev@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/apex-dev/8754d662-4948-4920-96f3-cb58f70d5f39%40googlegroups.com
>> <https://groups.google.com/d/msgid/apex-dev/8754d662-4948-4920-96f3-cb58f70d5f39%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "apex-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to apex-dev+unsubscribe@googlegroups.com.
> To post to this group, send email to apex-dev@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/apex-dev/CAMqituP83nSGd4Ln6phTe0okyojwsE%3DGq22unu%3D-yDgyf0Y8tA%40mail.gmail.com
> <https://groups.google.com/d/msgid/apex-dev/CAMqituP83nSGd4Ln6phTe0okyojwsE%3DGq22unu%3D-yDgyf0Y8tA%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>
> For more options, visit https://groups.google.com/d/optout.
>
Re: HDFS Space Utilization keeps on increasing
Posted by Himanshu Bari <hi...@gmail.com>.
Why does this happen?
On Aug 31, 2015 9:09 AM, "Gaurav Gupta" <ga...@datatorrent.com> wrote:
> Shashi,
>
> Are you running multiple instances of gateway with same license?
>
> Thanks
> - Gaurav
>
> > On Aug 31, 2015, at 8:53 AM, Gaurav Gupta <ga...@datatorrent.com>
> wrote:
> >
> > Shashi,
> > I see what is happening. For now, please stop gateway, clear
> /user/dtadmin/datatorrent/audit/ folder and start gateway again. This
> should resolve the issue for now.
> >
> >
> > Thanks
> > -Gaurav
> >
> > On Mon, Aug 31, 2015 at 7:07 AM, Shashi Vishwakarma <
> shashi.vish123@gmail.com <ma...@gmail.com>> wrote:
> > Hi All,
> >
> > Thanks for your reply. I believe you guys are right. There is data
> torrent application which keeps on restarting. I observed resource manager
> UI, I always see one application running even no one running app from my
> team.
> >
> > Chetan,
> >
> > yarn.resourcemanager.am.max-attempts property is currently set to 2. I
> checked a log for that application,there are some
> AlreadybeingCreatedException is coming.Attaching log along this mail.Can
> some one help me on this? <>
> >
> > <>
> > Thanks and Regards, <>
> > Shashi <>
> >
> > <>
> >
> > <>
> >
> > On Thu, Aug 27, 2015 at 1:01 AM, Chetan Narsude <chetan@iitbombay.org
> <ma...@iitbombay.org>> wrote:
> > Can you check: yarn.resourcemanager.am.max-attempts setting for YARN
> (yarn-site.xml or yarn-default.xml whichever you are using)?
> >
> > <>Also can you look at the application master logs for one of the app
> instances you did not start to see why it was shutdown?
> >
> >
> > --
> > <>Chetan
> >
> > <>
> >
> > On Wed, Aug 26, 2015 at 9:51 AM, Tushar Gosavi <tushargosavi@gmail.com
> <ma...@gmail.com>> wrote:
> > You can also check yarn resource manager ui and logs to verify which
> applications are getting restarted continuously.
> >
> > On Wed, Aug 26, 2015 at 9:08 AM, David Yan <david@datatorrent.com
> <ma...@datatorrent.com>> wrote:
> > That's a lot of applications. I suspect there is something that keeps
> starting the application, which causes the folder to keep increasing in
> size. Can you just run get-app-info on dtcli on just one application and
> see what is being spawned up?
> >
> > David
> >
> > On Tue, Aug 25, 2015 at 11:44 PM, Shashi Vishwakarma <
> shashi.vish123@gmail.com <ma...@gmail.com>> wrote:
> > Thanks David for detailed explanation. I checked apps directory in
> HDFS,there are around 12858 application in that folder each of having 6.2 M
> size. It will be a time consuming process to find status of each
> application by running get-app-info in dtcli. So logged in to web interface
> of datatorrent(port 9090) but there is no application running at this
> moment.
> >
> > Still HDFS space utilization is increasing,any pointers on this?
> >
> > Thanks and Regards,
> > Shashi
> >
> > On Wed, Aug 26, 2015 at 2:16 AM, Amol Kekre <amol@datatorrent.com
> <ma...@datatorrent.com>> wrote:
> >
> > Adding dev@apex.incubator.apache.org <mailto:
> dev@apex.incubator.apache.org>
> >
> > Thks,
> > Amol
> >
> >
> > On Tue, Aug 25, 2015 at 10:34 AM, David Yan <david@datatorrent.com
> <ma...@datatorrent.com>> wrote:
> > Hi Shashi,
> >
> > That directory is where Apex stores application information, like
> application jar files, checkpoints, container information, etc.
> > Please run this command to see which directory is taking the most space.
> >
> > $ hdfs dfs -du /user/dtadmin/datatorrent/apps
> >
> > Then open dtcli and use the get-app-info command look at the information
> of that application. For example:
> >
> > dt> get-app-info application_1439598948299_0557
> >
> > The field "state" will tell you whether the application is running or
> not.
> >
> > If you don't care about the application, you can safely kill it if it's
> running and delete the HDFS directory by doing hdfs dfs -rm -r
> /user/dtadmin/datatorrent/apps/application_xxx_yyy (replace xxx and yyy
> with appropriate values). Note that doing so will wipe all stored
> information about that application.
> >
> > David
> >
> > On Tue, Aug 25, 2015 at 6:32 AM, Shashi Vishwakarma <
> shashi.vish123@gmail.com <ma...@gmail.com>> wrote:
> > Hi,
> >
> > I have DataTorrent 3.x installed on my cluster.Even thought there is no
> data torrent application is running , still my hdfs space utilization goes
> on increasing. Below is hdfs path that has occupied most of the space.
> >
> > /user/dtadmin/datatorrent/apps
> >
> > Why this is happening? Am I missing something here?
> >
> > Thanks
> > Shashi
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups "apex-dev" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an email to apex-dev+unsubscribe@googlegroups.com <mailto:
> apex-dev+unsubscribe@googlegroups.com>.
> > To post to this group, send email to apex-dev@googlegroups.com <mailto:
> apex-dev@googlegroups.com>.
> > To view this discussion on the web visit
> https://groups.google.com/d/msgid/apex-dev/8754d662-4948-4920-96f3-cb58f70d5f39%40googlegroups.com
> <
> https://groups.google.com/d/msgid/apex-dev/8754d662-4948-4920-96f3-cb58f70d5f39%40googlegroups.com?utm_medium=email&utm_source=footer
> >.
> > For more options, visit https://groups.google.com/d/optout <
> https://groups.google.com/d/optout>.
> >
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups "apex-dev" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an email to apex-dev+unsubscribe@googlegroups.com <mailto:
> apex-dev+unsubscribe@googlegroups.com>.
> > To post to this group, send email to apex-dev@googlegroups.com <mailto:
> apex-dev@googlegroups.com>.
> > To view this discussion on the web visit
> https://groups.google.com/d/msgid/apex-dev/CAMqituP83nSGd4Ln6phTe0okyojwsE%3DGq22unu%3D-yDgyf0Y8tA%40mail.gmail.com
> <
> https://groups.google.com/d/msgid/apex-dev/CAMqituP83nSGd4Ln6phTe0okyojwsE%3DGq22unu%3D-yDgyf0Y8tA%40mail.gmail.com?utm_medium=email&utm_source=footer
> >.
> >
> > For more options, visit https://groups.google.com/d/optout <
> https://groups.google.com/d/optout>.
> >
> >
> >
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups "apex-dev" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an email to apex-dev+unsubscribe@googlegroups.com <mailto:
> apex-dev+unsubscribe@googlegroups.com>.
> > To post to this group, send email to apex-dev@googlegroups.com <mailto:
> apex-dev@googlegroups.com>.
> > To view this discussion on the web visit
> https://groups.google.com/d/msgid/apex-dev/CAMqituMeKHC84rJpFAHKbcFi-psC-zDqrOTRwQXhq75CbSQcBQ%40mail.gmail.com
> <
> https://groups.google.com/d/msgid/apex-dev/CAMqituMeKHC84rJpFAHKbcFi-psC-zDqrOTRwQXhq75CbSQcBQ%40mail.gmail.com?utm_medium=email&utm_source=footer
> >.
> >
> > For more options, visit https://groups.google.com/d/optout <
> https://groups.google.com/d/optout>.
> >
> >
> >
> > --
> > “I'd have blown my top, because I want to beat this damn thing,
> > as long as I've gone this far. I can't just leave it after I've found
> > out so much about it. I have to keep going to find out ultimately
> > what is the matter with it in the end."
> > Richard P. Feynman
> >
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups "apex-dev" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an email to apex-dev+unsubscribe@googlegroups.com <mailto:
> apex-dev+unsubscribe@googlegroups.com>.
> > To post to this group, send email to apex-dev@googlegroups.com <mailto:
> apex-dev@googlegroups.com>.
> > To view this discussion on the web visit
> https://groups.google.com/d/msgid/apex-dev/CAHYazdeHVNqPgn8ABwic92HEkSrEoWU%3D_cXDw%2Brb5Li4GoDpww%40mail.gmail.com
> <
> https://groups.google.com/d/msgid/apex-dev/CAHYazdeHVNqPgn8ABwic92HEkSrEoWU%3D_cXDw%2Brb5Li4GoDpww%40mail.gmail.com?utm_medium=email&utm_source=footer
> >.
> >
> > For more options, visit https://groups.google.com/d/optout <
> https://groups.google.com/d/optout>.
> >
> >
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups "apex-dev" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an email to apex-dev+unsubscribe@googlegroups.com <mailto:
> apex-dev+unsubscribe@googlegroups.com>.
> > To post to this group, send email to apex-dev@googlegroups.com <mailto:
> apex-dev@googlegroups.com>.
> > To view this discussion on the web visit
> https://groups.google.com/d/msgid/apex-dev/CA%2BaZ0XP872qRKPFxtqrP9aCJAHaLOZe6BBw2iMMA3rtVKmvYyA%40mail.gmail.com
> <
> https://groups.google.com/d/msgid/apex-dev/CA%2BaZ0XP872qRKPFxtqrP9aCJAHaLOZe6BBw2iMMA3rtVKmvYyA%40mail.gmail.com?utm_medium=email&utm_source=footer
> >.
> >
> > For more options, visit https://groups.google.com/d/optout <
> https://groups.google.com/d/optout>.
> >
>
>
Re: HDFS Space Utilization keeps on increasing
Posted by Gaurav Gupta <ga...@datatorrent.com>.
Shashi,
Are you running multiple instances of gateway with same license?
Thanks
- Gaurav
> On Aug 31, 2015, at 8:53 AM, Gaurav Gupta <ga...@datatorrent.com> wrote:
>
> Shashi,
> I see what is happening. For now, please stop gateway, clear /user/dtadmin/datatorrent/audit/ folder and start gateway again. This should resolve the issue for now.
>
>
> Thanks
> -Gaurav
>
> On Mon, Aug 31, 2015 at 7:07 AM, Shashi Vishwakarma <shashi.vish123@gmail.com <ma...@gmail.com>> wrote:
> Hi All,
>
> Thanks for your reply. I believe you guys are right. There is data torrent application which keeps on restarting. I observed resource manager UI, I always see one application running even no one running app from my team.
>
> Chetan,
>
> yarn.resourcemanager.am.max-attempts property is currently set to 2. I checked a log for that application,there are some AlreadybeingCreatedException is coming.Attaching log along this mail.Can some one help me on this? <>
>
> <>
> Thanks and Regards, <>
> Shashi <>
>
> <>
>
> <>
>
> On Thu, Aug 27, 2015 at 1:01 AM, Chetan Narsude <chetan@iitbombay.org <ma...@iitbombay.org>> wrote:
> Can you check: yarn.resourcemanager.am.max-attempts setting for YARN (yarn-site.xml or yarn-default.xml whichever you are using)?
>
> <>Also can you look at the application master logs for one of the app instances you did not start to see why it was shutdown?
>
>
> --
> <>Chetan
>
> <>
>
> On Wed, Aug 26, 2015 at 9:51 AM, Tushar Gosavi <tushargosavi@gmail.com <ma...@gmail.com>> wrote:
> You can also check yarn resource manager ui and logs to verify which applications are getting restarted continuously.
>
> On Wed, Aug 26, 2015 at 9:08 AM, David Yan <david@datatorrent.com <ma...@datatorrent.com>> wrote:
> That's a lot of applications. I suspect there is something that keeps starting the application, which causes the folder to keep increasing in size. Can you just run get-app-info on dtcli on just one application and see what is being spawned up?
>
> David
>
> On Tue, Aug 25, 2015 at 11:44 PM, Shashi Vishwakarma <shashi.vish123@gmail.com <ma...@gmail.com>> wrote:
> Thanks David for detailed explanation. I checked apps directory in HDFS,there are around 12858 application in that folder each of having 6.2 M size. It will be a time consuming process to find status of each application by running get-app-info in dtcli. So logged in to web interface of datatorrent(port 9090) but there is no application running at this moment.
>
> Still HDFS space utilization is increasing,any pointers on this?
>
> Thanks and Regards,
> Shashi
>
> On Wed, Aug 26, 2015 at 2:16 AM, Amol Kekre <amol@datatorrent.com <ma...@datatorrent.com>> wrote:
>
> Adding dev@apex.incubator.apache.org <ma...@apex.incubator.apache.org>
>
> Thks,
> Amol
>
>
> On Tue, Aug 25, 2015 at 10:34 AM, David Yan <david@datatorrent.com <ma...@datatorrent.com>> wrote:
> Hi Shashi,
>
> That directory is where Apex stores application information, like application jar files, checkpoints, container information, etc.
> Please run this command to see which directory is taking the most space.
>
> $ hdfs dfs -du /user/dtadmin/datatorrent/apps
>
> Then open dtcli and use the get-app-info command look at the information of that application. For example:
>
> dt> get-app-info application_1439598948299_0557
>
> The field "state" will tell you whether the application is running or not.
>
> If you don't care about the application, you can safely kill it if it's running and delete the HDFS directory by doing hdfs dfs -rm -r /user/dtadmin/datatorrent/apps/application_xxx_yyy (replace xxx and yyy with appropriate values). Note that doing so will wipe all stored information about that application.
>
> David
>
> On Tue, Aug 25, 2015 at 6:32 AM, Shashi Vishwakarma <shashi.vish123@gmail.com <ma...@gmail.com>> wrote:
> Hi,
>
> I have DataTorrent 3.x installed on my cluster.Even thought there is no data torrent application is running , still my hdfs space utilization goes on increasing. Below is hdfs path that has occupied most of the space.
>
> /user/dtadmin/datatorrent/apps
>
> Why this is happening? Am I missing something here?
>
> Thanks
> Shashi
>
> --
> You received this message because you are subscribed to the Google Groups "apex-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to apex-dev+unsubscribe@googlegroups.com <ma...@googlegroups.com>.
> To post to this group, send email to apex-dev@googlegroups.com <ma...@googlegroups.com>.
> To view this discussion on the web visit https://groups.google.com/d/msgid/apex-dev/8754d662-4948-4920-96f3-cb58f70d5f39%40googlegroups.com <https://groups.google.com/d/msgid/apex-dev/8754d662-4948-4920-96f3-cb58f70d5f39%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>.
>
>
> --
> You received this message because you are subscribed to the Google Groups "apex-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to apex-dev+unsubscribe@googlegroups.com <ma...@googlegroups.com>.
> To post to this group, send email to apex-dev@googlegroups.com <ma...@googlegroups.com>.
> To view this discussion on the web visit https://groups.google.com/d/msgid/apex-dev/CAMqituP83nSGd4Ln6phTe0okyojwsE%3DGq22unu%3D-yDgyf0Y8tA%40mail.gmail.com <https://groups.google.com/d/msgid/apex-dev/CAMqituP83nSGd4Ln6phTe0okyojwsE%3DGq22unu%3D-yDgyf0Y8tA%40mail.gmail.com?utm_medium=email&utm_source=footer>.
>
> For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>.
>
>
>
>
> --
> You received this message because you are subscribed to the Google Groups "apex-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to apex-dev+unsubscribe@googlegroups.com <ma...@googlegroups.com>.
> To post to this group, send email to apex-dev@googlegroups.com <ma...@googlegroups.com>.
> To view this discussion on the web visit https://groups.google.com/d/msgid/apex-dev/CAMqituMeKHC84rJpFAHKbcFi-psC-zDqrOTRwQXhq75CbSQcBQ%40mail.gmail.com <https://groups.google.com/d/msgid/apex-dev/CAMqituMeKHC84rJpFAHKbcFi-psC-zDqrOTRwQXhq75CbSQcBQ%40mail.gmail.com?utm_medium=email&utm_source=footer>.
>
> For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>.
>
>
>
> --
> “I'd have blown my top, because I want to beat this damn thing,
> as long as I've gone this far. I can't just leave it after I've found
> out so much about it. I have to keep going to find out ultimately
> what is the matter with it in the end."
> Richard P. Feynman
>
>
> --
> You received this message because you are subscribed to the Google Groups "apex-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to apex-dev+unsubscribe@googlegroups.com <ma...@googlegroups.com>.
> To post to this group, send email to apex-dev@googlegroups.com <ma...@googlegroups.com>.
> To view this discussion on the web visit https://groups.google.com/d/msgid/apex-dev/CAHYazdeHVNqPgn8ABwic92HEkSrEoWU%3D_cXDw%2Brb5Li4GoDpww%40mail.gmail.com <https://groups.google.com/d/msgid/apex-dev/CAHYazdeHVNqPgn8ABwic92HEkSrEoWU%3D_cXDw%2Brb5Li4GoDpww%40mail.gmail.com?utm_medium=email&utm_source=footer>.
>
> For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>.
>
>
>
> --
> You received this message because you are subscribed to the Google Groups "apex-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to apex-dev+unsubscribe@googlegroups.com <ma...@googlegroups.com>.
> To post to this group, send email to apex-dev@googlegroups.com <ma...@googlegroups.com>.
> To view this discussion on the web visit https://groups.google.com/d/msgid/apex-dev/CA%2BaZ0XP872qRKPFxtqrP9aCJAHaLOZe6BBw2iMMA3rtVKmvYyA%40mail.gmail.com <https://groups.google.com/d/msgid/apex-dev/CA%2BaZ0XP872qRKPFxtqrP9aCJAHaLOZe6BBw2iMMA3rtVKmvYyA%40mail.gmail.com?utm_medium=email&utm_source=footer>.
>
> For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>.
>
Re: HDFS Space Utilization keeps on increasing
Posted by Gaurav Gupta <ga...@datatorrent.com>.
Shashi,
I see what is happening. For now, please stop gateway, clear
/user/dtadmin/datatorrent/audit/
folder and start gateway again. This should resolve the issue for now.
Thanks
-Gaurav
On Mon, Aug 31, 2015 at 7:07 AM, Shashi Vishwakarma <
shashi.vish123@gmail.com> wrote:
> Hi All,
>
> Thanks for your reply. I believe you guys are right. There is data torrent
> application which keeps on restarting. I observed resource manager UI, I
> always see one application running even no one running app from my team.
>
> Chetan,
>
> yarn.resourcemanager.am.max-attempts property is currently set to 2. I
> checked a log for that application,there are some
> AlreadybeingCreatedException is coming.Attaching log along this mail.Can
> some one help me on this?
>
> Thanks and Regards,
> Shashi
>
>
>
> On Thu, Aug 27, 2015 at 1:01 AM, Chetan Narsude <ch...@iitbombay.org>
> wrote:
>
>> Can you check: yarn.resourcemanager.am.max-attempts setting for YARN
>> (yarn-site.xml or yarn-default.xml whichever you are using)?
>>
>> Also can you look at the application master logs for one of the app
>> instances you did not start to see why it was shutdown?
>>
>>
>> --
>> Chetan
>>
>>
>> On Wed, Aug 26, 2015 at 9:51 AM, Tushar Gosavi <tu...@gmail.com>
>> wrote:
>>
>>> You can also check yarn resource manager ui and logs to verify which
>>> applications are getting restarted continuously.
>>>
>>> On Wed, Aug 26, 2015 at 9:08 AM, David Yan <da...@datatorrent.com>
>>> wrote:
>>>
>>>> That's a lot of applications. I suspect there is something that keeps
>>>> starting the application, which causes the folder to keep increasing in
>>>> size. Can you just run get-app-info on dtcli on just one application and
>>>> see what is being spawned up?
>>>>
>>>> David
>>>>
>>>> On Tue, Aug 25, 2015 at 11:44 PM, Shashi Vishwakarma <
>>>> shashi.vish123@gmail.com> wrote:
>>>>
>>>>> Thanks David for detailed explanation. I checked apps directory in
>>>>> HDFS,there are around 12858 application in that folder each of having 6.2 M
>>>>> size. It will be a time consuming process to find status of each
>>>>> application by running get-app-info in dtcli. So logged in to web
>>>>> interface of datatorrent(port 9090) but there is no application running at
>>>>> this moment.
>>>>>
>>>>> Still HDFS space utilization is increasing,any pointers on this?
>>>>>
>>>>> Thanks and Regards,
>>>>> Shashi
>>>>>
>>>>> On Wed, Aug 26, 2015 at 2:16 AM, Amol Kekre <am...@datatorrent.com>
>>>>> wrote:
>>>>>
>>>>>>
>>>>>> Adding dev@apex.incubator.apache.org
>>>>>>
>>>>>> Thks,
>>>>>> Amol
>>>>>>
>>>>>>
>>>>>> On Tue, Aug 25, 2015 at 10:34 AM, David Yan <da...@datatorrent.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Shashi,
>>>>>>>
>>>>>>> That directory is where Apex stores application information, like
>>>>>>> application jar files, checkpoints, container information, etc.
>>>>>>> Please run this command to see which directory is taking the most
>>>>>>> space.
>>>>>>>
>>>>>>> $ hdfs dfs -du /user/dtadmin/datatorrent/apps
>>>>>>>
>>>>>>> Then open dtcli and use the get-app-info command look at the
>>>>>>> information of that application. For example:
>>>>>>>
>>>>>>> dt> get-app-info application_1439598948299_0557
>>>>>>>
>>>>>>> The field "state" will tell you whether the application is running
>>>>>>> or not.
>>>>>>>
>>>>>>> If you don't care about the application, you can safely kill it if
>>>>>>> it's running and delete the HDFS directory by doing hdfs dfs -rm -r
>>>>>>> /user/dtadmin/datatorrent/apps/application_xxx_yyy (replace xxx and yyy
>>>>>>> with appropriate values). Note that doing so will wipe all stored
>>>>>>> information about that application.
>>>>>>>
>>>>>>> David
>>>>>>>
>>>>>>> On Tue, Aug 25, 2015 at 6:32 AM, Shashi Vishwakarma <
>>>>>>> shashi.vish123@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I have DataTorrent 3.x installed on my cluster.Even thought there
>>>>>>>> is no data torrent application is running , still my hdfs space utilization
>>>>>>>> goes on increasing. Below is hdfs path that has occupied most of the space.
>>>>>>>>
>>>>>>>> /user/dtadmin/datatorrent/apps
>>>>>>>>
>>>>>>>> Why this is happening? Am I missing something here?
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>> Shashi
>>>>>>>>
>>>>>>>> --
>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>> Groups "apex-dev" group.
>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>>> send an email to apex-dev+unsubscribe@googlegroups.com.
>>>>>>>> To post to this group, send email to apex-dev@googlegroups.com.
>>>>>>>> To view this discussion on the web visit
>>>>>>>> https://groups.google.com/d/msgid/apex-dev/8754d662-4948-4920-96f3-cb58f70d5f39%40googlegroups.com
>>>>>>>> <https://groups.google.com/d/msgid/apex-dev/8754d662-4948-4920-96f3-cb58f70d5f39%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>>>> .
>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> You received this message because you are subscribed to the Google
>>>>>>> Groups "apex-dev" group.
>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>> send an email to apex-dev+unsubscribe@googlegroups.com.
>>>>>>> To post to this group, send email to apex-dev@googlegroups.com.
>>>>>>> To view this discussion on the web visit
>>>>>>> https://groups.google.com/d/msgid/apex-dev/CAMqituP83nSGd4Ln6phTe0okyojwsE%3DGq22unu%3D-yDgyf0Y8tA%40mail.gmail.com
>>>>>>> <https://groups.google.com/d/msgid/apex-dev/CAMqituP83nSGd4Ln6phTe0okyojwsE%3DGq22unu%3D-yDgyf0Y8tA%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>>>>> .
>>>>>>>
>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "apex-dev" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to apex-dev+unsubscribe@googlegroups.com.
>>>> To post to this group, send email to apex-dev@googlegroups.com.
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/apex-dev/CAMqituMeKHC84rJpFAHKbcFi-psC-zDqrOTRwQXhq75CbSQcBQ%40mail.gmail.com
>>>> <https://groups.google.com/d/msgid/apex-dev/CAMqituMeKHC84rJpFAHKbcFi-psC-zDqrOTRwQXhq75CbSQcBQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>> .
>>>>
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>
>>>
>>>
>>> --
>>> “I'd have blown my top, because I want to beat this damn thing,
>>> as long as I've gone this far. I can't just leave it after I've found
>>> out so much about it. I have to keep going to find out ultimately
>>> what is the matter with it in the end."
>>> Richard P. Feynman
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "apex-dev" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to apex-dev+unsubscribe@googlegroups.com.
>>> To post to this group, send email to apex-dev@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/apex-dev/CAHYazdeHVNqPgn8ABwic92HEkSrEoWU%3D_cXDw%2Brb5Li4GoDpww%40mail.gmail.com
>>> <https://groups.google.com/d/msgid/apex-dev/CAHYazdeHVNqPgn8ABwic92HEkSrEoWU%3D_cXDw%2Brb5Li4GoDpww%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
> --
> You received this message because you are subscribed to the Google Groups
> "apex-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to apex-dev+unsubscribe@googlegroups.com.
> To post to this group, send email to apex-dev@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/apex-dev/CA%2BaZ0XP872qRKPFxtqrP9aCJAHaLOZe6BBw2iMMA3rtVKmvYyA%40mail.gmail.com
> <https://groups.google.com/d/msgid/apex-dev/CA%2BaZ0XP872qRKPFxtqrP9aCJAHaLOZe6BBw2iMMA3rtVKmvYyA%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>
> For more options, visit https://groups.google.com/d/optout.
>
Re: HDFS Space Utilization keeps on increasing
Posted by Gaurav Gupta <ga...@datatorrent.com>.
Shashi,
I looked at the logs again and looks like you upgraded from 2.x to 3.x. Is
that the case?
>From 2.x to 3.x there is big change of how License behaves. In 3.x you
don't need any Licensing Agent. Please follow the following steps. You
should have Yarn Application running with Id dt-20150514010221-88gt3kvr
1. Shutdown Gateway (Make sure that gateway script is pointing to 3.x
binary file)
2. Kill Yarn Application with Id dt-20150514010221-88gt3kvr. You can kill
it using yarn application -kill dt-20150514010221-88gt3kvr
3. Start the Gateway
Thanks
-Gaurav
On Mon, Aug 31, 2015 at 7:07 AM, Shashi Vishwakarma <
shashi.vish123@gmail.com> wrote:
> Hi All,
>
> Thanks for your reply. I believe you guys are right. There is data torrent
> application which keeps on restarting. I observed resource manager UI, I
> always see one application running even no one running app from my team.
>
> Chetan,
>
> yarn.resourcemanager.am.max-attempts property is currently set to 2. I
> checked a log for that application,there are some
> AlreadybeingCreatedException is coming.Attaching log along this mail.Can
> some one help me on this?
>
> Thanks and Regards,
> Shashi
>
>
>
> On Thu, Aug 27, 2015 at 1:01 AM, Chetan Narsude <ch...@iitbombay.org>
> wrote:
>
>> Can you check: yarn.resourcemanager.am.max-attempts setting for YARN
>> (yarn-site.xml or yarn-default.xml whichever you are using)?
>>
>> Also can you look at the application master logs for one of the app
>> instances you did not start to see why it was shutdown?
>>
>>
>> --
>> Chetan
>>
>>
>> On Wed, Aug 26, 2015 at 9:51 AM, Tushar Gosavi <tu...@gmail.com>
>> wrote:
>>
>>> You can also check yarn resource manager ui and logs to verify which
>>> applications are getting restarted continuously.
>>>
>>> On Wed, Aug 26, 2015 at 9:08 AM, David Yan <da...@datatorrent.com>
>>> wrote:
>>>
>>>> That's a lot of applications. I suspect there is something that keeps
>>>> starting the application, which causes the folder to keep increasing in
>>>> size. Can you just run get-app-info on dtcli on just one application and
>>>> see what is being spawned up?
>>>>
>>>> David
>>>>
>>>> On Tue, Aug 25, 2015 at 11:44 PM, Shashi Vishwakarma <
>>>> shashi.vish123@gmail.com> wrote:
>>>>
>>>>> Thanks David for detailed explanation. I checked apps directory in
>>>>> HDFS,there are around 12858 application in that folder each of having 6.2 M
>>>>> size. It will be a time consuming process to find status of each
>>>>> application by running get-app-info in dtcli. So logged in to web
>>>>> interface of datatorrent(port 9090) but there is no application running at
>>>>> this moment.
>>>>>
>>>>> Still HDFS space utilization is increasing,any pointers on this?
>>>>>
>>>>> Thanks and Regards,
>>>>> Shashi
>>>>>
>>>>> On Wed, Aug 26, 2015 at 2:16 AM, Amol Kekre <am...@datatorrent.com>
>>>>> wrote:
>>>>>
>>>>>>
>>>>>> Adding dev@apex.incubator.apache.org
>>>>>>
>>>>>> Thks,
>>>>>> Amol
>>>>>>
>>>>>>
>>>>>> On Tue, Aug 25, 2015 at 10:34 AM, David Yan <da...@datatorrent.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Shashi,
>>>>>>>
>>>>>>> That directory is where Apex stores application information, like
>>>>>>> application jar files, checkpoints, container information, etc.
>>>>>>> Please run this command to see which directory is taking the most
>>>>>>> space.
>>>>>>>
>>>>>>> $ hdfs dfs -du /user/dtadmin/datatorrent/apps
>>>>>>>
>>>>>>> Then open dtcli and use the get-app-info command look at the
>>>>>>> information of that application. For example:
>>>>>>>
>>>>>>> dt> get-app-info application_1439598948299_0557
>>>>>>>
>>>>>>> The field "state" will tell you whether the application is running
>>>>>>> or not.
>>>>>>>
>>>>>>> If you don't care about the application, you can safely kill it if
>>>>>>> it's running and delete the HDFS directory by doing hdfs dfs -rm -r
>>>>>>> /user/dtadmin/datatorrent/apps/application_xxx_yyy (replace xxx and yyy
>>>>>>> with appropriate values). Note that doing so will wipe all stored
>>>>>>> information about that application.
>>>>>>>
>>>>>>> David
>>>>>>>
>>>>>>> On Tue, Aug 25, 2015 at 6:32 AM, Shashi Vishwakarma <
>>>>>>> shashi.vish123@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I have DataTorrent 3.x installed on my cluster.Even thought there
>>>>>>>> is no data torrent application is running , still my hdfs space utilization
>>>>>>>> goes on increasing. Below is hdfs path that has occupied most of the space.
>>>>>>>>
>>>>>>>> /user/dtadmin/datatorrent/apps
>>>>>>>>
>>>>>>>> Why this is happening? Am I missing something here?
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>> Shashi
>>>>>>>>
>>>>>>>> --
>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>> Groups "apex-dev" group.
>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>>> send an email to apex-dev+unsubscribe@googlegroups.com.
>>>>>>>> To post to this group, send email to apex-dev@googlegroups.com.
>>>>>>>> To view this discussion on the web visit
>>>>>>>> https://groups.google.com/d/msgid/apex-dev/8754d662-4948-4920-96f3-cb58f70d5f39%40googlegroups.com
>>>>>>>> <https://groups.google.com/d/msgid/apex-dev/8754d662-4948-4920-96f3-cb58f70d5f39%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>>>> .
>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> You received this message because you are subscribed to the Google
>>>>>>> Groups "apex-dev" group.
>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>> send an email to apex-dev+unsubscribe@googlegroups.com.
>>>>>>> To post to this group, send email to apex-dev@googlegroups.com.
>>>>>>> To view this discussion on the web visit
>>>>>>> https://groups.google.com/d/msgid/apex-dev/CAMqituP83nSGd4Ln6phTe0okyojwsE%3DGq22unu%3D-yDgyf0Y8tA%40mail.gmail.com
>>>>>>> <https://groups.google.com/d/msgid/apex-dev/CAMqituP83nSGd4Ln6phTe0okyojwsE%3DGq22unu%3D-yDgyf0Y8tA%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>>>>> .
>>>>>>>
>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "apex-dev" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to apex-dev+unsubscribe@googlegroups.com.
>>>> To post to this group, send email to apex-dev@googlegroups.com.
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/apex-dev/CAMqituMeKHC84rJpFAHKbcFi-psC-zDqrOTRwQXhq75CbSQcBQ%40mail.gmail.com
>>>> <https://groups.google.com/d/msgid/apex-dev/CAMqituMeKHC84rJpFAHKbcFi-psC-zDqrOTRwQXhq75CbSQcBQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>> .
>>>>
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>
>>>
>>>
>>> --
>>> “I'd have blown my top, because I want to beat this damn thing,
>>> as long as I've gone this far. I can't just leave it after I've found
>>> out so much about it. I have to keep going to find out ultimately
>>> what is the matter with it in the end."
>>> Richard P. Feynman
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "apex-dev" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to apex-dev+unsubscribe@googlegroups.com.
>>> To post to this group, send email to apex-dev@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/apex-dev/CAHYazdeHVNqPgn8ABwic92HEkSrEoWU%3D_cXDw%2Brb5Li4GoDpww%40mail.gmail.com
>>> <https://groups.google.com/d/msgid/apex-dev/CAHYazdeHVNqPgn8ABwic92HEkSrEoWU%3D_cXDw%2Brb5Li4GoDpww%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
> --
> You received this message because you are subscribed to the Google Groups
> "apex-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to apex-dev+unsubscribe@googlegroups.com.
> To post to this group, send email to apex-dev@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/apex-dev/CA%2BaZ0XP872qRKPFxtqrP9aCJAHaLOZe6BBw2iMMA3rtVKmvYyA%40mail.gmail.com
> <https://groups.google.com/d/msgid/apex-dev/CA%2BaZ0XP872qRKPFxtqrP9aCJAHaLOZe6BBw2iMMA3rtVKmvYyA%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>
> For more options, visit https://groups.google.com/d/optout.
>
Re: HDFS Space Utilization keeps on increasing
Posted by Shashi Vishwakarma <sh...@gmail.com>.
Hi All,
Thanks for your reply. I believe you guys are right. There is data torrent
application which keeps on restarting. I observed resource manager UI, I
always see one application running even no one running app from my team.
Chetan,
yarn.resourcemanager.am.max-attempts property is currently set to 2. I
checked a log for that application,there are some
AlreadybeingCreatedException is coming.Attaching log along this mail.Can
some one help me on this?
Thanks and Regards,
Shashi
On Thu, Aug 27, 2015 at 1:01 AM, Chetan Narsude <ch...@iitbombay.org>
wrote:
> Can you check: yarn.resourcemanager.am.max-attempts setting for YARN
> (yarn-site.xml or yarn-default.xml whichever you are using)?
>
> Also can you look at the application master logs for one of the app
> instances you did not start to see why it was shutdown?
>
>
> --
> Chetan
>
>
> On Wed, Aug 26, 2015 at 9:51 AM, Tushar Gosavi <tu...@gmail.com>
> wrote:
>
>> You can also check yarn resource manager ui and logs to verify which
>> applications are getting restarted continuously.
>>
>> On Wed, Aug 26, 2015 at 9:08 AM, David Yan <da...@datatorrent.com> wrote:
>>
>>> That's a lot of applications. I suspect there is something that keeps
>>> starting the application, which causes the folder to keep increasing in
>>> size. Can you just run get-app-info on dtcli on just one application and
>>> see what is being spawned up?
>>>
>>> David
>>>
>>> On Tue, Aug 25, 2015 at 11:44 PM, Shashi Vishwakarma <
>>> shashi.vish123@gmail.com> wrote:
>>>
>>>> Thanks David for detailed explanation. I checked apps directory in
>>>> HDFS,there are around 12858 application in that folder each of having 6.2 M
>>>> size. It will be a time consuming process to find status of each
>>>> application by running get-app-info in dtcli. So logged in to web
>>>> interface of datatorrent(port 9090) but there is no application running at
>>>> this moment.
>>>>
>>>> Still HDFS space utilization is increasing,any pointers on this?
>>>>
>>>> Thanks and Regards,
>>>> Shashi
>>>>
>>>> On Wed, Aug 26, 2015 at 2:16 AM, Amol Kekre <am...@datatorrent.com>
>>>> wrote:
>>>>
>>>>>
>>>>> Adding dev@apex.incubator.apache.org
>>>>>
>>>>> Thks,
>>>>> Amol
>>>>>
>>>>>
>>>>> On Tue, Aug 25, 2015 at 10:34 AM, David Yan <da...@datatorrent.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Shashi,
>>>>>>
>>>>>> That directory is where Apex stores application information, like
>>>>>> application jar files, checkpoints, container information, etc.
>>>>>> Please run this command to see which directory is taking the most
>>>>>> space.
>>>>>>
>>>>>> $ hdfs dfs -du /user/dtadmin/datatorrent/apps
>>>>>>
>>>>>> Then open dtcli and use the get-app-info command look at the
>>>>>> information of that application. For example:
>>>>>>
>>>>>> dt> get-app-info application_1439598948299_0557
>>>>>>
>>>>>> The field "state" will tell you whether the application is running or
>>>>>> not.
>>>>>>
>>>>>> If you don't care about the application, you can safely kill it if
>>>>>> it's running and delete the HDFS directory by doing hdfs dfs -rm -r
>>>>>> /user/dtadmin/datatorrent/apps/application_xxx_yyy (replace xxx and yyy
>>>>>> with appropriate values). Note that doing so will wipe all stored
>>>>>> information about that application.
>>>>>>
>>>>>> David
>>>>>>
>>>>>> On Tue, Aug 25, 2015 at 6:32 AM, Shashi Vishwakarma <
>>>>>> shashi.vish123@gmail.com> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I have DataTorrent 3.x installed on my cluster.Even thought there
>>>>>>> is no data torrent application is running , still my hdfs space utilization
>>>>>>> goes on increasing. Below is hdfs path that has occupied most of the space.
>>>>>>>
>>>>>>> /user/dtadmin/datatorrent/apps
>>>>>>>
>>>>>>> Why this is happening? Am I missing something here?
>>>>>>>
>>>>>>> Thanks
>>>>>>> Shashi
>>>>>>>
>>>>>>> --
>>>>>>> You received this message because you are subscribed to the Google
>>>>>>> Groups "apex-dev" group.
>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>> send an email to apex-dev+unsubscribe@googlegroups.com.
>>>>>>> To post to this group, send email to apex-dev@googlegroups.com.
>>>>>>> To view this discussion on the web visit
>>>>>>> https://groups.google.com/d/msgid/apex-dev/8754d662-4948-4920-96f3-cb58f70d5f39%40googlegroups.com
>>>>>>> <https://groups.google.com/d/msgid/apex-dev/8754d662-4948-4920-96f3-cb58f70d5f39%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>>> .
>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "apex-dev" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>> send an email to apex-dev+unsubscribe@googlegroups.com.
>>>>>> To post to this group, send email to apex-dev@googlegroups.com.
>>>>>> To view this discussion on the web visit
>>>>>> https://groups.google.com/d/msgid/apex-dev/CAMqituP83nSGd4Ln6phTe0okyojwsE%3DGq22unu%3D-yDgyf0Y8tA%40mail.gmail.com
>>>>>> <https://groups.google.com/d/msgid/apex-dev/CAMqituP83nSGd4Ln6phTe0okyojwsE%3DGq22unu%3D-yDgyf0Y8tA%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>>>> .
>>>>>>
>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>
>>>>>
>>>>>
>>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "apex-dev" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to apex-dev+unsubscribe@googlegroups.com.
>>> To post to this group, send email to apex-dev@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/apex-dev/CAMqituMeKHC84rJpFAHKbcFi-psC-zDqrOTRwQXhq75CbSQcBQ%40mail.gmail.com
>>> <https://groups.google.com/d/msgid/apex-dev/CAMqituMeKHC84rJpFAHKbcFi-psC-zDqrOTRwQXhq75CbSQcBQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
>>
>> --
>> “I'd have blown my top, because I want to beat this damn thing,
>> as long as I've gone this far. I can't just leave it after I've found
>> out so much about it. I have to keep going to find out ultimately
>> what is the matter with it in the end."
>> Richard P. Feynman
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "apex-dev" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to apex-dev+unsubscribe@googlegroups.com.
>> To post to this group, send email to apex-dev@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/apex-dev/CAHYazdeHVNqPgn8ABwic92HEkSrEoWU%3D_cXDw%2Brb5Li4GoDpww%40mail.gmail.com
>> <https://groups.google.com/d/msgid/apex-dev/CAHYazdeHVNqPgn8ABwic92HEkSrEoWU%3D_cXDw%2Brb5Li4GoDpww%40mail.gmail.com?utm_medium=email&utm_source=footer>
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
Re: HDFS Space Utilization keeps on increasing
Posted by Chetan Narsude <ch...@iitbombay.org>.
Can you check: yarn.resourcemanager.am.max-attempts setting for YARN
(yarn-site.xml or yarn-default.xml whichever you are using)?
Also can you look at the application master logs for one of the app
instances you did not start to see why it was shutdown?
--
Chetan
On Wed, Aug 26, 2015 at 9:51 AM, Tushar Gosavi <tu...@gmail.com>
wrote:
> You can also check yarn resource manager ui and logs to verify which
> applications are getting restarted continuously.
>
> On Wed, Aug 26, 2015 at 9:08 AM, David Yan <da...@datatorrent.com> wrote:
>
>> That's a lot of applications. I suspect there is something that keeps
>> starting the application, which causes the folder to keep increasing in
>> size. Can you just run get-app-info on dtcli on just one application and
>> see what is being spawned up?
>>
>> David
>>
>> On Tue, Aug 25, 2015 at 11:44 PM, Shashi Vishwakarma <
>> shashi.vish123@gmail.com> wrote:
>>
>>> Thanks David for detailed explanation. I checked apps directory in
>>> HDFS,there are around 12858 application in that folder each of having 6.2 M
>>> size. It will be a time consuming process to find status of each
>>> application by running get-app-info in dtcli. So logged in to web
>>> interface of datatorrent(port 9090) but there is no application running at
>>> this moment.
>>>
>>> Still HDFS space utilization is increasing,any pointers on this?
>>>
>>> Thanks and Regards,
>>> Shashi
>>>
>>> On Wed, Aug 26, 2015 at 2:16 AM, Amol Kekre <am...@datatorrent.com>
>>> wrote:
>>>
>>>>
>>>> Adding dev@apex.incubator.apache.org
>>>>
>>>> Thks,
>>>> Amol
>>>>
>>>>
>>>> On Tue, Aug 25, 2015 at 10:34 AM, David Yan <da...@datatorrent.com>
>>>> wrote:
>>>>
>>>>> Hi Shashi,
>>>>>
>>>>> That directory is where Apex stores application information, like
>>>>> application jar files, checkpoints, container information, etc.
>>>>> Please run this command to see which directory is taking the most
>>>>> space.
>>>>>
>>>>> $ hdfs dfs -du /user/dtadmin/datatorrent/apps
>>>>>
>>>>> Then open dtcli and use the get-app-info command look at the
>>>>> information of that application. For example:
>>>>>
>>>>> dt> get-app-info application_1439598948299_0557
>>>>>
>>>>> The field "state" will tell you whether the application is running or
>>>>> not.
>>>>>
>>>>> If you don't care about the application, you can safely kill it if
>>>>> it's running and delete the HDFS directory by doing hdfs dfs -rm -r
>>>>> /user/dtadmin/datatorrent/apps/application_xxx_yyy (replace xxx and yyy
>>>>> with appropriate values). Note that doing so will wipe all stored
>>>>> information about that application.
>>>>>
>>>>> David
>>>>>
>>>>> On Tue, Aug 25, 2015 at 6:32 AM, Shashi Vishwakarma <
>>>>> shashi.vish123@gmail.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I have DataTorrent 3.x installed on my cluster.Even thought there is
>>>>>> no data torrent application is running , still my hdfs space utilization
>>>>>> goes on increasing. Below is hdfs path that has occupied most of the space.
>>>>>>
>>>>>> /user/dtadmin/datatorrent/apps
>>>>>>
>>>>>> Why this is happening? Am I missing something here?
>>>>>>
>>>>>> Thanks
>>>>>> Shashi
>>>>>>
>>>>>> --
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "apex-dev" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>> send an email to apex-dev+unsubscribe@googlegroups.com.
>>>>>> To post to this group, send email to apex-dev@googlegroups.com.
>>>>>> To view this discussion on the web visit
>>>>>> https://groups.google.com/d/msgid/apex-dev/8754d662-4948-4920-96f3-cb58f70d5f39%40googlegroups.com
>>>>>> <https://groups.google.com/d/msgid/apex-dev/8754d662-4948-4920-96f3-cb58f70d5f39%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>> .
>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>
>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "apex-dev" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to apex-dev+unsubscribe@googlegroups.com.
>>>>> To post to this group, send email to apex-dev@googlegroups.com.
>>>>> To view this discussion on the web visit
>>>>> https://groups.google.com/d/msgid/apex-dev/CAMqituP83nSGd4Ln6phTe0okyojwsE%3DGq22unu%3D-yDgyf0Y8tA%40mail.gmail.com
>>>>> <https://groups.google.com/d/msgid/apex-dev/CAMqituP83nSGd4Ln6phTe0okyojwsE%3DGq22unu%3D-yDgyf0Y8tA%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>>> .
>>>>>
>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>
>>>>
>>>>
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "apex-dev" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to apex-dev+unsubscribe@googlegroups.com.
>> To post to this group, send email to apex-dev@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/apex-dev/CAMqituMeKHC84rJpFAHKbcFi-psC-zDqrOTRwQXhq75CbSQcBQ%40mail.gmail.com
>> <https://groups.google.com/d/msgid/apex-dev/CAMqituMeKHC84rJpFAHKbcFi-psC-zDqrOTRwQXhq75CbSQcBQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> --
> “I'd have blown my top, because I want to beat this damn thing,
> as long as I've gone this far. I can't just leave it after I've found
> out so much about it. I have to keep going to find out ultimately
> what is the matter with it in the end."
> Richard P. Feynman
>
> --
> You received this message because you are subscribed to the Google Groups
> "apex-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to apex-dev+unsubscribe@googlegroups.com.
> To post to this group, send email to apex-dev@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/apex-dev/CAHYazdeHVNqPgn8ABwic92HEkSrEoWU%3D_cXDw%2Brb5Li4GoDpww%40mail.gmail.com
> <https://groups.google.com/d/msgid/apex-dev/CAHYazdeHVNqPgn8ABwic92HEkSrEoWU%3D_cXDw%2Brb5Li4GoDpww%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>
> For more options, visit https://groups.google.com/d/optout.
>
Re: HDFS Space Utilization keeps on increasing
Posted by Tushar Gosavi <tu...@gmail.com>.
You can also check yarn resource manager ui and logs to verify which
applications are getting restarted continuously.
On Wed, Aug 26, 2015 at 9:08 AM, David Yan <da...@datatorrent.com> wrote:
> That's a lot of applications. I suspect there is something that keeps
> starting the application, which causes the folder to keep increasing in
> size. Can you just run get-app-info on dtcli on just one application and
> see what is being spawned up?
>
> David
>
> On Tue, Aug 25, 2015 at 11:44 PM, Shashi Vishwakarma <
> shashi.vish123@gmail.com> wrote:
>
>> Thanks David for detailed explanation. I checked apps directory in
>> HDFS,there are around 12858 application in that folder each of having 6.2 M
>> size. It will be a time consuming process to find status of each
>> application by running get-app-info in dtcli. So logged in to web
>> interface of datatorrent(port 9090) but there is no application running at
>> this moment.
>>
>> Still HDFS space utilization is increasing,any pointers on this?
>>
>> Thanks and Regards,
>> Shashi
>>
>> On Wed, Aug 26, 2015 at 2:16 AM, Amol Kekre <am...@datatorrent.com> wrote:
>>
>>>
>>> Adding dev@apex.incubator.apache.org
>>>
>>> Thks,
>>> Amol
>>>
>>>
>>> On Tue, Aug 25, 2015 at 10:34 AM, David Yan <da...@datatorrent.com>
>>> wrote:
>>>
>>>> Hi Shashi,
>>>>
>>>> That directory is where Apex stores application information, like
>>>> application jar files, checkpoints, container information, etc.
>>>> Please run this command to see which directory is taking the most space.
>>>>
>>>> $ hdfs dfs -du /user/dtadmin/datatorrent/apps
>>>>
>>>> Then open dtcli and use the get-app-info command look at the
>>>> information of that application. For example:
>>>>
>>>> dt> get-app-info application_1439598948299_0557
>>>>
>>>> The field "state" will tell you whether the application is running or
>>>> not.
>>>>
>>>> If you don't care about the application, you can safely kill it if it's
>>>> running and delete the HDFS directory by doing hdfs dfs -rm -r
>>>> /user/dtadmin/datatorrent/apps/application_xxx_yyy (replace xxx and yyy
>>>> with appropriate values). Note that doing so will wipe all stored
>>>> information about that application.
>>>>
>>>> David
>>>>
>>>> On Tue, Aug 25, 2015 at 6:32 AM, Shashi Vishwakarma <
>>>> shashi.vish123@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I have DataTorrent 3.x installed on my cluster.Even thought there is
>>>>> no data torrent application is running , still my hdfs space utilization
>>>>> goes on increasing. Below is hdfs path that has occupied most of the space.
>>>>>
>>>>> /user/dtadmin/datatorrent/apps
>>>>>
>>>>> Why this is happening? Am I missing something here?
>>>>>
>>>>> Thanks
>>>>> Shashi
>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "apex-dev" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to apex-dev+unsubscribe@googlegroups.com.
>>>>> To post to this group, send email to apex-dev@googlegroups.com.
>>>>> To view this discussion on the web visit
>>>>> https://groups.google.com/d/msgid/apex-dev/8754d662-4948-4920-96f3-cb58f70d5f39%40googlegroups.com
>>>>> <https://groups.google.com/d/msgid/apex-dev/8754d662-4948-4920-96f3-cb58f70d5f39%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>> .
>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "apex-dev" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to apex-dev+unsubscribe@googlegroups.com.
>>>> To post to this group, send email to apex-dev@googlegroups.com.
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/apex-dev/CAMqituP83nSGd4Ln6phTe0okyojwsE%3DGq22unu%3D-yDgyf0Y8tA%40mail.gmail.com
>>>> <https://groups.google.com/d/msgid/apex-dev/CAMqituP83nSGd4Ln6phTe0okyojwsE%3DGq22unu%3D-yDgyf0Y8tA%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>> .
>>>>
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>
>>>
>>
> --
> You received this message because you are subscribed to the Google Groups
> "apex-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to apex-dev+unsubscribe@googlegroups.com.
> To post to this group, send email to apex-dev@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/apex-dev/CAMqituMeKHC84rJpFAHKbcFi-psC-zDqrOTRwQXhq75CbSQcBQ%40mail.gmail.com
> <https://groups.google.com/d/msgid/apex-dev/CAMqituMeKHC84rJpFAHKbcFi-psC-zDqrOTRwQXhq75CbSQcBQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>
> For more options, visit https://groups.google.com/d/optout.
>
--
“I'd have blown my top, because I want to beat this damn thing,
as long as I've gone this far. I can't just leave it after I've found
out so much about it. I have to keep going to find out ultimately
what is the matter with it in the end."
Richard P. Feynman
Re: HDFS Space Utilization keeps on increasing
Posted by David Yan <da...@datatorrent.com>.
That's a lot of applications. I suspect there is something that keeps
starting the application, which causes the folder to keep increasing in
size. Can you just run get-app-info on dtcli on just one application and
see what is being spawned up?
David
On Tue, Aug 25, 2015 at 11:44 PM, Shashi Vishwakarma <
shashi.vish123@gmail.com> wrote:
> Thanks David for detailed explanation. I checked apps directory in
> HDFS,there are around 12858 application in that folder each of having 6.2 M
> size. It will be a time consuming process to find status of each
> application by running get-app-info in dtcli. So logged in to web
> interface of datatorrent(port 9090) but there is no application running at
> this moment.
>
> Still HDFS space utilization is increasing,any pointers on this?
>
> Thanks and Regards,
> Shashi
>
> On Wed, Aug 26, 2015 at 2:16 AM, Amol Kekre <am...@datatorrent.com> wrote:
>
>>
>> Adding dev@apex.incubator.apache.org
>>
>> Thks,
>> Amol
>>
>>
>> On Tue, Aug 25, 2015 at 10:34 AM, David Yan <da...@datatorrent.com>
>> wrote:
>>
>>> Hi Shashi,
>>>
>>> That directory is where Apex stores application information, like
>>> application jar files, checkpoints, container information, etc.
>>> Please run this command to see which directory is taking the most space.
>>>
>>> $ hdfs dfs -du /user/dtadmin/datatorrent/apps
>>>
>>> Then open dtcli and use the get-app-info command look at the information
>>> of that application. For example:
>>>
>>> dt> get-app-info application_1439598948299_0557
>>>
>>> The field "state" will tell you whether the application is running or
>>> not.
>>>
>>> If you don't care about the application, you can safely kill it if it's
>>> running and delete the HDFS directory by doing hdfs dfs -rm -r
>>> /user/dtadmin/datatorrent/apps/application_xxx_yyy (replace xxx and yyy
>>> with appropriate values). Note that doing so will wipe all stored
>>> information about that application.
>>>
>>> David
>>>
>>> On Tue, Aug 25, 2015 at 6:32 AM, Shashi Vishwakarma <
>>> shashi.vish123@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> I have DataTorrent 3.x installed on my cluster.Even thought there is
>>>> no data torrent application is running , still my hdfs space utilization
>>>> goes on increasing. Below is hdfs path that has occupied most of the space.
>>>>
>>>> /user/dtadmin/datatorrent/apps
>>>>
>>>> Why this is happening? Am I missing something here?
>>>>
>>>> Thanks
>>>> Shashi
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "apex-dev" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to apex-dev+unsubscribe@googlegroups.com.
>>>> To post to this group, send email to apex-dev@googlegroups.com.
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/apex-dev/8754d662-4948-4920-96f3-cb58f70d5f39%40googlegroups.com
>>>> <https://groups.google.com/d/msgid/apex-dev/8754d662-4948-4920-96f3-cb58f70d5f39%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>> .
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "apex-dev" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to apex-dev+unsubscribe@googlegroups.com.
>>> To post to this group, send email to apex-dev@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/apex-dev/CAMqituP83nSGd4Ln6phTe0okyojwsE%3DGq22unu%3D-yDgyf0Y8tA%40mail.gmail.com
>>> <https://groups.google.com/d/msgid/apex-dev/CAMqituP83nSGd4Ln6phTe0okyojwsE%3DGq22unu%3D-yDgyf0Y8tA%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
>
Re: HDFS Space Utilization keeps on increasing
Posted by Shashi Vishwakarma <sh...@gmail.com>.
Thanks David for detailed explanation. I checked apps directory in
HDFS,there are around 12858 application in that folder each of having 6.2 M
size. It will be a time consuming process to find status of each
application by running get-app-info in dtcli. So logged in to web interface
of datatorrent(port 9090) but there is no application running at this
moment.
Still HDFS space utilization is increasing,any pointers on this?
Thanks and Regards,
Shashi
On Wed, Aug 26, 2015 at 2:16 AM, Amol Kekre <am...@datatorrent.com> wrote:
>
> Adding dev@apex.incubator.apache.org
>
> Thks,
> Amol
>
>
> On Tue, Aug 25, 2015 at 10:34 AM, David Yan <da...@datatorrent.com> wrote:
>
>> Hi Shashi,
>>
>> That directory is where Apex stores application information, like
>> application jar files, checkpoints, container information, etc.
>> Please run this command to see which directory is taking the most space.
>>
>> $ hdfs dfs -du /user/dtadmin/datatorrent/apps
>>
>> Then open dtcli and use the get-app-info command look at the information
>> of that application. For example:
>>
>> dt> get-app-info application_1439598948299_0557
>>
>> The field "state" will tell you whether the application is running or
>> not.
>>
>> If you don't care about the application, you can safely kill it if it's
>> running and delete the HDFS directory by doing hdfs dfs -rm -r
>> /user/dtadmin/datatorrent/apps/application_xxx_yyy (replace xxx and yyy
>> with appropriate values). Note that doing so will wipe all stored
>> information about that application.
>>
>> David
>>
>> On Tue, Aug 25, 2015 at 6:32 AM, Shashi Vishwakarma <
>> shashi.vish123@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I have DataTorrent 3.x installed on my cluster.Even thought there is no
>>> data torrent application is running , still my hdfs space utilization goes
>>> on increasing. Below is hdfs path that has occupied most of the space.
>>>
>>> /user/dtadmin/datatorrent/apps
>>>
>>> Why this is happening? Am I missing something here?
>>>
>>> Thanks
>>> Shashi
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "apex-dev" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to apex-dev+unsubscribe@googlegroups.com.
>>> To post to this group, send email to apex-dev@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/apex-dev/8754d662-4948-4920-96f3-cb58f70d5f39%40googlegroups.com
>>> <https://groups.google.com/d/msgid/apex-dev/8754d662-4948-4920-96f3-cb58f70d5f39%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "apex-dev" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to apex-dev+unsubscribe@googlegroups.com.
>> To post to this group, send email to apex-dev@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/apex-dev/CAMqituP83nSGd4Ln6phTe0okyojwsE%3DGq22unu%3D-yDgyf0Y8tA%40mail.gmail.com
>> <https://groups.google.com/d/msgid/apex-dev/CAMqituP83nSGd4Ln6phTe0okyojwsE%3DGq22unu%3D-yDgyf0Y8tA%40mail.gmail.com?utm_medium=email&utm_source=footer>
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>