Re: HDFS Space Utilization keeps on increasing

[Himanshu Bari <hi...@gmail.com>]

Why does this happen?
On Aug 31, 2015 9:09 AM, "Gaurav Gupta" <ga...@datatorrent.com> wrote:

> Shashi,
>
> Are you running multiple instances of gateway with same license?
>
> Thanks
> - Gaurav
>
> > On Aug 31, 2015, at 8:53 AM, Gaurav Gupta <ga...@datatorrent.com>
> wrote:
> >
> > Shashi,
> > I see what is happening. For now, please stop gateway, clear
> /user/dtadmin/datatorrent/audit/ folder and start gateway again. This
> should resolve the issue for now.
> >
> >
> > Thanks
> > -Gaurav
> >
> > On Mon, Aug 31, 2015 at 7:07 AM, Shashi Vishwakarma <
> shashi.vish123@gmail.com <ma...@gmail.com>> wrote:
> > Hi All,
> >
> > Thanks for your reply. I believe you guys are right. There is data
> torrent application which keeps on restarting. I observed resource manager
> UI, I always see one application running even no one running app from my
> team.
> >
> > Chetan,
> >
> > yarn.resourcemanager.am.max-attempts property is currently set to 2. I
> checked a log for that application,there are some
> AlreadybeingCreatedException is coming.Attaching log along this mail.Can
> some one help me on this? <>
> >
> >  <>
> > Thanks and Regards, <>
> > Shashi <>
> >
> >  <>
> >
> >  <>
> >
> > On Thu, Aug 27, 2015 at 1:01 AM, Chetan Narsude <chetan@iitbombay.org
> <ma...@iitbombay.org>> wrote:
> > Can you check: yarn.resourcemanager.am.max-attempts setting for YARN
> (yarn-site.xml or yarn-default.xml whichever you are using)?
> >
> >  <>Also can you look at the application master logs for one of the app
> instances you did not start to see why it was shutdown?
> >
> >
> > --
> >  <>Chetan
> >
> >  <>
> >
> > On Wed, Aug 26, 2015 at 9:51 AM, Tushar Gosavi <tushargosavi@gmail.com
> <ma...@gmail.com>> wrote:
> > You can also check yarn resource manager ui and logs to verify which
> applications are getting restarted continuously.
> >
> > On Wed, Aug 26, 2015 at 9:08 AM, David Yan <david@datatorrent.com
> <ma...@datatorrent.com>> wrote:
> > That's a lot of applications.  I suspect there is something that keeps
> starting the application, which causes the folder to keep increasing in
> size. Can you just run get-app-info on dtcli on just one application and
> see what is being spawned up?
> >
> > David
> >
> > On Tue, Aug 25, 2015 at 11:44 PM, Shashi Vishwakarma <
> shashi.vish123@gmail.com <ma...@gmail.com>> wrote:
> > Thanks David for detailed explanation. I checked apps directory in
> HDFS,there are around 12858 application in that folder each of having 6.2 M
> size. It will be a time consuming process to find status of each
> application by running get-app-info in dtcli. So logged in to web interface
> of datatorrent(port 9090) but there is no application running at this
> moment.
> >
> > Still HDFS space utilization  is increasing,any pointers on this?
> >
> > Thanks and Regards,
> > Shashi
> >
> > On Wed, Aug 26, 2015 at 2:16 AM, Amol Kekre <amol@datatorrent.com
> <ma...@datatorrent.com>> wrote:
> >
> > Adding dev@apex.incubator.apache.org <mailto:
> dev@apex.incubator.apache.org>
> >
> > Thks,
> > Amol
> >
> >
> > On Tue, Aug 25, 2015 at 10:34 AM, David Yan <david@datatorrent.com
> <ma...@datatorrent.com>> wrote:
> > Hi Shashi,
> >
> > That directory is where Apex stores application information, like
> application jar files, checkpoints, container information, etc.
> > Please run this command to see which directory is taking the most space.
> >
> > $ hdfs dfs -du /user/dtadmin/datatorrent/apps
> >
> > Then open dtcli and use the get-app-info command look at the information
> of that application.  For example:
> >
> > dt> get-app-info application_1439598948299_0557
> >
> > The field "state" will tell you whether the application is running or
> not.
> >
> > If you don't care about the application, you can safely kill it if it's
> running and delete the HDFS directory by doing hdfs dfs -rm -r
> /user/dtadmin/datatorrent/apps/application_xxx_yyy (replace xxx and yyy
> with appropriate values).  Note that doing so will wipe all stored
> information about that application.
> >
> > David
> >
> > On Tue, Aug 25, 2015 at 6:32 AM, Shashi Vishwakarma <
> shashi.vish123@gmail.com <ma...@gmail.com>> wrote:
> > Hi,
> >
> > I have  DataTorrent 3.x installed on my cluster.Even thought there is no
> data torrent application is running , still my hdfs space utilization goes
> on increasing. Below is hdfs path that has occupied most of the space.
> >
> > /user/dtadmin/datatorrent/apps
> >
> > Why this is happening? Am I missing something here?
> >
> > Thanks
> > Shashi
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups "apex-dev" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an email to apex-dev+unsubscribe@googlegroups.com <mailto:
> apex-dev+unsubscribe@googlegroups.com>.
> > To post to this group, send email to apex-dev@googlegroups.com <mailto:
> apex-dev@googlegroups.com>.
> > To view this discussion on the web visit
> https://groups.google.com/d/msgid/apex-dev/8754d662-4948-4920-96f3-cb58f70d5f39%40googlegroups.com
> <
> https://groups.google.com/d/msgid/apex-dev/8754d662-4948-4920-96f3-cb58f70d5f39%40googlegroups.com?utm_medium=email&utm_source=footer
> >.
> > For more options, visit https://groups.google.com/d/optout <
> https://groups.google.com/d/optout>.
> >
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups "apex-dev" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an email to apex-dev+unsubscribe@googlegroups.com <mailto:
> apex-dev+unsubscribe@googlegroups.com>.
> > To post to this group, send email to apex-dev@googlegroups.com <mailto:
> apex-dev@googlegroups.com>.
> > To view this discussion on the web visit
> https://groups.google.com/d/msgid/apex-dev/CAMqituP83nSGd4Ln6phTe0okyojwsE%3DGq22unu%3D-yDgyf0Y8tA%40mail.gmail.com
> <
> https://groups.google.com/d/msgid/apex-dev/CAMqituP83nSGd4Ln6phTe0okyojwsE%3DGq22unu%3D-yDgyf0Y8tA%40mail.gmail.com?utm_medium=email&utm_source=footer
> >.
> >
> > For more options, visit https://groups.google.com/d/optout <
> https://groups.google.com/d/optout>.
> >
> >
> >
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups "apex-dev" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an email to apex-dev+unsubscribe@googlegroups.com <mailto:
> apex-dev+unsubscribe@googlegroups.com>.
> > To post to this group, send email to apex-dev@googlegroups.com <mailto:
> apex-dev@googlegroups.com>.
> > To view this discussion on the web visit
> https://groups.google.com/d/msgid/apex-dev/CAMqituMeKHC84rJpFAHKbcFi-psC-zDqrOTRwQXhq75CbSQcBQ%40mail.gmail.com
> <
> https://groups.google.com/d/msgid/apex-dev/CAMqituMeKHC84rJpFAHKbcFi-psC-zDqrOTRwQXhq75CbSQcBQ%40mail.gmail.com?utm_medium=email&utm_source=footer
> >.
> >
> > For more options, visit https://groups.google.com/d/optout <
> https://groups.google.com/d/optout>.
> >
> >
> >
> > --
> > “I'd have blown my top, because I want to beat this damn thing,
> >  as long as I've gone this far. I can't just leave it after I've found
> >  out so much about it. I have to keep going to find out ultimately
> > what is the matter with it in the end."
> >                 Richard P. Feynman
> >
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups "apex-dev" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an email to apex-dev+unsubscribe@googlegroups.com <mailto:
> apex-dev+unsubscribe@googlegroups.com>.
> > To post to this group, send email to apex-dev@googlegroups.com <mailto:
> apex-dev@googlegroups.com>.
> > To view this discussion on the web visit
> https://groups.google.com/d/msgid/apex-dev/CAHYazdeHVNqPgn8ABwic92HEkSrEoWU%3D_cXDw%2Brb5Li4GoDpww%40mail.gmail.com
> <
> https://groups.google.com/d/msgid/apex-dev/CAHYazdeHVNqPgn8ABwic92HEkSrEoWU%3D_cXDw%2Brb5Li4GoDpww%40mail.gmail.com?utm_medium=email&utm_source=footer
> >.
> >
> > For more options, visit https://groups.google.com/d/optout <
> https://groups.google.com/d/optout>.
> >
> >
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups "apex-dev" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an email to apex-dev+unsubscribe@googlegroups.com <mailto:
> apex-dev+unsubscribe@googlegroups.com>.
> > To post to this group, send email to apex-dev@googlegroups.com <mailto:
> apex-dev@googlegroups.com>.
> > To view this discussion on the web visit
> https://groups.google.com/d/msgid/apex-dev/CA%2BaZ0XP872qRKPFxtqrP9aCJAHaLOZe6BBw2iMMA3rtVKmvYyA%40mail.gmail.com
> <
> https://groups.google.com/d/msgid/apex-dev/CA%2BaZ0XP872qRKPFxtqrP9aCJAHaLOZe6BBw2iMMA3rtVKmvYyA%40mail.gmail.com?utm_medium=email&utm_source=footer
> >.
> >
> > For more options, visit https://groups.google.com/d/optout <
> https://groups.google.com/d/optout>.
> >
>
>