You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@continuum.apache.org by murali mohan <mu...@gmail.com> on 2008/03/20 21:14:41 UTC
OutOfMemory Error because of large build (log) files
I have been facing this error because of large build output files created by
builds in my projects. The logs have a threshold of INFO and they do create
large files even then because of the sheer size of the project.
This causes OutOfMemory errosr in two places:
1) The webapp - When we click on the build results page (And if the build
output file is huge)
2) The Reports Generation phase (Usually towards the end of the build) -
This is a major problem. Because the build does not complete even though it
has run all the phases till Test. And the result is that we dont have the
surefire reports link in the build result page and also there is no
notification email sent out.
For now, in my project, i have just put in a check for the size of the file
and if it is larger than 50MB, i just put in a messsage that the file is
Huge. This may not be an ideal patch for this problem. So the questions i
have are:
1) Can't we avoid reading the files into a String during report generation?
2) In the build results page, we can have a save file button instead of
opening the whole file in memory?
3) Is there any way we can continue the reports generation of a failed
Build? I am pretty sure it is not possible today. But , maybe in the future,
we can have some flexibility that, even though the Build failed in say phase
3 after successful completion phase 1 and 2, we should be able to continue
from phase 3. This will be very useful and we can have the full builds even
though it didnt run correctly during the scheduled time.
Any suggestions? What do people think of my patch? Should I submit it?
Murali.
Re: OutOfMemory Error because of large build (log) files
Posted by Emmanuel Venisse <em...@gmail.com>.
On Fri, Mar 21, 2008 at 3:17 PM, murali mohan <mu...@gmail.com> wrote:
> On Fri, Mar 21, 2008 at 10:52 AM, Emmanuel Venisse <
> emmanuel.venisse@gmail.com> wrote:
>
> > On Thu, Mar 20, 2008 at 9:14 PM, murali mohan <mu...@gmail.com>
> > wrote:
> >
> > > I have been facing this error because of large build output files
> > created
> > > by
> > > builds in my projects. The logs have a threshold of INFO and they do
> > > create
> > > large files even then because of the sheer size of the project.
> > >
> > > This causes OutOfMemory errosr in two places:
> > > 1) The webapp - When we click on the build results page (And if the
> > build
> > > output file is huge)
> > > 2) The Reports Generation phase (Usually towards the end of the build)
> -
> > > This is a major problem. Because the build does not complete even
> though
> > > it
> > > has run all the phases till Test. And the result is that we dont have
> > the
> > > surefire reports link in the build result page and also there is no
> > > notification email sent out.
> > >
> > > For now, in my project, i have just put in a check for the size of the
> > > file
> > > and if it is larger than 50MB, i just put in a messsage that the file
> is
> > > Huge. This may not be an ideal patch for this problem. So the
> questions
> > i
> > > have are:
> >
> >
> > Yes, it isn't an ideal patch.
> >
> >
> > > 1) Can't we avoid reading the files into a String during report
> > > generation?
> >
> >
> > You can try to set the include* parameter in application.xml for the
> > MailContinuumNotifier component;
> >
>
> But, there are two places where the build files are read in continuum. One
> in continuum-webapp (this is what we will try to change to a save file
> feature). The second one is in Continuum-commons project in
> DefaultConfigurationService.java. Is this read required at all?
It's an error. It should be read at a single place. Even if
defaultConfigurationService isn't the best place to it, it would be good to
modify the continuum-webapp to use it because I'm sure
DefaultConfigurationService.getBuildOutput(...) is use elsewhere until a
better placebut maybe you can find a better place at the same time.
>
>
> >
> >
> > > 2) In the build results page, we can have a save file button instead
> of
> > > opening the whole file in memory?
> >
> >
> > It would be good, can you write a patch?
> >
> > >
> > > 3) Is there any way we can continue the reports generation of a failed
> > > Build? I am pretty sure it is not possible today. But , maybe in the
> > > future,
> > > we can have some flexibility that, even though the Build failed in say
> > > phase
> > > 3 after successful completion phase 1 and 2, we should be able to
> > continue
> > > from phase 3. This will be very useful and we can have the full builds
> > > even
> > > though it didnt run correctly during the scheduled time.
> >
> >
> > hmm, I'm not sure to undrstand what you want to do.
> >
> > Any suggestions? What do people think of my patch? Should I submit it?
> > >
> > > Murali.
> > >
> >
>
Re: OutOfMemory Error because of large build (log) files
Posted by murali mohan <mu...@gmail.com>.
On Fri, Mar 21, 2008 at 10:52 AM, Emmanuel Venisse <
emmanuel.venisse@gmail.com> wrote:
> On Thu, Mar 20, 2008 at 9:14 PM, murali mohan <mu...@gmail.com>
> wrote:
>
> > I have been facing this error because of large build output files
> created
> > by
> > builds in my projects. The logs have a threshold of INFO and they do
> > create
> > large files even then because of the sheer size of the project.
> >
> > This causes OutOfMemory errosr in two places:
> > 1) The webapp - When we click on the build results page (And if the
> build
> > output file is huge)
> > 2) The Reports Generation phase (Usually towards the end of the build) -
> > This is a major problem. Because the build does not complete even though
> > it
> > has run all the phases till Test. And the result is that we dont have
> the
> > surefire reports link in the build result page and also there is no
> > notification email sent out.
> >
> > For now, in my project, i have just put in a check for the size of the
> > file
> > and if it is larger than 50MB, i just put in a messsage that the file is
> > Huge. This may not be an ideal patch for this problem. So the questions
> i
> > have are:
>
>
> Yes, it isn't an ideal patch.
>
>
> > 1) Can't we avoid reading the files into a String during report
> > generation?
>
>
> You can try to set the include* parameter in application.xml for the
> MailContinuumNotifier component;
>
But, there are two places where the build files are read in continuum. One
in continuum-webapp (this is what we will try to change to a save file
feature). The second one is in Continuum-commons project in
DefaultConfigurationService.java. Is this read required at all?
>
>
> > 2) In the build results page, we can have a save file button instead of
> > opening the whole file in memory?
>
>
> It would be good, can you write a patch?
>
> >
> > 3) Is there any way we can continue the reports generation of a failed
> > Build? I am pretty sure it is not possible today. But , maybe in the
> > future,
> > we can have some flexibility that, even though the Build failed in say
> > phase
> > 3 after successful completion phase 1 and 2, we should be able to
> continue
> > from phase 3. This will be very useful and we can have the full builds
> > even
> > though it didnt run correctly during the scheduled time.
>
>
> hmm, I'm not sure to undrstand what you want to do.
>
> Any suggestions? What do people think of my patch? Should I submit it?
> >
> > Murali.
> >
>
Re: OutOfMemory Error because of large build (log) files
Posted by murali mohan <mu...@gmail.com>.
On Fri, Mar 21, 2008 at 10:52 AM, Emmanuel Venisse <
emmanuel.venisse@gmail.com> wrote:
> On Thu, Mar 20, 2008 at 9:14 PM, murali mohan <mu...@gmail.com>
> wrote:
>
> > I have been facing this error because of large build output files
> created
> > by
> > builds in my projects. The logs have a threshold of INFO and they do
> > create
> > large files even then because of the sheer size of the project.
> >
> > This causes OutOfMemory errosr in two places:
> > 1) The webapp - When we click on the build results page (And if the
> build
> > output file is huge)
> > 2) The Reports Generation phase (Usually towards the end of the build) -
> > This is a major problem. Because the build does not complete even though
> > it
> > has run all the phases till Test. And the result is that we dont have
> the
> > surefire reports link in the build result page and also there is no
> > notification email sent out.
> >
> > For now, in my project, i have just put in a check for the size of the
> > file
> > and if it is larger than 50MB, i just put in a messsage that the file is
> > Huge. This may not be an ideal patch for this problem. So the questions
> i
> > have are:
>
>
> Yes, it isn't an ideal patch.
>
>
> > 1) Can't we avoid reading the files into a String during report
> > generation?
>
>
> You can try to set the include* parameter in application.xml for the
> MailContinuumNotifier component;
>
>
> > 2) In the build results page, we can have a save file button instead of
> > opening the whole file in memory?
>
>
> It would be good, can you write a patch?
I will try. Will update later this week.
>
>
> >
> > 3) Is there any way we can continue the reports generation of a failed
> > Build? I am pretty sure it is not possible today. But , maybe in the
> > future,
> > we can have some flexibility that, even though the Build failed in say
> > phase
> > 3 after successful completion phase 1 and 2, we should be able to
> continue
> > from phase 3. This will be very useful and we can have the full builds
> > even
> > though it didnt run correctly during the scheduled time.
>
>
> hmm, I'm not sure to undrstand what you want to do.
>
> Any suggestions? What do people think of my patch? Should I submit it?
> >
> > Murali.
> >
>
Re: OutOfMemory Error because of large build (log) files
Posted by Emmanuel Venisse <em...@gmail.com>.
On Thu, Mar 20, 2008 at 9:14 PM, murali mohan <mu...@gmail.com> wrote:
> I have been facing this error because of large build output files created
> by
> builds in my projects. The logs have a threshold of INFO and they do
> create
> large files even then because of the sheer size of the project.
>
> This causes OutOfMemory errosr in two places:
> 1) The webapp - When we click on the build results page (And if the build
> output file is huge)
> 2) The Reports Generation phase (Usually towards the end of the build) -
> This is a major problem. Because the build does not complete even though
> it
> has run all the phases till Test. And the result is that we dont have the
> surefire reports link in the build result page and also there is no
> notification email sent out.
>
> For now, in my project, i have just put in a check for the size of the
> file
> and if it is larger than 50MB, i just put in a messsage that the file is
> Huge. This may not be an ideal patch for this problem. So the questions i
> have are:
Yes, it isn't an ideal patch.
> 1) Can't we avoid reading the files into a String during report
> generation?
You can try to set the include* parameter in application.xml for the
MailContinuumNotifier component;
> 2) In the build results page, we can have a save file button instead of
> opening the whole file in memory?
It would be good, can you write a patch?
>
> 3) Is there any way we can continue the reports generation of a failed
> Build? I am pretty sure it is not possible today. But , maybe in the
> future,
> we can have some flexibility that, even though the Build failed in say
> phase
> 3 after successful completion phase 1 and 2, we should be able to continue
> from phase 3. This will be very useful and we can have the full builds
> even
> though it didnt run correctly during the scheduled time.
hmm, I'm not sure to undrstand what you want to do.
Any suggestions? What do people think of my patch? Should I submit it?
>
> Murali.
>