You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Mohit Jain <mo...@open-insights.com> on 2020/07/22 03:35:10 UTC
Urgent: HDFS processors throwing OOM - Compressed class space
exception
Hi team,
I’ve been facing the issue while using any HDFS processor, e.g. - PutHDFS throws the error -
Failed to write to HDFS due to compressed class space: java.lang.OutOFMemoryError
Eventually the node gets disconnected.
Any help would be appreciated.
Regards,
Mohit
[Image.jpeg]
Get Outlook for iOS<https://aka.ms/o0ukef>
Re: Urgent: HDFS processors throwing OOM - Compressed class space
exception
Posted by Mohit Jain <mo...@open-insights.com>.
Yes seems like the resource issue, CDSW is also running in the same node which eats up a lot of resources.
I’ll look into the configurations.
Thanks again.
Regards,
Mohit
Get Outlook for iOS<https://aka.ms/o0ukef>
________________________________
From: Joe Witt <jo...@gmail.com>
Sent: Wednesday, July 22, 2020 11:37:14 AM
To: dev@nifi.apache.org <de...@nifi.apache.org>
Subject: Re: Urgent: HDFS processors throwing OOM - Compressed class space exception
Yeah worth noting that such an error would present anywhere in the flow.
But the root cause is more likely related to config or system limitations
or possibly a leak of classes. Upgrading to latest is highly recommended.
On Tue, Jul 21, 2020 at 10:59 PM Mohit Jain <mo...@open-insights.com> wrote:
> Thanks Joe,
>
> Yes it is mostly happening with the HDFS processors, but I found a
> RouteText processor in the nifi-app.log throwing the same error as well.
>
> Sure I’ll look into the latest nifi version.
>
> Thanks,
> Mohit
>
>
> Get Outlook for iOS<https://aka.ms/o0ukef>
> ________________________________
> From: Joe Witt <jo...@gmail.com>
> Sent: Wednesday, July 22, 2020 11:04:14 AM
> To: dev@nifi.apache.org <de...@nifi.apache.org>
> Subject: Re: Urgent: HDFS processors throwing OOM - Compressed class space
> exception
>
> The fact that the class space is being mentioned is interesting. You should
> try a more recent version of nifi to see if this has been addressed as the
> hdfs processors tend to get a lot of attention and bugs there get dealt
> with quickly.
>
> Thanks
>
> On Tue, Jul 21, 2020 at 10:19 PM Mohit Jain <mo...@open-insights.com>
> wrote:
>
> > Hi Joe,
> >
> > Thanks for responding.
> >
> > Sure I’ll keep that in mind in my future mails.
> >
> > Following are the details:
> >
> > NiFi version: 1.9.0
> > Java version: java 8
> > OS: Linux based
> >
> > Nifi properties change:
> > There were issues with java heap space so we increased it to 16g
> >
> > Thanks,
> > Mohit
> >
> >
> > Get Outlook for iOS<https://aka.ms/o0ukef>
> > ________________________________
> > From: Joe Witt <jo...@gmail.com>
> > Sent: Wednesday, July 22, 2020 10:03:59 AM
> > To: dev@nifi.apache.org <de...@nifi.apache.org>
> > Subject: Re: Urgent: HDFS processors throwing OOM - Compressed class
> space
> > exception
> >
> > Mohit
> >
> > This is probably better for the users list.
> >
> > It is probably never ideal to send an email subject of ‘Urgent’ to an
> open
> > source community.
> >
> > Please when reporting an issue provide information on the version of
> nifi,
> > java, and os.
> >
> > Also in this case what settings changes have you made in nifi.properties
> > and such. This seems likely to be a config/install issue or setting that
> > needs to be changed.
> >
> > thanks
> >
> > On Tue, Jul 21, 2020 at 8:35 PM Mohit Jain <mo...@open-insights.com>
> > wrote:
> >
> > > Hi team,
> > >
> > > I’ve been facing the issue while using any HDFS processor, e.g. -
> PutHDFS
> > > throws the error -
> > > Failed to write to HDFS due to compressed class space:
> > > java.lang.OutOFMemoryError
> > >
> > > Eventually the node gets disconnected.
> > >
> > > Any help would be appreciated.
> > >
> > > Regards,
> > > Mohit
> > >
> > > [image: Image.jpeg]
> > > Get Outlook for iOS <https://aka.ms/o0ukef>
> > >
> >
>
Re: Urgent: HDFS processors throwing OOM - Compressed class space exception
Posted by Joe Witt <jo...@gmail.com>.
Yeah worth noting that such an error would present anywhere in the flow.
But the root cause is more likely related to config or system limitations
or possibly a leak of classes. Upgrading to latest is highly recommended.
On Tue, Jul 21, 2020 at 10:59 PM Mohit Jain <mo...@open-insights.com> wrote:
> Thanks Joe,
>
> Yes it is mostly happening with the HDFS processors, but I found a
> RouteText processor in the nifi-app.log throwing the same error as well.
>
> Sure I’ll look into the latest nifi version.
>
> Thanks,
> Mohit
>
>
> Get Outlook for iOS<https://aka.ms/o0ukef>
> ________________________________
> From: Joe Witt <jo...@gmail.com>
> Sent: Wednesday, July 22, 2020 11:04:14 AM
> To: dev@nifi.apache.org <de...@nifi.apache.org>
> Subject: Re: Urgent: HDFS processors throwing OOM - Compressed class space
> exception
>
> The fact that the class space is being mentioned is interesting. You should
> try a more recent version of nifi to see if this has been addressed as the
> hdfs processors tend to get a lot of attention and bugs there get dealt
> with quickly.
>
> Thanks
>
> On Tue, Jul 21, 2020 at 10:19 PM Mohit Jain <mo...@open-insights.com>
> wrote:
>
> > Hi Joe,
> >
> > Thanks for responding.
> >
> > Sure I’ll keep that in mind in my future mails.
> >
> > Following are the details:
> >
> > NiFi version: 1.9.0
> > Java version: java 8
> > OS: Linux based
> >
> > Nifi properties change:
> > There were issues with java heap space so we increased it to 16g
> >
> > Thanks,
> > Mohit
> >
> >
> > Get Outlook for iOS<https://aka.ms/o0ukef>
> > ________________________________
> > From: Joe Witt <jo...@gmail.com>
> > Sent: Wednesday, July 22, 2020 10:03:59 AM
> > To: dev@nifi.apache.org <de...@nifi.apache.org>
> > Subject: Re: Urgent: HDFS processors throwing OOM - Compressed class
> space
> > exception
> >
> > Mohit
> >
> > This is probably better for the users list.
> >
> > It is probably never ideal to send an email subject of ‘Urgent’ to an
> open
> > source community.
> >
> > Please when reporting an issue provide information on the version of
> nifi,
> > java, and os.
> >
> > Also in this case what settings changes have you made in nifi.properties
> > and such. This seems likely to be a config/install issue or setting that
> > needs to be changed.
> >
> > thanks
> >
> > On Tue, Jul 21, 2020 at 8:35 PM Mohit Jain <mo...@open-insights.com>
> > wrote:
> >
> > > Hi team,
> > >
> > > I’ve been facing the issue while using any HDFS processor, e.g. -
> PutHDFS
> > > throws the error -
> > > Failed to write to HDFS due to compressed class space:
> > > java.lang.OutOFMemoryError
> > >
> > > Eventually the node gets disconnected.
> > >
> > > Any help would be appreciated.
> > >
> > > Regards,
> > > Mohit
> > >
> > > [image: Image.jpeg]
> > > Get Outlook for iOS <https://aka.ms/o0ukef>
> > >
> >
>
Re: Urgent: HDFS processors throwing OOM - Compressed class space
exception
Posted by Mohit Jain <mo...@open-insights.com>.
Thanks Joe,
Yes it is mostly happening with the HDFS processors, but I found a RouteText processor in the nifi-app.log throwing the same error as well.
Sure I’ll look into the latest nifi version.
Thanks,
Mohit
Get Outlook for iOS<https://aka.ms/o0ukef>
________________________________
From: Joe Witt <jo...@gmail.com>
Sent: Wednesday, July 22, 2020 11:04:14 AM
To: dev@nifi.apache.org <de...@nifi.apache.org>
Subject: Re: Urgent: HDFS processors throwing OOM - Compressed class space exception
The fact that the class space is being mentioned is interesting. You should
try a more recent version of nifi to see if this has been addressed as the
hdfs processors tend to get a lot of attention and bugs there get dealt
with quickly.
Thanks
On Tue, Jul 21, 2020 at 10:19 PM Mohit Jain <mo...@open-insights.com> wrote:
> Hi Joe,
>
> Thanks for responding.
>
> Sure I’ll keep that in mind in my future mails.
>
> Following are the details:
>
> NiFi version: 1.9.0
> Java version: java 8
> OS: Linux based
>
> Nifi properties change:
> There were issues with java heap space so we increased it to 16g
>
> Thanks,
> Mohit
>
>
> Get Outlook for iOS<https://aka.ms/o0ukef>
> ________________________________
> From: Joe Witt <jo...@gmail.com>
> Sent: Wednesday, July 22, 2020 10:03:59 AM
> To: dev@nifi.apache.org <de...@nifi.apache.org>
> Subject: Re: Urgent: HDFS processors throwing OOM - Compressed class space
> exception
>
> Mohit
>
> This is probably better for the users list.
>
> It is probably never ideal to send an email subject of ‘Urgent’ to an open
> source community.
>
> Please when reporting an issue provide information on the version of nifi,
> java, and os.
>
> Also in this case what settings changes have you made in nifi.properties
> and such. This seems likely to be a config/install issue or setting that
> needs to be changed.
>
> thanks
>
> On Tue, Jul 21, 2020 at 8:35 PM Mohit Jain <mo...@open-insights.com>
> wrote:
>
> > Hi team,
> >
> > I’ve been facing the issue while using any HDFS processor, e.g. - PutHDFS
> > throws the error -
> > Failed to write to HDFS due to compressed class space:
> > java.lang.OutOFMemoryError
> >
> > Eventually the node gets disconnected.
> >
> > Any help would be appreciated.
> >
> > Regards,
> > Mohit
> >
> > [image: Image.jpeg]
> > Get Outlook for iOS <https://aka.ms/o0ukef>
> >
>
Re: Urgent: HDFS processors throwing OOM - Compressed class space exception
Posted by Joe Witt <jo...@gmail.com>.
The fact that the class space is being mentioned is interesting. You should
try a more recent version of nifi to see if this has been addressed as the
hdfs processors tend to get a lot of attention and bugs there get dealt
with quickly.
Thanks
On Tue, Jul 21, 2020 at 10:19 PM Mohit Jain <mo...@open-insights.com> wrote:
> Hi Joe,
>
> Thanks for responding.
>
> Sure I’ll keep that in mind in my future mails.
>
> Following are the details:
>
> NiFi version: 1.9.0
> Java version: java 8
> OS: Linux based
>
> Nifi properties change:
> There were issues with java heap space so we increased it to 16g
>
> Thanks,
> Mohit
>
>
> Get Outlook for iOS<https://aka.ms/o0ukef>
> ________________________________
> From: Joe Witt <jo...@gmail.com>
> Sent: Wednesday, July 22, 2020 10:03:59 AM
> To: dev@nifi.apache.org <de...@nifi.apache.org>
> Subject: Re: Urgent: HDFS processors throwing OOM - Compressed class space
> exception
>
> Mohit
>
> This is probably better for the users list.
>
> It is probably never ideal to send an email subject of ‘Urgent’ to an open
> source community.
>
> Please when reporting an issue provide information on the version of nifi,
> java, and os.
>
> Also in this case what settings changes have you made in nifi.properties
> and such. This seems likely to be a config/install issue or setting that
> needs to be changed.
>
> thanks
>
> On Tue, Jul 21, 2020 at 8:35 PM Mohit Jain <mo...@open-insights.com>
> wrote:
>
> > Hi team,
> >
> > I’ve been facing the issue while using any HDFS processor, e.g. - PutHDFS
> > throws the error -
> > Failed to write to HDFS due to compressed class space:
> > java.lang.OutOFMemoryError
> >
> > Eventually the node gets disconnected.
> >
> > Any help would be appreciated.
> >
> > Regards,
> > Mohit
> >
> > [image: Image.jpeg]
> > Get Outlook for iOS <https://aka.ms/o0ukef>
> >
>
Re: Urgent: HDFS processors throwing OOM - Compressed class space
exception
Posted by Mohit Jain <mo...@open-insights.com>.
Hi Joe,
Thanks for responding.
Sure I’ll keep that in mind in my future mails.
Following are the details:
NiFi version: 1.9.0
Java version: java 8
OS: Linux based
Nifi properties change:
There were issues with java heap space so we increased it to 16g
Thanks,
Mohit
Get Outlook for iOS<https://aka.ms/o0ukef>
________________________________
From: Joe Witt <jo...@gmail.com>
Sent: Wednesday, July 22, 2020 10:03:59 AM
To: dev@nifi.apache.org <de...@nifi.apache.org>
Subject: Re: Urgent: HDFS processors throwing OOM - Compressed class space exception
Mohit
This is probably better for the users list.
It is probably never ideal to send an email subject of ‘Urgent’ to an open
source community.
Please when reporting an issue provide information on the version of nifi,
java, and os.
Also in this case what settings changes have you made in nifi.properties
and such. This seems likely to be a config/install issue or setting that
needs to be changed.
thanks
On Tue, Jul 21, 2020 at 8:35 PM Mohit Jain <mo...@open-insights.com> wrote:
> Hi team,
>
> I’ve been facing the issue while using any HDFS processor, e.g. - PutHDFS
> throws the error -
> Failed to write to HDFS due to compressed class space:
> java.lang.OutOFMemoryError
>
> Eventually the node gets disconnected.
>
> Any help would be appreciated.
>
> Regards,
> Mohit
>
> [image: Image.jpeg]
> Get Outlook for iOS <https://aka.ms/o0ukef>
>
Re: Urgent: HDFS processors throwing OOM - Compressed class space exception
Posted by Joe Witt <jo...@gmail.com>.
Mohit
This is probably better for the users list.
It is probably never ideal to send an email subject of ‘Urgent’ to an open
source community.
Please when reporting an issue provide information on the version of nifi,
java, and os.
Also in this case what settings changes have you made in nifi.properties
and such. This seems likely to be a config/install issue or setting that
needs to be changed.
thanks
On Tue, Jul 21, 2020 at 8:35 PM Mohit Jain <mo...@open-insights.com> wrote:
> Hi team,
>
> I’ve been facing the issue while using any HDFS processor, e.g. - PutHDFS
> throws the error -
> Failed to write to HDFS due to compressed class space:
> java.lang.OutOFMemoryError
>
> Eventually the node gets disconnected.
>
> Any help would be appreciated.
>
> Regards,
> Mohit
>
> [image: Image.jpeg]
> Get Outlook for iOS <https://aka.ms/o0ukef>
>