You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Motty Cruz <mo...@gmail.com> on 2015/06/03 19:30:04 UTC
upgrade CDH 5.3.1 to 5.4.1 - Event Server health issue
Hello,
Just finished upgrading CDH from 5.3.1 to 5.4.1 using
http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/install_upgrade_to_cdh54_parcels.html
however Event health issues warnings " eventserver (cloudera1)
<http://cloudera1.sscs.ad:7180/cmf/services/6/instances/28/status>
Process Status, Unexpected Exits"
2015-02-24 14:18:06,012 INFO
com.cloudera.cmf.eventcatcher.server.EventCatcherService: Starting
EventCatcherService. JVM Args: [-XX:+UseConcMarkSweepGC,
-XX:-CMSConcurrentMTEnabled, -XX:+UseParNewGC,
-Dmgmt.log.file=mgmt-cmf-mgmt-EVENTSERVER-cloudera1.domain.ad.log.out,
-Djava.awt.headless=true, -Djava.net.preferIPv4Stack=true, -Xms52428800,
-Xmx52428800,
-XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh],
Args: [], Version: 5.3.1 (#191 built by jenkins on 20150123-2020 git:
b0377087cf605a686591e659eb14078923bc3c83)
Any ideas? we're using Java 8
Thanks,
Motty
Re: upgrade CDH 5.3.1 to 5.4.1 - Event Server health issue
Posted by Motty Cruz <mo...@gmail.com>.
Thank you very much! Terry; issues is fixed. I bumped that up Java heap
size to 512MB as well.
Thanks,
-Motty
On 06/24/2015 09:08 AM, Terry Siu wrote:
> Hi Motty,
>
> On your EventServer config, check what the java heap size setting is.
> Ours was set to 128MB, which was much less than the default of 1GB.
> After bumping up our value to 512MB, the EventServer started up fine
> and no longer restarted. Did you have see any stderr logs for your
> EventServer? The only ones I saw were from stdout and they did not
> record any OOM. One of the Cloudera support folks said that I should
> see OOM errors in the stderr logs, which is one cause of the
> EventServer restart. Hope this helps.
>
> -Terry
>
> On Tuesday, June 23, 2015 at 10:07:29 AM UTC-7, motty cruz wrote:
>
> yes, I did upgrade to 5.4.1:
> *Version*: Cloudera Express 5.4.1 (#197 built by jenkins on
> 20150509-0041 git: 003e06d761f80834d39c3a42431a266f0aaee736)
>
> I did not enable trace level as suggest!
>
> Please let me know if you find a solution, I had tried increasing
> memory to that service but no successful results yet.
>
> Thanks,
> -Motty
> On 06/23/2015 09:31 AM, Terry Siu wrote:
>> Motty,
>>
>> Did you upgrade your Cloudera Manager to 5.4.1? I was comparing
>> your INFO message for the EventCatcherService and noticed that
>> your Version number at the end says 5.3.1.
>>
>>
>> On Tuesday, June 23, 2015 at 9:26:42 AM UTC-7, Terry Siu wrote:
>>
>> Bit skeptical that enabling trace would "fix" this issue, but
>> I did and no change. Event Server keeps restarting and no
>> errors. Posted a message on the Cloudera Manager forum to see
>> if anybody has any other ideas.
>>
>> On Tuesday, June 23, 2015 at 9:01:08 AM UTC-7, Terry Siu wrote:
>>
>> Hi Motty,
>>
>> No, I haven't. The Event Server has been restarting
>> constantly since I left it last night and the logs do not
>> provide any errors on what might be wrong. Did you try
>> enabling the trace level as described by Atul? I'll see
>> if I can figure out where the setting is. This is getting
>> annoying and I'm going to engage Cloudera Support for this.
>>
>> -Terry
>>
>> On Tuesday, June 23, 2015 at 7:26:10 AM UTC-7, motty cruz
>> wrote:
>>
>> Hello Terry,
>> did you find a solution to this problem? We're
>> experiencing this issue, no solution thus far.
>>
>> Thanks,
>> -Motty
>>
>> On 06/22/2015 03:12 PM, Terry Siu wrote:
>>> Hi guys,
>>>
>>> I just updated CDH from 5.3.3 to 5.4.2 and am seeing
>>> the same issue with the Clouder Manager EventServer
>>> (unexpected exits). I tailed the event server log
>>> and saw not ERROR logs and it looked like the
>>> EventServer keeps restarting itself over and over
>>> again. The only notable thing I've seen in the log
>>> is WARN level message:
>>>
>>> 2015-06-22 15:04:10,885 WARN
>>> com.cloudera.cmf.event.publish.EventStorePublisherWithRetry:
>>> F
>>> ailed to publish event:
>>> SimpleEvent{attributes={ROLE_TYPE=[EVENTSERVER],
>>> CATEGORY=[LOG_MESS
>>> AGE],
>>> ROLE=[mgmt-EVENTSERVER-fd434ec7afa4142d2e2f91566bafb72b],
>>> SEVERITY=[IMPORTANT], SERVI
>>> CE=[mgmt],
>>> HOST_IDS=[10a65ece-add7-4512-82bf-9f6db5c6c7ec],
>>> SERVICE_TYPE=[MGMT], LOG_LEVEL=
>>> [WARN], HOSTS=[8K04.corp.pivotlink.com
>>> <http://8K04.corp.pivotlink.com>],
>>> EVENTCODE=[EV_LOG_EVENT]}, content=Category SERVIC
>>> E is not ROLE for input role type KMS-KMS.,
>>> timestamp=1435010650675}
>>>
>>> Anyone else got this resolved/found a workaround?
>>>
>>> Thanks,
>>> -Terry
>>>
>>> On Thursday, June 4, 2015 at 7:24:16 AM UTC-7, motty
>>> cruz wrote:
>>>
>>> Thanks for your reply Atul,
>>> I believe it has to do with version of Java. We
>>> Also updated Java to version 8 previously we
>>> were using java version 1.7-67.
>>>
>>> I have not resolve the issue yet. do you mind
>>> sharing the steps you taken to enable "trace".
>>>
>>> Thanks,
>>> Motty
>>>
>>> On 06/03/2015 06:18 PM, Atul Kulkarni wrote:
>>>> We had the same issue (upgrade from 5.3.3 to
>>>> 5.4.0) - while trying to debug I enabled trace
>>>> and it started working correctly - it was very
>>>> odd, in the mean time we did not care about the
>>>> previous events data, hence we also whacked
>>>> directory (after backing it up, of course!)
>>>> where it stores some intermediate data.
>>>>
>>>> As this is not a perfect or even sensible
>>>> solution this may not work for you, we are
>>>> still investigating the real reason but since
>>>> we have not been able reproduce it - we are not
>>>> sure how to debug further.
>>>>
>>>> Just thought of sharing this random act that we
>>>> still can't believe turned out to be the
>>>> solution. Let us know if this works for you too.
>>>>
>>>> Regards,
>>>> Atul.
>>>>
>>>> On Wednesday, June 3, 2015 at 10:30:13 AM
>>>> UTC-7, motty cruz wrote:
>>>>
>>>> Hello,
>>>> Just finished upgrading CDH from 5.3.1 to
>>>> 5.4.1 using
>>>>
>>>> http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/install_upgrade_to_cdh54_parcels.html
>>>> <http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/install_upgrade_to_cdh54_parcels.html>
>>>>
>>>>
>>>> however Event health issues warnings "
>>>> eventserver (cloudera1)
>>>> <http://cloudera1.sscs.ad:7180/cmf/services/6/instances/28/status>
>>>> Process Status, Unexpected Exits"
>>>>
>>>> 2015-02-24 14:18:06,012 INFO
>>>> com.cloudera.cmf.eventcatcher.server.EventCatcherService:
>>>> Starting EventCatcherService. JVM Args:
>>>> [-XX:+UseConcMarkSweepGC,
>>>> -XX:-CMSConcurrentMTEnabled,
>>>> -XX:+UseParNewGC,
>>>> -Dmgmt.log.file=mgmt-cmf-mgmt-EVENTSERVER-cloudera1.domain.ad.log.out,
>>>> -Djava.awt.headless=true,
>>>> -Djava.net.preferIPv4Stack=true,
>>>> -Xms52428800, -Xmx52428800,
>>>> -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh],
>>>> Args: [], Version: 5.3.1 (#191 built by
>>>> jenkins on 20150123-2020 git:
>>>> b0377087cf605a686591e659eb14078923bc3c83)
>>>>
>>>> Any ideas? we're using Java 8
>>>>
>>>> Thanks,
>>>> Motty
>>>>
>>>>
>>>> --
>>>>
>>>> ---
>>>> You received this message because you are
>>>> subscribed to the Google Groups "CDH Users" group.
>>>> To unsubscribe from this group and stop
>>>> receiving emails from it, send an email to
>>>> cdh-user+u...@cloudera.org.
>>>> For more options, visit
>>>> https://groups.google.com/a/cloudera.org/d/optout
>>>> <https://groups.google.com/a/cloudera.org/d/optout>.
>>>
>>> --
>>>
>>> ---
>>> You received this message because you are subscribed
>>> to the Google Groups "CDH Users" group.
>>> To unsubscribe from this group and stop receiving
>>> emails from it, send an email to
>>> cdh-user+u...@cloudera.org.
>>> For more options, visit
>>> https://groups.google.com/a/cloudera.org/d/optout
>>> <https://groups.google.com/a/cloudera.org/d/optout>.
>>
>> --
>>
>> ---
>> You received this message because you are subscribed to the
>> Google Groups "CDH Users" group.
>> To unsubscribe from this group and stop receiving emails from it,
>> send an email to cdh-user+u...@cloudera.org <javascript:>.
>> For more options, visit
>> https://groups.google.com/a/cloudera.org/d/optout
>> <https://groups.google.com/a/cloudera.org/d/optout>.
>
> --
>
> ---
> You received this message because you are subscribed to the Google
> Groups "CDH Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to cdh-user+unsubscribe@cloudera.org
> <ma...@cloudera.org>.
> For more options, visit https://groups.google.com/a/cloudera.org/d/optout.
Re: upgrade CDH 5.3.1 to 5.4.1 - Event Server health issue
Posted by Motty Cruz <mo...@gmail.com>.
Thank you very much! Terry; issues is fixed. I bumped that up Java heap
size to 512MB as well.
Thanks,
-Motty
On 06/24/2015 09:08 AM, Terry Siu wrote:
> Hi Motty,
>
> On your EventServer config, check what the java heap size setting is.
> Ours was set to 128MB, which was much less than the default of 1GB.
> After bumping up our value to 512MB, the EventServer started up fine
> and no longer restarted. Did you have see any stderr logs for your
> EventServer? The only ones I saw were from stdout and they did not
> record any OOM. One of the Cloudera support folks said that I should
> see OOM errors in the stderr logs, which is one cause of the
> EventServer restart. Hope this helps.
>
> -Terry
>
> On Tuesday, June 23, 2015 at 10:07:29 AM UTC-7, motty cruz wrote:
>
> yes, I did upgrade to 5.4.1:
> *Version*: Cloudera Express 5.4.1 (#197 built by jenkins on
> 20150509-0041 git: 003e06d761f80834d39c3a42431a266f0aaee736)
>
> I did not enable trace level as suggest!
>
> Please let me know if you find a solution, I had tried increasing
> memory to that service but no successful results yet.
>
> Thanks,
> -Motty
> On 06/23/2015 09:31 AM, Terry Siu wrote:
>> Motty,
>>
>> Did you upgrade your Cloudera Manager to 5.4.1? I was comparing
>> your INFO message for the EventCatcherService and noticed that
>> your Version number at the end says 5.3.1.
>>
>>
>> On Tuesday, June 23, 2015 at 9:26:42 AM UTC-7, Terry Siu wrote:
>>
>> Bit skeptical that enabling trace would "fix" this issue, but
>> I did and no change. Event Server keeps restarting and no
>> errors. Posted a message on the Cloudera Manager forum to see
>> if anybody has any other ideas.
>>
>> On Tuesday, June 23, 2015 at 9:01:08 AM UTC-7, Terry Siu wrote:
>>
>> Hi Motty,
>>
>> No, I haven't. The Event Server has been restarting
>> constantly since I left it last night and the logs do not
>> provide any errors on what might be wrong. Did you try
>> enabling the trace level as described by Atul? I'll see
>> if I can figure out where the setting is. This is getting
>> annoying and I'm going to engage Cloudera Support for this.
>>
>> -Terry
>>
>> On Tuesday, June 23, 2015 at 7:26:10 AM UTC-7, motty cruz
>> wrote:
>>
>> Hello Terry,
>> did you find a solution to this problem? We're
>> experiencing this issue, no solution thus far.
>>
>> Thanks,
>> -Motty
>>
>> On 06/22/2015 03:12 PM, Terry Siu wrote:
>>> Hi guys,
>>>
>>> I just updated CDH from 5.3.3 to 5.4.2 and am seeing
>>> the same issue with the Clouder Manager EventServer
>>> (unexpected exits). I tailed the event server log
>>> and saw not ERROR logs and it looked like the
>>> EventServer keeps restarting itself over and over
>>> again. The only notable thing I've seen in the log
>>> is WARN level message:
>>>
>>> 2015-06-22 15:04:10,885 WARN
>>> com.cloudera.cmf.event.publish.EventStorePublisherWithRetry:
>>> F
>>> ailed to publish event:
>>> SimpleEvent{attributes={ROLE_TYPE=[EVENTSERVER],
>>> CATEGORY=[LOG_MESS
>>> AGE],
>>> ROLE=[mgmt-EVENTSERVER-fd434ec7afa4142d2e2f91566bafb72b],
>>> SEVERITY=[IMPORTANT], SERVI
>>> CE=[mgmt],
>>> HOST_IDS=[10a65ece-add7-4512-82bf-9f6db5c6c7ec],
>>> SERVICE_TYPE=[MGMT], LOG_LEVEL=
>>> [WARN], HOSTS=[8K04.corp.pivotlink.com
>>> <http://8K04.corp.pivotlink.com>],
>>> EVENTCODE=[EV_LOG_EVENT]}, content=Category SERVIC
>>> E is not ROLE for input role type KMS-KMS.,
>>> timestamp=1435010650675}
>>>
>>> Anyone else got this resolved/found a workaround?
>>>
>>> Thanks,
>>> -Terry
>>>
>>> On Thursday, June 4, 2015 at 7:24:16 AM UTC-7, motty
>>> cruz wrote:
>>>
>>> Thanks for your reply Atul,
>>> I believe it has to do with version of Java. We
>>> Also updated Java to version 8 previously we
>>> were using java version 1.7-67.
>>>
>>> I have not resolve the issue yet. do you mind
>>> sharing the steps you taken to enable "trace".
>>>
>>> Thanks,
>>> Motty
>>>
>>> On 06/03/2015 06:18 PM, Atul Kulkarni wrote:
>>>> We had the same issue (upgrade from 5.3.3 to
>>>> 5.4.0) - while trying to debug I enabled trace
>>>> and it started working correctly - it was very
>>>> odd, in the mean time we did not care about the
>>>> previous events data, hence we also whacked
>>>> directory (after backing it up, of course!)
>>>> where it stores some intermediate data.
>>>>
>>>> As this is not a perfect or even sensible
>>>> solution this may not work for you, we are
>>>> still investigating the real reason but since
>>>> we have not been able reproduce it - we are not
>>>> sure how to debug further.
>>>>
>>>> Just thought of sharing this random act that we
>>>> still can't believe turned out to be the
>>>> solution. Let us know if this works for you too.
>>>>
>>>> Regards,
>>>> Atul.
>>>>
>>>> On Wednesday, June 3, 2015 at 10:30:13 AM
>>>> UTC-7, motty cruz wrote:
>>>>
>>>> Hello,
>>>> Just finished upgrading CDH from 5.3.1 to
>>>> 5.4.1 using
>>>>
>>>> http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/install_upgrade_to_cdh54_parcels.html
>>>> <http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/install_upgrade_to_cdh54_parcels.html>
>>>>
>>>>
>>>> however Event health issues warnings "
>>>> eventserver (cloudera1)
>>>> <http://cloudera1.sscs.ad:7180/cmf/services/6/instances/28/status>
>>>> Process Status, Unexpected Exits"
>>>>
>>>> 2015-02-24 14:18:06,012 INFO
>>>> com.cloudera.cmf.eventcatcher.server.EventCatcherService:
>>>> Starting EventCatcherService. JVM Args:
>>>> [-XX:+UseConcMarkSweepGC,
>>>> -XX:-CMSConcurrentMTEnabled,
>>>> -XX:+UseParNewGC,
>>>> -Dmgmt.log.file=mgmt-cmf-mgmt-EVENTSERVER-cloudera1.domain.ad.log.out,
>>>> -Djava.awt.headless=true,
>>>> -Djava.net.preferIPv4Stack=true,
>>>> -Xms52428800, -Xmx52428800,
>>>> -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh],
>>>> Args: [], Version: 5.3.1 (#191 built by
>>>> jenkins on 20150123-2020 git:
>>>> b0377087cf605a686591e659eb14078923bc3c83)
>>>>
>>>> Any ideas? we're using Java 8
>>>>
>>>> Thanks,
>>>> Motty
>>>>
>>>>
>>>> --
>>>>
>>>> ---
>>>> You received this message because you are
>>>> subscribed to the Google Groups "CDH Users" group.
>>>> To unsubscribe from this group and stop
>>>> receiving emails from it, send an email to
>>>> cdh-user+u...@cloudera.org.
>>>> For more options, visit
>>>> https://groups.google.com/a/cloudera.org/d/optout
>>>> <https://groups.google.com/a/cloudera.org/d/optout>.
>>>
>>> --
>>>
>>> ---
>>> You received this message because you are subscribed
>>> to the Google Groups "CDH Users" group.
>>> To unsubscribe from this group and stop receiving
>>> emails from it, send an email to
>>> cdh-user+u...@cloudera.org.
>>> For more options, visit
>>> https://groups.google.com/a/cloudera.org/d/optout
>>> <https://groups.google.com/a/cloudera.org/d/optout>.
>>
>> --
>>
>> ---
>> You received this message because you are subscribed to the
>> Google Groups "CDH Users" group.
>> To unsubscribe from this group and stop receiving emails from it,
>> send an email to cdh-user+u...@cloudera.org <javascript:>.
>> For more options, visit
>> https://groups.google.com/a/cloudera.org/d/optout
>> <https://groups.google.com/a/cloudera.org/d/optout>.
>
> --
>
> ---
> You received this message because you are subscribed to the Google
> Groups "CDH Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to cdh-user+unsubscribe@cloudera.org
> <ma...@cloudera.org>.
> For more options, visit https://groups.google.com/a/cloudera.org/d/optout.
Re: upgrade CDH 5.3.1 to 5.4.1 - Event Server health issue
Posted by Motty Cruz <mo...@gmail.com>.
Thank you very much! Terry; issues is fixed. I bumped that up Java heap
size to 512MB as well.
Thanks,
-Motty
On 06/24/2015 09:08 AM, Terry Siu wrote:
> Hi Motty,
>
> On your EventServer config, check what the java heap size setting is.
> Ours was set to 128MB, which was much less than the default of 1GB.
> After bumping up our value to 512MB, the EventServer started up fine
> and no longer restarted. Did you have see any stderr logs for your
> EventServer? The only ones I saw were from stdout and they did not
> record any OOM. One of the Cloudera support folks said that I should
> see OOM errors in the stderr logs, which is one cause of the
> EventServer restart. Hope this helps.
>
> -Terry
>
> On Tuesday, June 23, 2015 at 10:07:29 AM UTC-7, motty cruz wrote:
>
> yes, I did upgrade to 5.4.1:
> *Version*: Cloudera Express 5.4.1 (#197 built by jenkins on
> 20150509-0041 git: 003e06d761f80834d39c3a42431a266f0aaee736)
>
> I did not enable trace level as suggest!
>
> Please let me know if you find a solution, I had tried increasing
> memory to that service but no successful results yet.
>
> Thanks,
> -Motty
> On 06/23/2015 09:31 AM, Terry Siu wrote:
>> Motty,
>>
>> Did you upgrade your Cloudera Manager to 5.4.1? I was comparing
>> your INFO message for the EventCatcherService and noticed that
>> your Version number at the end says 5.3.1.
>>
>>
>> On Tuesday, June 23, 2015 at 9:26:42 AM UTC-7, Terry Siu wrote:
>>
>> Bit skeptical that enabling trace would "fix" this issue, but
>> I did and no change. Event Server keeps restarting and no
>> errors. Posted a message on the Cloudera Manager forum to see
>> if anybody has any other ideas.
>>
>> On Tuesday, June 23, 2015 at 9:01:08 AM UTC-7, Terry Siu wrote:
>>
>> Hi Motty,
>>
>> No, I haven't. The Event Server has been restarting
>> constantly since I left it last night and the logs do not
>> provide any errors on what might be wrong. Did you try
>> enabling the trace level as described by Atul? I'll see
>> if I can figure out where the setting is. This is getting
>> annoying and I'm going to engage Cloudera Support for this.
>>
>> -Terry
>>
>> On Tuesday, June 23, 2015 at 7:26:10 AM UTC-7, motty cruz
>> wrote:
>>
>> Hello Terry,
>> did you find a solution to this problem? We're
>> experiencing this issue, no solution thus far.
>>
>> Thanks,
>> -Motty
>>
>> On 06/22/2015 03:12 PM, Terry Siu wrote:
>>> Hi guys,
>>>
>>> I just updated CDH from 5.3.3 to 5.4.2 and am seeing
>>> the same issue with the Clouder Manager EventServer
>>> (unexpected exits). I tailed the event server log
>>> and saw not ERROR logs and it looked like the
>>> EventServer keeps restarting itself over and over
>>> again. The only notable thing I've seen in the log
>>> is WARN level message:
>>>
>>> 2015-06-22 15:04:10,885 WARN
>>> com.cloudera.cmf.event.publish.EventStorePublisherWithRetry:
>>> F
>>> ailed to publish event:
>>> SimpleEvent{attributes={ROLE_TYPE=[EVENTSERVER],
>>> CATEGORY=[LOG_MESS
>>> AGE],
>>> ROLE=[mgmt-EVENTSERVER-fd434ec7afa4142d2e2f91566bafb72b],
>>> SEVERITY=[IMPORTANT], SERVI
>>> CE=[mgmt],
>>> HOST_IDS=[10a65ece-add7-4512-82bf-9f6db5c6c7ec],
>>> SERVICE_TYPE=[MGMT], LOG_LEVEL=
>>> [WARN], HOSTS=[8K04.corp.pivotlink.com
>>> <http://8K04.corp.pivotlink.com>],
>>> EVENTCODE=[EV_LOG_EVENT]}, content=Category SERVIC
>>> E is not ROLE for input role type KMS-KMS.,
>>> timestamp=1435010650675}
>>>
>>> Anyone else got this resolved/found a workaround?
>>>
>>> Thanks,
>>> -Terry
>>>
>>> On Thursday, June 4, 2015 at 7:24:16 AM UTC-7, motty
>>> cruz wrote:
>>>
>>> Thanks for your reply Atul,
>>> I believe it has to do with version of Java. We
>>> Also updated Java to version 8 previously we
>>> were using java version 1.7-67.
>>>
>>> I have not resolve the issue yet. do you mind
>>> sharing the steps you taken to enable "trace".
>>>
>>> Thanks,
>>> Motty
>>>
>>> On 06/03/2015 06:18 PM, Atul Kulkarni wrote:
>>>> We had the same issue (upgrade from 5.3.3 to
>>>> 5.4.0) - while trying to debug I enabled trace
>>>> and it started working correctly - it was very
>>>> odd, in the mean time we did not care about the
>>>> previous events data, hence we also whacked
>>>> directory (after backing it up, of course!)
>>>> where it stores some intermediate data.
>>>>
>>>> As this is not a perfect or even sensible
>>>> solution this may not work for you, we are
>>>> still investigating the real reason but since
>>>> we have not been able reproduce it - we are not
>>>> sure how to debug further.
>>>>
>>>> Just thought of sharing this random act that we
>>>> still can't believe turned out to be the
>>>> solution. Let us know if this works for you too.
>>>>
>>>> Regards,
>>>> Atul.
>>>>
>>>> On Wednesday, June 3, 2015 at 10:30:13 AM
>>>> UTC-7, motty cruz wrote:
>>>>
>>>> Hello,
>>>> Just finished upgrading CDH from 5.3.1 to
>>>> 5.4.1 using
>>>>
>>>> http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/install_upgrade_to_cdh54_parcels.html
>>>> <http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/install_upgrade_to_cdh54_parcels.html>
>>>>
>>>>
>>>> however Event health issues warnings "
>>>> eventserver (cloudera1)
>>>> <http://cloudera1.sscs.ad:7180/cmf/services/6/instances/28/status>
>>>> Process Status, Unexpected Exits"
>>>>
>>>> 2015-02-24 14:18:06,012 INFO
>>>> com.cloudera.cmf.eventcatcher.server.EventCatcherService:
>>>> Starting EventCatcherService. JVM Args:
>>>> [-XX:+UseConcMarkSweepGC,
>>>> -XX:-CMSConcurrentMTEnabled,
>>>> -XX:+UseParNewGC,
>>>> -Dmgmt.log.file=mgmt-cmf-mgmt-EVENTSERVER-cloudera1.domain.ad.log.out,
>>>> -Djava.awt.headless=true,
>>>> -Djava.net.preferIPv4Stack=true,
>>>> -Xms52428800, -Xmx52428800,
>>>> -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh],
>>>> Args: [], Version: 5.3.1 (#191 built by
>>>> jenkins on 20150123-2020 git:
>>>> b0377087cf605a686591e659eb14078923bc3c83)
>>>>
>>>> Any ideas? we're using Java 8
>>>>
>>>> Thanks,
>>>> Motty
>>>>
>>>>
>>>> --
>>>>
>>>> ---
>>>> You received this message because you are
>>>> subscribed to the Google Groups "CDH Users" group.
>>>> To unsubscribe from this group and stop
>>>> receiving emails from it, send an email to
>>>> cdh-user+u...@cloudera.org.
>>>> For more options, visit
>>>> https://groups.google.com/a/cloudera.org/d/optout
>>>> <https://groups.google.com/a/cloudera.org/d/optout>.
>>>
>>> --
>>>
>>> ---
>>> You received this message because you are subscribed
>>> to the Google Groups "CDH Users" group.
>>> To unsubscribe from this group and stop receiving
>>> emails from it, send an email to
>>> cdh-user+u...@cloudera.org.
>>> For more options, visit
>>> https://groups.google.com/a/cloudera.org/d/optout
>>> <https://groups.google.com/a/cloudera.org/d/optout>.
>>
>> --
>>
>> ---
>> You received this message because you are subscribed to the
>> Google Groups "CDH Users" group.
>> To unsubscribe from this group and stop receiving emails from it,
>> send an email to cdh-user+u...@cloudera.org <javascript:>.
>> For more options, visit
>> https://groups.google.com/a/cloudera.org/d/optout
>> <https://groups.google.com/a/cloudera.org/d/optout>.
>
> --
>
> ---
> You received this message because you are subscribed to the Google
> Groups "CDH Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to cdh-user+unsubscribe@cloudera.org
> <ma...@cloudera.org>.
> For more options, visit https://groups.google.com/a/cloudera.org/d/optout.
Re: upgrade CDH 5.3.1 to 5.4.1 - Event Server health issue
Posted by Motty Cruz <mo...@gmail.com>.
Thank you very much! Terry; issues is fixed. I bumped that up Java heap
size to 512MB as well.
Thanks,
-Motty
On 06/24/2015 09:08 AM, Terry Siu wrote:
> Hi Motty,
>
> On your EventServer config, check what the java heap size setting is.
> Ours was set to 128MB, which was much less than the default of 1GB.
> After bumping up our value to 512MB, the EventServer started up fine
> and no longer restarted. Did you have see any stderr logs for your
> EventServer? The only ones I saw were from stdout and they did not
> record any OOM. One of the Cloudera support folks said that I should
> see OOM errors in the stderr logs, which is one cause of the
> EventServer restart. Hope this helps.
>
> -Terry
>
> On Tuesday, June 23, 2015 at 10:07:29 AM UTC-7, motty cruz wrote:
>
> yes, I did upgrade to 5.4.1:
> *Version*: Cloudera Express 5.4.1 (#197 built by jenkins on
> 20150509-0041 git: 003e06d761f80834d39c3a42431a266f0aaee736)
>
> I did not enable trace level as suggest!
>
> Please let me know if you find a solution, I had tried increasing
> memory to that service but no successful results yet.
>
> Thanks,
> -Motty
> On 06/23/2015 09:31 AM, Terry Siu wrote:
>> Motty,
>>
>> Did you upgrade your Cloudera Manager to 5.4.1? I was comparing
>> your INFO message for the EventCatcherService and noticed that
>> your Version number at the end says 5.3.1.
>>
>>
>> On Tuesday, June 23, 2015 at 9:26:42 AM UTC-7, Terry Siu wrote:
>>
>> Bit skeptical that enabling trace would "fix" this issue, but
>> I did and no change. Event Server keeps restarting and no
>> errors. Posted a message on the Cloudera Manager forum to see
>> if anybody has any other ideas.
>>
>> On Tuesday, June 23, 2015 at 9:01:08 AM UTC-7, Terry Siu wrote:
>>
>> Hi Motty,
>>
>> No, I haven't. The Event Server has been restarting
>> constantly since I left it last night and the logs do not
>> provide any errors on what might be wrong. Did you try
>> enabling the trace level as described by Atul? I'll see
>> if I can figure out where the setting is. This is getting
>> annoying and I'm going to engage Cloudera Support for this.
>>
>> -Terry
>>
>> On Tuesday, June 23, 2015 at 7:26:10 AM UTC-7, motty cruz
>> wrote:
>>
>> Hello Terry,
>> did you find a solution to this problem? We're
>> experiencing this issue, no solution thus far.
>>
>> Thanks,
>> -Motty
>>
>> On 06/22/2015 03:12 PM, Terry Siu wrote:
>>> Hi guys,
>>>
>>> I just updated CDH from 5.3.3 to 5.4.2 and am seeing
>>> the same issue with the Clouder Manager EventServer
>>> (unexpected exits). I tailed the event server log
>>> and saw not ERROR logs and it looked like the
>>> EventServer keeps restarting itself over and over
>>> again. The only notable thing I've seen in the log
>>> is WARN level message:
>>>
>>> 2015-06-22 15:04:10,885 WARN
>>> com.cloudera.cmf.event.publish.EventStorePublisherWithRetry:
>>> F
>>> ailed to publish event:
>>> SimpleEvent{attributes={ROLE_TYPE=[EVENTSERVER],
>>> CATEGORY=[LOG_MESS
>>> AGE],
>>> ROLE=[mgmt-EVENTSERVER-fd434ec7afa4142d2e2f91566bafb72b],
>>> SEVERITY=[IMPORTANT], SERVI
>>> CE=[mgmt],
>>> HOST_IDS=[10a65ece-add7-4512-82bf-9f6db5c6c7ec],
>>> SERVICE_TYPE=[MGMT], LOG_LEVEL=
>>> [WARN], HOSTS=[8K04.corp.pivotlink.com
>>> <http://8K04.corp.pivotlink.com>],
>>> EVENTCODE=[EV_LOG_EVENT]}, content=Category SERVIC
>>> E is not ROLE for input role type KMS-KMS.,
>>> timestamp=1435010650675}
>>>
>>> Anyone else got this resolved/found a workaround?
>>>
>>> Thanks,
>>> -Terry
>>>
>>> On Thursday, June 4, 2015 at 7:24:16 AM UTC-7, motty
>>> cruz wrote:
>>>
>>> Thanks for your reply Atul,
>>> I believe it has to do with version of Java. We
>>> Also updated Java to version 8 previously we
>>> were using java version 1.7-67.
>>>
>>> I have not resolve the issue yet. do you mind
>>> sharing the steps you taken to enable "trace".
>>>
>>> Thanks,
>>> Motty
>>>
>>> On 06/03/2015 06:18 PM, Atul Kulkarni wrote:
>>>> We had the same issue (upgrade from 5.3.3 to
>>>> 5.4.0) - while trying to debug I enabled trace
>>>> and it started working correctly - it was very
>>>> odd, in the mean time we did not care about the
>>>> previous events data, hence we also whacked
>>>> directory (after backing it up, of course!)
>>>> where it stores some intermediate data.
>>>>
>>>> As this is not a perfect or even sensible
>>>> solution this may not work for you, we are
>>>> still investigating the real reason but since
>>>> we have not been able reproduce it - we are not
>>>> sure how to debug further.
>>>>
>>>> Just thought of sharing this random act that we
>>>> still can't believe turned out to be the
>>>> solution. Let us know if this works for you too.
>>>>
>>>> Regards,
>>>> Atul.
>>>>
>>>> On Wednesday, June 3, 2015 at 10:30:13 AM
>>>> UTC-7, motty cruz wrote:
>>>>
>>>> Hello,
>>>> Just finished upgrading CDH from 5.3.1 to
>>>> 5.4.1 using
>>>>
>>>> http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/install_upgrade_to_cdh54_parcels.html
>>>> <http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/install_upgrade_to_cdh54_parcels.html>
>>>>
>>>>
>>>> however Event health issues warnings "
>>>> eventserver (cloudera1)
>>>> <http://cloudera1.sscs.ad:7180/cmf/services/6/instances/28/status>
>>>> Process Status, Unexpected Exits"
>>>>
>>>> 2015-02-24 14:18:06,012 INFO
>>>> com.cloudera.cmf.eventcatcher.server.EventCatcherService:
>>>> Starting EventCatcherService. JVM Args:
>>>> [-XX:+UseConcMarkSweepGC,
>>>> -XX:-CMSConcurrentMTEnabled,
>>>> -XX:+UseParNewGC,
>>>> -Dmgmt.log.file=mgmt-cmf-mgmt-EVENTSERVER-cloudera1.domain.ad.log.out,
>>>> -Djava.awt.headless=true,
>>>> -Djava.net.preferIPv4Stack=true,
>>>> -Xms52428800, -Xmx52428800,
>>>> -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh],
>>>> Args: [], Version: 5.3.1 (#191 built by
>>>> jenkins on 20150123-2020 git:
>>>> b0377087cf605a686591e659eb14078923bc3c83)
>>>>
>>>> Any ideas? we're using Java 8
>>>>
>>>> Thanks,
>>>> Motty
>>>>
>>>>
>>>> --
>>>>
>>>> ---
>>>> You received this message because you are
>>>> subscribed to the Google Groups "CDH Users" group.
>>>> To unsubscribe from this group and stop
>>>> receiving emails from it, send an email to
>>>> cdh-user+u...@cloudera.org.
>>>> For more options, visit
>>>> https://groups.google.com/a/cloudera.org/d/optout
>>>> <https://groups.google.com/a/cloudera.org/d/optout>.
>>>
>>> --
>>>
>>> ---
>>> You received this message because you are subscribed
>>> to the Google Groups "CDH Users" group.
>>> To unsubscribe from this group and stop receiving
>>> emails from it, send an email to
>>> cdh-user+u...@cloudera.org.
>>> For more options, visit
>>> https://groups.google.com/a/cloudera.org/d/optout
>>> <https://groups.google.com/a/cloudera.org/d/optout>.
>>
>> --
>>
>> ---
>> You received this message because you are subscribed to the
>> Google Groups "CDH Users" group.
>> To unsubscribe from this group and stop receiving emails from it,
>> send an email to cdh-user+u...@cloudera.org <javascript:>.
>> For more options, visit
>> https://groups.google.com/a/cloudera.org/d/optout
>> <https://groups.google.com/a/cloudera.org/d/optout>.
>
> --
>
> ---
> You received this message because you are subscribed to the Google
> Groups "CDH Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to cdh-user+unsubscribe@cloudera.org
> <ma...@cloudera.org>.
> For more options, visit https://groups.google.com/a/cloudera.org/d/optout.
Re: upgrade CDH 5.3.1 to 5.4.1 - Event Server health issue
Posted by Motty Cruz <mo...@gmail.com>.
yes, I did upgrade to 5.4.1:
*Version*: Cloudera Express 5.4.1 (#197 built by jenkins on
20150509-0041 git: 003e06d761f80834d39c3a42431a266f0aaee736)
I did not enable trace level as suggest!
Please let me know if you find a solution, I had tried increasing memory
to that service but no successful results yet.
Thanks,
-Motty
On 06/23/2015 09:31 AM, Terry Siu wrote:
> Motty,
>
> Did you upgrade your Cloudera Manager to 5.4.1? I was comparing your
> INFO message for the EventCatcherService and noticed that your Version
> number at the end says 5.3.1.
>
>
> On Tuesday, June 23, 2015 at 9:26:42 AM UTC-7, Terry Siu wrote:
>
> Bit skeptical that enabling trace would "fix" this issue, but I
> did and no change. Event Server keeps restarting and no errors.
> Posted a message on the Cloudera Manager forum to see if anybody
> has any other ideas.
>
> On Tuesday, June 23, 2015 at 9:01:08 AM UTC-7, Terry Siu wrote:
>
> Hi Motty,
>
> No, I haven't. The Event Server has been restarting constantly
> since I left it last night and the logs do not provide any
> errors on what might be wrong. Did you try enabling the trace
> level as described by Atul? I'll see if I can figure out where
> the setting is. This is getting annoying and I'm going to
> engage Cloudera Support for this.
>
> -Terry
>
> On Tuesday, June 23, 2015 at 7:26:10 AM UTC-7, motty cruz wrote:
>
> Hello Terry,
> did you find a solution to this problem? We're
> experiencing this issue, no solution thus far.
>
> Thanks,
> -Motty
>
> On 06/22/2015 03:12 PM, Terry Siu wrote:
>> Hi guys,
>>
>> I just updated CDH from 5.3.3 to 5.4.2 and am seeing the
>> same issue with the Clouder Manager EventServer
>> (unexpected exits). I tailed the event server log and saw
>> not ERROR logs and it looked like the EventServer keeps
>> restarting itself over and over again. The only notable
>> thing I've seen in the log is WARN level message:
>>
>> 2015-06-22 15:04:10,885 WARN
>> com.cloudera.cmf.event.publish.EventStorePublisherWithRetry:
>> F
>> ailed to publish event:
>> SimpleEvent{attributes={ROLE_TYPE=[EVENTSERVER],
>> CATEGORY=[LOG_MESS
>> AGE],
>> ROLE=[mgmt-EVENTSERVER-fd434ec7afa4142d2e2f91566bafb72b],
>> SEVERITY=[IMPORTANT], SERVI
>> CE=[mgmt],
>> HOST_IDS=[10a65ece-add7-4512-82bf-9f6db5c6c7ec],
>> SERVICE_TYPE=[MGMT], LOG_LEVEL=
>> [WARN], HOSTS=[8K04.corp.pivotlink.com
>> <http://8K04.corp.pivotlink.com>],
>> EVENTCODE=[EV_LOG_EVENT]}, content=Category SERVIC
>> E is not ROLE for input role type KMS-KMS.,
>> timestamp=1435010650675}
>>
>> Anyone else got this resolved/found a workaround?
>>
>> Thanks,
>> -Terry
>>
>> On Thursday, June 4, 2015 at 7:24:16 AM UTC-7, motty cruz
>> wrote:
>>
>> Thanks for your reply Atul,
>> I believe it has to do with version of Java. We Also
>> updated Java to version 8 previously we were using
>> java version 1.7-67.
>>
>> I have not resolve the issue yet. do you mind sharing
>> the steps you taken to enable "trace".
>>
>> Thanks,
>> Motty
>>
>> On 06/03/2015 06:18 PM, Atul Kulkarni wrote:
>>> We had the same issue (upgrade from 5.3.3 to 5.4.0)
>>> - while trying to debug I enabled trace and it
>>> started working correctly - it was very odd, in the
>>> mean time we did not care about the previous events
>>> data, hence we also whacked directory (after backing
>>> it up, of course!) where it stores some intermediate
>>> data.
>>>
>>> As this is not a perfect or even sensible solution
>>> this may not work for you, we are still
>>> investigating the real reason but since we have not
>>> been able reproduce it - we are not sure how to
>>> debug further.
>>>
>>> Just thought of sharing this random act that we
>>> still can't believe turned out to be the solution.
>>> Let us know if this works for you too.
>>>
>>> Regards,
>>> Atul.
>>>
>>> On Wednesday, June 3, 2015 at 10:30:13 AM UTC-7,
>>> motty cruz wrote:
>>>
>>> Hello,
>>> Just finished upgrading CDH from 5.3.1 to 5.4.1
>>> using
>>>
>>> http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/install_upgrade_to_cdh54_parcels.html
>>> <http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/install_upgrade_to_cdh54_parcels.html>
>>>
>>>
>>> however Event health issues warnings "
>>> eventserver (cloudera1)
>>> <http://cloudera1.sscs.ad:7180/cmf/services/6/instances/28/status>
>>> Process Status, Unexpected Exits"
>>>
>>> 2015-02-24 14:18:06,012 INFO
>>> com.cloudera.cmf.eventcatcher.server.EventCatcherService:
>>> Starting EventCatcherService. JVM Args:
>>> [-XX:+UseConcMarkSweepGC,
>>> -XX:-CMSConcurrentMTEnabled, -XX:+UseParNewGC,
>>> -Dmgmt.log.file=mgmt-cmf-mgmt-EVENTSERVER-cloudera1.domain.ad.log.out,
>>> -Djava.awt.headless=true,
>>> -Djava.net.preferIPv4Stack=true, -Xms52428800,
>>> -Xmx52428800,
>>> -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh],
>>> Args: [], Version: 5.3.1 (#191 built by jenkins
>>> on 20150123-2020 git:
>>> b0377087cf605a686591e659eb14078923bc3c83)
>>>
>>> Any ideas? we're using Java 8
>>>
>>> Thanks,
>>> Motty
>>>
>>>
>>> --
>>>
>>> ---
>>> You received this message because you are subscribed
>>> to the Google Groups "CDH Users" group.
>>> To unsubscribe from this group and stop receiving
>>> emails from it, send an email to
>>> cdh-user+u...@cloudera.org.
>>> For more options, visit
>>> https://groups.google.com/a/cloudera.org/d/optout
>>> <https://groups.google.com/a/cloudera.org/d/optout>.
>>
>> --
>>
>> ---
>> You received this message because you are subscribed to
>> the Google Groups "CDH Users" group.
>> To unsubscribe from this group and stop receiving emails
>> from it, send an email to cdh-user+u...@cloudera.org.
>> For more options, visit
>> https://groups.google.com/a/cloudera.org/d/optout
>> <https://groups.google.com/a/cloudera.org/d/optout>.
>
> --
>
> ---
> You received this message because you are subscribed to the Google
> Groups "CDH Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to cdh-user+unsubscribe@cloudera.org
> <ma...@cloudera.org>.
> For more options, visit https://groups.google.com/a/cloudera.org/d/optout.
Re: upgrade CDH 5.3.1 to 5.4.1 - Event Server health issue
Posted by Motty Cruz <mo...@gmail.com>.
yes, I did upgrade to 5.4.1:
*Version*: Cloudera Express 5.4.1 (#197 built by jenkins on
20150509-0041 git: 003e06d761f80834d39c3a42431a266f0aaee736)
I did not enable trace level as suggest!
Please let me know if you find a solution, I had tried increasing memory
to that service but no successful results yet.
Thanks,
-Motty
On 06/23/2015 09:31 AM, Terry Siu wrote:
> Motty,
>
> Did you upgrade your Cloudera Manager to 5.4.1? I was comparing your
> INFO message for the EventCatcherService and noticed that your Version
> number at the end says 5.3.1.
>
>
> On Tuesday, June 23, 2015 at 9:26:42 AM UTC-7, Terry Siu wrote:
>
> Bit skeptical that enabling trace would "fix" this issue, but I
> did and no change. Event Server keeps restarting and no errors.
> Posted a message on the Cloudera Manager forum to see if anybody
> has any other ideas.
>
> On Tuesday, June 23, 2015 at 9:01:08 AM UTC-7, Terry Siu wrote:
>
> Hi Motty,
>
> No, I haven't. The Event Server has been restarting constantly
> since I left it last night and the logs do not provide any
> errors on what might be wrong. Did you try enabling the trace
> level as described by Atul? I'll see if I can figure out where
> the setting is. This is getting annoying and I'm going to
> engage Cloudera Support for this.
>
> -Terry
>
> On Tuesday, June 23, 2015 at 7:26:10 AM UTC-7, motty cruz wrote:
>
> Hello Terry,
> did you find a solution to this problem? We're
> experiencing this issue, no solution thus far.
>
> Thanks,
> -Motty
>
> On 06/22/2015 03:12 PM, Terry Siu wrote:
>> Hi guys,
>>
>> I just updated CDH from 5.3.3 to 5.4.2 and am seeing the
>> same issue with the Clouder Manager EventServer
>> (unexpected exits). I tailed the event server log and saw
>> not ERROR logs and it looked like the EventServer keeps
>> restarting itself over and over again. The only notable
>> thing I've seen in the log is WARN level message:
>>
>> 2015-06-22 15:04:10,885 WARN
>> com.cloudera.cmf.event.publish.EventStorePublisherWithRetry:
>> F
>> ailed to publish event:
>> SimpleEvent{attributes={ROLE_TYPE=[EVENTSERVER],
>> CATEGORY=[LOG_MESS
>> AGE],
>> ROLE=[mgmt-EVENTSERVER-fd434ec7afa4142d2e2f91566bafb72b],
>> SEVERITY=[IMPORTANT], SERVI
>> CE=[mgmt],
>> HOST_IDS=[10a65ece-add7-4512-82bf-9f6db5c6c7ec],
>> SERVICE_TYPE=[MGMT], LOG_LEVEL=
>> [WARN], HOSTS=[8K04.corp.pivotlink.com
>> <http://8K04.corp.pivotlink.com>],
>> EVENTCODE=[EV_LOG_EVENT]}, content=Category SERVIC
>> E is not ROLE for input role type KMS-KMS.,
>> timestamp=1435010650675}
>>
>> Anyone else got this resolved/found a workaround?
>>
>> Thanks,
>> -Terry
>>
>> On Thursday, June 4, 2015 at 7:24:16 AM UTC-7, motty cruz
>> wrote:
>>
>> Thanks for your reply Atul,
>> I believe it has to do with version of Java. We Also
>> updated Java to version 8 previously we were using
>> java version 1.7-67.
>>
>> I have not resolve the issue yet. do you mind sharing
>> the steps you taken to enable "trace".
>>
>> Thanks,
>> Motty
>>
>> On 06/03/2015 06:18 PM, Atul Kulkarni wrote:
>>> We had the same issue (upgrade from 5.3.3 to 5.4.0)
>>> - while trying to debug I enabled trace and it
>>> started working correctly - it was very odd, in the
>>> mean time we did not care about the previous events
>>> data, hence we also whacked directory (after backing
>>> it up, of course!) where it stores some intermediate
>>> data.
>>>
>>> As this is not a perfect or even sensible solution
>>> this may not work for you, we are still
>>> investigating the real reason but since we have not
>>> been able reproduce it - we are not sure how to
>>> debug further.
>>>
>>> Just thought of sharing this random act that we
>>> still can't believe turned out to be the solution.
>>> Let us know if this works for you too.
>>>
>>> Regards,
>>> Atul.
>>>
>>> On Wednesday, June 3, 2015 at 10:30:13 AM UTC-7,
>>> motty cruz wrote:
>>>
>>> Hello,
>>> Just finished upgrading CDH from 5.3.1 to 5.4.1
>>> using
>>>
>>> http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/install_upgrade_to_cdh54_parcels.html
>>> <http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/install_upgrade_to_cdh54_parcels.html>
>>>
>>>
>>> however Event health issues warnings "
>>> eventserver (cloudera1)
>>> <http://cloudera1.sscs.ad:7180/cmf/services/6/instances/28/status>
>>> Process Status, Unexpected Exits"
>>>
>>> 2015-02-24 14:18:06,012 INFO
>>> com.cloudera.cmf.eventcatcher.server.EventCatcherService:
>>> Starting EventCatcherService. JVM Args:
>>> [-XX:+UseConcMarkSweepGC,
>>> -XX:-CMSConcurrentMTEnabled, -XX:+UseParNewGC,
>>> -Dmgmt.log.file=mgmt-cmf-mgmt-EVENTSERVER-cloudera1.domain.ad.log.out,
>>> -Djava.awt.headless=true,
>>> -Djava.net.preferIPv4Stack=true, -Xms52428800,
>>> -Xmx52428800,
>>> -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh],
>>> Args: [], Version: 5.3.1 (#191 built by jenkins
>>> on 20150123-2020 git:
>>> b0377087cf605a686591e659eb14078923bc3c83)
>>>
>>> Any ideas? we're using Java 8
>>>
>>> Thanks,
>>> Motty
>>>
>>>
>>> --
>>>
>>> ---
>>> You received this message because you are subscribed
>>> to the Google Groups "CDH Users" group.
>>> To unsubscribe from this group and stop receiving
>>> emails from it, send an email to
>>> cdh-user+u...@cloudera.org.
>>> For more options, visit
>>> https://groups.google.com/a/cloudera.org/d/optout
>>> <https://groups.google.com/a/cloudera.org/d/optout>.
>>
>> --
>>
>> ---
>> You received this message because you are subscribed to
>> the Google Groups "CDH Users" group.
>> To unsubscribe from this group and stop receiving emails
>> from it, send an email to cdh-user+u...@cloudera.org.
>> For more options, visit
>> https://groups.google.com/a/cloudera.org/d/optout
>> <https://groups.google.com/a/cloudera.org/d/optout>.
>
> --
>
> ---
> You received this message because you are subscribed to the Google
> Groups "CDH Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to cdh-user+unsubscribe@cloudera.org
> <ma...@cloudera.org>.
> For more options, visit https://groups.google.com/a/cloudera.org/d/optout.
Re: upgrade CDH 5.3.1 to 5.4.1 - Event Server health issue
Posted by Motty Cruz <mo...@gmail.com>.
yes, I did upgrade to 5.4.1:
*Version*: Cloudera Express 5.4.1 (#197 built by jenkins on
20150509-0041 git: 003e06d761f80834d39c3a42431a266f0aaee736)
I did not enable trace level as suggest!
Please let me know if you find a solution, I had tried increasing memory
to that service but no successful results yet.
Thanks,
-Motty
On 06/23/2015 09:31 AM, Terry Siu wrote:
> Motty,
>
> Did you upgrade your Cloudera Manager to 5.4.1? I was comparing your
> INFO message for the EventCatcherService and noticed that your Version
> number at the end says 5.3.1.
>
>
> On Tuesday, June 23, 2015 at 9:26:42 AM UTC-7, Terry Siu wrote:
>
> Bit skeptical that enabling trace would "fix" this issue, but I
> did and no change. Event Server keeps restarting and no errors.
> Posted a message on the Cloudera Manager forum to see if anybody
> has any other ideas.
>
> On Tuesday, June 23, 2015 at 9:01:08 AM UTC-7, Terry Siu wrote:
>
> Hi Motty,
>
> No, I haven't. The Event Server has been restarting constantly
> since I left it last night and the logs do not provide any
> errors on what might be wrong. Did you try enabling the trace
> level as described by Atul? I'll see if I can figure out where
> the setting is. This is getting annoying and I'm going to
> engage Cloudera Support for this.
>
> -Terry
>
> On Tuesday, June 23, 2015 at 7:26:10 AM UTC-7, motty cruz wrote:
>
> Hello Terry,
> did you find a solution to this problem? We're
> experiencing this issue, no solution thus far.
>
> Thanks,
> -Motty
>
> On 06/22/2015 03:12 PM, Terry Siu wrote:
>> Hi guys,
>>
>> I just updated CDH from 5.3.3 to 5.4.2 and am seeing the
>> same issue with the Clouder Manager EventServer
>> (unexpected exits). I tailed the event server log and saw
>> not ERROR logs and it looked like the EventServer keeps
>> restarting itself over and over again. The only notable
>> thing I've seen in the log is WARN level message:
>>
>> 2015-06-22 15:04:10,885 WARN
>> com.cloudera.cmf.event.publish.EventStorePublisherWithRetry:
>> F
>> ailed to publish event:
>> SimpleEvent{attributes={ROLE_TYPE=[EVENTSERVER],
>> CATEGORY=[LOG_MESS
>> AGE],
>> ROLE=[mgmt-EVENTSERVER-fd434ec7afa4142d2e2f91566bafb72b],
>> SEVERITY=[IMPORTANT], SERVI
>> CE=[mgmt],
>> HOST_IDS=[10a65ece-add7-4512-82bf-9f6db5c6c7ec],
>> SERVICE_TYPE=[MGMT], LOG_LEVEL=
>> [WARN], HOSTS=[8K04.corp.pivotlink.com
>> <http://8K04.corp.pivotlink.com>],
>> EVENTCODE=[EV_LOG_EVENT]}, content=Category SERVIC
>> E is not ROLE for input role type KMS-KMS.,
>> timestamp=1435010650675}
>>
>> Anyone else got this resolved/found a workaround?
>>
>> Thanks,
>> -Terry
>>
>> On Thursday, June 4, 2015 at 7:24:16 AM UTC-7, motty cruz
>> wrote:
>>
>> Thanks for your reply Atul,
>> I believe it has to do with version of Java. We Also
>> updated Java to version 8 previously we were using
>> java version 1.7-67.
>>
>> I have not resolve the issue yet. do you mind sharing
>> the steps you taken to enable "trace".
>>
>> Thanks,
>> Motty
>>
>> On 06/03/2015 06:18 PM, Atul Kulkarni wrote:
>>> We had the same issue (upgrade from 5.3.3 to 5.4.0)
>>> - while trying to debug I enabled trace and it
>>> started working correctly - it was very odd, in the
>>> mean time we did not care about the previous events
>>> data, hence we also whacked directory (after backing
>>> it up, of course!) where it stores some intermediate
>>> data.
>>>
>>> As this is not a perfect or even sensible solution
>>> this may not work for you, we are still
>>> investigating the real reason but since we have not
>>> been able reproduce it - we are not sure how to
>>> debug further.
>>>
>>> Just thought of sharing this random act that we
>>> still can't believe turned out to be the solution.
>>> Let us know if this works for you too.
>>>
>>> Regards,
>>> Atul.
>>>
>>> On Wednesday, June 3, 2015 at 10:30:13 AM UTC-7,
>>> motty cruz wrote:
>>>
>>> Hello,
>>> Just finished upgrading CDH from 5.3.1 to 5.4.1
>>> using
>>>
>>> http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/install_upgrade_to_cdh54_parcels.html
>>> <http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/install_upgrade_to_cdh54_parcels.html>
>>>
>>>
>>> however Event health issues warnings "
>>> eventserver (cloudera1)
>>> <http://cloudera1.sscs.ad:7180/cmf/services/6/instances/28/status>
>>> Process Status, Unexpected Exits"
>>>
>>> 2015-02-24 14:18:06,012 INFO
>>> com.cloudera.cmf.eventcatcher.server.EventCatcherService:
>>> Starting EventCatcherService. JVM Args:
>>> [-XX:+UseConcMarkSweepGC,
>>> -XX:-CMSConcurrentMTEnabled, -XX:+UseParNewGC,
>>> -Dmgmt.log.file=mgmt-cmf-mgmt-EVENTSERVER-cloudera1.domain.ad.log.out,
>>> -Djava.awt.headless=true,
>>> -Djava.net.preferIPv4Stack=true, -Xms52428800,
>>> -Xmx52428800,
>>> -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh],
>>> Args: [], Version: 5.3.1 (#191 built by jenkins
>>> on 20150123-2020 git:
>>> b0377087cf605a686591e659eb14078923bc3c83)
>>>
>>> Any ideas? we're using Java 8
>>>
>>> Thanks,
>>> Motty
>>>
>>>
>>> --
>>>
>>> ---
>>> You received this message because you are subscribed
>>> to the Google Groups "CDH Users" group.
>>> To unsubscribe from this group and stop receiving
>>> emails from it, send an email to
>>> cdh-user+u...@cloudera.org.
>>> For more options, visit
>>> https://groups.google.com/a/cloudera.org/d/optout
>>> <https://groups.google.com/a/cloudera.org/d/optout>.
>>
>> --
>>
>> ---
>> You received this message because you are subscribed to
>> the Google Groups "CDH Users" group.
>> To unsubscribe from this group and stop receiving emails
>> from it, send an email to cdh-user+u...@cloudera.org.
>> For more options, visit
>> https://groups.google.com/a/cloudera.org/d/optout
>> <https://groups.google.com/a/cloudera.org/d/optout>.
>
> --
>
> ---
> You received this message because you are subscribed to the Google
> Groups "CDH Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to cdh-user+unsubscribe@cloudera.org
> <ma...@cloudera.org>.
> For more options, visit https://groups.google.com/a/cloudera.org/d/optout.
Re: upgrade CDH 5.3.1 to 5.4.1 - Event Server health issue
Posted by Motty Cruz <mo...@gmail.com>.
yes, I did upgrade to 5.4.1:
*Version*: Cloudera Express 5.4.1 (#197 built by jenkins on
20150509-0041 git: 003e06d761f80834d39c3a42431a266f0aaee736)
I did not enable trace level as suggest!
Please let me know if you find a solution, I had tried increasing memory
to that service but no successful results yet.
Thanks,
-Motty
On 06/23/2015 09:31 AM, Terry Siu wrote:
> Motty,
>
> Did you upgrade your Cloudera Manager to 5.4.1? I was comparing your
> INFO message for the EventCatcherService and noticed that your Version
> number at the end says 5.3.1.
>
>
> On Tuesday, June 23, 2015 at 9:26:42 AM UTC-7, Terry Siu wrote:
>
> Bit skeptical that enabling trace would "fix" this issue, but I
> did and no change. Event Server keeps restarting and no errors.
> Posted a message on the Cloudera Manager forum to see if anybody
> has any other ideas.
>
> On Tuesday, June 23, 2015 at 9:01:08 AM UTC-7, Terry Siu wrote:
>
> Hi Motty,
>
> No, I haven't. The Event Server has been restarting constantly
> since I left it last night and the logs do not provide any
> errors on what might be wrong. Did you try enabling the trace
> level as described by Atul? I'll see if I can figure out where
> the setting is. This is getting annoying and I'm going to
> engage Cloudera Support for this.
>
> -Terry
>
> On Tuesday, June 23, 2015 at 7:26:10 AM UTC-7, motty cruz wrote:
>
> Hello Terry,
> did you find a solution to this problem? We're
> experiencing this issue, no solution thus far.
>
> Thanks,
> -Motty
>
> On 06/22/2015 03:12 PM, Terry Siu wrote:
>> Hi guys,
>>
>> I just updated CDH from 5.3.3 to 5.4.2 and am seeing the
>> same issue with the Clouder Manager EventServer
>> (unexpected exits). I tailed the event server log and saw
>> not ERROR logs and it looked like the EventServer keeps
>> restarting itself over and over again. The only notable
>> thing I've seen in the log is WARN level message:
>>
>> 2015-06-22 15:04:10,885 WARN
>> com.cloudera.cmf.event.publish.EventStorePublisherWithRetry:
>> F
>> ailed to publish event:
>> SimpleEvent{attributes={ROLE_TYPE=[EVENTSERVER],
>> CATEGORY=[LOG_MESS
>> AGE],
>> ROLE=[mgmt-EVENTSERVER-fd434ec7afa4142d2e2f91566bafb72b],
>> SEVERITY=[IMPORTANT], SERVI
>> CE=[mgmt],
>> HOST_IDS=[10a65ece-add7-4512-82bf-9f6db5c6c7ec],
>> SERVICE_TYPE=[MGMT], LOG_LEVEL=
>> [WARN], HOSTS=[8K04.corp.pivotlink.com
>> <http://8K04.corp.pivotlink.com>],
>> EVENTCODE=[EV_LOG_EVENT]}, content=Category SERVIC
>> E is not ROLE for input role type KMS-KMS.,
>> timestamp=1435010650675}
>>
>> Anyone else got this resolved/found a workaround?
>>
>> Thanks,
>> -Terry
>>
>> On Thursday, June 4, 2015 at 7:24:16 AM UTC-7, motty cruz
>> wrote:
>>
>> Thanks for your reply Atul,
>> I believe it has to do with version of Java. We Also
>> updated Java to version 8 previously we were using
>> java version 1.7-67.
>>
>> I have not resolve the issue yet. do you mind sharing
>> the steps you taken to enable "trace".
>>
>> Thanks,
>> Motty
>>
>> On 06/03/2015 06:18 PM, Atul Kulkarni wrote:
>>> We had the same issue (upgrade from 5.3.3 to 5.4.0)
>>> - while trying to debug I enabled trace and it
>>> started working correctly - it was very odd, in the
>>> mean time we did not care about the previous events
>>> data, hence we also whacked directory (after backing
>>> it up, of course!) where it stores some intermediate
>>> data.
>>>
>>> As this is not a perfect or even sensible solution
>>> this may not work for you, we are still
>>> investigating the real reason but since we have not
>>> been able reproduce it - we are not sure how to
>>> debug further.
>>>
>>> Just thought of sharing this random act that we
>>> still can't believe turned out to be the solution.
>>> Let us know if this works for you too.
>>>
>>> Regards,
>>> Atul.
>>>
>>> On Wednesday, June 3, 2015 at 10:30:13 AM UTC-7,
>>> motty cruz wrote:
>>>
>>> Hello,
>>> Just finished upgrading CDH from 5.3.1 to 5.4.1
>>> using
>>>
>>> http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/install_upgrade_to_cdh54_parcels.html
>>> <http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/install_upgrade_to_cdh54_parcels.html>
>>>
>>>
>>> however Event health issues warnings "
>>> eventserver (cloudera1)
>>> <http://cloudera1.sscs.ad:7180/cmf/services/6/instances/28/status>
>>> Process Status, Unexpected Exits"
>>>
>>> 2015-02-24 14:18:06,012 INFO
>>> com.cloudera.cmf.eventcatcher.server.EventCatcherService:
>>> Starting EventCatcherService. JVM Args:
>>> [-XX:+UseConcMarkSweepGC,
>>> -XX:-CMSConcurrentMTEnabled, -XX:+UseParNewGC,
>>> -Dmgmt.log.file=mgmt-cmf-mgmt-EVENTSERVER-cloudera1.domain.ad.log.out,
>>> -Djava.awt.headless=true,
>>> -Djava.net.preferIPv4Stack=true, -Xms52428800,
>>> -Xmx52428800,
>>> -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh],
>>> Args: [], Version: 5.3.1 (#191 built by jenkins
>>> on 20150123-2020 git:
>>> b0377087cf605a686591e659eb14078923bc3c83)
>>>
>>> Any ideas? we're using Java 8
>>>
>>> Thanks,
>>> Motty
>>>
>>>
>>> --
>>>
>>> ---
>>> You received this message because you are subscribed
>>> to the Google Groups "CDH Users" group.
>>> To unsubscribe from this group and stop receiving
>>> emails from it, send an email to
>>> cdh-user+u...@cloudera.org.
>>> For more options, visit
>>> https://groups.google.com/a/cloudera.org/d/optout
>>> <https://groups.google.com/a/cloudera.org/d/optout>.
>>
>> --
>>
>> ---
>> You received this message because you are subscribed to
>> the Google Groups "CDH Users" group.
>> To unsubscribe from this group and stop receiving emails
>> from it, send an email to cdh-user+u...@cloudera.org.
>> For more options, visit
>> https://groups.google.com/a/cloudera.org/d/optout
>> <https://groups.google.com/a/cloudera.org/d/optout>.
>
> --
>
> ---
> You received this message because you are subscribed to the Google
> Groups "CDH Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to cdh-user+unsubscribe@cloudera.org
> <ma...@cloudera.org>.
> For more options, visit https://groups.google.com/a/cloudera.org/d/optout.
Re: upgrade CDH 5.3.1 to 5.4.1 - Event Server health issue
Posted by Motty Cruz <mo...@gmail.com>.
Hello Terry,
did you find a solution to this problem? We're experiencing this issue,
no solution thus far.
Thanks,
-Motty
On 06/22/2015 03:12 PM, Terry Siu wrote:
> Hi guys,
>
> I just updated CDH from 5.3.3 to 5.4.2 and am seeing the same issue
> with the Clouder Manager EventServer (unexpected exits). I tailed the
> event server log and saw not ERROR logs and it looked like the
> EventServer keeps restarting itself over and over again. The only
> notable thing I've seen in the log is WARN level message:
>
> 2015-06-22 15:04:10,885 WARN
> com.cloudera.cmf.event.publish.EventStorePublisherWithRetry: F
> ailed to publish event:
> SimpleEvent{attributes={ROLE_TYPE=[EVENTSERVER], CATEGORY=[LOG_MESS
> AGE], ROLE=[mgmt-EVENTSERVER-fd434ec7afa4142d2e2f91566bafb72b],
> SEVERITY=[IMPORTANT], SERVI
> CE=[mgmt], HOST_IDS=[10a65ece-add7-4512-82bf-9f6db5c6c7ec],
> SERVICE_TYPE=[MGMT], LOG_LEVEL=
> [WARN], HOSTS=[8K04.corp.pivotlink.com], EVENTCODE=[EV_LOG_EVENT]},
> content=Category SERVIC
> E is not ROLE for input role type KMS-KMS., timestamp=1435010650675}
>
> Anyone else got this resolved/found a workaround?
>
> Thanks,
> -Terry
>
> On Thursday, June 4, 2015 at 7:24:16 AM UTC-7, motty cruz wrote:
>
> Thanks for your reply Atul,
> I believe it has to do with version of Java. We Also updated Java
> to version 8 previously we were using java version 1.7-67.
>
> I have not resolve the issue yet. do you mind sharing the steps
> you taken to enable "trace".
>
> Thanks,
> Motty
>
> On 06/03/2015 06:18 PM, Atul Kulkarni wrote:
>> We had the same issue (upgrade from 5.3.3 to 5.4.0) - while
>> trying to debug I enabled trace and it started working correctly
>> - it was very odd, in the mean time we did not care about the
>> previous events data, hence we also whacked directory (after
>> backing it up, of course!) where it stores some intermediate data.
>>
>> As this is not a perfect or even sensible solution this may not
>> work for you, we are still investigating the real reason but
>> since we have not been able reproduce it - we are not sure how to
>> debug further.
>>
>> Just thought of sharing this random act that we still can't
>> believe turned out to be the solution. Let us know if this works
>> for you too.
>>
>> Regards,
>> Atul.
>>
>> On Wednesday, June 3, 2015 at 10:30:13 AM UTC-7, motty cruz wrote:
>>
>> Hello,
>> Just finished upgrading CDH from 5.3.1 to 5.4.1 using
>>
>> http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/install_upgrade_to_cdh54_parcels.html
>> <http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/install_upgrade_to_cdh54_parcels.html>
>>
>>
>> however Event health issues warnings " eventserver
>> (cloudera1)
>> <http://cloudera1.sscs.ad:7180/cmf/services/6/instances/28/status>
>> Process Status, Unexpected Exits"
>>
>> 2015-02-24 14:18:06,012 INFO
>> com.cloudera.cmf.eventcatcher.server.EventCatcherService:
>> Starting EventCatcherService. JVM Args:
>> [-XX:+UseConcMarkSweepGC, -XX:-CMSConcurrentMTEnabled,
>> -XX:+UseParNewGC,
>> -Dmgmt.log.file=mgmt-cmf-mgmt-EVENTSERVER-cloudera1.domain.ad.log.out,
>> -Djava.awt.headless=true, -Djava.net.preferIPv4Stack=true,
>> -Xms52428800, -Xmx52428800,
>> -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh],
>> Args: [], Version: 5.3.1 (#191 built by jenkins on
>> 20150123-2020 git: b0377087cf605a686591e659eb14078923bc3c83)
>>
>> Any ideas? we're using Java 8
>>
>> Thanks,
>> Motty
>>
>>
>> --
>>
>> ---
>> You received this message because you are subscribed to the
>> Google Groups "CDH Users" group.
>> To unsubscribe from this group and stop receiving emails from it,
>> send an email to cdh-user+u...@cloudera.org <javascript:>.
>> For more options, visit
>> https://groups.google.com/a/cloudera.org/d/optout
>> <https://groups.google.com/a/cloudera.org/d/optout>.
>
> --
>
> ---
> You received this message because you are subscribed to the Google
> Groups "CDH Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to cdh-user+unsubscribe@cloudera.org
> <ma...@cloudera.org>.
> For more options, visit https://groups.google.com/a/cloudera.org/d/optout.
Re: upgrade CDH 5.3.1 to 5.4.1 - Event Server health issue
Posted by Motty Cruz <mo...@gmail.com>.
Hello Terry,
did you find a solution to this problem? We're experiencing this issue,
no solution thus far.
Thanks,
-Motty
On 06/22/2015 03:12 PM, Terry Siu wrote:
> Hi guys,
>
> I just updated CDH from 5.3.3 to 5.4.2 and am seeing the same issue
> with the Clouder Manager EventServer (unexpected exits). I tailed the
> event server log and saw not ERROR logs and it looked like the
> EventServer keeps restarting itself over and over again. The only
> notable thing I've seen in the log is WARN level message:
>
> 2015-06-22 15:04:10,885 WARN
> com.cloudera.cmf.event.publish.EventStorePublisherWithRetry: F
> ailed to publish event:
> SimpleEvent{attributes={ROLE_TYPE=[EVENTSERVER], CATEGORY=[LOG_MESS
> AGE], ROLE=[mgmt-EVENTSERVER-fd434ec7afa4142d2e2f91566bafb72b],
> SEVERITY=[IMPORTANT], SERVI
> CE=[mgmt], HOST_IDS=[10a65ece-add7-4512-82bf-9f6db5c6c7ec],
> SERVICE_TYPE=[MGMT], LOG_LEVEL=
> [WARN], HOSTS=[8K04.corp.pivotlink.com], EVENTCODE=[EV_LOG_EVENT]},
> content=Category SERVIC
> E is not ROLE for input role type KMS-KMS., timestamp=1435010650675}
>
> Anyone else got this resolved/found a workaround?
>
> Thanks,
> -Terry
>
> On Thursday, June 4, 2015 at 7:24:16 AM UTC-7, motty cruz wrote:
>
> Thanks for your reply Atul,
> I believe it has to do with version of Java. We Also updated Java
> to version 8 previously we were using java version 1.7-67.
>
> I have not resolve the issue yet. do you mind sharing the steps
> you taken to enable "trace".
>
> Thanks,
> Motty
>
> On 06/03/2015 06:18 PM, Atul Kulkarni wrote:
>> We had the same issue (upgrade from 5.3.3 to 5.4.0) - while
>> trying to debug I enabled trace and it started working correctly
>> - it was very odd, in the mean time we did not care about the
>> previous events data, hence we also whacked directory (after
>> backing it up, of course!) where it stores some intermediate data.
>>
>> As this is not a perfect or even sensible solution this may not
>> work for you, we are still investigating the real reason but
>> since we have not been able reproduce it - we are not sure how to
>> debug further.
>>
>> Just thought of sharing this random act that we still can't
>> believe turned out to be the solution. Let us know if this works
>> for you too.
>>
>> Regards,
>> Atul.
>>
>> On Wednesday, June 3, 2015 at 10:30:13 AM UTC-7, motty cruz wrote:
>>
>> Hello,
>> Just finished upgrading CDH from 5.3.1 to 5.4.1 using
>>
>> http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/install_upgrade_to_cdh54_parcels.html
>> <http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/install_upgrade_to_cdh54_parcels.html>
>>
>>
>> however Event health issues warnings " eventserver
>> (cloudera1)
>> <http://cloudera1.sscs.ad:7180/cmf/services/6/instances/28/status>
>> Process Status, Unexpected Exits"
>>
>> 2015-02-24 14:18:06,012 INFO
>> com.cloudera.cmf.eventcatcher.server.EventCatcherService:
>> Starting EventCatcherService. JVM Args:
>> [-XX:+UseConcMarkSweepGC, -XX:-CMSConcurrentMTEnabled,
>> -XX:+UseParNewGC,
>> -Dmgmt.log.file=mgmt-cmf-mgmt-EVENTSERVER-cloudera1.domain.ad.log.out,
>> -Djava.awt.headless=true, -Djava.net.preferIPv4Stack=true,
>> -Xms52428800, -Xmx52428800,
>> -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh],
>> Args: [], Version: 5.3.1 (#191 built by jenkins on
>> 20150123-2020 git: b0377087cf605a686591e659eb14078923bc3c83)
>>
>> Any ideas? we're using Java 8
>>
>> Thanks,
>> Motty
>>
>>
>> --
>>
>> ---
>> You received this message because you are subscribed to the
>> Google Groups "CDH Users" group.
>> To unsubscribe from this group and stop receiving emails from it,
>> send an email to cdh-user+u...@cloudera.org <javascript:>.
>> For more options, visit
>> https://groups.google.com/a/cloudera.org/d/optout
>> <https://groups.google.com/a/cloudera.org/d/optout>.
>
> --
>
> ---
> You received this message because you are subscribed to the Google
> Groups "CDH Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to cdh-user+unsubscribe@cloudera.org
> <ma...@cloudera.org>.
> For more options, visit https://groups.google.com/a/cloudera.org/d/optout.
Re: upgrade CDH 5.3.1 to 5.4.1 - Event Server health issue
Posted by Motty Cruz <mo...@gmail.com>.
Hello Terry,
did you find a solution to this problem? We're experiencing this issue,
no solution thus far.
Thanks,
-Motty
On 06/22/2015 03:12 PM, Terry Siu wrote:
> Hi guys,
>
> I just updated CDH from 5.3.3 to 5.4.2 and am seeing the same issue
> with the Clouder Manager EventServer (unexpected exits). I tailed the
> event server log and saw not ERROR logs and it looked like the
> EventServer keeps restarting itself over and over again. The only
> notable thing I've seen in the log is WARN level message:
>
> 2015-06-22 15:04:10,885 WARN
> com.cloudera.cmf.event.publish.EventStorePublisherWithRetry: F
> ailed to publish event:
> SimpleEvent{attributes={ROLE_TYPE=[EVENTSERVER], CATEGORY=[LOG_MESS
> AGE], ROLE=[mgmt-EVENTSERVER-fd434ec7afa4142d2e2f91566bafb72b],
> SEVERITY=[IMPORTANT], SERVI
> CE=[mgmt], HOST_IDS=[10a65ece-add7-4512-82bf-9f6db5c6c7ec],
> SERVICE_TYPE=[MGMT], LOG_LEVEL=
> [WARN], HOSTS=[8K04.corp.pivotlink.com], EVENTCODE=[EV_LOG_EVENT]},
> content=Category SERVIC
> E is not ROLE for input role type KMS-KMS., timestamp=1435010650675}
>
> Anyone else got this resolved/found a workaround?
>
> Thanks,
> -Terry
>
> On Thursday, June 4, 2015 at 7:24:16 AM UTC-7, motty cruz wrote:
>
> Thanks for your reply Atul,
> I believe it has to do with version of Java. We Also updated Java
> to version 8 previously we were using java version 1.7-67.
>
> I have not resolve the issue yet. do you mind sharing the steps
> you taken to enable "trace".
>
> Thanks,
> Motty
>
> On 06/03/2015 06:18 PM, Atul Kulkarni wrote:
>> We had the same issue (upgrade from 5.3.3 to 5.4.0) - while
>> trying to debug I enabled trace and it started working correctly
>> - it was very odd, in the mean time we did not care about the
>> previous events data, hence we also whacked directory (after
>> backing it up, of course!) where it stores some intermediate data.
>>
>> As this is not a perfect or even sensible solution this may not
>> work for you, we are still investigating the real reason but
>> since we have not been able reproduce it - we are not sure how to
>> debug further.
>>
>> Just thought of sharing this random act that we still can't
>> believe turned out to be the solution. Let us know if this works
>> for you too.
>>
>> Regards,
>> Atul.
>>
>> On Wednesday, June 3, 2015 at 10:30:13 AM UTC-7, motty cruz wrote:
>>
>> Hello,
>> Just finished upgrading CDH from 5.3.1 to 5.4.1 using
>>
>> http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/install_upgrade_to_cdh54_parcels.html
>> <http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/install_upgrade_to_cdh54_parcels.html>
>>
>>
>> however Event health issues warnings " eventserver
>> (cloudera1)
>> <http://cloudera1.sscs.ad:7180/cmf/services/6/instances/28/status>
>> Process Status, Unexpected Exits"
>>
>> 2015-02-24 14:18:06,012 INFO
>> com.cloudera.cmf.eventcatcher.server.EventCatcherService:
>> Starting EventCatcherService. JVM Args:
>> [-XX:+UseConcMarkSweepGC, -XX:-CMSConcurrentMTEnabled,
>> -XX:+UseParNewGC,
>> -Dmgmt.log.file=mgmt-cmf-mgmt-EVENTSERVER-cloudera1.domain.ad.log.out,
>> -Djava.awt.headless=true, -Djava.net.preferIPv4Stack=true,
>> -Xms52428800, -Xmx52428800,
>> -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh],
>> Args: [], Version: 5.3.1 (#191 built by jenkins on
>> 20150123-2020 git: b0377087cf605a686591e659eb14078923bc3c83)
>>
>> Any ideas? we're using Java 8
>>
>> Thanks,
>> Motty
>>
>>
>> --
>>
>> ---
>> You received this message because you are subscribed to the
>> Google Groups "CDH Users" group.
>> To unsubscribe from this group and stop receiving emails from it,
>> send an email to cdh-user+u...@cloudera.org <javascript:>.
>> For more options, visit
>> https://groups.google.com/a/cloudera.org/d/optout
>> <https://groups.google.com/a/cloudera.org/d/optout>.
>
> --
>
> ---
> You received this message because you are subscribed to the Google
> Groups "CDH Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to cdh-user+unsubscribe@cloudera.org
> <ma...@cloudera.org>.
> For more options, visit https://groups.google.com/a/cloudera.org/d/optout.
Re: upgrade CDH 5.3.1 to 5.4.1 - Event Server health issue
Posted by Motty Cruz <mo...@gmail.com>.
Hello Terry,
did you find a solution to this problem? We're experiencing this issue,
no solution thus far.
Thanks,
-Motty
On 06/22/2015 03:12 PM, Terry Siu wrote:
> Hi guys,
>
> I just updated CDH from 5.3.3 to 5.4.2 and am seeing the same issue
> with the Clouder Manager EventServer (unexpected exits). I tailed the
> event server log and saw not ERROR logs and it looked like the
> EventServer keeps restarting itself over and over again. The only
> notable thing I've seen in the log is WARN level message:
>
> 2015-06-22 15:04:10,885 WARN
> com.cloudera.cmf.event.publish.EventStorePublisherWithRetry: F
> ailed to publish event:
> SimpleEvent{attributes={ROLE_TYPE=[EVENTSERVER], CATEGORY=[LOG_MESS
> AGE], ROLE=[mgmt-EVENTSERVER-fd434ec7afa4142d2e2f91566bafb72b],
> SEVERITY=[IMPORTANT], SERVI
> CE=[mgmt], HOST_IDS=[10a65ece-add7-4512-82bf-9f6db5c6c7ec],
> SERVICE_TYPE=[MGMT], LOG_LEVEL=
> [WARN], HOSTS=[8K04.corp.pivotlink.com], EVENTCODE=[EV_LOG_EVENT]},
> content=Category SERVIC
> E is not ROLE for input role type KMS-KMS., timestamp=1435010650675}
>
> Anyone else got this resolved/found a workaround?
>
> Thanks,
> -Terry
>
> On Thursday, June 4, 2015 at 7:24:16 AM UTC-7, motty cruz wrote:
>
> Thanks for your reply Atul,
> I believe it has to do with version of Java. We Also updated Java
> to version 8 previously we were using java version 1.7-67.
>
> I have not resolve the issue yet. do you mind sharing the steps
> you taken to enable "trace".
>
> Thanks,
> Motty
>
> On 06/03/2015 06:18 PM, Atul Kulkarni wrote:
>> We had the same issue (upgrade from 5.3.3 to 5.4.0) - while
>> trying to debug I enabled trace and it started working correctly
>> - it was very odd, in the mean time we did not care about the
>> previous events data, hence we also whacked directory (after
>> backing it up, of course!) where it stores some intermediate data.
>>
>> As this is not a perfect or even sensible solution this may not
>> work for you, we are still investigating the real reason but
>> since we have not been able reproduce it - we are not sure how to
>> debug further.
>>
>> Just thought of sharing this random act that we still can't
>> believe turned out to be the solution. Let us know if this works
>> for you too.
>>
>> Regards,
>> Atul.
>>
>> On Wednesday, June 3, 2015 at 10:30:13 AM UTC-7, motty cruz wrote:
>>
>> Hello,
>> Just finished upgrading CDH from 5.3.1 to 5.4.1 using
>>
>> http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/install_upgrade_to_cdh54_parcels.html
>> <http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/install_upgrade_to_cdh54_parcels.html>
>>
>>
>> however Event health issues warnings " eventserver
>> (cloudera1)
>> <http://cloudera1.sscs.ad:7180/cmf/services/6/instances/28/status>
>> Process Status, Unexpected Exits"
>>
>> 2015-02-24 14:18:06,012 INFO
>> com.cloudera.cmf.eventcatcher.server.EventCatcherService:
>> Starting EventCatcherService. JVM Args:
>> [-XX:+UseConcMarkSweepGC, -XX:-CMSConcurrentMTEnabled,
>> -XX:+UseParNewGC,
>> -Dmgmt.log.file=mgmt-cmf-mgmt-EVENTSERVER-cloudera1.domain.ad.log.out,
>> -Djava.awt.headless=true, -Djava.net.preferIPv4Stack=true,
>> -Xms52428800, -Xmx52428800,
>> -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh],
>> Args: [], Version: 5.3.1 (#191 built by jenkins on
>> 20150123-2020 git: b0377087cf605a686591e659eb14078923bc3c83)
>>
>> Any ideas? we're using Java 8
>>
>> Thanks,
>> Motty
>>
>>
>> --
>>
>> ---
>> You received this message because you are subscribed to the
>> Google Groups "CDH Users" group.
>> To unsubscribe from this group and stop receiving emails from it,
>> send an email to cdh-user+u...@cloudera.org <javascript:>.
>> For more options, visit
>> https://groups.google.com/a/cloudera.org/d/optout
>> <https://groups.google.com/a/cloudera.org/d/optout>.
>
> --
>
> ---
> You received this message because you are subscribed to the Google
> Groups "CDH Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to cdh-user+unsubscribe@cloudera.org
> <ma...@cloudera.org>.
> For more options, visit https://groups.google.com/a/cloudera.org/d/optout.
Re: upgrade CDH 5.3.1 to 5.4.1 - Event Server health issue
Posted by Motty Cruz <mo...@gmail.com>.
Thanks for your reply Atul,
I believe it has to do with version of Java. We Also updated Java to
version 8 previously we were using java version 1.7-67.
I have not resolve the issue yet. do you mind sharing the steps you
taken to enable "trace".
Thanks,
Motty
On 06/03/2015 06:18 PM, Atul Kulkarni wrote:
> We had the same issue (upgrade from 5.3.3 to 5.4.0) - while trying to
> debug I enabled trace and it started working correctly - it was very
> odd, in the mean time we did not care about the previous events data,
> hence we also whacked directory (after backing it up, of course!)
> where it stores some intermediate data.
>
> As this is not a perfect or even sensible solution this may not work
> for you, we are still investigating the real reason but since we have
> not been able reproduce it - we are not sure how to debug further.
>
> Just thought of sharing this random act that we still can't believe
> turned out to be the solution. Let us know if this works for you too.
>
> Regards,
> Atul.
>
> On Wednesday, June 3, 2015 at 10:30:13 AM UTC-7, motty cruz wrote:
>
> Hello,
> Just finished upgrading CDH from 5.3.1 to 5.4.1 using
>
> http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/install_upgrade_to_cdh54_parcels.html
> <http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/install_upgrade_to_cdh54_parcels.html>
>
>
> however Event health issues warnings " eventserver (cloudera1)
> <http://cloudera1.sscs.ad:7180/cmf/services/6/instances/28/status>
> Process Status, Unexpected Exits"
>
> 2015-02-24 14:18:06,012 INFO
> com.cloudera.cmf.eventcatcher.server.EventCatcherService: Starting
> EventCatcherService. JVM Args: [-XX:+UseConcMarkSweepGC,
> -XX:-CMSConcurrentMTEnabled, -XX:+UseParNewGC,
> -Dmgmt.log.file=mgmt-cmf-mgmt-EVENTSERVER-cloudera1.domain.ad.log.out,
> -Djava.awt.headless=true, -Djava.net.preferIPv4Stack=true,
> -Xms52428800, -Xmx52428800,
> -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh],
> Args: [], Version: 5.3.1 (#191 built by jenkins on 20150123-2020
> git: b0377087cf605a686591e659eb14078923bc3c83)
>
> Any ideas? we're using Java 8
>
> Thanks,
> Motty
>
>
> --
>
> ---
> You received this message because you are subscribed to the Google
> Groups "CDH Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to cdh-user+unsubscribe@cloudera.org
> <ma...@cloudera.org>.
> For more options, visit https://groups.google.com/a/cloudera.org/d/optout.
Re: upgrade CDH 5.3.1 to 5.4.1 - Event Server health issue
Posted by Motty Cruz <mo...@gmail.com>.
Thanks for your reply Atul,
I believe it has to do with version of Java. We Also updated Java to
version 8 previously we were using java version 1.7-67.
I have not resolve the issue yet. do you mind sharing the steps you
taken to enable "trace".
Thanks,
Motty
On 06/03/2015 06:18 PM, Atul Kulkarni wrote:
> We had the same issue (upgrade from 5.3.3 to 5.4.0) - while trying to
> debug I enabled trace and it started working correctly - it was very
> odd, in the mean time we did not care about the previous events data,
> hence we also whacked directory (after backing it up, of course!)
> where it stores some intermediate data.
>
> As this is not a perfect or even sensible solution this may not work
> for you, we are still investigating the real reason but since we have
> not been able reproduce it - we are not sure how to debug further.
>
> Just thought of sharing this random act that we still can't believe
> turned out to be the solution. Let us know if this works for you too.
>
> Regards,
> Atul.
>
> On Wednesday, June 3, 2015 at 10:30:13 AM UTC-7, motty cruz wrote:
>
> Hello,
> Just finished upgrading CDH from 5.3.1 to 5.4.1 using
>
> http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/install_upgrade_to_cdh54_parcels.html
> <http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/install_upgrade_to_cdh54_parcels.html>
>
>
> however Event health issues warnings " eventserver (cloudera1)
> <http://cloudera1.sscs.ad:7180/cmf/services/6/instances/28/status>
> Process Status, Unexpected Exits"
>
> 2015-02-24 14:18:06,012 INFO
> com.cloudera.cmf.eventcatcher.server.EventCatcherService: Starting
> EventCatcherService. JVM Args: [-XX:+UseConcMarkSweepGC,
> -XX:-CMSConcurrentMTEnabled, -XX:+UseParNewGC,
> -Dmgmt.log.file=mgmt-cmf-mgmt-EVENTSERVER-cloudera1.domain.ad.log.out,
> -Djava.awt.headless=true, -Djava.net.preferIPv4Stack=true,
> -Xms52428800, -Xmx52428800,
> -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh],
> Args: [], Version: 5.3.1 (#191 built by jenkins on 20150123-2020
> git: b0377087cf605a686591e659eb14078923bc3c83)
>
> Any ideas? we're using Java 8
>
> Thanks,
> Motty
>
>
> --
>
> ---
> You received this message because you are subscribed to the Google
> Groups "CDH Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to cdh-user+unsubscribe@cloudera.org
> <ma...@cloudera.org>.
> For more options, visit https://groups.google.com/a/cloudera.org/d/optout.
Re: upgrade CDH 5.3.1 to 5.4.1 - Event Server health issue
Posted by Motty Cruz <mo...@gmail.com>.
Thanks for your reply Atul,
I believe it has to do with version of Java. We Also updated Java to
version 8 previously we were using java version 1.7-67.
I have not resolve the issue yet. do you mind sharing the steps you
taken to enable "trace".
Thanks,
Motty
On 06/03/2015 06:18 PM, Atul Kulkarni wrote:
> We had the same issue (upgrade from 5.3.3 to 5.4.0) - while trying to
> debug I enabled trace and it started working correctly - it was very
> odd, in the mean time we did not care about the previous events data,
> hence we also whacked directory (after backing it up, of course!)
> where it stores some intermediate data.
>
> As this is not a perfect or even sensible solution this may not work
> for you, we are still investigating the real reason but since we have
> not been able reproduce it - we are not sure how to debug further.
>
> Just thought of sharing this random act that we still can't believe
> turned out to be the solution. Let us know if this works for you too.
>
> Regards,
> Atul.
>
> On Wednesday, June 3, 2015 at 10:30:13 AM UTC-7, motty cruz wrote:
>
> Hello,
> Just finished upgrading CDH from 5.3.1 to 5.4.1 using
>
> http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/install_upgrade_to_cdh54_parcels.html
> <http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/install_upgrade_to_cdh54_parcels.html>
>
>
> however Event health issues warnings " eventserver (cloudera1)
> <http://cloudera1.sscs.ad:7180/cmf/services/6/instances/28/status>
> Process Status, Unexpected Exits"
>
> 2015-02-24 14:18:06,012 INFO
> com.cloudera.cmf.eventcatcher.server.EventCatcherService: Starting
> EventCatcherService. JVM Args: [-XX:+UseConcMarkSweepGC,
> -XX:-CMSConcurrentMTEnabled, -XX:+UseParNewGC,
> -Dmgmt.log.file=mgmt-cmf-mgmt-EVENTSERVER-cloudera1.domain.ad.log.out,
> -Djava.awt.headless=true, -Djava.net.preferIPv4Stack=true,
> -Xms52428800, -Xmx52428800,
> -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh],
> Args: [], Version: 5.3.1 (#191 built by jenkins on 20150123-2020
> git: b0377087cf605a686591e659eb14078923bc3c83)
>
> Any ideas? we're using Java 8
>
> Thanks,
> Motty
>
>
> --
>
> ---
> You received this message because you are subscribed to the Google
> Groups "CDH Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to cdh-user+unsubscribe@cloudera.org
> <ma...@cloudera.org>.
> For more options, visit https://groups.google.com/a/cloudera.org/d/optout.
Re: upgrade CDH 5.3.1 to 5.4.1 - Event Server health issue
Posted by Motty Cruz <mo...@gmail.com>.
Thanks for your reply Atul,
I believe it has to do with version of Java. We Also updated Java to
version 8 previously we were using java version 1.7-67.
I have not resolve the issue yet. do you mind sharing the steps you
taken to enable "trace".
Thanks,
Motty
On 06/03/2015 06:18 PM, Atul Kulkarni wrote:
> We had the same issue (upgrade from 5.3.3 to 5.4.0) - while trying to
> debug I enabled trace and it started working correctly - it was very
> odd, in the mean time we did not care about the previous events data,
> hence we also whacked directory (after backing it up, of course!)
> where it stores some intermediate data.
>
> As this is not a perfect or even sensible solution this may not work
> for you, we are still investigating the real reason but since we have
> not been able reproduce it - we are not sure how to debug further.
>
> Just thought of sharing this random act that we still can't believe
> turned out to be the solution. Let us know if this works for you too.
>
> Regards,
> Atul.
>
> On Wednesday, June 3, 2015 at 10:30:13 AM UTC-7, motty cruz wrote:
>
> Hello,
> Just finished upgrading CDH from 5.3.1 to 5.4.1 using
>
> http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/install_upgrade_to_cdh54_parcels.html
> <http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/install_upgrade_to_cdh54_parcels.html>
>
>
> however Event health issues warnings " eventserver (cloudera1)
> <http://cloudera1.sscs.ad:7180/cmf/services/6/instances/28/status>
> Process Status, Unexpected Exits"
>
> 2015-02-24 14:18:06,012 INFO
> com.cloudera.cmf.eventcatcher.server.EventCatcherService: Starting
> EventCatcherService. JVM Args: [-XX:+UseConcMarkSweepGC,
> -XX:-CMSConcurrentMTEnabled, -XX:+UseParNewGC,
> -Dmgmt.log.file=mgmt-cmf-mgmt-EVENTSERVER-cloudera1.domain.ad.log.out,
> -Djava.awt.headless=true, -Djava.net.preferIPv4Stack=true,
> -Xms52428800, -Xmx52428800,
> -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh],
> Args: [], Version: 5.3.1 (#191 built by jenkins on 20150123-2020
> git: b0377087cf605a686591e659eb14078923bc3c83)
>
> Any ideas? we're using Java 8
>
> Thanks,
> Motty
>
>
> --
>
> ---
> You received this message because you are subscribed to the Google
> Groups "CDH Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to cdh-user+unsubscribe@cloudera.org
> <ma...@cloudera.org>.
> For more options, visit https://groups.google.com/a/cloudera.org/d/optout.