You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@chukwa.apache.org by "Eric Yang (JIRA)" <ji...@apache.org> on 2009/12/08 20:53:18 UTC

[jira] Created: (CHUKWA-426) High frequency of adaptor registration and unregistration could prevent existing adaptor to stall

High frequency of adaptor registration and unregistration could prevent existing adaptor to stall
-------------------------------------------------------------------------------------------------

                 Key: CHUKWA-426
                 URL: https://issues.apache.org/jira/browse/CHUKWA-426
             Project: Hadoop Chukwa
          Issue Type: Bug
          Components: data collection
         Environment: Redhat EL 5.1, Java 6
            Reporter: Eric Yang
            Priority: Minor


The existing adaptor might not retrieve data because agent was designed to loop through the adaptors to send the information.  When the frequency of registering and unregistering is more frequent than chunk writer, like many tasks attempting to stream their metrics over, the existing adaptors do not get loops.  The easy workaround is to have socket appender ready, and we don't need to worry about this issue.  However, this is still an general issue, if the chunk writer write time is longer than registration interval with high registration frequencies.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (CHUKWA-426) High frequency of adaptor registration and unregistration could prevent existing adaptor to stall

Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CHUKWA-426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787717#action_12787717 ] 

Ari Rabkin commented on CHUKWA-426:
-----------------------------------

I don't completely understand this problem. Is this a collector or agent problem? Can you explain in a bit more detail?

> High frequency of adaptor registration and unregistration could prevent existing adaptor to stall
> -------------------------------------------------------------------------------------------------
>
>                 Key: CHUKWA-426
>                 URL: https://issues.apache.org/jira/browse/CHUKWA-426
>             Project: Hadoop Chukwa
>          Issue Type: Bug
>          Components: data collection
>         Environment: Redhat EL 5.1, Java 6
>            Reporter: Eric Yang
>            Priority: Minor
>
> The existing adaptor might not retrieve data because agent was designed to loop through the adaptors to send the information.  When the frequency of registering and unregistering is more frequent than chunk writer, like many tasks attempting to stream their metrics over, the existing adaptors do not get loops.  The easy workaround is to have socket appender ready, and we don't need to worry about this issue.  However, this is still an general issue, if the chunk writer write time is longer than registration interval with high registration frequencies.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (CHUKWA-426) High frequency of adaptor registration and unregistration could prevent existing adaptor to stall

Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CHUKWA-426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787774#action_12787774 ] 

Eric Yang commented on CHUKWA-426:
----------------------------------

I suspected it was stuck in org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.FileTailer run method.  Line 90 to 93.  I am not sure if that was possible.  If not possible, maybe invalidate this bug.  However, my system recovered after I shutdown the excessive adaptor registrations.

> High frequency of adaptor registration and unregistration could prevent existing adaptor to stall
> -------------------------------------------------------------------------------------------------
>
>                 Key: CHUKWA-426
>                 URL: https://issues.apache.org/jira/browse/CHUKWA-426
>             Project: Hadoop Chukwa
>          Issue Type: Bug
>          Components: data collection
>         Environment: Redhat EL 5.1, Java 6
>            Reporter: Eric Yang
>            Priority: Minor
>
> The existing adaptor might not retrieve data because agent was designed to loop through the adaptors to send the information.  When the frequency of registering and unregistering is more frequent than chunk writer, like many tasks attempting to stream their metrics over, the existing adaptors do not get loops.  The easy workaround is to have socket appender ready, and we don't need to worry about this issue.  However, this is still an general issue, if the chunk writer write time is longer than registration interval with high registration frequencies.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (CHUKWA-426) High frequency of adaptor registration and unregistration could prevent existing adaptor to stall

Posted by "Jerome Boulon (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CHUKWA-426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787728#action_12787728 ] 

Jerome Boulon commented on CHUKWA-426:
--------------------------------------

@Eric, are you referring to a potential problem when an application send a register command then an unregister command before the agent had a chance to look at/open the file? Is that what you're referring to?

If yes, this could happen if you're sending an hardStop instead of a stop. If you're sending a start, this will create an adaptor and sending a stop after that will move the adaptor in stop mode but should not destroy it before it gets a chance to send all the data.
Can you clarify?

I can see a use case for the socket adaptor (I should still have my prototype somewhere if needed) but what you're describing should just work with the current chukwa implementation.

> High frequency of adaptor registration and unregistration could prevent existing adaptor to stall
> -------------------------------------------------------------------------------------------------
>
>                 Key: CHUKWA-426
>                 URL: https://issues.apache.org/jira/browse/CHUKWA-426
>             Project: Hadoop Chukwa
>          Issue Type: Bug
>          Components: data collection
>         Environment: Redhat EL 5.1, Java 6
>            Reporter: Eric Yang
>            Priority: Minor
>
> The existing adaptor might not retrieve data because agent was designed to loop through the adaptors to send the information.  When the frequency of registering and unregistering is more frequent than chunk writer, like many tasks attempting to stream their metrics over, the existing adaptors do not get loops.  The easy workaround is to have socket appender ready, and we don't need to worry about this issue.  However, this is still an general issue, if the chunk writer write time is longer than registration interval with high registration frequencies.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.