You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "Hari Shreedharan (JIRA)" <ji...@apache.org> on 2012/07/05 00:47:34 UTC

[jira] [Created] (FLUME-1339) Implement an exec source like client API

Hari Shreedharan created FLUME-1339:
---------------------------------------

             Summary: Implement an exec source like client API
                 Key: FLUME-1339
                 URL: https://issues.apache.org/jira/browse/FLUME-1339
             Project: Flume
          Issue Type: Bug
            Reporter: Hari Shreedharan


Originally when the exec source was implemented there was no SDK. It would be useful to have an API provided in the sdk that works like the exec source, thus removing the requirement of having an entire agent on hosts where the exec source is used for tailing logs.

Maybe we can provide a standalone client that implements it too, which users can use off the shelf.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (FLUME-1339) Implement an exec source like client API

Posted by "Patrick Wendell (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/FLUME-1339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13408429#comment-13408429 ] 

Patrick Wendell commented on FLUME-1339:
----------------------------------------

Hey Hari - if someone just used an agent with a memory channel and an avro sink, wouldn't this enact fail-retry semantics? If an avro sink has a problem, it fails the txn and then attempts to reconnect, so this is a retry - right?

If the goal is to have partial durability and get better performance - one obvious extension would be to add some type of batching to the FileChannel, so that it doesn't sync at every txn, but there are sync points periodically. A disk seek takes 5 ms, so at a rate of 300 events per second you're looking at a lot of seek time if the disk is being used for other stuff than the flume logs. If you sync periodically, the data loss is bounded by the sync interval size.

Anyways a clear understanding of the performance bottleneck we are trying to solve would help navigate towards the correct solution. I guess maybe we can defer to Juhandi for that.
                
> Implement an exec source like client API
> ----------------------------------------
>
>                 Key: FLUME-1339
>                 URL: https://issues.apache.org/jira/browse/FLUME-1339
>             Project: Flume
>          Issue Type: Bug
>            Reporter: Hari Shreedharan
>            Assignee: Patrick Wendell
>
> Originally when the exec source was implemented there was no SDK. It would be useful to have an API provided in the sdk that works like the exec source, thus removing the requirement of having an entire agent on hosts where the exec source is used for tailing logs.
> Maybe we can provide a standalone client that implements it too, which users can use off the shelf.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Assigned] (FLUME-1339) Implement an exec source like client API

Posted by "Patrick Wendell (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/FLUME-1339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Patrick Wendell reassigned FLUME-1339:
--------------------------------------

    Assignee: Patrick Wendell
    
> Implement an exec source like client API
> ----------------------------------------
>
>                 Key: FLUME-1339
>                 URL: https://issues.apache.org/jira/browse/FLUME-1339
>             Project: Flume
>          Issue Type: Bug
>            Reporter: Hari Shreedharan
>            Assignee: Patrick Wendell
>
> Originally when the exec source was implemented there was no SDK. It would be useful to have an API provided in the sdk that works like the exec source, thus removing the requirement of having an entire agent on hosts where the exec source is used for tailing logs.
> Maybe we can provide a standalone client that implements it too, which users can use off the shelf.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (FLUME-1339) Implement an exec source like client API

Posted by "Juhani Connolly (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/FLUME-1339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13406806#comment-13406806 ] 

Juhani Connolly commented on FLUME-1339:
----------------------------------------

It's a good idea, but let's not drop the exec source if we do implement it. One reason for setting up a full agent is to survive network issues(if the next hop becomes temporarily unavailable things just stay in the file channel or whatever)
                
> Implement an exec source like client API
> ----------------------------------------
>
>                 Key: FLUME-1339
>                 URL: https://issues.apache.org/jira/browse/FLUME-1339
>             Project: Flume
>          Issue Type: Bug
>            Reporter: Hari Shreedharan
>
> Originally when the exec source was implemented there was no SDK. It would be useful to have an API provided in the sdk that works like the exec source, thus removing the requirement of having an entire agent on hosts where the exec source is used for tailing logs.
> Maybe we can provide a standalone client that implements it too, which users can use off the shelf.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (FLUME-1339) Implement an exec source like client API

Posted by "Hari Shreedharan (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/FLUME-1339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13408284#comment-13408284 ] 

Hari Shreedharan commented on FLUME-1339:
-----------------------------------------

Patrick:

Thanks for looking into this.
If strong durability is a requirement for something that tails the logs, then you are right. It would simply be better to just add a higher performance channel, and still use an agent locally. 

Also note that all our standalone clients/API just try to append to a flume agent and throw if the append fails. 

I am not sure of the use-cases, but users are likely to expect some amount of durability or fail-retry semantics. 

                
> Implement an exec source like client API
> ----------------------------------------
>
>                 Key: FLUME-1339
>                 URL: https://issues.apache.org/jira/browse/FLUME-1339
>             Project: Flume
>          Issue Type: Bug
>            Reporter: Hari Shreedharan
>            Assignee: Patrick Wendell
>
> Originally when the exec source was implemented there was no SDK. It would be useful to have an API provided in the sdk that works like the exec source, thus removing the requirement of having an entire agent on hosts where the exec source is used for tailing logs.
> Maybe we can provide a standalone client that implements it too, which users can use off the shelf.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (FLUME-1339) Implement an exec source like client API

Posted by "Hari Shreedharan (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/FLUME-1339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13406857#comment-13406857 ] 

Hari Shreedharan commented on FLUME-1339:
-----------------------------------------

I don't intend to remove the exec source. I just want to implement something like that as a client API.
                
> Implement an exec source like client API
> ----------------------------------------
>
>                 Key: FLUME-1339
>                 URL: https://issues.apache.org/jira/browse/FLUME-1339
>             Project: Flume
>          Issue Type: Bug
>            Reporter: Hari Shreedharan
>
> Originally when the exec source was implemented there was no SDK. It would be useful to have an API provided in the sdk that works like the exec source, thus removing the requirement of having an entire agent on hosts where the exec source is used for tailing logs.
> Maybe we can provide a standalone client that implements it too, which users can use off the shelf.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (FLUME-1339) Implement an exec source like client API

Posted by "Patrick Wendell (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/FLUME-1339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13408217#comment-13408217 ] 

Patrick Wendell commented on FLUME-1339:
----------------------------------------

I'm happy to take a look at this, but first I want to fully comprehend the problem being solved and the need for a standalone executable like this rather than just tuning an agent to provide the desired performance/durability trade-off.

Hari tells me that this JIRA originated from observed disk performance overhead of using file-based channel storage with an exec sink on production machines. The sink was simply tailing production logs - a very common use case.

If disk persistence is required here for durability, then I don't see how building this into a standalone client would make any difference, except that it would be a second implementation of disk-based durability to maintain in parallel with the existing agent code. I just want to be clear what overhead is being avoided here.

If I understand the problem correctly - an alternative solution would be to create a new channel which offers weaker durability semantics than the file storage, but stronger durability than the memory channel. In other words, find a middle point on this tradeoff curve but include it in Flume. Flume has pluggable channels exactly for this reason, right?
                
> Implement an exec source like client API
> ----------------------------------------
>
>                 Key: FLUME-1339
>                 URL: https://issues.apache.org/jira/browse/FLUME-1339
>             Project: Flume
>          Issue Type: Bug
>            Reporter: Hari Shreedharan
>            Assignee: Patrick Wendell
>
> Originally when the exec source was implemented there was no SDK. It would be useful to have an API provided in the sdk that works like the exec source, thus removing the requirement of having an entire agent on hosts where the exec source is used for tailing logs.
> Maybe we can provide a standalone client that implements it too, which users can use off the shelf.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira