You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "Dani Abel Rayan (JIRA)" <ji...@apache.org> on 2011/08/05 19:31:27 UTC

[jira] [Created] (FLUME-726) OpenTSDB Sink

OpenTSDB Sink
-------------

                 Key: FLUME-726
                 URL: https://issues.apache.org/jira/browse/FLUME-726
             Project: Flume
          Issue Type: New Feature
          Components: Sinks+Sources
            Reporter: Dani Abel Rayan


OpenTSDB is free software and is available under the LGPLv3+ license.  It allows you to collect many thousands of metrics from thousands of hosts and applications, at a high rate (every few seconds). and never deletes or downsample data and can easily store billions of data points. Under the hood it uses HBase (Asynchronous client) . Out of box, it gives you awesome graphs and hands-down options to tune your query.   http://opentsdb.net/    

OpenTSDB was collecting data using python scripts and Open-TSDB Flume plugin looked as natural alternative.


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Re: [jira] [Commented] (FLUME-726) OpenTSDB Sink

Posted by Jonathan Hsieh <jo...@cloudera.com>.
Thai,

I'd love to see some of the plugins people are working one.  If you make
them public feel free to add links to these here:

https://cwiki.apache.org/FLUME/flume-plugins.html

Thanks,
Jon.

On Thu, Sep 1, 2011 at 7:45 AM, Bao Thai Ngo <ba...@gmail.com> wrote:

> Hi,
>
> I have been working on this and can share if you are interested.
>
> ~Thai
>
> On Sat, Aug 6, 2011 at 5:51 AM, Huang, Zijian(Victor) <
> zijian.huang@etrade.com> wrote:
>
> > Hi,
> >  Is there anybody working on a MySqlSink? We are thinking of implementing
> > one. If no one is doing that right now, we might want to be able to
> > contribute to the project.
> >
> > Thanks
> >
> > Victor Huang
> >
> >
> > -----Original Message-----
> > From: Dani Abel Rayan (JIRA) [mailto:jira@apache.org]
> > Sent: Friday, August 05, 2011 3:30 PM
> > To: flume-dev@incubator.apache.org
> > Subject: [jira] [Commented] (FLUME-726) OpenTSDB Sink
> >
> >
> >    [
> >
> https://issues.apache.org/jira/browse/FLUME-726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13080269#comment-13080269
> ]
> >
> > Dani Abel Rayan commented on FLUME-726:
> > ---------------------------------------
> >
> > You can also see the maven changes in above link.
> >
> > > OpenTSDB Sink
> > > -------------
> > >
> > >                 Key: FLUME-726
> > >                 URL: https://issues.apache.org/jira/browse/FLUME-726
> > >             Project: Flume
> > >          Issue Type: New Feature
> > >          Components: Sinks+Sources
> > >            Reporter: Dani Abel Rayan
> > >            Assignee: Dani Abel Rayan
> > >
> > > OpenTSDB is free software and is available under the LGPLv3+ license.
>  It
> > allows you to collect many thousands of metrics from thousands of hosts
> and
> > applications, at a high rate (every few seconds). and never deletes or
> > downsample data and can easily store billions of data points. Under the
> hood
> > it uses HBase (Asynchronous client) . Out of box, it gives you awesome
> > graphs and hands-down options to tune your query.   http://opentsdb.net/
> > > OpenTSDB was collecting data using python scripts and Open-TSDB Flume
> > plugin looked as natural alternative.
> >
> > --
> > This message is automatically generated by JIRA.
> > For more information on JIRA, see:
> http://www.atlassian.com/software/jira
> >
> >
> >
>



-- 
// Jonathan Hsieh (shay)
// Software Engineer, Cloudera
// jon@cloudera.com

Re: [jira] [Commented] (FLUME-726) OpenTSDB Sink

Posted by Bao Thai Ngo <ba...@gmail.com>.
Hi,

I have been working on this and can share if you are interested.

~Thai

On Sat, Aug 6, 2011 at 5:51 AM, Huang, Zijian(Victor) <
zijian.huang@etrade.com> wrote:

> Hi,
>  Is there anybody working on a MySqlSink? We are thinking of implementing
> one. If no one is doing that right now, we might want to be able to
> contribute to the project.
>
> Thanks
>
> Victor Huang
>
>
> -----Original Message-----
> From: Dani Abel Rayan (JIRA) [mailto:jira@apache.org]
> Sent: Friday, August 05, 2011 3:30 PM
> To: flume-dev@incubator.apache.org
> Subject: [jira] [Commented] (FLUME-726) OpenTSDB Sink
>
>
>    [
> https://issues.apache.org/jira/browse/FLUME-726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13080269#comment-13080269]
>
> Dani Abel Rayan commented on FLUME-726:
> ---------------------------------------
>
> You can also see the maven changes in above link.
>
> > OpenTSDB Sink
> > -------------
> >
> >                 Key: FLUME-726
> >                 URL: https://issues.apache.org/jira/browse/FLUME-726
> >             Project: Flume
> >          Issue Type: New Feature
> >          Components: Sinks+Sources
> >            Reporter: Dani Abel Rayan
> >            Assignee: Dani Abel Rayan
> >
> > OpenTSDB is free software and is available under the LGPLv3+ license.  It
> allows you to collect many thousands of metrics from thousands of hosts and
> applications, at a high rate (every few seconds). and never deletes or
> downsample data and can easily store billions of data points. Under the hood
> it uses HBase (Asynchronous client) . Out of box, it gives you awesome
> graphs and hands-down options to tune your query.   http://opentsdb.net/
> > OpenTSDB was collecting data using python scripts and Open-TSDB Flume
> plugin looked as natural alternative.
>
> --
> This message is automatically generated by JIRA.
> For more information on JIRA, see: http://www.atlassian.com/software/jira
>
>
>

RE: [jira] [Commented] (FLUME-726) OpenTSDB Sink

Posted by "Huang, Zijian(Victor)" <zi...@etrade.com>.
Hi,
  Is there anybody working on a MySqlSink? We are thinking of implementing one. If no one is doing that right now, we might want to be able to contribute to the project.

Thanks

Victor Huang


-----Original Message-----
From: Dani Abel Rayan (JIRA) [mailto:jira@apache.org]
Sent: Friday, August 05, 2011 3:30 PM
To: flume-dev@incubator.apache.org
Subject: [jira] [Commented] (FLUME-726) OpenTSDB Sink


    [ https://issues.apache.org/jira/browse/FLUME-726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13080269#comment-13080269 ]

Dani Abel Rayan commented on FLUME-726:
---------------------------------------

You can also see the maven changes in above link.

> OpenTSDB Sink
> -------------
>
>                 Key: FLUME-726
>                 URL: https://issues.apache.org/jira/browse/FLUME-726
>             Project: Flume
>          Issue Type: New Feature
>          Components: Sinks+Sources
>            Reporter: Dani Abel Rayan
>            Assignee: Dani Abel Rayan
>
> OpenTSDB is free software and is available under the LGPLv3+ license.  It allows you to collect many thousands of metrics from thousands of hosts and applications, at a high rate (every few seconds). and never deletes or downsample data and can easily store billions of data points. Under the hood it uses HBase (Asynchronous client) . Out of box, it gives you awesome graphs and hands-down options to tune your query.   http://opentsdb.net/
> OpenTSDB was collecting data using python scripts and Open-TSDB Flume plugin looked as natural alternative.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira



[jira] [Commented] (FLUME-726) OpenTSDB Sink

Posted by "Tom White (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/FLUME-726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13080293#comment-13080293 ] 

Tom White commented on FLUME-726:
---------------------------------

LGPL-licensed dependencies are not allowed in Apache projects (see http://www.apache.org/legal/resolved.html#category-x) , so this can't be added to Flume, I'm afraid.

> OpenTSDB Sink
> -------------
>
>                 Key: FLUME-726
>                 URL: https://issues.apache.org/jira/browse/FLUME-726
>             Project: Flume
>          Issue Type: New Feature
>          Components: Sinks+Sources
>            Reporter: Dani Abel Rayan
>            Assignee: Dani Abel Rayan
>
> OpenTSDB is free software and is available under the LGPLv3+ license.  It allows you to collect many thousands of metrics from thousands of hosts and applications, at a high rate (every few seconds). and never deletes or downsample data and can easily store billions of data points. Under the hood it uses HBase (Asynchronous client) . Out of box, it gives you awesome graphs and hands-down options to tune your query.   http://opentsdb.net/    
> OpenTSDB was collecting data using python scripts and Open-TSDB Flume plugin looked as natural alternative.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (FLUME-726) OpenTSDB Sink

Posted by "Dani Abel Rayan (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/FLUME-726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13094230#comment-13094230 ] 

Dani Abel Rayan commented on FLUME-726:
---------------------------------------

A related discussion: Async HBase Client https://issues.apache.org/jira/browse/HBASE-2170?focusedCommentId=13007027&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13007027

> OpenTSDB Sink
> -------------
>
>                 Key: FLUME-726
>                 URL: https://issues.apache.org/jira/browse/FLUME-726
>             Project: Flume
>          Issue Type: New Feature
>          Components: Sinks+Sources
>            Reporter: Dani Abel Rayan
>            Assignee: Dani Abel Rayan
>
> OpenTSDB is free software and is available under the LGPLv3+ license.  It allows you to collect many thousands of metrics from thousands of hosts and applications, at a high rate (every few seconds). and never deletes or downsample data and can easily store billions of data points. Under the hood it uses HBase (Asynchronous client) . Out of box, it gives you awesome graphs and hands-down options to tune your query.   http://opentsdb.net/    
> OpenTSDB was collecting data using python scripts and Open-TSDB Flume plugin looked as natural alternative.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Assigned] (FLUME-726) OpenTSDB Sink

Posted by "Dani Abel Rayan (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/FLUME-726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dani Abel Rayan reassigned FLUME-726:
-------------------------------------

    Assignee: Dani Abel Rayan

> OpenTSDB Sink
> -------------
>
>                 Key: FLUME-726
>                 URL: https://issues.apache.org/jira/browse/FLUME-726
>             Project: Flume
>          Issue Type: New Feature
>          Components: Sinks+Sources
>            Reporter: Dani Abel Rayan
>            Assignee: Dani Abel Rayan
>
> OpenTSDB is free software and is available under the LGPLv3+ license.  It allows you to collect many thousands of metrics from thousands of hosts and applications, at a high rate (every few seconds). and never deletes or downsample data and can easily store billions of data points. Under the hood it uses HBase (Asynchronous client) . Out of box, it gives you awesome graphs and hands-down options to tune your query.   http://opentsdb.net/    
> OpenTSDB was collecting data using python scripts and Open-TSDB Flume plugin looked as natural alternative.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (FLUME-726) OpenTSDB Sink

Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/FLUME-726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13080261#comment-13080261 ] 

Jonathan Hsieh commented on FLUME-726:
--------------------------------------

@Dani,

This looks really cool!  

A few questions:
Is the goal here to fold this into a contrib/plugin dir?  
If so would committing this require the inclusion the LPGLv3 code/libs or can we use ivy/maven?

> OpenTSDB Sink
> -------------
>
>                 Key: FLUME-726
>                 URL: https://issues.apache.org/jira/browse/FLUME-726
>             Project: Flume
>          Issue Type: New Feature
>          Components: Sinks+Sources
>            Reporter: Dani Abel Rayan
>            Assignee: Dani Abel Rayan
>
> OpenTSDB is free software and is available under the LGPLv3+ license.  It allows you to collect many thousands of metrics from thousands of hosts and applications, at a high rate (every few seconds). and never deletes or downsample data and can easily store billions of data points. Under the hood it uses HBase (Asynchronous client) . Out of box, it gives you awesome graphs and hands-down options to tune your query.   http://opentsdb.net/    
> OpenTSDB was collecting data using python scripts and Open-TSDB Flume plugin looked as natural alternative.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (FLUME-726) OpenTSDB Sink

Posted by "Dani Abel Rayan (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/FLUME-726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13080268#comment-13080268 ] 

Dani Abel Rayan commented on FLUME-726:
---------------------------------------

@Jon, Yes! Under the hood, it is just a HBase (Async) client. 

I have put it under hbase plugn dir as separate plugin:
https://github.com/LeagueOfLegends/flume/commit/977c236effed7e092b52c9ec3fd815407f1b15dc

> OpenTSDB Sink
> -------------
>
>                 Key: FLUME-726
>                 URL: https://issues.apache.org/jira/browse/FLUME-726
>             Project: Flume
>          Issue Type: New Feature
>          Components: Sinks+Sources
>            Reporter: Dani Abel Rayan
>            Assignee: Dani Abel Rayan
>
> OpenTSDB is free software and is available under the LGPLv3+ license.  It allows you to collect many thousands of metrics from thousands of hosts and applications, at a high rate (every few seconds). and never deletes or downsample data and can easily store billions of data points. Under the hood it uses HBase (Asynchronous client) . Out of box, it gives you awesome graphs and hands-down options to tune your query.   http://opentsdb.net/    
> OpenTSDB was collecting data using python scripts and Open-TSDB Flume plugin looked as natural alternative.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (FLUME-726) OpenTSDB Sink

Posted by "Dani Abel Rayan (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/FLUME-726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13080269#comment-13080269 ] 

Dani Abel Rayan commented on FLUME-726:
---------------------------------------

You can also see the maven changes in above link.

> OpenTSDB Sink
> -------------
>
>                 Key: FLUME-726
>                 URL: https://issues.apache.org/jira/browse/FLUME-726
>             Project: Flume
>          Issue Type: New Feature
>          Components: Sinks+Sources
>            Reporter: Dani Abel Rayan
>            Assignee: Dani Abel Rayan
>
> OpenTSDB is free software and is available under the LGPLv3+ license.  It allows you to collect many thousands of metrics from thousands of hosts and applications, at a high rate (every few seconds). and never deletes or downsample data and can easily store billions of data points. Under the hood it uses HBase (Asynchronous client) . Out of box, it gives you awesome graphs and hands-down options to tune your query.   http://opentsdb.net/    
> OpenTSDB was collecting data using python scripts and Open-TSDB Flume plugin looked as natural alternative.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Issue Comment Edited] (FLUME-726) OpenTSDB Sink

Posted by "Dani Abel Rayan (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/FLUME-726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13094230#comment-13094230 ] 

Dani Abel Rayan edited comment on FLUME-726 at 8/31/11 1:18 AM:
----------------------------------------------------------------

A related discussion: Async HBase Client https://issues.apache.org/jira/browse/HBASE-2170?focusedCommentId=13007026&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13007026

      was (Author: danirayan):
    A related discussion: Async HBase Client https://issues.apache.org/jira/browse/HBASE-2170?focusedCommentId=13007027&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13007027
  
> OpenTSDB Sink
> -------------
>
>                 Key: FLUME-726
>                 URL: https://issues.apache.org/jira/browse/FLUME-726
>             Project: Flume
>          Issue Type: New Feature
>          Components: Sinks+Sources
>            Reporter: Dani Abel Rayan
>            Assignee: Dani Abel Rayan
>
> OpenTSDB is free software and is available under the LGPLv3+ license.  It allows you to collect many thousands of metrics from thousands of hosts and applications, at a high rate (every few seconds). and never deletes or downsample data and can easily store billions of data points. Under the hood it uses HBase (Asynchronous client) . Out of box, it gives you awesome graphs and hands-down options to tune your query.   http://opentsdb.net/    
> OpenTSDB was collecting data using python scripts and Open-TSDB Flume plugin looked as natural alternative.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira