You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by terreyshih <te...@gmail.com> on 2014/08/25 04:28:34 UTC
starting a thread in the customized sink ?
Hi,
If in my customized sink, I create some threads and schedule it to do some queueing. For example, I would like to drop the events off every 10 seconds. If I start the thread or an executor in the process method of the customized sink class, would have have some negative impact to the threading model in Flume ?
thanks,
-Terrey
Re: starting a thread in the customized sink ?
Posted by terreyshih <te...@gmail.com>.
thanks
On Aug 24, 2014, at 10:20 PM, Hari Shreedharan <hs...@cloudera.com> wrote:
> You are probably ok. There are a bunch of sinks that already do this (Asynchbase sink/Morphline Solr sink etc).
>
>
> On Sun, Aug 24, 2014 at 10:17 PM, terreyshih <te...@gmail.com> wrote:
> Thanks for answering my question.
>
> No, the transaction will not be shared by different threads. The transaction is committed after I convert the events to some data structure.
>
> I would only have a Timer/Executor thread that fires off at specific intervals to send these converted data structures (from events) to customized storage/consumer for processing.
>
> Do you see an issue with this approach ?
>
> thanks,
> Terrey.
>
>
>
> On Aug 24, 2014, at 9:59 PM, Hari Shreedharan <hs...@cloudera.com> wrote:
>
>> If you create a large number of threads in a sink - that could affect the performance of the agent - simply because there are too many threads. Also, transactions are thread-local - so make sure each transaction is completely owned and run by the same thread.
>>
>>
>> On Sun, Aug 24, 2014 at 7:28 PM, terreyshih <te...@gmail.com> wrote:
>> Hi,
>>
>> If in my customized sink, I create some threads and schedule it to do some queueing. For example, I would like to drop the events off every 10 seconds. If I start the thread or an executor in the process method of the customized sink class, would have have some negative impact to the threading model in Flume ?
>>
>> thanks,
>> -Terrey
>>
>
>
Re: starting a thread in the customized sink ?
Posted by Hari Shreedharan <hs...@cloudera.com>.
You are probably ok. There are a bunch of sinks that already do this
(Asynchbase sink/Morphline Solr sink etc).
On Sun, Aug 24, 2014 at 10:17 PM, terreyshih <te...@gmail.com> wrote:
> Thanks for answering my question.
>
> No, the transaction will not be shared by different threads. The
> transaction is committed after I convert the events to some data structure.
>
> I would only have a Timer/Executor thread that fires off at specific
> intervals to send these converted data structures (from events) to
> customized storage/consumer for processing.
>
> Do you see an issue with this approach ?
>
> thanks,
> Terrey.
>
>
>
> On Aug 24, 2014, at 9:59 PM, Hari Shreedharan <hs...@cloudera.com>
> wrote:
>
> If you create a large number of threads in a sink - that could affect the
> performance of the agent - simply because there are too many threads. Also,
> transactions are thread-local - so make sure each transaction is completely
> owned and run by the same thread.
>
>
> On Sun, Aug 24, 2014 at 7:28 PM, terreyshih <te...@gmail.com> wrote:
>
>> Hi,
>>
>> If in my customized sink, I create some threads and schedule it to do
>> some queueing. For example, I would like to drop the events off every 10
>> seconds. If I start the thread or an executor in the process method of the
>> customized sink class, would have have some negative impact to the
>> threading model in Flume ?
>>
>> thanks,
>> -Terrey
>
>
>
>
Re: starting a thread in the customized sink ?
Posted by terreyshih <te...@gmail.com>.
Thanks for answering my question.
No, the transaction will not be shared by different threads. The transaction is committed after I convert the events to some data structure.
I would only have a Timer/Executor thread that fires off at specific intervals to send these converted data structures (from events) to customized storage/consumer for processing.
Do you see an issue with this approach ?
thanks,
Terrey.
On Aug 24, 2014, at 9:59 PM, Hari Shreedharan <hs...@cloudera.com> wrote:
> If you create a large number of threads in a sink - that could affect the performance of the agent - simply because there are too many threads. Also, transactions are thread-local - so make sure each transaction is completely owned and run by the same thread.
>
>
> On Sun, Aug 24, 2014 at 7:28 PM, terreyshih <te...@gmail.com> wrote:
> Hi,
>
> If in my customized sink, I create some threads and schedule it to do some queueing. For example, I would like to drop the events off every 10 seconds. If I start the thread or an executor in the process method of the customized sink class, would have have some negative impact to the threading model in Flume ?
>
> thanks,
> -Terrey
>
Re: starting a thread in the customized sink ?
Posted by Hari Shreedharan <hs...@cloudera.com>.
If you create a large number of threads in a sink - that could affect the
performance of the agent - simply because there are too many threads. Also,
transactions are thread-local - so make sure each transaction is completely
owned and run by the same thread.
On Sun, Aug 24, 2014 at 7:28 PM, terreyshih <te...@gmail.com> wrote:
> Hi,
>
> If in my customized sink, I create some threads and schedule it to do some
> queueing. For example, I would like to drop the events off every 10
> seconds. If I start the thread or an executor in the process method of the
> customized sink class, would have have some negative impact to the
> threading model in Flume ?
>
> thanks,
> -Terrey