You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@chukwa.apache.org by GuoWei <we...@wbkit.com> on 2012/12/25 06:52:34 UTC
How to process Data when Server Connected Resused?
Dear all,
Recently, I write a my custom adaptor and map processor to store data to
Hbase. But when I shut down the Hbase, I found many many error logs in
Collector.log. My questions are :
1. When my custom adaptor can not connect to collector, do I need to store
data temporary myself ? Or chukwa can do this for me ?
2. When my custom demux map processor can not connect to Hbase, do I
need to store data temporary my self at collector side ? Or chukwa can do
this for me ?
Thanks in advance.
Best Regards / 商祺
郭伟 Guo Wei
-----------------------------------------------------
南京西桥科技有限公司
Western Bridge Tech Ltd., Nanjing
南京市玄武区花园路8号一号楼511
No. 511, Building 1, No. 8, Hua Yuan Road
Xuanwu District, Nanjing, PR China
Email: wei.guo@wbkit.com <ma...@wbkit.com>
Tel: +86 25 8528 4900 (Operator)
Mobile: +86 138 1589 8257
Fax: +86 25 8528 4980
Web: http://www.wbkit.com <http://www.wbkit.com/>
-----------------------------------------------------
WesternBridge Tech: Professional software service provider. Professional is
MANNER as well CAPABILITY.
Re: How to process Data when Server Connected Resused?
Posted by Eric Yang <er...@gmail.com>.
hi GuoWei,
The current offset of the adaptor is one way to track how much data are
sent. If your data is not stored in local disk of the agent, then data can
be lost when sending buffer is full. Collector buffers data in memory, it
does not store in collector disks by default. There is a local writer
which buffers data in disk and upload to hdfs. This is not turn on by
default. Hence, you can utilize local writer or implement your own writer
to achieve the same result.
See: http://incubator.apache.org/chukwa/docs/r0.5.0/collector.html for more
information about local writer.
We are planning to merge collector and agent together to provide ability to
spill to local disk, and reduce deployment complexity. This is in:
https://issues.apache.org/jira/browse/CHUKWA-674
If you are interested.
regards,
Eric
On Mon, Dec 24, 2012 at 9:52 PM, GuoWei <we...@wbkit.com> wrote:
> Dear all,
>
> Recently, I write a my custom adaptor and map processor to store data to
> Hbase. But when I shut down the Hbase, I found many many error logs in
> Collector.log. My questions are :
>
> 1. When my custom adaptor can not connect to collector, do I need to store
> data temporary myself ? Or chukwa can do this for me ?
> 2. When my custom demux map processor can not connect to Hbase, do I
> need to store data temporary my self at collector side ? Or chukwa can do
> this for me ?
>
> Thanks in advance.
>
> *Best Regards / **商祺**
> *郭伟 Guo Wei
> -----------------------------------------------------
> 南京西桥科技有限公司
> Western Bridge Tech Ltd., Nanjing
>
> 南京市玄武区花园路8号一号楼511
> No. 511, Building 1, No. 8, Hua Yuan Road
>
> Xuanwu District, Nanjing, PR China
>
> Email: wei.guo@wbkit.com
>
> Tel: +86 25 8528 4900 (Operator)
> Mobile: +86 138 1589 8257
>
> Fax: +86 25 8528 4980
>
> Web: http://www.wbkit.com
>
> ----------------------------------------------------- *******
> **WesternBridge Tech: Professional software service provider.
> Professional is MANNER as well CAPABILITY.*
>
> *
> *
>