You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@apex.apache.org by "Raja.Aravapalli" <Ra...@target.com> on 2016/06/10 22:07:31 UTC
Custom Kafka Offset Manager
Hi,
If I write my own implementation of OffsetManager for kafka input operator,
How should I configure kafka input operator to use my custom build offset manager ? I know about the property “OffsetManager”, But can someone please share a small example of how to set the offset manager ?
Thanks a lot.
Regards,
Raja.
Re: Custom Kafka Offset Manager
Posted by Munagala Ramanath <ra...@datatorrent.com>.
Raja,
Can you find the machine running YARN ?
Look for log files on that machine -- usually, they are under
*/var/log/hadoop-yarn* or */var/log/hadoop* or similar locations.
The files themselves will have names that vary depending on your
installation; some examples:
*yarn-<user>-resourcemanager-<host>.log*
*hadoop-cmf-yarn-RESOURCEMANAGER-<host>.log.out*
Look for lines similar to the following that reference your application id;
they will tell you what containers
were allocated on which nodes on behalf of this application and will show
if the containers were successfully
allocated or not.
Ram
-----------------------------------
2016-05-24 02:53:42,636 INFO
org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=<user>
OPERATION=AM Allocated Container TARGET=SchedulerApp
RESULT=SUCCESS APPID=application_1462948052533_0036
CONTAINERID=container_1462948052533_0036_01_022468
2016-05-24 02:53:42,636 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode:
Assigned container container_1462948052533_0036_01_022468 of capacity
<memory:1536, vCores:1> on host <host:port>, which has 9 containers,
<memory:24064, vCores:9> used and <memory:180736, vCores:15> available
after allocation
2016-05-24 02:53:42,636 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl:
container_1462948052533_0183_01_036933 Container Transitioned from
ALLOCATED to ACQUIRED
On Mon, Jun 13, 2016 at 8:46 AM, Raja.Aravapalli <Raja.Aravapalli@target.com
> wrote:
>
> I wrote a custom Hbase offset manager, and tried to set this for Kafka
> input operator as below!! When I trigger the application with this setting,
> application is staying long time in ACCEPTED state, and even it moves to
> RUNNING state… all the operators are staying INACTIVE without processing
> any records.
>
>
> HBaseOffsetManager hbaseOffMgr = new HBaseOffsetManager();
> kafkaInput.setOffsetManager(hbaseOffMgr);
>
>
> Any help will immensely help me. Thanks.
>
>
> Regards,
> Raja.
>
> From: "Raja.Aravapalli" <Ra...@target.com>
> Reply-To: "users@apex.apache.org" <us...@apex.apache.org>
> Date: Friday, June 10, 2016 at 5:07 PM
> To: "users@apex.apache.org" <us...@apex.apache.org>
> Subject: Custom Kafka Offset Manager
>
>
> Hi,
>
> If I write my own implementation of OffsetManager for kafka input
> operator,
>
> How should I configure kafka input operator to use my custom build offset
> manager ? I know about the property “OffsetManager”, But can someone please
> share a small example of how to set the offset manager ?
>
>
> Thanks a lot.
>
>
> Regards,
> Raja.
>
Re: Custom Kafka Offset Manager
Posted by "Raja.Aravapalli" <Ra...@target.com>.
I wrote a custom Hbase offset manager, and tried to set this for Kafka input operator as below!! When I trigger the application with this setting, application is staying long time in ACCEPTED state, and even it moves to RUNNING state… all the operators are staying INACTIVE without processing any records.
HBaseOffsetManager hbaseOffMgr = new HBaseOffsetManager();
kafkaInput.setOffsetManager(hbaseOffMgr);
Any help will immensely help me. Thanks.
Regards,
Raja.
From: "Raja.Aravapalli" <Ra...@target.com>>
Reply-To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Date: Friday, June 10, 2016 at 5:07 PM
To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Subject: Custom Kafka Offset Manager
Hi,
If I write my own implementation of OffsetManager for kafka input operator,
How should I configure kafka input operator to use my custom build offset manager ? I know about the property “OffsetManager”, But can someone please share a small example of how to set the offset manager ?
Thanks a lot.
Regards,
Raja.