You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Ia...@tdameritrade.com on 2016/09/26 18:38:58 UTC

HBase config files on multinode cluster

Hi,

I’ve been working on a storm topology. Our cluster is setup with storm bolts running on separate nodes from Hbase nodes. The configs are setup right now, so they match the services running on the node. Is this a best practice or should /etc/habse/conf/* exist on all nodes regardless?

The problem is that the storm bolt is looking for that file to pickup it’s zookeeper configs and others and they aren’t there. Configuration objects aren’t serializable, so we’re resorting to cherry picking properties from hbase-site.xml and serializing them off to the bolt.

Any advice would be appreciated.

Thanks,

Ian

Re: HBase config files on multinode cluster

Posted by Ia...@tdameritrade.com.
I ended up using a workaround. I turned the hbase-site.xml to a HashMap,
which can serialize to the bolts.


Thanks,

Ian





On 9/28/16, 4:51 PM, "saint.ack@gmail.com on behalf of Stack"
<saint.ack@gmail.com on behalf of stack@duboce.net> wrote:

>On Mon, Sep 26, 2016 at 11:38 AM, <Ia...@tdameritrade.com> wrote:
>
>> Hi,
>>
>> I¹ve been working on a storm topology. Our cluster is setup with storm
>> bolts running on separate nodes from Hbase nodes. The configs are setup
>> right now, so they match the services running on the node. Is this a
>>best
>> practice or should /etc/habse/conf/* exist on all nodes regardless?
>>
>>
>What is your deploy system like? Easy shipping config crafted by node
>character/role/profile?
>
>
>
>> The problem is that the storm bolt is looking for that file to pickup
>>it¹s
>> zookeeper configs and others and they aren¹t there. Configuration
>>objects
>> aren¹t serializable, so we¹re resorting to cherry picking properties
>>from
>> hbase-site.xml and serializing them off to the bolt.
>>
>>
>
>Serialize? (Sorry, not up on Storm config system -- a pointer would help).
>The bolt has an hbase client in it I suppose. How hard to add
>hbase-site.xml to the bolt CLASSPATH? How hard to bundle the config ... an
>hbase-site.xml ... into the bolt)?
>
>Thanks,
>St.Ack
>
>
>> Any advice would be appreciated.
>>
>> Thanks,
>>
>> Ian
>>

Re: HBase config files on multinode cluster

Posted by Stack <st...@duboce.net>.
On Mon, Sep 26, 2016 at 11:38 AM, <Ia...@tdameritrade.com> wrote:

> Hi,
>
> I’ve been working on a storm topology. Our cluster is setup with storm
> bolts running on separate nodes from Hbase nodes. The configs are setup
> right now, so they match the services running on the node. Is this a best
> practice or should /etc/habse/conf/* exist on all nodes regardless?
>
>
What is your deploy system like? Easy shipping config crafted by node
character/role/profile?



> The problem is that the storm bolt is looking for that file to pickup it’s
> zookeeper configs and others and they aren’t there. Configuration objects
> aren’t serializable, so we’re resorting to cherry picking properties from
> hbase-site.xml and serializing them off to the bolt.
>
>

Serialize? (Sorry, not up on Storm config system -- a pointer would help).
The bolt has an hbase client in it I suppose. How hard to add
hbase-site.xml to the bolt CLASSPATH? How hard to bundle the config ... an
hbase-site.xml ... into the bolt)?

Thanks,
St.Ack


> Any advice would be appreciated.
>
> Thanks,
>
> Ian
>