Is there a way to instantiate one object per worker?

[Banias H <ba...@gmail.com>]

Hi,

Is there a way to instantiate one object and share that object among
executors on the same worker?

One of the bolts of my topology reads from HBase using HTable. I configure
that bolt to have a couple hundred executors. Currently every executor
instantiates HTable in prepare() method like:

Configuration config = HBaseConfiguration.create();
config.set("hbase.zookeeper.property.clientPort", "2181");
config.set("hbase.zookeeper.quorum", "host1,host2");
table = new HTable(config, "table");

The problem of this is every executor has its own Configuration and
therefore each HTable is having its own underlying HConnection instance.
But it would be nice to share the underlying HConnection within the same
host. This means I will need to instantiate ONE Configuration per host (i.e
per worker as I have configured one worker per host for my topology) and
share it among all executors within the same worker.

Is there a way I can do this programatically?

Thanks,
BH

Re: Is there a way to instantiate one object per worker?

[John Reilly <jr...@inconspicuous.org>]

Move that code into a singleton returning the HTable and get the singleton
in your prepare method.

e.g.
class SingletonInjector {
  private static XYZ injector = null

  public synchronized XYZ getInjector(config: Config) {
    if(injector != null) {
      return injector
    } else {
      injector = new XYZ(config)
      return injector
    }
  }
}


On Tue, Jun 30, 2015 at 3:07 PM Banias H <ba...@gmail.com> wrote:

> Hi,
>
> Is there a way to instantiate one object and share that object among
> executors on the same worker?
>
> One of the bolts of my topology reads from HBase using HTable. I configure
> that bolt to have a couple hundred executors. Currently every executor
> instantiates HTable in prepare() method like:
>
> Configuration config = HBaseConfiguration.create();
> config.set("hbase.zookeeper.property.clientPort", "2181");
> config.set("hbase.zookeeper.quorum", "host1,host2");
> table = new HTable(config, "table");
>
> The problem of this is every executor has its own Configuration and
> therefore each HTable is having its own underlying HConnection instance.
> But it would be nice to share the underlying HConnection within the same
> host. This means I will need to instantiate ONE Configuration per host (i.e
> per worker as I have configured one worker per host for my topology) and
> share it among all executors within the same worker.
>
> Is there a way I can do this programatically?
>
> Thanks,
> BH
>