You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@metron.apache.org by Otto Fowler <ot...@gmail.com> on 2019/01/24 15:26:16 UTC

Re: How to provide hbase-site.xml to Stellar Processor Java API

Hi Anil,
Can you create a jira on this with these details and a general overview of
your use case?
It looks like the HbaseConfiguration we use in the HTableConnector
is done using the create() method, which creates from resources.

I think we would need to do some work to support the external file.

On January 24, 2019 at 10:14:46, Anil Donthireddy (
anil.donthireddy@sstech.us) wrote:

Hi,

I have written a java application which uses Stellar processor and execute
the stellar expressions. The issue I am facing is I am unable to connect
Hbase unless I place hbase-site.xml in src/main/resources/ folder of the
code. As it is not the proper way of packaging the hbase-site.xml with Jar,
I would like to understand how the hbase-site.xml is being set to classpath
while starting profiler topology.

The ways I tried are

1) Setting the classpath to hbase conf folder using command “java -cp
$CLASSPATH:/etc/hbase/conf:/etc/Hadoop/conf –jar myJar.jar”

2) Adding Hbase conf folder to HADOOP_CLASSPATH. Below is the Hadoop
classpath

/usr/hdp/2.6.1.0-129/hadoop/conf:/usr/hdp/2.6.1.0-129/hadoop/lib/*:/usr/hdp/2.6.1.0-129/hadoop/.//*:/usr/hdp/2.6.1.0-129/hadoop-hdfs/./:/usr/hdp/2.6.1.0-129/hadoop-hdfs/lib/*:/usr/hdp/2.6.1.0-129/hadoop-hdfs/.//*:/usr/hdp/2.6.1.0-129/hadoop-yarn/lib/*:/usr/hdp/2.6.1.0-129/hadoop-yarn/.//*:/usr/hdp/2.6.1.0-129/hadoop-mapreduce/lib/*:/usr/hdp/2.6.1.0-129/hadoop-mapreduce/.//*::mysql-connector-java.jar:postgresql-jdbc2ee.jar:postgresql-jdbc2.jar:postgresql-jdbc3.jar:postgresql-jdbc.jar:/etc/hbase/conf/:/usr/hdp/2.6.1.0-129/tez/*:/usr/hdp/2.6.1.0-129/tez/lib/*:/usr/hdp/2.6.1.0-129/tez/conf

One more step I would like to try to fix the issue is to set property “
*zookeeper.znode.parent*” for configuration object while instantiating
HbaseConnector. But it is in the scope of metron code to try this fix.

I would like to know if any one able to provide hbase-site.xml to any Java
Application or anyone able to extend Metron Stellar Processor and execute
profile definitions successfully.

Please provide any inputs to resolve the issue.

Thanking you.

Thanks,

Anil.

RE: How to provide hbase-site.xml to Stellar Processor Java API

Posted by Anil Donthireddy <An...@sstech.us>.

Hi All,

FYI

I am able to resolve the issue for my springboot application.

I added a plugin as below to my pom.xml file under <build>/<plugins>/ and passing files to be loaded while starting the application as “-Dloader.path=/etc/hbase/conf/,/etc/hadoop/conf/”

<plugin>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-maven-plugin</artifactId>
    <configuration>
        <layout>ZIP</layout><!-- enables PropertiesLauncher -->
    </configuration>
</plugin>


Thanks,
Anil.

From: Anil Donthireddy
Sent: Friday, January 25, 2019 5:32 PM
To: 'Otto Fowler' <ot...@gmail.com>; user@metron.apache.org; dev@metron.apache.org; Mohan Venkateshaiah <mv...@hortonworks.com>
Cc: Maxim Dashenko <Ma...@sstech.us>; Satish Abburi <Sa...@sstech.us>; James Sirota <js...@hortonworks.com>; Christopher Berry <Ch...@sstech.us>
Subject: RE: How to provide hbase-site.xml to Stellar Processor Java API

Hi Mohan DV,

I appreciate your time for sharing the much useful information.

As per my understanding there seems to be no scope to pass my own build Hbase configuration object to execute Stellar queries in my extended API. I may need to re-write lot of things in my extended API in the way Stellar processor works to override the Hbase configuration.

@Otto Fowler: I will raise a Jira case with the usecase and the issue.

Thanks,
Anil.

From: Otto Fowler [mailto:ottobackwards@gmail.com]
Sent: Thursday, January 24, 2019 11:15 PM
To: Anil Donthireddy <An...@sstech.us>>; user@metron.apache.org<ma...@metron.apache.org>; dev@metron.apache.org<ma...@metron.apache.org>; Mohan Venkateshaiah <mv...@hortonworks.com>>
Cc: Maxim Dashenko <Ma...@sstech.us>>; Satish Abburi <Sa...@sstech.us>>; James Sirota <js...@hortonworks.com>>; Christopher Berry <Ch...@sstech.us>>
Subject: Re: How to provide hbase-site.xml to Stellar Processor Java API

But still file a jira ;)



On January 24, 2019 at 12:19:34, Mohan Venkateshaiah (mvenkateshaiah@hortonworks.com<ma...@hortonworks.com>) wrote:
Hi Anil,

I had done similar to this in the past . In the stellar to get the hbase configuration we call HBaseConfiguration.create() , in that call hbase adds the hbase-site and core-site as resources to the config we probably SHOULD let people specify a base config.

What I had done was in the global config, I set a property called hbase.provider.impl, it's the fully qualified class name for a class that implements the TableProvider interface which has one method:

public HTableInterface getTable(Configuration config, String tableName) throws IOException

if you implement your own where you ignore the config argument and resolve the hbase table with your own injected config that will work

Thanks
Mohan DV

On 1/24/19, 8:56 PM, "Otto Fowler" <ot...@gmail.com>> wrote:

Hi Anil,
Can you create a jira on this with these details and a general overview of
your use case?
It looks like the HbaseConfiguration we use in the HTableConnector
is done using the create() method, which creates from resources.

I think we would need to do some work to support the external file.



On January 24, 2019 at 10:14:46, Anil Donthireddy (
anil.donthireddy@sstech.us<ma...@sstech.us>) wrote:

Hi,



I have written a java application which uses Stellar processor and execute
the stellar expressions. The issue I am facing is I am unable to connect
Hbase unless I place hbase-site.xml in src/main/resources/ folder of the
code. As it is not the proper way of packaging the hbase-site.xml with Jar,
I would like to understand how the hbase-site.xml is being set to classpath
while starting profiler topology.



The ways I tried are

1) Setting the classpath to hbase conf folder using command “java -cp
$CLASSPATH:/etc/hbase/conf:/etc/Hadoop/conf –jar myJar.jar”

2) Adding Hbase conf folder to HADOOP_CLASSPATH. Below is the Hadoop
classpath

/usr/hdp/2.6.1.0-129/hadoop/conf:/usr/hdp/2.6.1.0-129/hadoop/lib/*:/usr/hdp/2.6.1.0-129/hadoop/.//*:/usr/hdp/2.6.1.0-129/hadoop-hdfs/./:/usr/hdp/2.6.1.0-129/hadoop-hdfs/lib/*:/usr/hdp/2.6.1.0-129/hadoop-hdfs/.//*:/usr/hdp/2.6.1.0-129/hadoop-yarn/lib/*:/usr/hdp/2.6.1.0-129/hadoop-yarn/.//*:/usr/hdp/2.6.1.0-129/hadoop-mapreduce/lib/*:/usr/hdp/2.6.1.0-129/hadoop-mapreduce/.//*::mysql-connector-java.jar:postgresql-jdbc2ee.jar:postgresql-jdbc2.jar:postgresql-jdbc3.jar:postgresql-jdbc.jar:/etc/hbase/conf/:/usr/hdp/2.6.1.0-129/tez/*:/usr/hdp/2.6.1.0-129/tez/lib/*:/usr/hdp/2.6.1.0-129/tez/conf



One more step I would like to try to fix the issue is to set property “
*zookeeper.znode.parent*” for configuration object while instantiating
HbaseConnector. But it is in the scope of metron code to try this fix.



I would like to know if any one able to provide hbase-site.xml to any Java
Application or anyone able to extend Metron Stellar Processor and execute
profile definitions successfully.

Please provide any inputs to resolve the issue.



Thanking you.



Thanks,

Anil.

RE: How to provide hbase-site.xml to Stellar Processor Java API

Posted by Otto Fowler <ot...@gmail.com>.

Please link it to https://issues.apache.org/jira/browse/METRON-1409.
This is a jira around hosting stellar that I created a while ago.

On January 25, 2019 at 07:02:30, Anil Donthireddy (
anil.donthireddy@sstech.us) wrote:

Hi Mohan DV,

I appreciate your time for sharing the much useful information.

As per my understanding there seems to be no scope to pass my own build
Hbase configuration object to execute Stellar queries in my extended API. I
may need to re-write lot of things in my extended API in the way Stellar
processor works to override the Hbase configuration.

@Otto Fowler: I will raise a Jira case with the usecase and the issue.

Thanks,

Anil.

*From:* Otto Fowler [mailto:ottobackwards@gmail.com]
*Sent:* Thursday, January 24, 2019 11:15 PM
*To:* Anil Donthireddy <An...@sstech.us>; user@metron.apache.org;
dev@metron.apache.org; Mohan Venkateshaiah <mv...@hortonworks.com>
*Cc:* Maxim Dashenko <Ma...@sstech.us>; Satish Abburi <
Satish.Abburi@sstech.us>; James Sirota <js...@hortonworks.com>;
Christopher Berry <Ch...@sstech.us>
*Subject:* Re: How to provide hbase-site.xml to Stellar Processor Java API

But still file a jira ;)

On January 24, 2019 at 12:19:34, Mohan Venkateshaiah (
mvenkateshaiah@hortonworks.com) wrote:

Hi Anil,

I had done similar to this in the past . In the stellar to get the hbase
configuration we call HBaseConfiguration.create() , in that call hbase adds
the hbase-site and core-site as resources to the config we probably SHOULD
let people specify a base config.

What I had done was in the global config, I set a property called
hbase.provider.impl, it's the fully qualified class name for a class that
implements the TableProvider interface which has one method:

public HTableInterface getTable(Configuration config, String tableName)
throws IOException

if you implement your own where you ignore the config argument and resolve
the hbase table with your own injected config that will work

Thanks
Mohan DV

On 1/24/19, 8:56 PM, "Otto Fowler" <ot...@gmail.com> wrote:

I think we would need to do some work to support the external file.

On January 24, 2019 at 10:14:46, Anil Donthireddy (
anil.donthireddy@sstech.us) wrote:

Hi,

The ways I tried are

1) Setting the classpath to hbase conf folder using command “java -cp
$CLASSPATH:/etc/hbase/conf:/etc/Hadoop/conf –jar myJar.jar”

2) Adding Hbase conf folder to HADOOP_CLASSPATH. Below is the Hadoop
classpath

I would like to know if any one able to provide hbase-site.xml to any Java
Application or anyone able to extend Metron Stellar Processor and execute
profile definitions successfully.

Please provide any inputs to resolve the issue.

Thanking you.

Thanks,

Anil.

RE: How to provide hbase-site.xml to Stellar Processor Java API

Posted by Otto Fowler <ot...@gmail.com>.

Please link it to https://issues.apache.org/jira/browse/METRON-1409.
This is a jira around hosting stellar that I created a while ago.

On January 25, 2019 at 07:02:30, Anil Donthireddy (
anil.donthireddy@sstech.us) wrote:

Hi Mohan DV,

I appreciate your time for sharing the much useful information.

@Otto Fowler: I will raise a Jira case with the usecase and the issue.

Thanks,

Anil.

But still file a jira ;)

On January 24, 2019 at 12:19:34, Mohan Venkateshaiah (
mvenkateshaiah@hortonworks.com) wrote:

Hi Anil,

public HTableInterface getTable(Configuration config, String tableName)
throws IOException

if you implement your own where you ignore the config argument and resolve
the hbase table with your own injected config that will work

Thanks
Mohan DV

On 1/24/19, 8:56 PM, "Otto Fowler" <ot...@gmail.com> wrote:

I think we would need to do some work to support the external file.

On January 24, 2019 at 10:14:46, Anil Donthireddy (
anil.donthireddy@sstech.us) wrote:

Hi,

The ways I tried are

1) Setting the classpath to hbase conf folder using command “java -cp
$CLASSPATH:/etc/hbase/conf:/etc/Hadoop/conf –jar myJar.jar”

2) Adding Hbase conf folder to HADOOP_CLASSPATH. Below is the Hadoop
classpath

I would like to know if any one able to provide hbase-site.xml to any Java
Application or anyone able to extend Metron Stellar Processor and execute
profile definitions successfully.

Please provide any inputs to resolve the issue.

Thanking you.

Thanks,

Anil.

RE: How to provide hbase-site.xml to Stellar Processor Java API

Posted by Anil Donthireddy <An...@sstech.us>.

Hi Mohan DV,

I appreciate your time for sharing the much useful information.

As per my understanding there seems to be no scope to pass my own build Hbase configuration object to execute Stellar queries in my extended API. I may need to re-write lot of things in my extended API in the way Stellar processor works to override the Hbase configuration.

@Otto Fowler: I will raise a Jira case with the usecase and the issue.

Thanks,
Anil.

From: Otto Fowler [mailto:ottobackwards@gmail.com]
Sent: Thursday, January 24, 2019 11:15 PM
To: Anil Donthireddy <An...@sstech.us>; user@metron.apache.org; dev@metron.apache.org; Mohan Venkateshaiah <mv...@hortonworks.com>
Cc: Maxim Dashenko <Ma...@sstech.us>; Satish Abburi <Sa...@sstech.us>; James Sirota <js...@hortonworks.com>; Christopher Berry <Ch...@sstech.us>
Subject: Re: How to provide hbase-site.xml to Stellar Processor Java API

But still file a jira ;)



On January 24, 2019 at 12:19:34, Mohan Venkateshaiah (mvenkateshaiah@hortonworks.com<ma...@hortonworks.com>) wrote:
Hi Anil,

I had done similar to this in the past . In the stellar to get the hbase configuration we call HBaseConfiguration.create() , in that call hbase adds the hbase-site and core-site as resources to the config we probably SHOULD let people specify a base config.

What I had done was in the global config, I set a property called hbase.provider.impl, it's the fully qualified class name for a class that implements the TableProvider interface which has one method:

public HTableInterface getTable(Configuration config, String tableName) throws IOException

if you implement your own where you ignore the config argument and resolve the hbase table with your own injected config that will work

Thanks
Mohan DV

On 1/24/19, 8:56 PM, "Otto Fowler" <ot...@gmail.com>> wrote:

Hi Anil,
Can you create a jira on this with these details and a general overview of
your use case?
It looks like the HbaseConfiguration we use in the HTableConnector
is done using the create() method, which creates from resources.

I think we would need to do some work to support the external file.



On January 24, 2019 at 10:14:46, Anil Donthireddy (
anil.donthireddy@sstech.us<ma...@sstech.us>) wrote:

Hi,



I have written a java application which uses Stellar processor and execute
the stellar expressions. The issue I am facing is I am unable to connect
Hbase unless I place hbase-site.xml in src/main/resources/ folder of the
code. As it is not the proper way of packaging the hbase-site.xml with Jar,
I would like to understand how the hbase-site.xml is being set to classpath
while starting profiler topology.



The ways I tried are

1) Setting the classpath to hbase conf folder using command “java -cp
$CLASSPATH:/etc/hbase/conf:/etc/Hadoop/conf –jar myJar.jar”

2) Adding Hbase conf folder to HADOOP_CLASSPATH. Below is the Hadoop
classpath

/usr/hdp/2.6.1.0-129/hadoop/conf:/usr/hdp/2.6.1.0-129/hadoop/lib/*:/usr/hdp/2.6.1.0-129/hadoop/.//*:/usr/hdp/2.6.1.0-129/hadoop-hdfs/./:/usr/hdp/2.6.1.0-129/hadoop-hdfs/lib/*:/usr/hdp/2.6.1.0-129/hadoop-hdfs/.//*:/usr/hdp/2.6.1.0-129/hadoop-yarn/lib/*:/usr/hdp/2.6.1.0-129/hadoop-yarn/.//*:/usr/hdp/2.6.1.0-129/hadoop-mapreduce/lib/*:/usr/hdp/2.6.1.0-129/hadoop-mapreduce/.//*::mysql-connector-java.jar:postgresql-jdbc2ee.jar:postgresql-jdbc2.jar:postgresql-jdbc3.jar:postgresql-jdbc.jar:/etc/hbase/conf/:/usr/hdp/2.6.1.0-129/tez/*:/usr/hdp/2.6.1.0-129/tez/lib/*:/usr/hdp/2.6.1.0-129/tez/conf



One more step I would like to try to fix the issue is to set property “
*zookeeper.znode.parent*” for configuration object while instantiating
HbaseConnector. But it is in the scope of metron code to try this fix.



I would like to know if any one able to provide hbase-site.xml to any Java
Application or anyone able to extend Metron Stellar Processor and execute
profile definitions successfully.

Please provide any inputs to resolve the issue.



Thanking you.



Thanks,

Anil.

Re: How to provide hbase-site.xml to Stellar Processor Java API

Posted by Otto Fowler <ot...@gmail.com>.

But still file a jira ;)


On January 24, 2019 at 12:19:34, Mohan Venkateshaiah (
mvenkateshaiah@hortonworks.com) wrote:

Hi Anil,

I had done similar to this in the past . In the stellar to get the hbase
configuration we call HBaseConfiguration.create() , in that call hbase adds
the hbase-site and core-site as resources to the config we probably SHOULD
let people specify a base config.

What I had done was in the global config, I set a property called
hbase.provider.impl, it's the fully qualified class name for a class that
implements the TableProvider interface which has one method:

public HTableInterface getTable(Configuration config, String tableName)
throws IOException

if you implement your own where you ignore the config argument and resolve
the hbase table with your own injected config that will work

Thanks
Mohan DV

On 1/24/19, 8:56 PM, "Otto Fowler" <ot...@gmail.com> wrote:

Hi Anil,
Can you create a jira on this with these details and a general overview of
your use case?
It looks like the HbaseConfiguration we use in the HTableConnector
is done using the create() method, which creates from resources.

I think we would need to do some work to support the external file.



On January 24, 2019 at 10:14:46, Anil Donthireddy (
anil.donthireddy@sstech.us) wrote:

Hi,



I have written a java application which uses Stellar processor and execute
the stellar expressions. The issue I am facing is I am unable to connect
Hbase unless I place hbase-site.xml in src/main/resources/ folder of the
code. As it is not the proper way of packaging the hbase-site.xml with Jar,
I would like to understand how the hbase-site.xml is being set to classpath
while starting profiler topology.



The ways I tried are

1) Setting the classpath to hbase conf folder using command “java -cp
$CLASSPATH:/etc/hbase/conf:/etc/Hadoop/conf –jar myJar.jar”

2) Adding Hbase conf folder to HADOOP_CLASSPATH. Below is the Hadoop
classpath

/usr/hdp/2.6.1.0-129/hadoop/conf:/usr/hdp/2.6.1.0-129/hadoop/lib/*:/usr/hdp/2.6.1.0-129/hadoop/.//*:/usr/hdp/2.6.1.0-129/hadoop-hdfs/./:/usr/hdp/2.6.1.0-129/hadoop-hdfs/lib/*:/usr/hdp/2.6.1.0-129/hadoop-hdfs/.//*:/usr/hdp/2.6.1.0-129/hadoop-yarn/lib/*:/usr/hdp/2.6.1.0-129/hadoop-yarn/.//*:/usr/hdp/2.6.1.0-129/hadoop-mapreduce/lib/*:/usr/hdp/2.6.1.0-129/hadoop-mapreduce/.//*::mysql-connector-java.jar:postgresql-jdbc2ee.jar:postgresql-jdbc2.jar:postgresql-jdbc3.jar:postgresql-jdbc.jar:/etc/hbase/conf/:/usr/hdp/2.6.1.0-129/tez/*:/usr/hdp/2.6.1.0-129/tez/lib/*:/usr/hdp/2.6.1.0-129/tez/conf




One more step I would like to try to fix the issue is to set property “
*zookeeper.znode.parent*” for configuration object while instantiating
HbaseConnector. But it is in the scope of metron code to try this fix.



I would like to know if any one able to provide hbase-site.xml to any Java
Application or anyone able to extend Metron Stellar Processor and execute
profile definitions successfully.

Please provide any inputs to resolve the issue.



Thanking you.



Thanks,

Anil.

Re: How to provide hbase-site.xml to Stellar Processor Java API

Posted by Otto Fowler <ot...@gmail.com>.

But still file a jira ;)


On January 24, 2019 at 12:19:34, Mohan Venkateshaiah (
mvenkateshaiah@hortonworks.com) wrote:

Hi Anil,

I had done similar to this in the past . In the stellar to get the hbase
configuration we call HBaseConfiguration.create() , in that call hbase adds
the hbase-site and core-site as resources to the config we probably SHOULD
let people specify a base config.

What I had done was in the global config, I set a property called
hbase.provider.impl, it's the fully qualified class name for a class that
implements the TableProvider interface which has one method:

public HTableInterface getTable(Configuration config, String tableName)
throws IOException

if you implement your own where you ignore the config argument and resolve
the hbase table with your own injected config that will work

Thanks
Mohan DV

On 1/24/19, 8:56 PM, "Otto Fowler" <ot...@gmail.com> wrote:

Hi Anil,
Can you create a jira on this with these details and a general overview of
your use case?
It looks like the HbaseConfiguration we use in the HTableConnector
is done using the create() method, which creates from resources.

I think we would need to do some work to support the external file.



On January 24, 2019 at 10:14:46, Anil Donthireddy (
anil.donthireddy@sstech.us) wrote:

Hi,



I have written a java application which uses Stellar processor and execute
the stellar expressions. The issue I am facing is I am unable to connect
Hbase unless I place hbase-site.xml in src/main/resources/ folder of the
code. As it is not the proper way of packaging the hbase-site.xml with Jar,
I would like to understand how the hbase-site.xml is being set to classpath
while starting profiler topology.



The ways I tried are

1) Setting the classpath to hbase conf folder using command “java -cp
$CLASSPATH:/etc/hbase/conf:/etc/Hadoop/conf –jar myJar.jar”

2) Adding Hbase conf folder to HADOOP_CLASSPATH. Below is the Hadoop
classpath

/usr/hdp/2.6.1.0-129/hadoop/conf:/usr/hdp/2.6.1.0-129/hadoop/lib/*:/usr/hdp/2.6.1.0-129/hadoop/.//*:/usr/hdp/2.6.1.0-129/hadoop-hdfs/./:/usr/hdp/2.6.1.0-129/hadoop-hdfs/lib/*:/usr/hdp/2.6.1.0-129/hadoop-hdfs/.//*:/usr/hdp/2.6.1.0-129/hadoop-yarn/lib/*:/usr/hdp/2.6.1.0-129/hadoop-yarn/.//*:/usr/hdp/2.6.1.0-129/hadoop-mapreduce/lib/*:/usr/hdp/2.6.1.0-129/hadoop-mapreduce/.//*::mysql-connector-java.jar:postgresql-jdbc2ee.jar:postgresql-jdbc2.jar:postgresql-jdbc3.jar:postgresql-jdbc.jar:/etc/hbase/conf/:/usr/hdp/2.6.1.0-129/tez/*:/usr/hdp/2.6.1.0-129/tez/lib/*:/usr/hdp/2.6.1.0-129/tez/conf




One more step I would like to try to fix the issue is to set property “
*zookeeper.znode.parent*” for configuration object while instantiating
HbaseConnector. But it is in the scope of metron code to try this fix.



I would like to know if any one able to provide hbase-site.xml to any Java
Application or anyone able to extend Metron Stellar Processor and execute
profile definitions successfully.

Please provide any inputs to resolve the issue.



Thanking you.



Thanks,

Anil.

Re: How to provide hbase-site.xml to Stellar Processor Java API

Posted by Mohan Venkateshaiah <mv...@hortonworks.com>.

Hi Anil, 

I had done similar to this in the past . In the stellar to get the hbase configuration we call HBaseConfiguration.create() , in that call hbase adds the hbase-site and core-site as resources to the config we probably SHOULD let people specify a base config. 

What I had done was in the global config,  I set a property called hbase.provider.impl,  it's the fully qualified class name for a class that implements the TableProvider interface which has one method: 

    public HTableInterface getTable(Configuration config, String tableName) throws IOException

if you implement your own where you ignore the config argument and resolve the hbase table with your own injected config that will work

Thanks 
Mohan DV

On 1/24/19, 8:56 PM, "Otto Fowler" <ot...@gmail.com> wrote:

    Hi Anil,
    Can you create a jira on this with these details and a general overview of
    your use case?
    It looks like the HbaseConfiguration we use in the HTableConnector
    is done using the create() method, which creates from resources.
    
    I think we would need to do some work to support the external file.
    
    
    
    On January 24, 2019 at 10:14:46, Anil Donthireddy (
    anil.donthireddy@sstech.us) wrote:
    
    Hi,
    
    
    
    I have written a java application which uses Stellar processor and execute
    the stellar expressions. The issue I am facing is I am unable to connect
    Hbase unless I place hbase-site.xml in src/main/resources/ folder of the
    code. As it is not the proper way of packaging the hbase-site.xml with Jar,
    I would like to understand how the hbase-site.xml is being set to classpath
    while starting profiler topology.
    
    
    
    The ways I tried are
    
    1)      Setting the classpath to hbase conf folder using command “java -cp
    $CLASSPATH:/etc/hbase/conf:/etc/Hadoop/conf –jar myJar.jar”
    
    2)      Adding Hbase conf folder to HADOOP_CLASSPATH. Below is the Hadoop
    classpath
    
    /usr/hdp/2.6.1.0-129/hadoop/conf:/usr/hdp/2.6.1.0-129/hadoop/lib/*:/usr/hdp/2.6.1.0-129/hadoop/.//*:/usr/hdp/2.6.1.0-129/hadoop-hdfs/./:/usr/hdp/2.6.1.0-129/hadoop-hdfs/lib/*:/usr/hdp/2.6.1.0-129/hadoop-hdfs/.//*:/usr/hdp/2.6.1.0-129/hadoop-yarn/lib/*:/usr/hdp/2.6.1.0-129/hadoop-yarn/.//*:/usr/hdp/2.6.1.0-129/hadoop-mapreduce/lib/*:/usr/hdp/2.6.1.0-129/hadoop-mapreduce/.//*::mysql-connector-java.jar:postgresql-jdbc2ee.jar:postgresql-jdbc2.jar:postgresql-jdbc3.jar:postgresql-jdbc.jar:/etc/hbase/conf/:/usr/hdp/2.6.1.0-129/tez/*:/usr/hdp/2.6.1.0-129/tez/lib/*:/usr/hdp/2.6.1.0-129/tez/conf
    
    
    
    One more step I would like to try to fix the issue is to set property “
    *zookeeper.znode.parent*” for configuration object while instantiating
    HbaseConnector. But it is in the scope of metron code to try this fix.
    
    
    
    I would like to know if any one able to provide hbase-site.xml to any Java
    Application or anyone able to extend Metron Stellar Processor and execute
    profile definitions successfully.
    
    Please provide any inputs to resolve the issue.
    
    
    
    Thanking you.
    
    
    
    Thanks,
    
    Anil.

Re: How to provide hbase-site.xml to Stellar Processor Java API

Posted by Mohan Venkateshaiah <mv...@hortonworks.com>.

Hi Anil, 

I had done similar to this in the past . In the stellar to get the hbase configuration we call HBaseConfiguration.create() , in that call hbase adds the hbase-site and core-site as resources to the config we probably SHOULD let people specify a base config. 

What I had done was in the global config,  I set a property called hbase.provider.impl,  it's the fully qualified class name for a class that implements the TableProvider interface which has one method: 

    public HTableInterface getTable(Configuration config, String tableName) throws IOException

if you implement your own where you ignore the config argument and resolve the hbase table with your own injected config that will work

Thanks 
Mohan DV

On 1/24/19, 8:56 PM, "Otto Fowler" <ot...@gmail.com> wrote:

    Hi Anil,
    Can you create a jira on this with these details and a general overview of
    your use case?
    It looks like the HbaseConfiguration we use in the HTableConnector
    is done using the create() method, which creates from resources.
    
    I think we would need to do some work to support the external file.
    
    
    
    On January 24, 2019 at 10:14:46, Anil Donthireddy (
    anil.donthireddy@sstech.us) wrote:
    
    Hi,
    
    
    
    I have written a java application which uses Stellar processor and execute
    the stellar expressions. The issue I am facing is I am unable to connect
    Hbase unless I place hbase-site.xml in src/main/resources/ folder of the
    code. As it is not the proper way of packaging the hbase-site.xml with Jar,
    I would like to understand how the hbase-site.xml is being set to classpath
    while starting profiler topology.
    
    
    
    The ways I tried are
    
    1)      Setting the classpath to hbase conf folder using command “java -cp
    $CLASSPATH:/etc/hbase/conf:/etc/Hadoop/conf –jar myJar.jar”
    
    2)      Adding Hbase conf folder to HADOOP_CLASSPATH. Below is the Hadoop
    classpath
    
    /usr/hdp/2.6.1.0-129/hadoop/conf:/usr/hdp/2.6.1.0-129/hadoop/lib/*:/usr/hdp/2.6.1.0-129/hadoop/.//*:/usr/hdp/2.6.1.0-129/hadoop-hdfs/./:/usr/hdp/2.6.1.0-129/hadoop-hdfs/lib/*:/usr/hdp/2.6.1.0-129/hadoop-hdfs/.//*:/usr/hdp/2.6.1.0-129/hadoop-yarn/lib/*:/usr/hdp/2.6.1.0-129/hadoop-yarn/.//*:/usr/hdp/2.6.1.0-129/hadoop-mapreduce/lib/*:/usr/hdp/2.6.1.0-129/hadoop-mapreduce/.//*::mysql-connector-java.jar:postgresql-jdbc2ee.jar:postgresql-jdbc2.jar:postgresql-jdbc3.jar:postgresql-jdbc.jar:/etc/hbase/conf/:/usr/hdp/2.6.1.0-129/tez/*:/usr/hdp/2.6.1.0-129/tez/lib/*:/usr/hdp/2.6.1.0-129/tez/conf
    
    
    
    One more step I would like to try to fix the issue is to set property “
    *zookeeper.znode.parent*” for configuration object while instantiating
    HbaseConnector. But it is in the scope of metron code to try this fix.
    
    
    
    I would like to know if any one able to provide hbase-site.xml to any Java
    Application or anyone able to extend Metron Stellar Processor and execute
    profile definitions successfully.
    
    Please provide any inputs to resolve the issue.
    
    
    
    Thanking you.
    
    
    
    Thanks,
    
    Anil.