You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@trafodion.apache.org by Selva Govindarajan <se...@esgyn.com> on 2016/10/22 01:14:58 UTC
[DISCUSS] Introducing Trafodion customization using its own
configuration file.
Currently Trafodion uses the standard client side Hbase configuration file hbase-site.xml deployed by the distros. It is found that there are variations in this configuration file between distros. At times, the distro manager decides that a given property is not a client property and it is not added to the deployed hbase client configuration file. In addition, there are certain properties like hbase.coprocessor.region.classes need to be configured for Trafodion tables for the transaction management. Hence, I am planning to introduce a configuration file traf-site.xml specific to Trafodion similar to hbase configuration file hbase-site.xml. This configuration file extends the properties inherited from the standard hbase-site.xml.
By default, the traf-site.xml comes with the following properties
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put Trafodion site-specific property overrides in this file. -->
<configuration>
<property>
<name>hbase.hregion.impl</name>
<value>org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegion</value>
</property>
<property>
<name>hbase.coprocessor.region.classes</name>
<value>
org.apache.hadoop.hbase.coprocessor.transactional.TrxRegionObserver,
org.apache.hadoop.hbase.coprocessor.transactional.TrxRegionEndpoint,
org.apache.hadoop.hbase.coprocessor.AggregateImplementation
</value>
</property>
<property>
<name>hbase.client.scanner.timeout.period</name>
<value>3600000</value>
</property>
</configuration>
This property file will be installed if it doesn't exist at $MY_SQROOT/etc directory when sqgen is done. Any client side property can be added to this file and the client connections from Trafodion client processes will inherit them.
The hbase.coprocessor.region.classes are added as Table co-processor to the table descriptor when a Trafodion table is created. Trafodion installer will not be adding to these properties to hbase configuration file soon.
Please feel free to provide your valuable suggestions/comments.
Thanks
Selva
RE: [DISCUSS] Introducing Trafodion customization using its own
configuration file.
Posted by Roberta Marton <ro...@esgyn.com>.
This should work okay for now.
Roberta
-----Original Message-----
From: Selva Govindarajan [mailto:selva.govindarajan@esgyn.com]
Sent: Monday, October 24, 2016 8:10 AM
To: dev@trafodion.incubator.apache.org
Subject: RE: [DISCUSS] Introducing Trafodion customization using its own configuration file.
Hi Roberta,
The property hbase.coprocessor.region.classes can be comma separated list of coprocessor classes. Trafodion code parses this list of classes with comma as the delimiter and adds them as table coprocessor classes one by one. The security coprocessor classes can be added to this property in traf-site.xml.
Will that be sufficient for your use case scenario?
Selva
-----Original Message-----
From: Roberta Marton [mailto:roberta.marton@esgyn.com]
Sent: Monday, October 24, 2016 7:59 AM
To: dev@trafodion.incubator.apache.org
Subject: RE: [DISCUSS] Introducing Trafodion customization using its own configuration file.
This sender failed our fraud detection checks and may not be who they appear to be. Learn about spoofing at http://aka.ms/LearnAboutSpoofing
You mentioned that this change would "extend" HBase config changes and then you mentioned later that it would override settings if there are conficts.
As part of adding security features, we need to add certain co-processors to hbase.coprocessor.region.classes list. Today, special code is part of the installation scripts to make sure they continue to exist after adding transaction co-processor information. Would this code still be needed? That is, would you add the new co-processors to the existing list or replace the list. It would be better if the co-processors were added, not override the existing list.
Roberta
-----Original Message-----
From: Liu, Ming (Ming) [mailto:ming.liu@esgyn.cn]
Sent: Sunday, October 23, 2016 5:14 AM
To: dev@trafodion.incubator.apache.org
Subject: RE: [DISCUSS] Introducing Trafodion customization using its own configuration file.
I like this proposal very much. +1
Ming
-----Original Message-----
From: Selva Govindarajan [mailto:selva.govindarajan@esgyn.com]
Sent: Sunday, October 23, 2016 8:55 AM
To: dev@trafodion.incubator.apache.org
Subject: RE: [DISCUSS] Introducing Trafodion customization using its own configuration file.
With this change, there is no need to restart hbase if 'initialize trafodion' is done with this revision. You can also avoid restarting hbase if the table coprocessors are added manually to all Trafodion tables including the Trafodion meta data tables via hbase shell prior to installation of Trafodion release containing this change. However, I haven't tested it out to confirm it for sure.
Yes. We still need to copy the hbase-trx jar to hbase lib folder so that the region server processes can find the jar file containing the table coprocessor classes.
Currently, Trafodion modifies the ACL of some of the directories in hdfs. I believe, for these ACL to take effect Hadoop needs to be restarted and hence hbase also needs to be restarted.
Selva
-----Original Message-----
From: Zhang, Yi (Eason) [mailto:yi.zhang@esgyn.cn]
Sent: Saturday, October 22, 2016 12:12 AM
To: dev@trafodion.incubator.apache.org
Subject: Re: [DISCUSS] Introducing Trafodion customization using its own configuration file.
If using the new traf-site.ml config file:
1. Do we still need to restart HBase during installation?
2. Do we still need to copy hbase-trx jar to HBase lib folder?
Thanks,
Eason
On 22/10/2016, 10:16, "Selva Govindarajan" <se...@esgyn.com> wrote:
The properties added by the resource traf-site.xml will override the previously added resources, unless it was marked as final.
Selva
-----Original Message-----
From: Sean Broeder [mailto:sean.broeder@esgyn.com]
Sent: Friday, October 21, 2016 7:06 PM
To: dev@trafodion.incubator.apache.org
Subject: RE: [DISCUSS] Introducing Trafodion customization using its own configuration file.
I'm intrigued by this idea. Certainly there have been instances where the distros vary and that makes for difficult Trafodion installations.
What would the order of precedence be? How would it work if the standard hbase-site.xml file had a particular setting that Trafodion wanted to override? Or would HBase override the Trafodion settings? Would this create confusion if a user wanted to configure an HBase setting that Trafodion sets as well?
Thanks,
Sean
-----Original Message-----
From: Selva Govindarajan [mailto:selva.govindarajan@esgyn.com]
Sent: Friday, October 21, 2016 6:15 PM
To: dev@trafodion.incubator.apache.org
Subject: [DISCUSS] Introducing Trafodion customization using its own configuration file.
Currently Trafodion uses the standard client side Hbase configuration file hbase-site.xml deployed by the distros. It is found that there are variations in this configuration file between distros. At times, the distro manager decides that a given property is not a client property and it is not added to the deployed hbase client configuration file. In addition, there are certain properties like hbase.coprocessor.region.classes need to be configured for Trafodion tables for the transaction management. Hence, I am planning to introduce a configuration file traf-site.xml specific to Trafodion similar to hbase configuration file hbase-site.xml. This configuration file extends the properties inherited from the standard hbase-site.xml.
By default, the traf-site.xml comes with the following properties
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put Trafodion site-specific property overrides in this file. -->
<configuration>
<property>
<name>hbase.hregion.impl</name>
<value>org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegion</value>
</property>
<property>
<name>hbase.coprocessor.region.classes</name>
<value>
org.apache.hadoop.hbase.coprocessor.transactional.TrxRegionObserver,
org.apache.hadoop.hbase.coprocessor.transactional.TrxRegionEndpoint,
org.apache.hadoop.hbase.coprocessor.AggregateImplementation
</value>
</property>
<property>
<name>hbase.client.scanner.timeout.period</name>
<value>3600000</value>
</property>
</configuration>
This property file will be installed if it doesn't exist at $MY_SQROOT/etc directory when sqgen is done. Any client side property can be added to this file and the client connections from Trafodion client processes will inherit them.
The hbase.coprocessor.region.classes are added as Table co-processor to the table descriptor when a Trafodion table is created. Trafodion installer will not be adding to these properties to hbase configuration file soon.
Please feel free to provide your valuable suggestions/comments.
Thanks
Selva
RE: [DISCUSS] Introducing Trafodion customization using its own
configuration file.
Posted by Selva Govindarajan <se...@esgyn.com>.
Hi Roberta,
The property hbase.coprocessor.region.classes can be comma separated list of coprocessor classes. Trafodion code parses this list of classes with comma as the delimiter and adds them as table coprocessor classes one by one. The security coprocessor classes can be added to this property in traf-site.xml.
Will that be sufficient for your use case scenario?
Selva
-----Original Message-----
From: Roberta Marton [mailto:roberta.marton@esgyn.com]
Sent: Monday, October 24, 2016 7:59 AM
To: dev@trafodion.incubator.apache.org
Subject: RE: [DISCUSS] Introducing Trafodion customization using its own configuration file.
This sender failed our fraud detection checks and may not be who they appear to be. Learn about spoofing at http://aka.ms/LearnAboutSpoofing
You mentioned that this change would "extend" HBase config changes and then you mentioned later that it would override settings if there are conficts.
As part of adding security features, we need to add certain co-processors to hbase.coprocessor.region.classes list. Today, special code is part of the installation scripts to make sure they continue to exist after adding transaction co-processor information. Would this code still be needed? That is, would you add the new co-processors to the existing list or replace the list. It would be better if the co-processors were added, not override the existing list.
Roberta
-----Original Message-----
From: Liu, Ming (Ming) [mailto:ming.liu@esgyn.cn]
Sent: Sunday, October 23, 2016 5:14 AM
To: dev@trafodion.incubator.apache.org
Subject: RE: [DISCUSS] Introducing Trafodion customization using its own configuration file.
I like this proposal very much. +1
Ming
-----Original Message-----
From: Selva Govindarajan [mailto:selva.govindarajan@esgyn.com]
Sent: Sunday, October 23, 2016 8:55 AM
To: dev@trafodion.incubator.apache.org
Subject: RE: [DISCUSS] Introducing Trafodion customization using its own configuration file.
With this change, there is no need to restart hbase if 'initialize trafodion' is done with this revision. You can also avoid restarting hbase if the table coprocessors are added manually to all Trafodion tables including the Trafodion meta data tables via hbase shell prior to installation of Trafodion release containing this change. However, I haven't tested it out to confirm it for sure.
Yes. We still need to copy the hbase-trx jar to hbase lib folder so that the region server processes can find the jar file containing the table coprocessor classes.
Currently, Trafodion modifies the ACL of some of the directories in hdfs. I believe, for these ACL to take effect Hadoop needs to be restarted and hence hbase also needs to be restarted.
Selva
-----Original Message-----
From: Zhang, Yi (Eason) [mailto:yi.zhang@esgyn.cn]
Sent: Saturday, October 22, 2016 12:12 AM
To: dev@trafodion.incubator.apache.org
Subject: Re: [DISCUSS] Introducing Trafodion customization using its own configuration file.
If using the new traf-site.ml config file:
1. Do we still need to restart HBase during installation?
2. Do we still need to copy hbase-trx jar to HBase lib folder?
Thanks,
Eason
On 22/10/2016, 10:16, "Selva Govindarajan" <se...@esgyn.com> wrote:
The properties added by the resource traf-site.xml will override the previously added resources, unless it was marked as final.
Selva
-----Original Message-----
From: Sean Broeder [mailto:sean.broeder@esgyn.com]
Sent: Friday, October 21, 2016 7:06 PM
To: dev@trafodion.incubator.apache.org
Subject: RE: [DISCUSS] Introducing Trafodion customization using its own configuration file.
I'm intrigued by this idea. Certainly there have been instances where the distros vary and that makes for difficult Trafodion installations.
What would the order of precedence be? How would it work if the standard hbase-site.xml file had a particular setting that Trafodion wanted to override? Or would HBase override the Trafodion settings? Would this create confusion if a user wanted to configure an HBase setting that Trafodion sets as well?
Thanks,
Sean
-----Original Message-----
From: Selva Govindarajan [mailto:selva.govindarajan@esgyn.com]
Sent: Friday, October 21, 2016 6:15 PM
To: dev@trafodion.incubator.apache.org
Subject: [DISCUSS] Introducing Trafodion customization using its own configuration file.
Currently Trafodion uses the standard client side Hbase configuration file hbase-site.xml deployed by the distros. It is found that there are variations in this configuration file between distros. At times, the distro manager decides that a given property is not a client property and it is not added to the deployed hbase client configuration file. In addition, there are certain properties like hbase.coprocessor.region.classes need to be configured for Trafodion tables for the transaction management. Hence, I am planning to introduce a configuration file traf-site.xml specific to Trafodion similar to hbase configuration file hbase-site.xml. This configuration file extends the properties inherited from the standard hbase-site.xml.
By default, the traf-site.xml comes with the following properties
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put Trafodion site-specific property overrides in this file. -->
<configuration>
<property>
<name>hbase.hregion.impl</name>
<value>org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegion</value>
</property>
<property>
<name>hbase.coprocessor.region.classes</name>
<value>
org.apache.hadoop.hbase.coprocessor.transactional.TrxRegionObserver,
org.apache.hadoop.hbase.coprocessor.transactional.TrxRegionEndpoint,
org.apache.hadoop.hbase.coprocessor.AggregateImplementation
</value>
</property>
<property>
<name>hbase.client.scanner.timeout.period</name>
<value>3600000</value>
</property>
</configuration>
This property file will be installed if it doesn't exist at $MY_SQROOT/etc directory when sqgen is done. Any client side property can be added to this file and the client connections from Trafodion client processes will inherit them.
The hbase.coprocessor.region.classes are added as Table co-processor to the table descriptor when a Trafodion table is created. Trafodion installer will not be adding to these properties to hbase configuration file soon.
Please feel free to provide your valuable suggestions/comments.
Thanks
Selva
RE: [DISCUSS] Introducing Trafodion customization using its own
configuration file.
Posted by Roberta Marton <ro...@esgyn.com>.
You mentioned that this change would "extend" HBase config changes and then you mentioned later that it would override settings if there are conficts.
As part of adding security features, we need to add certain co-processors to hbase.coprocessor.region.classes list. Today, special code is part of the installation scripts to make sure they continue to exist after adding transaction co-processor information. Would this code still be needed? That is, would you add the new co-processors to the existing list or replace the list. It would be better if the co-processors were added, not override the existing list.
Roberta
-----Original Message-----
From: Liu, Ming (Ming) [mailto:ming.liu@esgyn.cn]
Sent: Sunday, October 23, 2016 5:14 AM
To: dev@trafodion.incubator.apache.org
Subject: RE: [DISCUSS] Introducing Trafodion customization using its own configuration file.
I like this proposal very much. +1
Ming
-----Original Message-----
From: Selva Govindarajan [mailto:selva.govindarajan@esgyn.com]
Sent: Sunday, October 23, 2016 8:55 AM
To: dev@trafodion.incubator.apache.org
Subject: RE: [DISCUSS] Introducing Trafodion customization using its own configuration file.
With this change, there is no need to restart hbase if 'initialize trafodion' is done with this revision. You can also avoid restarting hbase if the table coprocessors are added manually to all Trafodion tables including the Trafodion meta data tables via hbase shell prior to installation of Trafodion release containing this change. However, I haven't tested it out to confirm it for sure.
Yes. We still need to copy the hbase-trx jar to hbase lib folder so that the region server processes can find the jar file containing the table coprocessor classes.
Currently, Trafodion modifies the ACL of some of the directories in hdfs. I believe, for these ACL to take effect Hadoop needs to be restarted and hence hbase also needs to be restarted.
Selva
-----Original Message-----
From: Zhang, Yi (Eason) [mailto:yi.zhang@esgyn.cn]
Sent: Saturday, October 22, 2016 12:12 AM
To: dev@trafodion.incubator.apache.org
Subject: Re: [DISCUSS] Introducing Trafodion customization using its own configuration file.
If using the new traf-site.ml config file:
1. Do we still need to restart HBase during installation?
2. Do we still need to copy hbase-trx jar to HBase lib folder?
Thanks,
Eason
On 22/10/2016, 10:16, "Selva Govindarajan" <se...@esgyn.com> wrote:
The properties added by the resource traf-site.xml will override the previously added resources, unless it was marked as final.
Selva
-----Original Message-----
From: Sean Broeder [mailto:sean.broeder@esgyn.com]
Sent: Friday, October 21, 2016 7:06 PM
To: dev@trafodion.incubator.apache.org
Subject: RE: [DISCUSS] Introducing Trafodion customization using its own configuration file.
I'm intrigued by this idea. Certainly there have been instances where the distros vary and that makes for difficult Trafodion installations.
What would the order of precedence be? How would it work if the standard hbase-site.xml file had a particular setting that Trafodion wanted to override? Or would HBase override the Trafodion settings? Would this create confusion if a user wanted to configure an HBase setting that Trafodion sets as well?
Thanks,
Sean
-----Original Message-----
From: Selva Govindarajan [mailto:selva.govindarajan@esgyn.com]
Sent: Friday, October 21, 2016 6:15 PM
To: dev@trafodion.incubator.apache.org
Subject: [DISCUSS] Introducing Trafodion customization using its own configuration file.
Currently Trafodion uses the standard client side Hbase configuration file hbase-site.xml deployed by the distros. It is found that there are variations in this configuration file between distros. At times, the distro manager decides that a given property is not a client property and it is not added to the deployed hbase client configuration file. In addition, there are certain properties like hbase.coprocessor.region.classes need to be configured for Trafodion tables for the transaction management. Hence, I am planning to introduce a configuration file traf-site.xml specific to Trafodion similar to hbase configuration file hbase-site.xml. This configuration file extends the properties inherited from the standard hbase-site.xml.
By default, the traf-site.xml comes with the following properties
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put Trafodion site-specific property overrides in this file. -->
<configuration>
<property>
<name>hbase.hregion.impl</name>
<value>org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegion</value>
</property>
<property>
<name>hbase.coprocessor.region.classes</name>
<value>
org.apache.hadoop.hbase.coprocessor.transactional.TrxRegionObserver,
org.apache.hadoop.hbase.coprocessor.transactional.TrxRegionEndpoint,
org.apache.hadoop.hbase.coprocessor.AggregateImplementation
</value>
</property>
<property>
<name>hbase.client.scanner.timeout.period</name>
<value>3600000</value>
</property>
</configuration>
This property file will be installed if it doesn't exist at $MY_SQROOT/etc directory when sqgen is done. Any client side property can be added to this file and the client connections from Trafodion client processes will inherit them.
The hbase.coprocessor.region.classes are added as Table co-processor to the table descriptor when a Trafodion table is created. Trafodion installer will not be adding to these properties to hbase configuration file soon.
Please feel free to provide your valuable suggestions/comments.
Thanks
Selva
RE: [DISCUSS] Introducing Trafodion customization using its own
configuration file.
Posted by Selva Govindarajan <se...@esgyn.com>.
Thanks for all the suggestions and comments.
The resource file traf-site.xml being renamed as trafodion-site.xml is mainly for the client side properties. There is no need for hbase to be restarted for these properties to take effect, but trafodion need to be restarted.
As suggested by Hans/Steve, the resource file can contain the hbase property name prefixed with 'trafodion.'. Trafodion will strip 'trafodion.' and then add to the Configuration.
For details on this change, please see https://github.com/apache/incubator-trafodion/pull/777
Selva
-----Original Message-----
From: Qifan Chen [mailto:qifan.chen@esgyn.com]
Sent: Monday, October 24, 2016 11:14 AM
To: dev@trafodion.incubator.apache.org
Subject: Re: [DISCUSS] Introducing Trafodion customization using its own configuration file.
This sender failed our fraud detection checks and may not be who they appear to be. Learn about spoofing at http://aka.ms/LearnAboutSpoofing
We may also clarify (e.g. with a comment in the config file) whether trafodion.hbase.client.scanner.timeout.period applies only to HBase tables used by Trafodion, or to all Hbase tables after the restart.
--Qifan
________________________________
From: Dave Birdsall <da...@esgyn.com>
Sent: Monday, October 24, 2016 12:43:47 PM
To: dev@trafodion.incubator.apache.org
Subject: RE: [DISCUSS] Introducing Trafodion customization using its own configuration file.
This sender failed our fraud detection checks and may not be who they appear to be. Learn about spoofing at http://aka.ms/LearnAboutSpoofing
Hi,
We should take a look at the Trafodion HBaseClient.java module to see what other properties have been overridden.
hbase.client.scanner.timeout.period is one of them; thanks for getting that one.
Another one is in method estimateRowCount: HConstants.BUCKET_CACHE_IOENGINE_KEY (You'll have to look up the property name in the HBase HConstants.java file).
Dave
-----Original Message-----
From: Steve Varnau [mailto:steve.varnau@esgyn.com]
Sent: Monday, October 24, 2016 9:59 AM
To: dev@trafodion.incubator.apache.org
Subject: RE: [DISCUSS] Introducing Trafodion customization using its own configuration file.
This should make install/configuration of trafodion much easier, since it does not need to modify properties in other components. It will certainly help with the cluster-manager integration I'm working on.
I agree with Hans that adding trafodion or "traf" to the property names would help clarify.
--Steve
> -----Original Message-----
> From: Hans Zeller [mailto:hans.zeller@esgyn.com]
> Sent: Monday, October 24, 2016 9:06 AM
> To: dev@trafodion.incubator.apache.org
> Subject: RE: [DISCUSS] Introducing Trafodion customization using its
> own configuration file.
>
> This sender failed our fraud detection checks and may not be who they
> appear to be. Learn about spoofing at http://aka.ms/LearnAboutSpoofing
>
> Hi,
>
> Sounds good to me, too. I wonder, however, whether we shouldn't
> prepend "trafodion" to the properties in the file. So, instead of
> hbase.client.scanner.timeout.period, for example, what about calling
> it trafodion.hbase.client.scanner.timeout.period. This should avoid confusion.
>
> Thanks,
>
> Hans
>
> -----Original Message-----
> From: Liu, Ming (Ming) [mailto:ming.liu@esgyn.cn]
> Sent: Sunday, October 23, 2016 5:14 AM
> To: dev@trafodion.incubator.apache.org
> Subject: RE: [DISCUSS] Introducing Trafodion customization using its
> own configuration file.
>
> I like this proposal very much. +1
>
> Ming
>
> -----Original Message-----
> From: Selva Govindarajan [mailto:selva.govindarajan@esgyn.com]
> Sent: Sunday, October 23, 2016 8:55 AM
> To: dev@trafodion.incubator.apache.org
> Subject: RE: [DISCUSS] Introducing Trafodion customization using its
> own configuration file.
>
> With this change, there is no need to restart hbase if 'initialize
> trafodion' is done with this revision. You can also avoid restarting
> hbase if the table coprocessors are added manually to all Trafodion
> tables including the Trafodion meta data tables via hbase shell prior
> to installation of Trafodion release containing this change. However,
> I haven't tested it out to confirm it for sure.
>
> Yes. We still need to copy the hbase-trx jar to hbase lib folder so
> that the region server processes can find the jar file containing the
> table coprocessor classes.
>
> Currently, Trafodion modifies the ACL of some of the directories in
> hdfs. I believe, for these ACL to take effect Hadoop needs to be
> restarted and hence hbase also needs to be restarted.
>
>
> Selva
>
>
> -----Original Message-----
> From: Zhang, Yi (Eason) [mailto:yi.zhang@esgyn.cn]
> Sent: Saturday, October 22, 2016 12:12 AM
> To: dev@trafodion.incubator.apache.org
> Subject: Re: [DISCUSS] Introducing Trafodion customization using its
> own configuration file.
>
> If using the new traf-site.ml config file:
>
> 1. Do we still need to restart HBase during installation?
> 2. Do we still need to copy hbase-trx jar to HBase lib folder?
>
>
> Thanks,
> Eason
>
>
>
> On 22/10/2016, 10:16, "Selva Govindarajan"
> <se...@esgyn.com> wrote:
>
> The properties added by the resource traf-site.xml will override
> the previously added resources, unless it was marked as final.
>
> Selva
> -----Original Message-----
> From: Sean Broeder [mailto:sean.broeder@esgyn.com]
> Sent: Friday, October 21, 2016 7:06 PM
> To: dev@trafodion.incubator.apache.org
> Subject: RE: [DISCUSS] Introducing Trafodion customization using
> its own configuration file.
>
> I'm intrigued by this idea. Certainly there have been instances
> where the distros vary and that makes for difficult Trafodion installations.
>
> What would the order of precedence be? How would it work if the
> standard hbase-site.xml file had a particular setting that Trafodion
> wanted to override? Or would HBase override the Trafodion settings?
> Would this create confusion if a user wanted to configure an HBase
> setting that Trafodion sets as well?
>
> Thanks,
> Sean
>
> -----Original Message-----
> From: Selva Govindarajan [mailto:selva.govindarajan@esgyn.com]
> Sent: Friday, October 21, 2016 6:15 PM
> To: dev@trafodion.incubator.apache.org
> Subject: [DISCUSS] Introducing Trafodion customization using its
> own configuration file.
>
> Currently Trafodion uses the standard client side Hbase
> configuration file hbase-site.xml deployed by the distros. It is found
> that there are variations in this configuration file between distros.
> At times, the distro manager decides that a given property is not a
> client property and it is not added to the deployed hbase client
> configuration file. In addition, there are certain properties like
> hbase.coprocessor.region.classes need to be configured for Trafodion
> tables for the transaction management. Hence, I am planning to
> introduce a configuration file traf-site.xml specific to Trafodion
> similar to hbase configuration file hbase-site.xml. This configuration file extends the properties inherited from the standard hbase-site.xml.
>
> By default, the traf-site.xml comes with the following properties
>
> <?xml version="1.0"?>
> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>
> <!-- Put Trafodion site-specific property overrides in this file.
> -->
>
> <configuration>
> <property>
> <name>hbase.hregion.impl</name>
>
> <value>org.apache.hadoop.hbase.regionserver.transactional.Transactiona
> lR
> egion</value>
> </property>
> <property>
> <name>hbase.coprocessor.region.classes</name>
> <value>
>
> org.apache.hadoop.hbase.coprocessor.transactional.TrxRegionObserver,
>
> org.apache.hadoop.hbase.coprocessor.transactional.TrxRegionEndpoint,
> org.apache.hadoop.hbase.coprocessor.AggregateImplementation
> </value>
> </property>
> <property>
> <name>hbase.client.scanner.timeout.period</name>
> <value>3600000</value>
> </property>
> </configuration>
>
> This property file will be installed if it doesn't exist at
> $MY_SQROOT/etc directory when sqgen is done. Any client side property
> can be added to this file and the client connections from Trafodion
> client processes will inherit them.
>
> The hbase.coprocessor.region.classes are added as Table
> co-processor to the table descriptor when a Trafodion table is
> created. Trafodion installer will not be adding to these properties to hbase configuration file soon.
>
> Please feel free to provide your valuable suggestions/comments.
>
> Thanks
> Selva
>
>
>
>
>
Re: [DISCUSS] Introducing Trafodion customization using its own
configuration file.
Posted by Qifan Chen <qi...@esgyn.com>.
We may also clarify (e.g. with a comment in the config file) whether trafodion.hbase.client.scanner.timeout.period applies only to HBase tables used by Trafodion, or to all Hbase tables after the restart.
--Qifan
________________________________
From: Dave Birdsall <da...@esgyn.com>
Sent: Monday, October 24, 2016 12:43:47 PM
To: dev@trafodion.incubator.apache.org
Subject: RE: [DISCUSS] Introducing Trafodion customization using its own configuration file.
This sender failed our fraud detection checks and may not be who they appear to be. Learn about spoofing at http://aka.ms/LearnAboutSpoofing
Hi,
We should take a look at the Trafodion HBaseClient.java module to see what other properties have been overridden.
hbase.client.scanner.timeout.period is one of them; thanks for getting that one.
Another one is in method estimateRowCount: HConstants.BUCKET_CACHE_IOENGINE_KEY (You'll have to look up the property name in the HBase HConstants.java file).
Dave
-----Original Message-----
From: Steve Varnau [mailto:steve.varnau@esgyn.com]
Sent: Monday, October 24, 2016 9:59 AM
To: dev@trafodion.incubator.apache.org
Subject: RE: [DISCUSS] Introducing Trafodion customization using its own configuration file.
This should make install/configuration of trafodion much easier, since it does not need to modify properties in other components. It will certainly help with the cluster-manager integration I'm working on.
I agree with Hans that adding trafodion or "traf" to the property names would help clarify.
--Steve
> -----Original Message-----
> From: Hans Zeller [mailto:hans.zeller@esgyn.com]
> Sent: Monday, October 24, 2016 9:06 AM
> To: dev@trafodion.incubator.apache.org
> Subject: RE: [DISCUSS] Introducing Trafodion customization using its
> own configuration file.
>
> This sender failed our fraud detection checks and may not be who they
> appear to be. Learn about spoofing at http://aka.ms/LearnAboutSpoofing
>
> Hi,
>
> Sounds good to me, too. I wonder, however, whether we shouldn't
> prepend "trafodion" to the properties in the file. So, instead of
> hbase.client.scanner.timeout.period, for example, what about calling
> it trafodion.hbase.client.scanner.timeout.period. This should avoid confusion.
>
> Thanks,
>
> Hans
>
> -----Original Message-----
> From: Liu, Ming (Ming) [mailto:ming.liu@esgyn.cn]
> Sent: Sunday, October 23, 2016 5:14 AM
> To: dev@trafodion.incubator.apache.org
> Subject: RE: [DISCUSS] Introducing Trafodion customization using its
> own configuration file.
>
> I like this proposal very much. +1
>
> Ming
>
> -----Original Message-----
> From: Selva Govindarajan [mailto:selva.govindarajan@esgyn.com]
> Sent: Sunday, October 23, 2016 8:55 AM
> To: dev@trafodion.incubator.apache.org
> Subject: RE: [DISCUSS] Introducing Trafodion customization using its
> own configuration file.
>
> With this change, there is no need to restart hbase if 'initialize
> trafodion' is done with this revision. You can also avoid restarting
> hbase if the table coprocessors are added manually to all Trafodion
> tables including the Trafodion meta data tables via hbase shell prior
> to installation of Trafodion release containing this change. However,
> I haven't tested it out to confirm it for sure.
>
> Yes. We still need to copy the hbase-trx jar to hbase lib folder so
> that the region server processes can find the jar file containing the
> table coprocessor classes.
>
> Currently, Trafodion modifies the ACL of some of the directories in
> hdfs. I believe, for these ACL to take effect Hadoop needs to be
> restarted and hence hbase also needs to be restarted.
>
>
> Selva
>
>
> -----Original Message-----
> From: Zhang, Yi (Eason) [mailto:yi.zhang@esgyn.cn]
> Sent: Saturday, October 22, 2016 12:12 AM
> To: dev@trafodion.incubator.apache.org
> Subject: Re: [DISCUSS] Introducing Trafodion customization using its
> own configuration file.
>
> If using the new traf-site.ml config file:
>
> 1. Do we still need to restart HBase during installation?
> 2. Do we still need to copy hbase-trx jar to HBase lib folder?
>
>
> Thanks,
> Eason
>
>
>
> On 22/10/2016, 10:16, "Selva Govindarajan"
> <se...@esgyn.com> wrote:
>
> The properties added by the resource traf-site.xml will override
> the previously added resources, unless it was marked as final.
>
> Selva
> -----Original Message-----
> From: Sean Broeder [mailto:sean.broeder@esgyn.com]
> Sent: Friday, October 21, 2016 7:06 PM
> To: dev@trafodion.incubator.apache.org
> Subject: RE: [DISCUSS] Introducing Trafodion customization using
> its own configuration file.
>
> I'm intrigued by this idea. Certainly there have been instances
> where the distros vary and that makes for difficult Trafodion installations.
>
> What would the order of precedence be? How would it work if the
> standard hbase-site.xml file had a particular setting that Trafodion
> wanted to override? Or would HBase override the Trafodion settings?
> Would this create confusion if a user wanted to configure an HBase
> setting that Trafodion sets as well?
>
> Thanks,
> Sean
>
> -----Original Message-----
> From: Selva Govindarajan [mailto:selva.govindarajan@esgyn.com]
> Sent: Friday, October 21, 2016 6:15 PM
> To: dev@trafodion.incubator.apache.org
> Subject: [DISCUSS] Introducing Trafodion customization using its
> own configuration file.
>
> Currently Trafodion uses the standard client side Hbase
> configuration file hbase-site.xml deployed by the distros. It is found
> that there are variations in this configuration file between distros.
> At times, the distro manager decides that a given property is not a
> client property and it is not added to the deployed hbase client
> configuration file. In addition, there are certain properties like
> hbase.coprocessor.region.classes need to be configured for Trafodion
> tables for the transaction management. Hence, I am planning to
> introduce a configuration file traf-site.xml specific to Trafodion
> similar to hbase configuration file hbase-site.xml. This configuration file extends the properties inherited from the standard hbase-site.xml.
>
> By default, the traf-site.xml comes with the following properties
>
> <?xml version="1.0"?>
> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>
> <!-- Put Trafodion site-specific property overrides in this file.
> -->
>
> <configuration>
> <property>
> <name>hbase.hregion.impl</name>
>
> <value>org.apache.hadoop.hbase.regionserver.transactional.Transactiona
> lR
> egion</value>
> </property>
> <property>
> <name>hbase.coprocessor.region.classes</name>
> <value>
>
> org.apache.hadoop.hbase.coprocessor.transactional.TrxRegionObserver,
>
> org.apache.hadoop.hbase.coprocessor.transactional.TrxRegionEndpoint,
> org.apache.hadoop.hbase.coprocessor.AggregateImplementation
> </value>
> </property>
> <property>
> <name>hbase.client.scanner.timeout.period</name>
> <value>3600000</value>
> </property>
> </configuration>
>
> This property file will be installed if it doesn't exist at
> $MY_SQROOT/etc directory when sqgen is done. Any client side property
> can be added to this file and the client connections from Trafodion
> client processes will inherit them.
>
> The hbase.coprocessor.region.classes are added as Table
> co-processor to the table descriptor when a Trafodion table is
> created. Trafodion installer will not be adding to these properties to hbase configuration file soon.
>
> Please feel free to provide your valuable suggestions/comments.
>
> Thanks
> Selva
>
>
>
>
>
RE: [DISCUSS] Introducing Trafodion customization using its own
configuration file.
Posted by Dave Birdsall <da...@esgyn.com>.
Hi,
We should take a look at the Trafodion HBaseClient.java module to see what other properties have been overridden.
hbase.client.scanner.timeout.period is one of them; thanks for getting that one.
Another one is in method estimateRowCount: HConstants.BUCKET_CACHE_IOENGINE_KEY (You'll have to look up the property name in the HBase HConstants.java file).
Dave
-----Original Message-----
From: Steve Varnau [mailto:steve.varnau@esgyn.com]
Sent: Monday, October 24, 2016 9:59 AM
To: dev@trafodion.incubator.apache.org
Subject: RE: [DISCUSS] Introducing Trafodion customization using its own configuration file.
This should make install/configuration of trafodion much easier, since it does not need to modify properties in other components. It will certainly help with the cluster-manager integration I'm working on.
I agree with Hans that adding trafodion or "traf" to the property names would help clarify.
--Steve
> -----Original Message-----
> From: Hans Zeller [mailto:hans.zeller@esgyn.com]
> Sent: Monday, October 24, 2016 9:06 AM
> To: dev@trafodion.incubator.apache.org
> Subject: RE: [DISCUSS] Introducing Trafodion customization using its
> own configuration file.
>
> This sender failed our fraud detection checks and may not be who they
> appear to be. Learn about spoofing at http://aka.ms/LearnAboutSpoofing
>
> Hi,
>
> Sounds good to me, too. I wonder, however, whether we shouldn't
> prepend "trafodion" to the properties in the file. So, instead of
> hbase.client.scanner.timeout.period, for example, what about calling
> it trafodion.hbase.client.scanner.timeout.period. This should avoid confusion.
>
> Thanks,
>
> Hans
>
> -----Original Message-----
> From: Liu, Ming (Ming) [mailto:ming.liu@esgyn.cn]
> Sent: Sunday, October 23, 2016 5:14 AM
> To: dev@trafodion.incubator.apache.org
> Subject: RE: [DISCUSS] Introducing Trafodion customization using its
> own configuration file.
>
> I like this proposal very much. +1
>
> Ming
>
> -----Original Message-----
> From: Selva Govindarajan [mailto:selva.govindarajan@esgyn.com]
> Sent: Sunday, October 23, 2016 8:55 AM
> To: dev@trafodion.incubator.apache.org
> Subject: RE: [DISCUSS] Introducing Trafodion customization using its
> own configuration file.
>
> With this change, there is no need to restart hbase if 'initialize
> trafodion' is done with this revision. You can also avoid restarting
> hbase if the table coprocessors are added manually to all Trafodion
> tables including the Trafodion meta data tables via hbase shell prior
> to installation of Trafodion release containing this change. However,
> I haven't tested it out to confirm it for sure.
>
> Yes. We still need to copy the hbase-trx jar to hbase lib folder so
> that the region server processes can find the jar file containing the
> table coprocessor classes.
>
> Currently, Trafodion modifies the ACL of some of the directories in
> hdfs. I believe, for these ACL to take effect Hadoop needs to be
> restarted and hence hbase also needs to be restarted.
>
>
> Selva
>
>
> -----Original Message-----
> From: Zhang, Yi (Eason) [mailto:yi.zhang@esgyn.cn]
> Sent: Saturday, October 22, 2016 12:12 AM
> To: dev@trafodion.incubator.apache.org
> Subject: Re: [DISCUSS] Introducing Trafodion customization using its
> own configuration file.
>
> If using the new traf-site.ml config file:
>
> 1. Do we still need to restart HBase during installation?
> 2. Do we still need to copy hbase-trx jar to HBase lib folder?
>
>
> Thanks,
> Eason
>
>
>
> On 22/10/2016, 10:16, "Selva Govindarajan"
> <se...@esgyn.com> wrote:
>
> The properties added by the resource traf-site.xml will override
> the previously added resources, unless it was marked as final.
>
> Selva
> -----Original Message-----
> From: Sean Broeder [mailto:sean.broeder@esgyn.com]
> Sent: Friday, October 21, 2016 7:06 PM
> To: dev@trafodion.incubator.apache.org
> Subject: RE: [DISCUSS] Introducing Trafodion customization using
> its own configuration file.
>
> I'm intrigued by this idea. Certainly there have been instances
> where the distros vary and that makes for difficult Trafodion installations.
>
> What would the order of precedence be? How would it work if the
> standard hbase-site.xml file had a particular setting that Trafodion
> wanted to override? Or would HBase override the Trafodion settings?
> Would this create confusion if a user wanted to configure an HBase
> setting that Trafodion sets as well?
>
> Thanks,
> Sean
>
> -----Original Message-----
> From: Selva Govindarajan [mailto:selva.govindarajan@esgyn.com]
> Sent: Friday, October 21, 2016 6:15 PM
> To: dev@trafodion.incubator.apache.org
> Subject: [DISCUSS] Introducing Trafodion customization using its
> own configuration file.
>
> Currently Trafodion uses the standard client side Hbase
> configuration file hbase-site.xml deployed by the distros. It is found
> that there are variations in this configuration file between distros.
> At times, the distro manager decides that a given property is not a
> client property and it is not added to the deployed hbase client
> configuration file. In addition, there are certain properties like
> hbase.coprocessor.region.classes need to be configured for Trafodion
> tables for the transaction management. Hence, I am planning to
> introduce a configuration file traf-site.xml specific to Trafodion
> similar to hbase configuration file hbase-site.xml. This configuration file extends the properties inherited from the standard hbase-site.xml.
>
> By default, the traf-site.xml comes with the following properties
>
> <?xml version="1.0"?>
> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>
> <!-- Put Trafodion site-specific property overrides in this file.
> -->
>
> <configuration>
> <property>
> <name>hbase.hregion.impl</name>
>
> <value>org.apache.hadoop.hbase.regionserver.transactional.Transactiona
> lR
> egion</value>
> </property>
> <property>
> <name>hbase.coprocessor.region.classes</name>
> <value>
>
> org.apache.hadoop.hbase.coprocessor.transactional.TrxRegionObserver,
>
> org.apache.hadoop.hbase.coprocessor.transactional.TrxRegionEndpoint,
> org.apache.hadoop.hbase.coprocessor.AggregateImplementation
> </value>
> </property>
> <property>
> <name>hbase.client.scanner.timeout.period</name>
> <value>3600000</value>
> </property>
> </configuration>
>
> This property file will be installed if it doesn't exist at
> $MY_SQROOT/etc directory when sqgen is done. Any client side property
> can be added to this file and the client connections from Trafodion
> client processes will inherit them.
>
> The hbase.coprocessor.region.classes are added as Table
> co-processor to the table descriptor when a Trafodion table is
> created. Trafodion installer will not be adding to these properties to hbase configuration file soon.
>
> Please feel free to provide your valuable suggestions/comments.
>
> Thanks
> Selva
>
>
>
>
>
RE: [DISCUSS] Introducing Trafodion customization using its own
configuration file.
Posted by Steve Varnau <st...@esgyn.com>.
This should make install/configuration of trafodion much easier, since it does not need to modify properties in other components. It will certainly help with the cluster-manager integration I'm working on.
I agree with Hans that adding trafodion or "traf" to the property names would help clarify.
--Steve
> -----Original Message-----
> From: Hans Zeller [mailto:hans.zeller@esgyn.com]
> Sent: Monday, October 24, 2016 9:06 AM
> To: dev@trafodion.incubator.apache.org
> Subject: RE: [DISCUSS] Introducing Trafodion customization using its own
> configuration file.
>
> This sender failed our fraud detection checks and may not be who they
> appear to be. Learn about spoofing at http://aka.ms/LearnAboutSpoofing
>
> Hi,
>
> Sounds good to me, too. I wonder, however, whether we shouldn't prepend
> "trafodion" to the properties in the file. So, instead of
> hbase.client.scanner.timeout.period, for example, what about calling it
> trafodion.hbase.client.scanner.timeout.period. This should avoid confusion.
>
> Thanks,
>
> Hans
>
> -----Original Message-----
> From: Liu, Ming (Ming) [mailto:ming.liu@esgyn.cn]
> Sent: Sunday, October 23, 2016 5:14 AM
> To: dev@trafodion.incubator.apache.org
> Subject: RE: [DISCUSS] Introducing Trafodion customization using its own
> configuration file.
>
> I like this proposal very much. +1
>
> Ming
>
> -----Original Message-----
> From: Selva Govindarajan [mailto:selva.govindarajan@esgyn.com]
> Sent: Sunday, October 23, 2016 8:55 AM
> To: dev@trafodion.incubator.apache.org
> Subject: RE: [DISCUSS] Introducing Trafodion customization using its own
> configuration file.
>
> With this change, there is no need to restart hbase if 'initialize trafodion' is
> done with this revision. You can also avoid restarting hbase if the table
> coprocessors are added manually to all Trafodion tables including the
> Trafodion meta data tables via hbase shell prior to installation of Trafodion
> release containing this change. However, I haven't tested it out to confirm it
> for sure.
>
> Yes. We still need to copy the hbase-trx jar to hbase lib folder so that the
> region server processes can find the jar file containing the table coprocessor
> classes.
>
> Currently, Trafodion modifies the ACL of some of the directories in hdfs. I
> believe, for these ACL to take effect Hadoop needs to be restarted and
> hence hbase also needs to be restarted.
>
>
> Selva
>
>
> -----Original Message-----
> From: Zhang, Yi (Eason) [mailto:yi.zhang@esgyn.cn]
> Sent: Saturday, October 22, 2016 12:12 AM
> To: dev@trafodion.incubator.apache.org
> Subject: Re: [DISCUSS] Introducing Trafodion customization using its own
> configuration file.
>
> If using the new traf-site.ml config file:
>
> 1. Do we still need to restart HBase during installation?
> 2. Do we still need to copy hbase-trx jar to HBase lib folder?
>
>
> Thanks,
> Eason
>
>
>
> On 22/10/2016, 10:16, "Selva Govindarajan"
> <se...@esgyn.com> wrote:
>
> The properties added by the resource traf-site.xml will override the
> previously added resources, unless it was marked as final.
>
> Selva
> -----Original Message-----
> From: Sean Broeder [mailto:sean.broeder@esgyn.com]
> Sent: Friday, October 21, 2016 7:06 PM
> To: dev@trafodion.incubator.apache.org
> Subject: RE: [DISCUSS] Introducing Trafodion customization using its own
> configuration file.
>
> I'm intrigued by this idea. Certainly there have been instances where the
> distros vary and that makes for difficult Trafodion installations.
>
> What would the order of precedence be? How would it work if the
> standard hbase-site.xml file had a particular setting that Trafodion wanted to
> override? Or would HBase override the Trafodion settings? Would this
> create confusion if a user wanted to configure an HBase setting that
> Trafodion sets as well?
>
> Thanks,
> Sean
>
> -----Original Message-----
> From: Selva Govindarajan [mailto:selva.govindarajan@esgyn.com]
> Sent: Friday, October 21, 2016 6:15 PM
> To: dev@trafodion.incubator.apache.org
> Subject: [DISCUSS] Introducing Trafodion customization using its own
> configuration file.
>
> Currently Trafodion uses the standard client side Hbase configuration file
> hbase-site.xml deployed by the distros. It is found that there are variations in
> this configuration file between distros. At times, the distro manager decides
> that a given property is not a client property and it is not added to the
> deployed hbase client configuration file. In addition, there are certain
> properties like hbase.coprocessor.region.classes need to be configured for
> Trafodion tables for the transaction management. Hence, I am planning to
> introduce a configuration file traf-site.xml specific to Trafodion similar to
> hbase configuration file hbase-site.xml. This configuration file extends the
> properties inherited from the standard hbase-site.xml.
>
> By default, the traf-site.xml comes with the following properties
>
> <?xml version="1.0"?>
> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>
> <!-- Put Trafodion site-specific property overrides in this file. -->
>
> <configuration>
> <property>
> <name>hbase.hregion.impl</name>
>
> <value>org.apache.hadoop.hbase.regionserver.transactional.TransactionalR
> egion</value>
> </property>
> <property>
> <name>hbase.coprocessor.region.classes</name>
> <value>
>
> org.apache.hadoop.hbase.coprocessor.transactional.TrxRegionObserver,
>
> org.apache.hadoop.hbase.coprocessor.transactional.TrxRegionEndpoint,
> org.apache.hadoop.hbase.coprocessor.AggregateImplementation
> </value>
> </property>
> <property>
> <name>hbase.client.scanner.timeout.period</name>
> <value>3600000</value>
> </property>
> </configuration>
>
> This property file will be installed if it doesn't exist at $MY_SQROOT/etc
> directory when sqgen is done. Any client side property can be added to this
> file and the client connections from Trafodion client processes will inherit
> them.
>
> The hbase.coprocessor.region.classes are added as Table co-processor to
> the table descriptor when a Trafodion table is created. Trafodion installer will
> not be adding to these properties to hbase configuration file soon.
>
> Please feel free to provide your valuable suggestions/comments.
>
> Thanks
> Selva
>
>
>
>
>
RE: [DISCUSS] Introducing Trafodion customization using its own
configuration file.
Posted by Hans Zeller <ha...@esgyn.com>.
Hi,
Sounds good to me, too. I wonder, however, whether we shouldn't prepend "trafodion" to the properties in the file. So, instead of hbase.client.scanner.timeout.period, for example, what about calling it trafodion.hbase.client.scanner.timeout.period. This should avoid confusion.
Thanks,
Hans
-----Original Message-----
From: Liu, Ming (Ming) [mailto:ming.liu@esgyn.cn]
Sent: Sunday, October 23, 2016 5:14 AM
To: dev@trafodion.incubator.apache.org
Subject: RE: [DISCUSS] Introducing Trafodion customization using its own configuration file.
I like this proposal very much. +1
Ming
-----Original Message-----
From: Selva Govindarajan [mailto:selva.govindarajan@esgyn.com]
Sent: Sunday, October 23, 2016 8:55 AM
To: dev@trafodion.incubator.apache.org
Subject: RE: [DISCUSS] Introducing Trafodion customization using its own configuration file.
With this change, there is no need to restart hbase if 'initialize trafodion' is done with this revision. You can also avoid restarting hbase if the table coprocessors are added manually to all Trafodion tables including the Trafodion meta data tables via hbase shell prior to installation of Trafodion release containing this change. However, I haven't tested it out to confirm it for sure.
Yes. We still need to copy the hbase-trx jar to hbase lib folder so that the region server processes can find the jar file containing the table coprocessor classes.
Currently, Trafodion modifies the ACL of some of the directories in hdfs. I believe, for these ACL to take effect Hadoop needs to be restarted and hence hbase also needs to be restarted.
Selva
-----Original Message-----
From: Zhang, Yi (Eason) [mailto:yi.zhang@esgyn.cn]
Sent: Saturday, October 22, 2016 12:12 AM
To: dev@trafodion.incubator.apache.org
Subject: Re: [DISCUSS] Introducing Trafodion customization using its own configuration file.
If using the new traf-site.ml config file:
1. Do we still need to restart HBase during installation?
2. Do we still need to copy hbase-trx jar to HBase lib folder?
Thanks,
Eason
On 22/10/2016, 10:16, "Selva Govindarajan" <se...@esgyn.com> wrote:
The properties added by the resource traf-site.xml will override the previously added resources, unless it was marked as final.
Selva
-----Original Message-----
From: Sean Broeder [mailto:sean.broeder@esgyn.com]
Sent: Friday, October 21, 2016 7:06 PM
To: dev@trafodion.incubator.apache.org
Subject: RE: [DISCUSS] Introducing Trafodion customization using its own configuration file.
I'm intrigued by this idea. Certainly there have been instances where the distros vary and that makes for difficult Trafodion installations.
What would the order of precedence be? How would it work if the standard hbase-site.xml file had a particular setting that Trafodion wanted to override? Or would HBase override the Trafodion settings? Would this create confusion if a user wanted to configure an HBase setting that Trafodion sets as well?
Thanks,
Sean
-----Original Message-----
From: Selva Govindarajan [mailto:selva.govindarajan@esgyn.com]
Sent: Friday, October 21, 2016 6:15 PM
To: dev@trafodion.incubator.apache.org
Subject: [DISCUSS] Introducing Trafodion customization using its own configuration file.
Currently Trafodion uses the standard client side Hbase configuration file hbase-site.xml deployed by the distros. It is found that there are variations in this configuration file between distros. At times, the distro manager decides that a given property is not a client property and it is not added to the deployed hbase client configuration file. In addition, there are certain properties like hbase.coprocessor.region.classes need to be configured for Trafodion tables for the transaction management. Hence, I am planning to introduce a configuration file traf-site.xml specific to Trafodion similar to hbase configuration file hbase-site.xml. This configuration file extends the properties inherited from the standard hbase-site.xml.
By default, the traf-site.xml comes with the following properties
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put Trafodion site-specific property overrides in this file. -->
<configuration>
<property>
<name>hbase.hregion.impl</name>
<value>org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegion</value>
</property>
<property>
<name>hbase.coprocessor.region.classes</name>
<value>
org.apache.hadoop.hbase.coprocessor.transactional.TrxRegionObserver,
org.apache.hadoop.hbase.coprocessor.transactional.TrxRegionEndpoint,
org.apache.hadoop.hbase.coprocessor.AggregateImplementation
</value>
</property>
<property>
<name>hbase.client.scanner.timeout.period</name>
<value>3600000</value>
</property>
</configuration>
This property file will be installed if it doesn't exist at $MY_SQROOT/etc directory when sqgen is done. Any client side property can be added to this file and the client connections from Trafodion client processes will inherit them.
The hbase.coprocessor.region.classes are added as Table co-processor to the table descriptor when a Trafodion table is created. Trafodion installer will not be adding to these properties to hbase configuration file soon.
Please feel free to provide your valuable suggestions/comments.
Thanks
Selva
RE: [DISCUSS] Introducing Trafodion customization using its own
configuration file.
Posted by "Liu, Ming (Ming)" <mi...@esgyn.cn>.
I like this proposal very much. +1
Ming
-----Original Message-----
From: Selva Govindarajan [mailto:selva.govindarajan@esgyn.com]
Sent: Sunday, October 23, 2016 8:55 AM
To: dev@trafodion.incubator.apache.org
Subject: RE: [DISCUSS] Introducing Trafodion customization using its own configuration file.
With this change, there is no need to restart hbase if 'initialize trafodion' is done with this revision. You can also avoid restarting hbase if the table coprocessors are added manually to all Trafodion tables including the Trafodion meta data tables via hbase shell prior to installation of Trafodion release containing this change. However, I haven't tested it out to confirm it for sure.
Yes. We still need to copy the hbase-trx jar to hbase lib folder so that the region server processes can find the jar file containing the table coprocessor classes.
Currently, Trafodion modifies the ACL of some of the directories in hdfs. I believe, for these ACL to take effect Hadoop needs to be restarted and hence hbase also needs to be restarted.
Selva
-----Original Message-----
From: Zhang, Yi (Eason) [mailto:yi.zhang@esgyn.cn]
Sent: Saturday, October 22, 2016 12:12 AM
To: dev@trafodion.incubator.apache.org
Subject: Re: [DISCUSS] Introducing Trafodion customization using its own configuration file.
If using the new traf-site.ml config file:
1. Do we still need to restart HBase during installation?
2. Do we still need to copy hbase-trx jar to HBase lib folder?
Thanks,
Eason
On 22/10/2016, 10:16, "Selva Govindarajan" <se...@esgyn.com> wrote:
The properties added by the resource traf-site.xml will override the previously added resources, unless it was marked as final.
Selva
-----Original Message-----
From: Sean Broeder [mailto:sean.broeder@esgyn.com]
Sent: Friday, October 21, 2016 7:06 PM
To: dev@trafodion.incubator.apache.org
Subject: RE: [DISCUSS] Introducing Trafodion customization using its own configuration file.
I'm intrigued by this idea. Certainly there have been instances where the distros vary and that makes for difficult Trafodion installations.
What would the order of precedence be? How would it work if the standard hbase-site.xml file had a particular setting that Trafodion wanted to override? Or would HBase override the Trafodion settings? Would this create confusion if a user wanted to configure an HBase setting that Trafodion sets as well?
Thanks,
Sean
-----Original Message-----
From: Selva Govindarajan [mailto:selva.govindarajan@esgyn.com]
Sent: Friday, October 21, 2016 6:15 PM
To: dev@trafodion.incubator.apache.org
Subject: [DISCUSS] Introducing Trafodion customization using its own configuration file.
Currently Trafodion uses the standard client side Hbase configuration file hbase-site.xml deployed by the distros. It is found that there are variations in this configuration file between distros. At times, the distro manager decides that a given property is not a client property and it is not added to the deployed hbase client configuration file. In addition, there are certain properties like hbase.coprocessor.region.classes need to be configured for Trafodion tables for the transaction management. Hence, I am planning to introduce a configuration file traf-site.xml specific to Trafodion similar to hbase configuration file hbase-site.xml. This configuration file extends the properties inherited from the standard hbase-site.xml.
By default, the traf-site.xml comes with the following properties
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put Trafodion site-specific property overrides in this file. -->
<configuration>
<property>
<name>hbase.hregion.impl</name>
<value>org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegion</value>
</property>
<property>
<name>hbase.coprocessor.region.classes</name>
<value>
org.apache.hadoop.hbase.coprocessor.transactional.TrxRegionObserver,
org.apache.hadoop.hbase.coprocessor.transactional.TrxRegionEndpoint,
org.apache.hadoop.hbase.coprocessor.AggregateImplementation
</value>
</property>
<property>
<name>hbase.client.scanner.timeout.period</name>
<value>3600000</value>
</property>
</configuration>
This property file will be installed if it doesn't exist at $MY_SQROOT/etc directory when sqgen is done. Any client side property can be added to this file and the client connections from Trafodion client processes will inherit them.
The hbase.coprocessor.region.classes are added as Table co-processor to the table descriptor when a Trafodion table is created. Trafodion installer will not be adding to these properties to hbase configuration file soon.
Please feel free to provide your valuable suggestions/comments.
Thanks
Selva
RE: [DISCUSS] Introducing Trafodion customization using its own
configuration file.
Posted by Selva Govindarajan <se...@esgyn.com>.
With this change, there is no need to restart hbase if 'initialize trafodion' is done with this revision. You can also avoid restarting hbase if the table coprocessors are added manually to all Trafodion tables including the Trafodion meta data tables via hbase shell prior to installation of Trafodion release containing this change. However, I haven't tested it out to confirm it for sure.
Yes. We still need to copy the hbase-trx jar to hbase lib folder so that the region server processes can find the jar file containing the table coprocessor classes.
Currently, Trafodion modifies the ACL of some of the directories in hdfs. I believe, for these ACL to take effect Hadoop needs to be restarted and hence hbase also needs to be restarted.
Selva
-----Original Message-----
From: Zhang, Yi (Eason) [mailto:yi.zhang@esgyn.cn]
Sent: Saturday, October 22, 2016 12:12 AM
To: dev@trafodion.incubator.apache.org
Subject: Re: [DISCUSS] Introducing Trafodion customization using its own configuration file.
If using the new traf-site.ml config file:
1. Do we still need to restart HBase during installation?
2. Do we still need to copy hbase-trx jar to HBase lib folder?
Thanks,
Eason
On 22/10/2016, 10:16, "Selva Govindarajan" <se...@esgyn.com> wrote:
The properties added by the resource traf-site.xml will override the previously added resources, unless it was marked as final.
Selva
-----Original Message-----
From: Sean Broeder [mailto:sean.broeder@esgyn.com]
Sent: Friday, October 21, 2016 7:06 PM
To: dev@trafodion.incubator.apache.org
Subject: RE: [DISCUSS] Introducing Trafodion customization using its own configuration file.
I'm intrigued by this idea. Certainly there have been instances where the distros vary and that makes for difficult Trafodion installations.
What would the order of precedence be? How would it work if the standard hbase-site.xml file had a particular setting that Trafodion wanted to override? Or would HBase override the Trafodion settings? Would this create confusion if a user wanted to configure an HBase setting that Trafodion sets as well?
Thanks,
Sean
-----Original Message-----
From: Selva Govindarajan [mailto:selva.govindarajan@esgyn.com]
Sent: Friday, October 21, 2016 6:15 PM
To: dev@trafodion.incubator.apache.org
Subject: [DISCUSS] Introducing Trafodion customization using its own configuration file.
Currently Trafodion uses the standard client side Hbase configuration file hbase-site.xml deployed by the distros. It is found that there are variations in this configuration file between distros. At times, the distro manager decides that a given property is not a client property and it is not added to the deployed hbase client configuration file. In addition, there are certain properties like hbase.coprocessor.region.classes need to be configured for Trafodion tables for the transaction management. Hence, I am planning to introduce a configuration file traf-site.xml specific to Trafodion similar to hbase configuration file hbase-site.xml. This configuration file extends the properties inherited from the standard hbase-site.xml.
By default, the traf-site.xml comes with the following properties
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put Trafodion site-specific property overrides in this file. -->
<configuration>
<property>
<name>hbase.hregion.impl</name>
<value>org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegion</value>
</property>
<property>
<name>hbase.coprocessor.region.classes</name>
<value>
org.apache.hadoop.hbase.coprocessor.transactional.TrxRegionObserver,
org.apache.hadoop.hbase.coprocessor.transactional.TrxRegionEndpoint,
org.apache.hadoop.hbase.coprocessor.AggregateImplementation
</value>
</property>
<property>
<name>hbase.client.scanner.timeout.period</name>
<value>3600000</value>
</property>
</configuration>
This property file will be installed if it doesn't exist at $MY_SQROOT/etc directory when sqgen is done. Any client side property can be added to this file and the client connections from Trafodion client processes will inherit them.
The hbase.coprocessor.region.classes are added as Table co-processor to the table descriptor when a Trafodion table is created. Trafodion installer will not be adding to these properties to hbase configuration file soon.
Please feel free to provide your valuable suggestions/comments.
Thanks
Selva
Re: [DISCUSS] Introducing Trafodion customization using its own
configuration file.
Posted by "Zhang, Yi (Eason)" <yi...@esgyn.cn>.
If using the new traf-site.ml config file:
1. Do we still need to restart HBase during installation?
2. Do we still need to copy hbase-trx jar to HBase lib folder?
Thanks,
Eason
On 22/10/2016, 10:16, "Selva Govindarajan" <se...@esgyn.com> wrote:
The properties added by the resource traf-site.xml will override the previously added resources, unless it was marked as final.
Selva
-----Original Message-----
From: Sean Broeder [mailto:sean.broeder@esgyn.com]
Sent: Friday, October 21, 2016 7:06 PM
To: dev@trafodion.incubator.apache.org
Subject: RE: [DISCUSS] Introducing Trafodion customization using its own configuration file.
I'm intrigued by this idea. Certainly there have been instances where the distros vary and that makes for difficult Trafodion installations.
What would the order of precedence be? How would it work if the standard hbase-site.xml file had a particular setting that Trafodion wanted to override? Or would HBase override the Trafodion settings? Would this create confusion if a user wanted to configure an HBase setting that Trafodion sets as well?
Thanks,
Sean
-----Original Message-----
From: Selva Govindarajan [mailto:selva.govindarajan@esgyn.com]
Sent: Friday, October 21, 2016 6:15 PM
To: dev@trafodion.incubator.apache.org
Subject: [DISCUSS] Introducing Trafodion customization using its own configuration file.
Currently Trafodion uses the standard client side Hbase configuration file hbase-site.xml deployed by the distros. It is found that there are variations in this configuration file between distros. At times, the distro manager decides that a given property is not a client property and it is not added to the deployed hbase client configuration file. In addition, there are certain properties like hbase.coprocessor.region.classes need to be configured for Trafodion tables for the transaction management. Hence, I am planning to introduce a configuration file traf-site.xml specific to Trafodion similar to hbase configuration file hbase-site.xml. This configuration file extends the properties inherited from the standard hbase-site.xml.
By default, the traf-site.xml comes with the following properties
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put Trafodion site-specific property overrides in this file. -->
<configuration>
<property>
<name>hbase.hregion.impl</name>
<value>org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegion</value>
</property>
<property>
<name>hbase.coprocessor.region.classes</name>
<value>
org.apache.hadoop.hbase.coprocessor.transactional.TrxRegionObserver,
org.apache.hadoop.hbase.coprocessor.transactional.TrxRegionEndpoint,
org.apache.hadoop.hbase.coprocessor.AggregateImplementation
</value>
</property>
<property>
<name>hbase.client.scanner.timeout.period</name>
<value>3600000</value>
</property>
</configuration>
This property file will be installed if it doesn't exist at $MY_SQROOT/etc directory when sqgen is done. Any client side property can be added to this file and the client connections from Trafodion client processes will inherit them.
The hbase.coprocessor.region.classes are added as Table co-processor to the table descriptor when a Trafodion table is created. Trafodion installer will not be adding to these properties to hbase configuration file soon.
Please feel free to provide your valuable suggestions/comments.
Thanks
Selva
RE: [DISCUSS] Introducing Trafodion customization using its own
configuration file.
Posted by Selva Govindarajan <se...@esgyn.com>.
The properties added by the resource traf-site.xml will override the previously added resources, unless it was marked as final.
Selva
-----Original Message-----
From: Sean Broeder [mailto:sean.broeder@esgyn.com]
Sent: Friday, October 21, 2016 7:06 PM
To: dev@trafodion.incubator.apache.org
Subject: RE: [DISCUSS] Introducing Trafodion customization using its own configuration file.
I'm intrigued by this idea. Certainly there have been instances where the distros vary and that makes for difficult Trafodion installations.
What would the order of precedence be? How would it work if the standard hbase-site.xml file had a particular setting that Trafodion wanted to override? Or would HBase override the Trafodion settings? Would this create confusion if a user wanted to configure an HBase setting that Trafodion sets as well?
Thanks,
Sean
-----Original Message-----
From: Selva Govindarajan [mailto:selva.govindarajan@esgyn.com]
Sent: Friday, October 21, 2016 6:15 PM
To: dev@trafodion.incubator.apache.org
Subject: [DISCUSS] Introducing Trafodion customization using its own configuration file.
Currently Trafodion uses the standard client side Hbase configuration file hbase-site.xml deployed by the distros. It is found that there are variations in this configuration file between distros. At times, the distro manager decides that a given property is not a client property and it is not added to the deployed hbase client configuration file. In addition, there are certain properties like hbase.coprocessor.region.classes need to be configured for Trafodion tables for the transaction management. Hence, I am planning to introduce a configuration file traf-site.xml specific to Trafodion similar to hbase configuration file hbase-site.xml. This configuration file extends the properties inherited from the standard hbase-site.xml.
By default, the traf-site.xml comes with the following properties
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put Trafodion site-specific property overrides in this file. -->
<configuration>
<property>
<name>hbase.hregion.impl</name>
<value>org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegion</value>
</property>
<property>
<name>hbase.coprocessor.region.classes</name>
<value>
org.apache.hadoop.hbase.coprocessor.transactional.TrxRegionObserver,
org.apache.hadoop.hbase.coprocessor.transactional.TrxRegionEndpoint,
org.apache.hadoop.hbase.coprocessor.AggregateImplementation
</value>
</property>
<property>
<name>hbase.client.scanner.timeout.period</name>
<value>3600000</value>
</property>
</configuration>
This property file will be installed if it doesn't exist at $MY_SQROOT/etc directory when sqgen is done. Any client side property can be added to this file and the client connections from Trafodion client processes will inherit them.
The hbase.coprocessor.region.classes are added as Table co-processor to the table descriptor when a Trafodion table is created. Trafodion installer will not be adding to these properties to hbase configuration file soon.
Please feel free to provide your valuable suggestions/comments.
Thanks
Selva
RE: [DISCUSS] Introducing Trafodion customization using its own
configuration file.
Posted by Sean Broeder <se...@esgyn.com>.
I'm intrigued by this idea. Certainly there have been instances where the distros vary and that makes for difficult Trafodion installations.
What would the order of precedence be? How would it work if the standard hbase-site.xml file had a particular setting that Trafodion wanted to override? Or would HBase override the Trafodion settings? Would this create confusion if a user wanted to configure an HBase setting that Trafodion sets as well?
Thanks,
Sean
-----Original Message-----
From: Selva Govindarajan [mailto:selva.govindarajan@esgyn.com]
Sent: Friday, October 21, 2016 6:15 PM
To: dev@trafodion.incubator.apache.org
Subject: [DISCUSS] Introducing Trafodion customization using its own configuration file.
Currently Trafodion uses the standard client side Hbase configuration file hbase-site.xml deployed by the distros. It is found that there are variations in this configuration file between distros. At times, the distro manager decides that a given property is not a client property and it is not added to the deployed hbase client configuration file. In addition, there are certain properties like hbase.coprocessor.region.classes need to be configured for Trafodion tables for the transaction management. Hence, I am planning to introduce a configuration file traf-site.xml specific to Trafodion similar to hbase configuration file hbase-site.xml. This configuration file extends the properties inherited from the standard hbase-site.xml.
By default, the traf-site.xml comes with the following properties
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put Trafodion site-specific property overrides in this file. -->
<configuration>
<property>
<name>hbase.hregion.impl</name>
<value>org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegion</value>
</property>
<property>
<name>hbase.coprocessor.region.classes</name>
<value>
org.apache.hadoop.hbase.coprocessor.transactional.TrxRegionObserver,
org.apache.hadoop.hbase.coprocessor.transactional.TrxRegionEndpoint,
org.apache.hadoop.hbase.coprocessor.AggregateImplementation
</value>
</property>
<property>
<name>hbase.client.scanner.timeout.period</name>
<value>3600000</value>
</property>
</configuration>
This property file will be installed if it doesn't exist at $MY_SQROOT/etc directory when sqgen is done. Any client side property can be added to this file and the client connections from Trafodion client processes will inherit them.
The hbase.coprocessor.region.classes are added as Table co-processor to the table descriptor when a Trafodion table is created. Trafodion installer will not be adding to these properties to hbase configuration file soon.
Please feel free to provide your valuable suggestions/comments.
Thanks
Selva