You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@hawq.apache.org by lisakowen <gi...@git.apache.org> on 2016/11/10 18:15:28 UTC

[GitHub] incubator-hawq-docs pull request #60: HAWQ-1151 - add ambari procedures

GitHub user lisakowen opened a pull request:

    https://github.com/apache/incubator-hawq-docs/pull/60

    HAWQ-1151 - add ambari procedures

    update docs to identify specific ambari procedures when modifying server configuration parameters.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/lisakowen/incubator-hawq-docs feature/hawqcfg-ambari

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-hawq-docs/pull/60.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #60
    
----
commit 2964b845ca1d9264aab2921dd8d94a7ace3bc745
Author: Lisa Owen <lo...@pivotal.io>
Date:   2016-11-10T17:48:19Z

    add ambari procedures for setting pljava* cfg params

commit 546535f61b654a28d84af7d48e8c6b739c98d48d
Author: Lisa Owen <lo...@pivotal.io>
Date:   2016-11-10T17:53:56Z

    use => for SQL prompt

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq-docs pull request #60: HAWQ-1151 - add ambari procedures

Posted by dyozie <gi...@git.apache.org>.

Github user dyozie commented on a diff in the pull request:

https://github.com/apache/incubator-hawq-docs/pull/60#discussion_r87920277

--- Diff: clientaccess/client_auth.html.md.erb ---
@@ -101,51 +101,45 @@ For example, in `$GPHOME/etc/hawq-site.xml`:
</property>
```

-The following steps set the parameter values with the HAWQ utility `hawq config`.
-
-### <a id="ip142411"></a>To change the number of allowed connections
+**Note:** Raising the values of these parameters may cause HAWQ to request more shared memory. To mitigate this effect, consider decreasing other memory-related server configuration parameters such as [gp\_cached\_segworkers\_threshold](../reference/guc/parameter_definitions.html#gp_cached_segworkers_threshold).

-1. Log into the HAWQ master host as the HAWQ administrator and source the file `/usr/local/hawq/greenplum_path.sh`.
-2. Set the value of the `max_connections` parameter. This `hawq config` command sets the value to 100 on all HAWQ instances.

- ``` bash
- $ hawq config -c max_connections -v 100
- ```
+### <a id="ip142411"></a>Setting the number of allowed connections

- This configuration command sets the value for the master.
+You will perform different procedures to set connection-related server configuration parameters for your HAWQ cluster depending upon whether you manage your cluster from the command line or use Ambari. If you use Ambari to manage your HAWQ cluster, you must ensure that you update server configuration parameters via the Ambari Web UI only. If you manage your HAWQ cluster from the command line, you will use the `hawq config` command line utility to set server configuration parameters.

-3. Set the value of the `seg_max_connections` parameter. This `hawq config` command sets the value to 6400 on all HAWQ instances.
+If you use Ambari to manage your cluster:

- ``` bash
- $ hawq config -c seg_max_connections -v 6400
- ```
+1. Set the `max_connections`, `seg_max_connections`, and `max_prepared_transactions` configuration properties via the HAWQ service **Configs > Advanced > Custom hawq-site** drop down, providing values appropriate for your HAWQ deployment.
--- End diff --

I think I'd remove ", providing values appropriate for your HAWQ deployment" here.

---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq-docs pull request #60: HAWQ-1151 - add ambari procedures

Posted by dyozie <gi...@git.apache.org>.

Github user dyozie commented on a diff in the pull request:

https://github.com/apache/incubator-hawq-docs/pull/60#discussion_r87920362

--- Diff: clientaccess/client_auth.html.md.erb ---
@@ -101,51 +101,45 @@ For example, in `$GPHOME/etc/hawq-site.xml`:
</property>
```

- ``` bash
- $ hawq config -c max_connections -v 100
- ```
+### <a id="ip142411"></a>Setting the number of allowed connections

-3. Set the value of the `seg_max_connections` parameter. This `hawq config` command sets the value to 6400 on all HAWQ instances.
+If you use Ambari to manage your cluster:

- This configuration command sets the value for all segment instances.
+If you manage your cluster from the command line:

-4. Set the value of the `max_prepared_transactions` parameter. This `hawq config` command sets the value to 200 for the master and all segments.
+1. Log in to the HAWQ master host as the HAWQ administrator and source the file `/usr/local/hawq/greenplum_path.sh`.
--- End diff --

Change "the HAWQ Administrator" to "a HAWQ Administrator" ?

[GitHub] incubator-hawq-docs pull request #60: HAWQ-1151 - add ambari procedures

Posted by dyozie <gi...@git.apache.org>.

Github user dyozie commented on a diff in the pull request:

    https://github.com/apache/incubator-hawq-docs/pull/60#discussion_r87922594
  
    --- Diff: ddl/ddl-table.html.md.erb ---
    @@ -132,7 +132,7 @@ Policies for different application scenarios can be specified to optimize perfor
     
     Adjusting the values of the configuration parameters `default_hash_table_bucket_number` and `hawq_rm_nvseg_perquery_limit` can tune performance by controlling the number of virtual segments being used. In most circumstances, HAWQ's elastic runtime will dynamically allocate virtual segments to optimize performance, so further tuning should not be needed..
     
    -Hash tables are created using the value specified in `default_hash_table_bucket_number`. Explicitly setting this value can be useful in managing resources, as queries for hash tables use a fixed number of buckets, regardless of the amount of data present. If a larger or smaller number of hash buckets are desired, set this value before you CREATE TABLE. Resources are dynamically allocated to a multiple of the number of nodes. If setting the value of `default_hash_table_bucket_number` with hawq init --bucket\_number, the value should not exceed the value of `hawq_rm_nvseg_perquery_limit`, which defines the maximum number of virtual segments that can be used for a query \(default = 512, with a maximum of 65535\). Modifying the value to greater than 1000 segments is not recommended.
    +Hash tables are created using the value specified in `default_hash_table_bucket_number`. Explicitly setting this value can be useful in managing resources, as queries for hash tables use a fixed number of buckets, regardless of the amount of data present. If a larger or smaller number of hash buckets are desired, set this value before you CREATE TABLE. Resources are dynamically allocated to a multiple of the number of nodes. If using `hawq init --bucket_number` to set the value of `default_hash_table_bucket_number` during cluster initialization or expansion, the value should not exceed the value of `hawq_rm_nvseg_perquery_limit`, which defines the maximum number of virtual segments that can be used for a query \(default = 512, with a maximum of 65535\). Modifying the value to greater than 1000 segments is not recommended.
    --- End diff --
    
    change "CREATE TABLE" to "create tables"  Also might want to tackle some of the passive voice in this paragraph.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq-docs pull request #60: HAWQ-1151 - add ambari procedures

Posted by lisakowen <gi...@git.apache.org>.

Github user lisakowen closed the pull request at:

    https://github.com/apache/incubator-hawq-docs/pull/60


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq-docs pull request #60: HAWQ-1151 - add ambari procedures

Posted by dyozie <gi...@git.apache.org>.

Github user dyozie commented on a diff in the pull request:

https://github.com/apache/incubator-hawq-docs/pull/60#discussion_r87922081

--- Diff: ddl/ddl-table.html.md.erb ---
@@ -93,14 +93,14 @@ For any specific query, the first four factors are fixed values, while the confi

The `bucketnum` for a hash table specifies the number of hash buckets to be used in creating virtual segments. A HASH distributed table is created with `default_hash_table_bucket_number` buckets. The default bucket value can be changed in session level or in the `CREATE TABLE` DDL by using the `bucketnum` storage parameter.

-When initializing a cluster, you can use the `hawq init --bucket_number` parameter to explcitly set the default bucket number $`default_hash_table_bucket_number`$.
+In an Ambari-managed HAWQ cluster, the default bucket number $`default_hash_table_bucket_number`$ is derived from the number of segment nodes. In command-line-managed HAWQ environments, you can use the `--bucket_number` option of `hawq init` to explicitly set `default_hash_table_bucket_number` during cluster initialization.

-**Note:** For best performance with large tables, the number of buckets should not exceed the value of the `default_hash_table_bucket_number` parameter. Small tables can use one segment node, `with bucketnum=1`. For larger tables, the bucketnum is set to a multiple of the number of segment nodes, for the best load balancing on different segment nodes. The elastic runtime will attempt to find the optimal number of buckets for the number of nodes being processed. Larger tables need more virtual segments , and hence use larger numbers of buckets.
+**Note:** For best performance with large tables, the number of buckets should not exceed the value of the `default_hash_table_bucket_number` parameter. Small tables can use one segment node, `WITH bucketnum=1`. For larger tables, the `bucketnum` is set to a multiple of the number of segment nodes, for the best load balancing on different segment nodes. The elastic runtime will attempt to find the optimal number of buckets for the number of nodes being processed. Larger tables need more virtual segments , and hence use larger numbers of buckets.
--- End diff --

Might as well fix (remove) the space before the comma here.

[GitHub] incubator-hawq-docs pull request #60: HAWQ-1151 - add ambari procedures

Posted by dyozie <gi...@git.apache.org>.

Github user dyozie commented on a diff in the pull request:

https://github.com/apache/incubator-hawq-docs/pull/60#discussion_r87921329

--- Diff: clientaccess/roles_privs.html.md.erb ---
@@ -159,47 +159,44 @@ You can set your chosen encryption method system-wide or on a per-session basis.

#### <a id="topic11"></a>System-wide

-To set the `password_hash_algorithm` server parameter on a complete HAWQ system $master and its segments$:
+You will perform different procedures to set the encryption method system-wide depending upon whether you manage your cluster from the command line or use Ambari. If you use Ambari to manage your HAWQ cluster, you must ensure that you update encryption method configuration parameters via the Ambari Web UI only. If you manage your HAWQ cluster from the command line, you will use the `hawq config` command line utility to set encryption method configuration parameters.

-1. Log into your HAWQ instance as a superuser.
-2. Execute `hawq config` with the `password_hash_algorithm` set to `SHA-256` $or `SHA-256-FIPS` to use the FIPS-compliant libraries for SHA-256$
+To set the `password_hash_algorithm` server parameter on all nodes in your HAWQ cluster $master and its segments$:

- ``` bash
- $ hawq config -c password_hash_algorithm -v 'SHA-256'
- ```
-
- or:
-
- ``` bash
- $ hawq config -c password_hash_algorithm -v 'SHA-256-FIPS'
- ```
+1. If you use Ambari to manage your HAWQ cluster:

-3. Verify the setting:
-
- ``` bash
- $ hawq config -s
- ```
+ 1. Set the `password_hash_algorithm` configuration property via the HAWQ service **Configs > Advanced > Custom hawq-site** drop down. Valid values include `SHA-256` $or `SHA-256-FIPS` to use the FIPS-compliant libraries for SHA-256$.
+ 2. Select **Service Actions > Restart All** to load the updated configuration.
+2. If you manage your HAWQ cluster from the command line:
--- End diff --

Same here - make this a plain paragraph rather than a numbered step.

[GitHub] incubator-hawq-docs pull request #60: HAWQ-1151 - add ambari procedures

Posted by dyozie <gi...@git.apache.org>.

Github user dyozie commented on a diff in the pull request:

https://github.com/apache/incubator-hawq-docs/pull/60#discussion_r87921640

--- Diff: datamgmt/about_statistics.html.md.erb ---
@@ -142,11 +142,28 @@ There are several options for configuring HAWQ statistics collection.

The statistics target is the size of the `most_common_vals`, `most_common_freqs`, and `histogram_bounds` arrays for an individual column. By default, the target is 25. The default target can be changed by setting a server configuration parameter and the target can be set for any column using the `ALTER TABLE` command. Larger values increase the time needed to do `ANALYZE`, but may improve the quality of the legacy query optimizer (planner) estimates.

-Set the system default statistics target to a different value by setting the `default_statistics_target` server configuration parameter. The default value is usually sufficient, and you should only raise or lower it if your tests demonstrate that query plans improve with the new target. For example, to raise the default statistics target from 25 to 50 you can use the `hawq config` utility:
+Set the system default statistics target to a different value by setting the `default_statistics_target` server configuration parameter. The default value is usually sufficient, and you should only raise or lower it if your tests demonstrate that query plans improve with the new target.

Change to read "The following examples show how to raise the default statistics target from 25 to 50."

[GitHub] incubator-hawq-docs pull request #60: HAWQ-1151 - add ambari procedures

Posted by dyozie <gi...@git.apache.org>.

Github user dyozie commented on a diff in the pull request:

    https://github.com/apache/incubator-hawq-docs/pull/60#discussion_r87920864
  
    --- Diff: clientaccess/roles_privs.html.md.erb ---
    @@ -159,47 +159,44 @@ You can set your chosen encryption method system-wide or on a per-session basis.
     
     #### <a id="topic11"></a>System-wide 
     
    -To set the `password_hash_algorithm` server parameter on a complete HAWQ system \(master and its segments\):
    +You will perform different procedures to set the encryption method system-wide depending upon whether you manage your cluster from the command line or use Ambari. If you use Ambari to manage your HAWQ cluster, you must ensure that you update encryption method configuration parameters via the Ambari Web UI only. If you manage your HAWQ cluster from the command line, you will use the `hawq config` command line utility to set encryption method configuration parameters.
    --- End diff --
    
    See previous edit to this.  Also, change "You will perform different procedures to set the encryption method" to "You will perform different procedures to set the encryption method (`password_hash_algorithm` server parameter)"


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq-docs pull request #60: HAWQ-1151 - add ambari procedures

Posted by dyozie <gi...@git.apache.org>.

Github user dyozie commented on a diff in the pull request:

https://github.com/apache/incubator-hawq-docs/pull/60#discussion_r87920560

--- Diff: clientaccess/client_auth.html.md.erb ---
@@ -101,51 +101,45 @@ For example, in `$GPHOME/etc/hawq-site.xml`:
</property>
```

- ``` bash
- $ hawq config -c max_connections -v 100
- ```
+### <a id="ip142411"></a>Setting the number of allowed connections

-3. Set the value of the `seg_max_connections` parameter. This `hawq config` command sets the value to 6400 on all HAWQ instances.
+If you use Ambari to manage your cluster:

- This configuration command sets the value for all segment instances.
+If you manage your cluster from the command line:

Also, seems like sourcing the env script should be called out as a bash command block.

[GitHub] incubator-hawq-docs pull request #60: HAWQ-1151 - add ambari procedures

Posted by dyozie <gi...@git.apache.org>.

Github user dyozie commented on a diff in the pull request:

https://github.com/apache/incubator-hawq-docs/pull/60#discussion_r87921679

--- Diff: datamgmt/about_statistics.html.md.erb ---
@@ -142,11 +142,28 @@ There are several options for configuring HAWQ statistics collection.

-``` shell
-$ hawq config -c default_statistics_target -v 50
-```
+You will perform different procedures to set server configuration parameters for your whole HAWQ cluster depending upon whether you manage your cluster from the command line or use Ambari. If you use Ambari to manage your HAWQ cluster, you must ensure that you update server configuration parameters via the Ambari Web UI only. If you manage your HAWQ cluster from the command line, you will use the `hawq config` command line utility to set server configuration parameters.
+
+For example, to raise the default statistics target from 25 to 50:
+
+1. If you use Ambari to manage your HAWQ cluster:
+
+ 1. Set the `default_statistics_target` configuration property to `50` via the HAWQ service **Configs > Advanced > Custom hawq-site** drop down.
+ 2. Select **Service Actions > Restart All** to load the updated configuration.
+2. If you manage your HAWQ cluster from the command line:
--- End diff --

Demote to regular para

[GitHub] incubator-hawq-docs pull request #60: HAWQ-1151 - add ambari procedures

Posted by dyozie <gi...@git.apache.org>.

Github user dyozie commented on a diff in the pull request:

https://github.com/apache/incubator-hawq-docs/pull/60#discussion_r87921265

--- Diff: clientaccess/roles_privs.html.md.erb ---
@@ -159,47 +159,44 @@ You can set your chosen encryption method system-wide or on a per-session basis.

#### <a id="topic11"></a>System-wide

I think it's better to stay consistent with previous file's edits, and make this a plain paragraph rather than a numbered step.

[GitHub] incubator-hawq-docs pull request #60: HAWQ-1151 - add ambari procedures

Posted by dyozie <gi...@git.apache.org>.

Github user dyozie commented on a diff in the pull request:

https://github.com/apache/incubator-hawq-docs/pull/60#discussion_r87920132

--- Diff: clientaccess/client_auth.html.md.erb ---
@@ -101,51 +101,45 @@ For example, in `$GPHOME/etc/hawq-site.xml`:
</property>
```

- ``` bash
- $ hawq config -c max_connections -v 100
- ```
+### <a id="ip142411"></a>Setting the number of allowed connections

change "via the Ambari Web UI only" -> "only via the Ambari Web UI"

[GitHub] incubator-hawq-docs pull request #60: HAWQ-1151 - add ambari procedures

Posted by dyozie <gi...@git.apache.org>.

Github user dyozie commented on a diff in the pull request:

https://github.com/apache/incubator-hawq-docs/pull/60#discussion_r87921664

--- Diff: datamgmt/about_statistics.html.md.erb ---
@@ -142,11 +142,28 @@ There are several options for configuring HAWQ statistics collection.

Demote to regular para

[GitHub] incubator-hawq-docs pull request #60: HAWQ-1151 - add ambari procedures

Posted by dyozie <gi...@git.apache.org>.

Github user dyozie commented on a diff in the pull request:

    https://github.com/apache/incubator-hawq-docs/pull/60#discussion_r87922958
  
    --- Diff: plext/using_pljava.html.md.erb ---
    @@ -77,6 +77,41 @@ If TRUE, lingering savepoints will be released on function exit. If FALSE, they
     
     Defines the start up options for the Java VM.
     
    +### <a id="setting_serverconfigparams"></a>Setting PL/Java Configuration Parameters 
    +
    +You can set PL/Java server configuration parameters at the session level, or globally across your whole cluster. Your HAWQ cluster configuration must be reloaded after setting a server configuration value globally.
    --- End diff --
    
    See previous edits.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq-docs pull request #60: HAWQ-1151 - add ambari procedures

Posted by dyozie <gi...@git.apache.org>.

Github user dyozie commented on a diff in the pull request:

https://github.com/apache/incubator-hawq-docs/pull/60#discussion_r87921176

--- Diff: clientaccess/roles_privs.html.md.erb ---
@@ -159,47 +159,44 @@ You can set your chosen encryption method system-wide or on a per-session basis.

#### <a id="topic11"></a>System-wide

Remove this line.

[GitHub] incubator-hawq-docs pull request #60: HAWQ-1151 - add ambari procedures

Posted by dyozie <gi...@git.apache.org>.

Github user dyozie commented on a diff in the pull request:

https://github.com/apache/incubator-hawq-docs/pull/60#discussion_r87923659

--- Diff: query/gporca/query-gporca-enable.html.md.erb ---
@@ -20,35 +20,45 @@ As part of routine database maintenance, you should refresh statistics on the ro

When the configuration parameter `optimizer_analyze_root_partition` is set to `on`, root partition statistics will be collected when `ANALYZE` is run on a partitioned table. Root partition statistics are required by GPORCA.

-1. Log into the HAWQ master host as `gpadmin`, the HAWQ administrator.
-2. Set the values of the server configuration parameters. These HAWQ `hawq config` utility commands sets the value of the parameters to `on`:
+You will perform different procedures to set optimizer configuration parameters for your whole HAWQ cluster depending upon whether you manage your cluster from the command line or use Ambari. If you use Ambari to manage your HAWQ cluster, you must ensure that you update server configuration parameters via the Ambari Web UI only. If you manage your HAWQ cluster from the command line, you will use the `hawq config` command line utility to set optimizer server configuration parameters.
--- End diff --

See previous edits.

[GitHub] incubator-hawq-docs pull request #60: HAWQ-1151 - add ambari procedures

Posted by dyozie <gi...@git.apache.org>.

Github user dyozie commented on a diff in the pull request:

    https://github.com/apache/incubator-hawq-docs/pull/60#discussion_r87923400
  
    --- Diff: plext/using_pljava.html.md.erb ---
    @@ -179,15 +203,25 @@ Replace \<dbname\> with the name of the target database.
     
     When no databases have PL/Java as a registered language, remove the Java JAR files:
     
    -1. Remove the `pljava_classpath` server configuration parameter in the `hawq-site.xml` file.
    +1. Remove the `pljava_classpath` server configuration parameter:
    --- End diff --
    
    Better to make following procedure two separate procs, one for ambari and one cli.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---