You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@trafodion.apache.org by li...@apache.org on 2017/03/23 05:15:25 UTC
[1/4] incubator-trafodion git commit: add scratch_disks_CQD
Repository: incubator-trafodion
Updated Branches:
refs/heads/master c978ab556 -> 299b538b6
add scratch_disks_CQD
Project: http://git-wip-us.apache.org/repos/asf/incubator-trafodion/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-trafodion/commit/fb6c16e8
Tree: http://git-wip-us.apache.org/repos/asf/incubator-trafodion/tree/fb6c16e8
Diff: http://git-wip-us.apache.org/repos/asf/incubator-trafodion/diff/fb6c16e8
Branch: refs/heads/master
Commit: fb6c16e89679b23879bd62efac5bfc9426286340
Parents: d15b5b6
Author: liu.yu <yu...@esgyn.cn>
Authored: Tue Mar 7 19:04:56 2017 +0800
Committer: liu.yu <yu...@esgyn.cn>
Committed: Tue Mar 7 19:04:56 2017 +0800
----------------------------------------------------------------------
.../src/asciidoc/_chapters/query_execution.adoc | 44 +++++++++++++++-----
1 file changed, 34 insertions(+), 10 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/fb6c16e8/docs/cqd_reference/src/asciidoc/_chapters/query_execution.adoc
----------------------------------------------------------------------
diff --git a/docs/cqd_reference/src/asciidoc/_chapters/query_execution.adoc b/docs/cqd_reference/src/asciidoc/_chapters/query_execution.adoc
index 91d390a..8f21152 100644
--- a/docs/cqd_reference/src/asciidoc/_chapters/query_execution.adoc
+++ b/docs/cqd_reference/src/asciidoc/_chapters/query_execution.adoc
@@ -33,14 +33,14 @@ This section describes CQDs that are used to influence query execution.
[cols="25%h,75%"]
|===
-| *Description* | Allows index maintenance to be performed concurrently with base table operation.
+| *Description* | Allows index maintenance to be performed concurrently with base table operation.
| *Values* | *'ON'* Index maintenance is allowed. +
*'OFF'* Index maintenance is not allowed. +
+
The default value is *'ON'*.
-| *Usage* | HBase `put` operations are blocking. When the table has one or more indexes,
-then the insert/update/delete (IUD) operation response time is improved by executing the index maintenance
-operations concurrently with the base table operation: the put operations to these HBase tables are executed
+| *Usage* | HBase `put` operations are blocking. When the table has one or more indexes,
+then the insert/update/delete (IUD) operation response time is improved by executing the index maintenance
+operations concurrently with the base table operation: the put operations to these HBase tables are executed
in different threads.
| *Production Usage* | Yes. It is 'ON' by default. This feature can be disabled by setting this CQD to 'OFF'.
| *Impact* | IUD operations on tables with one or more indexes can become slower.
@@ -57,7 +57,7 @@ in different threads.
[cols="25%h,75%"]
|===
-| *Description* | Influences HBase to retain the data blocks in memory after they are read.
+| *Description* | Influences HBase to retain the data blocks in memory after they are read.
| *Values* |
*'ON'/'OFF'/'SYSTEM'* +
+
@@ -71,7 +71,7 @@ data blocks cached earlier thereby impacting the performance of the random reads
in such a case. +
+
Set the CQD <<hbase-region-server-max-heap-size,HBASE_REGION_SERVER_MAX_HEAP_SIZE>> value to reflect the amount of java heap size reserved for the region servers.
-This CQD is used by the {project-name} Optimizer to evaluate if the block cache should be turned off.
+This CQD is used by the {project-name} Optimizer to evaluate if the block cache should be turned off.
| *Production Usage* | Leave the setting to be 'SYSTEM' when HBase is configured to use LRU block cache. If needed,
you can override this settings with 'ON' or 'OFF'. +
+
@@ -150,7 +150,7 @@ The default value is *'10000'*.
a sequential scan of a table. +
+
You need to consider how soon the maximum number of rows are materialized on the Region Servers. When filtering is pushed down to Region Servers,
-then it can take a longer time depending upon the query and the predicates involved. This can result in HBase scanner timeouts.
+then it can take a longer time depending upon the query and the predicates involved. This can result in HBase scanner timeouts.
| *Production Usage* | Use the default setting and reduce the value to avoid HBase scanner timeouts. Please contact {project-support}.
if you think that you need to use this CQD.
| *Impact* | Not applicable.
@@ -167,7 +167,7 @@ if you think that you need to use this CQD.
[cols="25%h,75%"]
|===
-| *Description* | Allows INSERT, UPDATE, and DELETE (IUD) operations to be performed as an HBase batch `put` operation.
+| *Description* | Allows INSERT, UPDATE, and DELETE (IUD) operations to be performed as an HBase batch `put` operation.
| *Values* |
*'ON'*: Perform IUD operations as an HBase batch `put` operation. +
*'OFF'*: Do not perform IUD operations as an HBase batch `put` operation. +
@@ -192,7 +192,7 @@ If possible, then the query plan involves VSBB operators. The Virtual Sequential
[cols="25%h,75%"]
|===
-| *Description* | Determines the maximum number of rows in a batch `put` operation to HBase.
+| *Description* | Determines the maximum number of rows in a batch `put` operation to HBase.
| *Values* |
Numeric value.
+
@@ -216,7 +216,7 @@ You can adjust the maximum size to suit your application needs and thus tune it
[cols="25%h,75%"]
|===
-| *Description* | Enables {project-name} to leverage the HBase small scanner optimization. This optimization reduces I/O usage up to 66%
+| *Description* | Enables {project-name} to leverage the HBase small scanner optimization. This optimization reduces I/O usage up to 66%
and enables non-blocking reads for higher concurrency support. When a scan is known to require less than a HBASE BLOCK SIZE (default is 64K),
then enabling the HBase small scanner optimization increases performance.
| *Values* |
@@ -237,3 +237,27 @@ If you enable small scanner on large size scan incorrectly, then you are likely
| *Deprecated In Release* | Not applicable.
|===
+<<<
+[[scratch-disks]]
+== SCRATCH_DISKS
+
+[cols="25%h,75%"]
+|===
+| *Description* | Allows a CQD specification to override the env settings for STFS locations.
+| *Values* | Scratch disk locations for overflow (linux path names).
+| *Usage* | Currently the scratch directory locations are set at installation time. The STFS layer that manages the scratch file creation reads the envvar in ms.env to determine where to create the scratch files.
+
+The CQD is allowed to be specified that will override the envvar setting. This will give the user some flexibility to change scratch directory locations.
+
+If the CQD is not set (empty), the envvar setting will work.
+
+The requirement is that the directories MUST exists on every node and Trafodion user will need permissions to write to those directories. If not, an error will be returned when a query involving sort/hash overflow is executed.
+
+| *Production Usage* | Please contact {project-support}.
+| *Impact* | More volume of data can be processed, for example, large data set.
+| *Level* | System.
+| *Conflicts/Synergies* | Not applicable.
+| *Real Problem Addressed* | Not applicable.
+| *Introduced In Release* | Trafodion 2.1.
+| *Deprecated In Release* | Not applicable.
+|===
[4/4] incubator-trafodion git commit: Merge Fix for [TRAFODION-2521]
Add SCRATCH_DISKS CQD
Posted by li...@apache.org.
Merge Fix for [TRAFODION-2521] Add SCRATCH_DISKS CQD
Project: http://git-wip-us.apache.org/repos/asf/incubator-trafodion/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-trafodion/commit/299b538b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-trafodion/tree/299b538b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-trafodion/diff/299b538b
Branch: refs/heads/master
Commit: 299b538b630c9b38c33f6b8c388661bd7266f81a
Parents: c978ab5 b517466
Author: Liu Ming <ov...@sina.com>
Authored: Thu Mar 23 05:14:54 2017 +0000
Committer: Liu Ming <ov...@sina.com>
Committed: Thu Mar 23 05:14:54 2017 +0000
----------------------------------------------------------------------
.../src/asciidoc/_chapters/query_execution.adoc | 71 +++++++++++++++++---
1 file changed, 61 insertions(+), 10 deletions(-)
----------------------------------------------------------------------
[3/4] incubator-trafodion git commit: incorporate comments for
scratch dirs CQD 2
Posted by li...@apache.org.
incorporate comments for scratch dirs CQD 2
Project: http://git-wip-us.apache.org/repos/asf/incubator-trafodion/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-trafodion/commit/b5174665
Tree: http://git-wip-us.apache.org/repos/asf/incubator-trafodion/tree/b5174665
Diff: http://git-wip-us.apache.org/repos/asf/incubator-trafodion/diff/b5174665
Branch: refs/heads/master
Commit: b517466597a2d603a12d5ef8d4a4444562ca2def
Parents: 71f581f
Author: liu.yu <yu...@esgyn.cn>
Authored: Thu Mar 23 12:02:04 2017 +0800
Committer: liu.yu <yu...@esgyn.cn>
Committed: Thu Mar 23 12:02:04 2017 +0800
----------------------------------------------------------------------
.../src/asciidoc/_chapters/query_execution.adoc | 14 +++++++-------
1 file changed, 7 insertions(+), 7 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/b5174665/docs/cqd_reference/src/asciidoc/_chapters/query_execution.adoc
----------------------------------------------------------------------
diff --git a/docs/cqd_reference/src/asciidoc/_chapters/query_execution.adoc b/docs/cqd_reference/src/asciidoc/_chapters/query_execution.adoc
index 1f60546..5e41c99 100644
--- a/docs/cqd_reference/src/asciidoc/_chapters/query_execution.adoc
+++ b/docs/cqd_reference/src/asciidoc/_chapters/query_execution.adoc
@@ -243,7 +243,7 @@ If you enable small scanner on large size scan incorrectly, then you are likely
[cols="25%h,75%"]
|===
-| *Description* | Allows a CQD specification to override the envvar settings for scatch and temporary file locations, which gives user more flexibility to change scratch directory locations.
+| *Description* | Allows a CQD specification to override the environment variable settings for scatch and temporary file locations, which gives the user more flexibility to change scratch directory locations.
| *Values* |
Scratch disk locations for overflow (linux path names).
@@ -256,16 +256,16 @@ Multiple locations can be specified, separated by a colon. For example, /tmp/scr
* Specify this location at installation time.
+
-Currently the scratch directory locations are set at installation time. The layer of scratch and temporary file system that manages the scratch file creation and reads the envvar in $TRAF_HOME/etc/ms.env to determine where to create the scratch files.
+Currently the scratch directory locations are set at installation time. The code that manages scratch and temporary files reads environment variables in $TRAF_HOME/etc/ms.env to determine where to create scratch files.
+
-The CQD will override envvar settings, there are two envvar settings: one is STFS_HDD_LOACTION (default), the other is STFS_SSD_LOCATION (SSD disk).
+The CQD overrides the STFS_HDD_LOCATION (disk) and STFS_SSD_LOCATION (SSD disk) environment variables.
+
-If the CQD is not set (empty), the envvar setting will work.
+If the CQD is not set, the environment variable settings will be used.
+
-The requirement is that the directories MUST exist on every node and Trafodion user will need permission to write to those directories. If not, an error will be returned when a query involving sort/hash overflow is executed.
+The directories MUST exist on every node and the Trafodion user must have permission to write to those directories. If not, an error will be returned when a query involving sort/hash overflow is executed.
* Set it in sqconfig.
@@ -274,13 +274,13 @@ The requirement is that the directories MUST exist on every node and Trafodion u
* If the scratch directory location is not set at installation time, this CQD is valid only for the duration of the session.
+
-These settings will not be stored in $TRAF_HOME/etc/ms.env and will override the envvar settings as long as the CQD is in effect.
+These settings will not be stored in $TRAF_HOME/etc/ms.env and will override the environment variables settings as long as the CQD is in effect.
| *Production Usage* | Please contact {project-support}.
| *Impact* |
More volume of data can be processed. For example, large data set.
-Larger disk space can process larger data set.
+More disk space can process larger data sets.
| *Level* | System.
| *Conflicts/Synergies* | Not applicable.
[2/4] incubator-trafodion git commit: incorporate comments for
scratch dirs CQD
Posted by li...@apache.org.
incorporate comments for scratch dirs CQD
Project: http://git-wip-us.apache.org/repos/asf/incubator-trafodion/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-trafodion/commit/71f581fc
Tree: http://git-wip-us.apache.org/repos/asf/incubator-trafodion/tree/71f581fc
Diff: http://git-wip-us.apache.org/repos/asf/incubator-trafodion/diff/71f581fc
Branch: refs/heads/master
Commit: 71f581fcf0cfc8dc58c32adc0fb59bc90f3abd6f
Parents: fb6c16e
Author: liu.yu <yu...@esgyn.cn>
Authored: Mon Mar 13 13:19:15 2017 +0800
Committer: liu.yu <yu...@esgyn.cn>
Committed: Mon Mar 13 13:25:00 2017 +0800
----------------------------------------------------------------------
.../src/asciidoc/_chapters/query_execution.adoc | 43 ++++++++++++++++----
1 file changed, 35 insertions(+), 8 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/71f581fc/docs/cqd_reference/src/asciidoc/_chapters/query_execution.adoc
----------------------------------------------------------------------
diff --git a/docs/cqd_reference/src/asciidoc/_chapters/query_execution.adoc b/docs/cqd_reference/src/asciidoc/_chapters/query_execution.adoc
index 8f21152..1f60546 100644
--- a/docs/cqd_reference/src/asciidoc/_chapters/query_execution.adoc
+++ b/docs/cqd_reference/src/asciidoc/_chapters/query_execution.adoc
@@ -238,23 +238,50 @@ If you enable small scanner on large size scan incorrectly, then you are likely
|===
<<<
-[[scratch-disks]]
-== SCRATCH_DISKS
+[[scratch-dirs]]
+== SCRATCH_DIRS
[cols="25%h,75%"]
|===
-| *Description* | Allows a CQD specification to override the env settings for STFS locations.
-| *Values* | Scratch disk locations for overflow (linux path names).
-| *Usage* | Currently the scratch directory locations are set at installation time. The STFS layer that manages the scratch file creation reads the envvar in ms.env to determine where to create the scratch files.
+| *Description* | Allows a CQD specification to override the envvar settings for scatch and temporary file locations, which gives user more flexibility to change scratch directory locations.
+| *Values* |
+Scratch disk locations for overflow (linux path names).
+
+Multiple locations can be specified, separated by a colon. For example, /tmp/scratch1:/tmp/scratch2.
+
+| *Usage* a|
+
+*Two ways to change scratch directory location permanently*:
+
+* Specify this location at installation time.
+
++
+Currently the scratch directory locations are set at installation time. The layer of scratch and temporary file system that manages the scratch file creation and reads the envvar in $TRAF_HOME/etc/ms.env to determine where to create the scratch files.
-The CQD is allowed to be specified that will override the envvar setting. This will give the user some flexibility to change scratch directory locations.
++
+The CQD will override envvar settings, there are two envvar settings: one is STFS_HDD_LOACTION (default), the other is STFS_SSD_LOCATION (SSD disk).
++
If the CQD is not set (empty), the envvar setting will work.
-The requirement is that the directories MUST exists on every node and Trafodion user will need permissions to write to those directories. If not, an error will be returned when a query involving sort/hash overflow is executed.
++
+The requirement is that the directories MUST exist on every node and Trafodion user will need permission to write to those directories. If not, an error will be returned when a query involving sort/hash overflow is executed.
+
+* Set it in sqconfig.
+
+*One way to change scratch directory location temporarily*:
+
+* If the scratch directory location is not set at installation time, this CQD is valid only for the duration of the session.
+
++
+These settings will not be stored in $TRAF_HOME/etc/ms.env and will override the envvar settings as long as the CQD is in effect.
| *Production Usage* | Please contact {project-support}.
-| *Impact* | More volume of data can be processed, for example, large data set.
+| *Impact* |
+More volume of data can be processed. For example, large data set.
+
+Larger disk space can process larger data set.
+
| *Level* | System.
| *Conflicts/Synergies* | Not applicable.
| *Real Problem Addressed* | Not applicable.