You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tajo.apache.org by hyunsik <gi...@git.apache.org> on 2014/07/30 20:11:33 UTC
[GitHub] tajo pull request: TAJO-928: Session variables should override que...
GitHub user hyunsik opened a pull request:
https://github.com/apache/tajo/pull/98
TAJO-928: Session variables should override query configs in TajoConf.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/hyunsik/tajo TAJO-928
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/tajo/pull/98.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #98
----
commit a3b78642abb6c160b147eae2f29a10e362c14cac
Author: Hyunsik Choi <hy...@apache.org>
Date: 2014-07-08T08:47:42Z
Improve session variables to affect the query config.
commit 0a0035d9b259a1a05ba790b7a778a745251d27bd
Author: Hyunsik Choi <hy...@apache.org>
Date: 2014-07-08T12:54:32Z
Fixed.
commit 3fb54a6dde89d2d8e972253c1eccd17f334180d4
Author: Hyunsik Choi <hy...@apache.org>
Date: 2014-07-09T02:23:28Z
Completed output file rotating.
commit 8028f5f876af2050bb602e277026e76ca802619a
Author: Hyunsik Choi <hy...@apache.org>
Date: 2014-07-15T03:57:29Z
Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into OUTPUT_ROTATING
Conflicts:
tajo-core/src/main/java/org/apache/tajo/engine/planner/global/GlobalPlanner.java
tajo-core/src/main/java/org/apache/tajo/master/querymaster/Repartitioner.java
tajo-core/src/main/java/org/apache/tajo/master/querymaster/SubQuery.java
commit 50f6af418b42704ba14a4c7a084372f80c7ce1ec
Author: Hyunsik Choi <hy...@apache.org>
Date: 2014-07-15T06:25:09Z
Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into OUTPUT_ROTATING
commit 4d0abc0dfbf6c5898bce6bd0e1ecd4c995108571
Author: Hyunsik Choi <hy...@apache.org>
Date: 2014-07-15T11:13:55Z
Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into OUTPUT_ROTATING
Conflicts:
tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/HashBasedColPartitionStoreExec.java
tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/SortBasedColPartitionStoreExec.java
commit dd79f666d81875bf6a547478b76fc55b60f37d09
Author: Hyunsik Choi <hy...@apache.org>
Date: 2014-07-15T12:31:11Z
Added estimatedwrittensize.
commit da231ca89e5cf3638ea16faad281f8296854a9dd
Author: Hyunsik Choi <hy...@apache.org>
Date: 2014-07-17T03:03:37Z
Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into OUTPUT_ROTATING
commit c006382a3b16973872d753c9a0e0150da1c0f687
Author: Hyunsik Choi <hy...@apache.org>
Date: 2014-07-17T03:10:20Z
Reflect session variables to GlobalPlanner, Repartitioner, and PhysicalPlannerImpl.
commit bccd8528c84e09bc9861648bb427fb7706f8652d
Author: Hyunsik Choi <hy...@apache.org>
Date: 2014-07-25T01:20:33Z
Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into OUTPUT_ROTATING
commit 244318ccf8154c4b4e45dc24fdab23f85513e496
Author: Hyunsik Choi <hy...@apache.org>
Date: 2014-07-25T01:27:08Z
Reset output rotating part.
commit 244b2614f0986248cec43bd0012bce8c489ea95f
Author: Hyunsik Choi <hy...@apache.org>
Date: 2014-07-25T06:28:35Z
Add SessionVars and improve QueryContext.
commit ed05c03448e9b49fa1280266d6b4fc38e473f314
Author: Hyunsik Choi <hy...@apache.org>
Date: 2014-07-25T08:06:35Z
Improved QueryContext.
commit c8e53b4d140b7f6d656f10ded1767272b62a526e
Author: Hyunsik Choi <hy...@apache.org>
Date: 2014-07-29T07:29:05Z
Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-928
Conflicts:
tajo-core/src/main/java/org/apache/tajo/master/querymaster/Repartitioner.java
commit 2becff67dafba948e37463ef840c2d96ce8c56a1
Author: Hyunsik Choi <hy...@apache.org>
Date: 2014-07-29T10:16:01Z
Refactored QueryContext to take conf always.
commit 59861210fe4f16b50b076fd0b0456d6f7bd8a145
Author: Hyunsik Choi <hy...@apache.org>
Date: 2014-07-29T10:35:55Z
Refactored TaskAttemptContext to not take TajoConf.
commit ea952a4a3068f9ccff8cf2455300ddd7968da0fd
Author: Hyunsik Choi <hy...@apache.org>
Date: 2014-07-30T18:09:19Z
Fixed all unit tests.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-928: Session variables should override que...
Posted by hyunsik <gi...@git.apache.org>.
Github user hyunsik commented on the pull request:
https://github.com/apache/tajo/pull/98#issuecomment-51433096
I've described some changes.
The current way is to set some keys in TajoConf.ConfVars. I deprecated the way, but the way still works for a while. If you use the old way, you can see the deprecation warning.
```
Try \? for help.
default> \set tajo.dist-query.join.broadcast.threshold-bytes 50000
Warning: deprecated to directly use config key in TajoConf.ConfVars. Please execute '\help set'.
```
But, the conf key automatically is replaced by new key corresponding to the conf key.
```
default> \set
'BROADCAST_TABLE_SIZE_LIMIT'='50000'
'CURRENT_DATABASE'='default'
'SESSION_ID'='1a104a50-3f8d-4bac-a8a2-5a8efc936298'
'USERNAME'='hyunsik'
'SESSION_LAST_ACCESS_TIME'='1407388966573'
```
Also, I've added the help command and its alias. So you can use ```\? [command name]``` or ```\help [command name]```. I've extended some method for help to ```TajoShellCommand``` interface. The following is the help of set command.
```
default> \help set
Available Session Variables:
\set SESSION_EXPIRY_TIME [int value] - session expiry time (secs)
\set CLI_COLUMNS [int value] - Sets the width for the wrapped format
\set CLI_FORMATTER_CLASS [text value] - Sets the output format class to display results
\set CLI_NULL_CHAR [text value] - Sets the string to be printed in place of a null value.
\set CLI_PAGE_ROWS [int value] - Sets the number of rows for paging
\set CLI_PAGING_ENABLED [true or false] - Enable paging of result display
\set CLI_DISPLAY_ERROR_TRACE [true or false] - Enable display of error trace
\set ON_ERROR_STOP [true or false] - tsql will exist if an error occurs.
\set TZ [text value] - Sets timezone
\set DATE_ORDER [text value] - date order (default is YMD)
\set LANG [text value] - Language
\set LC_ALL [text value] - String sort order
\set LC_COLLATE [text value] - String sort order
\set LC_CTYPE [text value] - Character classification (What is a letter? Its upper-case equivalent?)
\set LC_MESSAGES [text value] - Language of messages
\set LC_MONETARY [text value] - Formatting of currency amounts
\set LC_NUMERIC [text value] - Formatting of numbers
\set LC_TIME [text value] - Formatting of dates and times
\set BROADCAST_TABLE_SIZE_LIMIT [long value] - limited size (bytes) of broadcast table
\set JOIN_TASK_INPUT_SIZE [int value] - join task input size (mb)
\set SORT_TASK_INPUT_SIZE [int value] - sort task input size (mb)
\set GROUPBY_TASK_INPUT_SIZE [int value] - group by task input size (mb)
\set JOIN_PER_SHUFFLE_SIZE [int value] - shuffle output size for join (mb)
\set GROUPBY_PER_SHUFFLE_SIZE [int value] - shuffle output size for sort (mb)
\set TABLE_PARTITION_PER_SHUFFLE_SIZE [int value] - shuffle output size for partition table write (mb)
\set EXTSORT_BUFFER_SIZE [long value] - sort buffer size for external sort (mb)
\set HASH_JOIN_SIZE_LIMIT [long value] - limited size for hash join (mb)
\set INNER_HASH_JOIN_SIZE_LIMIT [long value] - limited size for hash inner join (mb)
\set OUTER_HASH_JOIN_SIZE_LIMIT [long value] - limited size for hash outer join (mb)
\set HASH_GROUPBY_SIZE_LIMIT [long value] - limited size for hash groupby (mb)
\set MAX_OUTPUT_FILE_SIZE [int value] - Maximum per-output file size (mb). 0 means infinite.
\set NULL_CHAR [text value] - null char of text file output
\set ARITHABORT [true or false] - If true, a running query will be terminated when an overflow or divide-by-zero occurs.
\set DEBUG_ENABLED [true or false] - (debug only) debug mode enabled
```
Also, I've updated querydetail.jsp file to show session variables applied to the query.
I believe that this patch is ready to be committed. Please review this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-928: Session variables should override que...
Posted by hyunsik <gi...@git.apache.org>.
Github user hyunsik commented on the pull request:
https://github.com/apache/tajo/pull/98#issuecomment-51555782
If the failure occurs occasionally, we need to handle it in another jira issue.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-928: Session variables should override que...
Posted by blrunner <gi...@git.apache.org>.
Github user blrunner commented on the pull request:
https://github.com/apache/tajo/pull/98#issuecomment-51436220
Thank you for your detailed description and sorry but I'm hard to review it now.
Could I review it tonight?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-928: Session variables should override que...
Posted by hyunsik <gi...@git.apache.org>.
Github user hyunsik commented on the pull request:
https://github.com/apache/tajo/pull/98#issuecomment-50998335
It is ready to be reviewed. Please review the patch. I'll also add some documentation soon.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-928: Session variables should override que...
Posted by hyunsik <gi...@git.apache.org>.
Github user hyunsik commented on the pull request:
https://github.com/apache/tajo/pull/98#issuecomment-50657070
This is still ongoing work. The remain works are as follows:
* Some session variables only used in cli will be removed
* Add more comments and description
* Add new meta comment for tsql display configuration
* Add documentation about new session variable system
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-928: Session variables should override que...
Posted by blrunner <gi...@git.apache.org>.
Github user blrunner commented on the pull request:
https://github.com/apache/tajo/pull/98#issuecomment-51554635
Hi @hyunsik
Unfortunately, 'mvn clean install' failed as follows:
{code:xml}
Results :
Failed tests: testForwardedQuery(org.apache.tajo.scheduler.TestFifoScheduler): expected:<QUERY_RUNNING> but was:<QUERY_NOT_ASSIGNED>
Tests run: 1096, Failures: 1, Errors: 0, Skipped: 0
{code}
Could you check it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-928: Session variables should override que...
Posted by hyunsik <gi...@git.apache.org>.
Github user hyunsik commented on the pull request:
https://github.com/apache/tajo/pull/98#issuecomment-51436313
@blrunner Thanks in advance!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-928: Session variables should override que...
Posted by blrunner <gi...@git.apache.org>.
Github user blrunner commented on the pull request:
https://github.com/apache/tajo/pull/98#issuecomment-51738322
+1
Thank you for your great patch, it will be very useful to tajo.
It looks good overall and I tested it on my testing cluster.
For reference, It works expectedly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-928: Session variables should override que...
Posted by hyunsik <gi...@git.apache.org>.
Github user hyunsik commented on the pull request:
https://github.com/apache/tajo/pull/98#issuecomment-51734846
I've rebased the patch against the latest revision. Thank you for your review.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-928: Session variables should override que...
Posted by hyunsik <gi...@git.apache.org>.
Github user hyunsik commented on the pull request:
https://github.com/apache/tajo/pull/98#issuecomment-51555728
I haven't experienced the failure. The failure seems to occur occasionally due to unknown reason. In addition, I strongly believe that this change affects the test.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-928: Session variables should override que...
Posted by blrunner <gi...@git.apache.org>.
Github user blrunner commented on a diff in the pull request:
https://github.com/apache/tajo/pull/98#discussion_r15792495
--- Diff: tajo-common/src/main/java/org/apache/tajo/OverridableConf.java ---
@@ -0,0 +1,239 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.tajo;
+
+import com.google.common.base.Preconditions;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.tajo.conf.TajoConf;
+import org.apache.tajo.util.KeyValueSet;
+
+import java.lang.ref.WeakReference;
+import java.util.Map;
+import java.util.WeakHashMap;
+
+import static org.apache.tajo.ConfigKey.ConfigType;
+import static org.apache.tajo.rpc.protocolrecords.PrimitiveProtos.KeyValueSetProto;
+
+/**
+ * QueryContext provides a consolidated config system for a query instant.
+ *
+ * In Tajo, there are three configurable layers:
+ * <ul>
+ * <li>
+ * <ul>System Config - it comes from Hadoop's Configuration class. by tajo-site, catalog-site,
+ * catalog-default and TajoConf.</ul>
+ * <ul>Session variables - they are instantly configured by users.
+ * Each client session has it own set of session variables.</ul>
+ * <ul>Query config - it is internally used for meta information of a query instance.</ul>
+ * </li>
+ * </ul>
+ *
+ * System configs and session variables can set the same config in the same time. System configs are usually used to set
+ * default configs, and session variables is user-specified configs. So, session variables can override system configs.
+ *
+ * QueryContent provides a query with a uniform way to access various configs without considering their priorities.
+ */
+public class OverridableConf extends KeyValueSet {
+ private static final Log LOG = LogFactory.getLog(OverridableConf.class);
+ private ConfigType [] configTypes;
+
+ private static final Map<ClassLoader, Map<String, WeakReference<Class<?>>>>
+ CACHE_CLASSES = new WeakHashMap<ClassLoader, Map<String, WeakReference<Class<?>>>>();
+
+ private TajoConf conf;
+
+ public OverridableConf(final TajoConf conf, ConfigType...configTypes) {
+ this.conf = conf;
+ this.configTypes = configTypes;
+ }
+
+ public OverridableConf(final TajoConf conf, KeyValueSetProto proto, ConfigType...configTypes) {
+ super(proto);
+ this.conf = conf;
+ this.configTypes = configTypes;
+ }
+
+ public void setConf(TajoConf conf) {
+ this.conf = conf;
+ }
+
+ public TajoConf getConf() {
+ return conf;
+ }
+
+ public void setBool(ConfigKey key, boolean val) {
+ setBool(key.keyname(), val);
+ }
+
+ public boolean getBool(ConfigKey key, Boolean defaultVal) {
+ assertRegisteredEnum(key);
+
+ switch (key.type()) {
+ case QUERY:
+ return getBool(key.keyname());
+ case SESSION:
+ return getBool(key.keyname(), conf.getBoolVar(((SessionVars) key).getConfVars()));
+ case SYSTEM:
+ return conf.getBoolVar((TajoConf.ConfVars) key);
+ default:
+ return getBool(key.keyname(), defaultVal);
+ }
+ }
+
+ public boolean getBool(ConfigKey key) {
+ return getBool(key, null);
+ }
+
+ public int getInt(ConfigKey key, Integer defaultVal) {
+ assertRegisteredEnum(key);
+
+ if (key.type() != ConfigType.SESSION && key.type() != ConfigType.SYSTEM) {
+ return getInt(key.keyname());
+ } else {
+ switch (key.type()) {
+ case SESSION:
+ return getInt(key.keyname(), conf.getIntVar(((SessionVars) key).getConfVars()));
+ case SYSTEM:
+ return conf.getIntVar((TajoConf.ConfVars) key);
+ default:
+ throw new IllegalStateException("key does not belong to Session and System config sets");
+ }
+ }
+ }
+
+ public int getInt(ConfigKey key) {
+ return getInt(key, null);
+ }
+
+ public long getLong(ConfigKey key, Long defaultVal) {
+ assertRegisteredEnum(key);
+
+ if (key.type() != ConfigType.SESSION && key.type() != ConfigType.SYSTEM) {
+ return getLong(key.keyname());
+ } else {
+ switch (key.type()) {
+ case SESSION:
+ return getLong(key.keyname(), conf.getLongVar(((SessionVars) key).getConfVars()));
+ case SYSTEM:
+ return conf.getLongVar((TajoConf.ConfVars) key);
+ default:
+ throw new IllegalStateException("key does not belong to Session and System config sets");
+ }
+ }
+ }
+
+ public long getLong(ConfigKey key) {
+ return getLong(key, null);
+ }
+
+ public float getFloat(ConfigKey key, Float defaultVal) {
+ assertRegisteredEnum(key);
+
+ if (key.type() != ConfigType.SESSION && key.type() != ConfigType.SYSTEM) {
+ return getFloat(key.keyname());
+ } else {
+ switch (key.type()) {
+ case SESSION:
+ return getFloat(key.keyname(), conf.getFloatVar(((SessionVars) key).getConfVars()));
+ case SYSTEM:
+ return conf.getFloatVar((TajoConf.ConfVars) key);
+ default:
+ throw new IllegalStateException("key does not belong to Session and System config sets");
+ }
+ }
+ }
+
+ public float getFloat(ConfigKey key) {
+ return getLong(key, null);
+ }
+
+ public void put(ConfigKey key, String val) {
+ set(key.keyname(), val);
+ }
+
+ private void assertRegisteredEnum(ConfigKey key) {
+ boolean registered = false;
+
+ for (ConfigType c : configTypes) {
+ registered = key.type() == c;
+ }
+
+ registered |= key.type() == ConfigType.SESSION || key.type() != ConfigType.SYSTEM;
+
+ Preconditions.checkArgument(registered, key.keyname() + " (" + key.type() + ") is not allowed in " +
+ getClass().getSimpleName());
+ }
+
+ public String get(ConfigKey key, String defaultVal) {
+ assertRegisteredEnum(key);
+
+ if (key.type() != ConfigType.SESSION && key.type() != ConfigType.SYSTEM) {
+ return get(key.keyname(), defaultVal);
+ } else {
+ switch (key.type()) {
+ case SESSION:
+ return get(key.keyname(), conf.getVar(((SessionVars) key).getConfVars()));
+ case SYSTEM:
+ return conf.getVar((TajoConf.ConfVars) key);
+ default:
+ throw new IllegalStateException("key does not belong to Session and System config sets");
+ }
+ }
+ }
+
+ public String get(ConfigKey key) {
+ return get(key, null);
+ }
+
+ public Class<?> getClass(ConfigKey key) {
+// if (containsKey(key)) {
--- End diff --
Is this a comment?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-928: Session variables should override que...
Posted by hyunsik <gi...@git.apache.org>.
Github user hyunsik commented on a diff in the pull request:
https://github.com/apache/tajo/pull/98#discussion_r15918947
--- Diff: tajo-common/src/main/java/org/apache/tajo/OverridableConf.java ---
@@ -0,0 +1,239 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.tajo;
+
+import com.google.common.base.Preconditions;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.tajo.conf.TajoConf;
+import org.apache.tajo.util.KeyValueSet;
+
+import java.lang.ref.WeakReference;
+import java.util.Map;
+import java.util.WeakHashMap;
+
+import static org.apache.tajo.ConfigKey.ConfigType;
+import static org.apache.tajo.rpc.protocolrecords.PrimitiveProtos.KeyValueSetProto;
+
+/**
+ * QueryContext provides a consolidated config system for a query instant.
+ *
+ * In Tajo, there are three configurable layers:
+ * <ul>
+ * <li>
+ * <ul>System Config - it comes from Hadoop's Configuration class. by tajo-site, catalog-site,
+ * catalog-default and TajoConf.</ul>
+ * <ul>Session variables - they are instantly configured by users.
+ * Each client session has it own set of session variables.</ul>
+ * <ul>Query config - it is internally used for meta information of a query instance.</ul>
+ * </li>
+ * </ul>
+ *
+ * System configs and session variables can set the same config in the same time. System configs are usually used to set
+ * default configs, and session variables is user-specified configs. So, session variables can override system configs.
+ *
+ * QueryContent provides a query with a uniform way to access various configs without considering their priorities.
+ */
+public class OverridableConf extends KeyValueSet {
+ private static final Log LOG = LogFactory.getLog(OverridableConf.class);
+ private ConfigType [] configTypes;
+
+ private static final Map<ClassLoader, Map<String, WeakReference<Class<?>>>>
+ CACHE_CLASSES = new WeakHashMap<ClassLoader, Map<String, WeakReference<Class<?>>>>();
+
+ private TajoConf conf;
+
+ public OverridableConf(final TajoConf conf, ConfigType...configTypes) {
+ this.conf = conf;
+ this.configTypes = configTypes;
+ }
+
+ public OverridableConf(final TajoConf conf, KeyValueSetProto proto, ConfigType...configTypes) {
+ super(proto);
+ this.conf = conf;
+ this.configTypes = configTypes;
+ }
+
+ public void setConf(TajoConf conf) {
+ this.conf = conf;
+ }
+
+ public TajoConf getConf() {
+ return conf;
+ }
+
+ public void setBool(ConfigKey key, boolean val) {
+ setBool(key.keyname(), val);
+ }
+
+ public boolean getBool(ConfigKey key, Boolean defaultVal) {
+ assertRegisteredEnum(key);
+
+ switch (key.type()) {
+ case QUERY:
+ return getBool(key.keyname());
+ case SESSION:
+ return getBool(key.keyname(), conf.getBoolVar(((SessionVars) key).getConfVars()));
+ case SYSTEM:
+ return conf.getBoolVar((TajoConf.ConfVars) key);
+ default:
+ return getBool(key.keyname(), defaultVal);
+ }
+ }
+
+ public boolean getBool(ConfigKey key) {
+ return getBool(key, null);
+ }
+
+ public int getInt(ConfigKey key, Integer defaultVal) {
+ assertRegisteredEnum(key);
+
+ if (key.type() != ConfigType.SESSION && key.type() != ConfigType.SYSTEM) {
+ return getInt(key.keyname());
+ } else {
+ switch (key.type()) {
+ case SESSION:
+ return getInt(key.keyname(), conf.getIntVar(((SessionVars) key).getConfVars()));
+ case SYSTEM:
+ return conf.getIntVar((TajoConf.ConfVars) key);
+ default:
+ throw new IllegalStateException("key does not belong to Session and System config sets");
+ }
+ }
+ }
+
+ public int getInt(ConfigKey key) {
+ return getInt(key, null);
+ }
+
+ public long getLong(ConfigKey key, Long defaultVal) {
+ assertRegisteredEnum(key);
+
+ if (key.type() != ConfigType.SESSION && key.type() != ConfigType.SYSTEM) {
+ return getLong(key.keyname());
+ } else {
+ switch (key.type()) {
+ case SESSION:
+ return getLong(key.keyname(), conf.getLongVar(((SessionVars) key).getConfVars()));
+ case SYSTEM:
+ return conf.getLongVar((TajoConf.ConfVars) key);
+ default:
+ throw new IllegalStateException("key does not belong to Session and System config sets");
+ }
+ }
+ }
+
+ public long getLong(ConfigKey key) {
+ return getLong(key, null);
+ }
+
+ public float getFloat(ConfigKey key, Float defaultVal) {
+ assertRegisteredEnum(key);
+
+ if (key.type() != ConfigType.SESSION && key.type() != ConfigType.SYSTEM) {
+ return getFloat(key.keyname());
+ } else {
+ switch (key.type()) {
+ case SESSION:
+ return getFloat(key.keyname(), conf.getFloatVar(((SessionVars) key).getConfVars()));
+ case SYSTEM:
+ return conf.getFloatVar((TajoConf.ConfVars) key);
+ default:
+ throw new IllegalStateException("key does not belong to Session and System config sets");
+ }
+ }
+ }
+
+ public float getFloat(ConfigKey key) {
+ return getLong(key, null);
+ }
+
+ public void put(ConfigKey key, String val) {
+ set(key.keyname(), val);
+ }
+
+ private void assertRegisteredEnum(ConfigKey key) {
+ boolean registered = false;
+
+ for (ConfigType c : configTypes) {
+ registered = key.type() == c;
+ }
+
+ registered |= key.type() == ConfigType.SESSION || key.type() != ConfigType.SYSTEM;
+
+ Preconditions.checkArgument(registered, key.keyname() + " (" + key.type() + ") is not allowed in " +
+ getClass().getSimpleName());
+ }
+
+ public String get(ConfigKey key, String defaultVal) {
+ assertRegisteredEnum(key);
+
+ if (key.type() != ConfigType.SESSION && key.type() != ConfigType.SYSTEM) {
+ return get(key.keyname(), defaultVal);
+ } else {
+ switch (key.type()) {
+ case SESSION:
+ return get(key.keyname(), conf.getVar(((SessionVars) key).getConfVars()));
+ case SYSTEM:
+ return conf.getVar((TajoConf.ConfVars) key);
+ default:
+ throw new IllegalStateException("key does not belong to Session and System config sets");
+ }
+ }
+ }
+
+ public String get(ConfigKey key) {
+ return get(key, null);
+ }
+
+ public Class<?> getClass(ConfigKey key) {
+// if (containsKey(key)) {
--- End diff --
I removed the commented out lines.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-928: Session variables should override que...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/tajo/pull/98
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-928: Session variables should override que...
Posted by blrunner <gi...@git.apache.org>.
Github user blrunner commented on the pull request:
https://github.com/apache/tajo/pull/98#issuecomment-51734603
Sorry for my late review.
Could you rebase against the master branch?
I failed to merge it as follows:
{code:xml}
CONFLICT (content): Merge conflict in tajo-core/src/test/java/org/apache/tajo/cli/TestTajoCli.java
Auto-merging tajo-client/src/main/java/org/apache/tajo/cli/TajoCli.java
Automatic merge failed; fix conflicts and then commit the result.
{code}
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---