You are viewing a plain text version of this content. The canonical link for it is here.
Posted to codereview@trafodion.apache.org by zcorrea <gi...@git.apache.org> on 2017/05/02 01:57:42 UTC
[GitHub] incubator-trafodion pull request #1077: Trafodion 2001
GitHub user zcorrea opened a pull request:
https://github.com/apache/incubator-trafodion/pull/1077
Trafodion 2001
This is an initial pull request to validate changes which enable elasticity changes (JIRA: TRAFODION-2001).
These elasticity changes allow for Trafodion cluster node (hostname) membership to be changed while the instance is operational. Node can be added or deleted while the instance is running.
These changes include:
o Implementation of Trafodion Configuration API shared library
- Provides an interface to a repository for Trafodion Configuration and Registry
+ Initial repository storage method implemented is SQLite
- Allows for future repository storage methods such as Zookeeper, MySQL, etc. to added without affecting the rest of the Trafodion components
o Implementation of 'PERSIST' configuration section
- A template based persistent process definition
o Implementation of new shell commands (and enhancements to existing commands):
- node add node-name <node-name>,
cores {<first-core>} [ - <last-core>}],
processors {<processor-count>},
roles {connection|aggregation|storage}
- node config [<nid>|<node-name>]
- node delete <node-name>
- node down <nid>|<node-name> [, <reason-string>]
- node info [<nid>]
- node name <old-node-name> <new-node-name>
- node up <name>
- persist config [{keys}|<persist-process-prefix>]
- persist exec <persist-process-prefix>
- persist info [<persist-process-prefix>]
- persist kill <persist-process-prefix>
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zcorrea/incubator-trafodion TRAFODION-2001
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-trafodion/pull/1077.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1077
----
commit bc944cb865214e15e142688fc07e1e59882789f7
Author: Gonzalo Correa <go...@hp.com>
Date: 2015-04-01T21:40:40Z
Temporary deletion on shared repo.
commit 7ad9a9c114d6f0079980aa7a333dc4cccf78fcc2
Author: Dave George <d....@hp.com>
Date: 2015-04-14T15:06:31Z
remove extraneous from gensq.pl/sqconfig.
commit b240bf754a73bbf2d546cf499cfff6e5008257e1
Author: Dave George <d....@hp.com>
Date: 2015-04-14T15:08:34Z
refactor lobstart - phase I.
commit b36941153c28d753be8e1902078a62a2e0e3dcd0
Author: Dave George <d....@hp.com>
Date: 2015-04-14T15:18:38Z
Qualify percomstart.
commit 6660530c85d3cc4bbcda7cf3eae3cc4926c3c4b4
Author: Dave George <d....@hp.com>
Date: 2015-04-14T15:23:03Z
obsolete.
commit 89b8d4b47507b5b1b87d12ed9650a5ea61de2145
Author: Dave George <d....@hp.com>
Date: 2015-04-14T15:27:15Z
remove obsolete.
commit bd75eb612ac9ef7481b4836dacdff283042ee064
Author: Dave George <d....@hp.com>
Date: 2015-04-14T19:24:59Z
remove obsolete parts of sqstart.
commit 57d4de657238250491a5f1a08d4634999b8c5e6f
Author: Dave George <d....@hp.com>
Date: 2015-04-14T20:51:25Z
refactor gomon.cold.
commit cd18ee971ba05da2e02907a019ef1d581bb974c5
Author: Gonzalo Correa <go...@hp.com>
Date: 2015-05-11T21:56:53Z
Removed 'cluster.conf', added node conf in SQLite
commit 42e3e389bc4624b3bb67b9c57d7dbb4c554b6e2d
Author: Dave George <d....@hp.com>
Date: 2015-05-13T17:48:10Z
remove cluster.conf.
commit f07a3a998e88deb800623bc49150bfa722b824a2
Author: Gonzalo Correa <go...@hp.com>
Date: 2015-05-18T21:28:34Z
Merge remote branch 'gerrit/master' into traf/traf-config
commit 882d3349b30f722567965eb26b16a23561fbff18
Author: Gonzalo Correa <go...@hp.com>
Date: 2015-05-20T23:11:32Z
Merge branch 'master' into traf/opensrc
commit 5f8cb2e881bff92c324829722200ef8f415ad830
Author: Gonzalo Correa <go...@hp.com>
Date: 2015-05-20T23:13:12Z
Merge branch 'master' into traf/traf-config
commit 5f6ce4a2b2beeba9453e3b9a26419cc2cf2d4507
Author: Dave George <d....@hp.com>
Date: 2015-05-26T21:28:57Z
Add persistent process processing.
commit 8f49db80ae9667dc0cae6685a30e4bf10d3ccfb8
Author: Dave George <d....@hp.com>
Date: 2015-05-27T16:49:46Z
Remove trailing spaces from keys.
commit 53d651b42e5627df361ffdd57a4de30a618ce39e
Author: Gonzalo Correa <go...@hp.com>
Date: 2015-05-28T15:52:49Z
Merge branch 'master' into traf/traf-config
commit 2a7a336d66dbc2abced2d5d5b8524e2babf64220
Author: Dave George <d....@hp.com>
Date: 2015-05-28T16:47:53Z
Update parser for errors.
commit 96804eaac9d36881c3710bf216b1295640d61b26
Author: Dave George <d....@hp.com>
Date: 2015-05-28T18:55:00Z
Error handling.
commit 848e6a5dbd8b397915d02486a4b1e690dcf307a3
Author: Gonzalo Correa <go...@hp.com>
Date: 2015-05-28T23:06:44Z
Merge branch 'master' into traf/traf-config
commit 5f4ff7abe0013951f604a313f24806934a18aae2
Author: Dave George <d....@hp.com>
Date: 2015-06-01T13:51:06Z
Add node Added/Deleted.
commit 10929b27d82ed3982df8274f351e2546f49ddd0f
Author: Dave George <d....@hp.com>
Date: 2015-06-02T18:33:01Z
rework MsgType/ReqType/ReplyType.
commit 797b856a84e94ae1f96d0ff6d66e761e32a75b89
Author: Dave George <d....@hp.com>
Date: 2015-06-02T19:45:30Z
Fix permissions.
commit 97dd38ccc2ec941d5cb5c473c7127f215b2131fb
Author: Dave George <d....@hp.com>
Date: 2015-06-02T19:51:17Z
Fix permissions.
commit 9ac41179de0c31b6fe7640ef253f70c479310058
Author: Gonzalo Correa <go...@hp.com>
Date: 2015-06-02T22:38:10Z
Preliminary CClusterConfig changes
Change-Id: I8f61c7013e8553167bc08fd4d3d628d8f870d8b7
commit 7b010d94354607c374c0b99cea6df621d92d960b
Author: Gonzalo Correa <go...@hp.com>
Date: 2015-06-02T22:51:55Z
Merge branch 'master' into traf/traf-config
Conflicts:
sqf/monitor/linux/msgdef.h
Change-Id: Ib0387f04cefe26e49f36ed140f7ea6e05729ede8
commit ed6424e6aa9800d29dc38dc3ef5632fbf1d3459a
Author: Gonzalo Correa <go...@hp.com>
Date: 2015-06-03T00:14:24Z
Added process info pattern
commit 3c3b62875c4948da3b42b248e3758e9d8dd25e61
Author: Gonzalo Correa <go...@hp.com>
Date: 2015-06-03T00:24:57Z
Merge branch 'master' into traf/traf-config
Conflicts:
sqf/monitor/linux/msgdef.h
Change-Id: I869feff9fb564a59196bbc3e3e56eea56431c72e
commit 65a792259851c6bd3305fd6662280037eb394b89
Author: Gonzalo Correa <go...@hp.com>
Date: 2015-06-04T00:25:53Z
Code cleanup and process info pattern enabled
Change-Id: I30639358d5c30e02567f5a74ce7078cb2b08d433
commit 56c544a1d4ef958281b8406b2b18d2c959a7abf6
Author: Dave George <d....@hp.com>
Date: 2015-06-04T16:37:19Z
pstartd rework for new db.
commit f334fea8dd3416a2bf801f9ec8913dfedb3ea1c4
Author: Dave George <d....@hp.com>
Date: 2015-06-04T16:38:27Z
Merge branch 'master' of /archive/gcorrea/traf-config
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request #1077: Trafodion 2001
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-trafodion/pull/1077
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request #1077: Trafodion 2001
Posted by svarnau <gi...@git.apache.org>.
Github user svarnau commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/1077#discussion_r115350803
--- Diff: install/python-installer/scripts/traf_sqconfig.py ---
@@ -51,18 +52,15 @@ def run():
lines.append(line)
lines.append('end node\n')
- lines.append('\n')
- lines.append('begin overflow\n')
-
- for scratch_loc in scratch_locs:
- line = 'hdd %s\n' % scratch_loc
- lines.append(line)
-
- lines.append('end overflow\n')
+ # write out the node section
with open(sqconfig_file, 'w') as f:
f.writelines(lines)
+ # append the current persist section to the Trafodion configuration file
+ run_cmd('echo "sed \'1,20d\' %s >> %s"' % (sqconfig_persist_file, sqconfig_file))
+ run_cmd('sed \'1,20d\' %s >> %s' % (sqconfig_persist_file, sqconfig_file))
--- End diff --
In fact, why are the installers concatenating these files?
Why doesn't monitor just directly read both sqconfig and sqconfig.persist?
Or sqgen take some file name parameters?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request #1077: Trafodion 2001
Posted by mkby <gi...@git.apache.org>.
Github user mkby commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/1077#discussion_r115188436
--- Diff: core/sqf/sql/scripts/sqnodestatus ---
@@ -26,12 +26,11 @@
# Implements:
# ssh -M "ConnectTimeout 1" -o "ConnectionAttempts 3" n028 hostinfo 2>&1
#
-# Uses:
-# export MY_NODES=' -w n028 -w n044 -w n029 -w n045 -w n030 -w n031 -w n032 '
+# Uses the following enviroment variables:
# export SQ_MON_SSH_OPTIONS=' -o "ConnectTimeout 1" -o "ConnectionAttempts 3" '
#
-my $node_context=lc(readpipe("echo \$NODE_LIST"));
+my $node_context=readpipe("trafconf -name");
--- End diff --
looks good!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request #1077: Trafodion 2001
Posted by mkby <gi...@git.apache.org>.
Github user mkby commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/1077#discussion_r115133212
--- Diff: core/sqf/sql/scripts/sqnodestatus ---
@@ -26,12 +26,11 @@
# Implements:
# ssh -M "ConnectTimeout 1" -o "ConnectionAttempts 3" n028 hostinfo 2>&1
#
-# Uses:
-# export MY_NODES=' -w n028 -w n044 -w n029 -w n045 -w n030 -w n031 -w n032 '
+# Uses the following enviroment variables:
# export SQ_MON_SSH_OPTIONS=' -o "ConnectTimeout 1" -o "ConnectionAttempts 3" '
#
-my $node_context=lc(readpipe("echo \$NODE_LIST"));
+my $node_context=readpipe("trafconf -name");
--- End diff --
will `trafconf -name` return the lowercase hostname or the upper/lower case of hostname is addressed in other codes?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request #1077: Trafodion 2001
Posted by svarnau <gi...@git.apache.org>.
Github user svarnau commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/1077#discussion_r115345246
--- Diff: install/python-installer/scripts/traf_sqconfig.py ---
@@ -51,18 +52,15 @@ def run():
lines.append(line)
lines.append('end node\n')
- lines.append('\n')
- lines.append('begin overflow\n')
-
- for scratch_loc in scratch_locs:
- line = 'hdd %s\n' % scratch_loc
- lines.append(line)
-
- lines.append('end overflow\n')
+ # write out the node section
with open(sqconfig_file, 'w') as f:
f.writelines(lines)
+ # append the current persist section to the Trafodion configuration file
+ run_cmd('echo "sed \'1,20d\' %s >> %s"' % (sqconfig_persist_file, sqconfig_file))
+ run_cmd('sed \'1,20d\' %s >> %s' % (sqconfig_persist_file, sqconfig_file))
--- End diff --
why are we stripping header comments out?
seems like we ought to include file verbatim.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request #1077: Trafodion 2001
Posted by zcorrea <gi...@git.apache.org>.
Github user zcorrea commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/1077#discussion_r115490249
--- Diff: install/python-installer/scripts/traf_sqconfig.py ---
@@ -51,18 +52,15 @@ def run():
lines.append(line)
lines.append('end node\n')
- lines.append('\n')
- lines.append('begin overflow\n')
-
- for scratch_loc in scratch_locs:
- line = 'hdd %s\n' % scratch_loc
- lines.append(line)
-
- lines.append('end overflow\n')
+ # write out the node section
with open(sqconfig_file, 'w') as f:
f.writelines(lines)
+ # append the current persist section to the Trafodion configuration file
+ run_cmd('echo "sed \'1,20d\' %s >> %s"' % (sqconfig_persist_file, sqconfig_file))
+ run_cmd('sed \'1,20d\' %s >> %s' % (sqconfig_persist_file, sqconfig_file))
--- End diff --
On stripping the header comments out, the target file already has the same Copyright information.
On concatenating, yes, the sqgen should probably be modified to pull in the persist file directly after processing the node section on the 'sqconfig' file. Good suggestion!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request #1077: Trafodion 2001
Posted by zcorrea <gi...@git.apache.org>.
Github user zcorrea commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/1077#discussion_r115154601
--- Diff: core/sqf/sql/scripts/sqnodestatus ---
@@ -26,12 +26,11 @@
# Implements:
# ssh -M "ConnectTimeout 1" -o "ConnectionAttempts 3" n028 hostinfo 2>&1
#
-# Uses:
-# export MY_NODES=' -w n028 -w n044 -w n029 -w n045 -w n030 -w n031 -w n032 '
+# Uses the following enviroment variables:
# export SQ_MON_SSH_OPTIONS=' -o "ConnectTimeout 1" -o "ConnectionAttempts 3" '
#
-my $node_context=lc(readpipe("echo \$NODE_LIST"));
+my $node_context=readpipe("trafconf -name");
--- End diff --
The upper lower case is addressed in other code. Specifically, in the monitor code. The trafconf utility is new and was created to address the need of obtaining the current hostname (node name) membership in a Trafodion cluster instance. I will add more details to its purpose and usage in the JIRA: TRAFODION-2001. A brief summary is as follows:
- Node membership is initially derived from 'sqconfig' and used to populate the Trafodion Configuration database. After this, node membership is through the 'shell' node add/delete commands.
- The Trafodion Configuration database is the configuration of record once the instance is started.
- The 'trafconf' utility reads the contents of the Trafodion Configuration database to display the current node membership of the cluster.
- The 'trafconf' utility us used by scripts and by the Trafodion user .bashrc to populate environment variables with the current node membership. Specifically, the MY_NODES environment variable.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---