You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@asterixdb.apache.org by "Chris Hillery (Code Review)" <do...@asterixdb.incubator.apache.org> on 2016/06/24 08:58:37 UTC
Change in asterixdb[master]: Added NCServiceExecutionIT and implemented several fixes.
Chris Hillery has uploaded a new change for review.
https://asterix-gerrit.ics.uci.edu/958
Change subject: Added NCServiceExecutionIT and implemented several fixes.
......................................................................
Added NCServiceExecutionIT and implemented several fixes.
1. Fix handling of iodevices/storagedir
2. Proper handling of [nc] default section in all cases
3. Consolidate Ini handling
4. Pruned some dead code, and a bit of refactoring
Change-Id: If3eb450782a595cf85d04a2c2e9cc732564e65e6
---
M asterixdb/asterix-common/src/main/java/org/apache/asterix/common/config/AsterixMetadataProperties.java
M asterixdb/asterix-common/src/main/java/org/apache/asterix/common/config/AsterixPropertiesAccessor.java
M asterixdb/asterix-server/pom.xml
A asterixdb/asterix-server/src/test/java/org/apache/asterix/server/test/NCServiceExecutionIT.java
A asterixdb/asterix-server/src/test/resources/NCServiceExecutionIT/cc.conf
A asterixdb/asterix-server/src/test/resources/NCServiceExecutionIT/ncservice1.conf
A asterixdb/asterix-server/src/test/resources/NCServiceExecutionIT/ncservice2.conf
M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/ClusterControllerService.java
M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/work/TriggerNCWork.java
M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/application/IniApplicationConfig.java
M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/CCConfig.java
M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/IniUtils.java
M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/NCConfig.java
M hyracks-fullstack/hyracks/hyracks-control/hyracks-nc-service/src/main/java/org/apache/hyracks/control/nc/service/NCService.java
M hyracks-fullstack/hyracks/hyracks-server/pom.xml
D hyracks-fullstack/hyracks/hyracks-server/src/main/java/org/apache/hyracks/server/drivers/VirtualClusterDriver.java
M hyracks-fullstack/hyracks/hyracks-server/src/main/java/org/apache/hyracks/server/process/HyracksCCProcess.java
M hyracks-fullstack/hyracks/hyracks-server/src/main/java/org/apache/hyracks/server/process/HyracksNCProcess.java
M hyracks-fullstack/hyracks/hyracks-server/src/main/java/org/apache/hyracks/server/process/HyracksServerProcess.java
A hyracks-fullstack/hyracks/hyracks-server/src/main/java/org/apache/hyracks/server/process/HyracksVirtualCluster.java
M hyracks-fullstack/hyracks/hyracks-server/src/test/java/org/apache/hyracks/server/test/NCServiceIT.java
M hyracks-fullstack/hyracks/hyracks-server/src/test/resources/logging.properties
22 files changed, 536 insertions(+), 369 deletions(-)
git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/58/958/1
diff --git a/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/config/AsterixMetadataProperties.java b/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/config/AsterixMetadataProperties.java
index 9a8fba4..677fc78 100644
--- a/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/config/AsterixMetadataProperties.java
+++ b/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/config/AsterixMetadataProperties.java
@@ -39,7 +39,7 @@
}
public ClusterPartition getMetadataPartition() {
- return accessor.getMetadataPartiton();
+ return accessor.getMetadataPartition();
}
public Map<String, String[]> getStores() {
diff --git a/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/config/AsterixPropertiesAccessor.java b/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/config/AsterixPropertiesAccessor.java
index 507a393..7309f0c 100644
--- a/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/config/AsterixPropertiesAccessor.java
+++ b/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/config/AsterixPropertiesAccessor.java
@@ -18,6 +18,7 @@
*/
package org.apache.asterix.common.config;
+import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
@@ -53,10 +54,12 @@
private final List<String> nodeNames = new ArrayList<>();;
private final Map<String, String[]> stores = new HashMap<>();;
private final Map<String, String> coredumpConfig = new HashMap<>();
+
+ // This can be removed when asterix-configuration.xml is no longer required.
private final Map<String, Property> asterixConfigurationParams;
private final IApplicationConfig cfg;
private final Map<String, String> transactionLogDirs = new HashMap<>();
- private final Map<String, String> asterixBuildProperties;
+ private final Map<String, String> asterixBuildProperties = new HashMap<>();
private final Map<String, ClusterPartition[]> nodePartitionsMap;
private final SortedMap<Integer, ClusterPartition> clusterPartitions = new TreeMap<>();
@@ -94,6 +97,13 @@
List<Store> configuredStores = asterixConfiguration.getStore();
nodePartitionsMap = new HashMap<>();
int uniquePartitionId = 0;
+ // Here we iterate through all <store> elements in asterix-configuration.xml.
+ // For each one, we create an array of ClusterPartitions and store this array
+ // in nodePartitionsMap, keyed by the node name. The array is the same length
+ // as the comma-separated <storeDirs> child element, because Managix will have
+ // arranged for that element to be populated with the full paths to each
+ // partition directory (as formed by appending the <store> subdirectory to
+ // each <iodevices> path from the user's original cluster.xml).
for (Store store : configuredStores) {
String trimmedStoreDirs = store.getStoreDirs().trim();
String[] nodeStores = trimmedStoreDirs.split(",");
@@ -117,35 +127,29 @@
for (TransactionLogDir txnLogDir : asterixConfiguration.getTransactionLogDir()) {
transactionLogDirs.put(txnLogDir.getNcId(), txnLogDir.getTxnLogDirPath());
}
- Properties gitProperties = new Properties();
- try {
- gitProperties.load(getClass().getClassLoader().getResourceAsStream("git.properties"));
- asterixBuildProperties = new HashMap<String, String>();
- for (final String name : gitProperties.stringPropertyNames()) {
- asterixBuildProperties.put(name, gitProperties.getProperty(name));
- }
- } catch (IOException e) {
- throw new AsterixException(e);
- }
+ loadAsterixBuildProperties();
}
/**
* Constructor which wraps an IApplicationConfig.
*/
- public AsterixPropertiesAccessor(IApplicationConfig cfg) {
+ public AsterixPropertiesAccessor(IApplicationConfig cfg) throws AsterixException {
this.cfg = cfg;
instanceName = cfg.getString("asterix", "instance", "DEFAULT_INSTANCE");
String mdNode = null;
nodePartitionsMap = new HashMap<>();
int uniquePartitionId = 0;
+
+ // Iterate through each configured NC.
for (String section : cfg.getSections()) {
if (!section.startsWith("nc/")) {
continue;
}
String ncId = section.substring(3);
+ // Here we figure out which is the metadata node. If any NCs
+ // declare "metadata.port", use that one; otherwise just use the first.
if (mdNode == null) {
- // Default is first node == metadata node
mdNode = ncId;
}
if (cfg.getString(section, "metadata.port") != null) {
@@ -153,27 +157,46 @@
mdNode = ncId;
}
+ // Now we assign the coredump and txnlog directories for this node.
// QQQ Default values? Should they be specified here? Or should there
- // be a default.ini? They can't be inserted by TriggerNCWork except
- // possibly for hyracks-specified values. Certainly wherever they are,
- // they should be platform-dependent.
+ // be a default.ini? Certainly wherever they are, they should be platform-dependent.
coredumpConfig.put(ncId, cfg.getString(section, "coredumpdir", "/var/lib/asterixdb/coredump"));
transactionLogDirs.put(ncId, cfg.getString(section, "txnlogdir", "/var/lib/asterixdb/txn-log"));
- String[] storeDirs = cfg.getString(section, "storagedir", "storage").trim().split(",");
- ClusterPartition[] nodePartitions = new ClusterPartition[storeDirs.length];
+
+ // Now we create an array of ClusterPartitions for all the partitions
+ // on this NC.
+ String[] iodevices = cfg.getString(section, "iodevices", "/var/lib/asterixdb/iodevice").split(",");
+ String storageSubdir = cfg.getString(section, "storagedir", "storage");
+ String[] nodeStores = new String[iodevices.length];
+ ClusterPartition[] nodePartitions = new ClusterPartition[iodevices.length];
for (int i = 0; i < nodePartitions.length; i++) {
+ // Construct final storage path from iodevice dir + storage subdir.
+ nodeStores[i] = iodevices[i] + File.separator + storageSubdir;
+ // Create ClusterPartition instances for this NC.
ClusterPartition partition = new ClusterPartition(uniquePartitionId++, ncId, i);
clusterPartitions.put(partition.getPartitionId(), partition);
nodePartitions[i] = partition;
}
- stores.put(ncId, storeDirs);
+ stores.put(ncId, nodeStores);
nodePartitionsMap.put(ncId, nodePartitions);
nodeNames.add(ncId);
}
metadataNodeName = mdNode;
asterixConfigurationParams = null;
- asterixBuildProperties = null;
+ loadAsterixBuildProperties();
+ }
+
+ private void loadAsterixBuildProperties() throws AsterixException {
+ Properties gitProperties = new Properties();
+ try {
+ gitProperties.load(getClass().getClassLoader().getResourceAsStream("git.properties"));
+ for (final String name : gitProperties.stringPropertyNames()) {
+ asterixBuildProperties.put(name, gitProperties.getProperty(name));
+ }
+ } catch (IOException e) {
+ throw new AsterixException(e);
+ }
}
public String getMetadataNodeName() {
@@ -204,20 +227,6 @@
return asterixBuildProperties;
}
- public void putCoredumpPaths(String nodeId, String coredumpPath) {
- if (coredumpConfig.containsKey(nodeId)) {
- throw new IllegalStateException("Cannot override value for coredump path");
- }
- coredumpConfig.put(nodeId, coredumpPath);
- }
-
- public void putTransactionLogDir(String nodeId, String txnLogDir) {
- if (transactionLogDirs.containsKey(nodeId)) {
- throw new IllegalStateException("Cannot override value for txnLogDir");
- }
- transactionLogDirs.put(nodeId, txnLogDir);
- }
-
public <T> T getProperty(String property, T defaultValue, IPropertyInterpreter<T> interpreter) {
String value;
Property p = null;
@@ -246,18 +255,11 @@
}
}
- private static <T> void logConfigurationError(Property p, T defaultValue) {
- if (LOGGER.isLoggable(Level.SEVERE)) {
- LOGGER.severe("Invalid property value '" + p.getValue() + "' for property '" + p.getName()
- + "'.\n See the description: \n" + p.getDescription() + "\nDefault = " + defaultValue);
- }
- }
-
public String getInstanceName() {
return instanceName;
}
- public ClusterPartition getMetadataPartiton() {
+ public ClusterPartition getMetadataPartition() {
// metadata partition is always the first partition on the metadata node
return nodePartitionsMap.get(metadataNodeName)[0];
}
diff --git a/asterixdb/asterix-server/pom.xml b/asterixdb/asterix-server/pom.xml
index 812fd59..92fde73 100644
--- a/asterixdb/asterix-server/pom.xml
+++ b/asterixdb/asterix-server/pom.xml
@@ -125,6 +125,24 @@
</executions>
</plugin>
<plugin>
+ <artifactId>maven-antrun-plugin</artifactId>
+ <version>1.6</version>
+ <executions>
+ <execution>
+ <id>process-test-classes</id>
+ <phase>package</phase>
+ <configuration>
+ <target>
+ <chmod file="target/appassembler/bin/*" perm="755" />
+ </target>
+ </configuration>
+ <goals>
+ <goal>run</goal>
+ </goals>
+ </execution>
+ </executions>
+ </plugin>
+ <plugin>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.2-beta-5</version>
<executions>
@@ -181,11 +199,37 @@
<scope>compile</scope>
</dependency>
<dependency>
+ <groupId>org.apache.hyracks</groupId>
+ <artifactId>hyracks-server</artifactId>
+ <type>jar</type>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
<groupId>org.apache.asterix</groupId>
<artifactId>asterix-app</artifactId>
<version>0.8.9-SNAPSHOT</version>
</dependency>
<dependency>
+ <groupId>org.apache.asterix</groupId>
+ <artifactId>asterix-app</artifactId>
+ <version>0.8.9-SNAPSHOT</version>
+ <type>test-jar</type>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.asterix</groupId>
+ <artifactId>asterix-common</artifactId>
+ <version>0.8.9-SNAPSHOT</version>
+ <type>test-jar</type>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.asterix</groupId>
+ <artifactId>asterix-test-framework</artifactId>
+ <version>0.8.9-SNAPSHOT</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
<groupId>org.codehaus.mojo.appassembler</groupId>
<artifactId>appassembler-booter</artifactId>
<version>1.3.1</version>
diff --git a/asterixdb/asterix-server/src/test/java/org/apache/asterix/server/test/NCServiceExecutionIT.java b/asterixdb/asterix-server/src/test/java/org/apache/asterix/server/test/NCServiceExecutionIT.java
new file mode 100644
index 0000000..c179103
--- /dev/null
+++ b/asterixdb/asterix-server/src/test/java/org/apache/asterix/server/test/NCServiceExecutionIT.java
@@ -0,0 +1,186 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.asterix.server.test;
+
+import org.apache.asterix.test.aql.TestExecutor;
+import org.apache.asterix.test.runtime.HDFSCluster;
+import org.apache.asterix.testframework.context.TestCaseContext;
+import org.apache.asterix.testframework.xml.TestGroup;
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hyracks.server.process.HyracksVirtualCluster;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.FixMethodOrder;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.MethodSorters;
+import org.junit.runners.Parameterized;
+import org.junit.runners.Parameterized.Parameters;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.logging.Logger;
+
+@FixMethodOrder(MethodSorters.NAME_ASCENDING)
+@RunWith(Parameterized.class)
+public class NCServiceExecutionIT {
+
+ // Important paths and files for this test.
+
+ // The "target" subdirectory of asterix-server. All outputs go here.
+ private static final String TARGET_DIR = StringUtils
+ .join(new String[] { System.getProperty("basedir"), "target" }, File.separator);
+
+ // Directory where the NCs create and store all data, as configured by
+ // src/test/resources/NCServiceExecutionIT/cc.conf.
+ private static final String INSTANCE_DIR = StringUtils
+ .join(new String[] { TARGET_DIR, "tmp" }, File.separator);
+
+ // The log directory, where all CC, NCService, and NC logs are written. CC and
+ // NCService logs are configured on the HyracksVirtualCluster below. NC logs
+ // are configured in src/test/resources/NCServiceExecutionIT/ncservice*.conf.
+ private static final String LOG_DIR = StringUtils
+ .join(new String[] { TARGET_DIR, "failsafe-reports" }, File.separator);
+
+ // Directory where *.conf files are located.
+ private static final String CONF_DIR = StringUtils
+ .join(new String[] { TARGET_DIR, "test-classes", "NCServiceExecutionIT" },
+ File.separator);
+
+ // The app.home specified for HyracksVirtualCluster. The NCService expects
+ // to find the NC startup script in ${app.home}/bin.
+ private static final String APP_HOME = StringUtils
+ .join(new String[] { TARGET_DIR, "appassembler" }, File.separator);
+
+ // Path to the asterix-app directory. This is used as the current working
+ // directory for the CC and NCService processes, which allows relative file
+ // paths in "load" statements in test queries to find the right data. It is
+ // also used for HDFSCluster.
+ private static final String ASTERIX_APP_DIR = StringUtils
+ .join(new String[] { System.getProperty("basedir"), "..", "asterix-app" },
+ File.separator);
+
+ // Path to the actual AQL test files, which we borrow from asterix-app. This is
+ // passed to TestExecutor.
+ protected static final String TESTS_DIR = StringUtils
+ .join(new String[] { ASTERIX_APP_DIR, "src", "test", "resources", "runtimets" },
+ File.separator);
+
+ // Path that actual results are written to. We create and clean this directory
+ // here, and also pass it to TestExecutor which writes the test output there.
+ private static final String ACTUAL_RESULTS_DIR = StringUtils
+ .join(new String[] { TARGET_DIR, "ittest" }, File.separator);
+
+ private static final Logger LOGGER = Logger.getLogger(NCServiceExecutionIT.class.getName());
+
+ private final TestCaseContext tcCtx;
+ private static final TestExecutor testExecutor = new TestExecutor();
+ private static HyracksVirtualCluster cluster;
+
+ @BeforeClass
+ public static void setUp() throws Exception {
+ // Create actual-results output directory.
+ File outDir = new File(ACTUAL_RESULTS_DIR);
+ outDir.mkdirs();
+
+ // Remove any instance data from previous runs.
+ File instanceDir = new File(INSTANCE_DIR);
+ if (instanceDir.isDirectory()) {
+ FileUtils.deleteDirectory(instanceDir);
+ }
+
+ // HDFSCluster requires the input directory to end with a file separator.
+ HDFSCluster.getInstance().setup(ASTERIX_APP_DIR + File.separator);
+
+ cluster = new HyracksVirtualCluster(new File(APP_HOME), new File(ASTERIX_APP_DIR));
+ cluster.addNC(
+ new File(CONF_DIR, "ncservice1.conf"),
+ new File(LOG_DIR, "ncservice1.log")
+ );
+ cluster.addNC(
+ new File(CONF_DIR, "ncservice2.conf"),
+ new File(LOG_DIR, "ncservice2.log")
+ );
+
+ try {
+ Thread.sleep(2000);
+ }
+ catch (InterruptedException ignored) {
+ }
+
+ // Start CC
+ cluster.start(
+ new File(CONF_DIR, "cc.conf"),
+ new File(LOG_DIR, "cc.log")
+ );
+
+ LOGGER.info("Sleeping while cluster comes online...");
+ try {
+ Thread.sleep(6000);
+ }
+ catch (InterruptedException ignored) {
+ }
+ }
+
+ @AfterClass
+ public static void tearDown() throws Exception {
+ File outdir = new File(ACTUAL_RESULTS_DIR);
+ File[] files = outdir.listFiles();
+ if (files == null || files.length == 0) {
+ outdir.delete();
+ }
+ cluster.stop();
+ HDFSCluster.getInstance().cleanup();
+ }
+
+ @Parameters
+ public static Collection<Object[]> tests() throws Exception {
+ Collection<Object[]> testArgs = new ArrayList<Object[]>();
+ TestCaseContext.Builder b = new TestCaseContext.Builder();
+ for (TestCaseContext ctx : b.build(new File(TESTS_DIR))) {
+ if (!skip(ctx)) {
+ testArgs.add(new Object[]{ctx});
+ }
+ }
+ return testArgs;
+ }
+
+ private static boolean skip(TestCaseContext tcCtx) {
+ // For now we skip feeds tests and external-library tests.
+ for (TestGroup group : tcCtx.getTestGroups()) {
+ if (group.getName().startsWith("external-") || group.getName().equals("feeds")) {
+ LOGGER.info("Skipping test: " + tcCtx.toString());
+ return true;
+ }
+ }
+ return false;
+ }
+
+
+ public NCServiceExecutionIT(TestCaseContext ctx) {
+ this.tcCtx = ctx;
+ }
+
+ @Test
+ public void test() throws Exception {
+ testExecutor.executeTest(ACTUAL_RESULTS_DIR, tcCtx, null, false);
+ }
+}
diff --git a/asterixdb/asterix-server/src/test/resources/NCServiceExecutionIT/cc.conf b/asterixdb/asterix-server/src/test/resources/NCServiceExecutionIT/cc.conf
new file mode 100644
index 0000000..c4c76e6
--- /dev/null
+++ b/asterixdb/asterix-server/src/test/resources/NCServiceExecutionIT/cc.conf
@@ -0,0 +1,25 @@
+[nc/asterix_nc1]
+txnlogdir=../asterix-server/target/tmp/asterix_nc1/txnlog
+coredumpdir=../asterix-server/target/tmp/asterix_nc1/coredump
+iodevices=../asterix-server/target/tmp/asterix_nc1/iodevice1,../asterix-server/target/tmp/asterix_nc1/iodevice2
+
+[nc/asterix_nc2]
+port=9091
+txnlogdir=../asterix-server/target/tmp/asterix_nc2/txnlog
+coredumpdir=../asterix-server/target/tmp/asterix_nc2/coredump
+iodevices=../asterix-server/target/tmp/asterix_nc2/iodevice1,../asterix-server/target/tmp/asterix_nc2/iodevice2
+
+[nc]
+address=127.0.0.1
+command=asterixnc
+app.class=org.apache.asterix.hyracks.bootstrap.NCApplicationEntryPoint
+jvm.args=-Xmx4096m -Dnode.Resolver="org.apache.asterix.external.util.IdentitiyResolverFactory"
+storagedir=test_storage
+
+[cc]
+cluster.address = 127.0.0.1
+app.class=org.apache.asterix.hyracks.bootstrap.CCApplicationEntryPoint
+
+[asterix]
+storage.memorycomponent.globalbudget = 1073741824
+
diff --git a/asterixdb/asterix-server/src/test/resources/NCServiceExecutionIT/ncservice1.conf b/asterixdb/asterix-server/src/test/resources/NCServiceExecutionIT/ncservice1.conf
new file mode 100644
index 0000000..fa44fa2
--- /dev/null
+++ b/asterixdb/asterix-server/src/test/resources/NCServiceExecutionIT/ncservice1.conf
@@ -0,0 +1,3 @@
+[ncservice]
+logdir=../asterix-server/target/failsafe-reports
+
diff --git a/asterixdb/asterix-server/src/test/resources/NCServiceExecutionIT/ncservice2.conf b/asterixdb/asterix-server/src/test/resources/NCServiceExecutionIT/ncservice2.conf
new file mode 100644
index 0000000..53d8d9b
--- /dev/null
+++ b/asterixdb/asterix-server/src/test/resources/NCServiceExecutionIT/ncservice2.conf
@@ -0,0 +1,4 @@
+[ncservice]
+logdir=../asterix-server/target/failsafe-reports
+port=9091
+
diff --git a/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/ClusterControllerService.java b/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/ClusterControllerService.java
index 2aa3f37..4cddef1 100644
--- a/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/ClusterControllerService.java
+++ b/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/ClusterControllerService.java
@@ -276,7 +276,7 @@
continue;
}
String ncid = section.substring(3);
- String address = ini.get(section, "address");
+ String address = IniUtils.getString(ini, section, "address", null);
int port = IniUtils.getInt(ini, section, "port", 9090);
if (address == null) {
address = InetAddress.getLoopbackAddress().getHostAddress();
diff --git a/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/work/TriggerNCWork.java b/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/work/TriggerNCWork.java
index ee79d38..7d2ff25 100644
--- a/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/work/TriggerNCWork.java
+++ b/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/work/TriggerNCWork.java
@@ -87,37 +87,15 @@
}
/**
- * Utility routine to copy all keys from a named section in Ini a
- * to a named section in Ini b. We need to do this the hard way
- * because Ini4j reacts inscrutably when attempting to copy
- * Ini.Sections directly from one Ini to another.
- */
- private void copyIniSection(Ini a, String asect, Ini b, String bsect) {
- Ini.Section source = a.get(asect);
- for (String key : source.keySet()) {
- b.put(bsect, key, source.get(key));
- }
- }
- /**
* Given an Ini object, serialize it to String with some enhancements.
* @param ccini
*/
String serializeIni(Ini ccini) throws IOException {
- Ini ini = new Ini();
-
- // First copy the global [nc] section to a new section named for
- // *this* NC, so that those values serve as defaults.
- String ncsection = "nc/" + ncId;
- copyIniSection(ccini, "nc", ini, ncsection);
- // Now copy all sections to their same name in the derived config.
- for (String section : ccini.keySet()) {
- copyIniSection(ccini, section, ini, section);
- }
+ StringWriter iniString = new StringWriter();
+ ccini.store(iniString);
// Finally insert *this* NC's name into localnc section - this is a fixed
// entry point so that NCs can determine where all their config is.
- ini.put("localnc", "id", ncId);
- StringWriter iniString = new StringWriter();
- ini.store(iniString);
+ iniString.append("\n[localnc]\nid=" + ncId + "\n");
if (LOGGER.isLoggable(Level.FINE)) {
LOGGER.fine("Returning Ini file:\n" + iniString.toString());
}
diff --git a/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/application/IniApplicationConfig.java b/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/application/IniApplicationConfig.java
index 3a8a2de..22fe318 100644
--- a/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/application/IniApplicationConfig.java
+++ b/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/application/IniApplicationConfig.java
@@ -19,6 +19,7 @@
package org.apache.hyracks.control.common.application;
import org.apache.hyracks.api.application.IApplicationConfig;
+import org.apache.hyracks.control.common.controllers.IniUtils;
import org.ini4j.Ini;
import java.util.Set;
@@ -37,39 +38,34 @@
}
}
- private <T> T getIniValue(String section, String key, T default_value, Class<T> clazz) {
- T value = ini.get(section, key, clazz);
- return (value != null) ? value : default_value;
- }
-
@Override
public String getString(String section, String key) {
- return getIniValue(section, key, null, String.class);
+ return IniUtils.getString(ini, section, key, null);
}
@Override
public String getString(String section, String key, String defaultValue) {
- return getIniValue(section, key, defaultValue, String.class);
+ return IniUtils.getString(ini, section, key, defaultValue);
}
@Override
public int getInt(String section, String key) {
- return getIniValue(section, key, 0, Integer.class);
+ return IniUtils.getInt(ini, section, key, 0);
}
@Override
public int getInt(String section, String key, int defaultValue) {
- return getIniValue(section, key, defaultValue, Integer.class);
+ return IniUtils.getInt(ini, section, key, defaultValue);
}
@Override
public long getLong(String section, String key) {
- return getIniValue(section, key, (long) 0, Long.class);
+ return IniUtils.getLong(ini, section, key, (long) 0);
}
@Override
public long getLong(String section, String key, long defaultValue) {
- return getIniValue(section, key, defaultValue, Long.class);
+ return IniUtils.getLong(ini, section, key, defaultValue);
}
@Override
diff --git a/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/CCConfig.java b/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/CCConfig.java
index a04d750..64bd7d1 100644
--- a/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/CCConfig.java
+++ b/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/CCConfig.java
@@ -152,47 +152,4 @@
public IApplicationConfig getAppConfig() {
return new IniApplicationConfig(ini);
}
-
- public void toCommandLine(List<String> cList) {
- cList.add("-client-net-ip-address");
- cList.add(clientNetIpAddress);
- cList.add("-client-net-port");
- cList.add(String.valueOf(clientNetPort));
- cList.add("-cluster-net-ip-address");
- cList.add(clusterNetIpAddress);
- cList.add("-cluster-net-port");
- cList.add(String.valueOf(clusterNetPort));
- cList.add("-http-port");
- cList.add(String.valueOf(httpPort));
- cList.add("-heartbeat-period");
- cList.add(String.valueOf(heartbeatPeriod));
- cList.add("-max-heartbeat-lapse-periods");
- cList.add(String.valueOf(maxHeartbeatLapsePeriods));
- cList.add("-profile-dump-period");
- cList.add(String.valueOf(profileDumpPeriod));
- cList.add("-default-max-job-attempts");
- cList.add(String.valueOf(defaultMaxJobAttempts));
- cList.add("-job-history-size");
- cList.add(String.valueOf(jobHistorySize));
- cList.add("-result-time-to-live");
- cList.add(String.valueOf(resultTTL));
- cList.add("-result-sweep-threshold");
- cList.add(String.valueOf(resultSweepThreshold));
- cList.add("-cc-root");
- cList.add(ccRoot);
- if (clusterTopologyDefinition != null) {
- cList.add("-cluster-topology");
- cList.add(clusterTopologyDefinition.getAbsolutePath());
- }
- if (appCCMainClass != null) {
- cList.add("-app-cc-main-class");
- cList.add(appCCMainClass);
- }
- if (appArgs != null && !appArgs.isEmpty()) {
- cList.add("--");
- for (String appArg : appArgs) {
- cList.add(appArg);
- }
- }
- }
}
diff --git a/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/IniUtils.java b/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/IniUtils.java
index 9a5c9a0..538bb0b 100644
--- a/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/IniUtils.java
+++ b/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/IniUtils.java
@@ -26,21 +26,39 @@
/**
* Some utility functions for reading Ini4j objects with default values.
+ * For all getXxx() methods: if the 'section' contains a slash, and the 'key'
+ * is not found in that section, we will search for the key in the section named
+ * by stripping the leaf of the section name (final slash and anything following).
+ * eg. getInt(ini, "nc/red", "dir", null) will first look for the key "dir" in
+ * the section "nc/red", but if it is not found, will look in the section "nc".
*/
public class IniUtils {
+ private static <T> T getIniValue(Ini ini, String section, String key, T default_value, Class<T> clazz) {
+ T value;
+ while (true) {
+ value = ini.get(section, key, clazz);
+ if (value == null) {
+ int idx = section.lastIndexOf('/');
+ if (idx > -1) {
+ section = section.substring(0, idx);
+ continue;
+ }
+ }
+ break;
+ }
+ return (value != null) ? value : default_value;
+ }
+
public static String getString(Ini ini, String section, String key, String defaultValue) {
- String value = ini.get(section, key, String.class);
- return (value != null) ? value : defaultValue;
+ return getIniValue(ini, section, key, defaultValue, String.class);
}
public static int getInt(Ini ini, String section, String key, int defaultValue) {
- Integer value = ini.get(section, key, Integer.class);
- return (value != null) ? value : defaultValue;
+ return getIniValue(ini, section, key, defaultValue, Integer.class);
}
public static long getLong(Ini ini, String section, String key, long defaultValue) {
- Long value = ini.get(section, key, Long.class);
- return (value != null) ? value : defaultValue;
+ return getIniValue(ini, section, key, defaultValue, Long.class);
}
public static Ini loadINIFile(String configFile) throws IOException {
diff --git a/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/NCConfig.java b/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/NCConfig.java
index b408083..d08df60 100644
--- a/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/NCConfig.java
+++ b/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/NCConfig.java
@@ -191,66 +191,6 @@
return new IniApplicationConfig(ini);
}
- public void toCommandLine(List<String> cList) {
- cList.add("-cc-host");
- cList.add(ccHost);
- cList.add("-cc-port");
- cList.add(String.valueOf(ccPort));
- cList.add("-cluster-net-ip-address");
- cList.add(clusterNetIPAddress);
- cList.add("-cluster-net-port");
- cList.add(String.valueOf(clusterNetPort));
- cList.add("-cluster-net-public-ip-address");
- cList.add(clusterNetPublicIPAddress);
- cList.add("-cluster-net-public-port");
- cList.add(String.valueOf(clusterNetPublicPort));
- cList.add("-node-id");
- cList.add(nodeId);
- cList.add("-data-ip-address");
- cList.add(dataIPAddress);
- cList.add("-data-port");
- cList.add(String.valueOf(dataPort));
- cList.add("-data-public-ip-address");
- cList.add(dataPublicIPAddress);
- cList.add("-data-public-port");
- cList.add(String.valueOf(dataPublicPort));
- cList.add("-result-ip-address");
- cList.add(resultIPAddress);
- cList.add("-result-port");
- cList.add(String.valueOf(resultPort));
- cList.add("-result-public-ip-address");
- cList.add(resultPublicIPAddress);
- cList.add("-result-public-port");
- cList.add(String.valueOf(resultPublicPort));
- cList.add("-retries");
- cList.add(String.valueOf(retries));
- cList.add("-iodevices");
- cList.add(ioDevices);
- cList.add("-net-thread-count");
- cList.add(String.valueOf(nNetThreads));
- cList.add("-net-buffer-count");
- cList.add(String.valueOf(nNetBuffers));
- cList.add("-max-memory");
- cList.add(String.valueOf(maxMemory));
- cList.add("-result-time-to-live");
- cList.add(String.valueOf(resultTTL));
- cList.add("-result-sweep-threshold");
- cList.add(String.valueOf(resultSweepThreshold));
- cList.add("-result-manager-memory");
- cList.add(String.valueOf(resultManagerMemory));
-
- if (appNCMainClass != null) {
- cList.add("-app-nc-main-class");
- cList.add(appNCMainClass);
- }
- if (appArgs != null && !appArgs.isEmpty()) {
- cList.add("--");
- for (String appArg : appArgs) {
- cList.add(appArg);
- }
- }
- }
-
public void toMap(Map<String, String> configuration) {
configuration.put("cc-host", ccHost);
configuration.put("cc-port", (String.valueOf(ccPort)));
diff --git a/hyracks-fullstack/hyracks/hyracks-control/hyracks-nc-service/src/main/java/org/apache/hyracks/control/nc/service/NCService.java b/hyracks-fullstack/hyracks/hyracks-control/hyracks-nc-service/src/main/java/org/apache/hyracks/control/nc/service/NCService.java
index e3fe959..4102b4c 100644
--- a/hyracks-fullstack/hyracks/hyracks-control/hyracks-nc-service/src/main/java/org/apache/hyracks/control/nc/service/NCService.java
+++ b/hyracks-fullstack/hyracks/hyracks-control/hyracks-nc-service/src/main/java/org/apache/hyracks/control/nc/service/NCService.java
@@ -19,6 +19,7 @@
package org.apache.hyracks.control.nc.service;
import org.apache.commons.lang3.SystemUtils;
+import org.apache.hyracks.control.common.controllers.IniUtils;
import org.ini4j.Ini;
import org.kohsuke.args4j.CmdLineParser;
@@ -71,23 +72,13 @@
private static final String MAGIC_COOKIE = "hyncmagic";
- private static String getStringINIOpt(Ini ini, String section, String key, String default_value) {
- String value = ini.get(section, key, String.class);
- return (value != null) ? value : default_value;
- }
-
- private static int getIntINIOpt(Ini ini, String section, String key, int default_value) {
- Integer value = ini.get(section, key, Integer.class);
- return (value != null) ? value : default_value;
- }
-
private static List<String> buildCommand() throws IOException {
List<String> cList = new ArrayList<String>();
// Find the command to run. For now, we allow overriding the name, but
// still assume it's located in the bin/ directory of the deployment.
// Even this is likely more configurability than we need.
- String command = getStringINIOpt(ini, nodeSection, "command", "hyracksnc");
+ String command = IniUtils.getString(ini, nodeSection, "command", "hyracksnc");
// app.home is specified by the Maven appassembler plugin. If it isn't set,
// fall back to user's home dir. Again this is likely more flexibility
// than we need.
@@ -110,10 +101,16 @@
private static void configEnvironment(Map<String,String> env) {
if (env.containsKey("JAVA_OPTS")) {
+ if (LOGGER.isLoggable(Level.INFO)) {
+ LOGGER.info("Keeping JAVA_OPTS from environment");
+ }
return;
}
- String jvmargs = getStringINIOpt(ini, nodeSection, "jvm.args", "-Xmx1536m");
+ String jvmargs = IniUtils.getString(ini, nodeSection, "jvm.args", "-Xmx1536m");
env.put("JAVA_OPTS", jvmargs);
+ if (LOGGER.isLoggable(Level.INFO)) {
+ LOGGER.info("Setting JAVA_OPTS to " + jvmargs);
+ }
}
/**
@@ -146,6 +143,8 @@
// If the directory IS there, all is well
}
File logfile = new File(config.logdir, "nc-" + ncId + ".log");
+ // Don't care if this succeeds or fails:
+ logfile.delete();
pb.redirectOutput(ProcessBuilder.Redirect.appendTo(logfile));
if (LOGGER.isLoggable(Level.INFO)) {
LOGGER.info("Logging to " + logfile.getCanonicalPath());
@@ -192,7 +191,7 @@
}
String iniString = ois.readUTF();
ini = new Ini(new StringReader(iniString));
- ncId = getStringINIOpt(ini, "localnc", "id", "");
+ ncId = IniUtils.getString(ini, "localnc", "id", "");
nodeSection = "nc/" + ncId;
return launchNCProcess();
} catch (Exception e) {
diff --git a/hyracks-fullstack/hyracks/hyracks-server/pom.xml b/hyracks-fullstack/hyracks/hyracks-server/pom.xml
index 3bda1d3..52958f8 100644
--- a/hyracks-fullstack/hyracks/hyracks-server/pom.xml
+++ b/hyracks-fullstack/hyracks/hyracks-server/pom.xml
@@ -86,10 +86,6 @@
<mainClass>org.apache.hyracks.control.nc.service.NCService</mainClass>
<name>hyracksncservice</name>
</program>
- <program>
- <mainClass>org.apache.hyracks.server.drivers.VirtualClusterDriver</mainClass>
- <name>hyracks-virtual-cluster</name>
- </program>
</programs>
<repositoryLayout>flat</repositoryLayout>
<repositoryName>lib</repositoryName>
diff --git a/hyracks-fullstack/hyracks/hyracks-server/src/main/java/org/apache/hyracks/server/drivers/VirtualClusterDriver.java b/hyracks-fullstack/hyracks/hyracks-server/src/main/java/org/apache/hyracks/server/drivers/VirtualClusterDriver.java
deleted file mode 100644
index 41c14a7..0000000
--- a/hyracks-fullstack/hyracks/hyracks-server/src/main/java/org/apache/hyracks/server/drivers/VirtualClusterDriver.java
+++ /dev/null
@@ -1,82 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.hyracks.server.drivers;
-
-import org.kohsuke.args4j.CmdLineParser;
-import org.kohsuke.args4j.Option;
-
-import org.apache.hyracks.control.common.controllers.CCConfig;
-import org.apache.hyracks.control.common.controllers.NCConfig;
-import org.apache.hyracks.server.process.HyracksCCProcess;
-import org.apache.hyracks.server.process.HyracksNCProcess;
-
-public class VirtualClusterDriver {
- private static class Options {
- @Option(name = "-n", required = false, usage = "Number of node controllers (default: 2)")
- public int n = 2;
-
- @Option(name = "-cc-client-net-port", required = false, usage = "CC Port (default: 1098)")
- public int ccClientNetPort = 1098;
-
- @Option(name = "-cc-cluster-net-port", required = false, usage = "CC Port (default: 1099)")
- public int ccClusterNetPort = 1099;
-
- @Option(name = "-cc-http-port", required = false, usage = "CC Port (default: 16001)")
- public int ccHttpPort = 16001;
- }
-
- public static void main(String[] args) throws Exception {
- Options options = new Options();
- CmdLineParser cp = new CmdLineParser(options);
- try {
- cp.parseArgument(args);
- } catch (Exception e) {
- System.err.println(e.getMessage());
- cp.printUsage(System.err);
- return;
- }
-
- CCConfig ccConfig = new CCConfig();
- ccConfig.clusterNetIpAddress = "127.0.0.1";
- ccConfig.clusterNetPort = options.ccClusterNetPort;
- ccConfig.clientNetIpAddress = "127.0.0.1";
- ccConfig.clientNetPort = options.ccClientNetPort;
- ccConfig.httpPort = options.ccHttpPort;
- HyracksCCProcess ccp = new HyracksCCProcess(ccConfig);
- ccp.start();
-
- Thread.sleep(5000);
-
- HyracksNCProcess ncps[] = new HyracksNCProcess[options.n];
- for (int i = 0; i < options.n; ++i) {
- NCConfig ncConfig = new NCConfig();
- ncConfig.ccHost = "127.0.0.1";
- ncConfig.ccPort = options.ccClusterNetPort;
- ncConfig.clusterNetIPAddress = "127.0.0.1";
- ncConfig.nodeId = "nc" + i;
- ncConfig.dataIPAddress = "127.0.0.1";
- ncps[i] = new HyracksNCProcess(ncConfig);
- ncps[i].start();
- }
-
- while (true) {
- Thread.sleep(10000);
- }
- }
-}
diff --git a/hyracks-fullstack/hyracks/hyracks-server/src/main/java/org/apache/hyracks/server/process/HyracksCCProcess.java b/hyracks-fullstack/hyracks/hyracks-server/src/main/java/org/apache/hyracks/server/process/HyracksCCProcess.java
index d0d8d63..4a70120 100644
--- a/hyracks-fullstack/hyracks/hyracks-server/src/main/java/org/apache/hyracks/server/process/HyracksCCProcess.java
+++ b/hyracks-fullstack/hyracks/hyracks-server/src/main/java/org/apache/hyracks/server/process/HyracksCCProcess.java
@@ -18,25 +18,28 @@
*/
package org.apache.hyracks.server.process;
+import org.apache.hyracks.control.cc.CCDriver;
+
+import java.io.File;
import java.util.List;
-import org.apache.hyracks.control.cc.CCDriver;
-import org.apache.hyracks.control.common.controllers.CCConfig;
-
public class HyracksCCProcess extends HyracksServerProcess {
- private CCConfig config;
- public HyracksCCProcess(CCConfig config) {
- this.config = config;
- }
-
- @Override
- protected void addCmdLineArgs(List<String> cList) {
- config.toCommandLine(cList);
+ public HyracksCCProcess(File configFile, File logFile, File appHome, File workingDir) {
+ this.configFile = configFile;
+ this.logFile = logFile;
+ this.appHome = appHome;
+ this.workingDir = workingDir;
}
@Override
protected String getMainClassName() {
return CCDriver.class.getName();
}
+
+ @Override
+ protected void addJvmArgs(List<String> cList) {
+ // CC needs more than default memory
+ cList.add("-Xmx1024m");
+ }
}
diff --git a/hyracks-fullstack/hyracks/hyracks-server/src/main/java/org/apache/hyracks/server/process/HyracksNCProcess.java b/hyracks-fullstack/hyracks/hyracks-server/src/main/java/org/apache/hyracks/server/process/HyracksNCProcess.java
index c4517e6..8bc1694 100644
--- a/hyracks-fullstack/hyracks/hyracks-server/src/main/java/org/apache/hyracks/server/process/HyracksNCProcess.java
+++ b/hyracks-fullstack/hyracks/hyracks-server/src/main/java/org/apache/hyracks/server/process/HyracksNCProcess.java
@@ -18,25 +18,28 @@
*/
package org.apache.hyracks.server.process;
+import org.apache.hyracks.control.nc.service.NCService;
+
+import java.io.File;
import java.util.List;
-import org.apache.hyracks.control.common.controllers.NCConfig;
-import org.apache.hyracks.control.nc.NCDriver;
-
public class HyracksNCProcess extends HyracksServerProcess {
- private NCConfig config;
- public HyracksNCProcess(NCConfig config) {
- this.config = config;
- }
-
- @Override
- protected void addCmdLineArgs(List<String> cList) {
- config.toCommandLine(cList);
+ public HyracksNCProcess(File configFile, File logFile, File appHome, File workingDir) {
+ this.configFile = configFile;
+ this.logFile = logFile;
+ this.appHome = appHome;
+ this.workingDir = workingDir;
}
@Override
protected String getMainClassName() {
- return NCDriver.class.getName();
+ return NCService.class.getName();
+ }
+
+ @Override
+ protected void addJvmArgs(List<String> cList) {
+ // NCService needs little memory
+ cList.add("-Xmx128m");
}
}
diff --git a/hyracks-fullstack/hyracks/hyracks-server/src/main/java/org/apache/hyracks/server/process/HyracksServerProcess.java b/hyracks-fullstack/hyracks/hyracks-server/src/main/java/org/apache/hyracks/server/process/HyracksServerProcess.java
index 9dec0ec..13cb445 100644
--- a/hyracks-fullstack/hyracks/hyracks-server/src/main/java/org/apache/hyracks/server/process/HyracksServerProcess.java
+++ b/hyracks-fullstack/hyracks/hyracks-server/src/main/java/org/apache/hyracks/server/process/HyracksServerProcess.java
@@ -29,53 +29,69 @@
import java.util.logging.Level;
import java.util.logging.Logger;
-public abstract class HyracksServerProcess {
+abstract class HyracksServerProcess {
private static final Logger LOGGER = Logger.getLogger(HyracksServerProcess.class.getName());
protected Process process;
+ protected File configFile = null;
+ protected File logFile = null;
+ protected File appHome = null;
+ protected File workingDir = null;
public void start() throws IOException {
String[] cmd = buildCommand();
if (LOGGER.isLoggable(Level.INFO)) {
LOGGER.info("Starting command: " + Arrays.toString(cmd));
}
- process = Runtime.getRuntime().exec(cmd, null, null);
- dump(process.getInputStream());
- dump(process.getErrorStream());
+
+ ProcessBuilder pb = new ProcessBuilder(cmd);
+ pb.redirectErrorStream(true);
+ if (logFile != null) {
+ if (LOGGER.isLoggable(Level.INFO)) {
+ LOGGER.info("Logging to: " + logFile.getCanonicalPath());
+ }
+ logFile.getParentFile().mkdirs();
+ logFile.delete();
+ pb.redirectOutput(ProcessBuilder.Redirect.appendTo(logFile));
+ } else {
+ if (LOGGER.isLoggable(Level.INFO)) {
+ LOGGER.info("Logfile not set, subprocess will output to stdout");
+ }
+ }
+ pb.directory(workingDir);
+ process = pb.start();
}
- private void dump(InputStream input) {
- final int streamBufferSize = 1000;
- final Reader in = new InputStreamReader(input);
- new Thread(new Runnable() {
- public void run() {
- try {
- char[] chars = new char[streamBufferSize];
- int c;
- while ((c = in.read(chars)) != -1) {
- if (c > 0) {
- System.out.print(String.valueOf(chars, 0, c));
- }
- }
- } catch (IOException e) {
- }
- }
- }).start();
+ public void stop() {
+ process.destroy();
+ try {
+ process.waitFor();
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ }
}
private String[] buildCommand() {
List<String> cList = new ArrayList<String>();
cList.add(getJavaCommand());
- cList.add("-Dbasedir=" + System.getProperty("basedir"));
- cList.add("-Djava.rmi.server.hostname=127.0.0.1");
+ addJvmArgs(cList);
+ cList.add("-Dapp.home=" + appHome.getAbsolutePath());
cList.add("-classpath");
cList.add(getClasspath());
cList.add(getMainClassName());
+ if (configFile != null) {
+ cList.add("-config-file");
+ cList.add(configFile.getAbsolutePath());
+ }
addCmdLineArgs(cList);
return cList.toArray(new String[cList.size()]);
}
- protected abstract void addCmdLineArgs(List<String> cList);
+ protected void addJvmArgs(List<String> cList) {
+ }
+
+ protected void addCmdLineArgs(List<String> cList) {
+ }
protected abstract String getMainClassName();
@@ -83,7 +99,7 @@
return System.getProperty("java.class.path");
}
- protected final String getJavaCommand() {
+ private final String getJavaCommand() {
return System.getProperty("java.home") + File.separator + "bin" + File.separator + "java";
}
}
diff --git a/hyracks-fullstack/hyracks/hyracks-server/src/main/java/org/apache/hyracks/server/process/HyracksVirtualCluster.java b/hyracks-fullstack/hyracks/hyracks-server/src/main/java/org/apache/hyracks/server/process/HyracksVirtualCluster.java
new file mode 100644
index 0000000..f08bb43
--- /dev/null
+++ b/hyracks-fullstack/hyracks/hyracks-server/src/main/java/org/apache/hyracks/server/process/HyracksVirtualCluster.java
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hyracks.server.process;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * Starts a local hyracks-based cluster (NC and CC child processes).
+ */
+public class HyracksVirtualCluster {
+ private final File appHome;
+ private final File workingDir;
+ private List<HyracksNCProcess> ncProcs = new ArrayList<>(3);
+ private HyracksCCProcess ccProc = null;
+
+ /**
+ * Construct a Hyracks-based cluster.
+ * @param appHome - path to the installation root of the Hyracks application.
+ * At least bin/hyracksnc (or the equivalent NC script for
+ * the application) must exist in this directory.
+ * @param workingDir - directory to use as CWD for all child processes. May
+ * be null, in which case the CWD of the invoking process is used.
+ */
+ public HyracksVirtualCluster(File appHome, File workingDir) {
+ this.appHome = appHome;
+ this.workingDir = workingDir;
+ }
+
+ /**
+ * Creates and starts an NCService.
+ * @param configFile - full path to an ncservice.conf. May be null to accept all defaults.
+ * @throws IOException - if there are errors starting the process.
+ */
+ public void addNC(File configFile, File logFile) throws IOException {
+ HyracksNCProcess proc = new HyracksNCProcess(configFile, logFile, appHome, workingDir);
+ proc.start();
+ ncProcs.add(proc);
+ }
+
+ /**
+ * Starts the CC, initializing the cluster. Expects that any NCs referenced
+ * in the cluster configuration have already been started with addNC().
+ * @param ccConfigFile - full path to a cluster conf file. May be null to accept all
+ * defaults, although this is seldom useful since there are no NCs.
+ * @throws IOException - if there are errors starting the process.
+ */
+ public void start(File ccConfigFile, File logFile) throws IOException {
+ ccProc = new HyracksCCProcess(ccConfigFile, logFile, appHome, workingDir);
+ ccProc.start();
+ }
+
+ /**
+ * Stops all processes in the cluster.
+ * QQQ Someday this should probably do a graceful stop of NCs rather than
+ * killing the NCService.
+ */
+ public void stop() {
+ ccProc.stop();
+ for (HyracksNCProcess proc : ncProcs) {
+ proc.stop();
+ }
+ }
+}
+
diff --git a/hyracks-fullstack/hyracks/hyracks-server/src/test/java/org/apache/hyracks/server/test/NCServiceIT.java b/hyracks-fullstack/hyracks/hyracks-server/src/test/java/org/apache/hyracks/server/test/NCServiceIT.java
index bd99c8c..9a231a0 100644
--- a/hyracks-fullstack/hyracks/hyracks-server/src/test/java/org/apache/hyracks/server/test/NCServiceIT.java
+++ b/hyracks-fullstack/hyracks/hyracks-server/src/test/java/org/apache/hyracks/server/test/NCServiceIT.java
@@ -23,6 +23,7 @@
import org.apache.commons.httpclient.HttpStatus;
import org.apache.commons.httpclient.methods.GetMethod;
import org.apache.commons.lang3.StringUtils;
+import org.apache.hyracks.server.process.HyracksVirtualCluster;
import org.json.JSONArray;
import org.json.JSONObject;
import org.junit.AfterClass;
@@ -30,6 +31,7 @@
import org.junit.Test;
import java.io.File;
+import java.io.IOException;
import java.net.InetAddress;
import java.util.ArrayList;
import java.util.List;
@@ -38,23 +40,29 @@
public class NCServiceIT {
private static final String TARGET_DIR = StringUtils
- .join(new String[]{System.getProperty("basedir"), "target"}, File.separator);
+ .join(new String[] { System.getProperty("basedir"), "target" }, File.separator);
private static final String LOG_DIR = StringUtils
- .join(new String[]{TARGET_DIR, "surefire-reports"}, File.separator);
+ .join(new String[] { TARGET_DIR, "failsafe-reports" }, File.separator);
private static final String RESOURCE_DIR = StringUtils
- .join(new String[]{TARGET_DIR, "test-classes", "NCServiceIT"}, File.separator);
- private static final String APP_DIR = StringUtils
- .join(new String[]{TARGET_DIR, "appassembler", "bin"}, File.separator);
+ .join(new String[] { TARGET_DIR, "test-classes", "NCServiceIT" }, File.separator);
+ private static final String APP_HOME = StringUtils
+ .join(new String[] { TARGET_DIR, "appassembler" }, File.separator);
private static final Logger LOGGER = Logger.getLogger(NCServiceIT.class.getName());
- private static List<Process> procs = new ArrayList<>();
+
+ private static HyracksVirtualCluster cluster = null;
@BeforeClass
public static void setUp() throws Exception {
- // Start two NC Services - don't read their output as they don't terminate
- procs.add(invoke("nc-red.log", APP_DIR + File.separator + "hyracksncservice",
- "-config-file", RESOURCE_DIR + File.separator + "nc-red.conf"));
- procs.add(invoke("nc-blue.log", APP_DIR + File.separator + "hyracksncservice",
- "-config-file", RESOURCE_DIR + File.separator + "nc-blue.conf"));
+ cluster = new HyracksVirtualCluster(new File(APP_HOME), null);
+ cluster.addNC(
+ new File(RESOURCE_DIR, "nc-red.conf"),
+ new File(LOG_DIR, "nc-red.log")
+ );
+ cluster.addNC(
+ new File(RESOURCE_DIR, "nc-blue.conf"),
+ new File(LOG_DIR, "nc-blue.log")
+ );
+
try {
Thread.sleep(2000);
}
@@ -62,8 +70,11 @@
}
// Start CC
- procs.add(invoke("cc.log", APP_DIR + File.separator + "hyrackscc",
- "-config-file", RESOURCE_DIR + File.separator + "cc.conf"));
+ cluster.start(
+ new File(RESOURCE_DIR, "cc.conf"),
+ new File(LOG_DIR, "cc.log")
+ );
+
try {
Thread.sleep(10000);
}
@@ -72,11 +83,8 @@
}
@AfterClass
- public static void tearDown() throws Exception {
- for (Process p : procs) {
- p.destroy();
- p.waitFor();
- }
+ public static void tearDown() throws IOException {
+ cluster.stop();
}
private static String getHttp(String url) throws Exception {
@@ -95,18 +103,6 @@
} else {
throw new Exception("HTTP error " + statusCode + ":\n" + response);
}
- }
-
- private static Process invoke(String logfile, String... args) throws Exception {
- ProcessBuilder pb = new ProcessBuilder(args);
- pb.redirectErrorStream(true);
- File logDir = new File(LOG_DIR);
- logDir.mkdirs();
- File log = new File(logDir, logfile);
- log.delete();
- pb.redirectOutput(ProcessBuilder.Redirect.appendTo(log));
- Process p = pb.start();
- return p;
}
@Test
@@ -138,5 +134,4 @@
tearDown();
}
}
-
}
diff --git a/hyracks-fullstack/hyracks/hyracks-server/src/test/resources/logging.properties b/hyracks-fullstack/hyracks/hyracks-server/src/test/resources/logging.properties
index c888bb1..e9f8479 100644
--- a/hyracks-fullstack/hyracks/hyracks-server/src/test/resources/logging.properties
+++ b/hyracks-fullstack/hyracks/hyracks-server/src/test/resources/logging.properties
@@ -46,8 +46,8 @@
# Note that the ConsoleHandler also has a separate level
# setting to limit messages printed to the console.
-.level= WARNING
-# .level= INFO
+# .level= WARNING
+.level= INFO
# .level= FINE
# .level = FINEST
--
To view, visit https://asterix-gerrit.ics.uci.edu/958
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: If3eb450782a595cf85d04a2c2e9cc732564e65e6
Gerrit-PatchSet: 1
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Chris Hillery <ce...@lambda.nu>
Change in asterixdb[master]: ASTERIXDB-1482: Added NCServiceExecutionIT, HyracksVirtualCl...
Posted by "Ian Maxon (Code Review)" <do...@asterixdb.incubator.apache.org>.
Ian Maxon has posted comments on this change.
Change subject: ASTERIXDB-1482: Added NCServiceExecutionIT, HyracksVirtualCluster.
......................................................................
Patch Set 3: Code-Review+2
--
To view, visit https://asterix-gerrit.ics.uci.edu/958
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: If3eb450782a595cf85d04a2c2e9cc732564e65e6
Gerrit-PatchSet: 3
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Chris Hillery <ce...@lambda.nu>
Gerrit-Reviewer: Chris Hillery <ce...@lambda.nu>
Gerrit-Reviewer: Ian Maxon <im...@apache.org>
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-Reviewer: Till Westmann <ti...@apache.org>
Gerrit-HasComments: No
Change in asterixdb[master]: ASTERIXDB-1482: Added NCServiceExecutionIT, HyracksVirtualCl...
Posted by "Chris Hillery (Code Review)" <do...@asterixdb.incubator.apache.org>.
Chris Hillery has submitted this change and it was merged.
Change subject: ASTERIXDB-1482: Added NCServiceExecutionIT, HyracksVirtualCluster.
......................................................................
ASTERIXDB-1482: Added NCServiceExecutionIT, HyracksVirtualCluster.
NCServiceExecutionIT runs all execution tests against a local cluster
managed by the NCService deployment framework.
HyracksVirtualCluster offers programmatic NCService deployment
control along with improved HyracksNCProcess/HyracksCCProcess.
Further fixes and improvements:
1. Fix handling of iodevices/storagedir (ASTERIXDB-1482)
2. Proper handling of [nc] default section in all cases
3. Ensure asterixnc, etc. scripts are executable
4. Consolidate Ini handling
5. Pruned some dead code, including VirtualClusterDriver
6. A bit of refactoring and extended commenting
Change-Id: If3eb450782a595cf85d04a2c2e9cc732564e65e6
Reviewed-on: https://asterix-gerrit.ics.uci.edu/958
Tested-by: Jenkins <je...@fulliautomatix.ics.uci.edu>
Reviewed-by: Ian Maxon <im...@apache.org>
---
M asterixdb/asterix-common/src/main/java/org/apache/asterix/common/config/AsterixMetadataProperties.java
M asterixdb/asterix-common/src/main/java/org/apache/asterix/common/config/AsterixPropertiesAccessor.java
M asterixdb/asterix-server/pom.xml
A asterixdb/asterix-server/src/test/java/org/apache/asterix/server/test/NCServiceExecutionIT.java
A asterixdb/asterix-server/src/test/resources/NCServiceExecutionIT/cc.conf
A asterixdb/asterix-server/src/test/resources/NCServiceExecutionIT/ncservice1.conf
A asterixdb/asterix-server/src/test/resources/NCServiceExecutionIT/ncservice2.conf
M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/ClusterControllerService.java
M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/work/TriggerNCWork.java
M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/application/IniApplicationConfig.java
M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/CCConfig.java
M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/IniUtils.java
M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/NCConfig.java
M hyracks-fullstack/hyracks/hyracks-control/hyracks-nc-service/src/main/java/org/apache/hyracks/control/nc/service/NCService.java
M hyracks-fullstack/hyracks/hyracks-server/pom.xml
D hyracks-fullstack/hyracks/hyracks-server/src/main/java/org/apache/hyracks/server/drivers/VirtualClusterDriver.java
M hyracks-fullstack/hyracks/hyracks-server/src/main/java/org/apache/hyracks/server/process/HyracksCCProcess.java
M hyracks-fullstack/hyracks/hyracks-server/src/main/java/org/apache/hyracks/server/process/HyracksNCProcess.java
M hyracks-fullstack/hyracks/hyracks-server/src/main/java/org/apache/hyracks/server/process/HyracksServerProcess.java
A hyracks-fullstack/hyracks/hyracks-server/src/main/java/org/apache/hyracks/server/process/HyracksVirtualCluster.java
M hyracks-fullstack/hyracks/hyracks-server/src/test/java/org/apache/hyracks/server/test/NCServiceIT.java
M hyracks-fullstack/hyracks/hyracks-server/src/test/resources/logging.properties
22 files changed, 536 insertions(+), 369 deletions(-)
Approvals:
Ian Maxon: Looks good to me, approved
Jenkins: Verified
diff --git a/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/config/AsterixMetadataProperties.java b/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/config/AsterixMetadataProperties.java
index 9a8fba4..677fc78 100644
--- a/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/config/AsterixMetadataProperties.java
+++ b/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/config/AsterixMetadataProperties.java
@@ -39,7 +39,7 @@
}
public ClusterPartition getMetadataPartition() {
- return accessor.getMetadataPartiton();
+ return accessor.getMetadataPartition();
}
public Map<String, String[]> getStores() {
diff --git a/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/config/AsterixPropertiesAccessor.java b/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/config/AsterixPropertiesAccessor.java
index 507a393..7309f0c 100644
--- a/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/config/AsterixPropertiesAccessor.java
+++ b/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/config/AsterixPropertiesAccessor.java
@@ -18,6 +18,7 @@
*/
package org.apache.asterix.common.config;
+import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
@@ -53,10 +54,12 @@
private final List<String> nodeNames = new ArrayList<>();;
private final Map<String, String[]> stores = new HashMap<>();;
private final Map<String, String> coredumpConfig = new HashMap<>();
+
+ // This can be removed when asterix-configuration.xml is no longer required.
private final Map<String, Property> asterixConfigurationParams;
private final IApplicationConfig cfg;
private final Map<String, String> transactionLogDirs = new HashMap<>();
- private final Map<String, String> asterixBuildProperties;
+ private final Map<String, String> asterixBuildProperties = new HashMap<>();
private final Map<String, ClusterPartition[]> nodePartitionsMap;
private final SortedMap<Integer, ClusterPartition> clusterPartitions = new TreeMap<>();
@@ -94,6 +97,13 @@
List<Store> configuredStores = asterixConfiguration.getStore();
nodePartitionsMap = new HashMap<>();
int uniquePartitionId = 0;
+ // Here we iterate through all <store> elements in asterix-configuration.xml.
+ // For each one, we create an array of ClusterPartitions and store this array
+ // in nodePartitionsMap, keyed by the node name. The array is the same length
+ // as the comma-separated <storeDirs> child element, because Managix will have
+ // arranged for that element to be populated with the full paths to each
+ // partition directory (as formed by appending the <store> subdirectory to
+ // each <iodevices> path from the user's original cluster.xml).
for (Store store : configuredStores) {
String trimmedStoreDirs = store.getStoreDirs().trim();
String[] nodeStores = trimmedStoreDirs.split(",");
@@ -117,35 +127,29 @@
for (TransactionLogDir txnLogDir : asterixConfiguration.getTransactionLogDir()) {
transactionLogDirs.put(txnLogDir.getNcId(), txnLogDir.getTxnLogDirPath());
}
- Properties gitProperties = new Properties();
- try {
- gitProperties.load(getClass().getClassLoader().getResourceAsStream("git.properties"));
- asterixBuildProperties = new HashMap<String, String>();
- for (final String name : gitProperties.stringPropertyNames()) {
- asterixBuildProperties.put(name, gitProperties.getProperty(name));
- }
- } catch (IOException e) {
- throw new AsterixException(e);
- }
+ loadAsterixBuildProperties();
}
/**
* Constructor which wraps an IApplicationConfig.
*/
- public AsterixPropertiesAccessor(IApplicationConfig cfg) {
+ public AsterixPropertiesAccessor(IApplicationConfig cfg) throws AsterixException {
this.cfg = cfg;
instanceName = cfg.getString("asterix", "instance", "DEFAULT_INSTANCE");
String mdNode = null;
nodePartitionsMap = new HashMap<>();
int uniquePartitionId = 0;
+
+ // Iterate through each configured NC.
for (String section : cfg.getSections()) {
if (!section.startsWith("nc/")) {
continue;
}
String ncId = section.substring(3);
+ // Here we figure out which is the metadata node. If any NCs
+ // declare "metadata.port", use that one; otherwise just use the first.
if (mdNode == null) {
- // Default is first node == metadata node
mdNode = ncId;
}
if (cfg.getString(section, "metadata.port") != null) {
@@ -153,27 +157,46 @@
mdNode = ncId;
}
+ // Now we assign the coredump and txnlog directories for this node.
// QQQ Default values? Should they be specified here? Or should there
- // be a default.ini? They can't be inserted by TriggerNCWork except
- // possibly for hyracks-specified values. Certainly wherever they are,
- // they should be platform-dependent.
+ // be a default.ini? Certainly wherever they are, they should be platform-dependent.
coredumpConfig.put(ncId, cfg.getString(section, "coredumpdir", "/var/lib/asterixdb/coredump"));
transactionLogDirs.put(ncId, cfg.getString(section, "txnlogdir", "/var/lib/asterixdb/txn-log"));
- String[] storeDirs = cfg.getString(section, "storagedir", "storage").trim().split(",");
- ClusterPartition[] nodePartitions = new ClusterPartition[storeDirs.length];
+
+ // Now we create an array of ClusterPartitions for all the partitions
+ // on this NC.
+ String[] iodevices = cfg.getString(section, "iodevices", "/var/lib/asterixdb/iodevice").split(",");
+ String storageSubdir = cfg.getString(section, "storagedir", "storage");
+ String[] nodeStores = new String[iodevices.length];
+ ClusterPartition[] nodePartitions = new ClusterPartition[iodevices.length];
for (int i = 0; i < nodePartitions.length; i++) {
+ // Construct final storage path from iodevice dir + storage subdir.
+ nodeStores[i] = iodevices[i] + File.separator + storageSubdir;
+ // Create ClusterPartition instances for this NC.
ClusterPartition partition = new ClusterPartition(uniquePartitionId++, ncId, i);
clusterPartitions.put(partition.getPartitionId(), partition);
nodePartitions[i] = partition;
}
- stores.put(ncId, storeDirs);
+ stores.put(ncId, nodeStores);
nodePartitionsMap.put(ncId, nodePartitions);
nodeNames.add(ncId);
}
metadataNodeName = mdNode;
asterixConfigurationParams = null;
- asterixBuildProperties = null;
+ loadAsterixBuildProperties();
+ }
+
+ private void loadAsterixBuildProperties() throws AsterixException {
+ Properties gitProperties = new Properties();
+ try {
+ gitProperties.load(getClass().getClassLoader().getResourceAsStream("git.properties"));
+ for (final String name : gitProperties.stringPropertyNames()) {
+ asterixBuildProperties.put(name, gitProperties.getProperty(name));
+ }
+ } catch (IOException e) {
+ throw new AsterixException(e);
+ }
}
public String getMetadataNodeName() {
@@ -204,20 +227,6 @@
return asterixBuildProperties;
}
- public void putCoredumpPaths(String nodeId, String coredumpPath) {
- if (coredumpConfig.containsKey(nodeId)) {
- throw new IllegalStateException("Cannot override value for coredump path");
- }
- coredumpConfig.put(nodeId, coredumpPath);
- }
-
- public void putTransactionLogDir(String nodeId, String txnLogDir) {
- if (transactionLogDirs.containsKey(nodeId)) {
- throw new IllegalStateException("Cannot override value for txnLogDir");
- }
- transactionLogDirs.put(nodeId, txnLogDir);
- }
-
public <T> T getProperty(String property, T defaultValue, IPropertyInterpreter<T> interpreter) {
String value;
Property p = null;
@@ -246,18 +255,11 @@
}
}
- private static <T> void logConfigurationError(Property p, T defaultValue) {
- if (LOGGER.isLoggable(Level.SEVERE)) {
- LOGGER.severe("Invalid property value '" + p.getValue() + "' for property '" + p.getName()
- + "'.\n See the description: \n" + p.getDescription() + "\nDefault = " + defaultValue);
- }
- }
-
public String getInstanceName() {
return instanceName;
}
- public ClusterPartition getMetadataPartiton() {
+ public ClusterPartition getMetadataPartition() {
// metadata partition is always the first partition on the metadata node
return nodePartitionsMap.get(metadataNodeName)[0];
}
diff --git a/asterixdb/asterix-server/pom.xml b/asterixdb/asterix-server/pom.xml
index 812fd59..92fde73 100644
--- a/asterixdb/asterix-server/pom.xml
+++ b/asterixdb/asterix-server/pom.xml
@@ -125,6 +125,24 @@
</executions>
</plugin>
<plugin>
+ <artifactId>maven-antrun-plugin</artifactId>
+ <version>1.6</version>
+ <executions>
+ <execution>
+ <id>process-test-classes</id>
+ <phase>package</phase>
+ <configuration>
+ <target>
+ <chmod file="target/appassembler/bin/*" perm="755" />
+ </target>
+ </configuration>
+ <goals>
+ <goal>run</goal>
+ </goals>
+ </execution>
+ </executions>
+ </plugin>
+ <plugin>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.2-beta-5</version>
<executions>
@@ -181,11 +199,37 @@
<scope>compile</scope>
</dependency>
<dependency>
+ <groupId>org.apache.hyracks</groupId>
+ <artifactId>hyracks-server</artifactId>
+ <type>jar</type>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
<groupId>org.apache.asterix</groupId>
<artifactId>asterix-app</artifactId>
<version>0.8.9-SNAPSHOT</version>
</dependency>
<dependency>
+ <groupId>org.apache.asterix</groupId>
+ <artifactId>asterix-app</artifactId>
+ <version>0.8.9-SNAPSHOT</version>
+ <type>test-jar</type>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.asterix</groupId>
+ <artifactId>asterix-common</artifactId>
+ <version>0.8.9-SNAPSHOT</version>
+ <type>test-jar</type>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.asterix</groupId>
+ <artifactId>asterix-test-framework</artifactId>
+ <version>0.8.9-SNAPSHOT</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
<groupId>org.codehaus.mojo.appassembler</groupId>
<artifactId>appassembler-booter</artifactId>
<version>1.3.1</version>
diff --git a/asterixdb/asterix-server/src/test/java/org/apache/asterix/server/test/NCServiceExecutionIT.java b/asterixdb/asterix-server/src/test/java/org/apache/asterix/server/test/NCServiceExecutionIT.java
new file mode 100644
index 0000000..c179103
--- /dev/null
+++ b/asterixdb/asterix-server/src/test/java/org/apache/asterix/server/test/NCServiceExecutionIT.java
@@ -0,0 +1,186 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.asterix.server.test;
+
+import org.apache.asterix.test.aql.TestExecutor;
+import org.apache.asterix.test.runtime.HDFSCluster;
+import org.apache.asterix.testframework.context.TestCaseContext;
+import org.apache.asterix.testframework.xml.TestGroup;
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hyracks.server.process.HyracksVirtualCluster;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.FixMethodOrder;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.MethodSorters;
+import org.junit.runners.Parameterized;
+import org.junit.runners.Parameterized.Parameters;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.logging.Logger;
+
+@FixMethodOrder(MethodSorters.NAME_ASCENDING)
+@RunWith(Parameterized.class)
+public class NCServiceExecutionIT {
+
+ // Important paths and files for this test.
+
+ // The "target" subdirectory of asterix-server. All outputs go here.
+ private static final String TARGET_DIR = StringUtils
+ .join(new String[] { System.getProperty("basedir"), "target" }, File.separator);
+
+ // Directory where the NCs create and store all data, as configured by
+ // src/test/resources/NCServiceExecutionIT/cc.conf.
+ private static final String INSTANCE_DIR = StringUtils
+ .join(new String[] { TARGET_DIR, "tmp" }, File.separator);
+
+ // The log directory, where all CC, NCService, and NC logs are written. CC and
+ // NCService logs are configured on the HyracksVirtualCluster below. NC logs
+ // are configured in src/test/resources/NCServiceExecutionIT/ncservice*.conf.
+ private static final String LOG_DIR = StringUtils
+ .join(new String[] { TARGET_DIR, "failsafe-reports" }, File.separator);
+
+ // Directory where *.conf files are located.
+ private static final String CONF_DIR = StringUtils
+ .join(new String[] { TARGET_DIR, "test-classes", "NCServiceExecutionIT" },
+ File.separator);
+
+ // The app.home specified for HyracksVirtualCluster. The NCService expects
+ // to find the NC startup script in ${app.home}/bin.
+ private static final String APP_HOME = StringUtils
+ .join(new String[] { TARGET_DIR, "appassembler" }, File.separator);
+
+ // Path to the asterix-app directory. This is used as the current working
+ // directory for the CC and NCService processes, which allows relative file
+ // paths in "load" statements in test queries to find the right data. It is
+ // also used for HDFSCluster.
+ private static final String ASTERIX_APP_DIR = StringUtils
+ .join(new String[] { System.getProperty("basedir"), "..", "asterix-app" },
+ File.separator);
+
+ // Path to the actual AQL test files, which we borrow from asterix-app. This is
+ // passed to TestExecutor.
+ protected static final String TESTS_DIR = StringUtils
+ .join(new String[] { ASTERIX_APP_DIR, "src", "test", "resources", "runtimets" },
+ File.separator);
+
+ // Path that actual results are written to. We create and clean this directory
+ // here, and also pass it to TestExecutor which writes the test output there.
+ private static final String ACTUAL_RESULTS_DIR = StringUtils
+ .join(new String[] { TARGET_DIR, "ittest" }, File.separator);
+
+ private static final Logger LOGGER = Logger.getLogger(NCServiceExecutionIT.class.getName());
+
+ private final TestCaseContext tcCtx;
+ private static final TestExecutor testExecutor = new TestExecutor();
+ private static HyracksVirtualCluster cluster;
+
+ @BeforeClass
+ public static void setUp() throws Exception {
+ // Create actual-results output directory.
+ File outDir = new File(ACTUAL_RESULTS_DIR);
+ outDir.mkdirs();
+
+ // Remove any instance data from previous runs.
+ File instanceDir = new File(INSTANCE_DIR);
+ if (instanceDir.isDirectory()) {
+ FileUtils.deleteDirectory(instanceDir);
+ }
+
+ // HDFSCluster requires the input directory to end with a file separator.
+ HDFSCluster.getInstance().setup(ASTERIX_APP_DIR + File.separator);
+
+ cluster = new HyracksVirtualCluster(new File(APP_HOME), new File(ASTERIX_APP_DIR));
+ cluster.addNC(
+ new File(CONF_DIR, "ncservice1.conf"),
+ new File(LOG_DIR, "ncservice1.log")
+ );
+ cluster.addNC(
+ new File(CONF_DIR, "ncservice2.conf"),
+ new File(LOG_DIR, "ncservice2.log")
+ );
+
+ try {
+ Thread.sleep(2000);
+ }
+ catch (InterruptedException ignored) {
+ }
+
+ // Start CC
+ cluster.start(
+ new File(CONF_DIR, "cc.conf"),
+ new File(LOG_DIR, "cc.log")
+ );
+
+ LOGGER.info("Sleeping while cluster comes online...");
+ try {
+ Thread.sleep(6000);
+ }
+ catch (InterruptedException ignored) {
+ }
+ }
+
+ @AfterClass
+ public static void tearDown() throws Exception {
+ File outdir = new File(ACTUAL_RESULTS_DIR);
+ File[] files = outdir.listFiles();
+ if (files == null || files.length == 0) {
+ outdir.delete();
+ }
+ cluster.stop();
+ HDFSCluster.getInstance().cleanup();
+ }
+
+ @Parameters
+ public static Collection<Object[]> tests() throws Exception {
+ Collection<Object[]> testArgs = new ArrayList<Object[]>();
+ TestCaseContext.Builder b = new TestCaseContext.Builder();
+ for (TestCaseContext ctx : b.build(new File(TESTS_DIR))) {
+ if (!skip(ctx)) {
+ testArgs.add(new Object[]{ctx});
+ }
+ }
+ return testArgs;
+ }
+
+ private static boolean skip(TestCaseContext tcCtx) {
+ // For now we skip feeds tests and external-library tests.
+ for (TestGroup group : tcCtx.getTestGroups()) {
+ if (group.getName().startsWith("external-") || group.getName().equals("feeds")) {
+ LOGGER.info("Skipping test: " + tcCtx.toString());
+ return true;
+ }
+ }
+ return false;
+ }
+
+
+ public NCServiceExecutionIT(TestCaseContext ctx) {
+ this.tcCtx = ctx;
+ }
+
+ @Test
+ public void test() throws Exception {
+ testExecutor.executeTest(ACTUAL_RESULTS_DIR, tcCtx, null, false);
+ }
+}
diff --git a/asterixdb/asterix-server/src/test/resources/NCServiceExecutionIT/cc.conf b/asterixdb/asterix-server/src/test/resources/NCServiceExecutionIT/cc.conf
new file mode 100644
index 0000000..c4c76e6
--- /dev/null
+++ b/asterixdb/asterix-server/src/test/resources/NCServiceExecutionIT/cc.conf
@@ -0,0 +1,25 @@
+[nc/asterix_nc1]
+txnlogdir=../asterix-server/target/tmp/asterix_nc1/txnlog
+coredumpdir=../asterix-server/target/tmp/asterix_nc1/coredump
+iodevices=../asterix-server/target/tmp/asterix_nc1/iodevice1,../asterix-server/target/tmp/asterix_nc1/iodevice2
+
+[nc/asterix_nc2]
+port=9091
+txnlogdir=../asterix-server/target/tmp/asterix_nc2/txnlog
+coredumpdir=../asterix-server/target/tmp/asterix_nc2/coredump
+iodevices=../asterix-server/target/tmp/asterix_nc2/iodevice1,../asterix-server/target/tmp/asterix_nc2/iodevice2
+
+[nc]
+address=127.0.0.1
+command=asterixnc
+app.class=org.apache.asterix.hyracks.bootstrap.NCApplicationEntryPoint
+jvm.args=-Xmx4096m -Dnode.Resolver="org.apache.asterix.external.util.IdentitiyResolverFactory"
+storagedir=test_storage
+
+[cc]
+cluster.address = 127.0.0.1
+app.class=org.apache.asterix.hyracks.bootstrap.CCApplicationEntryPoint
+
+[asterix]
+storage.memorycomponent.globalbudget = 1073741824
+
diff --git a/asterixdb/asterix-server/src/test/resources/NCServiceExecutionIT/ncservice1.conf b/asterixdb/asterix-server/src/test/resources/NCServiceExecutionIT/ncservice1.conf
new file mode 100644
index 0000000..fa44fa2
--- /dev/null
+++ b/asterixdb/asterix-server/src/test/resources/NCServiceExecutionIT/ncservice1.conf
@@ -0,0 +1,3 @@
+[ncservice]
+logdir=../asterix-server/target/failsafe-reports
+
diff --git a/asterixdb/asterix-server/src/test/resources/NCServiceExecutionIT/ncservice2.conf b/asterixdb/asterix-server/src/test/resources/NCServiceExecutionIT/ncservice2.conf
new file mode 100644
index 0000000..53d8d9b
--- /dev/null
+++ b/asterixdb/asterix-server/src/test/resources/NCServiceExecutionIT/ncservice2.conf
@@ -0,0 +1,4 @@
+[ncservice]
+logdir=../asterix-server/target/failsafe-reports
+port=9091
+
diff --git a/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/ClusterControllerService.java b/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/ClusterControllerService.java
index 2aa3f37..4cddef1 100644
--- a/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/ClusterControllerService.java
+++ b/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/ClusterControllerService.java
@@ -276,7 +276,7 @@
continue;
}
String ncid = section.substring(3);
- String address = ini.get(section, "address");
+ String address = IniUtils.getString(ini, section, "address", null);
int port = IniUtils.getInt(ini, section, "port", 9090);
if (address == null) {
address = InetAddress.getLoopbackAddress().getHostAddress();
diff --git a/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/work/TriggerNCWork.java b/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/work/TriggerNCWork.java
index ee79d38..7d2ff25 100644
--- a/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/work/TriggerNCWork.java
+++ b/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/work/TriggerNCWork.java
@@ -87,37 +87,15 @@
}
/**
- * Utility routine to copy all keys from a named section in Ini a
- * to a named section in Ini b. We need to do this the hard way
- * because Ini4j reacts inscrutably when attempting to copy
- * Ini.Sections directly from one Ini to another.
- */
- private void copyIniSection(Ini a, String asect, Ini b, String bsect) {
- Ini.Section source = a.get(asect);
- for (String key : source.keySet()) {
- b.put(bsect, key, source.get(key));
- }
- }
- /**
* Given an Ini object, serialize it to String with some enhancements.
* @param ccini
*/
String serializeIni(Ini ccini) throws IOException {
- Ini ini = new Ini();
-
- // First copy the global [nc] section to a new section named for
- // *this* NC, so that those values serve as defaults.
- String ncsection = "nc/" + ncId;
- copyIniSection(ccini, "nc", ini, ncsection);
- // Now copy all sections to their same name in the derived config.
- for (String section : ccini.keySet()) {
- copyIniSection(ccini, section, ini, section);
- }
+ StringWriter iniString = new StringWriter();
+ ccini.store(iniString);
// Finally insert *this* NC's name into localnc section - this is a fixed
// entry point so that NCs can determine where all their config is.
- ini.put("localnc", "id", ncId);
- StringWriter iniString = new StringWriter();
- ini.store(iniString);
+ iniString.append("\n[localnc]\nid=" + ncId + "\n");
if (LOGGER.isLoggable(Level.FINE)) {
LOGGER.fine("Returning Ini file:\n" + iniString.toString());
}
diff --git a/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/application/IniApplicationConfig.java b/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/application/IniApplicationConfig.java
index 3a8a2de..22fe318 100644
--- a/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/application/IniApplicationConfig.java
+++ b/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/application/IniApplicationConfig.java
@@ -19,6 +19,7 @@
package org.apache.hyracks.control.common.application;
import org.apache.hyracks.api.application.IApplicationConfig;
+import org.apache.hyracks.control.common.controllers.IniUtils;
import org.ini4j.Ini;
import java.util.Set;
@@ -37,39 +38,34 @@
}
}
- private <T> T getIniValue(String section, String key, T default_value, Class<T> clazz) {
- T value = ini.get(section, key, clazz);
- return (value != null) ? value : default_value;
- }
-
@Override
public String getString(String section, String key) {
- return getIniValue(section, key, null, String.class);
+ return IniUtils.getString(ini, section, key, null);
}
@Override
public String getString(String section, String key, String defaultValue) {
- return getIniValue(section, key, defaultValue, String.class);
+ return IniUtils.getString(ini, section, key, defaultValue);
}
@Override
public int getInt(String section, String key) {
- return getIniValue(section, key, 0, Integer.class);
+ return IniUtils.getInt(ini, section, key, 0);
}
@Override
public int getInt(String section, String key, int defaultValue) {
- return getIniValue(section, key, defaultValue, Integer.class);
+ return IniUtils.getInt(ini, section, key, defaultValue);
}
@Override
public long getLong(String section, String key) {
- return getIniValue(section, key, (long) 0, Long.class);
+ return IniUtils.getLong(ini, section, key, (long) 0);
}
@Override
public long getLong(String section, String key, long defaultValue) {
- return getIniValue(section, key, defaultValue, Long.class);
+ return IniUtils.getLong(ini, section, key, defaultValue);
}
@Override
diff --git a/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/CCConfig.java b/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/CCConfig.java
index a04d750..64bd7d1 100644
--- a/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/CCConfig.java
+++ b/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/CCConfig.java
@@ -152,47 +152,4 @@
public IApplicationConfig getAppConfig() {
return new IniApplicationConfig(ini);
}
-
- public void toCommandLine(List<String> cList) {
- cList.add("-client-net-ip-address");
- cList.add(clientNetIpAddress);
- cList.add("-client-net-port");
- cList.add(String.valueOf(clientNetPort));
- cList.add("-cluster-net-ip-address");
- cList.add(clusterNetIpAddress);
- cList.add("-cluster-net-port");
- cList.add(String.valueOf(clusterNetPort));
- cList.add("-http-port");
- cList.add(String.valueOf(httpPort));
- cList.add("-heartbeat-period");
- cList.add(String.valueOf(heartbeatPeriod));
- cList.add("-max-heartbeat-lapse-periods");
- cList.add(String.valueOf(maxHeartbeatLapsePeriods));
- cList.add("-profile-dump-period");
- cList.add(String.valueOf(profileDumpPeriod));
- cList.add("-default-max-job-attempts");
- cList.add(String.valueOf(defaultMaxJobAttempts));
- cList.add("-job-history-size");
- cList.add(String.valueOf(jobHistorySize));
- cList.add("-result-time-to-live");
- cList.add(String.valueOf(resultTTL));
- cList.add("-result-sweep-threshold");
- cList.add(String.valueOf(resultSweepThreshold));
- cList.add("-cc-root");
- cList.add(ccRoot);
- if (clusterTopologyDefinition != null) {
- cList.add("-cluster-topology");
- cList.add(clusterTopologyDefinition.getAbsolutePath());
- }
- if (appCCMainClass != null) {
- cList.add("-app-cc-main-class");
- cList.add(appCCMainClass);
- }
- if (appArgs != null && !appArgs.isEmpty()) {
- cList.add("--");
- for (String appArg : appArgs) {
- cList.add(appArg);
- }
- }
- }
}
diff --git a/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/IniUtils.java b/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/IniUtils.java
index 9a5c9a0..538bb0b 100644
--- a/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/IniUtils.java
+++ b/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/IniUtils.java
@@ -26,21 +26,39 @@
/**
* Some utility functions for reading Ini4j objects with default values.
+ * For all getXxx() methods: if the 'section' contains a slash, and the 'key'
+ * is not found in that section, we will search for the key in the section named
+ * by stripping the leaf of the section name (final slash and anything following).
+ * eg. getInt(ini, "nc/red", "dir", null) will first look for the key "dir" in
+ * the section "nc/red", but if it is not found, will look in the section "nc".
*/
public class IniUtils {
+ private static <T> T getIniValue(Ini ini, String section, String key, T default_value, Class<T> clazz) {
+ T value;
+ while (true) {
+ value = ini.get(section, key, clazz);
+ if (value == null) {
+ int idx = section.lastIndexOf('/');
+ if (idx > -1) {
+ section = section.substring(0, idx);
+ continue;
+ }
+ }
+ break;
+ }
+ return (value != null) ? value : default_value;
+ }
+
public static String getString(Ini ini, String section, String key, String defaultValue) {
- String value = ini.get(section, key, String.class);
- return (value != null) ? value : defaultValue;
+ return getIniValue(ini, section, key, defaultValue, String.class);
}
public static int getInt(Ini ini, String section, String key, int defaultValue) {
- Integer value = ini.get(section, key, Integer.class);
- return (value != null) ? value : defaultValue;
+ return getIniValue(ini, section, key, defaultValue, Integer.class);
}
public static long getLong(Ini ini, String section, String key, long defaultValue) {
- Long value = ini.get(section, key, Long.class);
- return (value != null) ? value : defaultValue;
+ return getIniValue(ini, section, key, defaultValue, Long.class);
}
public static Ini loadINIFile(String configFile) throws IOException {
diff --git a/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/NCConfig.java b/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/NCConfig.java
index b408083..d08df60 100644
--- a/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/NCConfig.java
+++ b/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/NCConfig.java
@@ -191,66 +191,6 @@
return new IniApplicationConfig(ini);
}
- public void toCommandLine(List<String> cList) {
- cList.add("-cc-host");
- cList.add(ccHost);
- cList.add("-cc-port");
- cList.add(String.valueOf(ccPort));
- cList.add("-cluster-net-ip-address");
- cList.add(clusterNetIPAddress);
- cList.add("-cluster-net-port");
- cList.add(String.valueOf(clusterNetPort));
- cList.add("-cluster-net-public-ip-address");
- cList.add(clusterNetPublicIPAddress);
- cList.add("-cluster-net-public-port");
- cList.add(String.valueOf(clusterNetPublicPort));
- cList.add("-node-id");
- cList.add(nodeId);
- cList.add("-data-ip-address");
- cList.add(dataIPAddress);
- cList.add("-data-port");
- cList.add(String.valueOf(dataPort));
- cList.add("-data-public-ip-address");
- cList.add(dataPublicIPAddress);
- cList.add("-data-public-port");
- cList.add(String.valueOf(dataPublicPort));
- cList.add("-result-ip-address");
- cList.add(resultIPAddress);
- cList.add("-result-port");
- cList.add(String.valueOf(resultPort));
- cList.add("-result-public-ip-address");
- cList.add(resultPublicIPAddress);
- cList.add("-result-public-port");
- cList.add(String.valueOf(resultPublicPort));
- cList.add("-retries");
- cList.add(String.valueOf(retries));
- cList.add("-iodevices");
- cList.add(ioDevices);
- cList.add("-net-thread-count");
- cList.add(String.valueOf(nNetThreads));
- cList.add("-net-buffer-count");
- cList.add(String.valueOf(nNetBuffers));
- cList.add("-max-memory");
- cList.add(String.valueOf(maxMemory));
- cList.add("-result-time-to-live");
- cList.add(String.valueOf(resultTTL));
- cList.add("-result-sweep-threshold");
- cList.add(String.valueOf(resultSweepThreshold));
- cList.add("-result-manager-memory");
- cList.add(String.valueOf(resultManagerMemory));
-
- if (appNCMainClass != null) {
- cList.add("-app-nc-main-class");
- cList.add(appNCMainClass);
- }
- if (appArgs != null && !appArgs.isEmpty()) {
- cList.add("--");
- for (String appArg : appArgs) {
- cList.add(appArg);
- }
- }
- }
-
public void toMap(Map<String, String> configuration) {
configuration.put("cc-host", ccHost);
configuration.put("cc-port", (String.valueOf(ccPort)));
diff --git a/hyracks-fullstack/hyracks/hyracks-control/hyracks-nc-service/src/main/java/org/apache/hyracks/control/nc/service/NCService.java b/hyracks-fullstack/hyracks/hyracks-control/hyracks-nc-service/src/main/java/org/apache/hyracks/control/nc/service/NCService.java
index e3fe959..4102b4c 100644
--- a/hyracks-fullstack/hyracks/hyracks-control/hyracks-nc-service/src/main/java/org/apache/hyracks/control/nc/service/NCService.java
+++ b/hyracks-fullstack/hyracks/hyracks-control/hyracks-nc-service/src/main/java/org/apache/hyracks/control/nc/service/NCService.java
@@ -19,6 +19,7 @@
package org.apache.hyracks.control.nc.service;
import org.apache.commons.lang3.SystemUtils;
+import org.apache.hyracks.control.common.controllers.IniUtils;
import org.ini4j.Ini;
import org.kohsuke.args4j.CmdLineParser;
@@ -71,23 +72,13 @@
private static final String MAGIC_COOKIE = "hyncmagic";
- private static String getStringINIOpt(Ini ini, String section, String key, String default_value) {
- String value = ini.get(section, key, String.class);
- return (value != null) ? value : default_value;
- }
-
- private static int getIntINIOpt(Ini ini, String section, String key, int default_value) {
- Integer value = ini.get(section, key, Integer.class);
- return (value != null) ? value : default_value;
- }
-
private static List<String> buildCommand() throws IOException {
List<String> cList = new ArrayList<String>();
// Find the command to run. For now, we allow overriding the name, but
// still assume it's located in the bin/ directory of the deployment.
// Even this is likely more configurability than we need.
- String command = getStringINIOpt(ini, nodeSection, "command", "hyracksnc");
+ String command = IniUtils.getString(ini, nodeSection, "command", "hyracksnc");
// app.home is specified by the Maven appassembler plugin. If it isn't set,
// fall back to user's home dir. Again this is likely more flexibility
// than we need.
@@ -110,10 +101,16 @@
private static void configEnvironment(Map<String,String> env) {
if (env.containsKey("JAVA_OPTS")) {
+ if (LOGGER.isLoggable(Level.INFO)) {
+ LOGGER.info("Keeping JAVA_OPTS from environment");
+ }
return;
}
- String jvmargs = getStringINIOpt(ini, nodeSection, "jvm.args", "-Xmx1536m");
+ String jvmargs = IniUtils.getString(ini, nodeSection, "jvm.args", "-Xmx1536m");
env.put("JAVA_OPTS", jvmargs);
+ if (LOGGER.isLoggable(Level.INFO)) {
+ LOGGER.info("Setting JAVA_OPTS to " + jvmargs);
+ }
}
/**
@@ -146,6 +143,8 @@
// If the directory IS there, all is well
}
File logfile = new File(config.logdir, "nc-" + ncId + ".log");
+ // Don't care if this succeeds or fails:
+ logfile.delete();
pb.redirectOutput(ProcessBuilder.Redirect.appendTo(logfile));
if (LOGGER.isLoggable(Level.INFO)) {
LOGGER.info("Logging to " + logfile.getCanonicalPath());
@@ -192,7 +191,7 @@
}
String iniString = ois.readUTF();
ini = new Ini(new StringReader(iniString));
- ncId = getStringINIOpt(ini, "localnc", "id", "");
+ ncId = IniUtils.getString(ini, "localnc", "id", "");
nodeSection = "nc/" + ncId;
return launchNCProcess();
} catch (Exception e) {
diff --git a/hyracks-fullstack/hyracks/hyracks-server/pom.xml b/hyracks-fullstack/hyracks/hyracks-server/pom.xml
index 3bda1d3..52958f8 100644
--- a/hyracks-fullstack/hyracks/hyracks-server/pom.xml
+++ b/hyracks-fullstack/hyracks/hyracks-server/pom.xml
@@ -86,10 +86,6 @@
<mainClass>org.apache.hyracks.control.nc.service.NCService</mainClass>
<name>hyracksncservice</name>
</program>
- <program>
- <mainClass>org.apache.hyracks.server.drivers.VirtualClusterDriver</mainClass>
- <name>hyracks-virtual-cluster</name>
- </program>
</programs>
<repositoryLayout>flat</repositoryLayout>
<repositoryName>lib</repositoryName>
diff --git a/hyracks-fullstack/hyracks/hyracks-server/src/main/java/org/apache/hyracks/server/drivers/VirtualClusterDriver.java b/hyracks-fullstack/hyracks/hyracks-server/src/main/java/org/apache/hyracks/server/drivers/VirtualClusterDriver.java
deleted file mode 100644
index 41c14a7..0000000
--- a/hyracks-fullstack/hyracks/hyracks-server/src/main/java/org/apache/hyracks/server/drivers/VirtualClusterDriver.java
+++ /dev/null
@@ -1,82 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.hyracks.server.drivers;
-
-import org.kohsuke.args4j.CmdLineParser;
-import org.kohsuke.args4j.Option;
-
-import org.apache.hyracks.control.common.controllers.CCConfig;
-import org.apache.hyracks.control.common.controllers.NCConfig;
-import org.apache.hyracks.server.process.HyracksCCProcess;
-import org.apache.hyracks.server.process.HyracksNCProcess;
-
-public class VirtualClusterDriver {
- private static class Options {
- @Option(name = "-n", required = false, usage = "Number of node controllers (default: 2)")
- public int n = 2;
-
- @Option(name = "-cc-client-net-port", required = false, usage = "CC Port (default: 1098)")
- public int ccClientNetPort = 1098;
-
- @Option(name = "-cc-cluster-net-port", required = false, usage = "CC Port (default: 1099)")
- public int ccClusterNetPort = 1099;
-
- @Option(name = "-cc-http-port", required = false, usage = "CC Port (default: 16001)")
- public int ccHttpPort = 16001;
- }
-
- public static void main(String[] args) throws Exception {
- Options options = new Options();
- CmdLineParser cp = new CmdLineParser(options);
- try {
- cp.parseArgument(args);
- } catch (Exception e) {
- System.err.println(e.getMessage());
- cp.printUsage(System.err);
- return;
- }
-
- CCConfig ccConfig = new CCConfig();
- ccConfig.clusterNetIpAddress = "127.0.0.1";
- ccConfig.clusterNetPort = options.ccClusterNetPort;
- ccConfig.clientNetIpAddress = "127.0.0.1";
- ccConfig.clientNetPort = options.ccClientNetPort;
- ccConfig.httpPort = options.ccHttpPort;
- HyracksCCProcess ccp = new HyracksCCProcess(ccConfig);
- ccp.start();
-
- Thread.sleep(5000);
-
- HyracksNCProcess ncps[] = new HyracksNCProcess[options.n];
- for (int i = 0; i < options.n; ++i) {
- NCConfig ncConfig = new NCConfig();
- ncConfig.ccHost = "127.0.0.1";
- ncConfig.ccPort = options.ccClusterNetPort;
- ncConfig.clusterNetIPAddress = "127.0.0.1";
- ncConfig.nodeId = "nc" + i;
- ncConfig.dataIPAddress = "127.0.0.1";
- ncps[i] = new HyracksNCProcess(ncConfig);
- ncps[i].start();
- }
-
- while (true) {
- Thread.sleep(10000);
- }
- }
-}
diff --git a/hyracks-fullstack/hyracks/hyracks-server/src/main/java/org/apache/hyracks/server/process/HyracksCCProcess.java b/hyracks-fullstack/hyracks/hyracks-server/src/main/java/org/apache/hyracks/server/process/HyracksCCProcess.java
index d0d8d63..4a70120 100644
--- a/hyracks-fullstack/hyracks/hyracks-server/src/main/java/org/apache/hyracks/server/process/HyracksCCProcess.java
+++ b/hyracks-fullstack/hyracks/hyracks-server/src/main/java/org/apache/hyracks/server/process/HyracksCCProcess.java
@@ -18,25 +18,28 @@
*/
package org.apache.hyracks.server.process;
+import org.apache.hyracks.control.cc.CCDriver;
+
+import java.io.File;
import java.util.List;
-import org.apache.hyracks.control.cc.CCDriver;
-import org.apache.hyracks.control.common.controllers.CCConfig;
-
public class HyracksCCProcess extends HyracksServerProcess {
- private CCConfig config;
- public HyracksCCProcess(CCConfig config) {
- this.config = config;
- }
-
- @Override
- protected void addCmdLineArgs(List<String> cList) {
- config.toCommandLine(cList);
+ public HyracksCCProcess(File configFile, File logFile, File appHome, File workingDir) {
+ this.configFile = configFile;
+ this.logFile = logFile;
+ this.appHome = appHome;
+ this.workingDir = workingDir;
}
@Override
protected String getMainClassName() {
return CCDriver.class.getName();
}
+
+ @Override
+ protected void addJvmArgs(List<String> cList) {
+ // CC needs more than default memory
+ cList.add("-Xmx1024m");
+ }
}
diff --git a/hyracks-fullstack/hyracks/hyracks-server/src/main/java/org/apache/hyracks/server/process/HyracksNCProcess.java b/hyracks-fullstack/hyracks/hyracks-server/src/main/java/org/apache/hyracks/server/process/HyracksNCProcess.java
index c4517e6..8bc1694 100644
--- a/hyracks-fullstack/hyracks/hyracks-server/src/main/java/org/apache/hyracks/server/process/HyracksNCProcess.java
+++ b/hyracks-fullstack/hyracks/hyracks-server/src/main/java/org/apache/hyracks/server/process/HyracksNCProcess.java
@@ -18,25 +18,28 @@
*/
package org.apache.hyracks.server.process;
+import org.apache.hyracks.control.nc.service.NCService;
+
+import java.io.File;
import java.util.List;
-import org.apache.hyracks.control.common.controllers.NCConfig;
-import org.apache.hyracks.control.nc.NCDriver;
-
public class HyracksNCProcess extends HyracksServerProcess {
- private NCConfig config;
- public HyracksNCProcess(NCConfig config) {
- this.config = config;
- }
-
- @Override
- protected void addCmdLineArgs(List<String> cList) {
- config.toCommandLine(cList);
+ public HyracksNCProcess(File configFile, File logFile, File appHome, File workingDir) {
+ this.configFile = configFile;
+ this.logFile = logFile;
+ this.appHome = appHome;
+ this.workingDir = workingDir;
}
@Override
protected String getMainClassName() {
- return NCDriver.class.getName();
+ return NCService.class.getName();
+ }
+
+ @Override
+ protected void addJvmArgs(List<String> cList) {
+ // NCService needs little memory
+ cList.add("-Xmx128m");
}
}
diff --git a/hyracks-fullstack/hyracks/hyracks-server/src/main/java/org/apache/hyracks/server/process/HyracksServerProcess.java b/hyracks-fullstack/hyracks/hyracks-server/src/main/java/org/apache/hyracks/server/process/HyracksServerProcess.java
index 9dec0ec..13cb445 100644
--- a/hyracks-fullstack/hyracks/hyracks-server/src/main/java/org/apache/hyracks/server/process/HyracksServerProcess.java
+++ b/hyracks-fullstack/hyracks/hyracks-server/src/main/java/org/apache/hyracks/server/process/HyracksServerProcess.java
@@ -29,53 +29,69 @@
import java.util.logging.Level;
import java.util.logging.Logger;
-public abstract class HyracksServerProcess {
+abstract class HyracksServerProcess {
private static final Logger LOGGER = Logger.getLogger(HyracksServerProcess.class.getName());
protected Process process;
+ protected File configFile = null;
+ protected File logFile = null;
+ protected File appHome = null;
+ protected File workingDir = null;
public void start() throws IOException {
String[] cmd = buildCommand();
if (LOGGER.isLoggable(Level.INFO)) {
LOGGER.info("Starting command: " + Arrays.toString(cmd));
}
- process = Runtime.getRuntime().exec(cmd, null, null);
- dump(process.getInputStream());
- dump(process.getErrorStream());
+
+ ProcessBuilder pb = new ProcessBuilder(cmd);
+ pb.redirectErrorStream(true);
+ if (logFile != null) {
+ if (LOGGER.isLoggable(Level.INFO)) {
+ LOGGER.info("Logging to: " + logFile.getCanonicalPath());
+ }
+ logFile.getParentFile().mkdirs();
+ logFile.delete();
+ pb.redirectOutput(ProcessBuilder.Redirect.appendTo(logFile));
+ } else {
+ if (LOGGER.isLoggable(Level.INFO)) {
+ LOGGER.info("Logfile not set, subprocess will output to stdout");
+ }
+ }
+ pb.directory(workingDir);
+ process = pb.start();
}
- private void dump(InputStream input) {
- final int streamBufferSize = 1000;
- final Reader in = new InputStreamReader(input);
- new Thread(new Runnable() {
- public void run() {
- try {
- char[] chars = new char[streamBufferSize];
- int c;
- while ((c = in.read(chars)) != -1) {
- if (c > 0) {
- System.out.print(String.valueOf(chars, 0, c));
- }
- }
- } catch (IOException e) {
- }
- }
- }).start();
+ public void stop() {
+ process.destroy();
+ try {
+ process.waitFor();
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ }
}
private String[] buildCommand() {
List<String> cList = new ArrayList<String>();
cList.add(getJavaCommand());
- cList.add("-Dbasedir=" + System.getProperty("basedir"));
- cList.add("-Djava.rmi.server.hostname=127.0.0.1");
+ addJvmArgs(cList);
+ cList.add("-Dapp.home=" + appHome.getAbsolutePath());
cList.add("-classpath");
cList.add(getClasspath());
cList.add(getMainClassName());
+ if (configFile != null) {
+ cList.add("-config-file");
+ cList.add(configFile.getAbsolutePath());
+ }
addCmdLineArgs(cList);
return cList.toArray(new String[cList.size()]);
}
- protected abstract void addCmdLineArgs(List<String> cList);
+ protected void addJvmArgs(List<String> cList) {
+ }
+
+ protected void addCmdLineArgs(List<String> cList) {
+ }
protected abstract String getMainClassName();
@@ -83,7 +99,7 @@
return System.getProperty("java.class.path");
}
- protected final String getJavaCommand() {
+ private final String getJavaCommand() {
return System.getProperty("java.home") + File.separator + "bin" + File.separator + "java";
}
}
diff --git a/hyracks-fullstack/hyracks/hyracks-server/src/main/java/org/apache/hyracks/server/process/HyracksVirtualCluster.java b/hyracks-fullstack/hyracks/hyracks-server/src/main/java/org/apache/hyracks/server/process/HyracksVirtualCluster.java
new file mode 100644
index 0000000..f08bb43
--- /dev/null
+++ b/hyracks-fullstack/hyracks/hyracks-server/src/main/java/org/apache/hyracks/server/process/HyracksVirtualCluster.java
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hyracks.server.process;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * Starts a local hyracks-based cluster (NC and CC child processes).
+ */
+public class HyracksVirtualCluster {
+ private final File appHome;
+ private final File workingDir;
+ private List<HyracksNCProcess> ncProcs = new ArrayList<>(3);
+ private HyracksCCProcess ccProc = null;
+
+ /**
+ * Construct a Hyracks-based cluster.
+ * @param appHome - path to the installation root of the Hyracks application.
+ * At least bin/hyracksnc (or the equivalent NC script for
+ * the application) must exist in this directory.
+ * @param workingDir - directory to use as CWD for all child processes. May
+ * be null, in which case the CWD of the invoking process is used.
+ */
+ public HyracksVirtualCluster(File appHome, File workingDir) {
+ this.appHome = appHome;
+ this.workingDir = workingDir;
+ }
+
+ /**
+ * Creates and starts an NCService.
+ * @param configFile - full path to an ncservice.conf. May be null to accept all defaults.
+ * @throws IOException - if there are errors starting the process.
+ */
+ public void addNC(File configFile, File logFile) throws IOException {
+ HyracksNCProcess proc = new HyracksNCProcess(configFile, logFile, appHome, workingDir);
+ proc.start();
+ ncProcs.add(proc);
+ }
+
+ /**
+ * Starts the CC, initializing the cluster. Expects that any NCs referenced
+ * in the cluster configuration have already been started with addNC().
+ * @param ccConfigFile - full path to a cluster conf file. May be null to accept all
+ * defaults, although this is seldom useful since there are no NCs.
+ * @throws IOException - if there are errors starting the process.
+ */
+ public void start(File ccConfigFile, File logFile) throws IOException {
+ ccProc = new HyracksCCProcess(ccConfigFile, logFile, appHome, workingDir);
+ ccProc.start();
+ }
+
+ /**
+ * Stops all processes in the cluster.
+ * QQQ Someday this should probably do a graceful stop of NCs rather than
+ * killing the NCService.
+ */
+ public void stop() {
+ ccProc.stop();
+ for (HyracksNCProcess proc : ncProcs) {
+ proc.stop();
+ }
+ }
+}
+
diff --git a/hyracks-fullstack/hyracks/hyracks-server/src/test/java/org/apache/hyracks/server/test/NCServiceIT.java b/hyracks-fullstack/hyracks/hyracks-server/src/test/java/org/apache/hyracks/server/test/NCServiceIT.java
index bd99c8c..9a231a0 100644
--- a/hyracks-fullstack/hyracks/hyracks-server/src/test/java/org/apache/hyracks/server/test/NCServiceIT.java
+++ b/hyracks-fullstack/hyracks/hyracks-server/src/test/java/org/apache/hyracks/server/test/NCServiceIT.java
@@ -23,6 +23,7 @@
import org.apache.commons.httpclient.HttpStatus;
import org.apache.commons.httpclient.methods.GetMethod;
import org.apache.commons.lang3.StringUtils;
+import org.apache.hyracks.server.process.HyracksVirtualCluster;
import org.json.JSONArray;
import org.json.JSONObject;
import org.junit.AfterClass;
@@ -30,6 +31,7 @@
import org.junit.Test;
import java.io.File;
+import java.io.IOException;
import java.net.InetAddress;
import java.util.ArrayList;
import java.util.List;
@@ -38,23 +40,29 @@
public class NCServiceIT {
private static final String TARGET_DIR = StringUtils
- .join(new String[]{System.getProperty("basedir"), "target"}, File.separator);
+ .join(new String[] { System.getProperty("basedir"), "target" }, File.separator);
private static final String LOG_DIR = StringUtils
- .join(new String[]{TARGET_DIR, "surefire-reports"}, File.separator);
+ .join(new String[] { TARGET_DIR, "failsafe-reports" }, File.separator);
private static final String RESOURCE_DIR = StringUtils
- .join(new String[]{TARGET_DIR, "test-classes", "NCServiceIT"}, File.separator);
- private static final String APP_DIR = StringUtils
- .join(new String[]{TARGET_DIR, "appassembler", "bin"}, File.separator);
+ .join(new String[] { TARGET_DIR, "test-classes", "NCServiceIT" }, File.separator);
+ private static final String APP_HOME = StringUtils
+ .join(new String[] { TARGET_DIR, "appassembler" }, File.separator);
private static final Logger LOGGER = Logger.getLogger(NCServiceIT.class.getName());
- private static List<Process> procs = new ArrayList<>();
+
+ private static HyracksVirtualCluster cluster = null;
@BeforeClass
public static void setUp() throws Exception {
- // Start two NC Services - don't read their output as they don't terminate
- procs.add(invoke("nc-red.log", APP_DIR + File.separator + "hyracksncservice",
- "-config-file", RESOURCE_DIR + File.separator + "nc-red.conf"));
- procs.add(invoke("nc-blue.log", APP_DIR + File.separator + "hyracksncservice",
- "-config-file", RESOURCE_DIR + File.separator + "nc-blue.conf"));
+ cluster = new HyracksVirtualCluster(new File(APP_HOME), null);
+ cluster.addNC(
+ new File(RESOURCE_DIR, "nc-red.conf"),
+ new File(LOG_DIR, "nc-red.log")
+ );
+ cluster.addNC(
+ new File(RESOURCE_DIR, "nc-blue.conf"),
+ new File(LOG_DIR, "nc-blue.log")
+ );
+
try {
Thread.sleep(2000);
}
@@ -62,8 +70,11 @@
}
// Start CC
- procs.add(invoke("cc.log", APP_DIR + File.separator + "hyrackscc",
- "-config-file", RESOURCE_DIR + File.separator + "cc.conf"));
+ cluster.start(
+ new File(RESOURCE_DIR, "cc.conf"),
+ new File(LOG_DIR, "cc.log")
+ );
+
try {
Thread.sleep(10000);
}
@@ -72,11 +83,8 @@
}
@AfterClass
- public static void tearDown() throws Exception {
- for (Process p : procs) {
- p.destroy();
- p.waitFor();
- }
+ public static void tearDown() throws IOException {
+ cluster.stop();
}
private static String getHttp(String url) throws Exception {
@@ -95,18 +103,6 @@
} else {
throw new Exception("HTTP error " + statusCode + ":\n" + response);
}
- }
-
- private static Process invoke(String logfile, String... args) throws Exception {
- ProcessBuilder pb = new ProcessBuilder(args);
- pb.redirectErrorStream(true);
- File logDir = new File(LOG_DIR);
- logDir.mkdirs();
- File log = new File(logDir, logfile);
- log.delete();
- pb.redirectOutput(ProcessBuilder.Redirect.appendTo(log));
- Process p = pb.start();
- return p;
}
@Test
@@ -138,5 +134,4 @@
tearDown();
}
}
-
}
diff --git a/hyracks-fullstack/hyracks/hyracks-server/src/test/resources/logging.properties b/hyracks-fullstack/hyracks/hyracks-server/src/test/resources/logging.properties
index c888bb1..e9f8479 100644
--- a/hyracks-fullstack/hyracks/hyracks-server/src/test/resources/logging.properties
+++ b/hyracks-fullstack/hyracks/hyracks-server/src/test/resources/logging.properties
@@ -46,8 +46,8 @@
# Note that the ConsoleHandler also has a separate level
# setting to limit messages printed to the console.
-.level= WARNING
-# .level= INFO
+# .level= WARNING
+.level= INFO
# .level= FINE
# .level = FINEST
--
To view, visit https://asterix-gerrit.ics.uci.edu/958
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: If3eb450782a595cf85d04a2c2e9cc732564e65e6
Gerrit-PatchSet: 4
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Chris Hillery <ce...@lambda.nu>
Gerrit-Reviewer: Chris Hillery <ce...@lambda.nu>
Gerrit-Reviewer: Ian Maxon <im...@apache.org>
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-Reviewer: Till Westmann <ti...@apache.org>
Change in asterixdb[master]: Added NCServiceExecutionIT and programmatic NCService deploy...
Posted by "Chris Hillery (Code Review)" <do...@asterixdb.incubator.apache.org>.
Chris Hillery has posted comments on this change.
Change subject: Added NCServiceExecutionIT and programmatic NCService deployment control.
......................................................................
Patch Set 2:
I do not believe the cross-project changes here are dangerous. Largely I deleted VirtualClusterDriver which was unused by Asterix or VXQuery. I also deleted the now-unused toCommandLine() methods on CCConfig and NCConfig. There are API changes in HyracksCCProcess and HyracksNCProcess, but those classes were unused outside of VirtualClusterDriver.
--
To view, visit https://asterix-gerrit.ics.uci.edu/958
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: If3eb450782a595cf85d04a2c2e9cc732564e65e6
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Chris Hillery <ce...@lambda.nu>
Gerrit-Reviewer: Chris Hillery <ce...@lambda.nu>
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-HasComments: No
Change in asterixdb[master]: ASTERIXDB-1482: Added NCServiceExecutionIT, HyracksVirtualCl...
Posted by "Chris Hillery (Code Review)" <do...@asterixdb.incubator.apache.org>.
Hello Jenkins,
I'd like you to reexamine a change. Please visit
https://asterix-gerrit.ics.uci.edu/958
to look at the new patch set (#3).
Change subject: ASTERIXDB-1482: Added NCServiceExecutionIT, HyracksVirtualCluster.
......................................................................
ASTERIXDB-1482: Added NCServiceExecutionIT, HyracksVirtualCluster.
NCServiceExecutionIT runs all execution tests against a local cluster
managed by the NCService deployment framework.
HyracksVirtualCluster offers programmatic NCService deployment
control along with improved HyracksNCProcess/HyracksCCProcess.
Further fixes and improvements:
1. Fix handling of iodevices/storagedir (ASTERIXDB-1482)
2. Proper handling of [nc] default section in all cases
3. Ensure asterixnc, etc. scripts are executable
4. Consolidate Ini handling
5. Pruned some dead code, including VirtualClusterDriver
6. A bit of refactoring and extended commenting
Change-Id: If3eb450782a595cf85d04a2c2e9cc732564e65e6
---
M asterixdb/asterix-common/src/main/java/org/apache/asterix/common/config/AsterixMetadataProperties.java
M asterixdb/asterix-common/src/main/java/org/apache/asterix/common/config/AsterixPropertiesAccessor.java
M asterixdb/asterix-server/pom.xml
A asterixdb/asterix-server/src/test/java/org/apache/asterix/server/test/NCServiceExecutionIT.java
A asterixdb/asterix-server/src/test/resources/NCServiceExecutionIT/cc.conf
A asterixdb/asterix-server/src/test/resources/NCServiceExecutionIT/ncservice1.conf
A asterixdb/asterix-server/src/test/resources/NCServiceExecutionIT/ncservice2.conf
M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/ClusterControllerService.java
M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/work/TriggerNCWork.java
M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/application/IniApplicationConfig.java
M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/CCConfig.java
M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/IniUtils.java
M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/NCConfig.java
M hyracks-fullstack/hyracks/hyracks-control/hyracks-nc-service/src/main/java/org/apache/hyracks/control/nc/service/NCService.java
M hyracks-fullstack/hyracks/hyracks-server/pom.xml
D hyracks-fullstack/hyracks/hyracks-server/src/main/java/org/apache/hyracks/server/drivers/VirtualClusterDriver.java
M hyracks-fullstack/hyracks/hyracks-server/src/main/java/org/apache/hyracks/server/process/HyracksCCProcess.java
M hyracks-fullstack/hyracks/hyracks-server/src/main/java/org/apache/hyracks/server/process/HyracksNCProcess.java
M hyracks-fullstack/hyracks/hyracks-server/src/main/java/org/apache/hyracks/server/process/HyracksServerProcess.java
A hyracks-fullstack/hyracks/hyracks-server/src/main/java/org/apache/hyracks/server/process/HyracksVirtualCluster.java
M hyracks-fullstack/hyracks/hyracks-server/src/test/java/org/apache/hyracks/server/test/NCServiceIT.java
M hyracks-fullstack/hyracks/hyracks-server/src/test/resources/logging.properties
22 files changed, 536 insertions(+), 369 deletions(-)
git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/58/958/3
--
To view, visit https://asterix-gerrit.ics.uci.edu/958
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If3eb450782a595cf85d04a2c2e9cc732564e65e6
Gerrit-PatchSet: 3
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Chris Hillery <ce...@lambda.nu>
Gerrit-Reviewer: Chris Hillery <ce...@lambda.nu>
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Change in asterixdb[master]: Added NCServiceExecutionIT and implemented several fixes.
Posted by "Jenkins (Code Review)" <do...@asterixdb.incubator.apache.org>.
Jenkins has posted comments on this change.
Change subject: Added NCServiceExecutionIT and implemented several fixes.
......................................................................
Patch Set 1:
WARNING: THIS CHANGE CONTAINS CROSS-PRODUCT CHANGES IN:
* asterixdb
* hyracks-fullstack
PLEASE REVIEW CAREFULLY AND LOOK FOR API CHANGES!
--
To view, visit https://asterix-gerrit.ics.uci.edu/958
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: If3eb450782a595cf85d04a2c2e9cc732564e65e6
Gerrit-PatchSet: 1
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Chris Hillery <ce...@lambda.nu>
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-HasComments: No
Change in asterixdb[master]: ASTERIXDB-1482: Added NCServiceExecutionIT, HyracksVirtualCl...
Posted by "Ian Maxon (Code Review)" <do...@asterixdb.incubator.apache.org>.
Ian Maxon has posted comments on this change.
Change subject: ASTERIXDB-1482: Added NCServiceExecutionIT, HyracksVirtualCluster.
......................................................................
Patch Set 3: Code-Review+1
(3 comments)
A few thoughts/questions
https://asterix-gerrit.ics.uci.edu/#/c/958/3/asterixdb/asterix-server/pom.xml
File asterixdb/asterix-server/pom.xml:
Line 127: <plugin>
What's this plugin about?
https://asterix-gerrit.ics.uci.edu/#/c/958/3/asterixdb/asterix-server/src/test/java/org/apache/asterix/server/test/NCServiceExecutionIT.java
File asterixdb/asterix-server/src/test/java/org/apache/asterix/server/test/NCServiceExecutionIT.java:
Line 135: LOGGER.info("Sleeping while cluster comes online...");
Is there a better way to do this? Maybe trying to fetch against some port on the CC until it returns or a number of retries is reached?
https://asterix-gerrit.ics.uci.edu/#/c/958/3/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/CCConfig.java
File hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/CCConfig.java:
Line 155
Why remove this?
--
To view, visit https://asterix-gerrit.ics.uci.edu/958
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: If3eb450782a595cf85d04a2c2e9cc732564e65e6
Gerrit-PatchSet: 3
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Chris Hillery <ce...@lambda.nu>
Gerrit-Reviewer: Chris Hillery <ce...@lambda.nu>
Gerrit-Reviewer: Ian Maxon <im...@apache.org>
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-Reviewer: Till Westmann <ti...@apache.org>
Gerrit-HasComments: Yes
Change in asterixdb[master]: Added NCServiceExecutionIT and programmatic NCService deploy...
Posted by "Jenkins (Code Review)" <do...@asterixdb.incubator.apache.org>.
Jenkins has posted comments on this change.
Change subject: Added NCServiceExecutionIT and programmatic NCService deployment control.
......................................................................
Patch Set 2:
WARNING: THIS CHANGE CONTAINS CROSS-PRODUCT CHANGES IN:
* asterixdb
* hyracks-fullstack
PLEASE REVIEW CAREFULLY AND LOOK FOR API CHANGES!
--
To view, visit https://asterix-gerrit.ics.uci.edu/958
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: If3eb450782a595cf85d04a2c2e9cc732564e65e6
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Chris Hillery <ce...@lambda.nu>
Gerrit-Reviewer: Chris Hillery <ce...@lambda.nu>
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-HasComments: No
Change in asterixdb[master]: Added NCServiceExecutionIT and implemented several fixes.
Posted by "Jenkins (Code Review)" <do...@asterixdb.incubator.apache.org>.
Jenkins has posted comments on this change.
Change subject: Added NCServiceExecutionIT and implemented several fixes.
......................................................................
Patch Set 1:
Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-notopic/1733/
--
To view, visit https://asterix-gerrit.ics.uci.edu/958
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: If3eb450782a595cf85d04a2c2e9cc732564e65e6
Gerrit-PatchSet: 1
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Chris Hillery <ce...@lambda.nu>
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-HasComments: No
Change in asterixdb[master]: ASTERIXDB-1482: Added NCServiceExecutionIT, HyracksVirtualCl...
Posted by "Jenkins (Code Review)" <do...@asterixdb.incubator.apache.org>.
Jenkins has posted comments on this change.
Change subject: ASTERIXDB-1482: Added NCServiceExecutionIT, HyracksVirtualCluster.
......................................................................
Patch Set 3:
Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-notopic/1735/
--
To view, visit https://asterix-gerrit.ics.uci.edu/958
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: If3eb450782a595cf85d04a2c2e9cc732564e65e6
Gerrit-PatchSet: 3
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Chris Hillery <ce...@lambda.nu>
Gerrit-Reviewer: Chris Hillery <ce...@lambda.nu>
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-HasComments: No
Change in asterixdb[master]: ASTERIXDB-1482: Added NCServiceExecutionIT, HyracksVirtualCl...
Posted by "Chris Hillery (Code Review)" <do...@asterixdb.incubator.apache.org>.
Chris Hillery has posted comments on this change.
Change subject: ASTERIXDB-1482: Added NCServiceExecutionIT, HyracksVirtualCluster.
......................................................................
Patch Set 3:
(3 comments)
https://asterix-gerrit.ics.uci.edu/#/c/958/3/asterixdb/asterix-server/pom.xml
File asterixdb/asterix-server/pom.xml:
Line 127: <plugin>
> What's this plugin about?
I copied these plugin settings from hyracks-server, where it's doing the same thing: making sure that the appassembler-generated .sh scripts are executable.
https://asterix-gerrit.ics.uci.edu/#/c/958/3/asterixdb/asterix-server/src/test/java/org/apache/asterix/server/test/NCServiceExecutionIT.java
File asterixdb/asterix-server/src/test/java/org/apache/asterix/server/test/NCServiceExecutionIT.java:
Line 135: LOGGER.info("Sleeping while cluster comes online...");
> Is there a better way to do this? Maybe trying to fetch against some port o
I'm not sure.. I suppose it could poll the rest/nodes Hyracks endpoint until the expected number of NCs are connected?
https://asterix-gerrit.ics.uci.edu/#/c/958/3/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/CCConfig.java
File hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/CCConfig.java:
Line 155
> Why remove this?
Because it no longer serves any purpose, with VirtualClusterDriver removed. It was always an ugly hack that risked being out of sync, and with the NCService model I'm trying to minimize the use of command-line arguments anyway.
--
To view, visit https://asterix-gerrit.ics.uci.edu/958
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: If3eb450782a595cf85d04a2c2e9cc732564e65e6
Gerrit-PatchSet: 3
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Chris Hillery <ce...@lambda.nu>
Gerrit-Reviewer: Chris Hillery <ce...@lambda.nu>
Gerrit-Reviewer: Ian Maxon <im...@apache.org>
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-Reviewer: Till Westmann <ti...@apache.org>
Gerrit-HasComments: Yes
Change in asterixdb[master]: Added NCServiceExecutionIT and programmatic NCService deploy...
Posted by "Chris Hillery (Code Review)" <do...@asterixdb.incubator.apache.org>.
Chris Hillery has uploaded a new patch set (#2).
Change subject: Added NCServiceExecutionIT and programmatic NCService deployment control.
......................................................................
Added NCServiceExecutionIT and programmatic NCService deployment control.
NCServiceExecutionIT runs all execution tests against a local cluster
managed by the NCService deployment framework.
Programmatic NCService control comprises new HyracksVirtualCluster class
and improved HyracksNCProcess/HyracksCCProcess.
Further fixes and improvements:
1. Fix handling of iodevices/storagedir
2. Proper handling of [nc] default section in all cases
3. Ensure asterixnc, etc. scripts are executable
4. Consolidate Ini handling
5. Pruned some dead code, including VirtualClusterDriver
6. A bit of refactoring and extended commenting
Change-Id: If3eb450782a595cf85d04a2c2e9cc732564e65e6
---
M asterixdb/asterix-common/src/main/java/org/apache/asterix/common/config/AsterixMetadataProperties.java
M asterixdb/asterix-common/src/main/java/org/apache/asterix/common/config/AsterixPropertiesAccessor.java
M asterixdb/asterix-server/pom.xml
A asterixdb/asterix-server/src/test/java/org/apache/asterix/server/test/NCServiceExecutionIT.java
A asterixdb/asterix-server/src/test/resources/NCServiceExecutionIT/cc.conf
A asterixdb/asterix-server/src/test/resources/NCServiceExecutionIT/ncservice1.conf
A asterixdb/asterix-server/src/test/resources/NCServiceExecutionIT/ncservice2.conf
M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/ClusterControllerService.java
M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/work/TriggerNCWork.java
M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/application/IniApplicationConfig.java
M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/CCConfig.java
M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/IniUtils.java
M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/NCConfig.java
M hyracks-fullstack/hyracks/hyracks-control/hyracks-nc-service/src/main/java/org/apache/hyracks/control/nc/service/NCService.java
M hyracks-fullstack/hyracks/hyracks-server/pom.xml
D hyracks-fullstack/hyracks/hyracks-server/src/main/java/org/apache/hyracks/server/drivers/VirtualClusterDriver.java
M hyracks-fullstack/hyracks/hyracks-server/src/main/java/org/apache/hyracks/server/process/HyracksCCProcess.java
M hyracks-fullstack/hyracks/hyracks-server/src/main/java/org/apache/hyracks/server/process/HyracksNCProcess.java
M hyracks-fullstack/hyracks/hyracks-server/src/main/java/org/apache/hyracks/server/process/HyracksServerProcess.java
A hyracks-fullstack/hyracks/hyracks-server/src/main/java/org/apache/hyracks/server/process/HyracksVirtualCluster.java
M hyracks-fullstack/hyracks/hyracks-server/src/test/java/org/apache/hyracks/server/test/NCServiceIT.java
M hyracks-fullstack/hyracks/hyracks-server/src/test/resources/logging.properties
22 files changed, 536 insertions(+), 369 deletions(-)
git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/58/958/2
--
To view, visit https://asterix-gerrit.ics.uci.edu/958
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If3eb450782a595cf85d04a2c2e9cc732564e65e6
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Chris Hillery <ce...@lambda.nu>
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Change in asterixdb[master]: Added NCServiceExecutionIT and programmatic NCService deploy...
Posted by "Jenkins (Code Review)" <do...@asterixdb.incubator.apache.org>.
Jenkins has posted comments on this change.
Change subject: Added NCServiceExecutionIT and programmatic NCService deployment control.
......................................................................
Patch Set 2:
Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-notopic/1734/
--
To view, visit https://asterix-gerrit.ics.uci.edu/958
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: If3eb450782a595cf85d04a2c2e9cc732564e65e6
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Chris Hillery <ce...@lambda.nu>
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-HasComments: No
Change in asterixdb[master]: ASTERIXDB-1482: Added NCServiceExecutionIT, HyracksVirtualCl...
Posted by "Jenkins (Code Review)" <do...@asterixdb.incubator.apache.org>.
Jenkins has posted comments on this change.
Change subject: ASTERIXDB-1482: Added NCServiceExecutionIT, HyracksVirtualCluster.
......................................................................
Patch Set 3:
WARNING: THIS CHANGE CONTAINS CROSS-PRODUCT CHANGES IN:
* asterixdb
* hyracks-fullstack
PLEASE REVIEW CAREFULLY AND LOOK FOR API CHANGES!
--
To view, visit https://asterix-gerrit.ics.uci.edu/958
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: If3eb450782a595cf85d04a2c2e9cc732564e65e6
Gerrit-PatchSet: 3
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Chris Hillery <ce...@lambda.nu>
Gerrit-Reviewer: Chris Hillery <ce...@lambda.nu>
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-HasComments: No