You are viewing a plain text version of this content. The canonical link for it is here.
Posted to pr@cassandra.apache.org by GitBox <gi...@apache.org> on 2020/12/08 20:40:52 UTC
[GitHub] [cassandra-harry] grighetto opened a new pull request #5: Enable execution against external cluster / Cleanup
grighetto opened a new pull request #5:
URL: https://github.com/apache/cassandra-harry/pull/5
The goal of this PR is to enable execution of Harry against a running, remote/external, C* cluster. As of now, in master, the main HarryRunner is hardcoded to run in-JVM and the example config YAML is not used.
Summary of changes:
- Isolated in-JVM dtest dependency (cassandra-dtest-shaded) into `harry-integration` module, in such a way that the project can be built without that module if all we want to do is running against an external cluster. That means, we don't need to build C* nor the shaded jars beforehand. In order to build Harry without the in-JVM module, run: `mvn -pl '!harry-integration' clean install -DskipTests`.
- Created a new module for the execution against an external cluster, `harry-integration-external`. That Maven module produces an uber jar with all dependencies included for standalone execution.
- Added run-jvm.sh and run-external.sh scripts to help with kicking the tires off with the project.
- Fixed invalid properties in example.yaml.
- Added example YAML for execution against an external cluster.
- Removed hardcoded configuration from HarryRunner and relied on the provided config files instead.
- Fixed object mapping for `ApproximateMonotonicClockConfiguration`.
- Fixed logback configuration in pom.xml.
- Fixed NPE thrown at successful execution by HarryRunner.
- Adjusted the Dockerfile and related scripts to match the rest of the changes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra-harry] ifesdjeen commented on a change in pull request #5: Enable execution against external cluster / Cleanup
Posted by GitBox <gi...@apache.org>.
ifesdjeen commented on a change in pull request #5:
URL: https://github.com/apache/cassandra-harry/pull/5#discussion_r539131065
##########
File path: conf/example.yaml
##########
@@ -31,9 +31,9 @@ truncate_table: false
# to map it back to the logical timestamp of the operation that wrote this value.
clock:
approximate_monotonic:
- historySize: 7300
- epochPeriod: 1
- epochTimeUnit: "SECONDS"
+ history_size: 7300
+ epoch_length: 1
+ epoch_time_unit: "SECONDS"
Review comment:
well spotted! However, if this has happened, we need to rename java props as well, since jackson will attempt to dump properties with their field names.
##########
File path: harry-runner/src/harry/runner/HarryRunner.java
##########
@@ -36,21 +38,13 @@
import harry.corruptor.QueryResponseCorruptor;
import harry.model.ExhaustiveChecker;
import harry.model.OpSelectors;
-import harry.model.sut.InJvmSut;
-import org.apache.cassandra.distributed.impl.IsolatedExecutor;
-import org.apache.cassandra.distributed.test.TestBaseImpl;
-public class HarryRunner extends TestBaseImpl
+public interface HarryRunner
Review comment:
In this case, should we move inteface to `core`?
##########
File path: harry-runner/src/harry/runner/HarryRunner.java
##########
@@ -59,37 +53,7 @@ public void runWithInJvmDtest() throws Throwable
ScheduledThreadPoolExecutor executor = new ScheduledThreadPoolExecutor(1);
executor.setRemoveOnCancelPolicy(true);
- Configuration.ConfigurationBuilder configuration = new Configuration.ConfigurationBuilder();
-
- long seed = System.currentTimeMillis();
- configuration.setSeed(seed)
- .setSUT(new InJvmSut.InJvmSutConfiguration(3,
- 10,
- System.getProperty("harry.root", "/tmp/cassandra/harry/") + System.currentTimeMillis()))
- .setPartitionDescriptorSelector(new Configuration.DefaultPDSelectorConfiguration(10, 100))
- .setClusteringDescriptorSelector((builder) -> {
- builder.setNumberOfModificationsDistribution(new Configuration.ConstantDistributionConfig(10))
- .setRowsPerModificationDistribution(new Configuration.ConstantDistributionConfig(10))
- .setMaxPartitionSize(100)
- .setOperationKindWeights(new Configuration.OperationKindSelectorBuilder()
- .addWeight(OpSelectors.OperationKind.DELETE_ROW, 1)
- .addWeight(OpSelectors.OperationKind.DELETE_COLUMN, 1)
- .addWeight(OpSelectors.OperationKind.DELETE_RANGE, 1)
- .addWeight(OpSelectors.OperationKind.WRITE, 97)
- .build());
- })
- .setRowVisitor(new Configuration.DefaultRowVisitorConfiguration())
- .setClock(new Configuration.ApproximateMonotonicClockConfiguration((int) TimeUnit.HOURS.toSeconds(2) + 100,
- 1,
- TimeUnit.SECONDS))
- .setRunTime(2, TimeUnit.HOURS)
- .setCreateSchema(true)
- .setTruncateTable(false)
- .setDropSchema(false)
- .setModel(ExhaustiveChecker::new)
- .setRunner(new Configuration.ConcurrentRunnerConfig(1, 1, 1));
-
- Runner runner = configuration.build().createRunner();
Review comment:
This is great. I should've done it this way in the first place, but this code existed long before there was even any configuration. Great job!
##########
File path: conf/example.yaml
##########
@@ -31,9 +31,9 @@ truncate_table: false
# to map it back to the logical timestamp of the operation that wrote this value.
clock:
approximate_monotonic:
- historySize: 7300
- epochPeriod: 1
- epochTimeUnit: "SECONDS"
+ history_size: 7300
+ epoch_length: 1
+ epoch_time_unit: "SECONDS"
Review comment:
turns out there are more places like this. If you like you can just leave it and i'll fix it later, as you wish.
##########
File path: harry-runner/src/harry/runner/HarryRunner.java
##########
@@ -107,7 +71,8 @@ public void runWithInJvmDtest() throws Throwable
return a;
}).get(run.snapshot.run_time_unit.toSeconds(run.snapshot.run_time) + 30,
TimeUnit.SECONDS);
- ((Throwable) result).printStackTrace();
Review comment:
We can (and probably should) switch to `logger.error` here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra-harry] grighetto commented on a change in pull request #5: Enable execution against external cluster / Cleanup
Posted by GitBox <gi...@apache.org>.
grighetto commented on a change in pull request #5:
URL: https://github.com/apache/cassandra-harry/pull/5#discussion_r538852168
##########
File path: pom.xml
##########
@@ -98,8 +99,8 @@
--add-opens jdk.management/com.sun.management.internal=ALL-UNNAMED
-Dcassandra.config=file://${session.executionRootDirectory}/test/conf/cassandra.yaml
- -Dlog4j.configurationFile=file://${session.executionRootDirectory}/test/conf/log4j2-test.xml
- -Dcassandra.test.logConfigPath=${session.executionRootDirectory}/test/conf/log4j2-dtest.xml
Review comment:
the filename was incorrect so it was throwing a FileNotFound exception
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra-harry] ifesdjeen commented on a change in pull request #5: Enable execution against external cluster / Cleanup
Posted by GitBox <gi...@apache.org>.
ifesdjeen commented on a change in pull request #5:
URL: https://github.com/apache/cassandra-harry/pull/5#discussion_r540945554
##########
File path: conf/external.yaml
##########
@@ -0,0 +1,95 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+seed: 1596731732524
+
+# Default schema provider generates random schema
+schema_provider:
+ default: {}
+
+drop_schema: false
+create_schema: true
+truncate_table: false
+
+# Clock is a component responsible for mapping _logical_ timestamps to _real-time_ ones.
+#
+# When reproducing test failures, and for validation purposes, a snapshot of such clock can
+# be taken to map a real-time timestamp from the value retrieved from the database in order
+# to map it back to the logical timestamp of the operation that wrote this value.
+clock:
+ approximate_monotonic:
+ history_size: 7300
+ epoch_length: 1
+ epoch_time_unit: "SECONDS"
+
+# Runner is a is a component that schedules operations that change the cluster (system under test)
+# and model state.
+runner:
+ concurrent:
+ writer_threads: 2
+ round_robin_validator_threads: 1
+ recent_partition_validator_threads: 1
+
+run_time: 2
+run_time_unit: "HOURS"
+
+# System under test: a Cassandra node or cluster. Default implementation is in_jvm (in-jvm DTest cluster).
+# Harry also supports external clusters.
+system_under_test:
+ external:
+ contact_points: 127.0.0.1
+ port: 9042
+ username: null
+ password: null
+
+# Model is responsible for tracking logical timestamps that
+model:
+ exhaustive_checker:
+ max_seen_lts: 19
Review comment:
we can fix it after commit, but we should not set logical timestamps for a new checker, since it'll break verification
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra-harry] grighetto commented on a change in pull request #5: Enable execution against external cluster / Cleanup
Posted by GitBox <gi...@apache.org>.
grighetto commented on a change in pull request #5:
URL: https://github.com/apache/cassandra-harry/pull/5#discussion_r538845469
##########
File path: harry-core/pom.xml
##########
@@ -58,18 +58,6 @@
<artifactId>jackson-annotations</artifactId>
</dependency>
- <dependency>
- <groupId>org.apache.cassandra</groupId>
- <artifactId>cassandra-dtest-shaded</artifactId>
- <scope>test</scope>
- </dependency>
-
- <dependency>
- <groupId>org.apache.cassandra</groupId>
- <artifactId>dtest-api</artifactId>
- <scope>test</scope>
- </dependency>
-
Review comment:
this allows for building the project without pre-building the dtest shaded library, for execution against an external cluster
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra-harry] grighetto commented on a change in pull request #5: Enable execution against external cluster / Cleanup
Posted by GitBox <gi...@apache.org>.
grighetto commented on a change in pull request #5:
URL: https://github.com/apache/cassandra-harry/pull/5#discussion_r538852471
##########
File path: pom.xml
##########
@@ -98,8 +99,8 @@
--add-opens jdk.management/com.sun.management.internal=ALL-UNNAMED
-Dcassandra.config=file://${session.executionRootDirectory}/test/conf/cassandra.yaml
Review comment:
this file was missing, so I copied one from the C* codebase (you'll see it below)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra-harry] grighetto commented on a change in pull request #5: Enable execution against external cluster / Cleanup
Posted by GitBox <gi...@apache.org>.
grighetto commented on a change in pull request #5:
URL: https://github.com/apache/cassandra-harry/pull/5#discussion_r538847058
##########
File path: harry-integration-external/pom.xml
##########
@@ -0,0 +1,93 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
+
+ <modelVersion>4.0.0</modelVersion>
+ <packaging>jar</packaging>
+
+ <parent>
+ <groupId>org.apache.cassandra</groupId>
+ <version>0.0.1-SNAPSHOT</version>
+ <artifactId>harry-parent</artifactId>
+ </parent>
+
+ <artifactId>harry-integration-external</artifactId>
+ <name>Harry Integration - External</name>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.cassandra</groupId>
+ <artifactId>harry-core</artifactId>
+ <version>${project.parent.version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.cassandra</groupId>
+ <artifactId>harry-runner</artifactId>
+ <version>${project.parent.version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>junit</groupId>
+ <artifactId>junit</artifactId>
+ <scope>test</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>ch.qos.logback</groupId>
+ <artifactId>logback-classic</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.quicktheories</groupId>
+ <artifactId>quicktheories</artifactId>
+ <version>0.25</version>
+ <scope>test</scope>
+ </dependency>
+ </dependencies>
+
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-shade-plugin</artifactId>
+ <version>3.2.4</version>
+ <executions>
+ <execution>
+ <phase>package</phase>
+ <goals>
+ <goal>shade</goal>
+ </goals>
+ <configuration>
+ <transformers>
+ <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
+ <mainClass>harry.runner.external.HarryRunnerExternal</mainClass>
Review comment:
This pom is for the new `external` module and as such it doesn't depend on the in-JVM dtest libs.
Here we build an uber jar for standalone execution.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra-harry] grighetto commented on a change in pull request #5: Enable execution against external cluster / Cleanup
Posted by GitBox <gi...@apache.org>.
grighetto commented on a change in pull request #5:
URL: https://github.com/apache/cassandra-harry/pull/5#discussion_r538843283
##########
File path: conf/example.yaml
##########
@@ -31,9 +31,9 @@ truncate_table: false
# to map it back to the logical timestamp of the operation that wrote this value.
clock:
approximate_monotonic:
- historySize: 7300
- epochPeriod: 1
- epochTimeUnit: "SECONDS"
+ history_size: 7300
+ epoch_length: 1
+ epoch_time_unit: "SECONDS"
Review comment:
the prop names incorrectly matched the Java var names
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra-harry] grighetto commented on a change in pull request #5: Enable execution against external cluster / Cleanup
Posted by GitBox <gi...@apache.org>.
grighetto commented on a change in pull request #5:
URL: https://github.com/apache/cassandra-harry/pull/5#discussion_r539687200
##########
File path: harry-runner/src/harry/runner/HarryRunner.java
##########
@@ -107,7 +71,8 @@ public void runWithInJvmDtest() throws Throwable
return a;
}).get(run.snapshot.run_time_unit.toSeconds(run.snapshot.run_time) + 30,
TimeUnit.SECONDS);
- ((Throwable) result).printStackTrace();
Review comment:
Fixed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra-harry] grighetto commented on a change in pull request #5: Enable execution against external cluster / Cleanup
Posted by GitBox <gi...@apache.org>.
grighetto commented on a change in pull request #5:
URL: https://github.com/apache/cassandra-harry/pull/5#discussion_r538849169
##########
File path: harry-runner/src/harry/runner/HarryRunner.java
##########
@@ -36,21 +38,13 @@
import harry.corruptor.QueryResponseCorruptor;
import harry.model.ExhaustiveChecker;
import harry.model.OpSelectors;
-import harry.model.sut.InJvmSut;
-import org.apache.cassandra.distributed.impl.IsolatedExecutor;
-import org.apache.cassandra.distributed.test.TestBaseImpl;
-public class HarryRunner extends TestBaseImpl
+public interface HarryRunner
Review comment:
I turned this into an interface with a default implementation that's further implemented by both `HarryRunnerJvm` and `HarryRunnerExternal` and removed the dependency on `TestBaseImpl`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra-harry] ifesdjeen merged pull request #5: Enable execution against external cluster / Cleanup
Posted by GitBox <gi...@apache.org>.
ifesdjeen merged pull request #5:
URL: https://github.com/apache/cassandra-harry/pull/5
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra-harry] grighetto commented on a change in pull request #5: Enable execution against external cluster / Cleanup
Posted by GitBox <gi...@apache.org>.
grighetto commented on a change in pull request #5:
URL: https://github.com/apache/cassandra-harry/pull/5#discussion_r538844039
##########
File path: conf/example.yaml
##########
@@ -88,3 +88,7 @@ clustering_descriptor_selector:
DELETE_COLUMN: 1
column_mask_bitsets: null
max_partition_size: 100
+
+# Default Row Visitor
+row_visitor:
+ default: {}
Review comment:
this is a required config (without it it throws a NPE). It was set in the hardcoded runner, but not here in the YAML.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra-harry] grighetto commented on a change in pull request #5: Enable execution against external cluster / Cleanup
Posted by GitBox <gi...@apache.org>.
grighetto commented on a change in pull request #5:
URL: https://github.com/apache/cassandra-harry/pull/5#discussion_r538852909
##########
File path: pom.xml
##########
@@ -179,6 +180,13 @@
<artifactId>hamcrest-all</artifactId>
<version>1.3</version>
</dependency>
+
+ <!-- this should only be included in the uber jar for standalone execution or as a test dep -->
+ <dependency>
+ <groupId>ch.qos.logback</groupId>
+ <artifactId>logback-classic</artifactId>
+ <version>1.2.3</version>
+ </dependency>
Review comment:
we include this in the harry-integration-external uber jar for standalone execution
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra-harry] grighetto commented on a change in pull request #5: Enable execution against external cluster / Cleanup
Posted by GitBox <gi...@apache.org>.
grighetto commented on a change in pull request #5:
URL: https://github.com/apache/cassandra-harry/pull/5#discussion_r539686514
##########
File path: conf/example.yaml
##########
@@ -31,9 +31,9 @@ truncate_table: false
# to map it back to the logical timestamp of the operation that wrote this value.
clock:
approximate_monotonic:
- historySize: 7300
- epochPeriod: 1
- epochTimeUnit: "SECONDS"
+ history_size: 7300
+ epoch_length: 1
+ epoch_time_unit: "SECONDS"
Review comment:
Fixed remaining var names in `DebugApproximateMonotonicClockConfiguration`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra-harry] grighetto commented on a change in pull request #5: Enable execution against external cluster / Cleanup
Posted by GitBox <gi...@apache.org>.
grighetto commented on a change in pull request #5:
URL: https://github.com/apache/cassandra-harry/pull/5#discussion_r538845780
##########
File path: harry-core/src/harry/core/Configuration.java
##########
@@ -68,6 +68,7 @@
.disable(YAMLGenerator.Feature.CANONICAL_OUTPUT)
.enable(YAMLGenerator.Feature.INDENT_ARRAYS));
mapper.registerSubtypes(Configuration.DebugApproximateMonotonicClockConfiguration.class);
+ mapper.registerSubtypes(Configuration.ApproximateMonotonicClockConfiguration.class);
Review comment:
this was missing so it was complaining when reading the example config
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra-harry] grighetto commented on a change in pull request #5: Enable execution against external cluster / Cleanup
Posted by GitBox <gi...@apache.org>.
grighetto commented on a change in pull request #5:
URL: https://github.com/apache/cassandra-harry/pull/5#discussion_r538844698
##########
File path: docker/Dockerfile.local
##########
@@ -26,8 +26,11 @@ RUN mkdir -p /cassandra/harry
COPY ./harry-core/target/lib/* /opt/harry/lib/
COPY ./harry-core/target/*.jar /opt/harry/lib/
COPY ./harry-runner/target/lib/* /opt/harry/lib/
-COPY ./harry-runner/target/*.jar /opt/harry/
+COPY ./harry-runner/target/*.jar /opt/harry/lib/
+COPY ./harry-integration/target/lib/ /opt/harry/lib/
+COPY ./harry-integration/target/*.jar /opt/harry/
COPY ./test/conf/logback-dtest.xml /opt/harry/test/conf/logback-dtest.xml
+COPY ./conf/example.yaml /opt/harry/example.yaml
Review comment:
since the config is not hardcoded anymore, we have to include the example config
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra-harry] grighetto commented on a change in pull request #5: Enable execution against external cluster / Cleanup
Posted by GitBox <gi...@apache.org>.
grighetto commented on a change in pull request #5:
URL: https://github.com/apache/cassandra-harry/pull/5#discussion_r538850472
##########
File path: harry-runner/src/harry/runner/HarryRunner.java
##########
@@ -152,6 +117,28 @@ public void tryRun(IsolatedExecutor.ThrowingRunnable runnable)
}
}
+ /**
+ * Parses the command-line args and returns a File for the configuration YAML.
+ * @param args Command-line args.
+ * @return Configuration YAML file.
+ * @throws Exception If file is not found or cannot be read.
+ */
+ default File loadConfig(String[] args) throws Exception {
Review comment:
just some common validation for loading the config file
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra-harry] grighetto commented on pull request #5: Enable execution against external cluster / Cleanup
Posted by GitBox <gi...@apache.org>.
grighetto commented on pull request #5:
URL: https://github.com/apache/cassandra-harry/pull/5#issuecomment-742157313
@ifesdjeen I believe I addressed all the outstanding issues.
Also, I ran the unit and integration tests, the standalone in-JVM and external runners and the docker image, everything looks good.
Would you mind taking another pass at this PR?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra-harry] grighetto commented on a change in pull request #5: Enable execution against external cluster / Cleanup
Posted by GitBox <gi...@apache.org>.
grighetto commented on a change in pull request #5:
URL: https://github.com/apache/cassandra-harry/pull/5#discussion_r538849745
##########
File path: harry-runner/src/harry/runner/HarryRunner.java
##########
@@ -59,37 +53,7 @@ public void runWithInJvmDtest() throws Throwable
ScheduledThreadPoolExecutor executor = new ScheduledThreadPoolExecutor(1);
executor.setRemoveOnCancelPolicy(true);
- Configuration.ConfigurationBuilder configuration = new Configuration.ConfigurationBuilder();
-
- long seed = System.currentTimeMillis();
- configuration.setSeed(seed)
- .setSUT(new InJvmSut.InJvmSutConfiguration(3,
- 10,
- System.getProperty("harry.root", "/tmp/cassandra/harry/") + System.currentTimeMillis()))
- .setPartitionDescriptorSelector(new Configuration.DefaultPDSelectorConfiguration(10, 100))
- .setClusteringDescriptorSelector((builder) -> {
- builder.setNumberOfModificationsDistribution(new Configuration.ConstantDistributionConfig(10))
- .setRowsPerModificationDistribution(new Configuration.ConstantDistributionConfig(10))
- .setMaxPartitionSize(100)
- .setOperationKindWeights(new Configuration.OperationKindSelectorBuilder()
- .addWeight(OpSelectors.OperationKind.DELETE_ROW, 1)
- .addWeight(OpSelectors.OperationKind.DELETE_COLUMN, 1)
- .addWeight(OpSelectors.OperationKind.DELETE_RANGE, 1)
- .addWeight(OpSelectors.OperationKind.WRITE, 97)
- .build());
- })
- .setRowVisitor(new Configuration.DefaultRowVisitorConfiguration())
- .setClock(new Configuration.ApproximateMonotonicClockConfiguration((int) TimeUnit.HOURS.toSeconds(2) + 100,
- 1,
- TimeUnit.SECONDS))
- .setRunTime(2, TimeUnit.HOURS)
- .setCreateSchema(true)
- .setTruncateTable(false)
- .setDropSchema(false)
- .setModel(ExhaustiveChecker::new)
- .setRunner(new Configuration.ConcurrentRunnerConfig(1, 1, 1));
-
- Runner runner = configuration.build().createRunner();
Review comment:
all the config above will come from the YAML files now
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra-harry] grighetto commented on a change in pull request #5: Enable execution against external cluster / Cleanup
Posted by GitBox <gi...@apache.org>.
grighetto commented on a change in pull request #5:
URL: https://github.com/apache/cassandra-harry/pull/5#discussion_r539687018
##########
File path: harry-runner/src/harry/runner/HarryRunner.java
##########
@@ -36,21 +38,13 @@
import harry.corruptor.QueryResponseCorruptor;
import harry.model.ExhaustiveChecker;
import harry.model.OpSelectors;
-import harry.model.sut.InJvmSut;
-import org.apache.cassandra.distributed.impl.IsolatedExecutor;
-import org.apache.cassandra.distributed.test.TestBaseImpl;
-public class HarryRunner extends TestBaseImpl
+public interface HarryRunner
Review comment:
Done!
And removed the harry-runner module completely.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra-harry] grighetto commented on a change in pull request #5: Enable execution against external cluster / Cleanup
Posted by GitBox <gi...@apache.org>.
grighetto commented on a change in pull request #5:
URL: https://github.com/apache/cassandra-harry/pull/5#discussion_r538850040
##########
File path: harry-runner/src/harry/runner/HarryRunner.java
##########
@@ -107,7 +71,8 @@ public void runWithInJvmDtest() throws Throwable
return a;
}).get(run.snapshot.run_time_unit.toSeconds(run.snapshot.run_time) + 30,
TimeUnit.SECONDS);
- ((Throwable) result).printStackTrace();
Review comment:
this was throwing an NPE on successful execution
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra-harry] grighetto commented on a change in pull request #5: Enable execution against external cluster / Cleanup
Posted by GitBox <gi...@apache.org>.
grighetto commented on a change in pull request #5:
URL: https://github.com/apache/cassandra-harry/pull/5#discussion_r538844461
##########
File path: conf/external.yaml
##########
@@ -0,0 +1,95 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+seed: 1596731732524
+
+# Default schema provider generates random schema
+schema_provider:
+ default: {}
+
+drop_schema: false
+create_schema: true
+truncate_table: false
+
+# Clock is a component responsible for mapping _logical_ timestamps to _real-time_ ones.
+#
+# When reproducing test failures, and for validation purposes, a snapshot of such clock can
+# be taken to map a real-time timestamp from the value retrieved from the database in order
+# to map it back to the logical timestamp of the operation that wrote this value.
+clock:
+ approximate_monotonic:
+ history_size: 7300
+ epoch_length: 1
+ epoch_time_unit: "SECONDS"
+
+# Runner is a is a component that schedules operations that change the cluster (system under test)
+# and model state.
+runner:
+ concurrent:
+ writer_threads: 2
+ round_robin_validator_threads: 1
+ recent_partition_validator_threads: 1
+
+run_time: 2
+run_time_unit: "HOURS"
+
+# System under test: a Cassandra node or cluster. Default implementation is in_jvm (in-jvm DTest cluster).
+# Harry also supports external clusters.
+system_under_test:
+ external:
+ contact_points: 127.0.0.1
+ port: 9042
+ username: null
+ password: null
Review comment:
support for external cluster
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org