You are viewing a plain text version of this content. The canonical link for it is here.
Posted to oak-commits@jackrabbit.apache.org by md...@apache.org on 2018/04/24 08:57:00 UTC
svn commit: r1829969 - in /jackrabbit/oak/trunk:
oak-doc/src/site/markdown/nodestore/segment/ oak-run/
oak-run/src/main/java/org/apache/jackrabbit/oak/run/
oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/iotrace/
oak-segment-tar/sr...
Author: mduerig
Date: Tue Apr 24 08:56:59 2018
New Revision: 1829969
URL: http://svn.apache.org/viewvc?rev=1829969&view=rev
Log:
OAK-7402: Expose UI for collecting IO traces
Introduce a iotrace run mode in oak-run
Added:
jackrabbit/oak/trunk/oak-run/src/main/java/org/apache/jackrabbit/oak/run/IOTraceCommand.java
jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/iotrace/Traces.java
Removed:
jackrabbit/oak/trunk/oak-segment-tar/src/test/java/org/apache/jackrabbit/oak/segment/tool/iotrace/IOTracerRunner.java
Modified:
jackrabbit/oak/trunk/oak-doc/src/site/markdown/nodestore/segment/overview.md
jackrabbit/oak/trunk/oak-run/README.md
jackrabbit/oak/trunk/oak-run/src/main/java/org/apache/jackrabbit/oak/run/AvailableModes.java
Modified: jackrabbit/oak/trunk/oak-doc/src/site/markdown/nodestore/segment/overview.md
URL: http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-doc/src/site/markdown/nodestore/segment/overview.md?rev=1829969&r1=1829968&r2=1829969&view=diff
==============================================================================
--- jackrabbit/oak/trunk/oak-doc/src/site/markdown/nodestore/segment/overview.md (original)
+++ jackrabbit/oak/trunk/oak-doc/src/site/markdown/nodestore/segment/overview.md Tue Apr 24 08:56:59 2018
@@ -30,6 +30,7 @@
* [Check](#check)
* [Compact](#compact)
* [Debug](#debug)
+ * [IOTrace](#iotrace)
* [Diff](#diff)
* [History](#history)
@@ -819,6 +820,38 @@ The pair of record IDs can be followed b
When a node record ID range is specified, the tool will perform a diff between the two nodes pointed by the record IDs, optionally following the provided path.
The result of the diff will be printed in JSOP format.
+### <a name="iotrace"/> IOTrace
+
+````
+java -jar oak-run.jar iotrace PATH --trace DEPTH|BREADTH [--depth DEPTH] [--mmap MMAP] [--output OUTPUT] [--path PATH] [--segment-cache SEGMENT_CACHE]
+
+usage: iotrace path/to/segmentstore <options>
+Option (* = required) Description
+--------------------- -----------
+--depth <Integer> Maximal depth of the traversal (default: 5)
+--mmap <Boolean> use memory mapping for the file store (default: true)
+--output <File> output file where the IO trace is written to (default: iotrace.csv)
+--path <String> starting path for the traversal (default: /root)
+--segment-cache <Integer> size of the segment cache in MB (default: 256)
+--trace <Traces> (*) type of the traversal. Either of [DEPTH, BREADTH]
+````
+
+The `iotrace` command collects IO traces of read accesses to the segment store's back-end
+(e.g. disk). Traffic patterns can be specified via the `--trace` option. Permissible values
+are `DEPTH` for depth first traversal and `BREADTH` for breadth first traversal. The `--depth`
+option limits the maximum number of levels traversed. The `--path` option specifies the node
+where traversal starts (from the super root). The `--mmap` and `--segment-cache` options
+configure memory mapping and segment cache size of the segment store, respectively. The
+`--output` options specifies the file where the IO trace is stored. IO traces are stored in
+CSV format of the following form:
+
+```
+timestamp,file,segmentId,length,elapsed
+1522147945084,data01415a.tar,f81378df-b3f8-4b25-0000-00000002c450,181328,171849
+1522147945096,data01415a.tar,f81378df-b3f8-4b25-0000-00000002c450,181328,131272
+1522147945097,data01415a.tar,f81378df-b3f8-4b25-0000-00000002c450,181328,142766
+```
+
### <a name="diff"/> Diff
```
Modified: jackrabbit/oak/trunk/oak-run/README.md
URL: http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-run/README.md?rev=1829969&r1=1829968&r2=1829969&view=diff
==============================================================================
--- jackrabbit/oak/trunk/oak-run/README.md (original)
+++ jackrabbit/oak/trunk/oak-run/README.md Tue Apr 24 08:56:59 2018
@@ -27,6 +27,7 @@ The following runmodes are currently ava
* unlockUpgrade : Unlock a DocumentMK upgrade to a newer version
* upgrade : Migrate existing Jackrabbit 2.x repository to Oak.
* export : Export repository content as json
+ * iotrace : Collect a trace of segment store read accesses
Some of the features related to Jackrabbit 2.x are provided by oak-run-jr2 jar. See
@@ -78,6 +79,12 @@ Debug
See the [official documentation](http://jackrabbit.apache.org/oak/docs/nodestore/segment/overview.html#debug).
+IOTrace
+-----
+
+See the [official documentation](http://jackrabbit.apache.org/oak/docs/nodestore/segment/overview.html#iotrace).
+
+
Console
-------
Modified: jackrabbit/oak/trunk/oak-run/src/main/java/org/apache/jackrabbit/oak/run/AvailableModes.java
URL: http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-run/src/main/java/org/apache/jackrabbit/oak/run/AvailableModes.java?rev=1829969&r1=1829968&r2=1829969&view=diff
==============================================================================
--- jackrabbit/oak/trunk/oak-run/src/main/java/org/apache/jackrabbit/oak/run/AvailableModes.java (original)
+++ jackrabbit/oak/trunk/oak-run/src/main/java/org/apache/jackrabbit/oak/run/AvailableModes.java Tue Apr 24 08:56:59 2018
@@ -56,6 +56,7 @@ public final class AvailableModes {
.put("unlockupgrade", new UnlockUpgradeCommand())
.put(IndexCommand.NAME, new IndexCommand())
.put(NodeStateExportCommand.NAME, new NodeStateExportCommand())
+ .put(IOTraceCommand.NAME, new IOTraceCommand())
.put("server", new ServerCommand())
.build());
}
Added: jackrabbit/oak/trunk/oak-run/src/main/java/org/apache/jackrabbit/oak/run/IOTraceCommand.java
URL: http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-run/src/main/java/org/apache/jackrabbit/oak/run/IOTraceCommand.java?rev=1829969&view=auto
==============================================================================
--- jackrabbit/oak/trunk/oak-run/src/main/java/org/apache/jackrabbit/oak/run/IOTraceCommand.java (added)
+++ jackrabbit/oak/trunk/oak-run/src/main/java/org/apache/jackrabbit/oak/run/IOTraceCommand.java Tue Apr 24 08:56:59 2018
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.jackrabbit.oak.run;
+
+import static java.lang.String.format;
+import static org.apache.jackrabbit.oak.segment.FileStoreHelper.isValidFileStoreOrFail;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.PrintStream;
+import java.util.Arrays;
+
+import javax.annotation.Nonnull;
+
+import joptsimple.ArgumentAcceptingOptionSpec;
+import joptsimple.OptionException;
+import joptsimple.OptionParser;
+import joptsimple.OptionSet;
+import org.apache.jackrabbit.oak.run.commons.Command;
+import org.apache.jackrabbit.oak.segment.tool.iotrace.IOTracer;
+import org.apache.jackrabbit.oak.segment.tool.iotrace.Traces;
+
+/**
+ * Command line utility for collection {@link IOTracer io traces}
+ *
+ <pre>
+ usage: iotrace path/to/segmentstore <options>
+ Option (* = required) Description
+ --------------------- -----------
+ --depth <Integer> Maximal depth of the traversal (default: 5)
+ --mmap <Boolean> use memory mapping for the file store (default: true)
+ --output <File> output file where the IO trace is written to (default: iotrace.csv)
+ --path <String> starting path for the traversal (default: /root)
+ --segment-cache <Integer> size of the segment cache in MB (default: 256)
+ --trace <Traces> (*) type of the traversal. Either of [DEPTH, BREADTH]
+ </pre>
+ */
+class IOTraceCommand implements Command {
+ public static final String NAME = "iotrace";
+
+ @Override
+ public void execute(String... args) throws Exception {
+ OptionParser optionParser = new OptionParser();
+ ArgumentAcceptingOptionSpec<Traces> traceOption = optionParser
+ .accepts("trace", "type of the traversal. Either of " + Arrays.toString(Traces.values()))
+ .withRequiredArg()
+ .ofType(Traces.class)
+ .required();
+
+ ArgumentAcceptingOptionSpec<File> outputOption = optionParser
+ .accepts("output", "output file where the IO trace is written to")
+ .withRequiredArg()
+ .ofType(File.class)
+ .defaultsTo(new File("iotrace.csv"));
+
+ ArgumentAcceptingOptionSpec<Boolean> mmapOption = optionParser
+ .accepts("mmap", "use memory mapping for the file store")
+ .withRequiredArg()
+ .ofType(Boolean.class)
+ .defaultsTo(true);
+
+ ArgumentAcceptingOptionSpec<Integer> segmentCacheOption = optionParser
+ .accepts("segment-cache", "size of the segment cache in MB")
+ .withRequiredArg()
+ .ofType(Integer.class)
+ .defaultsTo(256);
+
+ ArgumentAcceptingOptionSpec<Integer> depthOption = optionParser
+ .accepts("depth", "Maximal depth of the traversal")
+ .withRequiredArg()
+ .ofType(Integer.class)
+ .defaultsTo(5);
+
+ ArgumentAcceptingOptionSpec<String> pathOption = optionParser
+ .accepts("path", "starting path for the traversal")
+ .withRequiredArg()
+ .ofType(String.class)
+ .defaultsTo("/root");
+
+ try {
+ OptionSet options = optionParser.parse(args);
+
+ if (options.nonOptionArguments().size() != 1) {
+ printUsage(optionParser, System.err);
+ System.exit(1);
+ }
+
+ File segmentStore = isValidFileStoreOrFail(new File(options.nonOptionArguments().get(0).toString()));
+ Traces trace = traceOption.value(options);
+ File output = outputOption.value(options);
+ Boolean mmap = mmapOption.value(options);
+ Integer segmentCache = segmentCacheOption.value(options);
+ Integer depth = depthOption.value(options);
+ String path = pathOption.value(options);
+
+ System.out.println(
+ format("%s first traversing %d levels of %s starting at %s", trace, depth, segmentStore, path));
+ System.out.println(
+ format("mmap=%b, segment cache=%d", mmap, segmentCache));
+ System.out.println(
+ format("Writing trace to %s", output));
+
+ trace.collectIOTrace(segmentStore, mmap, segmentCache, path, depth, output);
+ } catch (OptionException e) {
+ printUsage(optionParser, System.err, e.getMessage());
+ System.exit(1);
+ }
+ }
+
+ private static void printUsage(
+ @Nonnull OptionParser parser,
+ @Nonnull PrintStream err,
+ @Nonnull String... messages)
+ throws IOException {
+ for (String message : messages) {
+ err.println(message);
+ }
+
+ err.println("usage: " + NAME + " path/to/segmentstore <options>");
+ parser.printHelpOn(err);
+ }
+}
Added: jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/iotrace/Traces.java
URL: http://svn.apache.org/viewvc/jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/iotrace/Traces.java?rev=1829969&view=auto
==============================================================================
--- jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/iotrace/Traces.java (added)
+++ jackrabbit/oak/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/tool/iotrace/Traces.java Tue Apr 24 08:56:59 2018
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.jackrabbit.oak.segment.tool.iotrace;
+
+import static org.apache.jackrabbit.oak.segment.file.FileStoreBuilder.fileStoreBuilder;
+import static org.apache.jackrabbit.oak.segment.tool.iotrace.IOTracer.newIOTracer;
+
+import java.io.BufferedWriter;
+import java.io.File;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.util.function.Function;
+
+import javax.annotation.Nonnull;
+
+import org.apache.jackrabbit.oak.segment.file.FileStore;
+import org.apache.jackrabbit.oak.segment.file.FileStoreBuilder;
+import org.apache.jackrabbit.oak.segment.file.InvalidFileStoreVersionException;
+import org.apache.jackrabbit.oak.segment.spi.monitor.IOMonitor;
+
+/**
+ * Utility class for running the various {@link Trace} implementations.
+ */
+public enum Traces {
+
+ /**
+ * Utility for running a {@link DepthFirstTrace depth first trace}.
+ */
+ DEPTH {
+
+ /**
+ * Collect an IO trace on a segment store using a {@link DepthFirstTrace depth first trace}.
+ * @param segmentStore path to the segment store
+ * @param mmap whether to enable memory mapping
+ * @param segmentCacheSize size of the segment cache in MB
+ * @param path path to the root node (starting from the super root) from where
+ * to start traversing
+ * @param depth maximal number of levels to traverse
+ * @param output output file where the trace is written to
+ * @throws IOException
+ */
+ @Override
+ public void collectIOTrace(
+ @Nonnull File segmentStore,
+ boolean mmap,
+ int segmentCacheSize,
+ @Nonnull String path,
+ int depth,
+ @Nonnull File output)
+ throws IOException {
+ try (PrintWriter out = new PrintWriter(new BufferedWriter(new FileWriter(output, true)))) {
+ Function<IOMonitor, FileStore> factory = ioMonitor ->
+ newFileStore(fileStoreBuilder(segmentStore)
+ .withMemoryMapping(mmap)
+ .withSegmentCacheSize(segmentCacheSize)
+ .withIOMonitor(ioMonitor));
+
+ IOTracer ioTracer = newIOTracer(factory, out, DepthFirstTrace.CONTEXT_SPEC);
+ ioTracer.collectTrace(
+ new DepthFirstTrace(depth, path, ioTracer::setContext));
+ }
+ }
+
+ },
+
+ /**
+ * Utility for running a {@link BreadthFirstTrace breadth first trace}.
+ */
+ BREADTH {
+
+ /**
+ * Collect an IO trace on a segment store using a {@link BreadthFirstTrace breadth first trace}.
+ * @param segmentStore path to the segment store
+ * @param mmap whether to enable memory mapping
+ * @param segmentCacheSize size of the segment cache in MB
+ * @param path path to the root node (starting from the super root) from where
+ * to start traversing
+ * @param depth maximal number of levels to traverse
+ * @param output output file where the trace is written to
+ * @throws IOException
+ */
+ @Override
+ public void collectIOTrace(
+ @Nonnull File segmentStore,
+ boolean mmap,
+ int segmentCacheSize,
+ @Nonnull String path,
+ int depth,
+ @Nonnull File output)
+ throws IOException {
+ try (PrintWriter out = new PrintWriter(new BufferedWriter(new FileWriter(output, true)))) {
+ Function<IOMonitor, FileStore> factory = ioMonitor ->
+ newFileStore(fileStoreBuilder(segmentStore)
+ .withMemoryMapping(mmap)
+ .withSegmentCacheSize(segmentCacheSize)
+ .withIOMonitor(ioMonitor));
+
+ IOTracer ioTracer = newIOTracer(factory, out, BreadthFirstTrace.CONTEXT_SPEC);
+ ioTracer.collectTrace(
+ new BreadthFirstTrace(depth, path, ioTracer::setContext));
+ }
+ }
+ };
+
+ public abstract void collectIOTrace(
+ @Nonnull File segmentStore,
+ boolean mmap,
+ int segmentCacheSize,
+ @Nonnull String path,
+ int depth,
+ @Nonnull File output)
+ throws IOException, InvalidFileStoreVersionException;
+
+ @Nonnull
+ private static FileStore newFileStore(FileStoreBuilder fileStoreBuilder) {
+ try {
+ return fileStoreBuilder.build();
+ } catch (InvalidFileStoreVersionException | IOException e) {
+ throw new IllegalStateException(e);
+ }
+ }
+}