You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@crail.apache.org by pe...@apache.org on 2018/09/06 11:01:10 UTC
[5/5] incubator-crail git commit: Documentation: iobench & fsck
Documentation: iobench & fsck
Add iobench and fsck documentation.
Signed-off-by: Jonas Pfefferle <pe...@apache.org>
Project: http://git-wip-us.apache.org/repos/asf/incubator-crail/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-crail/commit/0e536ca9
Tree: http://git-wip-us.apache.org/repos/asf/incubator-crail/tree/0e536ca9
Diff: http://git-wip-us.apache.org/repos/asf/incubator-crail/diff/0e536ca9
Branch: refs/heads/master
Commit: 0e536ca9382622a4d3bb6602caf2f596d43ddbe5
Parents: f4d3147
Author: Jonas Pfefferle <pe...@apache.org>
Authored: Tue Aug 14 11:26:08 2018 +0200
Committer: Jonas Pfefferle <pe...@apache.org>
Committed: Thu Sep 6 12:59:41 2018 +0200
----------------------------------------------------------------------
doc/source/fsck.rst | 49 +++++++++++++
doc/source/index.rst | 9 ++-
doc/source/iobench.rst | 166 ++++++++++++++++++++++++++++++++++++++++++++
doc/source/shell.rst | 21 ++++++
doc/source/spark.rst | 13 ++++
5 files changed, 256 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-crail/blob/0e536ca9/doc/source/fsck.rst
----------------------------------------------------------------------
diff --git a/doc/source/fsck.rst b/doc/source/fsck.rst
new file mode 100644
index 0000000..f3e0600
--- /dev/null
+++ b/doc/source/fsck.rst
@@ -0,0 +1,49 @@
+fsck
+====
+
+The fsck is used to query Crail internals and perform management operations.
+
+
+Reference
+-----------------
+
+.. list-table::
+ :header-rows: 1
+
+ * - Argument
+ - Default
+ - Experiment type
+ - Description
+ * - :code:`-t <experiment>`
+ - *-*
+ - N/A
+ - * :code:`getLocations`
+ * :code:`directoryDump`
+ * :code:`namenodeDump`
+ * :code:`blockStatistics`
+ * :code:`ping`
+ * :code:`createDirectory`
+ * - :code:`-f <path>`
+ - /tmp.dat
+ - * :code:`getLocations`
+ * :code:`directoryDump`
+ * :code:`namenodeDump`
+ * :code:`blockStatistics`
+ * :code:`createDirectory`
+ - Path to perform operation with
+ * - :code:`-y <offset>`
+ - 0
+ - * :code:`getLocations`
+ - Offset into file
+ * - :code:`-l <length>`
+ - 1
+ - * :code:`getLocations`
+ - Length starting from offset (-y)
+ * - :code:`-c <storage_class>`
+ - 0
+ - * :code:`createDirectory`
+ - Storage class of directory
+ * - :code:`-p <location_class>`
+ - 0
+ - * :code:`createDirectory`
+ - Location class of directory
http://git-wip-us.apache.org/repos/asf/incubator-crail/blob/0e536ca9/doc/source/index.rst
----------------------------------------------------------------------
diff --git a/doc/source/index.rst b/doc/source/index.rst
index 5e8a965..eb8b573 100644
--- a/doc/source/index.rst
+++ b/doc/source/index.rst
@@ -20,11 +20,16 @@ Apache Crail incubating
.. toctree::
:maxdepth: 1
- :caption: Getting started
+ :caption: Applications
+
+ shell
+ iobench
+ fsck
+ spark
.. toctree::
:maxdepth: 1
- :caption: API
+ :caption: Development
.. toctree::
:maxdepth: 1
http://git-wip-us.apache.org/repos/asf/incubator-crail/blob/0e536ca9/doc/source/iobench.rst
----------------------------------------------------------------------
diff --git a/doc/source/iobench.rst b/doc/source/iobench.rst
new file mode 100644
index 0000000..8399bb3
--- /dev/null
+++ b/doc/source/iobench.rst
@@ -0,0 +1,166 @@
+iobench
+=======
+
+The iobench tool allows to perform microbenchmarks on Crail.
+
+Examples
+--------
+
+Synchronously write 1MB 1024 times to get a 1GB file:
+
+.. code-block:: bash
+
+ $CRAIL_HOME/crail iobench -t write -f /filename -s $((1024*1024)) -k 1024
+
+Read 1024 1MB buffers asynchronously with a batch size of 4:
+
+
+.. code-block:: bash
+
+ $CRAIL_HOME/crail iobench -t readSequentialAsync -f /filename -s $((1024*1024)) -k 1024 -b 4
+
+Command Reference
+-----------------
+
+.. list-table::
+ :header-rows: 1
+
+ * - Argument
+ - Default
+ - Experiment type
+ - Description
+ * - :code:`-t <experiment>`
+ - *-*
+ - N/A
+ - * :code:`write` - sequential sync write
+ * :code:`writeAsync` - sequential async write
+ * :code:`readSequential` - sequential sync read
+ * :code:`readRandom` - random sync read
+ * :code:`readSequentialAsync` - sequential async read
+ * :code:`readMultiStream` - multistream read
+ * :code:`createFile` - create file RPC
+ * :code:`createFileAsync` - create file async RPC
+ * :code:`createMultiFile` - create multifile
+ * :code:`getKey` - getKey RPC
+ * :code:`getFile` - getFile sync RPC
+ * :code:`getFileAsync` - getFile async RPC
+ * :code:`enumerateDir` - enumerate directory
+ * :code:`browseDir` - browse directory
+ * :code:`writeInt` - write integer
+ * :code:`readInt` - read integer
+ * :code:`seekInt` - seek integer
+ * :code:`readMultiStreamInt` - read integer multistream
+ * :code:`printLocationclass` - print machine's location class
+ * - :code:`-f <path>`
+ - /tmp.dat
+ - * :code:`write`
+ * :code:`writeAsync`
+ * :code:`readSequential`
+ * :code:`readRandom`
+ * :code:`readSequentialAsync`
+ * :code:`readMultiStream`
+ * :code:`createFile`
+ * :code:`createFileAsync`
+ * :code:`createMultiFile`
+ * :code:`getKey`
+ * :code:`getFile`
+ * :code:`getFileAsync`
+ * :code:`enumerateDir`
+ * :code:`browseDir`
+ * :code:`writeInt`
+ * :code:`readInt`
+ * :code:`seekInt`
+ * :code:`readMultiStreamInt`
+ - Path to perform operation with
+ * - :code:`-s <size>`
+ - :ref:`crail.buffersize <crail-site.conf>`
+ - * :code:`write`
+ * :code:`writeAsync`
+ * :code:`readSequential`
+ * :code:`readRandom`
+ * :code:`readSequentialAsync`
+ * :code:`readMultiStream`
+ * :code:`getKey`
+ - Buffer size in bytes. Only relevant for buffered experiments.
+ * - :code:`-k <n>`
+ - 1
+ - * :code:`write`
+ * :code:`writeAsync`
+ * :code:`readSequential`
+ * :code:`readRandom`
+ * :code:`readSequentialAsync`
+ * :code:`readMultiStream`
+ * :code:`createFile`
+ * :code:`createFileAsync`
+ * :code:`getKey`
+ * :code:`getFile`
+ * :code:`getFileAsync`
+ * :code:`writeInt`
+ * :code:`readInt`
+ * :code:`seekInt`
+ * :code:`readMultiStreamInt`
+ - Number of operations to perform
+ * - :code:`-b <size>`
+ - 1
+ - * :code:`writeAsync`
+ * :code:`readSequentialAsync`
+ * :code:`readMultiStream`
+ * :code:`createFileAsync`
+ * :code:`createMultiFile`
+ * :code:`getFileAsync`
+ * :code:`enumerateDir`
+ * :code:`readMultiStreamInt`
+ - Batch size of asynchronous requests.
+ * - :code:`-c <storage_class>`
+ - 0
+ - * :code:`write`
+ * :code:`writeAsync`
+ * :code:`createFile`
+ - Storage class of file.
+ * - :code:`-p <location_class>`
+ - 0
+ - * :code:`write`
+ * :code:`writeAsync`
+ * :code:`createFile`
+ - Location class of file
+ * - :code:`-w <n>`
+ - 32
+ - * :code:`write`
+ * :code:`writeAsync`
+ * :code:`readSequential`
+ * :code:`readRandom`
+ * :code:`readSequentialAsync`
+ * :code:`readMultiStream`
+ * :code:`createFile`
+ * :code:`createFileAsync`
+ * :code:`getFile`
+ * :code:`getFileAsync`
+ * :code:`enumerateDir`
+ - Number of warmup operations
+ * - :code:`-e <experiments>`
+ - 1
+ - * :code:`readSequential`
+ * :code:`readRandom`
+ * :code:`readSequentialAsync`
+ * :code:`readMultiStream`
+ - Number of experiments to run
+ * - :code:`-o <true/false>`
+ - false
+ - * :code:`readSequential`
+ * :code:`readRandom`
+ * :code:`readSequentialAsync`
+ * :code:`readMultiStream`
+ - Keep file system open between experiments
+ * - :code:`-d <true/false>`
+ - false
+ - * :code:`write`
+ * :code:`writeAsync`
+ - Skip writing directory record
+ * - :code:`-m <true/false`
+ - true
+ - * :code:`write`
+ * :code:`readSequential`
+ * :code:`readRandom`
+ - Use buffered streams
+
+
http://git-wip-us.apache.org/repos/asf/incubator-crail/blob/0e536ca9/doc/source/shell.rst
----------------------------------------------------------------------
diff --git a/doc/source/shell.rst b/doc/source/shell.rst
new file mode 100644
index 0000000..be3ba8b
--- /dev/null
+++ b/doc/source/shell.rst
@@ -0,0 +1,21 @@
+Shell
+=====
+
+Crail provides an implementation of the HDFS API thus allows interaction using the HDFS shell.
+For the HDFS adapter to work properly the :ref:`core-site.xml` needs to be configured
+properly.
+
+.. code-block:: bash
+
+ $CRAIL_HOME/crail fs
+
+Not all shell commands are support but the following operations have been tested to work:
+
+.. code-block:: bash
+
+ $CRAIL_HOME/crail fs -ls <crail_path>
+ $CRAIL_HOME/crail fs -mkdir <crail_path>
+ $CRAIL_HOME/crail fs -copyFromLocal <local_path> <crail_path>
+ $CRAIL_HOME/crail fs -copyToLocal <crail_path> <local_path>
+ $CRAIL_HOME/crail fs -cat <crail_path>
+
http://git-wip-us.apache.org/repos/asf/incubator-crail/blob/0e536ca9/doc/source/spark.rst
----------------------------------------------------------------------
diff --git a/doc/source/spark.rst b/doc/source/spark.rst
new file mode 100644
index 0000000..3f222ad
--- /dev/null
+++ b/doc/source/spark.rst
@@ -0,0 +1,13 @@
+Spark
+=====
+
+Spark-IO
+--------
+
+Crail-TeraSort
+--------------
+
+SQL
+---
+
+