You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@crail.apache.org by pe...@apache.org on 2018/10/18 14:44:45 UTC
[1/2] incubator-crail git commit: Documentation: Add programming
documentation
Repository: incubator-crail
Updated Branches:
refs/heads/master 34e49d29d -> 3a2c8098c
Documentation: Add programming documentation
Add documentation how to program against Crail.
https://jira.apache.org/jira/projects/CRAIL/issues/CRAIL-55
Signed-off-by: Jonas Pfefferle <pe...@apache.org>
Project: http://git-wip-us.apache.org/repos/asf/incubator-crail/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-crail/commit/3a2c8098
Tree: http://git-wip-us.apache.org/repos/asf/incubator-crail/tree/3a2c8098
Diff: http://git-wip-us.apache.org/repos/asf/incubator-crail/diff/3a2c8098
Branch: refs/heads/master
Commit: 3a2c8098c716bad8893bfdbec1bc2ea97811ecae
Parents: cdb5401
Author: Jonas Pfefferle <pe...@apache.org>
Authored: Thu Oct 18 16:30:42 2018 +0200
Committer: Jonas Pfefferle <pe...@apache.org>
Committed: Thu Oct 18 16:43:35 2018 +0200
----------------------------------------------------------------------
doc/source/develop.rst | 58 +++++++++++++++++++++++++++++++++++++++++++++
doc/source/index.rst | 2 ++
2 files changed, 60 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-crail/blob/3a2c8098/doc/source/develop.rst
----------------------------------------------------------------------
diff --git a/doc/source/develop.rst b/doc/source/develop.rst
new file mode 100644
index 0000000..d0c3845
--- /dev/null
+++ b/doc/source/develop.rst
@@ -0,0 +1,58 @@
+Programming against Crail
+=========================
+
+The best way to program against Crail is to use Maven. Make sure you have the Crail dependency specified in your application pom.xml file
+with the latest Crail version (e.g. 1.1-incubating):
+
+.. code-block:: xml
+
+ <dependency>
+ <groupId>org.apache.crail</groupId>
+ <artifactId>crail-client</artifactId>
+ <version>X.Y</version>
+ </dependency>
+
+Then, create a Crail client as follows:
+
+.. code-block:: Java
+
+ CrailConfiguration conf = new CrailConfiguration();
+ CrailStore store = CrailStore.newInstance(conf);
+
+Make sure the :code:`$CRAIL_HOME/conf` directory is part of the classpath.
+
+Crail supports different file types. The simplest way to create a file in Crail is as follows:
+
+.. code-block:: Java
+
+ CrailFile file = store.create(filename, CrailNodeType.DATAFILE, CrailStorageClass.DEFAULT, CrailLocationClass.DEFAULT).get().syncDir();
+
+Aside from the actual filename, the :code:`create()` call takes as input the storage and location classes which are preferences for the storage tier and physical location that this file should be created in. Crail tries to satisfy these preferences later when the file is written. In the example we do not request any particular storage or location affinity.
+
+This :code:`create()` command is non-blocking, calling :code:`get()` on the returning future object awaits the completion of the call. At that time, the file has been created, but its directory entry may not be visible. Therefore, the file may not yet show up in a file enumeration of the given parent directory. Calling :code:`syncDir()` waits to for the directory entry to be completed. Both the :code:`get()` and the :code:`syncDir()` operation can be deffered to a later time at which they may become non-blocking operations.
+
+Once the file is created, a file stream can be obtained for writing:
+
+.. code-block:: Java
+
+ CrailBufferedOutputStream outstream = file.getBufferedOutputStream(1024);
+
+Here, we create a buffered stream so that we can pass heap byte arrays as well. We could also create a non-buffered stream using
+
+.. code-block:: Java
+
+ CrailOutputStream outstream = file.getDirectOutputStream(1024);
+
+In both cases, we pass a write hint (1024 in the example) that indicates to Crail how much data we are intending to write. This allows Crail to optimize metadatanode lookups. Crail never prefetches data, but it may fetch the metadata of the very next operation concurrently with the current data operation if the write hint allows to do so.
+
+Once the stream has been obtained, there exist various ways to write a file. The code snippet below shows the use of the asynchronous interface:
+
+.. code-block:: Java
+
+ CrailBuffer dataBuf = fs.allocateBuffer();
+ Future<DataResult> future = outputStream.write(dataBuf);
+ ...
+ future.get();
+
+Reading files works very similar to writing. There exist various examples in org.apache.crail.tools.CrailBenchmark.
+
http://git-wip-us.apache.org/repos/asf/incubator-crail/blob/3a2c8098/doc/source/index.rst
----------------------------------------------------------------------
diff --git a/doc/source/index.rst b/doc/source/index.rst
index eb8b573..3a2666e 100644
--- a/doc/source/index.rst
+++ b/doc/source/index.rst
@@ -31,6 +31,8 @@ Apache Crail incubating
:maxdepth: 1
:caption: Development
+ develop
+
.. toctree::
:maxdepth: 1
:caption: Help
[2/2] incubator-crail git commit: HISTORY: bug fixes and improvements
of v1.1
Posted by pe...@apache.org.
HISTORY: bug fixes and improvements of v1.1
Add history of bug fixes and improvements of v1.1
release.
Signed-off-by: Jonas Pfefferle <pe...@apache.org>
Project: http://git-wip-us.apache.org/repos/asf/incubator-crail/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-crail/commit/cdb54016
Tree: http://git-wip-us.apache.org/repos/asf/incubator-crail/tree/cdb54016
Diff: http://git-wip-us.apache.org/repos/asf/incubator-crail/diff/cdb54016
Branch: refs/heads/master
Commit: cdb5401617afccce5def9f3b6c3ab0bdbfac34fa
Parents: 34e49d2
Author: Jonas Pfefferle <pe...@apache.org>
Authored: Thu Oct 18 15:42:20 2018 +0200
Committer: Jonas Pfefferle <pe...@apache.org>
Committed: Thu Oct 18 16:43:35 2018 +0200
----------------------------------------------------------------------
HISTORY.md | 31 +++++++++++++++++++++++++++++++
1 file changed, 31 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-crail/blob/cdb54016/HISTORY.md
----------------------------------------------------------------------
diff --git a/HISTORY.md b/HISTORY.md
index aeec25b..33626ff 100644
--- a/HISTORY.md
+++ b/HISTORY.md
@@ -17,6 +17,37 @@ limitations under the License.
{% endcomment %}
-->
+## [1.1](https://github.com/apache/incubator-crail/tree/v1.1) - 18.10.2018
+
+#### New features / improvements
+
+* [[CRAIL-5](https://jira.apache.org/jira/projects/CRAIL/issues/CRAIL-5)] How to I contribute to Crail?
+* [[CRAIL-15](https://jira.apache.org/jira/projects/CRAIL/issues/CRAIL-15)] Better default configuration
+* [[CRAIL-19](https://jira.apache.org/jira/projects/CRAIL/issues/CRAIL-19)] Add checkstyle plugin
+* [[CRAIL-20](https://jira.apache.org/jira/projects/CRAIL/issues/CRAIL-20)] Statistics for storage and rpc client
+* [[CRAIL-24](https://jira.apache.org/jira/projects/CRAIL/issues/CRAIL-24)] New documentation using readthedocs
+* [[CRAIL-27](https://jira.apache.org/jira/projects/CRAIL/issues/CRAIL-27)] HDFS adapter implementation improvement
+* [[CRAIL-34](https://jira.apache.org/jira/projects/CRAIL/issues/CRAIL-34)] Use Apache parent pom
+* [[CRAIL-36](https://jira.apache.org/jira/projects/CRAIL/issues/CRAIL-36)] Add incubation disclaimer to README
+* [[CRAIL-39](https://jira.apache.org/jira/projects/CRAIL/issues/CRAIL-39)] Add download page to website
+* [[CRAIL-41](https://jira.apache.org/jira/projects/CRAIL/issues/CRAIL-41)] Use DiSNI/DaRPC v1.6
+* [[CRAIL-49](https://jira.apache.org/jira/projects/CRAIL/issues/CRAIL-49)] Add Docker container
+* [[CRAIL-50](https://jira.apache.org/jira/projects/CRAIL/issues/CRAIL-50)] Use DiSNI/DaRPC v1.7
+* [[CRAIL-53](https://jira.apache.org/jira/projects/CRAIL/issues/CRAIL-53)] Add missing license headers and exclude configuration files from rat check
+* [[CRAIL-54](https://jira.apache.org/jira/projects/CRAIL/issues/CRAIL-54)] Tar source files / scm tag is now HEAD
+* [[CRAIL-55](https://jira.apache.org/jira/projects/CRAIL/issues/CRAIL-55)] Add programming documentation
+
+
+#### Bug fixes
+
+* [[CRAIL-26](https://jira.apache.org/jira/projects/CRAIL/issues/CRAIL-26)] HDFS statistics
+* [[CRAIL-29](https://jira.apache.org/jira/projects/CRAIL/issues/CRAIL-29)] Fix starting multiple datanodes on same host
+* [[CRAIL-40](https://jira.apache.org/jira/projects/CRAIL/issues/CRAIL-40)] Fix startup script memory requirement
+* [[CRAIL-45](https://jira.apache.org/jira/projects/CRAIL/issues/CRAIL-45)] Fix Nullpointer exception when datapath is not accessible
+* [[CRAIL-46](https://jira.apache.org/jira/projects/CRAIL/issues/CRAIL-46)] Clarify incubating status on website
+* [[CRAIL-51](https://jira.apache.org/jira/projects/CRAIL/issues/CRAIL-51)] Fix subsitution of core-site.xml in Docker container
+* [[CRAIL-52](https://jira.apache.org/jira/projects/CRAIL/issues/CRAIL-52)] Fix CRAIL_HOME in Docker container
+
## [1.0](https://github.com/apache/incubator-crail/tree/v1.0) - 23.05.2018
This is our first Apache incubator release. Below are new features and bug fixes since the import to Apache.