You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@marmotta.apache.org by wi...@apache.org on 2015/12/16 13:28:24 UTC
[1/3] marmotta git commit: MARMOTTA-621: latest changes to get the
ostrich running in docker
Repository: marmotta
Updated Branches:
refs/heads/develop dd1994a92 -> 27889eb17
MARMOTTA-621: latest changes to get the ostrich running in docker
Project: http://git-wip-us.apache.org/repos/asf/marmotta/repo
Commit: http://git-wip-us.apache.org/repos/asf/marmotta/commit/02d04c42
Tree: http://git-wip-us.apache.org/repos/asf/marmotta/tree/02d04c42
Diff: http://git-wip-us.apache.org/repos/asf/marmotta/diff/02d04c42
Branch: refs/heads/develop
Commit: 02d04c4276e703196384a79c9e339d624b0139af
Parents: dd1994a
Author: Sergio Fernández <wi...@apache.org>
Authored: Wed Dec 16 13:20:52 2015 +0100
Committer: Sergio Fernández <wi...@apache.org>
Committed: Wed Dec 16 13:20:52 2015 +0100
----------------------------------------------------------------------
libraries/ostrich/Dockerfile | 76 ++++++++++++++++++++++
libraries/ostrich/README.md | 101 ++++++++++++++++++++++++++++++
libraries/ostrich/backend/Dockerfile | 73 ---------------------
libraries/ostrich/backend/README.md | 99 -----------------------------
4 files changed, 177 insertions(+), 172 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/marmotta/blob/02d04c42/libraries/ostrich/Dockerfile
----------------------------------------------------------------------
diff --git a/libraries/ostrich/Dockerfile b/libraries/ostrich/Dockerfile
new file mode 100644
index 0000000..887684d
--- /dev/null
+++ b/libraries/ostrich/Dockerfile
@@ -0,0 +1,76 @@
+# Dockerfile for Apache Marmotta Ostrich Triplestore
+
+FROM debian:stretch
+MAINTAINER Sergio Fernández <wi...@apache.org>
+
+ADD . /src
+WORKDIR /src
+
+# configuration
+ENV DEBIAN_FRONTEND noninteractive
+ENV OSTRICH_PATH /opt/ostrich
+ENV DB_PATH /data/ostrich
+ENV DB_PORT 10000
+
+# base environment
+RUN apt-get update -qq \
+ && apt-get install -qq -y \
+ locales \
+ apt-utils \
+ git \
+ && localedef -i en_US -c -f UTF-8 -A /usr/share/locale/locale.alias en_US.UTF-8
+ENV LANG en_US.utf8
+
+RUN apt-get update -qq \
+ && apt-get install -y \
+ build-essential \
+ autoconf \
+ libtool \
+ cmake \
+ libc++-dev \
+ python-all-dev \
+ python-virtualenv \
+ && apt-get install -y \
+ libraptor2-dev \
+ librasqal3-dev \
+ libgoogle-glog-dev \
+ libgflags-dev \
+ libleveldb-dev \
+ zlib1g-dev \
+ libgflags-dev \
+ libgtest-dev \
+ libboost-all-dev \
+ libgoogle-perftools-dev
+
+RUN git clone https://github.com/grpc/grpc.git \
+ && cd grpc \
+ && git submodule update --init --recursive \
+ && make \
+ && cd third_party/protobuf \
+ && make install \
+ && cd ../.. \
+ && make install \
+ && cd
+
+# build
+RUN cd backend \
+ && mkdir build \
+ && cd build \
+ && cmake -D CMAKE_BUILD_TYPE=Release .. \
+ && make \
+ && cd
+RUN mkdir -p ${DB_PATH}
+RUN mkdir ${OSTRICH_PATH}
+RUN cp -r /src/backend/build/* ${OSTRICH_PATH}/
+
+# clean
+RUN apt-get -y clean -qq \
+ && apt-get -y autoclean -qq \
+ && apt-get -y autoremove -qq \
+ && rm -rf /var/lib/apt/lists/* \
+ && rm -rf /src
+
+EXPOSE 10000
+
+CMD ${OSTRICH_PATH}/persistence/marmotta_persistence -db ${DB_PATH} -port ${DB_PORT}
+
http://git-wip-us.apache.org/repos/asf/marmotta/blob/02d04c42/libraries/ostrich/README.md
----------------------------------------------------------------------
diff --git a/libraries/ostrich/README.md b/libraries/ostrich/README.md
new file mode 100644
index 0000000..b94fdf4
--- /dev/null
+++ b/libraries/ostrich/README.md
@@ -0,0 +1,101 @@
+# Apache Marmotta Ostrich Backend
+
+This repository implements an experimental high-performance backend for Apache Marmotta
+using LevelDB as storage and gRPC as communication channel between the Java frontend
+and the C++ backend.
+
+If it proves to be useful, the repository will eventually be merged into the main
+development branch of Apache Marmotta
+
+## Dependencies (C++)
+
+To compile the C++ backend, you need to have the following dependencies installed:
+
+ * libraptor (used for parsing/serializing in C++)
+ * librasqal (used for server-side SPARQL evaluation)
+ * libglog (logging)
+ * libgflags (command line arguments)
+ * libleveldb (database backend)
+ * libgrpc (gRPC runtime)
+ * libprotobuf (messaging, data model)
+
+With the exception of libgrpc and libprotobuf, all libraries are available in Linux repositories.
+Debian:
+
+ apt-get install libraptor2-dev librasqal3-dev libgoogle-glog-dev libgflags-dev libleveldb-dev
+
+The backend uses the new Proto 3 format and the gRPC SDK. These need to be installed separately;
+please follow the instructions at [https://github.com/grpc/grpc](https://github.com/grpc/grpc/blob/master/INSTALL).
+
+
+## Compilation (C++)
+
+The backend uses cmake to compile the modules. Create a new directory `build`, run cmake, and run make:
+
+ cd backend
+ mkdir build && cd build
+ cmake ..
+ make
+ cd ..
+
+## Compilation (Java)
+
+The frontend is compiled with Maven and depends on many Apache Marmotta modules to work. Build it with
+
+ mvn clean install
+
+## Running C++ Backend
+
+Start the backend from the cmake build directory as follows:
+
+ ./backend/build/persistence/marmotta_persistence -db /path/to/database -port 10000
+
+The binary accepts many different options. Please see `--help` for details.
+
+## Running Docker
+
+The C++ backend can be ran in the provided Docker image. Just build it:
+
+ docker build -t apachemarmotta/ostrich .
+
+Then you can run Ostrich as a container:
+
+ docker run -t -d -p 10000:10000 apachemarmotta/ostrich
+
+connecting normally to `localhost:10000`.
+
+## Running Sharding
+
+The repository contains an experimental implementation of a sharding server that proxies and
+distributes requests based on a hash calculation over statements. In heavy load environments,
+this is potentially much faster than running a single persistence backend. The setup requires
+several persistence backends (shards) and a sharding proxy. To experiment, you can start these
+on the same machine as follows:
+
+ ./backend/build/persistence/marmotta_persistence -db /path/to/shard1 -port 10001
+ ./backend/build/persistence/marmotta_persistence -db /path/to/shard2 -port 10002
+ ./sharding/marmotta_sharding --port 10000 --backends localhost:10001,localhost:10002
+
+You can then access the sharding server through Marmotta like the persistence server. Running all instances
+on the same host is only useful for testing. In production environments, you would of course run all three
+(or more) instances on different hosts. Note that the number and order of backends should not change once
+data has been imported, because otherwise the hashing algorithm will do the wrong thing.
+
+## Running Apache Marmotta
+
+There is a `ostrich` Maven profile to run the webapp launcher:
+
+ cd launchers/marmotta-webapp
+ mvn tomcat7:run -Postrich
+
+Afterwards, point your browser to [localhost:8080](http://localhost:8080/).
+
+## Command Line Client
+
+A C++ command line client is available for very fast bulk imports and simple queries. To import
+a large turtle file, run:
+
+ ./client/marmotta_client --format=turtle import file.ttl
+
+The client connects by default to `localhost:10000` (change with `--host` and `--port` flags).
+
http://git-wip-us.apache.org/repos/asf/marmotta/blob/02d04c42/libraries/ostrich/backend/Dockerfile
----------------------------------------------------------------------
diff --git a/libraries/ostrich/backend/Dockerfile b/libraries/ostrich/backend/Dockerfile
deleted file mode 100644
index 82e4da1..0000000
--- a/libraries/ostrich/backend/Dockerfile
+++ /dev/null
@@ -1,73 +0,0 @@
-# Dockerfile for Apache Marmotta Ostrich Triplestore
-
-FROM debian:stretch
-MAINTAINER Sergio Fernández <wi...@apache.org>
-
-ADD . /src
-WORKDIR /src
-
-# configuration
-ENV DEBIAN_FRONTEND noninteractive
-ENV DB_PATH /data/ostrich
-ENV DB_PORT 10000
-
-# base environment
-RUN apt-get update -qq \
- && apt-get install -qq -y \
- locales \
- apt-utils \
- git \
- && localedef -i en_US -c -f UTF-8 -A /usr/share/locale/locale.alias en_US.UTF-8
-ENV LANG en_US.utf8
-
-RUN apt-get update -qq \
- && apt-get install -y \
- build-essential \
- autoconf \
- libtool \
- cmake \
- libc++-dev \
- python-all-dev \
- python-virtualenv \
- && apt-get install -y \
- libraptor2-dev \
- librasqal3-dev \
- libgoogle-glog-dev \
- libgflags-dev \
- libleveldb-dev \
- zlib1g-dev \
- libgflags-dev \
- libgtest-dev \
- libboost-all-dev \
- libgoogle-perftools-dev
-
-RUN git clone https://github.com/grpc/grpc.git \
- && cd grpc \
- && git submodule update --init --recursive \
- && make \
- && cd third_party/protobuf \
- && make install \
- && cd ../.. \
- && make install \
- && cd ..
-
-# build
-RUN mkdir build \
- && cd build \
- && cmake -D CMAKE_BUILD_TYPE=Release .. \
- && make \
- && cd ..
-
-# clean
-RUN apt-get -y clean -qq \
- && apt-get -y autoclean -qq \
- && apt-get -y autoremove -qq \
- && rm -rf /var/lib/apt/lists/* \
- && rm -rf /src/grpc
-
-RUN mkdir -p ${DB_PATH}
-
-EXPOSE 10000
-
-CMD ["/src/build/persistence/marmotta_persistence", "-db", "${DB_PATH}", "-port" "${DB_PORT}"]
-
http://git-wip-us.apache.org/repos/asf/marmotta/blob/02d04c42/libraries/ostrich/backend/README.md
----------------------------------------------------------------------
diff --git a/libraries/ostrich/backend/README.md b/libraries/ostrich/backend/README.md
deleted file mode 100644
index 3eab294..0000000
--- a/libraries/ostrich/backend/README.md
+++ /dev/null
@@ -1,99 +0,0 @@
-# Apache Marmotta Ostrich Backend
-
-This repository implements an experimental high-performance backend for Apache Marmotta
-using LevelDB as storage and gRPC as communication channel between the Java frontend
-and the C++ backend.
-
-If it proves to be useful, the repository will eventually be merged into the main
-development branch of Apache Marmotta
-
-## Dependencies (C++)
-
-To compile the C++ backend, you need to have the following dependencies installed:
-
- * libraptor (used for parsing/serializing in C++)
- * librasqal (used for server-side SPARQL evaluation)
- * libglog (logging)
- * libgflags (command line arguments)
- * libleveldb (database backend)
- * libgrpc (gRPC runtime)
- * libprotobuf (messaging, data model)
-
-With the exception of libgrpc and libprotobuf, all libraries are available in Linux repositories.
-Debian:
-
- apt-get install libraptor2-dev librasqal3-dev libgoogle-glog-dev libgflags-dev libleveldb-dev
-
-The backend uses the new Proto 3 format and the gRPC SDK. These need to be installed separately;
-please follow the instructions at [https://github.com/grpc/grpc](https://github.com/grpc/grpc/blob/master/INSTALL).
-
-
-## Compilation (C++)
-
-The backend uses cmake to compile the modules. Create a new directory `build`, run cmake, and run make:
-
- mkdir build && cd build
- cmake ..
- make
-
-## Compilation (Java)
-
-The frontend is compiled with Maven and depends on many Apache Marmotta modules to work. Build it with
-
- cd java
- mvn clean install
-
-## Running C++ Backend
-
-Start the backend from the cmake build directory as follows:
-
- ./persistence/marmotta_persistence -db /path/to/database -port 10000
-
-The binary accepts many different options. Please see `--help` for details.
-
-## Running Docker
-
-The C++ backend can be ran in the provided Docker image. Just build it:
-
- docker build -t apachemarmotta/ostrich .
-
-Then you can run Ostrich as a container:
-
- docker run -t -d -p 8080:8080 -p 10000:10000 apachemarmotta/ostrich
-
-connecting normally to `localhost:10000`.
-
-## Running Sharding
-
-The repository contains an experimental implementation of a sharding server that proxies and
-distributes requests based on a hash calculation over statements. In heavy load environments,
-this is potentially much faster than running a single persistence backend. The setup requires
-several persistence backends (shards) and a sharding proxy. To experiment, you can start these
-on the same machine as follows:
-
- ./service/marmotta_persistence -db /path/to/shard1 -port 10001
- ./service/marmotta_persistence -db /path/to/shard2 -port 10002
- ./sharding/marmotta_sharding --port 10000 --backends localhost:10001,localhost:10002
-
-You can then access the sharding server through Marmotta like the persistence server. Running all instances
-on the same host is only useful for testing. In production environments, you would of course run all three
-(or more) instances on different hosts. Note that the number and order of backends should not change once
-data has been imported, because otherwise the hashing algorithm will do the wrong thing.
-
-## Running Apache Marmotta
-
-A preconfigured version of Apache Marmotta is available in `java/webapp`. It connects to
-`localhost:10000` by default and can be started with:
-
- mvn tomcat7:run
-
-Afterwards, point your browser to `localhost:8080`.
-
-## Command Line Client
-
-A C++ command line client is available for very fast bulk imports and simple queries. To import
-a large turtle file, run:
-
- ./client/marmotta_client --format=turtle import file.ttl
-
-The client connects by default to `localhost:10000` (change with `--host` and `--port` flags).
[2/3] marmotta git commit: MARMOTTA-621: added docker hub instructions
Posted by wi...@apache.org.
MARMOTTA-621: added docker hub instructions
Project: http://git-wip-us.apache.org/repos/asf/marmotta/repo
Commit: http://git-wip-us.apache.org/repos/asf/marmotta/commit/0eba24fc
Tree: http://git-wip-us.apache.org/repos/asf/marmotta/tree/0eba24fc
Diff: http://git-wip-us.apache.org/repos/asf/marmotta/diff/0eba24fc
Branch: refs/heads/develop
Commit: 0eba24fc7de7d23b807350172c2e05a5f2e408d4
Parents: 02d04c4
Author: Sergio Fernández <wi...@apache.org>
Authored: Wed Dec 16 13:23:45 2015 +0100
Committer: Sergio Fernández <wi...@apache.org>
Committed: Wed Dec 16 13:23:45 2015 +0100
----------------------------------------------------------------------
libraries/ostrich/README.md | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/marmotta/blob/0eba24fc/libraries/ostrich/README.md
----------------------------------------------------------------------
diff --git a/libraries/ostrich/README.md b/libraries/ostrich/README.md
index b94fdf4..6352f21 100644
--- a/libraries/ostrich/README.md
+++ b/libraries/ostrich/README.md
@@ -54,10 +54,14 @@ The binary accepts many different options. Please see `--help` for details.
## Running Docker
-The C++ backend can be ran in the provided Docker image. Just build it:
+The C++ backend can be ran in the provided Docker image. You can build it:
docker build -t apachemarmotta/ostrich .
+Or fetch it [from Docker Hub](https://hub.docker.com/r/apachemarmotta/ostrich/):
+
+ docker pull apachemarmotta/ostrich
+
Then you can run Ostrich as a container:
docker run -t -d -p 10000:10000 apachemarmotta/ostrich
[3/3] marmotta git commit: MARMOTTA-621: latest details
Posted by wi...@apache.org.
MARMOTTA-621: latest details
Project: http://git-wip-us.apache.org/repos/asf/marmotta/repo
Commit: http://git-wip-us.apache.org/repos/asf/marmotta/commit/27889eb1
Tree: http://git-wip-us.apache.org/repos/asf/marmotta/tree/27889eb1
Diff: http://git-wip-us.apache.org/repos/asf/marmotta/diff/27889eb1
Branch: refs/heads/develop
Commit: 27889eb173b0bf2c76c6d65047e6f00f07ed8676
Parents: 0eba24f
Author: Sergio Fernández <wi...@apache.org>
Authored: Wed Dec 16 13:27:33 2015 +0100
Committer: Sergio Fernández <wi...@apache.org>
Committed: Wed Dec 16 13:27:33 2015 +0100
----------------------------------------------------------------------
libraries/ostrich/Dockerfile | 1 +
libraries/ostrich/README.md | 2 +-
2 files changed, 2 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/marmotta/blob/27889eb1/libraries/ostrich/Dockerfile
----------------------------------------------------------------------
diff --git a/libraries/ostrich/Dockerfile b/libraries/ostrich/Dockerfile
index 887684d..6211776 100644
--- a/libraries/ostrich/Dockerfile
+++ b/libraries/ostrich/Dockerfile
@@ -70,6 +70,7 @@ RUN apt-get -y clean -qq \
&& rm -rf /var/lib/apt/lists/* \
&& rm -rf /src
+WORKDIR ${OSTRICH_PATH}
EXPOSE 10000
CMD ${OSTRICH_PATH}/persistence/marmotta_persistence -db ${DB_PATH} -port ${DB_PORT}
http://git-wip-us.apache.org/repos/asf/marmotta/blob/27889eb1/libraries/ostrich/README.md
----------------------------------------------------------------------
diff --git a/libraries/ostrich/README.md b/libraries/ostrich/README.md
index 6352f21..62db012 100644
--- a/libraries/ostrich/README.md
+++ b/libraries/ostrich/README.md
@@ -78,7 +78,7 @@ on the same machine as follows:
./backend/build/persistence/marmotta_persistence -db /path/to/shard1 -port 10001
./backend/build/persistence/marmotta_persistence -db /path/to/shard2 -port 10002
- ./sharding/marmotta_sharding --port 10000 --backends localhost:10001,localhost:10002
+ ./backend/build/sharding/marmotta_sharding --port 10000 --backends localhost:10001,localhost:10002
You can then access the sharding server through Marmotta like the persistence server. Running all instances
on the same host is only useful for testing. In production environments, you would of course run all three