You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@ozone.apache.org by ms...@apache.org on 2020/04/24 05:23:08 UTC
[hadoop-ozone] branch master updated: HDDS-3352. Support for native
ozone filesystem client using libhdfs. (#782)
This is an automated email from the ASF dual-hosted git repository.
msingh pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git
The following commit(s) were added to refs/heads/master by this push:
new a53209e HDDS-3352. Support for native ozone filesystem client using libhdfs. (#782)
a53209e is described below
commit a53209e90ee821f342b0089a55b27de357fcf91e
Author: Aryan Gupta <44...@users.noreply.github.com>
AuthorDate: Fri Apr 24 10:52:59 2020 +0530
HDDS-3352. Support for native ozone filesystem client using libhdfs. (#782)
---
hadoop-ozone/native-client/README.md | 96 ++++++++++++++++++++++
.../native-client/libo3fs-examples/libo3fs_read.c | 65 +++++++++++++++
.../native-client/libo3fs-examples/libo3fs_write.c | 91 ++++++++++++++++++++
hadoop-ozone/native-client/libo3fs/o3fs.c | 66 +++++++++++++++
hadoop-ozone/native-client/libo3fs/o3fs.h | 44 ++++++++++
5 files changed, 362 insertions(+)
diff --git a/hadoop-ozone/native-client/README.md b/hadoop-ozone/native-client/README.md
new file mode 100644
index 0000000..04bb6fa
--- /dev/null
+++ b/hadoop-ozone/native-client/README.md
@@ -0,0 +1,96 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+#Overview
+
+libo3fs is a JNI based C API for Ozone File System. It provides with read and write functionality on OzoneFileSystem. It also uses some functions from HDFS(Hadoop Distributed File System) for which it uses libhdfs, which is a JNI based C API for Hadoop’s Distributed File System (HDFS). It provides C APIs to a subset of the HDFS APIs to manipulate HDFS files and the filesystem. libhdfs is part of the Hadoop distribution and comes pre-compiled in $HADOOP_HDFS_HOME/lib/native/libhdfs.so .
+
+#The APIs
+
+The libo3fs APIs are a subset of the Ozone FileSystem APIs. The header file for libo3fs describes each API in detail and is available in ${OZONE_HOME}/native-client/libo3fs/o3fs.h.
+
+#Requirements:
+
+1.Hadoop with compiled libhdfs.so
+2.Linux kernel > 2.6.9
+3.Compiled Ozone
+
+#Compilation
+
+ 1.Compilation of .c files
+ In libo3fs directory there is one file o3fs.c.
+
+ Execute the following command to compile it:
+
+ gcc -fPIC -pthread -I {OZONE_HOME}/native-client/libo3fs -I {HADOOP_HOME}/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/include -g -c o3fs.c
+ In the libo3fs-examples directory there are two .c files: libo3fs_read.c and libo3fs_write.c.
+
+ Execute the following command to compile libo3fs_read.c and libo3fs_write.c:
+
+ gcc -fPIC -pthread -I {OZONE_HOME}/native-client/libo3fs -I {HADOOP_HOME}/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/include -g -c libo3fs_read.c
+ gcc -fPIC -pthread -I {OZONE_HOME}/native-client/libo3fs -I {HADOOP_HOME}/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/include -g -c libo3fs_write.c
+
+ 2. Generation of libo3fs.so
+ Execute the following command to generate a .so:
+
+ gcc -shared o3fs.o hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/mybuild/hdfs.o -o libo3fs.so.
+
+ 3. Generation of binary(executable)
+ Two binaries have to be generated namely o3fs_read and o3fs_write.
+ Execute the following command to generate o3fs_red:
+
+ gcc -L {HADOOP_HOME}/hadoop-hdfs-project/hadoop-hdfs-native-client/target/native/target/usr/local/lib -o o3fs_read libo3fs_read.o -lhdfs -pthread -L/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.242.b08-0.el7_7.x86_64/jre/lib/amd64/server -ljvm -L {OZONE_HOME}/native-client/libo3fs -lo3fs
+
+ Execute the following command to execute o3fs_write:
+
+ gcc -L {HADOOP_HOME}/hadoop-hdfs-project/hadoop-hdfs-native-client/target/native/target/usr/local/lib -o o3fs_write libo3fs_write.o -lhdfs -pthread -L/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.242.b08- 0.el7_7.x86_64/jre/lib/amd64/server -ljvm -L {OZONE_HOME}/native-client/libo3fs -lo3fs
+
+#Deploying
+
+In root shell execute the following:
+
+ ./o3fs_write filename file_size buffer_size host_name port_number bucket_name volume_name
+
+For example
+
+ ./o3fs_write file1 100 100 127.0.0.1 9862 bucket4 vol4 , where file1 is name of the file, 100 is file size and buffer size, "127.0.0.1 is host", 9862 is the port number, "bucket4" is bucket name and "vol4" is the volume name.
+
+#Common Problems:
+
+CLASSPATH is not set. CLASSPATH can be set using the following command:
+
+ export CLASSPATH=$({OZONE_HOME}/hadoop-ozone/dist/target/ozone-0.5.0-SNAPSHOT/bin/ozone classpath hadoop-ozone-filesystem --glob)
+
+ export CLASSPATH=$CLASSPATH:{OZONE_HOME}/hadoop-ozone/dist/target/ozone-0.5.0-SNAPSHOT/share/ozone/lib/hadoop-ozone-filesystem-0.5.0-SNAPSHOT.jar
+
+LD_LIBRARY_PATH is not set. LD_LIBRARY_PATH can be set using the following command:
+
+ export LD_LIBRARY_PATH={HADOOP_HOME}/hadoop-hdfs-project/hadoop-hdfs-native-client/target/native/target/usr/local/lib:
+ /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.242.b08-0.el7_7.x86_64/jre/lib/amd64/server:
+ {OZONE_HOME}/native-client/libo3fs
+
+Ozone-site.xml is not configured. Save the minimal snippet to hadoop-ozone/dist/target/ozone-*/etc/hadoop/ozone-site.xml in the compiled distribution.
+
+ <configuration>
+ <properties>
+ <property><name>ozone.scm.datanode.id.dir</name><value>/tmp/ozone/data</value></property>
+ <property><name>ozone.replication</name><value>1</value></property>
+ <property><name>ozone.metadata.dirs</name><value>/tmp/ozone/data/metadata</value></property>
+ <property><name>ozone.scm.names</name><value>localhost</value></property>
+ <property><name>ozone.om.address</name><value>localhost</value></property>
+ </properties>
+ </configuration>
diff --git a/hadoop-ozone/native-client/libo3fs-examples/libo3fs_read.c b/hadoop-ozone/native-client/libo3fs-examples/libo3fs_read.c
new file mode 100644
index 0000000..5cf727a
--- /dev/null
+++ b/hadoop-ozone/native-client/libo3fs-examples/libo3fs_read.c
@@ -0,0 +1,65 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "o3fs.h"
+#include <stdio.h>
+#include <stdlib.h>
+#include<string.h>
+
+int main(int argc, char **argv) {
+ o3fsFS fs;
+ const char *rfile = argv[1];
+ tSize bufferSize = strtoul(argv[3], NULL, 10);
+ const char *host = argv[4];
+ tPort port = atoi(argv[5]);
+ const char *bucket = argv[6];
+ const char *volume = argv[7];
+ o3fsFile readFile;
+ char* buffer;
+ tSize curSize;
+ char message[110] = "Usage: o3fs_read <filename> <filesize> <buffersize>";
+ strcat(message, " <host-name> <port> <bucket-name> <volume-name>\n");
+ if (argc != 8) {
+ fprintf(stderr, message);
+ exit(-1);
+ }
+ fs = o3fsConnect(host, port, bucket, volume);
+ if (!fs) {
+ fprintf(stderr, "Oops! Failed to connect to o3fs!\n");
+ exit(-1);
+ }
+ readFile = o3fsOpenFile(fs, rfile, O_RDONLY, bufferSize, 0, 0);
+ if (!readFile) {
+ fprintf(stderr, "Failed to open %s for writing!\n", rfile);
+ exit(-2);
+ }
+ // data to be written to the file
+ buffer = malloc(sizeof(char) * bufferSize);
+ if(buffer == NULL) {
+ return -2;
+ }
+ // read from the file
+ curSize = bufferSize;
+ for (; curSize == bufferSize;) {
+ curSize = o3fsRead(fs, readFile, (void*)buffer, curSize);
+ }
+ free(buffer);
+ o3fsCloseFile(fs, readFile);
+ o3fsDisconnect(fs);
+ return 0;
+}
\ No newline at end of file
diff --git a/hadoop-ozone/native-client/libo3fs-examples/libo3fs_write.c b/hadoop-ozone/native-client/libo3fs-examples/libo3fs_write.c
new file mode 100644
index 0000000..30c8683
--- /dev/null
+++ b/hadoop-ozone/native-client/libo3fs-examples/libo3fs_write.c
@@ -0,0 +1,91 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "o3fs.h"
+#include <limits.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <sys/types.h>
+#include<string.h>
+
+int main(int argc, char **argv) {
+ o3fsFS fs;
+ const char *writeFileName = argv[1];
+ off_t fileTotalSize = strtoul(argv[2], NULL, 10);
+ long long tmpBufferSize = strtoul(argv[3], NULL, 10);
+ const char *host = argv[4];
+ tPort port = atoi(argv[5]);
+ const char *bucket = argv[6];
+ const char *volume = argv[7];
+ tSize bufferSize;
+ o3fsFile writeFile;
+ char* buffer;
+ int i;
+ off_t nrRemaining;
+ tSize curSize;
+ tSize written;
+ char message[110] = "Usage: o3fs_write <filename> <filesize> <buffersize>";
+ strcat(message, " <host-name> <port> <bucket-name> <volume-name>\n");
+ if (argc != 8) {
+ fprintf(stderr, message);
+ exit(-1);
+ }
+ fs = o3fsConnect(host, port, bucket, volume);
+ if (!fs) {
+ fprintf(stderr, "Oops! Failed to connect to o3fs!\n");
+ exit(-1);
+ }
+ if(fileTotalSize == ULONG_MAX && errno == ERANGE) {
+ fprintf(stderr, "invalid file size %s - must be <= %lu\n",
+ argv[2], ULONG_MAX);
+ exit(-3);
+ }
+ if(tmpBufferSize > INT_MAX) {
+ fprintf(stderr,
+ "invalid buffer size libhdfs API write chunks must be <= %d\n", INT_MAX);
+ exit(-3);
+ }
+ bufferSize = (tSize)tmpBufferSize;
+ writeFile = o3fsOpenFile(fs, writeFileName, O_WRONLY, bufferSize, 0, 0);
+ if (!writeFile) {
+ fprintf(stderr, "Failed to open %s for writing!\n", writeFileName);
+ exit(-2);
+ }
+ buffer = malloc(sizeof(char) * bufferSize);
+ if(buffer == NULL) {
+ fprintf(stderr, "Could not allocate buffer of size %d\n", bufferSize);
+ return -2;
+ }
+ for (i=0; i < bufferSize; ++i) {
+ buffer[i] = 'a' + (i%26);
+ }
+ for (nrRemaining = fileTotalSize; nrRemaining > 0;
+ nrRemaining -= bufferSize ) {
+ curSize = ( bufferSize < nrRemaining ) ? bufferSize : (tSize)nrRemaining;
+ if ((written = o3fsWrite(fs, writeFile, (void*)buffer,
+ curSize)) != curSize) {
+ fprintf(stderr, "ERROR: o3fsWrite returned an error on write: %d\n",
+ written);
+ exit(-3);
+ }
+ }
+ free(buffer);
+ o3fsCloseFile(fs, writeFile);
+ o3fsDisconnect(fs);
+ return 0;
+}
\ No newline at end of file
diff --git a/hadoop-ozone/native-client/libo3fs/o3fs.c b/hadoop-ozone/native-client/libo3fs/o3fs.c
new file mode 100644
index 0000000..421b7a4
--- /dev/null
+++ b/hadoop-ozone/native-client/libo3fs/o3fs.c
@@ -0,0 +1,66 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "o3fs.h"
+#include "hdfs/hdfs.h"
+#include <fcntl.h>
+#include <inttypes.h>
+#include <stdio.h>
+#include <string.h>
+#define O3FS "o3fs://"
+
+o3fsFS o3fsConnect(const char *host, tPort port,
+ const char *bucket, const char *vol)
+{
+ struct hdfsBuilder *bld = hdfsNewBuilder();
+ int len = 0;
+ if (!bld){
+ return NULL;
+ }
+ len = strlen(host) + strlen(bucket) + strlen(vol) + strlen(O3FS);
+ char string[len + 2];
+ snprintf(string, len + 3, "%s%s.%s.%s", O3FS, bucket, vol, host);
+ // After snprintf command,
+ // string = o3fs://bucket4.vol4.127.0.0.1 (URI without port)
+ //port will be added to URI in hdfsBuilerConnect() function below.
+ //finally URI: o3fs://bucket4.vol4.127.0.0.1:9862
+ hdfsBuilderSetNameNode(bld, string);
+ hdfsBuilderSetNameNodePort(bld, port);
+ return (o3fsFS)hdfsBuilderConnect(bld);
+}
+o3fsFile o3fsOpenFile(o3fsFS fs, const char *path, int flags, int bufferSize,
+ short replication, tSize blockSize){
+ return (o3fsFile)hdfsOpenFile((hdfsFS)fs, path, flags, bufferSize,
+ replication, blockSize);
+}
+
+tSize o3fsRead(o3fsFS fs, o3fsFile f, void* buffer, tSize length){
+ return hdfsRead((hdfsFS)fs, (hdfsFile)f, buffer, length);
+}
+
+int o3fsCloseFile(o3fsFS fs, o3fsFile file){
+ return hdfsCloseFile((hdfsFS)fs, (hdfsFile)file);
+}
+
+int o3fsDisconnect(o3fsFS fs){
+ return hdfsDisconnect((hdfsFS)fs);
+}
+
+tSize o3fsWrite(o3fsFS fs, o3fsFile f, const void* buffer, tSize length){
+ return hdfsWrite((hdfsFS)fs, (hdfsFile)f, buffer, length);
+}
diff --git a/hadoop-ozone/native-client/libo3fs/o3fs.h b/hadoop-ozone/native-client/libo3fs/o3fs.h
new file mode 100644
index 0000000..d908900
--- /dev/null
+++ b/hadoop-ozone/native-client/libo3fs/o3fs.h
@@ -0,0 +1,44 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#ifndef O3FS_DOT_H
+#define O3FS_DOT_H
+
+#include "hdfs/hdfs.h"
+
+struct hdfs_internal;
+typedef struct hdfs_internal* o3fsFS;
+
+struct hdfsFile_internal;
+typedef struct hdfsFile_internal* o3fsFile;
+
+o3fsFS o3fsConnect(const char* nn, tPort port, const char* bucket,
+ const char* volume);
+
+o3fsFile o3fsOpenFile(o3fsFS fs, const char *path, int flags, int bufferSize,
+ short replication, tSize blockSize);
+
+tSize o3fsRead(o3fsFS fs, o3fsFile f, void* buffer, tSize length);
+
+int o3fsCloseFile(o3fsFS fs, o3fsFile file);
+
+int o3fsDisconnect(o3fsFS fs);
+
+tSize o3fsWrite(o3fsFS fs, o3fsFile f, const void* buffer, tSize length);
+
+#endif
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-commits-help@hadoop.apache.org